acceptodds
Withdrawn from ICLR 2027

MOIRE: Objective-Specific Adaptation and Residual Readout for Multi-Objective Self-Supervised Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-objective training in self-supervised learning (SSL) combines several pre-training objectives to learn representations for downstream tasks. However, training these objectives together requires coordinating their losses through a shared backbone, where updates for one objective can impair predictions for others. We therefore introduce two complementary diagnostic protocols to quantify cross-objective damage under isolated and coordinated updates, and use them to examine existing loss-weighting and gradient-coordination methods from multi-task learning. Our measurements show that these methods only modestly reduce cross-objective damage compared with a fixed-weight sum of losses. Moreover, even effective coordination of self-supervised objectives need not improve downstream performance, because pretraining objectives may emphasize different information from what downstream tasks and evaluation metrics reward. To address both challenges, we propose MOIRE (Multi-Objective Interaction and Residual Evidence). MOIRE gives each objective a separate adaptation path, reducing interference with the shared representation. It retains the resulting objective-specific changes as residual evidence. This evidence can then be combined with the shared representation to better align the learned representation with downstream tasks. Mechanistic experiments show that MOIRE has the lowest relative cross-objective damage among the compared methods. Its downstream performance is similarly strong: across eight datasets spanning graph representation learning and sequential recommendation, it achieves the best average rank in both domains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, multi-objective learning, multi-task learning, cross-objective interference, graph representation learning, sequential recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aWGeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multi-objective SSL where different pretraining losses share one backbone. The authors propose protocols that measure how much one objectives affects the other in frozen readouts and ultimately, propose MOIRE that gives each objective a small adapter on top of the share representation. During downstream eval, the method uses the validation data to finalize combination.

Critical strengths
  • I think the most useful aspect in this paper is the directed damage diagnostic. This is different from a gradient-conflict view since it tries to ask whether training one objective hurts the learned knowledge of others. Seems useful even as an interpretability tool.

  • The paper covers multiple domains and also reports when the results do not favor the method, which I appreciate since it gives clarity when it works and not.

Critical weaknesses
  • I do not think the evidence shows that MOIRE is cleanly better than the baselines. At downstream time, MOIRE seems to select residual gates or readout methods separately for downstream metrics, has up to 6x representation capacity, and the residual stage incurs additional tuning cost. I believe there are confounders that make the comparison not clear. In fact, the authors' own ablation shows the impact without the readout, MOIRE's graph rank drops from 2.78 to 4.83, leaving open questions on evaluation fairness.

  • Also, the design overlaps with architectural multi-task work that is not discussed such as per-task adapters, including but not limited to AdapterFusion. MOIRE seems to be more of an architectural change but is compared only against optimizers, so I cannot tell how much of the gain comes from its specific design.

  • The damage audit only measures how fixed readouts of the shared backbone representation change. However, MOIRE is designed to keep objective-specific changes in separate adaptation paths and these paths are bypassed during source-only continuation. The damage metric is also normalized by the target's initial loss (not by how much the source objective improves), so a method making smaller updates could appear less damaging. Overall, some design choices make the mechanistic evidence less conclusive.

  • Also the figures are too small and very hard to read...

Questions for authors

Q1. How much of MOIRE's advantage remains when all methods are compared under matched downstream conditions, meaning the same representation width, parameter count and selection budget, and either per-metric readout selection for every method or a single readout configuration per dataset for MOIRE? The backbone-only ablation is encouraging, but a fully matched comparison would clarify how much of the full-model gain comes from the method itself.

Q2. How does MOIRE compare with architectural multi-task baselines at matched parameter count, such as per-task adapters with AdapterFusion-style combination, or MMoE (Multi-gate Mixture-of-Experts) and PLE (Progressive Layered Extraction)? How do the authors position MOIRE relative to this line of work?

Q3. Since MOIRE's adaptation paths are bypassed during source-only continuation, could the authors report, for each method, the initial audit loss and how much the source objective's own loss improved during the audited updates? Presenting damage relative to that improvement would show whether MOIRE's lower damage reflects better coordination or simply weaker readouts or smaller updates.

Reviewer_KLPPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MOIRE, a framework for multi-objective self-supervised learning. The method uses objective-specific adapters to preserve information learned from different objectives. The experiments cover eight benchmarks across graph representation learning and compare against existing multi-task learning baselines.

Critical strengths

The paper presents a clearn motivation and illustrate the motivation concretely.

The experments cover a wide spectrum of datasets, covering multiple existing powerful baselines.

Critical weaknesses

I am not fully convinced by the technical novelty. Many components are based on existing works.

It is unclear whether the gain comes from the framework design or the increased parameters.

Questions for authors

What is the main novelty of MOIRE compared with existing methods?

Can the authors compare MOIRE with baselines of similar model size to rule out the effect of additional parameters?

Reviewer_BQ9gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates two challenges in multi-objective self-supervised learning (SSL): cross-objective interference arising from updates to a shared backbone and the potential mismatch between pretraining objectives and downstream evaluation criteria. To characterize cross-objective interference, the authors employ two diagnostic protocols that measure how updates associated with one objective affect the frozen predictive readouts of others under isolated and coordinated training. Their findings suggest that existing loss-weighting and gradient-coordination methods provide only modest reductions in cross-objective damage compared with linear scalarization. To address these challenges, the paper proposes MOIRE, a framework that introduces objective-specific adaptation paths to reduce interference with the shared representation, along with directed interactions that allow information exchange across objectives. MOIRE further preserves objective-specific representation changes as residual evidence, which can be selectively combined with the shared representation to better align with downstream tasks. Mechanistic experiments demonstrate reduced cross-objective damage, while downstream evaluations across eight datasets spanning graph representation learning and sequential recommendation show that MOIRE achieves the best average rank in both domains among the compared methods.

Critical strengths
  1. The paper introduces two diagnostic protocols to quantify cross-objective damage under isolated and coordinated updates.

  2. MOIRE provides a framework that combines objective-specific adaptation, cross-objective interaction, and residual readout to address interference and downstream alignment.

  3. The paper presents extensive experiments across eight datasets spanning graph representation learning and sequential recommendation, demonstrating consistent improvements over a wide range of MTL baselines.

Critical weaknesses
  1. The paper primarily compares MOIRE with loss-weighting and gradient-coordination methods, while existing approaches such as BERT and PALs [1], Cross-Stitch Networks [2], and AdapterFusion [3] have already explored task-specific adaptation and representation fusion. In particular, AdapterFusion [3] similarly preserves task-specific knowledge through separate adapters and combines it for downstream tasks. The lack of discussion of these closely related approaches, as well as comparisons with relevant task-specific adaptation or representation-sharing baselines, makes it difficult to assess the novelty and significance of the proposed framework.

  2. The effectiveness of directed cross-objective interaction is not sufficiently demonstrated. In Appendix A.4.2, removing this component improves all five reported backbone metrics on Actor, while the results on other datasets are mixed, raising questions about its necessity.

  3. The proposed cross-objective damage measurements rely on frozen target-specific readouts. However, increased frozen-readout loss may reflect changes in representation geometry rather than an actual loss of recoverable information. It remains unclear how well these measurements capture meaningful representation interference or correlate with downstream performance.

  4. The theoretical analysis provides a local feasibility argument showing that additional adaptation directions can reduce the minimum attainable shared modification cost. Although the analysis also gives conditional bounds on directed damage, it does not establish that practical optimization will achieve reduced interference or improved downstream performance.

References:

[1] Stickland, Asa Cooper, and Iain Murray. "Bert and pals: Projected attention layers for efficient adaptation in multi-task learning." International conference on machine learning. PMLR, 2019.

[2] Misra, Ishan, et al. "Cross-stitch networks for multi-task learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[3] Pfeiffer, Jonas, et al. "Adapterfusion: Non-destructive task composition for transfer learning." Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume. 2021.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.