- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies multi-objective SSL where different pretraining losses share one backbone. The authors propose protocols that measure how much one objectives affects the other in frozen readouts and ultimately, propose MOIRE that gives each objective a small adapter on top of the share representation. During downstream eval, the method uses the validation data to finalize combination.
Critical strengths
-
I think the most useful aspect in this paper is the directed damage diagnostic. This is different from a gradient-conflict view since it tries to ask whether training one objective hurts the learned knowledge of others. Seems useful even as an interpretability tool.
-
The paper covers multiple domains and also reports when the results do not favor the method, which I appreciate since it gives clarity when it works and not.
Critical weaknesses
-
I do not think the evidence shows that MOIRE is cleanly better than the baselines. At downstream time, MOIRE seems to select residual gates or readout methods separately for downstream metrics, has up to 6x representation capacity, and the residual stage incurs additional tuning cost. I believe there are confounders that make the comparison not clear. In fact, the authors' own ablation shows the impact without the readout, MOIRE's graph rank drops from 2.78 to 4.83, leaving open questions on evaluation fairness.
-
Also, the design overlaps with architectural multi-task work that is not discussed such as per-task adapters, including but not limited to AdapterFusion. MOIRE seems to be more of an architectural change but is compared only against optimizers, so I cannot tell how much of the gain comes from its specific design.
-
The damage audit only measures how fixed readouts of the shared backbone representation change. However, MOIRE is designed to keep objective-specific changes in separate adaptation paths and these paths are bypassed during source-only continuation. The damage metric is also normalized by the target's initial loss (not by how much the source objective improves), so a method making smaller updates could appear less damaging. Overall, some design choices make the mechanistic evidence less conclusive.
-
Also the figures are too small and very hard to read...
Questions for authors
Q1. How much of MOIRE's advantage remains when all methods are compared under matched downstream conditions, meaning the same representation width, parameter count and selection budget, and either per-metric readout selection for every method or a single readout configuration per dataset for MOIRE? The backbone-only ablation is encouraging, but a fully matched comparison would clarify how much of the full-model gain comes from the method itself.
Q2. How does MOIRE compare with architectural multi-task baselines at matched parameter count, such as per-task adapters with AdapterFusion-style combination, or MMoE (Multi-gate Mixture-of-Experts) and PLE (Progressive Layered Extraction)? How do the authors position MOIRE relative to this line of work?
Q3. Since MOIRE's adaptation paths are bypassed during source-only continuation, could the authors report, for each method, the initial audit loss and how much the source objective's own loss improved during the audited updates? Presenting damage relative to that improvement would show whether MOIRE's lower damage reflects better coordination or simply weaker readouts or smaller updates.