- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether gradient-conflict metrics predict the understanding–generation trade-off in unified multimodal models. It introduces a synthetic grid-based testbed and compares several gradient-combination methods, data ratios, and random seeds. It also varies the strength of gradient projection to test whether reducing conflict improves performance. The results suggest that directional conflict metrics are weak predictors in this setting, while training loss and some representation-based measures show stronger associations.
Critical strengths
The paper asks an important diagnostic question. Gradient conflict is often used to motivate training decisions, so directly checking whether these measurements predict downstream performance is worthwhile.
The study goes beyond reporting a single correlation. It includes multiple optimization strategies, training seeds, progress-controlled analyses, and a projection intervention. Reporting unsuccessful methods and negative results also makes the evaluation informative.
Critical weaknesses
The testbed provides a limited test of the understanding–generation trade-off. In Table 1, generation accuracy is already around 99.6–100% for most methods, while other methods exhibit substantial generation failure. Much of the analysis therefore compares understanding performance under saturated generation, or distinguishes failed runs from successful ones. A harder setting where both tasks remain below saturation would provide stronger evidence about whether conflict predicts a genuine trade-off.
Some statistical conclusions are stronger than the reported evidence supports. For example, the early conflict-magnitude correlation has a confidence interval of [-0.463, 0.079], which still allows a moderately strong negative association. Also, the reported bootstrap interval of 0.729–1.041 compares signed correlations of 0.702 and -0.187, rather than their absolute predictive strengths. Finally, a negative correlation for cosine and positive correlations for conflict rate and removed energy are not inherently sign-inconsistent, since these metrics have opposite orientations. These issues weaken the claims about metric invalidity and the superiority of alternative predictors.
Questions for authors
Do the conclusions hold in a harder version of the testbed where neither understanding nor generation reaches saturation?
2 comments
rebuttal_AzNr - part 2
Q1 — A harder testbed where neither task saturates
Same commitment as above (E6): harder grammar and/or shortened budget, verified by pilot to yield (a) non-saturated generation in normally trained configurations and (b) measurable two-axis competition (raising one task's data share measurably lowers the other's accuracy); then the full correlation and intervention analyses are repeated and reported, whatever the outcome.
Summary of revisions
| Concern | Action |
|---|---|
| W1 | Within-mastering-regime numbers foregrounded; E6 committed |
| W2 | "Sign-inconsistent" wording corrected everywhere; orientation-corrected restatement (associations uniformly opposite to the premise) strengthens the thesis |
| Q1 | E6 harder regime with two-axis competition; results reported either way |
rebuttal_AzNr - part 1
Response to Reviewer AzNr (Rating 3, Confidence 4)
We thank Reviewer AzNr for both the substantive recognition (the diagnostic question matters; the audit goes beyond a single correlation) and for a methodological catch that improves the paper: our "mutually sign-inconsistent" wording is wrong, and correcting it makes the paper's argument stronger. We answer the two weaknesses and the question below.
Bottom line. (1) The concern that the analysis is driven by saturated-vs.-failed contrast is answerable from the paper's within-regime analysis — the conclusion holds among the 47 configurations that master generation — though we agree a fully unsaturated regime (committed as E6) is the decisive version. (2) On the sign-inconsistency point the reviewer is right, and we restate the result correctly: the concurrent associations are internally consistent once metric orientations are accounted for — and they consistently point opposite to the direction the field's premise predicts, before dissolving under the regime cut.
W1 — Saturated generation limits the test; analysis driven by the saturated-vs.-failed contrast
The within-regime analysis is already in the paper and does not rest on that contrast. Among the 47 configurations that master generation (135 measured checkpoints):
The failed-generation methods (SOAP-style at 0.000, token-normalized at 0.607 ± 0.307, MGDA at understanding 0.389) are analyzed as pathologies with their own diagnostic value — e.g., only the norm ratio flags the SOAP-style collapse — not as the source of the correlation result.
[New Experiment] E6. We nevertheless agree the decisive test is a regime where neither task saturates and both compete; we commit to the harder-grammar/shorter-budget variant with the full E1 correlation sweep and E3 intervention repeated (design detailed in our response to Reviewer eBTo; ≈12 min per run on the original hardware).
W2 — "Sign-inconsistent" is the wrong description of the concurrent associations
The reviewer is correct, and we thank them for the catch. The concurrent configuration-level associations are global cosine −0.32 [−0.52,−0.10], conflict rate +0.34 [0.13,0.54], removed energy +0.37 [0.16,0.56]. Because cosine is oriented positively with agreement while conflict rate and removed energy are oriented positively with conflict, a naive sign comparison is meaningless. Orientation-corrected, the three metrics agree: all three indicate that configurations with more measured conflict land at higher Pareto scores. The corrected statement replaces "mutually sign-inconsistent" with:
This is a sharper result than the wording the reviewer criticized: the metrics do not merely fail to predict the trade-off; where they do associate, the association runs against the premise's predicted sign. We will make this correction in the abstract, Sec. 4.2, and the discussion.
We also note the same correction applies inside the within-regime cut (the four small associations listed under W1 are likewise mutually consistent, all pointing the same "wrong" way), and the revision will use the corrected orientation-consistent description there as well.