acceptodds
Under review as a conference paper at ICLR 2027

Does Gradient Conflict Predict the Understanding–Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) are increasingly designed and optimized around the notion of gradient conflict between the understanding and generation objectives, measured as cosine similarities between task gradients or conflict rates across layers. The premise that reducing these metrics improves the downstream understanding–generation trade-off has never been tested directly. We audit it in a controlled testbed, GRIDUMM, which mirrors the structural ingredients of UMM training—a fully shared trunk serving two objectives with asymmetric token budgets and task difficulty—while making the ground-truth trade-off exactly computable. Across 63 configurations (7 gradient-combination strategies × 3 data-mixing ratios × 3 seeds) and 372 measured checkpoints, no directional conflict metric—global or per-layer cosine, conflict rate, removed energy—reaches |ρ| ≥ 0.3 with a confidence interval excluding zero for conflict measured during training against the eventual trade-off (strongest |ρ| = 0.145, interval crossing zero), and the larger concurrent configuration-level associations are mutually sign-inconsistent. A dose–response intervention that monotonically suppresses conflict (level α) leaves the trade-off flat, separating correlation from causation. The one gradient-geometric quantity with association, the norm ratio, is a generation-failure detector, null among configurations that master generation. The functional interference measure eff_rank leads every directional conflict metric (Δρ CI 0.729–1.041) and retains its advantage inside that regime. Training loss, the outcome most often reported in place of benchmarks, tracks the tradeoff strongly—the failure is specific to gradient-conflict geometry. Our results do not show that conflict is useless; they show that its validity as a diagnostic target must be established, not assumed, and we release the audit protocol as a reusable standard.

Primary area:
optimization
Keywords:
Unified Multimodal Models, Gradient Conflict, Multi-Task Learning, Gradient Geometry, Functional Interference, Understanding–Generation Trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AzNrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether gradient-conflict metrics predict the understanding–generation trade-off in unified multimodal models. It introduces a synthetic grid-based testbed and compares several gradient-combination methods, data ratios, and random seeds. It also varies the strength of gradient projection to test whether reducing conflict improves performance. The results suggest that directional conflict metrics are weak predictors in this setting, while training loss and some representation-based measures show stronger associations.

Critical strengths

The paper asks an important diagnostic question. Gradient conflict is often used to motivate training decisions, so directly checking whether these measurements predict downstream performance is worthwhile.

The study goes beyond reporting a single correlation. It includes multiple optimization strategies, training seeds, progress-controlled analyses, and a projection intervention. Reporting unsuccessful methods and negative results also makes the evaluation informative.

Critical weaknesses

The testbed provides a limited test of the understanding–generation trade-off. In Table 1, generation accuracy is already around 99.6–100% for most methods, while other methods exhibit substantial generation failure. Much of the analysis therefore compares understanding performance under saturated generation, or distinguishes failed runs from successful ones. A harder setting where both tasks remain below saturation would provide stronger evidence about whether conflict predicts a genuine trade-off.

Some statistical conclusions are stronger than the reported evidence supports. For example, the early conflict-magnitude correlation has a confidence interval of [-0.463, 0.079], which still allows a moderately strong negative association. Also, the reported bootstrap interval of 0.729–1.041 compares signed correlations of 0.702 and -0.187, rather than their absolute predictive strengths. Finally, a negative correlation for cosine and positive correlations for conflict rate and removed energy are not inherently sign-inconsistent, since these metrics have opposite orientations. These issues weaken the claims about metric invalidity and the superiority of alternative predictors.

Questions for authors

Do the conclusions hold in a harder version of the testbed where neither understanding nor generation reaches saturation?

2 comments
AuthorsOpenReview ↗

rebuttal_AzNr - part 1

Response to Reviewer AzNr (Rating 3, Confidence 4)

We thank Reviewer AzNr for both the substantive recognition (the diagnostic question matters; the audit goes beyond a single correlation) and for a methodological catch that improves the paper: our "mutually sign-inconsistent" wording is wrong, and correcting it makes the paper's argument stronger. We answer the two weaknesses and the question below.

Bottom line. (1) The concern that the analysis is driven by saturated-vs.-failed contrast is answerable from the paper's within-regime analysis — the conclusion holds among the 47 configurations that master generation — though we agree a fully unsaturated regime (committed as E6) is the decisive version. (2) On the sign-inconsistency point the reviewer is right, and we restate the result correctly: the concurrent associations are internally consistent once metric orientations are accounted for — and they consistently point opposite to the direction the field's premise predicts, before dissolving under the regime cut.


W1 — Saturated generation limits the test; analysis driven by the saturated-vs.-failed contrast

The within-regime analysis is already in the paper and does not rest on that contrast. Among the 47 configurations that master generation (135 measured checkpoints):

  • every directional conflict metric stays weak — global cosine −0.211-0.211 [−0.347,−0.060][-0.347, -0.060], conflict rate +0.230+0.230 [0.062,0.390][0.062, 0.390], conflict magnitude +0.168+0.168 [0.027,0.299][0.027, 0.299], removed energy +0.229+0.229 [0.062,0.394][0.062, 0.394] — none reaches ∣ρ∣≥0.3|\rho| \geq 0.3;
  • the norm ratio, the one gradient-geometric quantity with a strong full-population association (−0.651-0.651 at the checkpoint level, −0.796-0.796 against generation accuracy specifically), drops to null (+0.02+0.02, CI [−0.17,0.18][-0.17, 0.18]) — confirming its role is failure detection, not trade-off prediction;
  • the functional measures stay strong where the field would deploy them (best functional predictor ρ=0.854\rho = 0.854 [0.795,0.903][0.795, 0.903] vs. best directional 0.2300.230; paired Δρ\Delta\rho CI [0.47,0.78][0.47, 0.78]).

The failed-generation methods (SOAP-style at 0.000, token-normalized at 0.607 ±\pm 0.307, MGDA at understanding 0.389) are analyzed as pathologies with their own diagnostic value — e.g., only the norm ratio flags the SOAP-style collapse — not as the source of the correlation result.

[New Experiment] E6. We nevertheless agree the decisive test is a regime where neither task saturates and both compete; we commit to the harder-grammar/shorter-budget variant with the full E1 correlation sweep and E3 intervention repeated (design detailed in our response to Reviewer eBTo; ≈12 min per run on the original hardware).

W2 — "Sign-inconsistent" is the wrong description of the concurrent associations

The reviewer is correct, and we thank them for the catch. The concurrent configuration-level associations are global cosine −0.32-0.32 [−0.52,−0.10][-0.52, -0.10], conflict rate +0.34+0.34 [0.13,0.54][0.13, 0.54], removed energy +0.37+0.37 [0.16,0.56][0.16, 0.56]. Because cosine is oriented positively with agreement while conflict rate and removed energy are oriented positively with conflict, a naive sign comparison is meaningless. Orientation-corrected, the three metrics agree: all three indicate that configurations with more measured conflict land at higher Pareto scores. The corrected statement replaces "mutually sign-inconsistent" with:

the concurrent associations are moderate but uniformly oriented opposite to the field's premise — the metrics, where they associate at all, point toward more conflict predicting a better trade-off — and they do not survive the regime cut (norm ratio null within mastering configurations; directional metrics weak throughout).

This is a sharper result than the wording the reviewer criticized: the metrics do not merely fail to predict the trade-off; where they do associate, the association runs against the premise's predicted sign. We will make this correction in the abstract, Sec. 4.2, and the discussion.

We also note the same correction applies inside the within-regime cut (the four small associations listed under W1 are likewise mutually consistent, all pointing the same "wrong" way), and the revision will use the corrected orientation-consistent description there as well.

AuthorsOpenReview ↗

rebuttal_AzNr - part 2

Q1 — A harder testbed where neither task saturates

Same commitment as above (E6): harder grammar and/or shortened budget, verified by pilot to yield (a) non-saturated generation in normally trained configurations and (b) measurable two-axis competition (raising one task's data share measurably lowers the other's accuracy); then the full correlation and intervention analyses are repeated and reported, whatever the outcome.


Summary of revisions

ConcernAction
W1Within-mastering-regime numbers foregrounded; E6 committed
W2"Sign-inconsistent" wording corrected everywhere; orientation-corrected restatement (associations uniformly opposite to the premise) strengthens the thesis
Q1E6 harder regime with two-axis competition; results reported either way
Reviewer_eBToOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether gradient-conflict metrics predict the understanding–generation trade-off in unified models. It uses a synthetic grid task to compare gradient-combination methods, measure correlations with downstream performance, and test an intervention that progressively suppresses gradient conflict.

Critical strengths

Questioning the validity of gradient-conflict metrics is worthwhile. The paper goes beyond observational correlations by including a controlled intervention and reports uncertainty estimates for its main analyses.

Critical weaknesses
  1. The testbed does not establish a substantial understanding–generation trade-off in the first place. Generation accuracy is nearly saturated in most normally trained configurations, so the joint score largely reflects understanding accuracy. Configurations with failed generation do not necessarily demonstrate a trade-off either: the SOAP-style variant performs worse on both tasks than naive training. Consequently, the results do not distinguish an uninformative conflict metric from a setting with little meaningful task competition to explain.

  2. This limits the interpretation of the intervention. Suppressing conflict without improving performance is informative only within the tested regime; it does not establish that conflict reduction is ineffective when both tasks remain challenging and compete for shared capacity. All experiments use the same synthetic grammar with small models, and the second model scale does not address this limitation. Although the paper acknowledges these constraints, its broader framing is not adequately supported by the experiments.

Questions for authors
  1. Can the authors demonstrate a regime where neither task is saturated and improving one task measurably compromises the other, then repeat the correlation and intervention analyses?

  2. How do the jointly trained models compare with task-specific models under matched capacity and training budgets? Is there measurable negative transfer for the conflict metrics to explain?

2 comments
AuthorsOpenReview ↗

rebuttal_eBTo - part 1

Response to Reviewer eBTo (Rating 2, Confidence 3)

We thank Reviewer eBTo for a review that pinpoints the two deepest issues in the design — that the testbed must first establish a trade-off worth explaining, and that a null intervention is only interpretable inside the regime it was run in. We answer both with existing numbers where they exist and with a committed new experiment where they do not.

Bottom line. The population does contain a trade-off signal to explain — understanding accuracy varies 0.389–0.641 across the 47 configurations that master generation, and the paper's within-regime analysis (135 checkpoints) is exactly the "competition without failure" cut the reviewer asks for — but we agree the strongest version of the audit needs a regime where both tasks remain unsaturated and compete, and we commit to building and re-running the full audit (correlation + intervention) in it.


W1 — The testbed does not establish a substantial trade-off

What the existing analyses already establish. The audit's target is the premise's operational content: a practitioner measures conflict during training and forecasts where the configuration will land. For that question the relevant variation exists and is not confined to failed runs:

  1. Joint scores span 0.271–0.820; among the 47 configurations that master generation (generation object-cell accuracy ≈ 1.0), understanding accuracy still spans 0.389–0.641 (mean ≈ 0.63) — genuine, non-saturated outcome variation. By measurement the generation axis carries almost no variance among healthy methods (s.d. ≈ 0.004): the trade-off variation lives on the understanding axis, and that is the axis the audit predicts.
  2. The within-mastering-regime analysis is reported in the paper (135 checkpoints): directional conflict metrics stay weak (largest ∣ρ∣=0.23|\rho| = 0.23), the norm ratio goes null (+0.02+0.02, CI [−0.17,0.18][-0.17, 0.18]), and functional measures retain their advantage (best functional predictor ρ=0.854\rho = 0.854 [0.795,0.903][0.795, 0.903] vs. best directional 0.2300.230; paired Δρ\Delta\rho CI [0.47,0.78][0.47, 0.78]). So the conclusion does not hinge on contrasting saturated against collapsed runs.
  3. On the SOAP-style variant being "worse on both tasks": we agree it does not instantiate a trade-off, and the paper does not use it as one — it is analyzed as a pathology whose invisibility to the directional metrics is the finding (conflict statistics inside the healthy range while generation is 0.000).

What we concede. A regime with genuine two-axis competition would make the design conclusively answer the reviewer's alternative hypothesis — that conflict metrics are uninformative only where there is little competition to explain. We commit to exactly that experiment (Q1).

W2 — The intervention's interpretation is limited to the tested regime

We accept this and will bound the claim in text. The causal statement the paper defends is deliberately narrow and we will scope it accordingly: within this regime, an intervention that monotonically suppresses the measured conflict to zero — while leaving data, schedule, and parameters otherwise fixed — moves the joint score by only −0.002-0.002 by seed (understanding −0.004-0.004, generation object-cell +0.001+0.001) against a seed-to-seed spread of 0.0370.037, with a total dose-response range of 0.005 against a mean per-dose s.d. of 0.028. What this rules out is the strongest rescue of the premise in this regime ("the correlation is diluted but causally real"); what it does not establish is ineffectiveness in regimes with active capacity competition. The revision will state both sentences explicitly, and the second one is testable with the committed experiment below rather than left as a caveat.

AuthorsOpenReview ↗

rebuttal_eBTo - part 2

Q1 — A regime where neither task saturates and improving one measurably compromises the other

[New Experiment] E6 — harder regime, full re-audit. Design:

  1. Regime construction: harder grammar (more objects per scene, longer randomized descriptions, compositional question types) and/or a shortened budget, with a pilot pass verifying that (a) normally trained configurations do not reach ceiling generation by the budget's end, and (b) increasing the understanding data share measurably lowers generation (and vice versa) — i.e., the two axes actively compete.
  2. Full re-audit: E1 correlation sweep (3×3×3=273 \times 3 \times 3 = 27 configurations) and the E3 dose-response intervention, same fixed probe batch and evaluation protocol.
  3. Interpretation pre-registered: if the directional metrics gain predictive power in this regime, the paper's conclusion becomes regime-conditional in a precise, useful way; if they remain null while a real trade-off exists, the validity critique is strengthened. Either outcome is informative and will be reported.

Each run costs ≈12 minutes on the original hardware (the paper's 93 runs took 18.8 aggregate hours), so the full E6 sweep is roughly 6–8 run-hours plus the pilot.

Q2 — Jointly trained vs. task-specific models under matched capacity and budget

[New Experiment] E7 — negative-transfer measurement. Train task-specific single-task models (understanding-only; generation-only) at matched capacity and budget, 3 seeds each, and measure the transfer gap Δtask=single−joint\Delta_{\text{task}} = \text{single} - \text{joint} per axis. Then test the reviewer's exact question: do the conflict metrics predict the transfer gap across configurations? This is the sharpest possible test of the premise — if conflict geometry explains interference, it should track how much joint training costs each task relative to its dedicated baseline. We will report the gap table and its correlation with every conflict and functional metric.


Summary of revisions

ConcernAction
W1Within-mastering numbers foregrounded; scope of the saturation caveat sharpened
W2Causal claim explicitly bounded to the tested regime
Q1E6: harder regime, both axes unsaturated; E1 + E3 fully repeated
Q2E7: task-specific baselines, negative-transfer gap vs. conflict metrics
Reviewer_8bavOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether gradient-conflict metrics in unified multimodal models predict the final understanding-generation trade-off. Using a synthetic testbed (GridUMM: a 3.2M-parameter shared transformer that generates and answers questions about 5x5 color grids), the authors train 63 configurations, run a dose-response intervention, and report that conflict metrics correlate weakly with the outcome, that suppressing conflict changes nothing, and that training loss and representation-based measures predict much better.

Critical strengths
  • The question is worth asking; conflict is widely used as motivation without validation.
  • The audit design (correlation, intervention, reliability, power) is reasonable and clearly reported.
Critical weaknesses
  1. The testbed lacks the mechanism under study. Generation here is filling 25 fixed slots with one of 8 colors, fully determined by the description. This is closer to text-to-table generation than to image generation. Both tasks use the same symbolic representation, so the usual suspected source of conflict, namely semantic abstraction vs. low-level detail, is absent. A null result here says little about real UMMs.
  2. There is little conflict or trade-off to explain. Cosines are ~0.00x and generation saturates near 1.0, so most variance comes from a few failing methods (SOAP-style at 0.000 generation in all runs, MGDA, token-normalized). The claimed match to production gradient imbalance also seems reversed: ∥∥gg∥∥<<∥∥gu∥∥\|\|g_g\|\|<< \|\|g_u\|\| here.
  3. The conflict metrics are weak. Conflict rate over 8 near-zero cosines is close to a coin flip (median ~0.5 in Fig. 6). Group-level cosines on raw gradients ignore AdamW and can hide localized conflict. A functional measure, such as lookahead loss change, would be more meaningful. At most, the results show these metrics are poor, not that interference is unimportant.
  4. E3 does not support a causal claim. The projection also removes aligned components. Raw conflict does not drop (Fig. 7), so the manipulation check is tautological. The removed component is roughly 1% of the update or less, and 3 seeds per dose only rule out large effects.
  5. The conclusions overreach. There is no experiment on a real UMM, and the limitations section does not address the mechanism mismatch. There is also no new method, so the contribution rests on this limited finding.
  6. Several numbers are inconsistent across text, tables, and figures. Examples: the abstract gives ∥ρ∥=0.145\|\rho \|= 0.145 vs. 0.198 in Table 5; the shallow- vs. depth-weighted values in Sec. 4.2 do not match Table 5; the “∥ρ∥\|\rho \| gap 0.51” in Sec. 4.5 does not match its CI; the Fig. 5 caption and the text disagree.
Questions for authors
  • Can the testbed be modified so that the two tasks need different representations?
  • Is the SOAP-style collapse caused by the modified implementation? How do the results change without the failing methods?
3 comments
AuthorsOpenReview ↗

rebuttal_8bav - part 1

Response to Reviewer 8bav (Rating 1, Confidence 3)

We thank Reviewer 8bav for the most demanding review, and we respond to it in the spirit it was written: several criticisms are correct, two are correctable presentation errors that we have now located precisely, and the central scientific concerns are answerable with a combination of existing analyses, reanalyses we commit to, and new experiment designs. All numbers below are from the submission's own tables and figures.

Bottom line. (1) The number inconsistencies the reviewer lists are real and we fix each one explicitly (W6). (2) The claim that the concurrent associations are "mutually sign-inconsistent" is incorrectly worded — Reviewer AzNr independently makes the same point — and the corrected statement is stronger than the original. (3) The mechanism-mismatch concern (W1) is the honest boundary of a controlled testbed; we commit to the heterogeneous-representation variant (Q1) that introduces the missing axis. (4) The "results without failing methods" reanalysis the reviewer requests (Q2) substantially exists in the paper's healthy-regime cut, and we commit to the explicit three-method-excluded table.


W1 — The testbed lacks the mechanism under study

We accept the factual premise and sharpen the claim's scope. In GRIDUMM both tasks operate on the same interleaved marker/color token stream, so the semantic-abstraction-vs.-low-level-detail axis that motivates conflict claims in production UMMs is indeed absent. What the testbed does establish is narrower and still useful:

  1. It implements the strongest form of the design point the premise is made on — every parameter shared by both tasks, asymmetric token budgets (26 generation target tokens vs. 1 answer token), asymmetric task difficulty.
  2. It reproduces the conflict phenomenology the field's premise operates on: conflict concentrated in the learning phase, order-of-magnitude task-norm imbalance (converged norm ratios 0.004–0.476 across methods), and the pathologies the balancing literature targets (MGDA collapsing onto the minority-gradient task at 0.389 understanding / 1.000 generation; SOAP-style collapse to 0.000 generation with conflict statistics inside the healthy range — cosine 0.002, rate 0.389 vs. naive 0.001/0.417).
  3. The audit's conclusion is therefore conditional and stated as such: in this controlled instance of the design point, the directional metrics the field keys on carry no valid signal — so their validity must be established per setting, not assumed. That is the paper's actual thesis, and we will reword the abstract and limitations so no reader can mistake the null for an unconditional claim about production UMMs.

[New Experiment] Q1 — heterogeneous-representation variant (E8). Keep the trunk, data, and protocol fixed; change only the generation output side to a different representation space (the grid rendered as a low-resolution image target through a separate output head, with the description still symbolic). This introduces exactly the abstraction-vs.-surface axis the reviewer identifies while preserving the controlled ground truth. We will repeat E1's correlation sweep and report whether the directional metrics gain signal when the mechanism is present — a result informative in either direction.

W2 — Little conflict or trade-off to explain; imbalance direction "reversed"

On degeneracy. The reviewer's observation is the finding, not a confound, and the degeneracy is measured, not asserted. The author-measured phenomenology of the easy regime: learning-phase mean global cosine ≈0.00\approx 0.00 and converged cosine ≈0.00\approx 0.00; conflict-rate median ≈0.52\approx 0.52; generation accuracy varies by only ≈0.004\approx 0.004 across healthy methods. At the predictive level every directional metric's confidence interval crosses zero (global cosine −0.146-0.146 [−0.411,0.147][-0.411, 0.147]; conflict rate +0.014+0.014 [−0.237,0.258][-0.237, 0.258]; conflict magnitude −0.198-0.198 [−0.463,0.079][-0.463, 0.079]; removed energy +0.041+0.041 [−0.204,0.291][-0.204, 0.291]), and the method-level cosines span −0.003-0.003 to 0.1230.123 — the metrics are near-degenerate exactly where the field reads them. The failing methods are not excluded from the analysis; they are part of what the metrics fail to see (the worst method, SOAP-style, is invisible to every directional metric — only the norm ratio, 0.476 vs. naive 0.025, flags it).

On the imbalance direction. The paper states (Sec. 2.3) that generation has the larger token budget but the smaller gradient norm at convergence, "reproducing the imbalance structure reported on production UMMs." We will re-examine the direction claim against the cited production measurement and state the norm-ratio direction explicitly with its citation; if the production direction is the opposite of what our wording implies, the imbalance magnitude match stands and the direction sentence will be corrected. We will not let an ambiguous sentence carry a matching claim.

AuthorsOpenReview ↗

rebuttal_8bav - part 2

W3 — The conflict metrics are weak; a functional measure would be more meaningful

The paper's claim is precisely the first half of the reviewer's sentence — these metrics are poor — and we quantify the two mechanisms the reviewer names. First, the sign-count degeneracy: with a conflict-rate median of ≈0.52\approx 0.52 over near-zero group cosines (block-level median ≈0.5\approx 0.5), the rate is statistically indistinguishable from a coin flip in this regime. Second, raw-gradient geometry ignores the optimizer: the checkpoint-level raw global cosine is −0.074-0.074, i.e., essentially no association even before considering AdamW's per-coordinate scaling. The constructive counterpart the reviewer asks for already exists in E4: effective rank reaches ρ=0.702\rho = 0.702 [0.603,0.782][0.603, 0.782] at the checkpoint level versus 0.1870.187 for the best directional conflict metric (conflict magnitude), with the norm ratio (−0.651-0.651) exposed as a generation-failure detector that goes null (+0.02+0.02) within configurations that master generation. We agree this shows interference is not unimportant — it is measurable functionally — and the revision will say exactly that.

[New Experiment] metric upgrades. We will add (i) AdamW-aware gradient geometry (cosines computed on preconditioned/whitened gradients, addressing the reviewer's point that raw-gradient cosines ignore the optimizer's per-coordinate scaling and can hide localized conflict), and (ii) the reviewer's suggested lookahead measure — the change in the other task's loss under a provisional step — as a sixth functional baseline in E4.

W4 — E3 does not support a causal claim

We take the four sub-points in turn.

  1. "The projection also removes aligned components." Correct, and it makes the test conservative in our disfavor: the intervention removes more than the conflict component, so a null outcome cannot be explained by the intervention being too gentle. We will state this direction explicitly.
  2. "Raw conflict does not drop, so the manipulation check is tautological." The suppressed-level dose response (0.005→0.0000.005 \to 0.000, Spearman ρ=1.000\rho = 1.000) is by construction, and we agree it is not independent evidence. The non-trivial facts are (a) the raw level does not fall — the adapted gradients re-create conflict at comparable levels, so the intervention is not trivially reducing conflict in the running system — and (b) the outcome is flat anyway.
  3. "The removed component is ~1% of the update or less." We accept the reviewer's estimate and will adopt it: this is why the flat outcome is informative rather than vacuous — even the near-complete suppression actually applied moves the joint score by only −0.002-0.002 by seed (understanding −0.004-0.004, generation +0.001+0.001) against a seed-to-seed spread of 0.0370.037.
  4. "3 seeds per dose only rule out large effects." Correct, and the dose–outcome correlation is correspondingly weak evidence on its own (ρ=−0.600\rho = -0.600, p=0.285p = 0.285, n=12n=12). The claim we will keep is the bounded one — the total dose-response range (0.005) is far inside seed noise (0.028) — and E6's harder regime will re-run the intervention with more seeds per dose.

W5 — Conclusions overreach; no new method

We accept the framing criticism. The contribution is the audit result plus the released protocol plus the first predictor competition (E4) — not a method. The limitations section will gain an explicit mechanism-mismatch paragraph (W1) and a regime paragraph (W2/FA8b-W2), and the abstract's final sentence will be scoped to "in this testbed and at the second scale tested."

AuthorsOpenReview ↗

rebuttal_8bav - part 3

W6 — Numerical inconsistencies

All four are real; here are the diagnoses and fixes.

  1. Abstract 0.145 vs. Table's 0.198. The 0.198 is the conflict-magnitude row of the correlation table (predictive level); the 0.145 in the abstract/summary claim cites a different analysis level's largest ∣ρ∣|\rho|. Both intervals cross zero. Fix: recompute and make every quoted ∣ρ∣|\rho| in the abstract traceable to exactly one named table row; state the level next to the number.
  2. Shallow- vs. depth-weighted values (Sec. 4.2) vs. the table. The text quotes the early-concurrent weighted values (shallow −0.291-0.291 [−0.553,−0.010][-0.553, -0.010]; depth −0.149-0.149 [−0.407,0.121][-0.407, 0.121]), while the table lists weighted variants only at the concurrent level (shallow −0.171-0.171 [−0.428,0.083][-0.428, 0.083]; depth −0.295-0.295 [−0.535,−0.040][-0.535, -0.040]). Fix: add the early-concurrent weighted rows to the table so text and table cite the same numbers.
  3. "|ρρ| gap 0.51" vs. its CI [0.729,1.041][0.729, 1.041]. The quoted CI is the paired-bootstrap interval for the gap against global cosine (0.702−(−0.074)=0.7760.702 - (-0.074) = 0.776, inside the interval); the text paired it with the gap against conflict magnitude (0.702−0.187=0.5150.702 - 0.187 = 0.515). Fix: report both gaps with their own intervals; the significance conclusion is unchanged (both exclude zero).
  4. Fig. 5 caption vs. text. Fig. 5 is the per-layer profile. The caption is correct — blocks 5 and 6 have intervals excluding zero (−0.356-0.356 [−0.590,−0.109][-0.590, -0.109] and −0.279-0.279 [−0.525,−0.015][-0.525, -0.015]) — while the text names only block 5. Fix: correct the text.

Q2 — Is the SOAP-style collapse an implementation artifact? Results without failing methods

On the implementation. The deviation is documented (Appendix): we omit SOAP's momentum-translation scaling and apply RMS normalization to keep learning rates comparable across optimizer families. We cannot exclude that this contributes to the collapse; the paper's use of the method is as a pathology instance — a real method family member whose failure is invisible to the conflict metrics — not an evaluation of SOAP itself, and the revision will say this in one sentence.

On the reanalysis. The paper's healthy-regime cut is exactly "results without the failing configurations": within the 135 checkpoints of the 47 generation-mastering configurations, directional conflict metrics stay weak (largest ∣ρ∣=0.23|\rho| = 0.23), the norm ratio goes null (+0.02+0.02, CI [−0.17,0.18][-0.17, 0.18]), and the best functional predictor retains its advantage (ρ=0.854\rho = 0.854 [0.795,0.903][0.795, 0.903], vs. best directional 0.2300.230; paired Δρ\Delta\rho CI [0.47,0.78][0.47, 0.78]). [Reanalysis committed] we will additionally report the explicit configuration-level table with SOAP-style, MGDA, and token-normalized excluded (N=36N = 36 configurations) so the reviewer can check the exclusion directly.


Summary of revisions

ConcernAction
W1, Q1E8 heterogeneous-representation variant (generation in a different output space); abstract/limitations rescoped
W2Norm-ratio direction statement corrected with explicit citation
W3AdamW-whitened cosines + lookahead-Δ\Deltaloss added to E4
W4E3 claim bounded; conservative-direction statement; more seeds in E6
W5Limitations gain mechanism-mismatch and regime paragraphs
W6All four inconsistencies fixed as diagnosed above
Q2Healthy-cut numbers cited; explicit 3-method-excluded table committed
Reviewer_FA8bOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the influence of gradient conflict in unified multimodal models. Through experiments in a synthetic testbed, it finds that common conflict metrics have limited associations with performance and that reducing conflict provides no clear performance gains.

Critical strengths

Relatively comprehensive experiments within the testbed. The study covers multiple training strategies, data-mixing ratios, random seeds, and model sizes.

Critical weaknesses
  1. Limited representativeness. The experiments use only 3.2M–6.7M models and a narrow set of tasks, limiting the generalizability of the findings to realistic tasks.
  2. Inappropriate task selection weakens the conclusions. Most methods achieve nearly 100% generation accuracy, making the conclusions about the role of gradient conflict less convincing.
Questions for authors
  1. Could the authors provide more comprehensive experiments with larger models and more tasks to verify whether the conclusions still hold?
1 comment
AuthorsOpenReview ↗

rebuttal_FA8b

Response to Reviewer FA8b (Rating 2, Confidence 3)

We thank Reviewer FA8b for the constructive assessment. The two concerns — scale and regime saturation — are the same two we identify as the testbed's main limitations, and one of them (scale) is partially answerable with data already in the paper (E5). We answer both and commit to the new experiments the concerns jointly motivate.

Bottom line. The scale concern is already partially addressed by E5 (18 additional runs at a second scale), which replicates the paper's sign structure; the saturation concern is real and we commit to a harder-regime variant (E6) in which neither task saturates, repeating the full correlation and intervention analyses. All runs are ≈12 minutes each on the same hardware (18.8 aggregate run-hours for all 93 runs in the paper), so both commitments are executable within revision.


W1 — Limited representativeness (3.2M–6.7M models, narrow task set)

What the paper already contains. E5 replicates the audit at a second scale (d=320d{=}320, 8 layers, 6.7M parameters, 18 runs of three representative methods) and yields the same sign structure:

  • directional conflict metrics remain within noise at the larger scale: largest ∣ρ∣=0.15|\rho| = 0.15 (CI [−0.33,0.04][-0.33, 0.04]), against 0.19 at the base scale;
  • the norm-ratio association persists (ρ=−0.42\rho = -0.42, CI [−0.64,−0.20][-0.64, -0.20]).

What we concede and commit to. The paper states in Limitations (i) that the audit's logic and protocol transfer to production scales while its quantitative conclusions do not automatically. We will strengthen this with two commitments: (i) a third scale point as compute permits, using the released protocol (its cost scales linearly in the number of configurations); (ii) task-set extension beyond the three question types (cell color, color count, pair adjacency) in the harder-regime variant below, which broadens the task distribution at the same time as it removes saturation.

We note the testbed is not intended as a miniature production UMM; it is an instrument for testing metric validity with exactly computable ground truth, and the pathologies it reproduces are documented ones (MGDA's minority-gradient collapse at understanding 0.389 vs. generation 1.000; SOAP-style collapse to generation 0.000 in all nine runs — both mirroring modality-imbalance complaints in the production literature we cite).

W2 — Nearly saturated generation weakens the conclusions

Direct answer. Saturation is a property of the population, not of the analysis, and the analysis is designed for it:

  1. The audit's operative claim is about the predictive reading — conflict measured during training forecasting where a configuration lands. The population spans joint scores 0.271–0.820 with understanding accuracy 0.389–0.641 among generation-mastering configurations (mean ≈0.63\approx 0.63), so there is substantial variation to predict — and by measurement it lives on the understanding axis: across healthy methods the generation axis has essentially no variance (s.d. ≈0.004\approx 0.004).
  2. We already report the analysis restricted to configurations that master generation (47 of 63; 135 checkpoints): every directional conflict metric stays weak (largest ∣ρ∣=0.23|\rho| = 0.23), the norm ratio drops to null (ρ=+0.02\rho = +0.02), and the functional measures retain their advantage. The conclusion is not an artifact of the failing runs.
  3. We agree, however, that a regime where both axes vary would make the design airtight, and we commit to it (E6, below).

[New Experiment] E6 — harder regime. Increase grammar difficulty (more objects per scene, longer randomized descriptions, compositional questions) and/or shorten the training budget so that generation is not mastered by the budget's end in normally trained configurations; verify mid-training non-saturation on a pilot before the full sweep; then repeat E1 (correlation, 3×3×3=273\times3\times3 = 27 configurations) and E3 (dose-response). This directly answers the reviewer's question of whether the conclusions hold when there is a genuine two-axis trade-off to explain.

Q1 — Larger models and more tasks

Covered by the E5 discussion and the E6 commitment above. We will also state in the revision that the released protocol makes any of these extensions a configuration change rather than a methods contribution, and report the E6 results in the revision if the rebuttal window does not suffice.


Summary of revisions

ConcernAction
W1E5 numbers foregrounded; third-scale and task-extension commitments
W2, Q1E6 harder-regime variant: correlation + intervention repeated with both axes unsaturated

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.