acceptodds
Withdrawn from ICLR 2027

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to interfere. Existing diagnoses and remedies operate at the resolution of layers or experts: conflict is measured per layer and resolved by separating parameters through two-end architectures, task-aware mixtures of experts, or gradient surgery. We argue that this resolution hides an orthogonal axis. Generation in a UMM is next-token prediction over a raster sequence of visual tokens whose roles vary systematically with position—early tokens fix global layout and low-frequency structure, late tokens fill in texture—so how strongly a generation gradient interferes with understanding should depend on where in the sequence it originates. We introduce a position-resolved interference map that attributes the understanding–generation gradient conflict to visual-token positions within every layer, computed from a single backward pass with position-grouped gradient hooks at the cost of a standard backward pass. On Show-o and Janus-Pro, position explains a substantial share of conflict variance after controlling for depth (partial vs. for layer on Show-o; vs. on Janus-Pro, whose understanding branch bypasses the visual-token sequence): the first quarter of the sequence has a mean gradient cosine of against understanding, the last quarter . The dependence survives per-position gradient-norm normalization, retaining of its effect size, so it is directional rather than a magnitude artifact, and conflict strength tracks the semantic content of a position (Spearman ). Building on the map, we propose position-aware modulation (PAM), which removes the anti-aligned component of generation gradients only at high-conflict positions without changing the architecture. Under a matched trainable-parameter budget PAM improves over layer-wise separation by MME and GenEval points while matching it on POPE and overall FID, whereas a random-position control with the same number of modulated tokens recovers only about of the gain; the cost is a relative degradation of the high-frequency component of FID, with overall FID unchanged. Position-based and layer-based separation are complementary degrees of freedom and can be combined.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Multimodal Models, Multimodal Understanding and Generation, Gradient Conflict, Position-Resolved Interference, Position-Aware Modulation, Visual Token Positions, Multi-Task Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BA4NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies how understanding–generation gradient conflict varies across visual-token positions and layers. It proposes position-aware gradient modulation and reports improvements over layer-wise separation on two unified multimodal models.
Critical strengths
  • Position-resolved analysis offers a useful perspective beyond layer-level measurements.
  • Random-position controls, normalization checks, and explicit reporting of the texture-quality trade-off make the evaluation more informative.
Critical weaknesses
  • The hook decomposition groups gradient contributions by intermediate activation position. With attention mixing positions, this is not generally equivalent to grouping gradients by output-token loss, yet the paper appears to conflate these interpretations.
  • The reported 0.3% training overhead needs reconciliation with running a probe at every generation step whose backward-pass overhead is approximately 20%.
Questions for authors
  1. Clarification: Does the brute-force self-test mask output-token losses or intermediate activation gradients? Please explain which positional attribution is actually validated in the full attention model.
  2. Evidence request: Can you provide a timing and memory breakdown covering bucket-gradient accumulation, projection, and understanding-gradient refresh? This is necessary to assess the claimed training overhead.
Reviewer_s6wVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies understanding–generation gradient conflict in unified multimodal models at the level of layers and visual-token positions. It introduces a position-resolved conflict map and PAM, which projects generation gradients at selected positions away from an understanding-gradient direction. Experiments on Show-o and Janus-Pro report improvements on both tasks, with some loss of high-frequency detail.

Critical strengths
  1. Understanding when the two tasks can share parameters effectively is an important problem. A reliable position-level analysis could complement existing layer-level diagnoses and guide more targeted optimization.

  2. The evaluation includes matched parameter budgets, random-position controls, single-task references, and frequency-band measurements, considering both benefits and costs.

Critical weaknesses
  1. The central positional interpretation is not established. Show-o uses discrete diffusion with full attention among image tokens, whereas this paper describes it as raster-order autoregressive generation. Were its objective and attention mask changed? Even for an autoregressive model, raster position does not inherently separate global structure from texture. Moreover, Eq. (2) decomposes the total loss gradient by internal activation position; after attention mixes positions, this is generally different from differentiating only the prediction losses in that bucket. Please clarify what is actually measured and demonstrate the claimed equivalence to per-bucket backward passes in a model with cross-position attention. These are substantive issues for the meaning of the map and its semantic interpretation.

  2. The methodological contribution is insufficiently isolated. PAM uses the basic conflicting-component projection of PCGrad, so its distinctive contribution rests on position selection. Random-position controls do not establish an advantage over global or layer-wise projection under the same budget, or over simply selecting a fixed prefix. Could the authors provide these comparisons and specify the training datasets and sampling procedure? The appendix mainly supplies optimizer settings. The reported gains suggest that the intervention can help, but do not establish that the detailed conflict map is necessary.

Questions for authors

See Weaknesses.

Reviewer_d1xROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows position-dependent gradient conflicts between understanding and generation branches in Unified Multimodal Models (UMMs). Based on this, they propose Position-Aware Modulation (PAM), a parameter-free gradient modulation strategy that prunes the components of generation gradient that are conflict with understanding gradient, while preserving the remaining components. Experiments on Show-o and Janus-Pro demonstrate improvements on benchmarks including GenEval and MME compared with joint training and layer-wise separation baselines.

Critical strengths
  1. The paper proposes that across different image token positions, the gradients conflict between generation and understanding are different.

  2. PAM only prunes the components of generation gradient that are conflict with understanding gradient, while preserving the remaining components. This design provides a better understanding learning while retaining useful learning signals for generation.

Critical weaknesses
  1. The paper assumes that early visual tokens mainly encode global semantics and layout, while later tokens focus on local textures and details. But the visual tokens in raster-order image generation mainly correspond to different spatial locations. And Show-o employs discrete denoising diffusion with full attention for image generation rather than raster-order autoregressive prediction. It's in conflict with paper's assumption.

  2. The benchmarks used in paper are very narrow, and the PAM only gains modest improvements compared to layer-wise separation baseline. Could you please provide more benchmarks like Qwen-Image-Bench, OneIG-Bench, MMMU, MMBench.

And the figure/chart layout is a disaster.

Questions for authors

Please see my comments in weaknesses.

What's more, focusing on improving the balance between understanding and generation, the paper provides no qualitative comparisons of generated images. This is particularly concerning given the reported degradation in high-frequency FID. Visual comparisons would help demonstrate whether PAM actually improves semantic alignment and global structure, and whether such improvements come at the cost of visual details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.