acceptodds
Desk rejected at ICLR 2027

AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal models (UMMs) integrate image understanding and generation within a single architecture, yet the two tasks require different visual representations, creating inherent conflicts during joint training. Recent methods increasingly improve UMMs through architectural decoupling (e.g., decoupled visual encoders, MoE/MoT architectures, or frozen MLLMs), but why decoupling works remains poorly understood. In this work, we revisit architectural decoupling through layer-wise cross-modal attention interaction patterns. Our analysis reveals a previously overlooked phenomenon: as architectural decoupling becomes stronger, understanding and generation retain distinct interaction regimes, while each task's interaction pattern becomes increasingly similar to that of a strong task-specific reference model. This suggests that the benefit of architectural decoupling may stem from enabling **task-specific cross-modal interaction specialization**, rather than driving the two tasks toward a shared interaction behavior. Motivated by this finding, we propose **A**ttention **I**nteraction **A**lignment (**AIA**), which adaptively transfers task-specific reference interaction behavior without further architectural decoupling. Specifically, **stage-level interaction alignment** transfers the overall cross-layer interaction knowledge of the reference while preserving the student's own representation organization, whereas a **Huber loss** retains the model's ability to adaptively adjust interaction across individual layers. Applied to Emu3 during supervised fine-tuning and Janus-Pro during post-training, AIA consistently improves both understanding and generation, and further improves interleaved cross-task performance. These findings suggest that strong unified multimodal performance does not require understanding and generation to converge to a shared interaction regime; instead, each task can retain and exploit its own appropriate interaction behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Multimodal Model, Intelligent Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MqkJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Attention Interaction Alignment (AIA). By calculating the cross-modal interaction intensity, it can generally reflect the attention pattern between image and text among each layer. And by Stage-level Averaging & Huber loss, the student can learn the attention pattern from the task-specific teacher (generation/understanding models). Both generation and understanding tasks of the UMM can be enhanced during the process, so as to get a better performance in both task's benchmark.

Critical strengths
  1. The paper introduces cross-modal Interaction Intensity to characterize the cross-modal attention mass, providing an intuitive way to analyze layer-wise attention interaction patterns in UMMs.

  2. The proposed stage-level alignment guides the student to learn the overall interaction trends of the teacher instead of directly matching individual layers. This preserves flexibility in student's layer-wise representations, reducing the risk of overly restrictive alignment.

Critical weaknesses
  1. The proposed Interaction Intensity is sensitive to the sequence lengths of different modalities. For example, when Ltext<<LimageL_{text}<<L_{image}, the text attention mass may be relatively low even when the text contains important conditioning information. It would be helpful to provide an analysis of how token ratios affect the measured interaction intensity.

  2. The training datasets include BLIP3-o and ShareGPT-4o-Image which highly overlaps with GenEval. This calls into question whether the reported scores (meanwhile, some benefit itself is very marginal, e.g. 0.01 in Janus-Pro) are a true measure of performance or an artifact of the similar data distributions. Could the authors provide additional results excluding potentially overlapping samples?

Questions for authors

Please see the weakness.

1 comment

Thanks for the authors’ detailed responses. I will increase the rating from 2 to 3.

Reviewer_8NiqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why architectural decoupling helps unified models that handle both image understanding and generation. It measures per-layer cross-modal attention intensity in Emu3, Janus-Pro, and BAGEL and finds that the more decoupled a model is, the closer its profile is to that of a task-specific reference model.

Critical strengths

AIA is a light intervention: it needs only precomputed per-layer scalars and no teacher at training time. What makes it more than a diagnostic is that the authors act on the attention statistic instead of only observing it, and they do so on two quite different backbones (Table 3) with a matched post-training baseline (Table 5). If the causal story held up, this would be a practical recipe for unified models.

Critical weaknesses

First, the central explanation is correlational. Fig. 1 compares models that differ in data, scale and training recipe, so "closer to a strong reference" is confounded with "is a strong model". Sec. 4.4 says AIA helps at every degree of decoupling, but in Table 4 only the purely unified model has an AIA row. The experiment that would test the claim, AIA applied to already-decoupled variants, is missing. Without it, the paper shows that AIA helps a unified model, not that decoupling works through the proposed mechanism.

Second, the empirical gains are small and the evidence is thin. Several Janus-Pro improvements are a point or less (GenEval 0.80 to 0.81, MMVP 47.3 to 48.0), reported without seeds or intervals, and the hyperparameters and reference model appear to be tuned on the reported benchmarks. The same number also changes between tables: MMVet for Janus-Pro 7B + AIA is 51.2 in Tables 3 and 5 but 49.8 in Table A1. The Emu3 baseline is very weak (MMVP 8.7), which makes its gains easy to obtain and hard to interpret.

Questions for authors

Can you add AIA rows, over several seeds, for the double-encoder and MoT variants in Table 4? A positive result there would change my view of the main claim.

Which MMVet figure for Janus-Pro 7B + AIA is correct, 51.2 or 49.8?

Were the weight λ, the stage boundaries, and the choice of reference model selected on held-out data rather than on the reported benchmarks?

A flat target and a shuffled-stage target would be stronger controls than the random target in the current ablation.

Reviewer_YP93OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why architectural decoupling improves unified multimodal models by analyzing layer-wise cross-modal attention mass across understanding and generation tasks. Observing that stronger decoupling moves each task's depth-wise attention profile closer to that of a single-task reference model, the authors propose Attention Interaction Alignment, which uses a stage-level Huber loss to guide each task toward its reference interaction regime without altering the shared architecture. Experiments on a controlled 1.7B study, Emu3-8B SFT, and Janus-Pro-7B post-training show improvements in understanding, generation, and cross-task transfer with zero inference overhead.

Critical strengths

(1) The paper investigates an important and interesting question.

(2) A controlled from-scratch study on Qwen3-1.7B confirms the observed attention trend and shows that AIA recovers most of the performance gap on a purely unified model.

(3) AIA relies only on pre-computed scalar stage targets, avoiding online teacher forward passes during training, adding zero inference latency, and outperforming feature-level and output-level distillation baselines.

Critical weaknesses

(1) The interaction metric does not normalize for the text-to-image token ratio, so on high-resolution models like Emu3 the trained generation curve remains far below the target intensity. Because the deviation always exceeds the Huber threshold, the loss reduces to a constant linear push on text attention rather than aligning to the reference target.

(2) The stage partitioning lacks a well-defined criterion: the paper does not explain how turning points are quantitatively determined from the continuous reference curves, nor does it evaluate sensitivity to the stage boundary positions.

(3) On Janus-Pro 7B, the generation improvement on GenEval is negligible across data scales, and the MMVet scores for the same 7B model are inconsistent between the main text and the appendix.

(4) The cross-task evaluation only reports the "Understanding aids Generation" split of RealUnify on Janus-Pro, omitting the reverse direction and Emu3.

Questions for authors

Please see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.