- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates bidirectional information flow in diffusion-based multimodal large language models (DMLLMs). Through attention knockout and representation analysis, the authors identify a staged integration pattern in which language-to-vision information flow dominates in shallow layers and cooperates with vision-to-language flow in deeper layers. Motivated by these findings, they introduce reverse attention into autoregressive MLLMs, enabling question-conditioned visual representations and improving performance on several vision-centric benchmarks.
Critical strengths
Interesting mechanistic insights into multimodal integration. The paper extends information-flow analysis beyond conventional autoregressive MLLMs and provides evidence of staged bidirectional interactions in diffusion-based architectures.
Well-connected analysis and empirical validation. The authors combine attention knockout, semantic alignment analysis, and controlled architectural modifications. Experiments across different language backbones and visual encoders support the benefits of language-to-vision information flow.
Critical weaknesses
Limited novelty of bidirectional cross-modal interaction. Question-conditioned visual representations and bidirectional vision-language interactions have been explored in previous multimodal architectures. The paper should more clearly distinguish its contributions from existing approaches that allow visual features to incorporate linguistic information.
Potential confounding factors in architectural comparisons. The diffusion and autoregressive models differ in training objectives, architectures, and training data. Consequently, the observed differences in information-flow patterns cannot be attributed solely to bidirectional attention.
Incomplete causal validation of the proposed mechanism. Although attention knockout demonstrates the importance of cross-modal connections, it does not fully establish the claimed causal progression from language-to-vision alignment to improved vision-to-language transfer and downstream reasoning.
Relatively modest downstream improvements. Despite substantial gains in question-conditioned visual representation metrics, improvements on downstream benchmarks are generally moderate and occasionally negative. The practical benefits of the proposed modification could be demonstrated more convincingly.
Insufficient comparisons with alternative bidirectional interaction designs. The experiments primarily compare standard causal attention with the proposed reverse attention mechanism. Stronger comparisons with existing cross-attention and question-conditioned visual processing architectures would help establish the specific advantages of the proposed design.
Questions for authors
NA
3 comments
Rebuttal for Reviewer 5amF (Part2)
(3) Comparison with other bidirectional attention designs
As suggested, it is also worth exploring the effects of alternative bidirectional prefix attention designs under the same training settings. To this end, we further compare our design with full-prefix bidirectional attention (PaliGemma [1]) and another variant that additionally enables bidirectional attention among visual tokens based on our setting.
We find that full-prefix bidirectional attention achieves higher performance on some tasks. However, since causal + reverse already enables both directions of cross-modal flow, the remaining gap arises entirely from intra-modal bidirectional attention rather than cross-modal information flow. For example, on CV-Bench 2D, where the gap is most pronounced, the additional gain of full-prefix attention over causal plus reverse (+7.78) comes from bidirectional attention within visual tokens (+3.64) and within text tokens (+4.14). This is consistent with prior findings that bidirectional attention among image tokens alleviates the context starvation of early image tokens under causal masking and improves spatial consistency [7], which benefits spatial tasks such as CV-Bench 2D but cannot be directly credited to cross-modal information flow.
Moreover, our design achieves its gains with far fewer additional attention edges. Let and denote the numbers of visual and question tokens, respectively. Our design only adds edges from each visual token to each question token, i.e., × edges, whereas bidirectional attention among visual tokens alone adds edges between every pair of visual tokens, i.e., about edges. Since a question typically contains far fewer tokens than an image (), our design introduces far fewer edges ().
All these further support our choice of the causal plus reverse attention setting for a targeted analysis of language-to-vision information flow and highlight that language-to-vision flow brings notable gains with minimal modification.
| Attention setting | Additional attention edges( ) | RWQA | CV-Bench 2D | MMVP | MMStar | BLINK (4-task avg.) |
|---|---|---|---|---|---|---|
| Causal | – | 48.63 | 50.76 | 27.33 | 33.27 | 41.17 |
| Full-prefix bidirectional (PaliGemma-style) | 49.93 | 59.94 | 28.67 | 33.67 | 44.38 | |
| Causal + reverse + bidirectional in vision | 48.89 | 55.80 | 28.00 | 31.47 | 44.26 | |
| Causal + reverse | 50.07 | 52.16 | 27.67 | 32.21 | 43.14 |
Rebuttal for Reviewer 5amF (Part3)
W2: More fair comparison of information-flow patterns
We thank the reviewer for raising this important concern.
(1) Why we compare diffusion MLLMs with AR MLLMs
Conventional causal AR models do not permit bidirectional information flow between image and question tokens within the multimodal prefix. We therefore study DMLLMs, which were trained with bidirectional attention throughout multimodal training, allowing us to examine how the two directions of information flow operate and cooperate. Across the different DMLLMs examined, we consistently observe a two-stage information-flow pattern.
(2) Controlled comparison of causal and bidirectional attention
As suggested, we conduct a more controlled comparison of causal and bidirectional prefix attention under the same LLaVA-1.5 training framework, with the attention mask being the only difference during SFT. We examine their information-flow patterns through attention knockout.
As shown in the newly added Figure 15 of the revised manuscript (as figures cannot be directly included in OpenReview responses), the bidirectional setting exhibits a similar two-stage information flow pattern, with language-to-vision flow dominating in shallow layers, followed by stronger vision-to-language flow cooperating with the reverse direction in later layers.
These findings provide more direct evidence that enabling bidirectional attention is a key factor underlying the emergence of the two-stage information-flow pattern.
W3: Causal relationships between effects
(1) What we claim in our original analyses
It should be noted that our original analyses separately examine how language-to-vision flow affects visual semantic alignment (Sec. 5.2), subsequent vision-to-language transfer (Sec. 5.3), and downstream performance in AR models (Sec. 6), rather than directly establishing causal relationships among these effects.
(2) Causal effect of visual representations on subsequent information transfer
Motivated by this insightful question, we further conduct activation-patching experiments on LLaDA-V to investigate whether early visual representations contribute to subsequent information transfer. Concretely, we restore the original visual hidden states after blocking language-to-vision flow in shallow layers.
As shown in the following table, restoring these visual representations partially recovers the subsequent vision-to-language attention enrichment on task-relevant visual regions. This suggests that more semantically aligned visual representations shaped by early language-to-vision flow help facilitate subsequent information transfer, further strengthening our understanding of the underlying mechanism.
| Intervention (layers 7–15) | Subsequent V→L attention enrichment (layers 16–24) |
|---|---|
| Original | 4.832× |
| Early L→V knockout | 2.902× |
| Early L→V knockout + original visual-state patching | 4.335× |
W4: Downstream benefits of language-to-vision flow in AR
As mentioned above, our primary goal is to examine the benefits of introducing language-to-vision information flow into conventional AR models, rather than introducing an optimal bidirectional prefix attention design.
Probe-based metrics have directly assessed how language-to-vision flow changes visual representations, showing substantial improvements in their question-conditioned organization. Moreover, we observe notable gains on several vision-centric downstream tasks, including object localization (+9.02 with Vicuna/DINOv2), MMVP (+5.33 with Qwen2.5/SigLIP2), and spatial relations (+4.90 with Qwen2.5/CLIP).
While showing occasional negative changes, these benefits are achieved through a simple modification of the attention mask, without introducing any extra design and training. Further exploring how to better leverage language-to-vision flow for downstream tasks is a promising direction for improving future MLLMs.
[1] PaliGemma: A Versatile 3B VLM for Transfer. arXiv:2407.07726, 2024.
[2] Cross-modal Information Flow in Multimodal Large Language Models. CVPR, 2025.
[3] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs. ICLR, 2026.
[4] From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs. NeurIPS, 2026.
[5] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. NeurIPS, 2019.
[6] Hierarchical Question-Image Co-Attention for Visual Question Answering. NeurIPS, 2016.
[7] From drop-off to recovery: A mechanistic analysis of segmentation in MLLMs. ECCV, 2026.
We look forward to any further questions or suggestions you may have regarding our work.
Rebuttal for Reviewer 5amF (Part1)
We sincerely thank the reviewer for the positive assessment and helpful suggestions. We are glad that the reviewer finds our mechanistic insights interesting and recognizes how we combine analysis with empirical validation. Below, we address each weakness (W). All new experiments and discussions are added into the revised manuscript and marked in blue.
We sincerely thank the reviewer for the valuable suggestions regarding the clarification of our contributions and comparisons with existing approaches.
(1) Novelty and contributions
Indeed, previous multimodal architectures have extensively explored bidirectional designs such as PaliGemma [1]. Our novelty and contribution, however, lie in systematically investigating how the two directions of information flow operate and cooperate within MLLMs, rather than introducing bidirectional interaction itself.
This is significant because prior information-flow studies [2, 3, 4] have focused only on one-way vision-to-language transfer when understanding multimodal integration in MLLMs. How language influences visual representations, and further, whether and how the two directions of information flow benefit and collaborate with each other remain unexplored, which are indispensable aspects of modality integration in multimodal learning [5, 6]
To fill this gap, we systematically analyze the mechanisms and effects of bidirectional cross-modal information flow in MLLMs.
Our key findings are:
(2) Our goal and why we use this design
Our exploration in AR aims to examine how introducing language-to-vision flow into conventional causal models could bring benefits, rather than introducing an optimal bidirectional prefix attention design.
Therefore, in the AR paradigm, we introduce only the additional cross-modal attention edges while preserving causal attention within each modality, so the gains can be solely credited to the newly added language-to-vision flow.
Methods such as PaliGemma [1] also enable language-to-vision flow through full bidirectional attention over multimodal prefixes. However, their performance gains cannot be attributed solely to this language-to-vision flow, as they additionally allow bidirectional attention within visual and text tokens, which may bring extra benefits.