acceptodds
Under review as a conference paper at ICLR 2027

Language Shapes Vision: Bidirectional Information Flow in Multimodal LLMs

Abstract

MLLMs have shown strong capabilities in jointly processing visual and linguistic information, making the underlying mechanism of multimodal integration a fundamental question. However, existing studies mainly examine autoregressive (AR) MLLMs, where causal attention has largely confined our understanding of this mechanism to a unidirectional flow from vision to language. In this work, we revisit multimodal integration mechanisms by studying diffusion MLLMs (DMLLMs) with full attention between vision and language. Surprisingly, we find that DMLLMs coordinate bidirectional information flow in a distinctly staged manner: language-to-vision flow dominates in shallow layers, and cooperates with vision-to-language flow in deeper layers. We further reveal that this language-to-vision flow prompts the alignment of question-relevant visual states with semantics and facilitates subsequent vision-to-language transfer. Motivated by this finding, we introduce language-to-vision flow into the training of conventional AR MLLMs and find that it brings more question-targeted processing of visual information, improving vision-centric understanding ability. Our findings reveal bidirectional information flow as an important mechanism underlying multimodal integration and provide insights for designing next-generation MLLMs.

Primary area:
interpretability and explainable AI
Keywords:
Information Flow, Multimodal Large Language Models, Multimodal Integration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5amFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates bidirectional information flow in diffusion-based multimodal large language models (DMLLMs). Through attention knockout and representation analysis, the authors identify a staged integration pattern in which language-to-vision information flow dominates in shallow layers and cooperates with vision-to-language flow in deeper layers. Motivated by these findings, they introduce reverse attention into autoregressive MLLMs, enabling question-conditioned visual representations and improving performance on several vision-centric benchmarks.

Critical strengths

Interesting mechanistic insights into multimodal integration. The paper extends information-flow analysis beyond conventional autoregressive MLLMs and provides evidence of staged bidirectional interactions in diffusion-based architectures.

Well-connected analysis and empirical validation. The authors combine attention knockout, semantic alignment analysis, and controlled architectural modifications. Experiments across different language backbones and visual encoders support the benefits of language-to-vision information flow.

Critical weaknesses

Limited novelty of bidirectional cross-modal interaction. Question-conditioned visual representations and bidirectional vision-language interactions have been explored in previous multimodal architectures. The paper should more clearly distinguish its contributions from existing approaches that allow visual features to incorporate linguistic information.

Potential confounding factors in architectural comparisons. The diffusion and autoregressive models differ in training objectives, architectures, and training data. Consequently, the observed differences in information-flow patterns cannot be attributed solely to bidirectional attention.

Incomplete causal validation of the proposed mechanism. Although attention knockout demonstrates the importance of cross-modal connections, it does not fully establish the claimed causal progression from language-to-vision alignment to improved vision-to-language transfer and downstream reasoning.

Relatively modest downstream improvements. Despite substantial gains in question-conditioned visual representation metrics, improvements on downstream benchmarks are generally moderate and occasionally negative. The practical benefits of the proposed modification could be demonstrated more convincingly.

Insufficient comparisons with alternative bidirectional interaction designs. The experiments primarily compare standard causal attention with the proposed reverse attention mechanism. Stronger comparisons with existing cross-attention and question-conditioned visual processing architectures would help establish the specific advantages of the proposed design.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 5amF (Part1)

We sincerely thank the reviewer for the positive assessment and helpful suggestions. We are glad that the reviewer finds our mechanistic insights interesting and recognizes how we combine analysis with empirical validation. Below, we address each weakness (W). All new experiments and discussions are added into the revised manuscript and marked in blue.

W1&5: Novelty and comparison with existing bidirectional designs

We sincerely thank the reviewer for the valuable suggestions regarding the clarification of our contributions and comparisons with existing approaches.

(1) Novelty and contributions

Indeed, previous multimodal architectures have extensively explored bidirectional designs such as PaliGemma [1]. Our novelty and contribution, however, lie in systematically investigating how the two directions of information flow operate and cooperate within MLLMs, rather than introducing bidirectional interaction itself.

This is significant because prior information-flow studies [2, 3, 4] have focused only on one-way vision-to-language transfer when understanding multimodal integration in MLLMs. How language influences visual representations, and further, whether and how the two directions of information flow benefit and collaborate with each other remain unexplored, which are indispensable aspects of modality integration in multimodal learning [5, 6]

To fill this gap, we systematically analyze the mechanisms and effects of bidirectional cross-modal information flow in MLLMs.

Our key findings are:

  • A two-stage bidirectional information-flow pattern. In DMLLMs, language-to-vision flow dominates in shallow layers, and then cooperates with vision-to-language flow in deeper layers.
  • The role of language-to-vision flow. It aligns question-relevant visual representations with semantic concepts and facilitates subsequent vision-to-language transfer.
  • Benefits of introducing bidirectional cross-modal flow into AR MLLMs. It makes visual representations question-conditioned and improves vision-centric downstream performance.

(2) Our goal and why we use this design

Our exploration in AR aims to examine how introducing language-to-vision flow into conventional causal models could bring benefits, rather than introducing an optimal bidirectional prefix attention design.

Therefore, in the AR paradigm, we introduce only the additional cross-modal attention edges while preserving causal attention within each modality, so the gains can be solely credited to the newly added language-to-vision flow.

Methods such as PaliGemma [1] also enable language-to-vision flow through full bidirectional attention over multimodal prefixes. However, their performance gains cannot be attributed solely to this language-to-vision flow, as they additionally allow bidirectional attention within visual and text tokens, which may bring extra benefits.

AuthorsOpenReview ↗

Rebuttal for Reviewer 5amF (Part2)

(3) Comparison with other bidirectional attention designs

As suggested, it is also worth exploring the effects of alternative bidirectional prefix attention designs under the same training settings. To this end, we further compare our design with full-prefix bidirectional attention (PaliGemma [1]) and another variant that additionally enables bidirectional attention among visual tokens based on our setting.

We find that full-prefix bidirectional attention achieves higher performance on some tasks. However, since causal + reverse already enables both directions of cross-modal flow, the remaining gap arises entirely from intra-modal bidirectional attention rather than cross-modal information flow. For example, on CV-Bench 2D, where the gap is most pronounced, the additional gain of full-prefix attention over causal plus reverse (+7.78) comes from bidirectional attention within visual tokens (+3.64) and within text tokens (+4.14). This is consistent with prior findings that bidirectional attention among image tokens alleviates the context starvation of early image tokens under causal masking and improves spatial consistency [7], which benefits spatial tasks such as CV-Bench 2D but cannot be directly credited to cross-modal information flow.

Moreover, our design achieves its gains with far fewer additional attention edges. Let NvN_{v} and NqN_{q} denote the numbers of visual and question tokens, respectively. Our design only adds edges from each visual token to each question token, i.e., NvN_{v} × NqN_{q} edges, whereas bidirectional attention among visual tokens alone adds edges between every pair of visual tokens, i.e., about (Nv)2/2(N_{v})^2/2 edges. Since a question typically contains far fewer tokens than an image (Nq≪NvN_{q} ≪ N_{v}), our design introduces far fewer edges (Nv×Nq≪(Nv)2/2N_{v} × N_{q} ≪ (N_{v})^2/2).

All these further support our choice of the causal plus reverse attention setting for a targeted analysis of language-to-vision information flow and highlight that language-to-vision flow brings notable gains with minimal modification.

Attention settingAdditional attention edges( Nq≪NvN_{q} ≪ N_{v} )RWQACV-Bench 2DMMVPMMStarBLINK (4-task avg.)
Causal–48.6350.7627.3333.2741.17
Full-prefix bidirectional (PaliGemma-style)Nv×Nq+Nv22+Nq22N_{v}\times N_{q}+\frac{N_{v}^{2}}{2}+\frac{N_{q}^{2}}{2}49.9359.9428.6733.6744.38
Causal + reverse + bidirectional in visionNv×Nq+Nv22N_{v}\times N_{q}+\frac{N_{v}^{2}}{2}48.8955.8028.0031.4744.26
Causal + reverseNv×NqN_{v} × N_{q}50.0752.1627.6732.2143.14
AuthorsOpenReview ↗

Rebuttal for Reviewer 5amF (Part3)

W2: More fair comparison of information-flow patterns

We thank the reviewer for raising this important concern.

(1) Why we compare diffusion MLLMs with AR MLLMs

Conventional causal AR models do not permit bidirectional information flow between image and question tokens within the multimodal prefix. We therefore study DMLLMs, which were trained with bidirectional attention throughout multimodal training, allowing us to examine how the two directions of information flow operate and cooperate. Across the different DMLLMs examined, we consistently observe a two-stage information-flow pattern.

(2) Controlled comparison of causal and bidirectional attention

As suggested, we conduct a more controlled comparison of causal and bidirectional prefix attention under the same LLaVA-1.5 training framework, with the attention mask being the only difference during SFT. We examine their information-flow patterns through attention knockout.

As shown in the newly added Figure 15 of the revised manuscript (as figures cannot be directly included in OpenReview responses), the bidirectional setting exhibits a similar two-stage information flow pattern, with language-to-vision flow dominating in shallow layers, followed by stronger vision-to-language flow cooperating with the reverse direction in later layers.

These findings provide more direct evidence that enabling bidirectional attention is a key factor underlying the emergence of the two-stage information-flow pattern.

W3: Causal relationships between effects

(1) What we claim in our original analyses

It should be noted that our original analyses separately examine how language-to-vision flow affects visual semantic alignment (Sec. 5.2), subsequent vision-to-language transfer (Sec. 5.3), and downstream performance in AR models (Sec. 6), rather than directly establishing causal relationships among these effects.

(2) Causal effect of visual representations on subsequent information transfer

Motivated by this insightful question, we further conduct activation-patching experiments on LLaDA-V to investigate whether early visual representations contribute to subsequent information transfer. Concretely, we restore the original visual hidden states after blocking language-to-vision flow in shallow layers.

As shown in the following table, restoring these visual representations partially recovers the subsequent vision-to-language attention enrichment on task-relevant visual regions. This suggests that more semantically aligned visual representations shaped by early language-to-vision flow help facilitate subsequent information transfer, further strengthening our understanding of the underlying mechanism.

Intervention (layers 7–15)Subsequent V→L attention enrichment (layers 16–24)
Original4.832×
Early L→V knockout2.902×
Early L→V knockout + original visual-state patching4.335×

W4: Downstream benefits of language-to-vision flow in AR

As mentioned above, our primary goal is to examine the benefits of introducing language-to-vision information flow into conventional AR models, rather than introducing an optimal bidirectional prefix attention design.

Probe-based metrics have directly assessed how language-to-vision flow changes visual representations, showing substantial improvements in their question-conditioned organization. Moreover, we observe notable gains on several vision-centric downstream tasks, including object localization (+9.02 with Vicuna/DINOv2), MMVP (+5.33 with Qwen2.5/SigLIP2), and spatial relations (+4.90 with Qwen2.5/CLIP).

While showing occasional negative changes, these benefits are achieved through a simple modification of the attention mask, without introducing any extra design and training. Further exploring how to better leverage language-to-vision flow for downstream tasks is a promising direction for improving future MLLMs.

[1] PaliGemma: A Versatile 3B VLM for Transfer. arXiv:2407.07726, 2024.

[2] Cross-modal Information Flow in Multimodal Large Language Models. CVPR, 2025.

[3] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs. ICLR, 2026.

[4] From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs. NeurIPS, 2026.

[5] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. NeurIPS, 2019.

[6] Hierarchical Question-Image Co-Attention for Visual Question Answering. NeurIPS, 2016.

[7] From drop-off to recovery: A mechanistic analysis of segmentation in MLLMs. ECCV, 2026.

We look forward to any further questions or suggestions you may have regarding our work.

Reviewer_hWbLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how vision and language mix inside MLLMs. In autoregressive models, causal attention lets information flow only from vision to language. The authors study diffusion MLLMs, which use full bidirectional attention, and find a staged pattern: in shallow layers, language shapes vision first; later, the two directions work together. They also show that question words get tied to image regions and concepts inside the model. For autoregressive models, they relax the causal mask during instruction tuning so image tokens can attend to question tokens. This gives small downstream gains, with a few drops.

Critical strengths

The staged pattern is a real finding: language shapes vision first, then vision feeds language. The paper also closes the loop: it takes the finding and builds a concrete fix for AR models. Eq. (3) is clean — only image-to-question edges are added, and attention inside each modality stays causal (Appendix C.2), so the effect can be credited to the new edges.

Critical weaknesses

The paper never tests the simple ordering control. Its input puts image tokens before question tokens, and reverse attention exists to let image tokens see the question. But in LLaVA-style training, the image sometimes comes before the question and sometimes after, and the order makes little difference. If the image is placed after the question, image tokens can already attend to the question under the normal causal mask, with no new edges. The paper never runs this baseline. Without it, the gain from reverse attention may come from something the ordering already gives for free, not from the new attention paths.

Questions for authors

What happens if the image is placed after the question in the AR baseline, with the normal causal mask? Image tokens can then attend to the question with no new edges. How much of the reverse-attention gain does this simple change recover?

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer hWbL (Part1)

We sincerely thank the reviewer for this insightful question, which helps us provide more direct evidence supporting our findings and a more complete analysis of bidirectional information flow in AR models. All new experiments and discussions are added to the revised manuscript and marked in blue.

Response 1. Placing the question before the image under causal attention

We evaluate the suggested setting, causal([Q, V]), in which the question is placed before the image under the standard causal mask. For a complete comparison, we further add reverse attention to this setting, i.e., causal + reverse([Q, V]), which enables vision-to-language flow alongside the existing language-to-vision flow.  Causal([V, Q]) serves as the baseline, and causal + reverse([V, Q]) is our original setting in the paper. Experiments are conducted using Vicuna-7B-v1.5 with CLIP.

The results show that:

(1) causal([Q, V]) already strengthens the question-conditioned organization of visual representations compared with causal([V, Q]), since changing the input order enables language-to-vision flow.

Attention settingAUROC ↑NDCG@all ↑Target-shift AUROC ↑
causal([V, Q])77.6057.3049.51
causal([Q, V])80.7661.3477.41
causal + reverse ([Q, V])82.6561.0977.21
causal + reverse([V, Q])82.2163.9079.28

(2) To examine whether this improvement in visual representations further yields better downstream visual understanding, we further evaluate these four settings on vision-centric benchmarks.

As shown in the following table, the causal([Q, V]) setting underperforms our original causal + reverse([V, Q]) setting on downstream vision-centric tasks, showing no clear overall improvement compared with the causal([V, Q]) baseline.

In fact, causal([Q, V]) still enables only unidirectional language-to-vision flow within the multimodal prefix, while vision-to-language flow is absent. We therefore further add reverse attention, i.e., causal + reverse([Q, V]). With both directions of information flow enabled, causal + reverse([Q, V]) improves downstream performance over causal([Q, V]).

Overall, bidirectional information flow provides additional benefits over unidirectional flow, regardless of the input order. We further investigate the cooperation between the two directions of information flow in the next part.

Attention settingRWQACV-Bench 2DMMVPMMStarBLINK (4-task avg.)
causal([V, Q])48.6350.7627.3333.2741.17
causal([Q, V])43.9251.7419.3327.0742.23
causal + reverse([Q, V])49.9352.4326.6732.6742.35
causal + reverse([V, Q])50.0752.1627.6732.2143.14
AuthorsOpenReview ↗

Rebuttal for Reviewer hWbL (Part2)

Response 2. Beyond causal: collaboration of bidirectional flow in AR models

To further understand the benefits brought by bidirectional information flows, we observe how the newly introduced direction affects the already existing one, following the analysis method in Sec. 5.3. Specifically, comparing causal([V, Q]) with causal + reverse([V, Q]) reveals how introducing language-to-vision flow affects the existing vision-to-language flow, and vice versa for causal([Q, V]) and causal + reverse([Q, V]).

As shown in the following table, under both input orders, the newly introduced flow consistently helps the existing flow concentrate on relevant cross-modal content, especially in the middle layers (L9–24) where cross-modal interaction becomes prominent. Specifically, introducing language-to-vision flow makes vision-to-language flow more concentrated on task-relevant visual regions (e.g., 3.273× → 4.548× in L9–16). Conversely, introducing vision-to-language flow makes language-to-vision flow more concentrated on the question keywords (e.g., 3.187× → 5.548× in L9–16).

LayerCausal([V, Q])Causal + reverse([V, Q])Causal([Q, V])Causal+reverse([Q, V])
L1–81.459×1.466×1.412×1.430×
L9–163.273×4.548×3.187×5.548×
L17–243.364×4.768×3.796×4.953×
L25–321.765×1.982×1.362×1.605×

This further shows that the two directions of information flow facilitate each other in AR models and helps explain why adding reverse attention edges yields additional downstream gains. We incorporate these experiments and analyses in the revised manuscript (Appendix D.1 and D.2).

We would be grateful for any further discussion. We hope that our additional evidence and analyses can help address your concerns and earn your support.

Dear reviewer,

As the rebuttal discussion period nears its end, we would appreciate your feedback on whether our responses have successfully addressed your concerns. We would be happy to answer any remaining questions.

Thank you again for your time and effort.

Dear Reviewer,

As the rebuttal period is ending soon, we would like to know whether our responses addressed your concern. Your valuable suggestion helps us further improve our work. We are happy to answer any further questions or concerns.

Thank you again for your time and thoughtful consideration.

Reviewer_VxSrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates bidirectional information flow between visual and language tokens in multimodal LLMs. Through attention knockout experiments on three diffusion MLLMs, the authors identify a two-stage pattern where language-to-vision flow is more important in shallow layers, followed by cooperation with vision-to-language flow in deeper layers. They further analyze how this reverse flow affects visual representations and introduce it into autoregressive MLLMs by modifying the attention mask. Experiments show that reverse attention makes visual representations more question-dependent and improves performance on several visual understanding benchmarks.

Critical strengths
  1. Interesting research question and insightful analysis. The paper provides a new perspective on multimodal information integration by examining both directions of information flow. The two-stage pattern observed across multiple diffusion MLLMs is interesting, and the attention knockout experiments provide useful evidence for understanding their internal mechanisms.
  2. Clear experimental design and useful findings. The authors further validate their observations by introducing reverse attention into autoregressive MLLMs. Experiments across different language backbones and visual encoders show more question-dependent visual representations and improvements on several downstream tasks.
Critical weaknesses
  1. The interpretation of semantic alignment needs further support. The paper uses logit lens scores to show that language-to-vision flow strengthens semantic information in visual representations. However, the higher scores may partly result from question keywords being directly incorporated into visual tokens, rather than improved visual understanding. The current experiments do not fully distinguish these possibilities.
  2. Limited downstream improvements and methodological novelty. Although reverse attention substantially improves the probe-based metrics, the gains on downstream benchmarks are generally modest, with some tasks showing performance decreases. The paper also does not report variance across training runs. Furthermore, related architectures such as PaliGemma have already explored bidirectional attention over multimodal prefixes, so the practical advantage of the proposed design could be better established.
Questions for authors
  1. In Figure 3(b), how can the authors distinguish genuine visual semantic alignment from the direct incorporation of question keywords into visual tokens? Could an experiment with mismatched or perturbed visual evidence help clarify this?
  2. Many improvements in Table 2 are relatively small. Could the authors provide statistical significance analysis to demonstrate their reliability?
  3. How does the proposed reverse attention design compare with existing bidirectional prefix attention methods, such as PaliGemma, under comparable training settings? A direct comparison would help clarify its practical contribution.
  4. The information flow analysis in Figure 2 uses only correctly answered questions. Do the same two-stage patterns hold for incorrectly answered samples? This would help clarify whether the observed mechanism is specific to successful predictions.
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer VxSr (Part1)

We sincerely thank the reviewer for the positive feedback and constructive suggestions. We are glad that the reviewer finds our analysis insightful, the observed pattern interesting, the experimental design clear, and our findings useful. Below, we address each weakness (W) and question (Q). All new experiments and discussions are added to the revised manuscript and marked in blue.

W1&Q1: Further support for semantic alignment

We thank the reviewer for the valuable suggestion. As suggested, we provide more comprehensive evidence to verify the genuine visual semantic alignment effect of language-to-vision flow.

As suggested, we conduct additional experiments by keeping the question fixed while replacing the original image with a mismatched one or perturbing the visual evidence (following the pipeline used by BEAF [1]). We find that language-to-vision flow produces substantially larger increases in logit-lens scores for question keywords in the corresponding visual regions of the original images than when the images are mismatched or perturbed. This shows that language-to-vision flow aligns visual representations with semantic concepts based on the visual evidence they contain.

QuestionImageChange in Logit-lens score
Original QuestionOriginal image13%
UnchangedMismatched image0.6%
UnchangedPerturbed image1.7%

More importantly, we further examine the visual attribute (color) that is present in the image but not explicitly mentioned in the question (so visual tokens cannot obtain this information directly from the question). The semantic scores for this visual attribute in the original image also increase with language-to-vision flow, but such improvements are not observed under mismatched or perturbed images.

QuestionImageChange in Logit-lens score
Original QuestionOriginal image11%
UnchangedMismatched image-0.8%
UnchangedPerturbed image-2.5%

In addition, as visualized in Fig. 4 of the manuscript, the visual tokens with increased semantic scores are primarily concentrated within the bounding boxes of the corresponding visual concepts, while tokens outside these regions show limited improvements despite also being able to access question information.

All these observations further support that language-to-vision flow leads to genuine visual semantic alignment, rather than merely the direct incorporation of question keywords into visual tokens.

AuthorsOpenReview ↗

Rebuttal for Reviewer VxSr (Part2)

W2&Q3: Comparison with existing bidirectional attention designs

(1) Our goal and why we use this design

Indeed, previous multimodal architectures have extensively explored bidirectional designs such as PaliGemma [2]. Our goal lies in systematically investigating how the two directions of information flow operate and cooperate within MLLMs, rather than introducing bidirectional interaction itself.

This is significant because prior information-flow studies [3, 4, 5] have focused only on one-way vision-to-language transfer when understanding multimodal integration in MLLMs. How language influences visual representations, and further, whether and how the two directions of information flow benefit and collaborate with each other remain unexplored, which are indispensable aspects of modality integration in multimodal learning [6, 7].

Following this goal, our design in AR models serves as a controlled examination of how introducing language-to-vision flow affects conventional causal models, rather than an optimal bidirectional prefix attention design. So we add attention edges from image tokens to question tokens while preserving causal attention within each modality, so that the observed effects can be attributed to the newly introduced language-to-vision flow.

Methods such as PaliGemma [2] also enable language-to-vision flow through full bidirectional attention over multimodal prefixes. However, their performance gains cannot be attributed solely to this language-to-vision flow, as they additionally allow bidirectional attention within visual and text tokens, which may bring extra benefits.

(2) Comparison with other bidirectional attention designs

As suggested, it is also worth exploring the effects of alternative bidirectional prefix attention designs. To this end, under the same training settings, we further compare our design with full-prefix bidirectional attention (PaliGemma) and another variant that additionally enables bidirectional attention among visual tokens based on our setting.

We find that full-prefix bidirectional attention achieves higher performance on some tasks. However, since causal + reverse already enables both directions of cross-modal flow, the remaining gap arises entirely from intra-modal bidirectional attention rather than cross-modal information flow. For example, on CV-Bench 2D, where the gap is most pronounced, the additional gain of full-prefix attention over causal plus reverse (+7.78) comes from bidirectional attention within visual tokens (+3.64) and within text tokens (+4.14). This is consistent with prior findings that bidirectional attention among image tokens alleviates the context starvation of early image tokens under causal masking and improves spatial consistency [8], which benefits spatial tasks such as CV-Bench 2D but cannot be directly credited to cross-modal information flow.

Moreover, our design achieves its gains with far fewer additional attention edges. Let NvN_{v} and NqN_{q} denote the numbers of visual and question tokens, respectively. Our design only adds edges from each visual token to each question token, i.e., NvN_{v} × NqN_{q} edges, whereas bidirectional attention among visual tokens alone adds edges between every pair of visual tokens, i.e., about (Nv)2/2(N_{v})^2/2 edges. Since a question typically contains far fewer tokens than an image (Nq≪NvN_{q} ≪ N_{v}), our design introduces far fewer edges (Nv×Nq≪(Nv)2/2N_{v} × N_{q} ≪ (N_{v})^2/2).

All these further support our choice of the causal plus reverse attention setting for a targeted analysis of language-to-vision information flow and highlight that language-to-vision flow brings notable gains with minimal modification.

Attention settingAdditional attention edges( Nq≪NvN_{q} ≪ N_{v} )RWQACV-Bench 2DMMVPMMStarBLINK (4-task avg.)
Causal–48.6350.7627.3333.2741.17
Full-prefix bidirectional (PaliGemma-style)Nv×Nq+Nv22+Nq22N_{v}\times N_{q}+\frac{N_{v}^{2}}{2}+\frac{N_{q}^{2}}{2}49.9359.9428.6733.6744.38
Causal + reverse + bidirectional in visionNv×Nq+Nv22N_{v}\times N_{q}+\frac{N_{v}^{2}}{2}48.8955.8028.0031.4744.26
Causal + reverseNv×NqN_{v} × N_{q}50.0752.1627.6732.2143.14
AuthorsOpenReview ↗

Rebuttal for Reviewer VxSr (Part3)

Q4: Information-flow patterns in failure samples

We thank the reviewer for this valuable suggestion. Following prior information-flow analyses of MLLMs [3, 4, 5], our original experiments examine information-flow patterns on correctly predicted samples. Examining failure samples can help us further assess the generality of the two-stage collaboration mechanism and better understand these failures.

(1) The two-stage pattern persists in failure samples

We conduct attention-knockout analysis on failure samples of LLaDA-V. As shown in the newly added Figure 14 of the revised manuscript (as figures cannot be directly included in OpenReview responses), these samples exhibit a similar two-stage information-flow pattern, with language-to-vision flow dominating in shallow layers, followed by stronger vision-to-language flow in later layers.

(2) The joint contribution of the two information flows is weaker

We can also observe that the joint knockout of the two cross-modal flows produces noticeably smaller probability drops than in successful samples. This suggests that although a two-stage pattern is present, the joint contribution of the two information flows is weaker, which may help explain the model's incorrect predictions.

W2&Q2: Statistical significance of downstream improvements

We thank the reviewer for this valuable suggestion regarding the statistical reliability of our downstream improvements.

As mentioned above, our primary goal is to examine the benefits of introducing language-to-vision information flow into conventional AR models, rather than seeking a SOTA bidirectional attention design on the multimodal prefix.

We conduct additional experiments with 5 random seeds across different backbones, reporting the mean, standard deviation and the mean score change in the following table. As shown in the results, introducing reverse attention yields general improvements across vision-centric tasks, further demonstrating the benefits of language-to-vision information flow in AR models.

Backbone/EncoderAttention settingRWQA (mean ± SD)CV-Bench 2D (mean ± SD)MMVP (pair; mean ± SD)MMStar (mean ± SD)BLINK (4-task avg.; mean ± SD)
Vicuna-7B-v1.5/CLIPcausal48.86 ± 0.4551.17 ± 0.4928.40 ± 1.2133.73 ± 0.6142.31 ± 0.85
causal + reverse49.99 ± 0.6052.45 ± 0.5628.67 ± 1.0532.29 ± 1.1443.89 ± 0.79
Mean Δ+1.12+1.28+0.27−1.44+1.58
Vicuna-7B-v1.5/SigLIP2causal50.85 ± 0.3657.33 ± 0.4228.13 ± 1.1033.64 ± 0.4844.39 ± 0.50
causal + reverse51.56 ± 0.3157.52 ± 0.4029.33 ± 0.8234.03 ± 0.3546.71 ± 1.16
Mean Δ+0.71+0.19+1.20+0.39+2.32
Qwen2.5-7B/CLIPcausal52.68 ± 0.5555.58 ± 0.3831.60 ± 1.7438.87 ± 0.5446.94 ± 1.05
causal + reverse54.01 ± 0.6356.88 ± 0.4230.93 ± 1.4642.29 ± 1.0848.89 ± 0.78
Mean Δ+1.33+1.31−0.67+3.43+1.95

[1] BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models. ECCV, 2024.

[2] PaliGemma: A Versatile 3B VLM for Transfer. arXiv:2407.07726, 2024.

[3] Cross-modal Information Flow in Multimodal Large Language Models. CVPR, 2025.

[4] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs. ICLR, 2026.

[5] From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs. NeurIPS, 2026.

[6] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. NeurIPS, 2019.

[7] Hierarchical Question-Image Co-Attention for Visual Question Answering. NeurIPS, 2016.

[8] From drop-off to recovery: A mechanistic analysis of segmentation in MLLMs. ECCV, 2026.

We would greatly appreciate any further questions or suggestions you may have regarding our work.

Comments on the paper

2 comments

Dear Reviewers,

The rebuttal period is coming to an end. Please read the authors' responses and update your reviews if appropriate. It would be helpful to share your thoughts as soon as possible to allow enough time for a productive discussion. Thank you!

Best regards, Area Chair

AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We sincerely thank the Area Chair and all reviewers for their thoughtful feedback and careful help. We greatly appreciate the reviewers' recognition of our interesting research question, insightful mechanistic analysis, the solid and useful findings, and clear experimental design.

Our work systematically investigates how bidirectional cross-modal information flow operates and cooperates in MLLMs. We identify a two-stage information-flow pattern in diffusion MLLMs, reveal how language-to-vision flow facilitates visual semantic alignment and subsequent vision-to-language transfer, and further demonstrate the benefits of bidirectional flow in conventional AR models.

The main concerns were the influence of input-order change (hWbL), further evidence and causal relationships of discovered effects (VxSr, 5amF), comparisons with existing bidirectional designs (VxSr, 5amF), and controlled validation and generality of the observed information-flow pattern (5amF, VxSr). We address all these through additional experiments and analyses:

  1. Input order change under causal and bidirectional attention (hWbL). We demonstrate that changing the input order alone improves question-conditioned visual representations but yields no clear overall downstream gains. Bidirectional information flow provides additional benefits under both input orderings, and the two directions can facilitate each other in AR models.
  2. Comparisons with related designs in AR models(VxSr, 5amF). Our design serves as a targeted probe of language-to-vision information flow and demonstrates that it brings gains in vision-centric understanding with minimal modification.
  3. Validation and generality of the info-flow pattern(VxSr, 5amF). Controlled comparison shows that bidirectional attention is a key factor underlying the two-stage information-flow pattern, which also persists in incorrectly answered samples.
  4. Further evidence and causal relationships between effects (VxSr, 5amF). Semantic alignment depends on visual evidence rather than merely incorporating the question keyword, and this alignment contributes to later vision-to-language transfer.

Reviewer VxSr acknowledged that we addressed their concern about visual semantic alignment ability, found our failure-sample analysis useful, and recognized our design as a controlled probe of language-to-vision information flow. We have also carefully addressed the concerns raised by Reviewers hWbL and 5amF and are awaiting their feedback. We would greatly appreciate the reviewers’ and the Area Chair's further consideration of our additional evidence and analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.