acceptodds
Under review as a conference paper at ICLR 2027

Content-Bounded Phase Consensus: Mitigating Position-Driven Source Preference in Multi-Image Vision-Language Models

Abstract

Multi-image vision–language models (VLMs) must associate visual evidence with its source, yet their predictions can depend on source position. We study this behavior through Boundary Keys, the Key representations of model-native tokens delimiting visual sources. Across representative models from Qwen3-VL, InternVL3.5, and Molmo2-O, identical visual inputs exhibit position-dependent source attention, while early-layer Boundary Keys contain a dominant source-shared component. RoPE maps this shared component to source-dependent realizations, and counterfactual phase reassignment changes boundary source preference even when native source-specific residuals are retained. Motivated by this analysis, we propose Content-Bounded Phase Consensus (CBPC), a lightweight training-free intervention that selectively contracts the RoPE-induced phase deviation while preserving the cross-source mean. CBPC derives an input- and layer-dependent correction strength from native visual-key-centroid statistics without additional parameters, answer labels, or strength search. Across Mantis, MuirBench, MIRB, and QBench2, CBPC yields consistent gains, improves correctness under meaning-preserving image reorderings, and incurs modest LM forward overhead with negligible additional peak memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Image Vision-Language Models, Positional Bias, Attention Routing, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NBh5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates position-dependent source preference in multi-image vision–language models by analyzing the Key representations of visual boundary tokens. The authors find that early-layer Boundary Keys contain a dominant source-shared component whose RoPE-induced phase deviations can influence source attention. Based on this observation, they propose Content-Bounded Phase Consensus (CBPC), a training-free method that selectively reduces these phase deviations while preserving the cross-source mean. The correction strength is determined adaptively using visual-key-centroid statistics. Experiments across three model families and four multi-image benchmarks show consistent accuracy improvements, along with better robustness to image reordering and modest computational overhead. My main concerns are whether the proposed mechanism sufficiently explains the observed improvements and whether some design choices are adequately justified.

Critical strengths

(1) The paper investigates position-dependent source preference in multi-image VLMs, which is a relevant problem for multi-image understanding.

(2) The analysis of Boundary Keys provides an interesting perspective on how RoPE may influence source attention. In particular, the decomposition into source-shared components and source-specific residuals, together with the counterfactual phase-reassignment experiments, helps characterize the contribution of positional encoding to source preference.

(3) CBPC is relatively simple and does not require additional training or model parameters. The experiments cover nine model configurations from three model families and four benchmarks, with consistent improvements over vanilla inference.

Critical weaknesses

(1) Figure 1 demonstrates asymmetric source attention under identical visual inputs, while Figure 3 shows that reassigning the RoPE phases of the shared Boundary-Key component changes source preference. However, attention asymmetry does not necessarily indicate incorrect source attribution, and the phase-reassignment results are partly expected from the experimental construction.

(2) In Section 4.2, the authors use the cross-source variance of visual-key centroid scores as an upper bound for the RoPE-induced phase variation. However, the rationale behind this choice is not entirely clear. Visual-key-centroid variance may reflect both semantic content differences and positional effects, and it is not obvious why exceeding this reference necessarily indicates excessive phase-induced bias. Furthermore, since the centroids are computed by averaging visual Keys, their statistical properties may depend on image resolution, visual token counts, and token organization across architectures, potentially affecting the adaptive correction strength.

(3) The authors apply CBPC to the first two decoder layers, and Table 6 shows that this configuration outperforms other layer selections. However, it remains unclear why these two layers are particularly effective. Is this related to the stronger source-shared component in early layers? Examining how the shared component and RoPE-induced phase deviation vary across layers could help explain these results. Furthermore, although the method avoids searching over correction strengths, the choice of the first two layers still appears heuristic.

(4) Table 4 reports modest LM forward-time overhead (2.01%–8.33%) and negligible additional peak memory usage. However, the evaluation uses only 100 Mantis samples and reports LM forward time rather than end-to-end inference latency. Since CBPC requires computing visual-key centroids and QK variance statistics, its overhead may increase with the number of images, visual tokens, and context length. Reporting prefill and end-to-end latency under different input scales would provide a clearer picture of the method's practical efficiency.

Questions for authors

Please refer to the Critical Weaknesses section.

2 comments

Thank you for your detailed responses and clarifications. My main concerns have now been largely addressed. After also considering the comments and discussions from other reviewers, I am happy to increase my score. I appreciate your efforts throughout the rebuttal process and wish you the best of luck!

We sincerely thank you for taking the time to revisit our response and reconsider your assessment. We are truly grateful for your thoughtful and constructive feedback, which has helped us clarify CBPC's motivation and strengthen our experimental analysis. We will carefully incorporate these additional findings and clarifications into the revised manuscript. Thank you once again for your time, careful consideration, and encouraging feedback.

Reviewer_SKZNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies position-dependent source preference in multi-image VLMs. The key observation is that Boundary Keys from different image sources have a dominant shared component before RoPE. However, RoPE applies different position-dependent rotations to this shared component, introducing source-specific variations in the resulting Boundary-Key representations, which can eventually influence the VLM's source preference. Based on this insight, the authors propose Content-Bounded Phase Consensus (CBPC), a training-free approach that adaptively corrects these RoPE-induced deviations using visual-key centroid statistics. Experiments across multiple VLMs and multi-image benchmarks show that CBPC generally outperforms vanilla VLMs, DTS and FOCUS, and CBPC also improves robustness to image-order variations.

Critical strengths
  • The paper provides an interesting analysis of how RoPE-induced variations in the Boundary Keys at the first decoding layer can impact source preference in multi-image VLMs.
  • The above observation also motivates a clean and lightweight design of CBPC, which adaptively corrects these variations without additional training.
Critical weaknesses
  • The motivation/intuition behind the visual-key-centroid reference is not well justified. It remains unclear to me why the source-wise QK variance of visual-key centroids is empirically chosen to be the upper bound for the RoPE-induced phase variation of Boundary Keys.
  • The analysis of this RoPE-induced variances is only done at the first layer. It is unsure if this still holds in deeper layers. Moreover, the evaluation in Appendix shows that CBPC provides smaller improvements when applied to middle or later layers, which would benefit from further analysis.
  • From the ablation, it seems that the benefit of adaptive correction is somewhat limited over simpler alternatives, such as Equal-phase and Full consensus. More elaboration on when and why the adaptive correction is necessary would be helpful.
  • It would be interesting to see a deeper analysis of how RoPE induced variances in Boundary Keys affect attention to actual visual tokens and ultimately influence source preferences and predictions.
Questions for authors

See Critical Weaknesses.

Reviewer_bi3LOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Content-Bounded Phase Consensus (CBPC), a training-free method to mitigate position-dependent source preference in multi-image vision-language models. The authors identify that RoPE-induced phase deviations in Boundary Keys contribute to source preference and propose an adaptive correction mechanism that selectively reduces these deviations while preserving source-specific information. Experiments across multiple models and benchmarks demonstrate consistent performance improvements and improved robustness to image reordering.

Critical strengths
  1. No Hyperparameter Tuning: The method does not require hyperparameter tuning, as the correction strength is determined adaptively.

  2. Consistent Performance Improvements: The proposed method demonstrates consistent improvements across multiple models and benchmarks.

  3. Well-Motivated Approach: The method is motivated by an analysis of position-dependent source preference and RoPE-induced phase deviations in Boundary Keys.

Critical weaknesses
  1. I am curious about how RoPE-induced position bias contributes to actual prediction errors. For example, in cases where vanilla inference produces incorrect predictions due to image reordering but CBPC yields correct predictions, does vanilla inference exhibit a Boundary Key attention preference toward an incorrect image? Does applying CBPC shift this preference toward the correct target image? Conversely, how does the attention preference change in cases where CBPC degrades prediction accuracy? Such an analysis would help clarify the relationship between the proposed RoPE-induced position bias and actual prediction errors, as well as whether the performance improvements of CBPC are indeed attributable to mitigating this bias.

  2. Computational Cost: How does the computational overhead of CBPC scale with the number of images and visual tokens?

  3. Qualitative Results: More qualitative examples are needed to demonstrate how CBPC improves predictions compared to vanilla inference.

I will consider adjusting my score based on the authors' rebuttal response.

Questions for authors

Baseline Hyperparameters: How were the hyperparameters for DTS and FOCUS selected, particularly for model and benchmark settings not covered in their original papers?

2 comments

Thank you for the detailed rebuttal and additional experiments. The authors have addressed most of my concerns, particularly regarding computational overhead, qualitative examples, and baseline configurations. Although the relationship between attention bias and prediction errors could be further investigated, the additional TV-distance analysis provides useful supporting evidence.

Overall, I appreciate the authors' efforts and have decided to raise my score.

Thank you for your response and for reconsidering your assessment. We appreciate your feedback and will incorporate the additional analyses into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.