- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates position-dependent source preference in multi-image vision–language models by analyzing the Key representations of visual boundary tokens. The authors find that early-layer Boundary Keys contain a dominant source-shared component whose RoPE-induced phase deviations can influence source attention. Based on this observation, they propose Content-Bounded Phase Consensus (CBPC), a training-free method that selectively reduces these phase deviations while preserving the cross-source mean. The correction strength is determined adaptively using visual-key-centroid statistics. Experiments across three model families and four multi-image benchmarks show consistent accuracy improvements, along with better robustness to image reordering and modest computational overhead. My main concerns are whether the proposed mechanism sufficiently explains the observed improvements and whether some design choices are adequately justified.
Critical strengths
(1) The paper investigates position-dependent source preference in multi-image VLMs, which is a relevant problem for multi-image understanding.
(2) The analysis of Boundary Keys provides an interesting perspective on how RoPE may influence source attention. In particular, the decomposition into source-shared components and source-specific residuals, together with the counterfactual phase-reassignment experiments, helps characterize the contribution of positional encoding to source preference.
(3) CBPC is relatively simple and does not require additional training or model parameters. The experiments cover nine model configurations from three model families and four benchmarks, with consistent improvements over vanilla inference.
Critical weaknesses
(1) Figure 1 demonstrates asymmetric source attention under identical visual inputs, while Figure 3 shows that reassigning the RoPE phases of the shared Boundary-Key component changes source preference. However, attention asymmetry does not necessarily indicate incorrect source attribution, and the phase-reassignment results are partly expected from the experimental construction.
(2) In Section 4.2, the authors use the cross-source variance of visual-key centroid scores as an upper bound for the RoPE-induced phase variation. However, the rationale behind this choice is not entirely clear. Visual-key-centroid variance may reflect both semantic content differences and positional effects, and it is not obvious why exceeding this reference necessarily indicates excessive phase-induced bias. Furthermore, since the centroids are computed by averaging visual Keys, their statistical properties may depend on image resolution, visual token counts, and token organization across architectures, potentially affecting the adaptive correction strength.
(3) The authors apply CBPC to the first two decoder layers, and Table 6 shows that this configuration outperforms other layer selections. However, it remains unclear why these two layers are particularly effective. Is this related to the stronger source-shared component in early layers? Examining how the shared component and RoPE-induced phase deviation vary across layers could help explain these results. Furthermore, although the method avoids searching over correction strengths, the choice of the first two layers still appears heuristic.
(4) Table 4 reports modest LM forward-time overhead (2.01%–8.33%) and negligible additional peak memory usage. However, the evaluation uses only 100 Mantis samples and reports LM forward time rather than end-to-end inference latency. Since CBPC requires computing visual-key centroids and QK variance statistics, its overhead may increase with the number of images, visual tokens, and context length. Reporting prefill and end-to-end latency under different input scales would provide a clearer picture of the method's practical efficiency.
Questions for authors
Please refer to the Critical Weaknesses section.
2 comments
We sincerely thank you for taking the time to revisit our response and reconsider your assessment. We are truly grateful for your thoughtful and constructive feedback, which has helped us clarify CBPC's motivation and strengthen our experimental analysis. We will carefully incorporate these additional findings and clarifications into the revised manuscript. Thank you once again for your time, careful consideration, and encouraging feedback.
Thank you for your detailed responses and clarifications. My main concerns have now been largely addressed. After also considering the comments and discussions from other reviewers, I am happy to increase my score. I appreciate your efforts throughout the rebuttal process and wish you the best of luck!