- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies fine-grained visual reasoning failures in vision-language models from the perspective of decoder-side attention routing. The authors observe a depth-dependent pattern in which early and final decoder layers exhibit relatively prompt-invariant visual attention sinks, while middle layers are more responsive to the input query. They characterize these two regimes as the Prompt-Invariant Sink Regime (PISR) and the Prompt-Conditioned Evidence Window (PCEW), respectively, using both cross-prompt sink overlap and spectral concentration analyses. Motivated by this observation, the paper proposes SPAR, a training-free, two-pass decoding method that first estimates a visual ROI from the model’s own middle-layer attention and then applies a push-pull bias to visual attention logits during generation in order to preserve this intermediate evidence. Experiments across several VLM backbones and multimodal benchmarks show consistent improvements over the original models and random-ROI controls, together with higher evidence-retention and lower mid-to-final evidence-drop metrics. The paper further provides ROI-source and layer-sweep ablations suggesting that middle layer attention is a more reliable source for constructing the intervention region.
Critical strengths
-
Clear writing and well-motivated problem formulation. The paper is generally well written, and the motivation is presented clearly. The distinction between prompt-invariant sink behavior and prompt-conditioned middle-layer attention is easy to follow, making the proposed method intuitive and well connected to the diagnostic analysis.
-
Comprehensive and carefully controlled experiments. The evaluation covers multiple VLM backbones and a broad range of multimodal benchmarks. The paper also provides useful controls and ablations, including random-ROI comparisons, ROI-source ablations, and target-layer sweeps, which help support that the observed gains are tied to middle-layer prompt-conditioned information rather than arbitrary attention perturbation.
Critical weaknesses
-
The prompt-invariance diagnostic may be confounded by legitimate overlap in visual evidence across prompts. Sink IoU is computed by comparing highly attended visual tokens for different prompts on the same image, and high overlap is interpreted as prompt-invariant sink behavior. However, the paper does not clearly specify how these prompt sets are constructed or whether the prompts are controlled to require spatially distinct evidence. Semantically different questions can still legitimately depend on the same image region, in which case high Sink IoU would not necessarily indicate an undesirable attention sink. A stronger diagnostic would explicitly compare prompts that require disjoint evidence regions, ideally with ground-truth localization.
-
The paper does not fully establish that PCEW attention corresponds to the truly task-critical evidence. The diagnostic analysis shows that middle-layer attention is less sink-dominated and more prompt-conditioned, but this does not by itself imply that the selected ROI contains the correct causal evidence. An oracle-ROI positive control, where ground-truth evidence regions are available, would help disentangle the quality of the PCEW-based ROI estimator from the effectiveness of the push-pull intervention itself. This is particularly important because the authors acknowledge that SPAR can be less effective when the relevant evidence is not captured by middle-layer attention.
Questions for authors
1. How are the multiple prompts used in the Sink-IoU diagnostic constructed? The paper interprets high overlap of attended visual tokens across prompts for the same image as evidence of prompt-invariant sink behavior. However, different prompts may legitimately require the same visual evidence (e.g., questions about the number, identity, and location of the same object). Could the authors clarify how prompt pairs are selected and whether the diagnostic controls for overlap in the task-relevant visual regions? In particular, do the main PISR results remain when restricting the analysis to prompts that require spatially distinct evidence? This would materially affect how strongly I interpret Sink IoU as measuring an undesirable prompt-invariant sink rather than legitimate evidence reuse.
2. To what extent does the PCEW-derived ROI correspond to truly task-critical visual evidence? The current random, shuffled, and layer-source controls show that middle-layer ROIs are more useful than several alternatives, but being less sink-dominated and more prompt-conditioned does not necessarily imply that these tokens are the correct evidence required for answering the question. Could the authors evaluate the alignment between the PCEW ROI and ground-truth evidence on a subset where localized annotations are available, or provide an oracle-ROI positive control? Such an experiment would separate the quality of the PCEW ROI estimator from the effectiveness of the push-pull intervention itself, and would substantially strengthen—or qualify—the paper’s “evidence-preservation” interpretation.
3. Could the authors more clearly distinguish SPAR from existing hallucination-oriented attention-steering methods? The paper already evaluates POPE and compares against VCD and OPERA in the appendix, but the conceptual boundary with prior attention-steering approaches remains somewhat unclear to me. Is the main novelty of SPAR the use of middle-layer self-referential attention as the ROI source, the depth-specific intervention, the push-pull reweighting mechanism, or the evidence-retention interpretation? It would be helpful to provide a more explicit comparison with hallucination mitigation methods that also strengthen or redirect visual attention, and to clarify whether SPAR is expected to address only hallucinations caused by late-stage loss of already-available visual evidence, or more general hallucination failures as well. If the latter is part of the intended scope, evidence beyond object-existence-style evaluation would affect my assessment of the claimed generality.
If the authors can satisfactorily address these concerns in the rebuttal, I would consider increasing my rating.