acceptodds
Withdrawn from ICLR 2027

SPAR: Self-Referential Prompt-Conditioned Attention Reweighting for VLM Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models often fail on fine-grained visual queries even when the required evidence is present in the image. We show that this failure arises from a decoder-side routing problem rather than a lack of visual perception. Across VLM families, early and final decoder layers fall into a Prompt-Invariant Sink Regime, where attention repeatedly collapses onto stable visual sink tokens across prompts. In contrast, middle layers form a Prompt-Conditioned Evidence Window, where attention is less sink-dominated and more responsive to the input query. Motivated by this depth asymmetry, we introduce SPAR, a self-referential and training-free attention reweighting method for frozen VLMs. SPAR estimates a region of interest from the model's own middle-layer visual attention and uses this prompt-conditioned evidence signal to guide visual attention during decoding. Across multiple benchmarks and VLM families, SPAR improves answer accuracy while increasing evidence retention and reducing mid-to-final evidence drop. Ablations over ROI construction, random controls, and layer choice show that the gains come from prompt-conditioned middle-layer evidence rather than generic attention perturbation or early-layer attention cues. Our results suggest that reliable fine-grained VLM decoding requires preserving evidence-bearing middle-layer signals through later decoder computation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language model, Attention map

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kPmBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fine-grained visual reasoning failures in vision-language models from the perspective of decoder-side attention routing. The authors observe a depth-dependent pattern in which early and final decoder layers exhibit relatively prompt-invariant visual attention sinks, while middle layers are more responsive to the input query. They characterize these two regimes as the Prompt-Invariant Sink Regime (PISR) and the Prompt-Conditioned Evidence Window (PCEW), respectively, using both cross-prompt sink overlap and spectral concentration analyses. Motivated by this observation, the paper proposes SPAR, a training-free, two-pass decoding method that first estimates a visual ROI from the model’s own middle-layer attention and then applies a push-pull bias to visual attention logits during generation in order to preserve this intermediate evidence. Experiments across several VLM backbones and multimodal benchmarks show consistent improvements over the original models and random-ROI controls, together with higher evidence-retention and lower mid-to-final evidence-drop metrics. The paper further provides ROI-source and layer-sweep ablations suggesting that middle layer attention is a more reliable source for constructing the intervention region.

Critical strengths
  1. Clear writing and well-motivated problem formulation. The paper is generally well written, and the motivation is presented clearly. The distinction between prompt-invariant sink behavior and prompt-conditioned middle-layer attention is easy to follow, making the proposed method intuitive and well connected to the diagnostic analysis.

  2. Comprehensive and carefully controlled experiments. The evaluation covers multiple VLM backbones and a broad range of multimodal benchmarks. The paper also provides useful controls and ablations, including random-ROI comparisons, ROI-source ablations, and target-layer sweeps, which help support that the observed gains are tied to middle-layer prompt-conditioned information rather than arbitrary attention perturbation.

Critical weaknesses
  1. The prompt-invariance diagnostic may be confounded by legitimate overlap in visual evidence across prompts. Sink IoU is computed by comparing highly attended visual tokens for different prompts on the same image, and high overlap is interpreted as prompt-invariant sink behavior. However, the paper does not clearly specify how these prompt sets are constructed or whether the prompts are controlled to require spatially distinct evidence. Semantically different questions can still legitimately depend on the same image region, in which case high Sink IoU would not necessarily indicate an undesirable attention sink. A stronger diagnostic would explicitly compare prompts that require disjoint evidence regions, ideally with ground-truth localization.

  2. The paper does not fully establish that PCEW attention corresponds to the truly task-critical evidence. The diagnostic analysis shows that middle-layer attention is less sink-dominated and more prompt-conditioned, but this does not by itself imply that the selected ROI contains the correct causal evidence. An oracle-ROI positive control, where ground-truth evidence regions are available, would help disentangle the quality of the PCEW-based ROI estimator from the effectiveness of the push-pull intervention itself. This is particularly important because the authors acknowledge that SPAR can be less effective when the relevant evidence is not captured by middle-layer attention.

Questions for authors

1. How are the multiple prompts used in the Sink-IoU diagnostic constructed? The paper interprets high overlap of attended visual tokens across prompts for the same image as evidence of prompt-invariant sink behavior. However, different prompts may legitimately require the same visual evidence (e.g., questions about the number, identity, and location of the same object). Could the authors clarify how prompt pairs are selected and whether the diagnostic controls for overlap in the task-relevant visual regions? In particular, do the main PISR results remain when restricting the analysis to prompts that require spatially distinct evidence? This would materially affect how strongly I interpret Sink IoU as measuring an undesirable prompt-invariant sink rather than legitimate evidence reuse.

2. To what extent does the PCEW-derived ROI correspond to truly task-critical visual evidence? The current random, shuffled, and layer-source controls show that middle-layer ROIs are more useful than several alternatives, but being less sink-dominated and more prompt-conditioned does not necessarily imply that these tokens are the correct evidence required for answering the question. Could the authors evaluate the alignment between the PCEW ROI and ground-truth evidence on a subset where localized annotations are available, or provide an oracle-ROI positive control? Such an experiment would separate the quality of the PCEW ROI estimator from the effectiveness of the push-pull intervention itself, and would substantially strengthen—or qualify—the paper’s “evidence-preservation” interpretation.

3. Could the authors more clearly distinguish SPAR from existing hallucination-oriented attention-steering methods? The paper already evaluates POPE and compares against VCD and OPERA in the appendix, but the conceptual boundary with prior attention-steering approaches remains somewhat unclear to me. Is the main novelty of SPAR the use of middle-layer self-referential attention as the ROI source, the depth-specific intervention, the push-pull reweighting mechanism, or the evidence-retention interpretation? It would be helpful to provide a more explicit comparison with hallucination mitigation methods that also strengthen or redirect visual attention, and to clarify whether SPAR is expected to address only hallucinations caused by late-stage loss of already-available visual evidence, or more general hallucination failures as well. If the latter is part of the intended scope, evidence beyond object-existence-style evaluation would affect my assessment of the claimed generality.

If the authors can satisfactorily address these concerns in the rebuttal, I would consider increasing my rating.

Reviewer_QeogOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper discovers that the failure of VLM in fine-grained visual recognition may stem from the fact that information from the intermediate layers of the network is not being passed to the deeper layers, and names this phenomenon the prompt-invariant sink regime. Based on this, this paper proposes a method called SPAR, which predicts the ROI of the task through the attention of the intermediate layers of the network, and then increases the attention of tokens within the ROI by /alpha in selected layers, while decreasing the attention of tokens outside the ROI by /beta. Experiments show that SPAR achieves performance improvements on different benchmarks.

Critical strengths

The proposed method is easy to implement and performs well.

Critical weaknesses

I have two major concerns about this paper.

  1. Refinement of the output of transformers using the attention has been widly dicussed by previous works[1,2,3,4]. However, these related works are neither compared against nor adequately differentiated.

  2. SPAR contains several contradictions. For example, it emphasizes that the performance loss is due to the inability of information from the intermediate layer to reach the final layer. Intuitively, the solution should be to establish a path from the intermediate layer to the output layer, that is, to add the ROI from the intermediate layer to the output layer. However, the current approach involves two forward passes: the first calculates the ROI, and the second injects the ROI into the intermediate layer. This contradicts the paper's proposal and introduces additional overhead, significantly reducing the overall reliability and contribution of the method. Furthermore, the paper claims that the intermediate layer has a Prompt-Conditioned Evidence Window; however, the ablation experiments in Table 3 appear to be not significantly different from selecting the final layer or all layers.

[1] Liu, Shi, Kecheng Zheng, and Wei Chen. "Paying more attention to image: A training-free method for alleviating hallucination in lvlms." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024. [2] Kang, Seil, et al. "See what you are told: Visual attention sink in large multimodal models." International Conference on Learning Representations. Vol. 2025. 2025. [3] Jiang, Zhangqi, et al. "Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [4] Shi, Yuheng, et al. "Catching the details: Self-distilled roi predictors for fine-grained mllm perception." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

This paper contains many strange numerical results.

  1. Many results in Table 1 do not match the official results. For example, the official MME for qwen vl 7b is 2326.8, but the table shows 1552.03. Why is this? There are many other instances of this mismatch; please explain.

  2. In Table 3, the delta acc intermediate layer is +0.925. Under what configuration was this obtained? It seems inconsistent with other tables.

  3. The POPE and MMVP results in Table 1 and Table 14 are the same, but the GQA results do not match. Why is this?

  4. When calculating ρ1\rho_1, was the mean subtracted? Otherwise, there is a high probability that ρ1\rho_1 will be naturally large due to constant offset.

Reviewer_cPmiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how frozen vision-language models allocate visual attention across decoder depth. It identifies more prompt-invariant visual sinks in early and final layers, whereas middle layers exhibit more query-dependent attention, and associates sink overlap with spectral concentration of attention logits. Motivated by these observations, the authors propose SPAR, a training-free, two-pass decoding method. The first pass selects visual tokens using attention averaged over middle layers and heads; the second pass promotes selected tokens and suppresses the remaining visual tokens through attention-logit biases at a single decoder layer. Experiments on four VLM backbones report improvements in benchmark scores and attention-retention metrics, supported by ROI-source ablations, random and shuffled controls, and hyperparameter analyses.

Critical strengths

S1. Clear motivation and a coherent connection between analysis and method. The paper presents an understandable hypothesis about the underuse of query-relevant visual information and translates it into a simple inference-time intervention (Sections 3–4). The motivation, method, and supporting analyses are clearly organized, making the proposed approach easy to understand and assess.

S2. Useful controlled ablations of the ROI source. Table 3 compares early, middle, final, all-layer, single-layer, shuffled, and random ROI sources while keeping the token budget and reweighting rule fixed. These controls provide meaningful evidence that the selected visual-token locations matter and that middle-layer averaging is a promising design choice. Together with the four-backbone evaluation, this strengthens the empirical case beyond an isolated improvement on one model.

Critical weaknesses

W1. The incremental novelty and mechanistic contribution are not sufficiently established. The importance of middle decoder layers for visual information processing and the existence of prompt-insensitive visual attention sinks have already been investigated in closely related work. Section 2 itself acknowledges related depth-dependent observations. The potentially new contributions therefore lie in the spectral characterization and the particular self-referential reweighting procedure. However, SPAR does not directly intervene on the singular directions identified by the analysis, and improved ROI attention-retention metrics do not independently establish that these directions cause evidence loss. This leaves the strongest mechanistic claims insufficiently supported. The paper would benefit from a more precise statement of its advance and a targeted intervention testing the proposed low-rank mechanism.

W2. The comparative evaluation leaves the practical advantage and generality uncertain. The main results primarily compare SPAR with the unmodified model and random ROIs. Appendix G adds VCD and OPERA only on LLaVA-1.5, but omits more directly related attention interventions such as VAR and VGA. These comparisons are important for establishing what SPAR’s additional ROI-estimation pass provides. Although the evaluation includes InternVL3 and NVILA, Qwen3-VL appears only in diagnostic analyses, without corresponding SPAR task-performance results. Architectural changes such as Qwen3-VL’s DeepStack integration make this an unresolved transfer question. A focused comparison with the closest methods, reporting accuracy and latency on a contemporary backbone and a demanding fine-grained perception task, would substantially strengthen the evidence.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.