acceptodds
Withdrawn from ICLR 2027

Beyond Attention Black Holes: Energy Landscape Guidance for Rectifying Feature Stagnation in VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models have achieved remarkable success in multimodal reasoning, yet they remain surprisingly vulnerable to low-level visual distraction. In this work, we identify and formalize a ubiquitous but overlooked phenomenon: Attentional Drift induced by artificial image padding that corresponds to redundant regions in natural scenes. We observe that semantically vacant black pixels and low-entropy background patches often act as attentional black holes, disproportionately absorbing the model's focus and leading to severe reasoning degradation. Our investigation reveals that this drift is triggered by a Dual Trap inherent in the Transformer architecture: 1) a positional bias toward the end of the sequence, and 2) an attention sink induced by feature stagnation, where stable and uninformative tokens act as attention sinks. To mitigate this universal issue, we propose Energy Landscape Guidance (ELG), a training-free, plug-and-play framework that conceptualizes the evolution of hidden states as trajectories within a high-dimensional manifold. ELG introduces an Energy Scoring Module that leverages spectral analysis to capture evolution flux, thereby isolating various forms of pathological tokens without the need for prior masks. This is followed by a Latent Rectification Module that steers hidden states toward high-information semantic basins via gradient-based intervention during inference. Extensive evaluations on VMCBench demonstrate that ELG consistently rehabilitates various VLMs from visual illusions. Notably, Qwen2.5-VL-7B equipped with ELG outperforms 72B-scale competitors and proprietary models like GPT-4o.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Multimodal Reasoning, Attention Mechanisms, Visual Grounding, Hidden-State Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_waG1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the pervasive attention drift problem in VLMs and attributes its root cause to the Dual Trap of Transformers: raster scan tokenization places uninformative tokens at the sequence tail, autoregressive training induces recency bias, and feature stagnation arising from uninformative regions produces stable keys that trigger the attention sink effect to rob valid objects of considerable attention; to tackle this problem, it proposes Energy Landscape Guidance, which regards layer wise hidden-state evolution as trajectories on a high dimensional manifold, builds a composite energy function to identify pathological tokens, and rectifies hidden-states via single-step gradient based intervention to steer representations toward high-information regions.

Critical strengths
  1. It analyzes the pervasive attention‑drift issue in VLMs and identifies its root cause as the Transformer’s Dual Trap: raster-scan tokenization places uninformative tokens at the tail of the sequence, autoregressive training induces recency bias, and feature stagnation in uninformative regions yields stable keys that trigger the attention-sink effect, which siphons substantial attention away from valid objects.
  2. The proposed ELG is training-free and plug-and-play at inference time, granting it higher practical value compared with baselines relying on segmentation-oriented pre-training or fine-tuning.
  3. Theoretical derivations and hyperparameter analyses are provided.
Critical weaknesses
  1. The paper attributes the performance drop to sequence-position bias and feature stagnation induced by black padding regions, and accordingly proposes the Dual Trap hypothesis. However, the existing experiments mainly demonstrate the performance drop caused by padded images and the performance recovery after ELG intervention; they do not sufficiently prove that these two mechanisms are indeed the main causes of the performance drop. For example, the results in Table 1 only show that accuracy decreases after adding black borders.
  2. The experiments almost exclusively rely on VMCBench, a single multiple-choice benchmark. The comparisons with various external models in Table 2 do not specify whether they were rerun under the same prompt, parsing, preprocessing, and evaluation protocol.
  3. ELG combines frequency-domain changes in hidden states, token sequence positions, and cosine distance from the original representation into an energy function, and modifies hidden states via gradient descent. However, the paper does not prove that a stable causal relationship exists between these metrics and harmful attention: small hidden-state changes do not necessarily imply insufficient semantic information, later sequence positions do not necessarily imply that a token will produce harmful attention bias, and the cosine-distance constraint only limits semantic drift; it does not guarantee that the correction direction points to genuinely useful visual information.
Questions for authors
  1. Can causal evidence be provided to demonstrate that positional bias and feature stagnation are the primary causes of attentional drift, rather than merely correlating with performance degradation?
  2. Can the quantitative performance gains of this method be reproduced on diverse low-information regions, such as blurred backgrounds and repetitive textures? Could more data be adopted in experiments to verify its generalization?
  3. Do the practical benefits brought by ELG justify its additional inference cost? Under identical hardware and inference settings, what are the end-to-end latency and memory overhead of ELG compared with the original unmodified model?
Reviewer_unJ6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies attention drift in vision-language models (VLMs), motivated by the performance degradation observed when black borders are added to images. The authors attribute this behavior to a "Dual Trap" involving end-of-sequence positional bias and feature stagnation. They propose Energy Landscape Guidance (ELG), a training-free inference-time method that combines spectral measures of layer-wise hidden-state evolution, a position-dependent prior, and a semantic-consistency term into an energy objective for hidden-state rectification. Experiments on VMCBench report accuracy improvements across several VLM backbones.

Critical strengths
  • The paper addresses a relevant problem concerning visual attention and robustness in VLMs.
  • ELG does not require parameter updates and can be applied to pretrained VLMs at inference time.
  • The evaluation covers multiple VLM architectures and includes comparisons with existing methods, component ablations, and hyperparameter sensitivity analyses.
Critical weaknesses

1. The proposed attention-drift phenomenon and its underlying mechanism are not adequately validated.

A central claim is that feature-stagnant visual tokens act as harmful attention sinks, particularly when combined with positional bias. However, the experiments primarily report downstream accuracy rather than direct measurements of this phenomenon. Table 1 shows accuracy degradation under letterboxing for four VLMs, while Table 5 reports gains from ELG on five backbones. These results do not establish that the same feature-stagnation and attention-sink behaviors occur across these architectures. The qualitative analyses in Fig. 3 and Appendix Figs. 5–6 are limited to selected examples, mainly using Qwen2.5-VL-7B. There is no systematic cross-model analysis of attention assigned to padding or background tokens, layer-wise evolution flux, or their relationship to prediction errors. Furthermore, letterboxing can also alter foreground scale and visual-token distribution, and these effects are not separated from the proposed positional and stagnation mechanisms. The extension from artificial padding to redundant regions in natural images is therefore not sufficiently supported. This limits the evidence for the paper's central claim of identifying a general failure mechanism in VLMs.

2. The semantic-consistency term is difficult to reconcile with the stated optimization procedure.

Eq. (8) defines semantic consistency as the cosine distance between a hidden state and its pre-rectification counterpart. Yet Eq. (10) specifies a single gradient step starting from that same pre-rectification state. At this point, the gradient of the cosine-distance term with respect to the current hidden state is zero, so this term should not directly affect the update as written. Nevertheless, Table 3 reports an improvement from 88.23% to 88.60% after adding semantic consistency, and Appendix Table 9 reports sensitivity to its coefficient. The manuscript does not explain how these results arise under the stated formulation. This apparent inconsistency between the optimization procedure and the reported ablations raises a substantive concern about the technical validity of this component.

3. The contribution and necessity of the energy-guided formulation are insufficiently established.

Visual attention sinks and training-free attention correction have been investigated in prior work, including See What You Are Told (Kang et al., ICLR 2025). More directly, Attention Debiasing for Token Pruning (Zhao et al., 2025) studies recency bias and attention to semantically empty padding tokens, albeit in a token-pruning setting. The paper does not sufficiently distinguish the proposed feature-stagnation explanation from these existing accounts. Moreover, Eqs. (3)–(4) sum the squared magnitudes of all nonzero DFT components, which, by Parseval's identity, amounts to a monotonic transformation of centered layer-wise feature variation. The advantage over simpler variance or feature-difference statistics is not demonstrated. Likewise, the "boundary-aware" prior depends on sequential position rather than explicit image-boundary information. While Appendix A shows that the proposed gradient step can locally decrease the constructed energy, it does not establish that decreasing this energy reduces harmful attention or improves semantic grounding. The reported gains therefore do not clearly isolate the benefit of the proposed formulation over simpler interventions.

Minor Comments

  • The implementation should specify the evolution-flux window size KK, the procedure for estimating the adaptive threshold τ\tau, and the smoothing parameter δ\delta used in Eqs. (3)–(7).
  • The attention visualizations in Fig. 3 and Appendix Figs. 5–6 should clarify how attention maps are extracted and aggregated across layers and heads.
  • The efficiency evaluation in Fig. 4 should clarify the timing protocol and distinguish aggregate throughput from end-to-end per-sample latency, particularly given the additional gradient computations.
Questions for authors
  1. Can the authors provide quantitative evidence that low evolution flux is associated with excessive attention to uninformative regions and with prediction errors, including on natural images and across the different VLM backbones? How are the effects of feature stagnation and positional bias disentangled from changes introduced by letterboxing?

  2. How exactly is the semantic-consistency term used in the single-step rectification? Given Eqs. (8) and (10), its gradient appears to vanish at the stated initialization. What implementation detail explains the improvements in Table 3 and the sensitivity to its weight in Appendix Table 9?

  3. What does the evolution-flux formulation add beyond prior attention-sink analyses [1, 2] and simpler layer-wise variance or feature-difference statistics? Can the authors isolate the benefit of the proposed energy-guided update over these alternatives?

Reviewer_S4jrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper identifies "attentional drift" in VLMs: semantically vacant black padding pixels (letterboxing) and low-entropy background regions disproportionately absorb attention, degrading reasoning. It attributes this to a "Dual Trap": (1) an end-of-sequence positional bias, since bottom padding falls at the tail of the rasterized visual token sequence, and (2) a feature-stagnation-induced attention sink, since padding hidden states barely change across layers (Δh ≈ 0). To mitigate it, the paper proposes Energy Landscape Guidance (ELG), a training-free, plug-and-play framework with two modules. The Energy Scoring Module defines a composite pathological energy E = α·E_stag + β·Ψ_geo + γ·Ω_cons, where E_stag = −log(Φ_flux + ε) converts a DFT-based layer-wise evolution flux Φ_flux over the last K hidden states into a stagnation penalty, Ψ_geo = s_t·ln(1+pos(t)) weights stagnant tokens by sequential position through a differentiable sigmoid gate s_t = σ((τ−Φ_flux)/δ), and Ω_cons = 1 − cos(h, ĥ) anchors rectified states to the original ones. The Latent Rectification Module applies a single gradient step h* = h − η·∇E per layer at inference. The default configuration is α=1.0, β=0.5, γ=0.2, η=0.02, tuned on Qwen2.5-VL-7B and kept fixed elsewhere. Evaluation is on VMCBench (9,450 multiple-choice samples from 20 VQA datasets, four dimensions: General, Reasoning, OCR, Doc & Chart; accuracy metric). Key reported results: (i) pilot study (Table 1): letterboxing drops accuracy by 10.7–15.8 points across four VLMs, and ELG recovers +10.1 to +11.9 points on letterboxed inputs (e.g., Qwen2.5-VL-7B 73.2 → 84.9); (ii) main comparison (Table 2, mean±std over three runs): ELG on Qwen2.5-VL-7B reaches 88.60±0.32 vs. 83.90±0.68 base, exceeding Qwen2-VL-72B (87.00), Qwen2.5-VL-14B (87.20), GPT-4o (82.90) and Claude-3.5-Sonnet (80.53), with per-dimension gains +4.6/+5.9/+1.9/+6.4 and a 97.2 OCR score; (iii) ablation (Table 3): 83.90 → 86.78 (+E_stag) → 88.23 (+Ψ_geo) → 88.60 (full); (iv) comparison with attention-correction methods on the same backbone (Table 4): 88.60 vs. 87.80 best baseline; (v) cross-architecture transfer of one fixed configuration (Table 5): +4.7 to +6.2 points (average +5.6) over five backbones; (vi) efficiency: 62 imgs/sec peak throughput and 0.016 s/sample latency, claimed Pareto-optimal. The paper concludes that ELG sets a new state-of-the-art on VMCBench and lets a 7B model surpass 72B-scale and proprietary models.

Critical strengths

S1. The pilot study (Table 1) quantifies letterbox-induced degradation consistently across four architectures (drops of −15.8/−11.7/−11.6/−10.7 points for LLaVA-1.5-7B, Phi-3-Vision, Qwen2-VL-7B, Qwen2.5-VL-7B) and shows ELG recovers most of it on padded inputs (+11.9/+10.1/+10.9/+11.7; e.g., Qwen2.5-VL-7B 73.2 → 84.9). This matters for the acceptance decision because, if the reporting inconsistencies listed below are resolved, it documents a cheap, deployment-relevant failure mode of standard preprocessing together with a training-free mitigation — a finding the community can use immediately. S2. The method transfers as a single fixed configuration (α,β,γ,η = 1.0,0.5,0.2,0.02, tuned once on Qwen2.5-VL-7B) to five heterogeneous backbones with uniform gains (Table 5: +5.8/+6.2/+6.1/+4.7/+5.3, average +5.6), and the sensitivity tables (Tables 6–9) show broad performance plateaus rather than knife-edge optima (e.g., η∈{0.01,0.02,0.05} gives 87.2/88.6/87.8). This matters because plug-and-play transferability without per-model tuning is the paper's central practical claim; if reproducible, it is a substantive strength over training-based grounding methods.

Critical weaknesses

W-A. The experimental record is internally inconsistent and partially unverifiable. Table 1's caption states accuracy is "averaged over the spatial-reasoning tasks in the benchmark," yet its Original / Original+ELG columns are digit-identical to the full-VMCBench averages in Table 2 and Table 5 (LLaVA-1.5-7B 58.2/64.0; Qwen2-VL-7B 80.9/87.1; Qwen2.5-VL-7B 83.9/88.6) — a subset mean and a full-benchmark mean cannot coincide to 0.1% for six values across three models, so at least one caption is false and the motivating pilot has no auditable protocol (letterbox geometry never specified). Table 4, which carries the state-of-the-art claim, lists VAR and LGS with no citation or definition anywhere in the paper (each string occurs exactly once, in the table), and attributes R-VLMs to You et al. 2023 (Ferret) and AGE-VLMs to Cai et al. 2024 (VIP-LLaVA), neither of which matches the AGE-VLM citation list in Section 2.2; the Reproducibility Statement promises "implementation settings, the VMCBench evaluation protocol" in Appendix C, but Appendix C is "Limitations and Future Work," and K, τ's estimation rule, δ, ε, the rectified layer set, pos(t)'s reference frame, and the decoding/answer-extraction protocol never appear. This is fundamental rather than fixable-in-rebuttal in the sense that every headline number currently rests on an unauditable record; it drives Soundness to 1 and is the main reason for the rejection recommendation. W-B. The efficiency and statistical claims do not survive arithmetic or the paper's own figure. The reported "minimal latency of 0.016s/sample" is exactly the reciprocal of the reported "peak throughput of 62 imgs/sec" (1/62 ≈ 0.016), i.e., an amortized throughput-derived quantity presented as latency, and Figure 4 (verified in the PDF) plots only {Conscious Gaze, Direct Visual Grounding, AGE-VLM, R-VLM, ELG} — the unmodified base model is absent even though LRM adds a per-layer gradient computation — while ELG sits at the highest GFLOPS in the third panel (~150–155 vs. ~115–135 for all baselines), contradicting "negligible computational overhead." Statistically, only Table 2 reports variance (mean±std over three runs); Tables 3–5 are single numbers, the final component's gain (Ω_cons: 88.23 → 88.60, +0.37) is inside Table 2's own reported std (±0.28–0.32), and Table 3's OCR regresses 97.8 → 97.2 when Ω_cons is added without comment. This is fixable by releasing the already-computed per-run values plus base-model overhead numbers, but as written two of the paper's selling points (efficiency, consistent component-wise gains) are unsupported, which caps the rating.

Questions for authors
  1. (clarification) Table 1's caption says accuracies are averaged over "the spatial-reasoning tasks in the benchmark," but its Original and Original+ELG columns are identical to the full-VMCBench averages in Tables 2 and 5 (58.2/64.0, 80.9/87.1, 83.9/88.6). Which scope is correct? If Table 1 is a subset, please give the subset's per-model numbers and define the subset; if it is the full benchmark, please correct the caption. Also specify the letterboxing protocol (border thickness / target aspect ratio / resizing) used in the pilot.
  2. (material) Please define VAR and LGS in Table 4 (full method names, citations, inference-time vs. trained) and describe how each was re-implemented on Qwen2.5-VL-7B, including any training budget; and reconcile the Table 4 attributions R-VLMs→You et al. 2023 (Ferret) and AGE-VLMs→Cai et al. 2024 (VIP-LLaVA) with Section 2.2, where the AGE-VLM citation list does not include Cai et al. 2024.
  3. (material) Please provide the missing implementation values: flux window K, the rule estimating τ from the flux distribution (mean? percentile?), δ, ε, the set of layers at which LRM is applied, and whether pos(t) is relative to the visual-token block or the full sequence; plus the VMCBench protocol (prompt template, multiple-choice answer extraction, decoding settings) and what varies across the "three independent runs."
  4. (clarification) Please report the unmodified Qwen2.5-VL-7B (no ELG) throughput, latency, and GFLOPS on the same 8×H20 setup, state whether 0.016 s/sample is the reciprocal of the 62 imgs/sec peak throughput (amortized) or a measured per-sample latency, and reconcile Figure 4's GFLOPS panel — where ELG is the most expensive method (~150–155 vs. ~115–135) — with the claim of "negligible computational overhead."
  5. (clarification) In Table 3, OCR drops from 97.8 (+Geometric Prior) to 97.2 (Full) and the Ω_cons gain (+0.37 Avg.) is within the ±0.28–0.32 std reported in Table 2. Please provide the per-configuration mean±std from the runs you already have, or justify retaining Ω_cons.
  6. (clarification) For the proprietary API models in Table 2 (GPT-4o ±0.19, Claude-3.5-Sonnet ±0.26), what varies across the three runs? If decoding is greedy, the std should be ~0; if sampling is used, please state the temperature.
  7. (material) Please identify the exact public release of VMCBench corresponding to Zhang et al. (2025b), give the per-category sample counts underlying the unweighted four-way "Avg.", and define the "spatial-reasoning tasks" subset referenced in Table 1's caption.
  8. (clarification) Were any counterfactual tests of the "Dual Trap" performed or observed — e.g., padding at the top/left instead of the bottom, non-black (gray/noise) padding, permuted padding positions, or quantitative attention mass on padding tokens? If such data exist, please share the numbers; if not, please acknowledge that the mechanism claim currently rests on qualitative attention maps (Figs. 1, 3, 5, 6) only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.