- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the pervasive attention drift problem in VLMs and attributes its root cause to the Dual Trap of Transformers: raster scan tokenization places uninformative tokens at the sequence tail, autoregressive training induces recency bias, and feature stagnation arising from uninformative regions produces stable keys that trigger the attention sink effect to rob valid objects of considerable attention; to tackle this problem, it proposes Energy Landscape Guidance, which regards layer wise hidden-state evolution as trajectories on a high dimensional manifold, builds a composite energy function to identify pathological tokens, and rectifies hidden-states via single-step gradient based intervention to steer representations toward high-information regions.
Critical strengths
- It analyzes the pervasive attention‑drift issue in VLMs and identifies its root cause as the Transformer’s Dual Trap: raster-scan tokenization places uninformative tokens at the tail of the sequence, autoregressive training induces recency bias, and feature stagnation in uninformative regions yields stable keys that trigger the attention-sink effect, which siphons substantial attention away from valid objects.
- The proposed ELG is training-free and plug-and-play at inference time, granting it higher practical value compared with baselines relying on segmentation-oriented pre-training or fine-tuning.
- Theoretical derivations and hyperparameter analyses are provided.
Critical weaknesses
- The paper attributes the performance drop to sequence-position bias and feature stagnation induced by black padding regions, and accordingly proposes the Dual Trap hypothesis. However, the existing experiments mainly demonstrate the performance drop caused by padded images and the performance recovery after ELG intervention; they do not sufficiently prove that these two mechanisms are indeed the main causes of the performance drop. For example, the results in Table 1 only show that accuracy decreases after adding black borders.
- The experiments almost exclusively rely on VMCBench, a single multiple-choice benchmark. The comparisons with various external models in Table 2 do not specify whether they were rerun under the same prompt, parsing, preprocessing, and evaluation protocol.
- ELG combines frequency-domain changes in hidden states, token sequence positions, and cosine distance from the original representation into an energy function, and modifies hidden states via gradient descent. However, the paper does not prove that a stable causal relationship exists between these metrics and harmful attention: small hidden-state changes do not necessarily imply insufficient semantic information, later sequence positions do not necessarily imply that a token will produce harmful attention bias, and the cosine-distance constraint only limits semantic drift; it does not guarantee that the correction direction points to genuinely useful visual information.
Questions for authors
- Can causal evidence be provided to demonstrate that positional bias and feature stagnation are the primary causes of attentional drift, rather than merely correlating with performance degradation?
- Can the quantitative performance gains of this method be reproduced on diverse low-information regions, such as blurred backgrounds and repetitive textures? Could more data be adopted in experiments to verify its generalization?
- Do the practical benefits brought by ELG justify its additional inference cost? Under identical hardware and inference settings, what are the end-to-end latency and memory overhead of ELG compared with the original unmodified model?