- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how query phrasing affects the stability of Vision-Language Models (VLMs) under image perturbations, reporting two opposing effects: wrapping queries in polite, verbose phrasing reduces perturbation-induced drift in the gold-answer log-likelihood, whereas semantically more complex questions exhibit greater sensitivity to perturbations. To explain the underlying mechanism, the authors average the text-to-vision attention maps across all attention heads and text tokens, apply a 2D FFT followed by radial binning, and obtain a task-conditioned frequency filter ; they argue that verbose prompts broaden , while complex queries narrow it. Furthermore, the authors propose an "overlap law," positing that the inner product between and the power spectrum of the perturbation pixel difference () predicts log-likelihood drift, achieving correlation coefficients of approximately to . The experiments employ an 8-level mirror design, where L1-L4 are concise questions, and L5-L8 are their corresponding verbose counterparts, evaluated across 78 perturbation types. The primary experiments are conducted on Qwen3-VL-2B and 8B using GQA and CLEVR, with additional replication on LLaVA-OneVision-0.5B and 7B using GQA only.
Critical strengths
Meaningful problem and thoughtful experimental design: Studying how instruction phrasing affects VLM robustness against natural perturbations addresses a valuable question. The use of a mirrored-query design on the same image to disentangle semantic complexity from prompt length is well-motivated. To the best of my knowledge, framing prompt phrasing as a moderating variable for perturbation robustness and explaining it through the spectral properties of attention maps is a novel angle without an exact precedent.
Awareness of ceiling effects: The authors proactively supplement their analysis with scale-invariant metrics, including margin, flip rate, and accuracy, in Table 12, and conduct prompt-template control experiments in Appendix C.
Strong transparency: The codebase is open-sourced, the authors candidly disclose an FFT shift bug in the perturbation operator in Appendix N, and the paper includes layer-selection sensitivity analyses as well as cross-architecture replications.
Critical weaknesses
Conceptual misinterpretation of the core mechanism: captures the spatial power spectrum of the attention map over the patch grid, not the model's selectivity for image frequency components.
- Inverted physical meaning: When attention is sharply concentrated on a small object, its spatial spectrum is broadband; conversely, when attention is distributed uniformly across space, all energy falls into the DC component. Thus, a "broadband" actually corresponds to spatially sharper attention rather than semantic selectivity for high-frequency visual details.
- Mismatched frequency scales: The proposed overlap law directly computes the inner product by matching radial bin indices between (defined on the patch grid up to ) and (defined at the pixel level up to ). Because each visual token on GQA covers roughly 30 pixels, taking the inner product across identically numbered bins has no physical meaning.
- Contradictory internal evidence: The caption of Table 5 admits that when the overlap law is computed in the vision-feature space, its sign flips completely ( to ), yet the authors dismiss this contradiction simply as "not load-bearing."
- Unaddressed attention sinks: Averaging attention maps across all heads and text tokens is heavily dominated by visual attention sinks, which the paper fails to account for.
Mechanistic claims remain purely correlational and exhibit key empirical inconsistencies. The paper performs no causal interventions on attention maps. Furthermore: (1) in Figure 2, the "narrow" regime exhibits higher drift than the "broadband" regime in only 9 out of 16 rows, with CLEVR-8B completely reversed; (2) bandwidth differences are minuscule and reported without confidence intervals (5.82 vs. 5.97 on GQA-8B, and 7.98 vs. 7.90 in the opposite direction on CLEVR-8B); (3) the semantic complexity coefficient is 0.075 on GQA for Qwen-8B, drops to 0.017 on LLaVA-0.5B, and flips sign to on LLaVA-7B, contradicting the main text's claim of consistent signs across settings; and (4) framing the sign reversal on CLEVR-8B as a "strong empirical vindication" reads as post-hoc rationalization. In addition, the four-stage layerwise trajectory is an incremental description building on Cross-modal Information Flow in MLLMs and Lifting the Veil (both CVPR 2025), neither of which is cited.
The conclusion that high-frequency perturbations cause minimal damage is an artifact of perturbation-bank composition and binning resolution rather than an intrinsic model property. Finding that 76% of perturbations are low-frequency merely reflects the authors' chosen perturbation suite. Similarly, the observation that the highest-frequency bin holds only 3%-5% of the spectral mass depends directly on the number of radial bins (dropping to when ). Moreover, the spatial frequencies of the attention map and those of the pixel image do not reside on the same physical scale.
Questions for authors
Q1. What is the correlation coefficient when predicting volatility using perturbation MSE alone? Furthermore, what is when is randomly permuted across images or across queries?
Q2. Why does the sign of the overlap law flip when computed in the vision-feature space, and which space do the authors consider to be the theoretically appropriate one?
Q3. If a minimal formatting instruction, e.g., "Answer with one word", is appended to the concise queries, or if different surface forms are marginalized over during scoring, how much of the verbose prompt's advantage remains? Additionally, how does a length-matched control that simply repeats the original query verbatim perform?
Q4. What are the corresponding values for each of the control templates evaluated in Appendix C?
Q5. Do the bandwidth differences and the overlap law still hold after removing visual attention sink tokens?
Q6. Can the authors validate the claimed causal direction by directly intervening on the attention maps, for example, by spatially smoothing or sharpening the attention maps at the last_2 layer during inference?