acceptodds
Withdrawn from ICLR 2027

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust—e.g., rephrasing “Is there a cat?“ into “Please look carefully and answer: is there a cat?“. Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., “what colour is the cup left of the chair?“ instead of “is there a cup?“. Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70–81% on the 8B models. The practical recipe—“pad the prompt“—further yields measurable gains in accuracy, even under image corruption.

Primary area:
interpretability and explainable AI
Keywords:
Vision Language Models, Model Robustness, Mechanistic Interpretability, Prompt Sensitivity, Cross Modal Attention, Frequency Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_e4M3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how query phrasing affects the stability of Vision-Language Models (VLMs) under image perturbations, reporting two opposing effects: wrapping queries in polite, verbose phrasing reduces perturbation-induced drift in the gold-answer log-likelihood, whereas semantically more complex questions exhibit greater sensitivity to perturbations. To explain the underlying mechanism, the authors average the text-to-vision attention maps across all attention heads and text tokens, apply a 2D FFT followed by radial binning, and obtain a task-conditioned frequency filter WtW_t; they argue that verbose prompts broaden WtW_t, while complex queries narrow it. Furthermore, the authors propose an "overlap law," positing that the inner product between WtW_t and the power spectrum of the perturbation pixel difference (ΔF\Delta F) predicts log-likelihood drift, achieving correlation coefficients of approximately 0.260.26 to 0.420.42. The experiments employ an 8-level mirror design, where L1-L4 are concise questions, and L5-L8 are their corresponding verbose counterparts, evaluated across 78 perturbation types. The primary experiments are conducted on Qwen3-VL-2B and 8B using GQA and CLEVR, with additional replication on LLaVA-OneVision-0.5B and 7B using GQA only.

Critical strengths

Meaningful problem and thoughtful experimental design: Studying how instruction phrasing affects VLM robustness against natural perturbations addresses a valuable question. The use of a mirrored-query design on the same image to disentangle semantic complexity from prompt length is well-motivated. To the best of my knowledge, framing prompt phrasing as a moderating variable for perturbation robustness and explaining it through the spectral properties of attention maps is a novel angle without an exact precedent.

Awareness of ceiling effects: The authors proactively supplement their analysis with scale-invariant metrics, including margin, flip rate, and accuracy, in Table 12, and conduct prompt-template control experiments in Appendix C.

Strong transparency: The codebase is open-sourced, the authors candidly disclose an FFT shift bug in the perturbation operator in Appendix N, and the paper includes layer-selection sensitivity analyses as well as cross-architecture replications.

Critical weaknesses

Conceptual misinterpretation of the core mechanism: WtW_t captures the spatial power spectrum of the attention map over the patch grid, not the model's selectivity for image frequency components.

  1. Inverted physical meaning: When attention is sharply concentrated on a small object, its spatial spectrum is broadband; conversely, when attention is distributed uniformly across space, all energy falls into the DC component. Thus, a "broadband" WtW_t actually corresponds to spatially sharper attention rather than semantic selectivity for high-frequency visual details.
  2. Mismatched frequency scales: The proposed overlap law directly computes the inner product by matching radial bin indices between WtW_t (defined on the patch grid up to ≈0.5 cycle/patch\approx 0.5\text{ cycle/patch}) and ΔF\Delta F (defined at the pixel level up to ≈0.5 cycle/pixel\approx 0.5\text{ cycle/pixel}). Because each visual token on GQA covers roughly 30 pixels, taking the inner product across identically numbered bins has no physical meaning.
  3. Contradictory internal evidence: The caption of Table 5 admits that when the overlap law is computed in the vision-feature space, its sign flips completely (r=−0.31r = -0.31 to −0.38-0.38), yet the authors dismiss this contradiction simply as "not load-bearing."
  4. Unaddressed attention sinks: Averaging attention maps across all heads and text tokens is heavily dominated by visual attention sinks, which the paper fails to account for.

Mechanistic claims remain purely correlational and exhibit key empirical inconsistencies. The paper performs no causal interventions on attention maps. Furthermore: (1) in Figure 2, the "narrow" regime exhibits higher drift than the "broadband" regime in only 9 out of 16 rows, with CLEVR-8B completely reversed; (2) bandwidth differences are minuscule and reported without confidence intervals (5.82 vs. 5.97 on GQA-8B, and 7.98 vs. 7.90 in the opposite direction on CLEVR-8B); (3) the semantic complexity coefficient is 0.075 on GQA for Qwen-8B, drops to 0.017 on LLaVA-0.5B, and flips sign to −0.035-0.035 on LLaVA-7B, contradicting the main text's claim of consistent signs across settings; and (4) framing the sign reversal on CLEVR-8B as a "strong empirical vindication" reads as post-hoc rationalization. In addition, the four-stage layerwise trajectory is an incremental description building on Cross-modal Information Flow in MLLMs and Lifting the Veil (both CVPR 2025), neither of which is cited.

The conclusion that high-frequency perturbations cause minimal damage is an artifact of perturbation-bank composition and binning resolution rather than an intrinsic model property. Finding that 76% of perturbations are low-frequency merely reflects the authors' chosen perturbation suite. Similarly, the observation that the highest-frequency bin holds only 3%-5% of the spectral mass depends directly on the number of radial bins BB (dropping to ≈1%\approx 1\% when B=20B = 20). Moreover, the spatial frequencies of the attention map and those of the pixel image do not reside on the same physical scale.

Questions for authors

Q1. What is the correlation coefficient rr when predicting volatility using perturbation MSE alone? Furthermore, what is rr when WtW_t is randomly permuted across images or across queries?

Q2. Why does the sign of the overlap law flip when computed in the vision-feature space, and which space do the authors consider to be the theoretically appropriate one?

Q3. If a minimal formatting instruction, e.g., "Answer with one word", is appended to the concise queries, or if different surface forms are marginalized over during scoring, how much of the verbose prompt's advantage remains? Additionally, how does a length-matched control that simply repeats the original query verbatim perform?

Q4. What are the corresponding GG values for each of the control templates evaluated in Appendix C?

Q5. Do the bandwidth differences and the overlap law still hold after removing visual attention sink tokens?

Q6. Can the authors validate the claimed causal direction by directly intervening on the attention maps, for example, by spatially smoothing or sharpening the attention maps at the last_2 layer during inference?

Reviewer_if99OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at text-visual cross attention in VLMs, and argues that some "coarse" questions attend to smoothly to many image regions, while some "fine" questions attend to localized image regions. By padding prompts to make them verbose, this paper points out that answers become more robust because they smooth over more image regions. This is largely an empirical analysis paper that looks at the spatial frequency of cross attention, and points out interesting phenomena. One concrete actionable result of the analysis is the suggestion to pad prompts, which empirically leads to smoother cross attention maps with more reliance to image noise.

Critical strengths
  • Analyzing the behaviour of cross-attention vision-language maps, as a function of image and prompt pertubations, is an interesting endeavour.
Critical weaknesses
  • While the introduction is relatively straightforward to understand, much of the technical presentation and the analysis in Sec 5 is poorly written and difficult to parse, which is particularly problematic for an analysis paper such as this.

  • It is quite difficult to articulate the take-home messages of the analysis. For example, L77 states "The model’s answer drifts most when the bands emphasised by this filter overlap the bands occupied by the perturbation", but this framing obscures the real-world behavior. Is it that global degradations (like motion blur) will affect global questions, while local degradations (like region masking) will effect questions about that local region? If so, this seems somewhat superficial and doesn't seem to require a frequency encoding/analysis.

  • It appears that the central metric is log-likelihood of the correct answer, and specifically, the variance of the log-likelihood when the input image is perturbed. Are the reductions in variance from prompt padding (Table 2) meaningful? No example images or results are presented, which makes it difficult to understand the impact of the proposed prompt padding.

  • L38 is oddly aggressive "yet no prior framework predicts before any behavioural test how a given task-corruption combination will fail."; I would suggest removing it.

Questions for authors

Can you clearly articulate the surprising and actionable conclusions of your analysis, with concrete examples?

Reviewer_FNnBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats cross-modal attention in vision-language models as a task-conditioned spatial-frequency filter, finding that prompt wording affects model robustness by tuning the filter’s bandwidth. It further proposes an overlap law to predict perturbation-induced answer drift from the alignment between attention and corruption spectra. Its main contributions are a task-conditioned spectral framework for VLM robustness, disentangled opposing effects of semantic complexity and prompt verbosity on attentional bandwidth, a mechanism-driven drift predictor relying only on internal model signals, a layerwise bandwidth trajectory, and a reinterpretation of the high-frequency cutoff effect. Evaluations use an 8-level mirrored prompt design with controlled difficulty across four model–dataset pairs (plus LLaVA-OneVision), covering 78 perturbation conditions and 312k samples, with support from regression, correlation, and prompt ablation analyses.

Critical strengths

I greatly appreciate the experimental design of this paper, whose strengths lie in the following aspects:

First, the mirrored 8-level design cleanly disentangles "complex questioning" from "question length" for the first time. In typical prompt studies, these two factors are inherently confounded: complex questions tend to be longer by nature, making it impossible to isolate their individual effects. This work generates eight questions for the same image. Versions L5–L8 are verbatim copies of L1–L4, augmented only with polite prefatory phrasing, which manipulates token count alone. Critically, L4 reuses the options, categories and ground-truth answers from L2; only one relative clause is added from L2 to L4, holding the difficulty of answer options constant. I cross-compared this setup with Tinted Frames during my review: Tinted Frames only enforces semantic equivalence and does not control for option difficulty. This paradigm is reusable for future prompt engineering studies.

Second, the overlap law is the only genuinely novel contribution I can confirm. The observation that rephrasing a question shifts the frequency bands the model relies on may not be unprecedented, to my knowledge. However, the approach of directly multiplying the model’s internal attention spectrum with an external perturbation spectrum to predict answer drift—without any training or parameter tuning—is a combination I have not seen reported before.

Third, the findings are not limited to a single model. Consistent trends hold across two architectures (Qwen3-VL, LLaVA-OneVision), four model sizes (0.5B to 8B), and two image domains (real photographs from GQA, synthetic CLEVR scenes). The practical recommendations are concrete and empirically effective.

Overall, the paper’s core strengths are well-posed research questions and a rigorously controlled experimental setup.

Critical weaknesses
  1. The magnitude of the proposed mechanism cannot account for the magnitude of the observed effect.**
    This is the most fundamental flaw. The bandwidth measure �0��0� ranges over [1,B][1, B], with �1��1� on GQA, yet the observed difference between the long and short prompt versions only shifts from 5.75 to 5.92 — a displacement of roughly 3%. Meanwhile, the pairwise drift variance in behavior drops to only 0.19–0.30. An almost unchanged intermediate variable is used to explain the main effect, which leaves only two interpretations: either this causal pathway is not the primary driver of the effect, or �2��2� is an overly coarse summary statistic. The authors do not directly address either possibility, and only note in the limitations that WtW_t represents a "lower bound on correlation".

  2. The core finding fails to generalize across architectures, yet the corresponding conclusion is not retracted.**
    The entire argument rests on FWL residualization — the claim that "semantic complexity independently induces fragility after controlling for word count". However, on LLaVA-OneVision the two coefficients are �3��3� and −0.035-0.035, effectively zero; and only �4��4� remains on GQA-8B. The authors report these values honestly, but do not retract the claim at the end of §1 that the effect is "neither model-specific nor dataset-specific" — a statement that underpins the very framing of the result as a general "mechanism".

  3. Correlation is presented as causation, and no dissociative intervention experiments are provided.**
    The group-level rr values for the overlap law range from 0.14 to 0.42, indicating correct directionality but not meaningful predictive utility. The p≈0p \approx 0 result across 312,000 rows of clustering data reflects statistical significance, not effect size. Slopes as small as �5��5� and �6��6� in Appendix G are still counted toward the "12/12 positive" tally.
    To establish a mechanistic account, a single intervention experiment would suffice: apply band-windowing or masking to visual tokens during inference to artificially manipulate the bandwidth of WtW_t, and test whether drift changes accordingly. This experiment would be far less costly than the existing 310,000-row scoring analysis, yet it was not conducted.

  4. An unruled alternative explanation remains unaddressed.**
    In Table 6, the clean accuracy of Qwen-2B on CLEVR rises from 78.85 to 97.30 (+18.5 percentage points), and from 73.35 to 84.25 on GQA. A polite filler phrase with no semantic content cannot produce an effect of this magnitude. It is far more plausible that the short prompt itself was improperly formatted — what appears to be "better performance with the long version" may simply be "worse performance with the short version".
    The five-template control in Appendix C moves in the right direction, but it uses only 100 images and a single model. Critically, its own finding — that longer random neutral wrappers yield worse stability — directly supports this alternative explanation.

  5. Novelty claims are overstated, with key prior work omitted.**
    Section 1 lists three "first" contributions, two of which are not certain:

    • (i) "condition the Fourier view on a task-specific prompt" — Although the HAFI-VLM paper is a preprint, its authors already achieved this via a text-conditioned band gate. In their Figure 2, they present three-band weights for different questions on the same image (0.580/0.265/0.155 vs. 0.070/0.307/0.623), following the exact same pattern reported here: attribute and category queries bias toward mid-high frequencies, while relation and object queries bias toward low-mid frequencies.
    • (iii) "disentangle semantic from lexical load" — The Tinted Frames paper (March 2026) already decomposed the respective contributions of question framing and additional instructions using quartile coefficients of variation.

Neither work is cited in Section 2. Tinted Frames predates this manuscript by half a year and addresses exactly the same line of inquiry, so recency cannot explain the omission.

  1. The physical scales of the two spectral measures are not aligned.*
    The attention spectrum is defined on a 12×1612 \times 16 patch grid (for GQA), where the highest frequency band corresponds to a checkerboard pattern alternating every other patch, with a wavelength of approximately two patches. The perturbation spectrum, by contrast, is defined on pixel-level luminance differences, with a highest-band wavelength of roughly two pixels. On a 500-pixel-wide image with a 16-column grid, this corresponds to a roughly 30-fold difference in physical scale.

Section 3.2 argues that the overlap is "well-defined dimensionally" because both spectra are divided into the same BB bands. This holds only in the relative sense of normalized spatial frequency, not in terms of physical scale. Correspondingly, the claim that "the model does not attend to the finest layer" should be phrased as "the attention map is smooth", not "the model does not use high-frequency details" — fine-grained visual details reside in the feature vectors of attended patches, not in the spatial fluctuations of the attention map itself.

Questions for authors

Major Comments

  1. If �0��0� is not the appropriate mediator variable to explain the magnitude of the observed effect, what alternative statistic do the authors propose? Could the analysis be refitted using metrics beyond bandwidth (e.g., band centroid, overlap ratio of effective support sets) to examine whether the effect size becomes consistent?

  2. Could the authors supplement an intervention experiment at inference time: artificially widening or narrowing the band support of WtW_t to verify that drift changes monotonically with bandwidth? Such an experiment would elevate the finding from correlation to causation.

  3. How should the +18.5 pp improvement in clean accuracy on CLEVR-2B be interpreted? Are the prompt templates for the terse baseline and the verbose version strictly equivalent, with the only difference being the wrapper text?

  4. Given that FWL residuals are nearly zero on LLaVA, what evidence supports the claim at the end of §1 that the effect is "neither model-specific nor dataset-specific"?

  5. Please elaborate on the relation of this work to HAFI-VLM and Tinted Frames. If these prior works are intentionally omitted from Section 2, please provide the rationale.

  6. In Figure 2, the narrow-band vs wide-band pattern holds for only 9 out of 16 rows. All four rows for CLEVR-8B show the opposite trend, and the final-layer long version on CLEVR-8B actually reads 7.90 < 7.98. These constitute multiple boundary tests of the same core prediction. Would the authors agree to restate this result as "the prediction holds for three out of four settings"?

Minor Issues

  • For the 78.85 → 97.30 jump in Table 6: it seems implausible that a generic polite framing phrase with no semantic content could drive such a large accuracy gain. This pattern seems more consistent with a formatting defect in the short prompt itself, rather than an effect of the proposed frequency filter mechanism. This alternative interpretation deserves at least a discussion in the text.
  • The abstract reports a 70–81% range, while the introduction states 59–87%. It is unclear whether these describe the same metric; please clarify the discrepancy.
  • In Figure 4, the "primary question" placeholder in the prompt template was not populated. The figure caption and Appendix E still carry the internal version label "v8 design". Appendix B references a "Concern-B regime" that is never defined in the main text.
  • Appendix N notes that LowBandNoise and HighBandNoise have swapped frequency bands due to an fftshift misalignment. Although the analysis uses the empirically measured spectra, this means the perturbation library did not function at its intended design frequencies. We recommend disclosing this issue in the main text and providing corrected operator spectrum plots.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.