acceptodds
Under review as a conference paper at ICLR 2027

Frequency Boundaries of Visual Semantics for Grounded Answering in Frozen Vision-Language Models

Abstract

Vision-language models answer questions across diverse visual tasks, yet how image frequency bands affect answer correctness and semantic content remains unclear. We introduce a paired analysis framework based on independent band suppression to characterize the frequency boundaries of visual semantics in frozen models. Holding the model and question fixed, we independently generate eight radial band-stop variants of each image and compare original and filtered responses using five semantic labels based on the question and reference answers, distinguishing preservation, degradation, and error correction. Across four tasks and five models from two families, degradation follows task-dependent frequency-band patterns shared across models. Some initially incorrect answers also become correct under band suppression. For shared instances answered correctly by all five models on unfiltered images, InternVL3 has significantly lower correct-to-incorrect transition rates than Qwen2.5-VL across multiple bands in text-centric tasks. Within families, larger models show similar reductions, mainly in document understanding. Human validation yields 91.72% weighted response-level agreement between final automatic labels and human judgments. These findings link suppressed bands to the direction and type of semantic transition, revealing how frequency boundaries vary with task, family, and scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Spatial Frequency, Frequency-Domain Analysis, Visual Question Answering, Semantic Transitions, Model Robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JdYBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how suppressing image frequency bands changes VLM answers. It evaluates eight band-stop interventions across four VQA datasets and five configurations from two model families, using semantic labels to track answer preservation, degradation, and correction.

Critical strengths

The paired design controls the model and question across interventions. Tracking answer transitions provides more information than aggregate accuracy, particularly when degradation and correction coexist. Human annotation checks and statistical comparisons strengthen the behavioral analysis.

Critical weaknesses

The main limitation is the gap between observing answer changes and explaining visual understanding. The paper does not distinguish destroyed visual evidence from failures in visual encoding, question-conditioned information selection, or reliance on language priors. Its discussion offers little explanation of the observed patterns or guidance for visual–language interaction.

VQA is inherently question-dependent: different questions about the same image require different regions and details. Dataset-level frequency comparisons do not isolate this dependence and may instead reflect differences in text size, object scale, or image statistics.

The eight-band partition is justified by spectral approximation error, which does not establish its suitability for semantic analysis. Sensitivity to band boundaries, removed energy, and image scale remains unclear. Likewise, improved answers after filtering could reflect successful guessing rather than improved grounding.

Finally, two model families at 2B–14B provide limited support for generalization across VLM architectures, particularly those without a separate vision encoder.

Questions for authors

What mechanistic hypothesis explains the observed patterns, and how could it be tested?

Does frequency sensitivity change with different questions about the same image, particularly when filtering relevant versus irrelevant regions?

Are filtered images still answerable, and how are visual evidence loss and language shortcuts distinguished?

Do the conclusions hold under different frequency partitions, low-/high-pass filtering, resolutions, and model architectures?

Reviewer_6HaHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how suppressing spatial-frequency bands affects the answers of frozen VLMs. The authors evaluate five models across four VLM datasets. They compare responses to original images and eight filtered images. They compare the changes in correctness, specificity, completeness, and abstention using a five-category semantic rubric. Their analysis shows a task-dependent degradation pattern and differences related to model family and size.

Critical strengths
  • The experimental design provides a systematic way to analyze answer changes while keeping the question and model fixed.
  • The transition analysis was an informative contribution. It helps to understand both degradation and correction, which can be hidden in the overall accuracy score.
  • Thanks for providing supplementary material. It helped me understand the method much better.
  • It was a clear and well-written paper.
Critical weaknesses
  • The origin of the five semantic labels is unclear: Please explain whether they came from previous work, were defined by the authors, or were derived from empirical analysis.
  • The annotation procedure described in the paper differs from that in the supplementary code. The paper says it evaluates one response per request. The code says the preserved results used five responses per request. Please clarify which procedure generated the reported results, and provide the prompt (or, if I misunderstood, clarifying here would be much appreciated).
Questions for authors

Check weaknesses.

Reviewer_KYeVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the impact of removing radial frequency bands from images on the accuracy of VLMs’ answers about these images. They state that the current practice of reporting only aggregrate statistics on how answers can hide the true complexity of VLMs behavior. For instance, while answers might be more correct on average across datasets after a frequency-removal intervention, some previously correct answers might now the answered incorrectly. To investigate these fine-grained patterns, they independently remove one of eight uniform-width frequency bands (1-8) from the input image and track the answer change across five categories corresponding to different levels of answer quality, beyond just “hard” correctness. To enable this more fine-grained analysis of the answers at scale, they employ an LLM as a judge approach, and verify the LLM-judge’s answers with a single annotator (~90% agreement). Overall they find that removal of low-to-mid bands seems to hurt text-heavy tasks (TextVQA, DocVQA). Removing band 1 hurts on GQA and SLaKE while removing higher bands can sometimes improve answers.

Critical strengths

The paper provides thorough empirical insights on the nuanced changes to VLMs predictions based on frequency removal and a relatively fine-grained rubric, which are easily obscured if only looking at aggregate statistics. The authors analyze the behavior across five open models across model families and model sizes and four datasets, providing a strong empirical basis for their observed behavior. The statistical analysis is thorough and honest

Critical weaknesses

I have three concerns regarding soundness and one regarding presentation:

Methodological:

I am not entirely convinced that the chosen construction of the frequency bands indeed reflects the authors intended goal, i.e., showing how image frequency information affects answering behavior. First, it is well known (see, e.g., Field 1987) that the power spectrum of natural images decays as a function of the radial frequency roughly on the order of 1/ρ21/\rho^2, and dividing the spectrum into equal-width bands will result in drastically varying spectral power per band. The paper’s first uniform-width band would contain most of it, which makes it hard to disentangle spectral power removed vs frequency removed. Second, these band-widths seem to be applied across datasets, independent of the nature of the images. Text-heavy images or medical images would have significantly different spectra on average. I question of whether comparing the same bands across these very different image datasets is even a sensible comparison, as the division of the spectral mass across bands will vary a lot. The authors observe “error correction” by removing bands 6-8 on GQA/SLaKE, but could this have been observed with any other trivial baseline intervention, like JPEG compression? Some important details on the human annotation procedure as missing (how many annotators, inter-annotator agreement if multiple where asked, etc.). Also, as reported in appendix D.2, the per-class F1 scores for the classes B-D (which are instrumental to the authors claimed contribution) are relatively low, making this agreement statistic mainly a statement about classes A/E (the hard, correct/incorrect ones the paper wanted to depart from). This weakens the validity of the LLM as a judge approach. Presentation: The tables are hard to read. I believe multiples of confusion matrices would have been a better choice to visualize the data (instead of stacking the elements of this matrix vertically)

Questions for authors

Band construction (Concern 1):

  1. For each dataset and band, what fraction of the image's spectral power does the band mask remove, averaged over images?
  2. Did the authors consider log-spaced (octave) bands, which carry roughly equal power under a 1/ρ21/\rho^2 spectrum? If not, could a single model and dataset be rerun with octave bands to check whether the task-dependent patterns survive?

Baseline (Concern 2): 3. What are the A-to-E and E-to-A transition rates under a perturbation without spectral design, such as JPEG re-encoding at a moderate quality factor? Do the correction rates in bands 6 to 8 on GQA and SLaKE exceed that baseline?

Human validation (Concern 3): 4. Who performed the human annotation (number of annotators, how they were recruited, and their expertise, in particular for SLaKE)? If more than one annotator was involved, what was the inter-annotator agreement?

Reporting (Concern 5): 5. Could the authors report the unfiltered label distribution and accuracy per model and dataset, together with the number of eligible pairs per transition and band?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.