acceptodds
Withdrawn from ICLR 2027

Do MLLMs Exhibit Structured Perception Process?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While mechanistic interpretability has yielded profound insights into the inner workings of Large Language Models (LLMs), the internal mechanisms of Multimodal Large Language Models (MLLMs) remain largely unclear, particularly regarding how they retrieve and organize visual evidence for complex visual understanding. In this work, we elucidate the mechanisms of visual understanding in MLLMs by analyzing their internal attention dynamics. We find that MLLMs exhibit two distinct patterns of perception (dubbed semantic and reasoning perception) which are concentrated in the middle and later layers of the model. Specifically, **semantic perception** attends to semantic targets of the query, showing highly visual–language alignment. **Reasoning perception** retrieves answer-critical visual evidence at query-boundary tokens. These findings indicate that MLLM visual understanding is not a single-step process but a structured perceptual process. We further demonstrate the functional necessity of these patterns: suppressing the identified heads results in distinct error profiles. Motivated by this insight, we propose Structured Attention Focusing (SAF), a training-free method that emphasizes semantic and reasoning regions via attention-guided enhancement. Across four MLLM backbones and eight benchmarks, SAF consistently improves task scores, including an average gain of 3.3% on Qwen2-VL and 4.1% on Gemma. Our work provides critical insights into MLLM's internal visual mechanisms and introduces an interpretable strategy for enhancing multimodal visual understanding.

Primary area:
interpretability and explainable AI
Keywords:
multimodal large language models, mechanistic interpretability, test-time augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k4wAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the internal attention dynamics of multimodal large language models (MLLMs) and proposes that visual understanding involves two distinct patterns: semantic perception and reasoning perception. Based on these observations, the authors introduce Structured Attention Focusing (SAF), a training-free method that extracts and combines attention maps to enhance task-relevant visual regions. Experiments across four MLLMs and eight benchmarks demonstrate consistent performance improvements.

Critical strengths

Interesting analysis of visual attention patterns. The paper investigates how attention distributions vary across query tokens and model layers, providing potentially useful observations about visual grounding in MLLMs.

Consistent empirical improvements. The proposed SAF method achieves positive gains across multiple MLLM architectures and eight benchmarks. The authors also provide ablation studies to examine different components of the framework.

Critical weaknesses

Limited novelty compared with existing attention-based approaches. Previous studies have extensively investigated visual attention patterns, relevant region localization, and attention-guided visual enhancement in MLLMs. The proposed SAF framework follows a similar pipeline of extracting attention maps, selecting important regions, and enhancing visual inputs. The paper does not sufficiently establish its methodological differences from closely related work.

Insufficient evidence for the claimed structured perception mechanism. The distinction between semantic perception and reasoning perception is primarily established through attention patterns associated with semantic tokens and query-boundary tokens. However, increased attention to answer-relevant regions near the end of a question may simply result from causal attention and contextual aggregation, rather than indicating a distinct reasoning perception mechanism.

Limited causal validation of the proposed interpretation. Although the authors conduct attention-head ablation experiments, performance degradation after removing selected heads does not necessarily establish their claimed functional specialization. More rigorous causal interventions are needed to demonstrate that semantic and reasoning perception constitute distinct mechanisms.

Heuristic design of the proposed method. SAF relies on several manually designed components, including entropy-based token selection, position-based token selection, layer filtering, attention smoothing, and mask fusion. The necessity of these specific choices is not sufficiently justified, and it remains unclear whether the improvements arise from the proposed structured perception mechanism or from generic attention-based image enhancement.

Insufficient comparisons with closely related methods. The experimental comparisons mainly focus on the original models and relatively simple attention-mask variants. Stronger comparisons with existing visual evidence selection, attention steering, and visual enhancement approaches are necessary to establish the advantages of SAF. Furthermore, the additional inference overhead is not adequately evaluated.

Questions for authors

NA

4 comments

We thank the reviewer for recognizing the value of the attention analysis and the experimental results across models and tasks. We will further emphasize the questions addressed by structured perception and separately explain the mechanistic evidence, implementation choices, and method comparisons.

W1: Novelty in structured perception

(1) Our core question is how MLLMs organize visual information serving distinct roles. Rather than claiming attention-map extraction or visual-region enhancement as our main novelty, we investigate MLLM internal perceptual mechanisms during complex VQA, uncovering a structured perceptual process that motivates our visual enhancement method. By separately annotating question semantic anchors and answer-relevant evidence, we characterize their divergent patterns across tokens, layers, and heads, and validate their functional roles through head and path interventions—shifting the focus from “where the model looks” to “which query representations attend to which evidence, and how these links govern the answer.”

(2) This decomposition also determines SAF’s specific design: low-entropy tokens provide semantic localization signals, query-boundary tokens provide answer-evidence signals, perception-layer selection reduces interference from layers with weaker signals, and fusion preserves complementary content. The generic-mask controls in Table A9 and comparisons with related methods in Table A8 of the original submission validate the method’s effectiveness.

MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
Random same-area62.742.255.353.4
Random layer64.142.355.554.0
Center same-area64.542.656.754.6
All-query attention63.041.858.654.5
Last-token attention63.443.458.155.0
Reasoning only64.544.260.356.3
Semantic only64.843.654.554.3
Full SAF65.445.660.757.2
MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
VAR / VisAttnSink adjusted63.942.957.654.8
ViCrop / MLLMs-Know crop64.441.957.654.6
Full SAF65.445.660.757.2

W2: Reasoning perception goes beyond terminal position

(1) Thank you for the insightful observation. We agree that causal-context accumulation is related to reasoning perception. Our view is that when the model reaches both a syntactic boundary and a query terminal position, it aggregates the preceding question context and directs attention toward answer-critical visual evidence.

(2) Reasoning perception cannot be reduced to a simple last-token effect. To test this, we conducted two controlled experiments on Qwen2-VL-7B:

  1. We replaced the sentence-final "?" with ".", ",", "#", or removed it entirely. The results show that "?", ".", and "," elicit reasoning perception, whereas "#" does not. Removing punctuation also substantially reduces reasoning-perception intensity at the terminal-position token. Thus, terminal position alone does not always induce reasoning perception; specific non-semantic boundary tokens, such as sentence punctuation ("?", ".", ","), also matter.
  2. When suffix tokens are appended after the sentence, reasoning perception drops substantially. When "?" is inserted into the middle of the sentence, reasoning perception does not emerge. When additional queries (e.g., "Where is the [semantic token]?") are inserted before or after the original query, strong reasoning perception still appears regardless of whether the original query appears first or last. This shows that reasoning perception does not simply occur at the end of the input sequence; instead, it is tied to the interaction between query context and boundary tokens such as "?".
Terminal variantTermTok→SemTermTok→ReasOtherTok→SemOtherTok→Reas
?0.1623.0680.1460.418
no punctuation1.9490.8690.1220.422
.0.0732.6120.1500.413
,0.1702.2050.1430.433
#0.7730.3260.1370.458
VariantProbe roleProbeTok→SemProbeTok→Reas
dummy suffixappended suffix-0.2181.168
moved aggregationappended Answer:-0.1631.485
random question markinserted ?-0.0270.523
multi-questionreal boundary ? #10.1813.114
reordered multi-questionreal boundary ? #30.0402.402

W3: Interventions demonstrate functional selectivity

Thank you for pointing this out.

(1) The current results show that top semantic and reasoning heads are behaviorally important, and the existing token→region path ablation shows larger drops for structured paths than random-token controls.

Existing path ablationAccuracy (change)
Base58.3 (0.0)
SemTok→semantic bbox51.7 (-6.6)
TermTok→reasoning bbox53.3 (-5.0)
RandomTok→annotated regions57.2±0.78 (-1.1)

(2) We ablate heads and examine the resulting attention patterns. Ablating reasoning heads substantially reduces attention to evidence while having a smaller effect on attention to semantic objects, indicating that these heads perform independent, specific functions. Similarly, ablating semantic heads substantially affects attention to semantic objects. The independent functions of these two types of heads indicate that the two types of perception constitute independent mechanisms within the model.

InterventionAccuracyΔAccSemTok→SemΔTermTok→ReasΔ
Base43.3—4.232—2.766—
Semantic heads30.0-13.33.015-1.2172.665-0.101
Reasoning heads15.0-28.33.832-0.4001.315-1.451
Random heads34.7±4.52-8.74.290±0.280+0.0592.540±0.331

W4: Structured selection has empirical support

(1) Low-entropy and query-boundary token selection: evidence from attention patterns. Figure A6 in the submission presents both an example of token-level attention entropy and the dataset-level distribution of entropy versus semantic BalGR. Human-annotated semantic tokens tend to occupy the low-entropy, high-semantic-BalGR region, providing empirical support for using low entropy to select semantic candidates. Figures 3–4 compare perceptual intensity across token categories and relative positions, showing that reasoning perception strengthens near query boundaries.

Table A1 further examines this boundary effect by replacing or removing the terminal marker. The results support selecting tokens near query boundaries.

Terminal variantTermTok→SemTermTok→ReasOtherTok→SemOtherTok→Reas
?0.1623.0680.1460.418
No punctuation1.9490.8690.1220.422
.0.0732.6120.1500.413
,0.1702.2050.1430.433
#0.7730.3260.1370.458
Paraphrase0.1383.0800.1030.310

(2) Token count: comparing a small token subset with all query tokens. Table 4(A) jointly varies the number of semantic tokens k and reasoning tokens m. All tested settings outperform Base in average accuracy across eight datasets. The k=m=2 setting achieves the best result of 65.7, compared with 64.6 when all tokens are selected. This supports using a small token subset as the default configuration.

SettingkmAvg. accuracy
Base——62.4
A1165.1
B2265.7
C3364.7
D5564.8
Eallall64.6

(3) Perception-layer selection: comparing layer ranges with selection by perceptual intensity. Figure 3(a) shows the layer-wise intensity distributions of the two types of perception, which are concentrated in middle and later layers. Table 4(B) then compares early, middle, late, full, and selected layers within the same enhancement framework. The benefits of layer selection are consistent with the observed distribution of perceptual signals.

SettingLayersAvg. accuracy
Base—62.4
AEarly Layers64.5
BMiddle Layers65.2
CLate Layers65.4
DFull Layers65.3
ESelected Layers65.7

(4) Two-branch fusion and structured masking: evaluation against single-branch and generic masking controls. Table 3(A) evaluates the semantic and reasoning branches separately; Table 3(B) compares all-query and last-token attention; and Table A9 adds random same-area regions, central same-area regions, and random-layer controls.

MethodA-OKVQAInfoVQAV*
Base61.942.157.1
Random same-area62.742.255.3
Random layer64.142.355.5
Center same-area64.542.656.7
All-query attention63.041.858.6
Last-token attention63.443.458.1
Semantic only64.843.654.5
Reasoning only64.544.260.3
SAF65.445.660.7

SAF outperforms both single branches and all listed generic masking controls on all three datasets. These results directly support the contributions of structured selection and two-branch fusion.

(5) Smoothing, thresholding, and connected-component processing: stability under parameter variation. Table A11 varies Gaussian smoothing, median filter size, threshold coefficient, and connected-region minimum area individually on A-OKVQA while holding the other parameters fixed.

Varied componentAccuracy
Gaussian smoothing64.7±0.74
Median filter size65.4±0.39
Threshold coefficient65.4±0.10
Connected-region minimum area65.8±0.39

W5a: Comparisons with related methods

(1) We compare methods using the same model and evaluation protocol on three representative datasets reported in the paper: A-OKVQA, InfoVQA, and V*. The baselines include random same-area masking (retaining a random square region of the same area), random layer (aggregating attention from several randomly selected layers to construct a mask), center same-area masking (retaining a central square region of the same area), all-query-token attention, and last-token attention. SAF outperforms these generic masking baselines, suggesting that the gains do not arise solely from removing background information or retaining regions favored by a center or objectness prior. SAF also outperforms the all-query-token attention baseline, indicating that the structured semantic/reasoning decomposition contributes beyond standard saliency enhancement.

MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
Random same-area62.742.255.353.4
Random layer64.142.355.554.0
Center same-area64.542.656.754.6
All-query attention63.041.858.654.5
Last-token attention63.443.458.155.0
Reasoning only64.544.260.356.7
Semantic only64.843.654.554.3
Full SAF65.445.660.757.2

(2) As shown in Table A8, SAF also outperforms the related methods [1] and [2].

MethodA-OKVQAInfoVQAV*Avg.
VisAttnSink adjusted63.942.957.654.8
MLLMs-Know crop64.441.957.654.6
Full SAF65.445.660.757.2

[1] See what you are told: Visual attention sink in large multimodal models

[2] Mllms know where to look: Training-free perception of small visual details with multimodal llms

(3) SAF is designed around the internal attention dynamics of MLLMs, providing both interpretability and generalizability.

W5b: SAF’s inference overhead is quantified

(1) Thank you for the question. SAF does introduce additional computational overhead. To quantify this, we measure the FLOPs and latency of Base and SAF. The results summarized below:

MethodEnd-to-end latency (s)FLOPs (T)Relative latencyRelative FLOPs
Base5.0710.151.00×1.00×
SAF online10.0926.151.99×2.58×

(2) We would like to clarify that our primary goal is not to propose a low-cost inference strategy, but to investigate MLLM internal perceptual mechanisms and verify whether they can provide effective visual enhancement signals.

Reviewer_EexFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates attention dynamics in MLLMs and identifies two perception patterns: semantic perception, which focuses on query-related visual objects, and reasoning perception, which attends to answer-relevant evidence at query boundaries. Based on these findings, the authors propose SAF, a training-free method that combines both attention maps to enhance relevant image regions. Experiments across four MLLMs and eight benchmarks demonstrate consistent performance improvements.

Critical strengths
  1. The paper provides a systematic analysis of attention patterns in MLLMs, supported by a manually annotated dataset and attention ablation experiments. The proposed training-free SAF method further demonstrates consistent improvements across multiple models and benchmarks, with relatively comprehensive experimental validation.
Critical weaknesses
  1. The novelty of the proposed structured perception process is somewhat limited. Semantic tokens attending to corresponding visual regions is consistent with existing findings, while the attention concentration at query boundaries may naturally arise from causal attention and the accumulation of question context. It remains unclear whether these observations sufficiently establish two distinct perception mechanisms.
  2. The necessity of the dual-branch SAF design is not fully convincing. In Table A9, Reasoning-only already achieves an average accuracy of 56.7%, compared to 57.2% for the full SAF method. The additional contribution of semantic perception appears relatively limited.
Questions for authors
  1. Similar attention patterns have been observed in prior work. For example, LAST (2026) also uses the final query token's attention to locate relevant visual regions. Could the authors clarify the novelty of the proposed structured perception beyond these existing observations?
  2. Given the limited additional benefit of the semantic branch,could the authors further justify the necessity of the dual-branch design.
2 comments

We thank the reviewer for recognizing our systematic analysis, manually annotated diagnostic data, and training-free enhancement results. Your questions help us clarify the functional evidence, its relationship to existing attention observations, and the value of the two-branch design.

W1: Semantic and reasoning perception serve distinct functions

(1) Our argument rests on consistency across multiple types of evidence. In the original submission, Table 1 shows that ablating the two types of heads leads to distinct accuracy drops and error distributions, while our token- and layer-level analyses further reveal that the two signal groups favor different visual targets.

Table 1(A): Accuracy after ablating the top 5% of functional heads.

Head interventionA-OKVQA
Base61.9
Semantic24.5
Reasoning52.7
Random59.4±3.0

Table 1(B): Distribution of error types after head ablation (%).

Head interventionPerception errorReasoning error
Semantic42.241.3
Reasoning22.255.3
Random30.3±2.846.3±1.3

(2) To further test this distinction, we add joint measurements of grounding and task accuracy after head ablation. Specifically, we identify functional heads on the TextVQA/DocVQA diagnostic sets and then separately ablate semantic, reasoning, and random heads on held-out GQA data, measuring BalGR for semantic-token →\rightarrow semantic-region and boundary-token →\rightarrow answer-region attention. Crucially, the impairment is selective : ablating semantic heads reduces semantic BalGR by ~28.8% (from 4.232 to 3.015) versus only ~3.7% for reasoning BalGR, whereas ablating reasoning heads reduces reasoning BalGR by ~52.5% versus ~9.5% for semantic BalGR. These results confirm that the two head types support independent functional mechanisms.

InterventionAccuracyΔAccSemTok→SemΔTermTok→ReasΔ
Base43.3—4.232—2.766—
Semantic heads30.0-13.33.015-1.2172.665-0.101
Reasoning heads15.0-28.33.832-0.4001.315-1.451
Random heads34.7±4.52-8.74.290±0.280+0.0592.540±0.331-0.226

W2: The semantic branch provides complementary gains

(1) We apologize for a calculation error: the correct average for Reasoning-only is 56.3.

(2) The reasoning branch provides the primary signal for answer evidence. The purpose of the two-branch design is to additionally preserve complementary semantic context, rather than to require equal gains from both branches. A referent in the question can determine how the answer region should be interpreted. For example, identifying “the appliance to the right of the lamp” requires not only seeing the appliance but also retaining the lamp as a spatial reference.

(3) Adding the semantic branch increases performance from 64.5 to 65.4 on A-OKVQA and from 44.2 to 45.6 on InfoVQA, raising the average across the three datasets from 56.3 to 57.2.

MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
Semantic only64.843.654.554.3
Reasoning only64.544.260.356.3
Full SAF65.445.660.757.2

(4) More direct evidence comes from the human-annotated box retention experiment in Figure A5: retaining only the answer-evidence regions lowers accuracy from 0.633 to 0.583, whereas retaining both semantic and reasoning regions boosts it to 0.667. This contrast shows that answer regions alone cannot always substitute for the semantic references needed to interpret them.

Image conditionAccuracy
Original0.633
Keep reasoning target only0.583
Keep semantic + reasoning targets0.667

Q1: Contributions beyond LAST

(1) We thank the reviewer for identifying this closely related work. LAST uses a compact edge-side VLM’s last-query-token attention as an importance signal for visual-token pruning under an edge–cloud token budget.

(2) Our work further investigates how semantic targets referenced in a question and evidence needed for the final answer are separately located and used within the same model. To this end, we separately annotate the two types of visual targets, use BalGR to analyze their distributions across tokens, layers, and heads, conduct boundary-position controls and functional interventions, and use both signals for image enhancement. Our novelty lies in identifying and investigating structured perceptual mechanisms within MLLMs and developing an interpretable visual enhancement method.

(3) To test the value of SAF beyond last-token saliency alone, we introduce a LAST-style masking control that uses attention from the last query token without SAF’s perception-layer selection, while keeping the enhancement and task evaluation procedures the same. The results show that SAF provides further performance gains over using only the last token.

MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
LAST-style64.545.258.756.1
SAF65.445.660.757.2

Q2: Two-branch fusion preserves complementary evidence

(1) Semantic and reasoning maps provide different types of information: the former preserve the targets and reference relations in the query, while the latter highlight the evidence needed to answer it. For spatial relations, comparisons between multiple objects, and text recognition that requires context, retaining only the answer region may remove the references needed to interpret it.

(2) More direct evidence comes from Figure A5’s human-box retention experiment: keeping only answer-evidence regions drops accuracy from 0.633 to 0.583, while keeping both semantic and reasoning regions raises it to 0.667, demonstrating that answer regions alone cannot always replace the semantic references required to interpret them.

Image conditionAccuracy
Original0.633
Keep reasoning target only0.583
Keep semantic + reasoning targets0.667

(3) The reasoning branch primarily focuses on answer-relevant evidence, while the semantic branch preserves semantic anchors referenced by the query. For questions involving spatial relations or the localization of multiple objects, these semantic references can provide essential context for interpreting the answer evidence. Table A9 already shows that SAF outperforms Reasoning-only on all three datasets.

MethodA-OKVQAInfoVQAV*Avg.
Base61.942.157.153.7
Semantic only64.843.654.554.3
Reasoning only64.544.260.356.3
Full SAF65.445.660.757.2
Reviewer_MLBFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a new perspective on MLLMs and a useful method to improve MLLMs without training. However, the experiments are insufficient to demonstrate its superiority, and the evaluation protocol seems to be strange. Based on the above weaknesses, I lean toward rejecting this paper.

AI statement: Deepseek v4 flash is used to retrieve related works,suggest drawbacks, and fix errors in spelling and grammar. All reviews are wirtten by myself.

Critical strengths

The proposed method is easy to implement and seems to be general. Besides, the proposed semantic perception, reasoning perception as well as the BalGR provides a new perspective for researching the mechanism of MLLMs.

Critical weaknesses

Current experiments do not fully demonstrate the characteristics of the proposed method. Firstly, among the compared MLLMs, QWEN2 is relatively old; why not conduct experiments on the latest models, which would be more meaningful and ensure the method's practicality? On the other hand, this paper only compares performance with other methods but does not address efficiency issues. SAF relies on two forward passes, but many of the compared methods only require one; this efficiency imbalance makes the experiments less convincing.

Questions for authors

The hyperparameter selection is not rigorous enough. For example, why was m=2 chosen? Why were the last two tokens selected? Could this cause problems in complex scenarios? Also, why was the semantic token chosen based on the lowest entropy criterion? Could you compare this with Oracle's settings to see if the lowest entropy is truly the best?

The evaluation protocol used in this paper also seems quite strange, with many performance metrics not matching those in the original paper. For example, the original report for qwen vl2 8b showed a DocVQA performance of 94.5 [1], while this paper only shows 78.6. Such mismatches seem to exist for almost every metric in Table 2. This mismatch should be explained here. If the issue stems from inconsistencies in the evaluation protocol, is such an evaluation protocol even meaningful?

[1] https://ar5iv.labs.arxiv.org/html/2409.12191

2 comments

We thank the reviewer for recognizing our work as useful perspectives for studying the internal mechanisms of MLLMs, and for the suggestions on practical applicability and experimental rigor.

W1: Evaluation on newer models

Thank you for the suggestion. We add paired Base/SAF evaluations on Qwen3-VL-8B to directly test its performance on a newer model.

ModelMethodA-OKVQATextVQADocVQAInfoVQAMMEPOPEHallusionBenchV*
Qwen3-VL-8BBase84.683.587.751.787.688.067.359.2
Qwen3-VL-8BSAF87.484.988.155.088.689.271.260.6
Δ+2.8+1.4+0.4+3.3+1.0+1.2+3.9+1.4

W2: Two-stage inference adds measurable cost

(1) Thank you for the question. SAF does introduce additional computational overhead. To quantify this, we measure the FLOPs and latency of Base and SAF. The results summarized below:

MethodEnd-to-end latency (s)FLOPs (T)Relative latencyRelative FLOPs
Base5.0710.151.00×1.00×
SAF online10.0926.151.99×2.58×

(2) We would like to clarify that our primary goal is not to propose a low-cost inference strategy, but to investigate MLLM internal perceptual mechanisms and verify whether they can provide effective visual enhancement signals.

Q1: Rationale for the last two tokens

(1) This position choice is motivated by perceptual dynamics across the question sequence. As shown in Figure 2, attention to answer evidence typically strengthens near the question boundary, where contextual representations have already integrated the preceding question context. Selecting tokens at the end of the question is therefore a simple way to leverage this statistical pattern.

(2) The token count choice is supported by the ablation in Table 4(A). Across eight datasets, all tested settings outperform Base in average accuracy, with k=2,m=2 achieving the best overall result.

SettingkmAvg. accuracy
Base——62.4
A1165.1
B2265.7
C3364.7
D5564.8
Eallall64.6

(3) These results support k=m=2 as a practical default that requires no dataset-specific tuning. Compared with using all tokens, it retains localized perceptual signals while limiting the inclusion of irrelevant attention.

Q2: SAF handles complex queries

(1) We thank the reviewer for this insightful question. In Figure A4, we examine our identified structured perception across distinct data domains (GQA, TextVQA, and DocVQA) and models. We observe consistent perceptual dynamics throughout, showing that these patterns are independent of scene complexity. Consequently, scene complexity does not undermine the mechanistic foundation of our method, supporting its cross-domain applicability.

(2) To further address your question on complexity, we fix k=2,m=2 and use structural information from GQA functional programs to examine Base/SAF performance across query groups. The results show that SAF also yields performance gains in complex scenarios.

Complexity groupBaseSAF (k=2,m=2)Δ
Simple76.081.0+5.0
Medium74.075.0+1.0
Complex71.477.7+6.3
Overall74.076.0+2.0

Q3: Low-entropy selection and oracle validation

(1) Low-entropy selection is motivated by the empirical relationship between the spatial concentration of attention and semantic grounding. When a query token aligns with a semantic target in the image, its visual attention tends to be more concentrated. The entropy analysis in Figure A6 of the original submission shows that human-annotated semantic tokens tend to have both lower entropy and higher semantic BalGR. We therefore use low entropy to rank semantic candidates without additional training.

(2) To directly test this rationale, we compare human-annotated semantic tokens with selected tokens on the diagnostic sets (GQA, TextVQA and DocVQA). The compared selection methods include low-entropy selection, visual-attention mass, and random selection under the same token budget. Low-entropy selection substantially boosts Hit@2 from 29.4% (visual-attention mass) to 46.4%, demonstrating its superior effectiveness in accurately locating semantic tokens.

SelectorHit@1Hit@2Token F1MRRAUPRCSemantic BalGR
Random matched0.7%1.3%0.8%4.2%0.8%0.233
Visual-attention mass17.9%29.4%19.4%34.3%7.0%3.320
Low entropy29.4%46.4%30.8%48.6%15.2%3.595

(3) For the oracle comparison, we further evaluate the effect of using exact human-annotated semantic tokens on the annotated GQA, TextVQA, and DocVQA datasets. The results show that low-entropy token selection yields performance gains comparable to the oracle annotations.

Semantic token sourceGQATextVQADocVQAMacro-average across three datasets
Base46.669.087.767.7
Low entropy (k=2)50.072.688.270.3
Human-token Oracle48.873.588.969.1

Q4: Protocol differences explain score gaps

(1) Thank you for the question. First, as detailed in Appendix A3 (Implementation Details), we adopt a unified evaluation protocol across all models and datasets, measuring accuracy via exact matching between reference answers and extracted model responses. Because this criterion is stricter than the official implementations, it naturally yields lower accuracy scores. Second, the official evaluation uses the DocVQA test split, whereas we use the validation split to reduce evaluation costs.

(2) We strictly apply the same testing and evaluation protocol across all compared methods. We believe that the experimental conclusions remain reliable as long as the comparison is fair.

(3) Table A10 in the original submission already examines different random seeds, prompts, decoding settings, and official answer matching on A-OKVQA. The reported paired gains range from +3.0 to +3.3 percentage points, showing that these evaluation choices do not change the main conclusion.

Evaluation settingBaseSAFΔ
Different random seeds62.5±0.4665.6±0.18+3.1
Different prompts62.6±1.2565.7±1.76+3.1
Different decoding settings62.3±0.4965.3±0.12+3.0
Official A-OKVQA answer matching76.479.7+3.3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.