- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the internal attention dynamics of multimodal large language models (MLLMs) and proposes that visual understanding involves two distinct patterns: semantic perception and reasoning perception. Based on these observations, the authors introduce Structured Attention Focusing (SAF), a training-free method that extracts and combines attention maps to enhance task-relevant visual regions. Experiments across four MLLMs and eight benchmarks demonstrate consistent performance improvements.
Critical strengths
Interesting analysis of visual attention patterns. The paper investigates how attention distributions vary across query tokens and model layers, providing potentially useful observations about visual grounding in MLLMs.
Consistent empirical improvements. The proposed SAF method achieves positive gains across multiple MLLM architectures and eight benchmarks. The authors also provide ablation studies to examine different components of the framework.
Critical weaknesses
Limited novelty compared with existing attention-based approaches. Previous studies have extensively investigated visual attention patterns, relevant region localization, and attention-guided visual enhancement in MLLMs. The proposed SAF framework follows a similar pipeline of extracting attention maps, selecting important regions, and enhancing visual inputs. The paper does not sufficiently establish its methodological differences from closely related work.
Insufficient evidence for the claimed structured perception mechanism. The distinction between semantic perception and reasoning perception is primarily established through attention patterns associated with semantic tokens and query-boundary tokens. However, increased attention to answer-relevant regions near the end of a question may simply result from causal attention and contextual aggregation, rather than indicating a distinct reasoning perception mechanism.
Limited causal validation of the proposed interpretation. Although the authors conduct attention-head ablation experiments, performance degradation after removing selected heads does not necessarily establish their claimed functional specialization. More rigorous causal interventions are needed to demonstrate that semantic and reasoning perception constitute distinct mechanisms.
Heuristic design of the proposed method. SAF relies on several manually designed components, including entropy-based token selection, position-based token selection, layer filtering, attention smoothing, and mask fusion. The necessity of these specific choices is not sufficiently justified, and it remains unclear whether the improvements arise from the proposed structured perception mechanism or from generic attention-based image enhancement.
Insufficient comparisons with closely related methods. The experimental comparisons mainly focus on the original models and relatively simple attention-mask variants. Stronger comparisons with existing visual evidence selection, attention steering, and visual enhancement approaches are necessary to establish the advantages of SAF. Furthermore, the additional inference overhead is not adequately evaluated.
Questions for authors
NA
4 comments
W2: Reasoning perception goes beyond terminal position
(1) Thank you for the insightful observation. We agree that causal-context accumulation is related to reasoning perception. Our view is that when the model reaches both a syntactic boundary and a query terminal position, it aggregates the preceding question context and directs attention toward answer-critical visual evidence.
(2) Reasoning perception cannot be reduced to a simple last-token effect. To test this, we conducted two controlled experiments on Qwen2-VL-7B:
- We replaced the sentence-final "?" with ".", ",", "#", or removed it entirely. The results show that "?", ".", and "," elicit reasoning perception, whereas "#" does not. Removing punctuation also substantially reduces reasoning-perception intensity at the terminal-position token. Thus, terminal position alone does not always induce reasoning perception; specific non-semantic boundary tokens, such as sentence punctuation ("?", ".", ","), also matter.
- When suffix tokens are appended after the sentence, reasoning perception drops substantially. When "?" is inserted into the middle of the sentence, reasoning perception does not emerge. When additional queries (e.g., "Where is the [semantic token]?") are inserted before or after the original query, strong reasoning perception still appears regardless of whether the original query appears first or last. This shows that reasoning perception does not simply occur at the end of the input sequence; instead, it is tied to the interaction between query context and boundary tokens such as "?".
| Terminal variant | TermTok→Sem | TermTok→Reas | OtherTok→Sem | OtherTok→Reas |
|---|---|---|---|---|
? | 0.162 | 3.068 | 0.146 | 0.418 |
| no punctuation | 1.949 | 0.869 | 0.122 | 0.422 |
. | 0.073 | 2.612 | 0.150 | 0.413 |
, | 0.170 | 2.205 | 0.143 | 0.433 |
# | 0.773 | 0.326 | 0.137 | 0.458 |
| Variant | Probe role | ProbeTok→Sem | ProbeTok→Reas |
|---|---|---|---|
| dummy suffix | appended suffix | -0.218 | 1.168 |
| moved aggregation | appended Answer: | -0.163 | 1.485 |
| random question mark | inserted ? | -0.027 | 0.523 |
| multi-question | real boundary ? #1 | 0.181 | 3.114 |
| reordered multi-question | real boundary ? #3 | 0.040 | 2.402 |
W3: Interventions demonstrate functional selectivity
Thank you for pointing this out.
(1) The current results show that top semantic and reasoning heads are behaviorally important, and the existing token→region path ablation shows larger drops for structured paths than random-token controls.
| Existing path ablation | Accuracy (change) |
|---|---|
| Base | 58.3 (0.0) |
| SemTok→semantic bbox | 51.7 (-6.6) |
| TermTok→reasoning bbox | 53.3 (-5.0) |
| RandomTok→annotated regions | 57.2±0.78 (-1.1) |
(2) We ablate heads and examine the resulting attention patterns. Ablating reasoning heads substantially reduces attention to evidence while having a smaller effect on attention to semantic objects, indicating that these heads perform independent, specific functions. Similarly, ablating semantic heads substantially affects attention to semantic objects. The independent functions of these two types of heads indicate that the two types of perception constitute independent mechanisms within the model.
| Intervention | Accuracy | ΔAcc | SemTok→Sem | Δ | TermTok→Reas | Δ |
|---|---|---|---|---|---|---|
| Base | 43.3 | — | 4.232 | — | 2.766 | — |
| Semantic heads | 30.0 | -13.3 | 3.015 | -1.217 | 2.665 | -0.101 |
| Reasoning heads | 15.0 | -28.3 | 3.832 | -0.400 | 1.315 | -1.451 |
| Random heads | 34.7±4.52 | -8.7 | 4.290±0.280 | +0.059 | 2.540±0.331 |
W4: Structured selection has empirical support
(1) Low-entropy and query-boundary token selection: evidence from attention patterns. Figure A6 in the submission presents both an example of token-level attention entropy and the dataset-level distribution of entropy versus semantic BalGR. Human-annotated semantic tokens tend to occupy the low-entropy, high-semantic-BalGR region, providing empirical support for using low entropy to select semantic candidates. Figures 3–4 compare perceptual intensity across token categories and relative positions, showing that reasoning perception strengthens near query boundaries.
Table A1 further examines this boundary effect by replacing or removing the terminal marker. The results support selecting tokens near query boundaries.
| Terminal variant | TermTok→Sem | TermTok→Reas | OtherTok→Sem | OtherTok→Reas |
|---|---|---|---|---|
? | 0.162 | 3.068 | 0.146 | 0.418 |
| No punctuation | 1.949 | 0.869 | 0.122 | 0.422 |
. | 0.073 | 2.612 | 0.150 | 0.413 |
, | 0.170 | 2.205 | 0.143 | 0.433 |
# | 0.773 | 0.326 | 0.137 | 0.458 |
| Paraphrase | 0.138 | 3.080 | 0.103 | 0.310 |
(2) Token count: comparing a small token subset with all query tokens. Table 4(A) jointly varies the number of semantic tokens k and reasoning tokens m. All tested settings outperform Base in average accuracy across eight datasets. The k=m=2 setting achieves the best result of 65.7, compared with 64.6 when all tokens are selected. This supports using a small token subset as the default configuration.
| Setting | k | m | Avg. accuracy |
|---|---|---|---|
| Base | — | — | 62.4 |
| A | 1 | 1 | 65.1 |
| B | 2 | 2 | 65.7 |
| C | 3 | 3 | 64.7 |
| D | 5 | 5 | 64.8 |
| E | all | all | 64.6 |
(3) Perception-layer selection: comparing layer ranges with selection by perceptual intensity. Figure 3(a) shows the layer-wise intensity distributions of the two types of perception, which are concentrated in middle and later layers. Table 4(B) then compares early, middle, late, full, and selected layers within the same enhancement framework. The benefits of layer selection are consistent with the observed distribution of perceptual signals.
| Setting | Layers | Avg. accuracy |
|---|---|---|
| Base | — | 62.4 |
| A | Early Layers | 64.5 |
| B | Middle Layers | 65.2 |
| C | Late Layers | 65.4 |
| D | Full Layers | 65.3 |
| E | Selected Layers | 65.7 |
(4) Two-branch fusion and structured masking: evaluation against single-branch and generic masking controls. Table 3(A) evaluates the semantic and reasoning branches separately; Table 3(B) compares all-query and last-token attention; and Table A9 adds random same-area regions, central same-area regions, and random-layer controls.
| Method | A-OKVQA | InfoVQA | V* |
|---|---|---|---|
| Base | 61.9 | 42.1 | 57.1 |
| Random same-area | 62.7 | 42.2 | 55.3 |
| Random layer | 64.1 | 42.3 | 55.5 |
| Center same-area | 64.5 | 42.6 | 56.7 |
| All-query attention | 63.0 | 41.8 | 58.6 |
| Last-token attention | 63.4 | 43.4 | 58.1 |
| Semantic only | 64.8 | 43.6 | 54.5 |
| Reasoning only | 64.5 | 44.2 | 60.3 |
| SAF | 65.4 | 45.6 | 60.7 |
SAF outperforms both single branches and all listed generic masking controls on all three datasets. These results directly support the contributions of structured selection and two-branch fusion.
(5) Smoothing, thresholding, and connected-component processing: stability under parameter variation. Table A11 varies Gaussian smoothing, median filter size, threshold coefficient, and connected-region minimum area individually on A-OKVQA while holding the other parameters fixed.
| Varied component | Accuracy |
|---|---|
| Gaussian smoothing | 64.7±0.74 |
| Median filter size | 65.4±0.39 |
| Threshold coefficient | 65.4±0.10 |
| Connected-region minimum area | 65.8±0.39 |
W5a: Comparisons with related methods
(1) We compare methods using the same model and evaluation protocol on three representative datasets reported in the paper: A-OKVQA, InfoVQA, and V*. The baselines include random same-area masking (retaining a random square region of the same area), random layer (aggregating attention from several randomly selected layers to construct a mask), center same-area masking (retaining a central square region of the same area), all-query-token attention, and last-token attention. SAF outperforms these generic masking baselines, suggesting that the gains do not arise solely from removing background information or retaining regions favored by a center or objectness prior. SAF also outperforms the all-query-token attention baseline, indicating that the structured semantic/reasoning decomposition contributes beyond standard saliency enhancement.
| Method | A-OKVQA | InfoVQA | V* | Avg. |
|---|---|---|---|---|
| Base | 61.9 | 42.1 | 57.1 | 53.7 |
| Random same-area | 62.7 | 42.2 | 55.3 | 53.4 |
| Random layer | 64.1 | 42.3 | 55.5 | 54.0 |
| Center same-area | 64.5 | 42.6 | 56.7 | 54.6 |
| All-query attention | 63.0 | 41.8 | 58.6 | 54.5 |
| Last-token attention | 63.4 | 43.4 | 58.1 | 55.0 |
| Reasoning only | 64.5 | 44.2 | 60.3 | 56.7 |
| Semantic only | 64.8 | 43.6 | 54.5 | 54.3 |
| Full SAF | 65.4 | 45.6 | 60.7 | 57.2 |
(2) As shown in Table A8, SAF also outperforms the related methods [1] and [2].
| Method | A-OKVQA | InfoVQA | V* | Avg. |
|---|---|---|---|---|
| VisAttnSink adjusted | 63.9 | 42.9 | 57.6 | 54.8 |
| MLLMs-Know crop | 64.4 | 41.9 | 57.6 | 54.6 |
| Full SAF | 65.4 | 45.6 | 60.7 | 57.2 |
[1] See what you are told: Visual attention sink in large multimodal models
[2] Mllms know where to look: Training-free perception of small visual details with multimodal llms
(3) SAF is designed around the internal attention dynamics of MLLMs, providing both interpretability and generalizability.
W5b: SAF’s inference overhead is quantified
(1) Thank you for the question. SAF does introduce additional computational overhead. To quantify this, we measure the FLOPs and latency of Base and SAF. The results summarized below:
| Method | End-to-end latency (s) | FLOPs (T) | Relative latency | Relative FLOPs |
|---|---|---|---|---|
| Base | 5.07 | 10.15 | 1.00× | 1.00× |
| SAF online | 10.09 | 26.15 | 1.99× | 2.58× |
(2) We would like to clarify that our primary goal is not to propose a low-cost inference strategy, but to investigate MLLM internal perceptual mechanisms and verify whether they can provide effective visual enhancement signals.
We thank the reviewer for recognizing the value of the attention analysis and the experimental results across models and tasks. We will further emphasize the questions addressed by structured perception and separately explain the mechanistic evidence, implementation choices, and method comparisons.
W1: Novelty in structured perception
(1) Our core question is how MLLMs organize visual information serving distinct roles. Rather than claiming attention-map extraction or visual-region enhancement as our main novelty, we investigate MLLM internal perceptual mechanisms during complex VQA, uncovering a structured perceptual process that motivates our visual enhancement method. By separately annotating question semantic anchors and answer-relevant evidence, we characterize their divergent patterns across tokens, layers, and heads, and validate their functional roles through head and path interventions—shifting the focus from “where the model looks” to “which query representations attend to which evidence, and how these links govern the answer.”
(2) This decomposition also determines SAF’s specific design: low-entropy tokens provide semantic localization signals, query-boundary tokens provide answer-evidence signals, perception-layer selection reduces interference from layers with weaker signals, and fusion preserves complementary content. The generic-mask controls in Table A9 and comparisons with related methods in Table A8 of the original submission validate the method’s effectiveness.