acceptodds
Under review as a conference paper at ICLR 2027

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

Abstract

Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge-Based Visual Question Answering, Retrieval-Augmented Generation, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d36GOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Look Twice (LoT), a training-free KB-VQA framework that uses internal attention to identify relevant visual regions and textual evidence. The framework extracts object-to-visual attention to locate relevant image regions, filters the attention sink, and selects informative sentences from the retrieved documents. It then highlights the selected evidence using prompt tokens while removing irrelevant content before generating the final answer. The authors conducted experiments on 4 KB-VQA benchmarks and 10 MLLMs, demonstrating consistent performance improvements.

Critical strengths
  1. The proposed LoT framework leverages the internal attention mechanisms of pre-trained MLLMs to jointly select visual and textual evidence, without requiring additional training or architectural modifications. This provides a practical method for improving evidence utilization in KB-VQA and can be integrated into various architectures as a plug-in.
  2. The authors present extensive experiments across various MLLMs and benchmarks. Ablation studies evaluate the contributions of visual and textual highlighting, evidence localization, and different selection strategies; the experiments are thorough and comprehensive, providing strong empirical support for the framework’s effectiveness.
Critical weaknesses
  1. The novelty of the proposed method has not been fully elucidated. In the "Related Work" section, the authors mention several related inference-level attention analysis methods, such as Self-Elicit, FOCUS, and visual attention pooling methods. However, LoT shares several core mechanisms with these methods, including attention-based evidence selection, visual cropping, and evidence highlighting. The primary distinction appears to be the integration of visual and textual evidence refinement, but the authors do not explicitly explain the technical novelty of this combination. Furthermore, the experiments lack direct comparisons with these closely related methods, making it difficult to assess the actual advantages of LoT over existing approaches.
  2. Compared to text refinement, the effectiveness of visual evidence selection is relatively limited. As shown in Table 3, most performance improvements come from text evidence highlighting, while visual evidence selection provides only modest additional gains across settings, as shown in the relevant Qwen2.5-VL-7B tests. Whether the benefits of visual selection justify the added complexity and computational overhead remains to be thoroughly investigated.
Questions for authors
  1. Could you clarify the key technical differences between LoT and closely related methods such as SelfElicit and FOCUS? What are the main innovations beyond combining existing visual and textual evidence selection mechanisms?
  2. Could you provide direct comparisons with these related attention-based methods under the same models, datasets, and inference budgets?
  3. According to Table 3, visual evidence selection provides relatively limited additional improvements over textual refinement. Could you explain why this happens and identify the scenarios where visual highlighting is particularly beneficial?
3 comments

We thank the reviewer for the positive and constructive assessment, and for recognizing that LoT provides a practical, plug-in way to improve evidence utilization in KB-VQA without additional training or architectural changes, and that our experiments and ablations are thorough and comprehensive, providing strong empirical support for the framework. We address each concern below; new results will be included in the revised paper.

[W1 & Q1] Technical novelty with respect to SelfElicit, FOCUS and visual attention methods

LoT builds on insights from prior work, but its contribution is a general, training-free framework designed around the specific evidence-utilization challenges of KB-VQA, where the model must identify the relevant visual cues while reasoning over retrieved passages that contain a large amount of distracting or irrelevant content.

LoT addresses these challenges with a unified, question-conditioned evidence-refinement pipeline, in which a single analysis pass produces both the visual and the textual selection:

  • Textual refinement vs. SelfElicit. We adopt SelfElicit's attention-based sentence scoring (last-token attention, sentence-level averaging, last half of the layers, half-maximum threshold), and we will acknowledge this explicitly in Sec. 3.2. The key difference is how the scores are used. SelfElicit only highlights evidence and keeps the full context. LoT applies a three-way refinement: it highlights highly relevant sentences, keeps intermediate ones as supporting context, and removes low-relevance content (72.9% of the context tokens for Qwen2.5-VL-3B on E-VQA, Appendix E). As shown below, this removal tier accounts for most of the gain over SelfElicit (e.g., +5.3 points on InfoSeek with the 3B model).
  • Visual refinement vs. FOCUS and visual attention methods. These methods also rely on internal MLLM signals, but differ from LoT in the relevance signal, the region selection and the way the evidence is presented:
    • Relevance signal. FOCUS identifies target objects through an additional in-context generation step, queries them with an object-existence prompt, and estimates relevance through cosine similarity between the cached value features of target and visual tokens. LoT instead identifies the target with a lightweight parser and extracts query–key attention from its tokens in the original question, during the shared KB-VQA analysis pass.
    • Sink filtering. Methods that pool or redistribute visual attention (e.g., Kang et al., 2025b) modify attention during decoding. LoT uses the sink dimensions identified by Kang et al. for a different goal: to remove sink tokens before localization, which improves localization (Table 13: IoU 0.388 → 0.489).
    • Region selection and cost. FOCUS proposes several candidate regions and verifies them with additional object-existence forward passes. LoT derives a single bounding box from the weighted centroid and spread of the filtered map, with no verification passes beyond the analysis pass it shares with textual selection.
    • Presentation. The resulting crop replaces the original image and is enclosed in evidence markers, whereas ViCrop keeps the original image alongside the crop. Table 5 supports crop-only presentation in our setting: with highlighting in both configurations, E-VQA accuracy for the 7B model is 30.2 with the crop alone versus 29.6 with both views.
  • Common refinement interface. LoT obtains both selections from a shared analysis pass and presents the refined visual and textual evidence through a common marker-based interface, without updating the answer model’s parameters. The framework is evaluated across ten backbones and four benchmarks. The controlled comparisons below assess its benefit over straightforward combinations of existing textual and visual refinement methods.
  • Efficient access to internal attention. Attention-based ViCrop variants require explicit query–key attention weights, which efficient attention implementations do not ordinarily materialize. FOCUS instead uses cached value-feature similarities. LoT selectively reconstructs the required query–key slices, of size (∣Tobj∣+1)×S(|T_{\mathrm{obj}}|+1)\times S, with ∣Tobj∣≈2|T_{\mathrm{obj}}|\approx2, rather than materializing the full S×SS\times S matrix. It retains FlashAttention-2 for the model’s attention computation. Both FOCUS and LoT therefore support efficient attention implementations, through different mechanisms.

[Q2] Direct comparison with related attention-based methods

Following the reviewer’s suggestion, we evaluated the closest methods individually and in combination using the same backbones, retrieved inputs (n=3)(n=3), final answer-prompt template, and answer-generation limit. SelfElicit-style applies SelfElicit’s sentence scoring and highlighting without sentence removal. For FOCUS and ViCrop, we use their official implementations adapted to our KB-VQA setting.

Qwen2.5-VL-3B

MethodE-VQA SingleE-VQA AllInfoSeek U-QInfoSeek U-EInfoSeek All
Qwen2.5-VL-3B (no refinement)30.328.022.622.222.4
Text refinement
Self-Elicit-style31.729.124.524.024.2
LoT (Text)31.729.129.829.229.5
Visual refinement
FOCUS31.629.423.923.223.6
ViCrop31.429.324.023.323.7
LoT (Visual)31.629.424.123.723.9
Multimodal refinement
FOCUS + Self-Elicit-style32.529.925.924.925.4
ViCrop + Self-Elicit-style32.229.926.125.025.5
FOCUS + LoT (Text)32.630.129.728.429.1
ViCrop + LoT (Text)32.229.730.028.729.3
LoT (Full)32.630.230.329.830.1

Qwen2.5-VL-7B

MethodE-VQA SingleE-VQA AllInfoSeek U-QInfoSeek U-EInfoSeek All
Qwen2.5-VL-7B (no refinement)32.130.223.925.124.5
Text refinement
Self-Elicit-style33.631.424.925.425.1
LoT (Text)34.732.128.829.929.4
Visual refinement
FOCUS31.629.622.322.822.6
ViCrop31.929.722.923.623.2
LoT (Visual)32.430.224.425.424.9
Multimodal refinement
FOCUS + Self-Elicit-style33.230.923.923.623.7
ViCrop + Self-Elicit-style33.431.124.724.724.7
FOCUS + LoT (Text)34.131.428.329.428.8
ViCrop + LoT (Text)34.031.429.030.129.5
LoT (Full)35.032.229.330.930.1

Both combinations of prior methods score below full LoT across the four backbone–dataset configurations, considering E-VQA All and InfoSeek All. LoT exceeds FOCUS + SelfElicit-style by +0.3/+1.3 points on E-VQA and +4.7/+6.4 on InfoSeek for 3B/7B, respectively. The corresponding gains over ViCrop + SelfElicit-style are +0.3/+1.1 and +4.6/+5.4 points. Averaged across these configurations, the improvements are +3.2 and +2.9 points, respectively. Thus, the tested combinations do not fully recover LoT’s performance.

The component comparisons further clarify these differences:

  • Textual. LoT text-only improves over Self-Elicit-style by +2.6 points on average, with the largest gains on InfoSeek (+5.3/+4.3), while the two are close on E-VQA (0.0/+0.7). This is consistent with the role of the removal tier: it helps most when the retrieved context contains many distracting passages.
  • Visual. Both FOCUS and ViCrop fall below the vanilla baseline for the 7B model on both datasets (E-VQA: 29.6 and 29.7 vs. 30.2; InfoSeek: 22.6 and 23.2 vs. 24.5). On E-VQA and InfoSeek, LoT visual-only never falls below the baseline, and it matches or exceeds both methods in all four settings (+0.8 on average over FOCUS, +0.6 over ViCrop).
  • Composition. With LoT's textual refinement held fixed, the LoT visual selector is the best complement, outperforming both FOCUS (+0.1/+0.8 on E-VQA, +1.0/+1.3 on InfoSeek) and ViCrop (+0.5/+0.8 on E-VQA, +0.8/+0.6 on InfoSeek) in all four settings.

Paired uncertainty for the closest comparisons. To assess whether the smaller differences are conclusive, we report paired-bootstrap 95% confidence intervals for LoT (Full) minus each baseline. Differences are expressed in percentage points using the E-VQA All and official InfoSeek All metrics.

ComparisonModelE-VQA Δ [95% CI]InfoSeek Δ [95% CI]
LoT − (FOCUS + LoT Text)3B+0.1 [-0.40, +0.61]+1.0 [+0.22, +1.78]
LoT − (FOCUS + LoT Text)7B+0.8 [+0.37, +1.23]+1.3 [+0.93, +1.67]
LoT − (ViCrop + LoT Text)3B+0.5 [+0.12, +0.88]+0.8 [+0.21, +1.39]
LoT − (ViCrop + LoT Text)7B+0.8 [+0.27, +1.33]+0.6 [+0.15, +1.05]
LoT − (FOCUS + Self-Elicit-style)3B+0.3 [-0.08, +0.68]+4.7 [+4.29, +5.11]
LoT − (FOCUS + Self-Elicit-style)7B+1.3 [+0.96, +1.64]+6.4 [+6.05, +6.75]
LoT − (ViCrop + Self-Elicit-style)3B+0.3 [-0.12, +0.72]+4.6 [+4.1, +5.1]
LoT − (ViCrop + Self-Elicit-style)7B+1.1 [+0.77, +1.43]+5.4 [+5.02, +5.78]

[W2 & Q3] Contribution of visual evidence selection

We agree that textual refinement accounts for the larger share of the improvement in Table 3. In KB-VQA, the image supports entity identification, while the requested facts often come from retrieved passages. The full image has also already been used upstream for retrieval. Visual refinement therefore addresses remaining localization and disambiguation difficulties whose importance varies across examples.

Incremental contribution. We distinguish visual refinement in isolation from its contribution within the complete framework:

Visual-only refinement vs. vanilla

DatasetModelVanillaVisual-onlyΔ
E-VQA3B28.029.4+1.4
E-VQA7B30.230.20.0
InfoSeek3B22.423.9+1.5
InfoSeek7B24.524.9+0.4

Full refinement vs. text-only refinement

DatasetModelText-onlyFullΔ
E-VQA3B29.130.2+1.1
E-VQA7B32.132.2+0.1
InfoSeek3B29.530.1+0.6
InfoSeek7B29.430.1+0.7

Across all four benchmarks in Table 3, adding visual refinement to text-only LoT yields positive accuracy differences in all eight model–dataset configurations. Paired pointwise 95% bootstrap intervals exclude zero in the positive direction in six of eight configurations; the small gains on E-VQA 7B and ViQuAE 3B remain inconclusive. The full intervals and protocol are reported in our response to Reviewer 5Sxc (W2/Q3).

Visual-only refinement improves six configurations, matches one, and reduces accuracy on ViQuAE 7B (36.4 → 34.3). We explicitly acknowledge this regression. On the same configuration, adding visual refinement to text-only LoT improves accuracy from 49.6 to 50.5. Thus, its effect depends on the accompanying textual evidence, and cropping is not uniformly beneficial.

Settings where visual refinement is more impactful. Within KB-VQA, the largest visual-only gain is on OVEN with 3B (+5.0 points). Table 7 provides complementary evidence from benchmarks without retrieval: visual refinement improves all five evaluated backbones by +1.0 to +4.2 points on average, with gains of up to +8.3 on V*. Those experiments supplement the original image with the crop rather than replacing it, supporting the usefulness of the selected regions under a different presentation strategy. These results identify settings where visual refinement is effective, although they do not establish which individual examples will benefit.

Please also see our response to Reviewer 5Sxc for the paired bootstrap intervals on the visual gains and the analysis of cases where cropping reduces accuracy (W2/Q3), the marker and sink-filtering ablations (W3/Q2), and the end-to-end latency analysis (Q4); and our response to Reviewer qoBm for a budget-matched joint baseline with BM25 and a center crop (W2/Q2).

Reviewer_qoBmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Look Twice (LoT), a training-free, two-pass inference method for knowledge-based visual question answering. In its first pass, the model's internal attention identifies a query-related image region and relevant sentences in retrieved documents. LoT filters high sink-score visual tokens, crops the image, removes low-score sentences, and marks selected evidence in a revised prompt for final answer generation. Across E-VQA, InfoSeek, OVEN, and ViQuAE, all ten evaluated backbones improve, with mean gains of 3.1 to 12.5 accuracy points over the four benchmark scores.

Critical strengths

S1. The empirical breadth is a clear strength. Table 1 evaluates ten backbones from four model families, spanning 2B to 38B parameters, on four knowledge-based VQA benchmarks. Every backbone improves on the reported four-benchmark average, including the larger models. This makes the result more persuasive than a gain confined to one architecture or dataset. S2. The paper includes several useful controls. Table 3 separates visual and textual components; Tables 4–6 examine budget-matched selection and alternative highlighting choices; Figure 4 distinguishes retrieval hits, retrieval misses, oracle evidence, and retrieval depth. These analyses help establish that LoT's gains involve evidence selection and are not explained solely by shorter inputs. S3. The design requires no parameter updates to the answer model and is compatible with existing inference strategies (Table 2). Appendix E describes computing only the needed attention slices while retaining FlashAttention-2. For Qwen2.5-VL-3B on E-VQA, the second-pass input shrinks by 71.93%; reported model-side latency rises by 14.1%, while total FLOPs rise by 42.79%. Reporting both measures makes the practical trade-off easier to assess.

Critical weaknesses

W1. The proposed attention-sink mechanism is not yet isolated from generic sparsification. With the sink threshold at the 25th percentile, roughly three-quarters of visual tokens are excluded from localization when scores are distinct. Table 13 compares this filter with no filter, but not with equal-sparsity random, activation-based, or attention-based masks. Its localization test uses COCO objects occupying 25–50% of the image area; the E-VQA localization analysis in Table 12 uses GroundingDINO predictions as proxy boxes. The observed IoU improvement is valuable, but it does not by itself establish that the BOS-derived dimensions specifically identify irrelevant attention sinks or that this step improves final answers beyond a similarly sized crop. W2. The incremental value of joint visual selection is uneven, and the strongest comparison is still missing. In Table 3, visual-only LoT on Qwen2.5-VL-7B matches the E-VQA All baseline at 30.2 and lowers ViQuAE from 36.4 to 34.3. Full LoT exceeds text-only LoT by just 0.1 point on E-VQA All and 0.7 on InfoSeek All for that model. Table 4 gives useful budget-matched controls, but evaluates text selectors and image crops separately; Tables 5–6 show that markers and sentence removal also matter. A joint baseline combining a strong non-attention text selector with a crop, under the same prompt and budget, is needed to support the claimed contribution of the attention-based multimodal combination. W3. The evidence-quality analysis does not directly measure whether the retained sentences support the answer. In Table 10, sentence precision is defined by membership in the oracle document, which is a weaker test than answer-support precision. E-VQA includes two-hop questions, yet Table 1 reports Single-Hop and All rather than a separate two-hop score. Because Appendix E reports removing 72.90% of context tokens in the measured setting, it would be useful to know how often bridge facts are discarded and whether the method helps or harms multi-hop cases. Human-checked answer-support labels on a sample and an error breakdown would make the selection claim more concrete. W4. The size and cost of the smaller incremental gains need more context. Some additions over refined baselines in Table 2 are only about 0.4–1.2 points, without paired uncertainty estimates or significance tests. Greedy decoding eliminates sampling variation in generation but does not remove uncertainty from a finite evaluation set. The 14.1% latency and 42.79% FLOP overhead in Appendix E are reported for one 3B backbone on E-VQA, while the limitations note additional CPU alignment work. End-to-end latency and memory measurements for larger backbones would clarify the practical value of these gains.

Questions for authors

Q1. For W1, can the authors compare the BOS-dimension filter against equal-sparsity random and non-BOS activation masks, and report both localization and downstream VQA accuracy with fixed crop-area distributions? This would test whether the claimed sink mechanism is needed. Q2. For W2–W3, can the authors provide a prompt- and budget-matched joint baseline, such as BM25 or prompted sentence selection plus a center crop with identical evidence markers? Please also report E-VQA two-hop accuracy and the recall of genuinely answer-supporting sentences on an annotated subset. Q3. For W4, can the authors give paired bootstrap intervals for the smaller Table 2 gains and end-to-end timing and peak memory on representative small and large backbones, including CPU preprocessing? These figures would help interpret the accuracy–cost trade-off.

5 comments

We thank the reviewer for the constructive review, and for recognizing the empirical breadth of our evaluation (ten backbones, four families, 2B–38B, all improving), the usefulness of our controls in showing that the gains come from evidence selection rather than shorter inputs, and the practicality of LoT: no parameter updates, compatibility with existing inference strategies, and FlashAttention-2 support. We address each concern below; new results will be included in the revised paper.

[W1 & Q1] BOS-based filtering versus equal-sparsity alternatives

We thank the reviewer for suggesting this control. We agree that comparison with no filtering alone does not distinguish the BOS-derived signal from generic sparsification. We therefore compare it with non-BOS activation-based masks and random masks, evaluating both COCO localization and downstream KB-VQA accuracy.

Baseline construction.

  • Non-BOS activations. On an independent calibration set of 100 images, we rank hidden-state dimensions by their mean normalized absolute activation over visual tokens, explicitly excluding BOS-associated dimensions. We select the same number of dimensions as in the BOS method and apply the same token-scoring rule using these alternative dimensions. The highest-scoring visual tokens are excluded from localization. No localization or answer-accuracy results are used to select the dimensions.
  • Random masks. We sample visual-token positions uniformly without replacement across the entire visual grid, excluding exactly the same number of tokens as the BOS method for each example. We evaluate five independent seeds and report their mean rather than selecting the best realization.

Equal sparsity and matched crop geometry. All masks exclude the same number of visual tokens per example from localization, not directly from the final model input. We distinguish natural crops, whose position and size are determined by each mask, from size-matched crops. In the latter, each control uses its own estimated center but exactly the BOS crop width and height for the same example, shifting the box within image boundaries rather than clipping it. Thus, crop area and aspect ratio are controlled per example, not merely in distribution.

COCO localization - mean IoU

Mask3B natural3B size-matched7B natural7B size-matched
BOS (Ours)0.4890.4890.5170.517
Non-BOS activations0.3840.4010.4130.422
Random global0.3250.3970.3160.391

Localization is evaluated on a held-out set of 475 COCO objects using ground-truth bounding boxes. Random results are averaged across five seeds.

Matching crop dimensions improves localization for both alternatives, indicating that crop size explains part of the natural-crop gap. Nevertheless, BOS retains an IoU advantage of 0.088/0.095 over non-BOS activations and 0.092/0.126 over random masks for 3B/7B, respectively. Thus, its localization advantage is not explained solely by crop dimensions.

Downstream accuracy - size-matched crops only (%)

ModelMaskE-VQA singleE-VQA allInfoSeek UQInfoSeek UEInfoSeek all
3BBOS (Ours)32.630.230.329.830.1
3BNon-BOS activations31.129.429.729.129.4
3BRandom global (5-seed mean)31.029.329.729.129.4
7BBOS (Ours)35.032.229.330.930.1
7BNon-BOS activations34.531.828.629.729.2
7BRandom global (5-seed mean)34.331.728.629.629.2

All downstream comparisons use identical per-example crop dimensions, frozen LoT-refined text, identical prompts and evidence markers, and identical realized input-token budgets. Thus, differences cannot be attributed to retaining more textual evidence or using larger crops.

Under these controls, BOS filtering yields higher downstream point estimates for both backbones and datasets. Compared with non-BOS activation masks, it improves E-VQA All by 0.8/0.4 percentage points and InfoSeek All by 0.7/0.9 points for 3B/7B. Compared with random masks, the corresponding gains are 0.9/0.5 points on E-VQA and 0.7/0.9 points on InfoSeek.

Together, these results support the utility of BOS-associated dimensions beyond generic sparsification: the localization advantage persists after matching crop geometry, and downstream point estimates favor BOS under identical crop dimensions and textual inputs. We interpret this as evidence for the usefulness of the BOS-derived selection signal, rather than proof that every excluded token is an irrelevant attention sink.

[W2 & Q2] Joint non-attention baseline and incremental value of visual selection

Joint non-attention baseline. We agree that separate evaluations of textual selectors and visual crops do not establish the benefit of their combination. We therefore added the requested BM25 + center crop comparison on Qwen2.5-VL-3B and 7B, on both E-VQA and InfoSeek.

The baseline ranks retrieved sentences with BM25, retains them under a target budget of 30% of the original context tokens, and highlights the highest-ranked retained sentences under a 20% retained-token budget. Selected sentences preserve their original order. The visual component uses a centered crop covering approximately 46% of the original image area. The backbone, retrieved inputs, final prompt template, textual and visual markers, and maximum answer length (128 tokens) are identical to LoT. Both budgets are set to match LoT’s average statistics on E-VQA (~70% context reduction, ~46% crop area). LoT decides per sample how much text to keep and where to crop, with no preset budget.

For completeness, we also report the attention-based combination of the closest prior methods (see our response to Reviewer 5Sxc, [W1 & Q1]).

MethodE-VQA 3BE-VQA 7BInfoSeek 3BInfoSeek 7B
Vanilla28.030.222.424.5
FOCUS + Self-Elicit-style29.930.925.423.7
ViCrop + Self-Elicit-style29.931.125.524.7
BM25 + center crop29.231.828.729.6
LoT (Full)30.232.230.130.1
Δ LoT − (BM25 + center crop)+1.0+0.4+1.4+0.5

The joint BM25 + center crop baseline improves over vanilla inference in all four configurations, confirming the value of combining textual and visual refinement even with non-attention selectors. LoT further improves accuracy across all four settings, by +1.0/+0.4 points on E-VQA and +1.4/+0.5 on InfoSeek for 3B/7B, respectively (+0.83 points on average). It also outperforms both combinations of prior methods throughout. These results support the benefit of LoT’s adaptive evidence selection within the shared refinement framework, while showing that a joint non-attention baseline already captures a substantial part of the overall improvement.

Paired uncertainty for the closest comparisons. We report paired-bootstrap 95% confidence intervals for LoT (Full) minus each baseline. Differences are expressed in percentage points using E-VQA All and the official InfoSeek All metric.

ComparisonModelE-VQA Δ [95% CI]InfoSeek Δ [95% CI]
LoT − (BM25 + center crop)3B+1.0 [+0.33, +1.67]+1.4 [+1.09, +1.71]
LoT − (BM25 + center crop)7B+0.4 [+0.05, +0.75]+0.5 [+0.08, +0.92]

Incremental value of visual selection. We agree that most of the improvement comes from textual refinement in Table 3. This is expected in KB-VQA. The information needed to answer is mostly contained in the retrieved passages rather than in visible attributes and the full image is already exploited upstream for entity retrieval.

Visual refinement nevertheless provides a consistent complementary contribution (Table 3):

Visual-only refinement vs. vanilla

DatasetModelVanillaVisual-onlyΔ
E-VQA3B28.029.4+1.4
E-VQA7B30.230.20.0
InfoSeek3B22.423.9+1.5
InfoSeek7B24.524.9+0.4

Full refinement vs. text-only refinement

DatasetModelText-onlyFullΔ
E-VQA3B29.130.2+1.1
E-VQA7B32.132.2+0.1
InfoSeek3B29.530.1+0.6
InfoSeek7B29.430.1+0.7

Visual-only refinement improves accuracy in three of the four evaluated configurations and matches the baseline in the remaining one, with gains of up to +1.5 points. When added to text-only LoT, it yields positive differences across all four settings: +1.1/+0.1 points on E-VQA and +0.6/+0.7 on InfoSeek for 3B/7B, respectively. These results indicate a complementary contribution whose magnitude depends on the backbone and dataset.

The E-VQA two-hop accuracy and the recall of answer-supporting sentences on an annotated subset (second part of Q2) are reported in W3/Q2 below. Please also see our response to Reviewer 5Sxc for the comparison with combinations of prior attention-based methods (W1/Q1), the paired bootstrap intervals on the visual gains and the analysis of cases where cropping reduces accuracy (W2/Q3), and the marker ablation with identical retained context and the downstream sink-filtering ablation (W3/Q2). The sink-filtering controls requested by the reviewer are discussed in W1/Q1 above, and the latency analysis in W4/Q3 below.

[W3 & Q2] Answer-support quality of the selected evidence and two-hop questions

We thank the reviewer for this suggestion. We agree that measuring sentence precision by membership in the oracle document (Table 10) is a weaker test than measuring whether the retained sentences actually support the answer. We therefore added two analyses: (i) separate E-VQA two-hop accuracy and (ii) human annotation of example-level answer-support retention and highlighting.

E-VQA two-hop accuracy

The table below reports the official E-VQA scores on the 4,750 single-hop and 1,000 two-hop test questions.

ModelMethodSingle-hopTwo-hopAll
Qwen2.5-VL-3BBaseline30.317.228.0
LoT32.618.430.2
Qwen2.5-VL-7BBaseline32.120.830.2
LoT35.019.332.2

On two-hop questions, LoT improves accuracy by +1.2 points for 3B and lowers it by −1.5 points for 7B.

To contextualize these results, E-VQA’s chained two-hop questions require first identifying a bridge entity linked to the entity in the image, then retrieving information about that bridge entity to answer the question (Mensink et al., 2023, Sec. 4.3). Our method instead focuses on refining evidence from a fixed retrieved context and does not explicitly implement this sequential retrieval process. So, we report these results for completeness, while emphasizing that addressing sequential multi-hop retrieval is outside LoT’s scope, which is to improve the utilization of already retrieved evidence to generate a more grounded answer.

Human annotation of answer-support retention and highlighting

To assess whether LoT preserves answer-supporting evidence, we conducted a human annotation study on 100 E-VQA examples per generator, Qwen2.5-VL-3B and Qwen2.5-VL-7B, randomly sampled from retrieval-hit cases where the gold entity document was included in the context. Examples were not filtered by the correctness of the generated answer.

Annotators inspected the original and refined contexts and assessed whether the refined context contained at least one sentence supporting the reference answer, including semantically equivalent formulations. They also recorded whether at least one supporting sentence was highlighted. This distinguishes the availability of answer support in the retained context from its explicit highlighting.

GeneratorAnnotated examplesAnswer support retainedAnswer support highlighted
Qwen2.5-VL-3B10081/100 (81%)75/100 (75%)
Qwen2.5-VL-7B10076/100 (76%)62/100 (62%)

These results distinguish preserving useful evidence from explicitly highlighting it. LoT retains at least one answer-supporting sentence in 81%/76% of the annotated retrieval-hit examples for 3B/7B, respectively. In 6/14 examples, support survives only in the unhighlighted context, illustrating the value of evaluating all retained sentences rather than highlighted evidence alone. Conversely, no answer-supporting sentence remains in 19%/24% of cases, identifying evidence loss as a remaining limitation.

Reviewer_5SxcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper propos LoT, a training-free inference procedure for knowledge-based visual question answering. An initial forward pass extracts internal attention signals to identify relevant image regions and retrieved sentences. The proposed method filters visual attention sinks, crop the image, removes low-scoring sentences, and hightlight selected evidence through prompt markers. The same model then generates an answer from the refined input.

Experiments across four benchmarks and ten multimodal models report consistent improvements in average accuracy, with additional benefits when combined with existing evidence-refinement methods.

Critical strengths
  1. Evaluating ten backbones spanning 2B–38B parameters across four benchmarks provides meaningful evidence that the overall procedure is useful. Comparisons against the same backbone and retrieval pipeline help isolate the effect of adding LoT.
  2. The paper includes modality ablations, alternative selection strategies, highlighting variants, and analyses of retrieval depth and evidence availability. Improvements when the ground-truth document is supplied are particularly useful evidence that better evidence utilization matters independently of retrieval quality.
Critical weaknesses
  1. The advance over closely related methods needs clearer characterization. The textual component closely resembles SelfElicit, including first-token attention extraction, sentence-level averaging, aggregation over the final half of the layers, a half-maximum threshold, and explicit evidence markers. Visual sink detection also closely follows "See What You Are Told: Visual Attention Sink in Large Multimodal Models", while internal-signal-based cropping has been explored by MLLMs Know Where to Look and FOCUS. The contribution therefore appears primarily to be their adaptation and integration for KB-VQA, together with selective sentence removal. Although these works are cited, the paper would benefit from a more explicit account of the incremental contribution and a comparison against a straightforward combination of the closest methods. Table 6’s highlighting-without-discarding variant provides a relevant partial control.
  2. The added contribution of visual refinement is modest and not uniformly beneficial. Text refinement accounts for most of the gains. For Qwen2.5-VL-7B, adding visual refinement to the text-only variant improves E-VQA, InfoSeek, OVEN, and ViQuAE by 0.1, 0.7, 1.3, and 0.9 points, respectively. Visual-only refinement reduces ViQuAE accuracy from 36.4 to 34.3. These results support some complementarity, but the evidence for a substantial advantage from refining both modalities is limited.
  3. Several mechanism claims remain incompletely isolated. The textual ablations do not compare identical retained context with and without highlighting markers; the “keep only evidence” variant also changes the filtering threshold. Likewise, the sink-filtering ablation demonstrates improved localization but does not directly quantify its contribution to downstream KB-VQA accuracy.
Questions for authors
  1. What specific methodological contribution distinguishes LoT from adapting SelfElicit to multimodal inputs and combining it with an existing visual cropping method? Can you provide a controlled comparison against such a combination?
  2. With the retained sentences held fixed, how much improvement comes from the highlighting markers? Similarly, how much does sink filtering improve downstream answer accuracy when all other components remain unchanged?
  3. Can you provide paired confidence intervals for the gains from adding visual refinement, and characterize cases where cropping removes useful context or reduces accuracy?
  4. What is the end-to-end latency, including parsing, token alignment, attention processing, and image preprocessing? How does this overhead change with model size and context length?
6 comments

We thank the reviewer for the thoughtful feedback, and for recognizing that our evaluation across ten backbones (2B–38B) and four KB-VQA benchmarks, run under the same backbone and retrieval pipeline, provides meaningful evidence of LoT's usefulness. We also appreciate the positive assessment of our ablations and analyses, in particular the oracle-evidence results, which show that better evidence utilization matters independently of retrieval quality. We address each concern below; new results will be included in the revised paper.

[W1] & [Q1] Difference with existing works

We agree that LoT should distinguish more explicitly between the mechanisms adopted from prior work and the contribution of their integration and adaptation to KB-VQA. We clarify these distinctions below and provide the requested comparisons against existing methods and their combinations.

What LoT adopts and what it adds

Textual refinement vs. SelfElicit. LoT builds on SelfElicit's attention-based sentence scoring (last-token attention, sentence-level averaging, last half of the layers, half-maximum highlighting threshold), and we will acknowledge this explicitly in Sec. 3.2. The difference lies in how the scores are used. SelfElicit highlights the selected sentences but keeps the full context, whereas LoT applies a three-way refinement: it highlights high-scoring sentences, keeps intermediate-scoring ones as supporting context, and removes low-scoring ones. This design targets the large amount of distracting content in retrieved KB-VQA passages: for Qwen2.5-VL-3B on E-VQA (Appendix E), it removes 72.9% of the context tokens. The comparison below shows that this refinement, rather than highlighting alone, accounts for most of the gain over SelfElicit (e.g., +5.3 points on InfoSeek with the 3B model).

Visual refinement vs. FOCUS and prior cropping methods. These methods use internal MLLM signals for cropping, but differ in their relevance estimates, region selection, and input presentation:

  • Relevance signal. FOCUS identifies target objects through an additional in-context generation step, then uses an object-existence prompt to compute cosine similarity between cached target-token and visual-token value features. LoT instead identifies the target with a lightweight parser and extracts query–key attention from its tokens in the original question, during the analysis pass on the KB-VQA input. Unlike attention-based ViCrop variants that require explicit attention matrices, LoT reconstructs only the necessary attention slices while retaining FlashAttention-2. Compatibility with efficient attention is therefore shared with FOCUS, but achieved through a different mechanism.
  • Sink filtering. We adopt the sink-dimension signal from Kang et al. (2025b), but use it to filter visual tokens before localization, rather than to redistribute attention during decoding. This improves localization (Table 13: IoU 0.388 → 0.489). Our new downstream ablation (W3/Q2) additionally shows small, non-negative accuracy changes across the evaluated configurations.
  • Region selection and cost. FOCUS proposes candidate regions and reranks them through additional object-existence queries. LoT directly derives a single bounding box from the weighted centroid and spread of the filtered map. It requires no candidate-verification passes and shares its analysis pass with textual evidence selection.
  • Presentation. LoT replaces the original image with the selected crop and encloses it in evidence markers, whereas ViCrop supplies the original image alongside the crop. Table 5 supports crop-only presentation in our setting: with highlighting in both configurations, E-VQA accuracy for the 7B model is 30.2 with the crop alone versus 29.6 with both views.

Common refinement interface. The selected visual and textual evidence is passed to the answer model through the same marker-based interface, with no parameter updates or auxiliary models. We position this as a contribution in the design and empirical validation of multimodal evidence refinement, rather than claiming that attention-based selection, cropping, or highlighting are individually new.

Controlled comparisons

Following the reviewer's suggestion, we re-implemented the closest methods and evaluated them under the same backbones, retrieval pipeline (n=3n=3), prompt template and decoding settings. We include individual textual and visual methods, their straightforward combinations, and combinations that hold LoT’s textual refinement fixed while changing the visual selector.

Qwen2.5-VL-3B

MethodE-VQA SingleE-VQA AllInfoSeek U-QInfoSeek U-EInfoSeek All
Qwen2.5-VL-3B (no refinement)30.328.022.622.222.4
Text refinement
Self-Elicit-style31.729.124.524.024.2
LoT (Text)31.729.129.829.229.5
Visual refinement
FOCUS31.629.423.923.223.6
ViCrop31.429.324.023.323.7
LoT (Visual)31.629.424.123.723.9
Multimodal refinement
FOCUS + Self-Elicit-style32.529.925.924.925.4
ViCrop + Self-Elicit-style32.229.926.125.025.5
FOCUS + LoT (Text)32.630.129.728.429.1
ViCrop + LoT (Text)32.229.730.028.729.3
LoT (Full)32.630.230.329.830.1

Qwen2.5-VL-7B

MethodE-VQA SingleE-VQA AllInfoSeek U-QInfoSeek U-EInfoSeek All
Qwen2.5-VL-7B (no refinement)32.130.223.925.124.5
Text refinement
Self-Elicit-style33.631.424.925.425.1
LoT (Text)34.732.128.829.929.4
Visual refinement
FOCUS31.629.622.322.822.6
ViCrop31.929.722.923.623.2
LoT (Visual)32.430.224.425.424.9
Multimodal refinement
FOCUS + Self-Elicit-style33.230.923.923.623.7
ViCrop + Self-Elicit-style33.431.124.724.724.7
FOCUS + LoT (Text)34.131.428.329.428.8
ViCrop + LoT (Text)34.031.429.030.129.5
LoT (Full)35.032.229.330.930.1

These adaptations provide useful baselines, but their straightforward combination remains below LoT full in all four evaluated settings: LoT improves over FOCUS + Self-Elicit-style refinement by +0.3/+1.3 points on E-VQA and +4.7/+6.4 points on InfoSeek for 3B/7B, respectively (+3.2 on average). Thus, the benefit of LoT is not recovered simply by placing an existing textual highlighting strategy alongside an existing visual cropping strategy.

Looking at the individual components:

  • Textual. LoT text-only improves over Self-Elicit-style by +2.6 points on average, with the largest gains on InfoSeek (+5.3/+4.3). On E-VQA the two are close (0.0/+0.7). This is consistent with the role of the removal tier: it helps most when the retrieved context contains many distracting passages.
  • Visual. Both FOCUS and ViCrop fall below the vanilla baseline for the 7B model on both datasets (E-VQA: 29.6 and 29.7 vs. 30.2; InfoSeek: 22.6 and 23.2 vs. 24.5). On E-VQA and InfoSeek, LoT visual-only never falls below the baseline, and it matches or exceeds both methods in all four settings (+0.8 on average over FOCUS, +0.6 over ViCrop).
  • Composition. With LoT's textual refinement held fixed, the LoT visual selector is the best complement, outperforming both FOCUS (+0.1/+0.8 on E-VQA, +1.0/+1.3 on InfoSeek) and ViCrop (+0.5/+0.8 on E-VQA, +0.8/+0.6 on InfoSeek) in all four settings.

Paired uncertainty for the closest comparisons. To assess whether the smaller differences are conclusive, we report paired-bootstrap 95% confidence intervals for LoT (Full) minus each baseline. Differences are expressed in percentage points using the E-VQA All and official InfoSeek All metrics.

ComparisonModelE-VQA Δ [95% CI]InfoSeek Δ [95% CI]
LoT − (FOCUS + LoT Text)3B+0.1 [-0.40, +0.61]+1.0 [+0.22, +1.78]
LoT − (FOCUS + LoT Text)7B+0.8 [+0.37, +1.23]+1.3 [+0.93, +1.67]
LoT − (ViCrop + LoT Text)3B+0.5 [+0.12, +0.88]+0.8 [+0.21, +1.39]
LoT − (ViCrop + LoT Text)7B+0.8 [+0.27, +1.33]+0.6 [+0.15, +1.05]
LoT − (FOCUS + Self-Elicit-style)3B+0.3 [-0.08, +0.68]+4.7 [+4.29, +5.11]
LoT − (FOCUS + Self-Elicit-style)7B+1.3 [+0.96, +1.64]+6.4 [+6.05, +6.75]
LoT − (ViCrop + Self-Elicit-style)3B+0.3 [-0.12, +0.72]+4.6 [+4.1, +5.1]
LoT − (ViCrop + Self-Elicit-style)7B+1.1 [+0.77, +1.43]+5.4 [+5.02, +5.78]

[W2 & Q3] Contribution of visual refinement

We thank the reviewer for highlighting the need to quantify the contribution of visual refinement and characterize its limitations.

Why the contribution varies

In KB-VQA, the image serves mainly to identify the entity, while the requested facts are usually found in the retrieved passages. In addition, the full image has already been used upstream to retrieve these passages. Visual refinement therefore addresses a residual problem, localizing or disambiguating the entity, whose importance varies from example to example: it matters when the entity is small or surrounded by distractors, and much less when it already dominates the image. This plausibly explains why the incremental visual gains are smaller and less uniform than the textual ones.

Nevertheless, visual refinement provides consistent complementary gains (Table 3):

  • Adding visual refinement on top of textual refinement improves accuracy in all 8 model–dataset settings. The gains are +1.1/+0.6/+1.5/+0.1 for the 3B model and +0.1/+0.7/+1.3/+0.9 for the 7B model on E-VQA/InfoSeek/OVEN/ViQuAE.
  • Visual-only refinement improves over the baseline in 6 of 8 settings and matches it in a seventh, including +5.0 on OVEN for the 3B model. The only decrease (ViQuAE, 7B: 36.4 → 34.3) does not persist in the full framework, where visual refinement improves ViQuAE over text-only LoT (49.6 → 50.5). This suggests that, with a noisy context, the model relies on the full image to disambiguate the entity, whereas once the context has been refined the crop becomes beneficial.
  • When the bottleneck is visual rather than textual, the effect is larger: on standard MLLM benchmarks without retrieval (Table 7), visual-only LoT improves all five backbones (+1.0 to +4.2 on average, up to +8.3 on V*). Although these experiments added the crop to the original image instead of replacing it, they still hint that the evidence extracted helps the model to exploit better the visual evidence.
  • The crop also removes 54% of the visual tokens in the second pass.

Paired confidence intervals

Following the reviewer's suggestion, we computed paired bootstrap 95% CIs on Qwen2.5-VL-3B/7B, comparing (i) visual refinement in isolation, which measures visual refinement with the original retrieved text; and (ii) full LoT − text-only LoT, which measures the incremental contribution of visual refinement within the complete framework.

Within each contrast, the backbone, retrieved inputs, answer-generation limit and textual-refinement hyperparameters are identical across conditions.

Visual refinement in isolation

DatasetModelVanillaVisual-onlyΔ [95% CI]
E-VQA3B28.029.4+1.4 [+0.75, +2.05]
E-VQA7B30.230.2+0.0 [-0.62, +0.62]
InfoSeek3B22.423.9+1.5 [+1.33, +1.67]
InfoSeek7B24.524.9+0.4 [+0.05, +0.75]
Oven3B11.616.6+5.0 [+4.32, +5.68]
Oven7B20.221.0+0.8 [+0.35, +1.25]
Viquae3B22.923.6+0.7 [+0.22, +1.18]
Viquae7B36.434.3-2.1 [-3.20, -1.00]

Full refinement versus text-only refinement

DatasetModelText-onlyFullΔ [95% CI]
E-VQA3B29.130.2+1.1 [+0.49, +1.72]
E-VQA7B32.132.2+0.1 [-0.50, +0.71]
InfoSeek3B29.530.1+0.6 [+0.32, +0.88]
InfoSeek7B29.430.1+0.7 [+0.55, +0.85]
Oven3B28.029.5+1.5 [+0.55, +2.45]
Oven7B28.730.0+1.3 [+0.83, +1.77]
Viquae3B34.634.7+0.1 [-0.46, + 0.66]
Viquae7B49.650.5+0.9 [+0.21, +1.59]

Bootstrap protocol. We use 10,000 paired question-level bootstrap replicates (seed 42), resampling the same question IDs for both methods in each comparison. E-VQA contains 5,750 examples and InfoSeek contains 71,335. For InfoSeek, resampling is stratified by the unseen-question and unseen-entity splits, preserving their sizes and recomputing the official harmonic-mean score in each replicate. We report pointwise percentile intervals without multiple-comparison correction. These intervals quantify finite-evaluation-set uncertainty, rather than variability across generation seeds.

Comment. For full LoT versus text-only LoT, the pointwise 95% confidence intervals exclude zero in the positive direction in six of eight configurations. The small gains on E-VQA 7B and ViQuAE 3B remain inconclusive. Thus, the results support a complementary contribution from visual refinement across several settings, without establishing an improvement in every configuration.

Comments on the paper

2 comments

We thank all reviewers for their thoughtful feedback and constructive suggestions. We have posted individual responses addressing their comments, including additional experiments and analyses. We are incorporating these results and clarifications into the paper and will upload a revised version in the coming days. In the meantime, we remain available for any further questions or clarifications.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

We sincerely thank all reviewers for their constructive feedback and valuable suggestions, which have helped us strengthen the paper.

We have uploaded a revised version of the manuscript incorporating the additional experiments, analyses, and clarifications discussed during the rebuttal. All newly added content is highlighted in red throughout the revised manuscript.

In particular, the revised paper includes:

  • Expanded comparisons with prior methods, including SelfElicit, FOCUS, ViCrop, their combinations, and a joint non-attention-based baseline.
  • Additional ablation studies isolating the contributions of textual highlighting markers and BOS-based attention-sink filtering, including controlled localization experiments and evaluations on vision-centric benchmarks.
  • Further evidence-quality analysis, including human annotations of answer-supporting sentence retention and highlighting.
  • Extended computational analysis, covering end-to-end latency, context-length scaling, and peak GPU memory.

We have also revised the methodological descriptions and discussions to better clarify our design choices, experimental findings, and limitations.

We thank the reviewers again for their time and thoughtful feedback. We remain available to address any further questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.