- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose Look Twice (LoT), a training-free KB-VQA framework that uses internal attention to identify relevant visual regions and textual evidence. The framework extracts object-to-visual attention to locate relevant image regions, filters the attention sink, and selects informative sentences from the retrieved documents. It then highlights the selected evidence using prompt tokens while removing irrelevant content before generating the final answer. The authors conducted experiments on 4 KB-VQA benchmarks and 10 MLLMs, demonstrating consistent performance improvements.
Critical strengths
- The proposed LoT framework leverages the internal attention mechanisms of pre-trained MLLMs to jointly select visual and textual evidence, without requiring additional training or architectural modifications. This provides a practical method for improving evidence utilization in KB-VQA and can be integrated into various architectures as a plug-in.
- The authors present extensive experiments across various MLLMs and benchmarks. Ablation studies evaluate the contributions of visual and textual highlighting, evidence localization, and different selection strategies; the experiments are thorough and comprehensive, providing strong empirical support for the framework’s effectiveness.
Critical weaknesses
- The novelty of the proposed method has not been fully elucidated. In the "Related Work" section, the authors mention several related inference-level attention analysis methods, such as Self-Elicit, FOCUS, and visual attention pooling methods. However, LoT shares several core mechanisms with these methods, including attention-based evidence selection, visual cropping, and evidence highlighting. The primary distinction appears to be the integration of visual and textual evidence refinement, but the authors do not explicitly explain the technical novelty of this combination. Furthermore, the experiments lack direct comparisons with these closely related methods, making it difficult to assess the actual advantages of LoT over existing approaches.
- Compared to text refinement, the effectiveness of visual evidence selection is relatively limited. As shown in Table 3, most performance improvements come from text evidence highlighting, while visual evidence selection provides only modest additional gains across settings, as shown in the relevant Qwen2.5-VL-7B tests. Whether the benefits of visual selection justify the added complexity and computational overhead remains to be thoroughly investigated.
Questions for authors
- Could you clarify the key technical differences between LoT and closely related methods such as SelfElicit and FOCUS? What are the main innovations beyond combining existing visual and textual evidence selection mechanisms?
- Could you provide direct comparisons with these related attention-based methods under the same models, datasets, and inference budgets?
- According to Table 3, visual evidence selection provides relatively limited additional improvements over textual refinement. Could you explain why this happens and identify the scenarios where visual highlighting is particularly beneficial?
3 comments
[Q2] Direct comparison with related attention-based methods
Following the reviewer’s suggestion, we evaluated the closest methods individually and in combination using the same backbones, retrieved inputs , final answer-prompt template, and answer-generation limit. SelfElicit-style applies SelfElicit’s sentence scoring and highlighting without sentence removal. For FOCUS and ViCrop, we use their official implementations adapted to our KB-VQA setting.
Qwen2.5-VL-3B
| Method | E-VQA Single | E-VQA All | InfoSeek U-Q | InfoSeek U-E | InfoSeek All |
|---|---|---|---|---|---|
| Qwen2.5-VL-3B (no refinement) | 30.3 | 28.0 | 22.6 | 22.2 | 22.4 |
| Text refinement | |||||
| Self-Elicit-style | 31.7 | 29.1 | 24.5 | 24.0 | 24.2 |
| LoT (Text) | 31.7 | 29.1 | 29.8 | 29.2 | 29.5 |
| Visual refinement | |||||
| FOCUS | 31.6 | 29.4 | 23.9 | 23.2 | 23.6 |
| ViCrop | 31.4 | 29.3 | 24.0 | 23.3 | 23.7 |
| LoT (Visual) | 31.6 | 29.4 | 24.1 | 23.7 | 23.9 |
| Multimodal refinement | |||||
| FOCUS + Self-Elicit-style | 32.5 | 29.9 | 25.9 | 24.9 | 25.4 |
| ViCrop + Self-Elicit-style | 32.2 | 29.9 | 26.1 | 25.0 | 25.5 |
| FOCUS + LoT (Text) | 32.6 | 30.1 | 29.7 | 28.4 | 29.1 |
| ViCrop + LoT (Text) | 32.2 | 29.7 | 30.0 | 28.7 | 29.3 |
| LoT (Full) | 32.6 | 30.2 | 30.3 | 29.8 | 30.1 |
Qwen2.5-VL-7B
| Method | E-VQA Single | E-VQA All | InfoSeek U-Q | InfoSeek U-E | InfoSeek All |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B (no refinement) | 32.1 | 30.2 | 23.9 | 25.1 | 24.5 |
| Text refinement | |||||
| Self-Elicit-style | 33.6 | 31.4 | 24.9 | 25.4 | 25.1 |
| LoT (Text) | 34.7 | 32.1 | 28.8 | 29.9 | 29.4 |
| Visual refinement | |||||
| FOCUS | 31.6 | 29.6 | 22.3 | 22.8 | 22.6 |
| ViCrop | 31.9 | 29.7 | 22.9 | 23.6 | 23.2 |
| LoT (Visual) | 32.4 | 30.2 | 24.4 | 25.4 | 24.9 |
| Multimodal refinement | |||||
| FOCUS + Self-Elicit-style | 33.2 | 30.9 | 23.9 | 23.6 | 23.7 |
| ViCrop + Self-Elicit-style | 33.4 | 31.1 | 24.7 | 24.7 | 24.7 |
| FOCUS + LoT (Text) | 34.1 | 31.4 | 28.3 | 29.4 | 28.8 |
| ViCrop + LoT (Text) | 34.0 | 31.4 | 29.0 | 30.1 | 29.5 |
| LoT (Full) | 35.0 | 32.2 | 29.3 | 30.9 | 30.1 |
Both combinations of prior methods score below full LoT across the four backbone–dataset configurations, considering E-VQA All and InfoSeek All. LoT exceeds FOCUS + SelfElicit-style by +0.3/+1.3 points on E-VQA and +4.7/+6.4 on InfoSeek for 3B/7B, respectively. The corresponding gains over ViCrop + SelfElicit-style are +0.3/+1.1 and +4.6/+5.4 points. Averaged across these configurations, the improvements are +3.2 and +2.9 points, respectively. Thus, the tested combinations do not fully recover LoT’s performance.
The component comparisons further clarify these differences:
- Textual. LoT text-only improves over Self-Elicit-style by +2.6 points on average, with the largest gains on InfoSeek (+5.3/+4.3), while the two are close on E-VQA (0.0/+0.7). This is consistent with the role of the removal tier: it helps most when the retrieved context contains many distracting passages.
- Visual. Both FOCUS and ViCrop fall below the vanilla baseline for the 7B model on both datasets (E-VQA: 29.6 and 29.7 vs. 30.2; InfoSeek: 22.6 and 23.2 vs. 24.5). On E-VQA and InfoSeek, LoT visual-only never falls below the baseline, and it matches or exceeds both methods in all four settings (+0.8 on average over FOCUS, +0.6 over ViCrop).
- Composition. With LoT's textual refinement held fixed, the LoT visual selector is the best complement, outperforming both FOCUS (+0.1/+0.8 on E-VQA, +1.0/+1.3 on InfoSeek) and ViCrop (+0.5/+0.8 on E-VQA, +0.8/+0.6 on InfoSeek) in all four settings.
Paired uncertainty for the closest comparisons. To assess whether the smaller differences are conclusive, we report paired-bootstrap 95% confidence intervals for LoT (Full) minus each baseline. Differences are expressed in percentage points using the E-VQA All and official InfoSeek All metrics.
| Comparison | Model | E-VQA Δ [95% CI] | InfoSeek Δ [95% CI] |
|---|---|---|---|
| LoT − (FOCUS + LoT Text) | 3B | +0.1 [-0.40, +0.61] | +1.0 [+0.22, +1.78] |
| LoT − (FOCUS + LoT Text) | 7B | +0.8 [+0.37, +1.23] | +1.3 [+0.93, +1.67] |
| LoT − (ViCrop + LoT Text) | 3B | +0.5 [+0.12, +0.88] | +0.8 [+0.21, +1.39] |
| LoT − (ViCrop + LoT Text) | 7B | +0.8 [+0.27, +1.33] | +0.6 [+0.15, +1.05] |
| LoT − (FOCUS + Self-Elicit-style) | 3B | +0.3 [-0.08, +0.68] | +4.7 [+4.29, +5.11] |
| LoT − (FOCUS + Self-Elicit-style) | 7B | +1.3 [+0.96, +1.64] | +6.4 [+6.05, +6.75] |
| LoT − (ViCrop + Self-Elicit-style) | 3B | +0.3 [-0.12, +0.72] | +4.6 [+4.1, +5.1] |
| LoT − (ViCrop + Self-Elicit-style) | 7B | +1.1 [+0.77, +1.43] | +5.4 [+5.02, +5.78] |
[W2 & Q3] Contribution of visual evidence selection
We agree that textual refinement accounts for the larger share of the improvement in Table 3. In KB-VQA, the image supports entity identification, while the requested facts often come from retrieved passages. The full image has also already been used upstream for retrieval. Visual refinement therefore addresses remaining localization and disambiguation difficulties whose importance varies across examples.
Incremental contribution. We distinguish visual refinement in isolation from its contribution within the complete framework:
Visual-only refinement vs. vanilla
| Dataset | Model | Vanilla | Visual-only | Δ |
|---|---|---|---|---|
| E-VQA | 3B | 28.0 | 29.4 | +1.4 |
| E-VQA | 7B | 30.2 | 30.2 | 0.0 |
| InfoSeek | 3B | 22.4 | 23.9 | +1.5 |
| InfoSeek | 7B | 24.5 | 24.9 | +0.4 |
Full refinement vs. text-only refinement
| Dataset | Model | Text-only | Full | Δ |
|---|---|---|---|---|
| E-VQA | 3B | 29.1 | 30.2 | +1.1 |
| E-VQA | 7B | 32.1 | 32.2 | +0.1 |
| InfoSeek | 3B | 29.5 | 30.1 | +0.6 |
| InfoSeek | 7B | 29.4 | 30.1 | +0.7 |
Across all four benchmarks in Table 3, adding visual refinement to text-only LoT yields positive accuracy differences in all eight model–dataset configurations. Paired pointwise 95% bootstrap intervals exclude zero in the positive direction in six of eight configurations; the small gains on E-VQA 7B and ViQuAE 3B remain inconclusive. The full intervals and protocol are reported in our response to Reviewer 5Sxc (W2/Q3).
Visual-only refinement improves six configurations, matches one, and reduces accuracy on ViQuAE 7B (36.4 → 34.3). We explicitly acknowledge this regression. On the same configuration, adding visual refinement to text-only LoT improves accuracy from 49.6 to 50.5. Thus, its effect depends on the accompanying textual evidence, and cropping is not uniformly beneficial.
Settings where visual refinement is more impactful. Within KB-VQA, the largest visual-only gain is on OVEN with 3B (+5.0 points). Table 7 provides complementary evidence from benchmarks without retrieval: visual refinement improves all five evaluated backbones by +1.0 to +4.2 points on average, with gains of up to +8.3 on V*. Those experiments supplement the original image with the crop rather than replacing it, supporting the usefulness of the selected regions under a different presentation strategy. These results identify settings where visual refinement is effective, although they do not establish which individual examples will benefit.
Please also see our response to Reviewer 5Sxc for the paired bootstrap intervals on the visual gains and the analysis of cases where cropping reduces accuracy (W2/Q3), the marker and sink-filtering ablations (W3/Q2), and the end-to-end latency analysis (Q4); and our response to Reviewer qoBm for a budget-matched joint baseline with BM25 and a center crop (W2/Q2).
We thank the reviewer for the positive and constructive assessment, and for recognizing that LoT provides a practical, plug-in way to improve evidence utilization in KB-VQA without additional training or architectural changes, and that our experiments and ablations are thorough and comprehensive, providing strong empirical support for the framework. We address each concern below; new results will be included in the revised paper.
[W1 & Q1] Technical novelty with respect to SelfElicit, FOCUS and visual attention methods
LoT builds on insights from prior work, but its contribution is a general, training-free framework designed around the specific evidence-utilization challenges of KB-VQA, where the model must identify the relevant visual cues while reasoning over retrieved passages that contain a large amount of distracting or irrelevant content.
LoT addresses these challenges with a unified, question-conditioned evidence-refinement pipeline, in which a single analysis pass produces both the visual and the textual selection: