- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies entity-grounded knowledge-based visual question answering, where a system must both identify the visual entity depicted in an image and retrieve facts about it. The authors identify a failure mode called the grounding trap: retrieved facts about a visually similar but wrong entity can support an internally consistent yet incorrect answer, so factual support alone does not guarantee that the recognized entity matches the image. The paper proposes DiVE, a test-time scaling framework that combines direct visual exploration with cross-trajectory verification (CTV). Independent rollouts explore alternative entity hypotheses through visual grounding tools (search, crop, compare) and entity-scoped direct interaction with candidate entity pages. CTV then organizes completed trajectories into an entity-evidence forest, grouping trajectories by their assigned final entity and sharing duplicate observations, and a frozen large language model ranks the branches to select an existing answer without rewriting it. The contributions are threefold: identifying the grounding-trap failure mode and motivating explicit comparison of alternative hypotheses; proposing the DiVE framework coupling direct visual exploration with CTV; and demonstrating improvements over retrieval-augmented, agentic search, and test-time-scaling baselines on E-VQA and InfoSeek, with better verifier-stage token efficiency on a shared rollout pool. Validation includes experiments across two benchmarks and four backbones, controlled ablations isolating the direct tools, SFT, and CTV components, mechanism analyses of exploration coverage and verifier behavior, and generalization experiments on three additional benchmarks.
Critical strengths
The paper advances test-time scaling from answer/process/sub-question selection to entity–evidence-provenance-aware selection, with a crisp, reproducible mechanism: an explicit entity-hypothesis dimension, an Entity-Evidence Forest grouping trajectories by terminal entity with shared-node deduplication, and cross-entity evidence comparison selecting an existing answer. It targets the grounding trap that voting, process-reward, and decomposition selectors cannot perceive.
Critical weaknesses
- The motivating failure mode is not established as prevalent. The paper's central premise, that the grounding trap is a frequent failure warranting dedicated machinery, rests on a single exploratory audit by GPT-5.6 Sol (556 of 2,251 errors, 24.70%) reported as the authors' own descriptive tally, with no independent validation, no inter-annotator agreement, and no direct link between this measured prevalence and the coverage headroom the framework is said to exploit. If the trap is in fact rare or confined to a narrow class of questions, the problem framing, the cross-trajectory machinery, and the claimed gains lose much of their standing. This is consequential because it sets the significance and novelty of the entire contribution on a weakly evidenced premise.
2.Statistical support is likewise uneven: paired bootstrap and McNemar significance are reported for E-VQA, but the InfoSeek main table carries no significance testing, and the ablations report essentially single-sample rollout results without quantifying sampling variance — leaving the claim that "every component contributes" (gaps of about 4 ~14 points) resting on comparisons whose noise is not characterized. This matters because the framework's value depends on reliable selection, and the evidence for that reliability is currently partial.
Questions for authors
- Trap audit criteria and reliability. What is the exact audit prompt and the operationalized definition of an explicit wrong-entity commitment? Was there any human inter-annotator agreement, and was the audit validated on held-out data, other backbones, or InfoSeek and generalization sets to support the 24.70 percent prevalence?
- Attribution of the headroom. Of the 556 audit-flagged trap-type errors, how many does DiVE-CTV actually repair? What fraction of the Pass@1 (46.2 percent) versus Oracle@8 (72.61 percent) gap is attributable to trap-type failures rather than retrieval, reasoning, or knowledge-missing causes?
- Verifier choice and sensitivity. Why is Qwen3-VL-8B the sole verifier? Have you evaluated other model sizes or sources, and how stable is CTV's relative ranking under different verifiers and forest serialization formats (branch order, evidence-node inclusion)?
- Statistical completeness. Can you provide paired bootstrap and McNemar significance (with Holm correction) and multi-seed variance for the InfoSeek main table and the Table 2 ablations, to confirm that the 3.9 to 14.6 point gaps exceed rollout sampling noise?
3 comments
W2&Q4: Statistical significance and sampling variance
We have added paired tests for the full InfoSeek main-table evaluation, report completed results from three independent generation pools, and align the key ablation comparisons with the original E-VQA Table 2. Bootstrapping fixed outputs measures query-sampling uncertainty, while repeated experiments with newly generated pools measure rollout-sampling variability; we report them separately.
Paired tests for the InfoSeek main table. Using saved outputs for the original 71,335 queries, we perform 10,000 paired bootstrap resamples stratified by Unseen-Entity and Unseen-Question, and conduct exact McNemar tests separately within each split. Within each split, Holm correction covers the 12 prespecified comparisons between four generators and three TTS baselines. In the table below, Overall is the harmonic mean of the two split scores; intervals are individual 95% intervals for Overall differences, and p-values correspond to the respective splits:
| Generator | Baseline | Δ Overall (pp) | 95% CI (pp) | Holm p: Un-E | Holm p: Un-Q |
|---|---|---|---|---|---|
| Qwen3-VL-8B | Self-consistency | +1.61 | [1.34, 1.89] | 7.37e−33 | 2.15e−10 |
| Qwen3-VL-8B | VisualPRM-BoN | +4.51 | [4.20, 4.80] | 1.73e−220 | 1.23e−50 |
| Qwen3-VL-8B | FineVerify | +2.34 | [2.05, 2.63] | 1.15e−55 | 1.21e−17 |
| Qwen3-VL-2B | Self-consistency | +2.40 | [2.12, 2.68] | 9.06e−70 | 2.85e−17 |
| Qwen3-VL-2B | VisualPRM-BoN | +4.45 | [4.14, 4.76] | 3.07e−172 | 1.63e−53 |
| Qwen3-VL-2B | FineVerify | +3.21 | [2.90, 3.51] | 1.27e−80 | 1.70e−32 |
| InternVL3.5-8B | Self-consistency | +1.42 | [1.13, 1.72] | 1.20e−23 | 9.15e−7 |
| InternVL3.5-8B | VisualPRM-BoN | +3.84 | [3.52, 4.16] | 4.07e−124 | 2.14e−38 |
| InternVL3.5-8B | FineVerify | +1.69 | [1.37, 2.02] | 7.77e−21 | 5.33e−10 |
| Qwen3.5-9B | Self-consistency | +1.88 | [1.62, 2.15] | 1.25e−46 | 6.26e−13 |
| Qwen3.5-9B | VisualPRM-BoN | +4.70 | [4.40, 5.00] | 6.41e−219 | 3.85e−60 |
| Qwen3.5-9B | FineVerify | +2.22 | [1.92, 2.50] | 1.09e−58 | 6.26e−13 |
All 12 Overall difference intervals lie above zero, and the Holm-adjusted p-values are below 0.05 in both splits.
Sampling variance across repeated generation runs. To quantify rollout-sampling variability, we repeat generation three times on the same 1,000 queries per dataset, producing a fresh eight-rollout pool each time and evaluating full CTV on each pool. Pass@1 always uses the first trajectory in each pool. The table below reports the completed comparisons between full DiVE and the configuration without direct corpus interaction, both retaining SFT; “±” denotes the sample standard deviation across the three pools. E-VQA uses normalized BEM, while InfoSeek uses the per-query mean of Modified RA (micro) in this table:
| Dataset | Generation setting | Pass@1: mean ± SD (%) | CTV: mean ± SD (%) | Paired gain: mean ± SD (pp) |
|---|---|---|---|---|
| E-VQA | Full DiVE | 46.83 ± 0.06 | 54.93 ± 0.35 | +8.10 ± 0.30 |
| E-VQA | Without direct corpus interaction | 39.13 ± 0.15 | 43.83 ± 0.71 | +4.70 ± 0.70 |
| InfoSeek | Full DiVE | 47.93 ± 0.84 | 53.47 ± 1.27 | +5.53 ± 2.11 |
| InfoSeek | Without direct corpus interaction | 44.20 ± 1.15 | 47.93 ± 0.81 | +3.73 ± 0.67 |
All per-pool gains above are relative to Pass@1 from the first trajectory in that pool. Across the three pools, the gains are +8.1/+8.4/+7.8 percentage points on E-VQA and +3.1/+6.7/+6.8 points on InfoSeek. These results support gains over a single trajectory in all three pools.
Repeated-run statistics for the original E-VQA Table 2. The table below uses three independent generation runs on the same fixed 1,000 queries, with configurations corresponding to the original table; completed entries are drawn from the table above.
| Direct Tool | SFT | CTV | Accuracy across three runs: mean ± SD (%) |
|---|---|---|---|
| 0 | 0 | 0 | 36.33 ± 0.81 |
| 1 | 0 | 0 | 41.00 ± 1.01 |
| 0 | 1 | 0 | 39.13 ± 0.15 |
| 1 | 1 | 0 | 46.83 ± 0.06 |
| 0 | 0 | 1 | 38.97 ± 0.90 |
| 0 | 1 | 1 | 43.83 ± 0.71 |
| 1 | 0 | 1 | 49.93 ± 1.22 |
| 1 | 1 | flat | 51.97 ± 1.00 |
| 1 | 1 | 1 | 54.93 ± 0.35 |
| Group | Mean gain ± SD (pp) | Holm-adjusted p-values for the three runs |
|---|---|---|
| Without CTV | +8.10 ± 0.30 | 2.11e−7 / 1.78e−9 / 6.33e−8 |
| Without Tool | +11.10 ± 0.92 | 1.04e−9 / 1.20e−13 / 2.58e−11 |
| Without SFT | +5.00 ± 1.49 | 0.0230 / 0.0000155 / 0.00761 |
| CTV-flat | +2.96 ± 1.06 | 0.0383 / 0.0195 / 0.000264 |
Manuscript revisions: We have added the full InfoSeek paired significance tests to App. H.1 (Tables 10–13), and three-run variability and repeated E-VQA component ablations to App. H.2.1 (Table 15).
Q3: Verifier and serialization sensitivity
CTV can use verifiers from non-Qwen families; on fixed generation pools, a verifier from the same family as the generator is not always optimal. We fix each of the four generators' eight-rollout candidate pools and the corresponding serialized inputs, changing only the verifier to Ministral-3-8B, Llama-3.1-8B, InternVL3.5-8B, or Qwen3.6-35B-A3B. The table below reports accuracy and correct-answer counts.
| Generation pool | Queries | Qwen3-VL-8B | Ministral-3-8B | Llama-3.1-8B | InternVL3.5-8B | Qwen3.6-35B-A3B |
|---|---|---|---|---|---|---|
| evqa | 3750 | 54.13% (2030) | 54.21% (2033) | 51.23% (1921) | 52.93% (1985) | 56.16% (2106) |
| infoseek | 3000 | 54.23% (1627) | 53.13% (1594) | 51.00% (1530) | 52.37% (1571) | 56.47% (1694) |
| evqa_qwen2b | 3750 | 50.03% (1876) | 49.41% (1853) | 45.95% (1723) | 48.29% (1811) | 52.16% (1956) |
| infoseek_qwen2b | 3000 | 49.77% (1493) | 48.27% (1448) | 44.93% (1348) | 46.57% (1397) | 51.77% (1553) |
| evqa_internvl8b | 3750 | 51.12% (1917) | 51.47% (1930) | 47.79% (1792) | 49.04% (1839) | 53.71% (2014) |
| infoseek_internvl8b | 3000 | 49.03% (1471) | 47.27% (1418) | 42.63% (1279) | 45.37% (1361) | 52.27% (1568) |
| evqa_qwen35_9b | 3750 | 55.47% (2080) | 56.59% (2122) | 53.15% (1993) | 53.28% (1998) | 56.05% (2102) |
| infoseek_qwen35_9b | 3000 | 52.40% (1572) | 51.37% (1541) | 48.73% (1462) | 50.87% (1526) | 53.43% (1603) |
Ranking relative to the baselines. On the fixed Qwen3-VL-8B E-VQA pool, for example, App. H.1 reports same-pool accuracies of 51.36%, 50.88%, and 48.37% for Self-consistency, FineVerify, and VisualPRM, respectively. With Ministral-3-8B or InternVL3.5-8B as the verifier, CTV reaches 54.21% and 52.93%, respectively, remaining above all three baselines. Llama-3.1-8B reaches 51.23%, close to but slightly below Self-consistency, indicating that relative rankings still depend on verifier choice. This comparison shows that CTV's selection advantage can be retained with the tested non-Qwen verifiers and does not depend solely on the generator and verifier belonging to the same family.
Results with the additional verifier. Qwen3.6-35B-A3B achieves the highest accuracy in seven of the eight generator–dataset settings and exceeds the original Qwen3-VL-8B verifier in all eight. On the Qwen3.5-9B E-VQA generation pool, Ministral-3-8B reaches 56.59%, whereas Qwen3.6-35B-A3B reaches 56.05%. These results show that changing the verifier can further improve answer selection on fixed candidate pools.
App. H.2.3, Table 17, reports the cross-verifier results and their scope. The Llama results serve as a text-only model control.
Verifier size and choice. The main experiments use Qwen3-VL-8B, matching the size of the primary 8B generator, and fix it as the common verifier across all generation pools to control verifier choice. The additional experiments include several approximately 8B model families as well as Qwen3.6-35B-A3B, examining selection performance across model sources and size configurations.
Serialization order and evidence nodes. For each dataset, we fix 1,000 queries and the same candidate pools, using three predefined root, branch, and label orderings. The top candidate list, candidate answers, evidence-node set, and evidence text remain unchanged across the three settings. This experiment only changes forest ordering and does not add or remove evidence nodes. InfoSeek uses Modified RA micro.
| Dataset | Original ordering | Ordering 1 | Ordering 2 | Accuracy range across the three orderings |
|---|---|---|---|---|
| E-VQA | 54.30% | 55.00% | 53.90% | 1.10 pp |
| InfoSeek | 53.00% | 52.90% | 54.50% | 1.60 pp |
These results quantify sensitivity to forest ordering with evidence content fixed: the accuracy ranges are 1.10 and 1.60 percentage points on E-VQA and InfoSeek, respectively.
Evidence-node inclusion. On 1,000 fixed queries per dataset, we compare retaining versus removing EVIDENCE nodes using the Qwen3-VL-8B verifier. Both conditions hide reasoning text to prevent evidence leakage through that text. Removing evidence reduces E-VQA accuracy from 53.20% to 49.40% (−3.80 pp; paired 95% CI [−6.20, −1.50]; Holm-adjusted McNemar p=0.00334). On InfoSeek, accuracy decreases from 54.00% to 52.60% (−1.40 pp; paired 95% CI [−3.30, +0.40]; Holm-adjusted McNemar p=0.23922). Removing evidence significantly reduces accuracy on E-VQA; the smaller decrease on InfoSeek is inconclusive. These conclusions apply only to the evaluated 1,000 queries per dataset under this controlled setting.
Manuscript revisions: We have added the cross-verifier evaluation (Table 17) and serialization-order sensitivity analysis to App. H.2.3, with their settings and scope explicitly stated.
We hope our responses and manuscript revisions have addressed all your concerns. We welcome any further questions or discussion.
Thank you for your careful review and constructive feedback. We address your comments and questions below.
W1&Q1&Q2: Grounding-trap validation and headroom attribution
Q1: Audit definition and reliability. The 24.70% figure is a descriptive tally by the original GPT 5.6-sol auditor on the 2,251 E-VQA errors made by the vanilla retrieval agent. We add cross-model audits of the same error set and human review of consensus-positive cases.
App. B.5 requires all four conditions to hold: the predicted answer substantively differs from the reference answer; the trajectory's explicit entity commitment differs from the gold entity; the record contains verbatim tool evidence from that wrong entity; and this evidence supports the predicted answer for the same queried attribute. Gold entities and reference answers are used only for post-hoc auditing, not inference. Figure 15 presents the audit prompt, and App. J.1 reports the added cross-model audits and human review.
All three LLM auditors jointly identify 434 queries. We draw a random sample of 200 cases from this consensus-positive set for review by one human annotator. The current review summary records wrong-entity identification in all 200 cases, including 17 borderline cases in which the predicted and reference answers partially overlap; setting these aside leaves 183/200 (91.5%).
Q2: Recovery on grounding-trap queries.
To improve the reliability of trap labels in the recovery analysis, we use cross-auditor consensus: among the 556 originally flagged queries, we select the 434 jointly confirmed by all three auditors as the evaluation set. All recovery counts and rates below refer to this consensus subset.
Among the eight-rollout candidate pools for these 434 queries, 194 contain at least one correct answer. All methods use the same candidate pools and answer labels, with results as follows:
CTV improves over the flat verifier and self-consistency by 5.15 and 9.28 percentage points, respectively. This flat control uses the same model, retained evidence, and verification rubric, removing only entity grouping.
Where do the remaining errors occur? For the 333 queries that CTV answers incorrectly, we further distinguish candidate-coverage failures from selection failures:
The first two categories primarily indicate room for improvement in entity exploration and candidate-answer generation, while the last category directly reflects remaining verifier-stage errors. This suggests that, although our method substantially mitigates grounding traps, improving multimodal retrieval tools and addressing semantic ambiguity and knowledge alignment remain promising directions for future work.
headroom between pass@1 and oracle@8 We next quantify the proportions of different error types within the headroom between DiVE Pass@1 and Oracle@8. There are 989 queries with a correct answer among the eight rollouts but an incorrect Pass@1 answer. Their failure types and proportions are reported below:
Manuscript revisions: We have added cross-model audits, human review, consensus-case recovery (Table 23), and the headroom decomposition (Table 24) to App. J.1, included the audit prompt in Figure 15, and clarified the audit limitations in App. B.5.