acceptodds
Under review as a conference paper at ICLR 2027

DiVE: Test-Time Scaling of Direct Visual Exploration via Cross-Trajectory Verification

Abstract

Knowledge-based visual question answering (KB-VQA) requires identifying the entity depicted in an image and retrieving facts about that entity. However, visually similar long-tail entities can confound grounding. We observe a failure pattern, termed a grounding trap: an agent may retrieve genuine facts about a visually similar but incorrect entity, yielding an internally coherent yet incorrect trajectory. To address this failure, we propose DiVE, a test-time scaling framework that enables direct visual exploration. DiVE combines fine-grained evidence access with direct interaction tools. At inference time, independent rollouts explore alternative entity hypotheses. An off-the-shelf verifier compares the recorded candidate entities and supporting evidence across these trajectories to select an existing answer. Compared with RAG, agentic-search, and alternative test-time scaling baselines, DiVE achieves state-of-the-art performance on E-VQA and InfoSeek. On shared rollout pools, CTV achieves higher accuracy with fewer verifier-stage tokens than the TTS baselines, indicating better verifier-stage efficiency. Our code is available anonymously.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Question Answering, Agentic Search, Test-time Scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uv7gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies entity-grounded knowledge-based visual question answering, where a system must both identify the visual entity depicted in an image and retrieve facts about it. The authors identify a failure mode called the grounding trap: retrieved facts about a visually similar but wrong entity can support an internally consistent yet incorrect answer, so factual support alone does not guarantee that the recognized entity matches the image. The paper proposes DiVE, a test-time scaling framework that combines direct visual exploration with cross-trajectory verification (CTV). Independent rollouts explore alternative entity hypotheses through visual grounding tools (search, crop, compare) and entity-scoped direct interaction with candidate entity pages. CTV then organizes completed trajectories into an entity-evidence forest, grouping trajectories by their assigned final entity and sharing duplicate observations, and a frozen large language model ranks the branches to select an existing answer without rewriting it. The contributions are threefold: identifying the grounding-trap failure mode and motivating explicit comparison of alternative hypotheses; proposing the DiVE framework coupling direct visual exploration with CTV; and demonstrating improvements over retrieval-augmented, agentic search, and test-time-scaling baselines on E-VQA and InfoSeek, with better verifier-stage token efficiency on a shared rollout pool. Validation includes experiments across two benchmarks and four backbones, controlled ablations isolating the direct tools, SFT, and CTV components, mechanism analyses of exploration coverage and verifier behavior, and generalization experiments on three additional benchmarks.

Critical strengths

The paper advances test-time scaling from answer/process/sub-question selection to entity–evidence-provenance-aware selection, with a crisp, reproducible mechanism: an explicit entity-hypothesis dimension, an Entity-Evidence Forest grouping trajectories by terminal entity with shared-node deduplication, and cross-entity evidence comparison selecting an existing answer. It targets the grounding trap that voting, process-reward, and decomposition selectors cannot perceive.

Critical weaknesses
  1. The motivating failure mode is not established as prevalent. The paper's central premise, that the grounding trap is a frequent failure warranting dedicated machinery, rests on a single exploratory audit by GPT-5.6 Sol (556 of 2,251 errors, 24.70%) reported as the authors' own descriptive tally, with no independent validation, no inter-annotator agreement, and no direct link between this measured prevalence and the coverage headroom the framework is said to exploit. If the trap is in fact rare or confined to a narrow class of questions, the problem framing, the cross-trajectory machinery, and the claimed gains lose much of their standing. This is consequential because it sets the significance and novelty of the entire contribution on a weakly evidenced premise.

2.Statistical support is likewise uneven: paired bootstrap and McNemar significance are reported for E-VQA, but the InfoSeek main table carries no significance testing, and the ablations report essentially single-sample rollout results without quantifying sampling variance — leaving the claim that "every component contributes" (gaps of about 4 ~14 points) resting on comparisons whose noise is not characterized. This matters because the framework's value depends on reliable selection, and the evidence for that reliability is currently partial.

Questions for authors
  1. Trap audit criteria and reliability. What is the exact audit prompt and the operationalized definition of an explicit wrong-entity commitment? Was there any human inter-annotator agreement, and was the audit validated on held-out data, other backbones, or InfoSeek and generalization sets to support the 24.70 percent prevalence?
  2. Attribution of the headroom. Of the 556 audit-flagged trap-type errors, how many does DiVE-CTV actually repair? What fraction of the Pass@1 (46.2 percent) versus Oracle@8 (72.61 percent) gap is attributable to trap-type failures rather than retrieval, reasoning, or knowledge-missing causes?
  3. Verifier choice and sensitivity. Why is Qwen3-VL-8B the sole verifier? Have you evaluated other model sizes or sources, and how stable is CTV's relative ranking under different verifiers and forest serialization formats (branch order, evidence-node inclusion)?
  4. Statistical completeness. Can you provide paired bootstrap and McNemar significance (with Holm correction) and multi-seed variance for the InfoSeek main table and the Table 2 ablations, to confirm that the 3.9 to 14.6 point gaps exceed rollout sampling noise?
3 comments

Thank you for your careful review and constructive feedback. We address your comments and questions below.

W1&Q1&Q2: Grounding-trap validation and headroom attribution

Q1: Audit definition and reliability. The 24.70% figure is a descriptive tally by the original GPT 5.6-sol auditor on the 2,251 E-VQA errors made by the vanilla retrieval agent. We add cross-model audits of the same error set and human review of consensus-positive cases.

App. B.5 requires all four conditions to hold: the predicted answer substantively differs from the reference answer; the trajectory's explicit entity commitment differs from the gold entity; the record contains verbatim tool evidence from that wrong entity; and this evidence supports the predicted answer for the same queried attribute. Gold entities and reference answers are used only for post-hoc auditing, not inference. Figure 15 presents the audit prompt, and App. J.1 reports the added cross-model audits and human review.

LLM auditorQueries identified as trapsShare of the same 2,251 errors
Original audit55624.70%
Qwen3-VL-235B67730.08%
Llama3-405B61727.41%

All three LLM auditors jointly identify 434 queries. We draw a random sample of 200 cases from this consensus-positive set for review by one human annotator. The current review summary records wrong-entity identification in all 200 cases, including 17 borderline cases in which the predicted and reference answers partially overlap; setting these aside leaves 183/200 (91.5%).

Q2: Recovery on grounding-trap queries.

To improve the reliability of trap labels in the recovery analysis, we use cross-auditor consensus: among the 556 originally flagged queries, we select the 434 jointly confirmed by all three auditors as the evaluation set. All recovery counts and rates below refer to this consensus subset.

Among the eight-rollout candidate pools for these 434 queries, 194 contain at least one correct answer. All methods use the same candidate pools and answer labels, with results as follows:

SelectorCorrect answers on the 434 historical trap queriesAnswer accuracy on the full setSelection accuracy when a correct candidate is available (denominator: 194)
Self-consistency8319.12%42.78%
Flat verifier9120.97%46.91%
Full CTV10123.27%52.06%

CTV improves over the flat verifier and self-consistency by 5.15 and 9.28 percentage points, respectively. This flat control uses the same model, retained evidence, and verification rubric, removing only entity grouping.

Where do the remaining errors occur? For the 333 queries that CTV answers incorrectly, we further distinguish candidate-coverage failures from selection failures:

Observed outcomeQueriesShare of all 434 queries
No gold-entity name detected, and all eight trajectories answer incorrectly15535.71%
Gold-entity name appears in candidates, tool-call targets, or evidence titles, but all eight trajectories answer incorrectly8018.43%
Only body-text mentions or related names appear, without confirming a correct entity hypothesis; all eight trajectories answer incorrectly51.15%
A correct candidate is available and included in the verifier's selectable set, but is not selected9321.43%

The first two categories primarily indicate room for improvement in entity exploration and candidate-answer generation, while the last category directly reflects remaining verifier-stage errors. This suggests that, although our method substantially mitigates grounding traps, improving multimodal retrieval tools and addressing semantic ambiguity and knowledge alignment remain promising directions for future work.

headroom between pass@1 and oracle@8 We next quantify the proportions of different error types within the headroom between DiVE Pass@1 and Oracle@8. There are 989 queries with a correct answer among the eight rollouts but an incorrect Pass@1 answer. Their failure types and proportions are reported below:

CategoryCountShare
Trap: identified under the original protocol25725.99%
Retrieval: non-trap, gold entity not retrieved28128.41%
Reasoning: correct entity retrieved and recognized, but answer incorrect15815.98%
Gold entity retrieved, wrong entity chosen, unsupported answer28228.51%
Gold entity retrieved, wrong entity chosen, answer unfinished111.11%

Manuscript revisions: We have added cross-model audits, human review, consensus-case recovery (Table 23), and the headroom decomposition (Table 24) to App. J.1, included the audit prompt in Figure 15, and clarified the audit limitations in App. B.5.

W2&Q4: Statistical significance and sampling variance

We have added paired tests for the full InfoSeek main-table evaluation, report completed results from three independent generation pools, and align the key ablation comparisons with the original E-VQA Table 2. Bootstrapping fixed outputs measures query-sampling uncertainty, while repeated experiments with newly generated pools measure rollout-sampling variability; we report them separately.

Paired tests for the InfoSeek main table. Using saved outputs for the original 71,335 queries, we perform 10,000 paired bootstrap resamples stratified by Unseen-Entity and Unseen-Question, and conduct exact McNemar tests separately within each split. Within each split, Holm correction covers the 12 prespecified comparisons between four generators and three TTS baselines. In the table below, Overall is the harmonic mean of the two split scores; intervals are individual 95% intervals for Overall differences, and p-values correspond to the respective splits:

GeneratorBaselineΔ Overall (pp)95% CI (pp)Holm p: Un-EHolm p: Un-Q
Qwen3-VL-8BSelf-consistency+1.61[1.34, 1.89]7.37e−332.15e−10
Qwen3-VL-8BVisualPRM-BoN+4.51[4.20, 4.80]1.73e−2201.23e−50
Qwen3-VL-8BFineVerify+2.34[2.05, 2.63]1.15e−551.21e−17
Qwen3-VL-2BSelf-consistency+2.40[2.12, 2.68]9.06e−702.85e−17
Qwen3-VL-2BVisualPRM-BoN+4.45[4.14, 4.76]3.07e−1721.63e−53
Qwen3-VL-2BFineVerify+3.21[2.90, 3.51]1.27e−801.70e−32
InternVL3.5-8BSelf-consistency+1.42[1.13, 1.72]1.20e−239.15e−7
InternVL3.5-8BVisualPRM-BoN+3.84[3.52, 4.16]4.07e−1242.14e−38
InternVL3.5-8BFineVerify+1.69[1.37, 2.02]7.77e−215.33e−10
Qwen3.5-9BSelf-consistency+1.88[1.62, 2.15]1.25e−466.26e−13
Qwen3.5-9BVisualPRM-BoN+4.70[4.40, 5.00]6.41e−2193.85e−60
Qwen3.5-9BFineVerify+2.22[1.92, 2.50]1.09e−586.26e−13

All 12 Overall difference intervals lie above zero, and the Holm-adjusted p-values are below 0.05 in both splits.

Sampling variance across repeated generation runs. To quantify rollout-sampling variability, we repeat generation three times on the same 1,000 queries per dataset, producing a fresh eight-rollout pool each time and evaluating full CTV on each pool. Pass@1 always uses the first trajectory in each pool. The table below reports the completed comparisons between full DiVE and the configuration without direct corpus interaction, both retaining SFT; “±” denotes the sample standard deviation across the three pools. E-VQA uses normalized BEM, while InfoSeek uses the per-query mean of Modified RA (micro) in this table:

DatasetGeneration settingPass@1: mean ± SD (%)CTV: mean ± SD (%)Paired gain: mean ± SD (pp)
E-VQAFull DiVE46.83 ± 0.0654.93 ± 0.35+8.10 ± 0.30
E-VQAWithout direct corpus interaction39.13 ± 0.1543.83 ± 0.71+4.70 ± 0.70
InfoSeekFull DiVE47.93 ± 0.8453.47 ± 1.27+5.53 ± 2.11
InfoSeekWithout direct corpus interaction44.20 ± 1.1547.93 ± 0.81+3.73 ± 0.67

All per-pool gains above are relative to Pass@1 from the first trajectory in that pool. Across the three pools, the gains are +8.1/+8.4/+7.8 percentage points on E-VQA and +3.1/+6.7/+6.8 points on InfoSeek. These results support gains over a single trajectory in all three pools.

Repeated-run statistics for the original E-VQA Table 2. The table below uses three independent generation runs on the same fixed 1,000 queries, with configurations corresponding to the original table; completed entries are drawn from the table above.

Direct ToolSFTCTVAccuracy across three runs: mean ± SD (%)
00036.33 ± 0.81
10041.00 ± 1.01
01039.13 ± 0.15
11046.83 ± 0.06
00138.97 ± 0.90
01143.83 ± 0.71
10149.93 ± 1.22
11flat51.97 ± 1.00
11154.93 ± 0.35
GroupMean gain ± SD (pp)Holm-adjusted p-values for the three runs
Without CTV+8.10 ± 0.302.11e−7 / 1.78e−9 / 6.33e−8
Without Tool+11.10 ± 0.921.04e−9 / 1.20e−13 / 2.58e−11
Without SFT+5.00 ± 1.490.0230 / 0.0000155 / 0.00761
CTV-flat+2.96 ± 1.060.0383 / 0.0195 / 0.000264

Manuscript revisions: We have added the full InfoSeek paired significance tests to App. H.1 (Tables 10–13), and three-run variability and repeated E-VQA component ablations to App. H.2.1 (Table 15).

Q3: Verifier and serialization sensitivity

CTV can use verifiers from non-Qwen families; on fixed generation pools, a verifier from the same family as the generator is not always optimal. We fix each of the four generators' eight-rollout candidate pools and the corresponding serialized inputs, changing only the verifier to Ministral-3-8B, Llama-3.1-8B, InternVL3.5-8B, or Qwen3.6-35B-A3B. The table below reports accuracy and correct-answer counts.

Generation poolQueriesQwen3-VL-8BMinistral-3-8BLlama-3.1-8BInternVL3.5-8BQwen3.6-35B-A3B
evqa375054.13% (2030)54.21% (2033)51.23% (1921)52.93% (1985)56.16% (2106)
infoseek300054.23% (1627)53.13% (1594)51.00% (1530)52.37% (1571)56.47% (1694)
evqa_qwen2b375050.03% (1876)49.41% (1853)45.95% (1723)48.29% (1811)52.16% (1956)
infoseek_qwen2b300049.77% (1493)48.27% (1448)44.93% (1348)46.57% (1397)51.77% (1553)
evqa_internvl8b375051.12% (1917)51.47% (1930)47.79% (1792)49.04% (1839)53.71% (2014)
infoseek_internvl8b300049.03% (1471)47.27% (1418)42.63% (1279)45.37% (1361)52.27% (1568)
evqa_qwen35_9b375055.47% (2080)56.59% (2122)53.15% (1993)53.28% (1998)56.05% (2102)
infoseek_qwen35_9b300052.40% (1572)51.37% (1541)48.73% (1462)50.87% (1526)53.43% (1603)

Ranking relative to the baselines. On the fixed Qwen3-VL-8B E-VQA pool, for example, App. H.1 reports same-pool accuracies of 51.36%, 50.88%, and 48.37% for Self-consistency, FineVerify, and VisualPRM, respectively. With Ministral-3-8B or InternVL3.5-8B as the verifier, CTV reaches 54.21% and 52.93%, respectively, remaining above all three baselines. Llama-3.1-8B reaches 51.23%, close to but slightly below Self-consistency, indicating that relative rankings still depend on verifier choice. This comparison shows that CTV's selection advantage can be retained with the tested non-Qwen verifiers and does not depend solely on the generator and verifier belonging to the same family.

Results with the additional verifier. Qwen3.6-35B-A3B achieves the highest accuracy in seven of the eight generator–dataset settings and exceeds the original Qwen3-VL-8B verifier in all eight. On the Qwen3.5-9B E-VQA generation pool, Ministral-3-8B reaches 56.59%, whereas Qwen3.6-35B-A3B reaches 56.05%. These results show that changing the verifier can further improve answer selection on fixed candidate pools.

App. H.2.3, Table 17, reports the cross-verifier results and their scope. The Llama results serve as a text-only model control.

Verifier size and choice. The main experiments use Qwen3-VL-8B, matching the size of the primary 8B generator, and fix it as the common verifier across all generation pools to control verifier choice. The additional experiments include several approximately 8B model families as well as Qwen3.6-35B-A3B, examining selection performance across model sources and size configurations.

Serialization order and evidence nodes. For each dataset, we fix 1,000 queries and the same candidate pools, using three predefined root, branch, and label orderings. The top candidate list, candidate answers, evidence-node set, and evidence text remain unchanged across the three settings. This experiment only changes forest ordering and does not add or remove evidence nodes. InfoSeek uses Modified RA micro.

DatasetOriginal orderingOrdering 1Ordering 2Accuracy range across the three orderings
E-VQA54.30%55.00%53.90%1.10 pp
InfoSeek53.00%52.90%54.50%1.60 pp

These results quantify sensitivity to forest ordering with evidence content fixed: the accuracy ranges are 1.10 and 1.60 percentage points on E-VQA and InfoSeek, respectively.

Evidence-node inclusion. On 1,000 fixed queries per dataset, we compare retaining versus removing EVIDENCE nodes using the Qwen3-VL-8B verifier. Both conditions hide reasoning text to prevent evidence leakage through that text. Removing evidence reduces E-VQA accuracy from 53.20% to 49.40% (−3.80 pp; paired 95% CI [−6.20, −1.50]; Holm-adjusted McNemar p=0.00334). On InfoSeek, accuracy decreases from 54.00% to 52.60% (−1.40 pp; paired 95% CI [−3.30, +0.40]; Holm-adjusted McNemar p=0.23922). Removing evidence significantly reduces accuracy on E-VQA; the smaller decrease on InfoSeek is inconclusive. These conclusions apply only to the evaluated 1,000 queries per dataset under this controlled setting.

Manuscript revisions: We have added the cross-verifier evaluation (Table 17) and serialization-order sensitivity analysis to App. H.2.3, with their settings and scope explicitly stated.

We hope our responses and manuscript revisions have addressed all your concerns. We welcome any further questions or discussion.

Reviewer_NiA9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Entity-centric KB-VQA requires identifying which depicted entity an image shows before retrieving facts about it. The paper argues that the dominant failure is not factual error but a "grounding trap" (Sec. 3, Fig. 2): an agent commits to a visually similar entity ẽ ≠ e*, retrieves genuine facts about ẽ, and returns an internally coherent but wrong answer — formally ã = f_θ(x; K_ẽ) ≠ a* (Eq. 5). Support for this claim is an exploratory post-hoc audit (App. B.5, Table 3) of a vanilla EVA-CLIP+BGE search agent on the 3,750-question E-VQA Single test set: of 2,251 BEM errors, 556 (24.70%) show explicit wrong-entity commitment plus verbatim same-entity tool evidence; 1,932 (85.83%) involve a wrong entity overall.

DiVE has two components. First, a search agent with a direct-interaction toolset: image_search_tool (three EVA-CLIP indexes over titles/images/summaries, min–max normalized and fused), image_source_tool (reference-image inspection), image_crop_tool (0–1000-grid regions), and text_search_tool restricted to ≤5 named entity pages with regex windows (≤3 per page, 220 chars/side default, 4,096-char CLI cap) plus a BGE fallback (Table 4, App. C.2); ≤15 interaction steps per rollout. The agent is SFT-distilled (Eq. 6) from Qwen3-VL-235B-A22B-Instruct on a "Balanced-2K" set of 2,000 rejection-sampled, answer-correct, structurally valid trajectories (600 stable + 400 challenging per benchmark, λ = 0.6, Eq. 19–20), updating only LM parameters. Second, Cross-Trajectory Verification (CTV, Sec. 4.2): each trajectory is assigned a terminal entity Ê_i by Eq. 7 (entities named in the frozen answer, else last text-search entity, else last visual entity, else ⊥; 99.07% of 30,000 E-VQA trajectories get a root, Table 15), trajectories are grouped under shared roots with navigation/grounding/evidence node dedup (Eq. 8, Alg. 1), and the serialized forest is ranked once by an off-the-shelf Qwen3-VL-8B verifier that returns a preference order; the first valid leaf's answer is copied unchanged.

Experiments show DiVE-CTV beating the three TTS selectors on every backbone on shared frozen 8-rollout pools: E-VQA BEM 50.0/51.1/54.1/55.5 vs self-consistency 47.5/48.5/51.4/51.1 and FineVerify 45.5/49.5/50.9/49.6 (Table 1); "All" 53.6 for Qwen3-VL-8B vs 51.5 self-consistency. The ablation grid (Table 2) runs 34.9 (all off) → 54.1 (full), with Direct Tool + CTV at 49.5 and a flat listwise verifier at 50.2, isolating the forest at +3.9. Paired bootstrap and exact McNemar tests (Tables 6–9) confirm all 12 comparisons significant after Holm correction (Qwen3-VL-8B: +2.77 vs self-consistency, CI [1.60, 3.95]). Diagnostics localize the gain (Table 11: Entity Match 80.4 vs 77.9, Within-Entity Answer 92.2 vs 90.3). CTV is also the cheapest verifier (10,940 input tok/q vs VisualPRM's 33,353).

Critical strengths
  • Experimental discipline. Every TTS selector consumes the same frozen 8-rollout pool per backbone (App. F.4), and the paper reports paired bootstrap CIs, exact McNemar tests, and Holm correction over all 12 comparisons (App. H.1) — rare and welcome in this area. Counts (e.g., CTV 2,030/3,750 = 54.13%) are given, not just percentages.
  • Candor about what is not shown. App. F.5 reports the adapted TTSP run at 13.63% vs CTV's 53.81% and states it "matches only trajectory count—not pools, tokens, models, or tools." App. H.3 states the token metric "neither holds generation cost fixed across different N nor equates token counts with FLOPs or latency," and that tool-execution cost is excluded. Sec. 5.4 notes CTV gains "do not fully keep pace" with Oracle@N. This makes the paper far more trustworthy than typical.
  • Complete and informative ablations. All eight Direct Tool × SFT × CTV combinations (Table 2), fine-grained tool ablations showing mean rollout BEM 46.58 → 44.55 (no crop/source) → 41.05 and Oracle@8 72.61 → 71.92 → 64.56 without direct entity-page access (Table 10), and difficulty-stratified selection rates (Fig. 4: on the 373 questions with only 1/8 correct rollouts, CTV 25.2% vs FineVerify 22.8%, self-consistency 15.0%).
  • A mechanistic account, not just a leaderboard. Eq. 14–17 decompose accuracy into coverage 1−(1−p_B)^N and conditional selection α_N, and Table 11 reports entity-level diagnostics showing CTV improves both axes (End-to-End Answer 74.1 vs 70.4 for self-consistency, Qwen3-VL-8B).
  • Genuine transfer evidence. With frozen policy and verifier, CTV still helps on three unseen benchmarks: ViQuAE 53.38→59.82 EM, M3-VQA 25.74→30.67, MC-SEARCH 38.76→41.21 (Tables 12–14), with format adaptations disclosed rather than hidden.
Critical weaknesses
  • The headline is compute-confounded. Every DiVE-CTV number uses N=8 rollouts (Fig. 7 goes to 32), while RAG and agentic baselines in Table 1 are single-trajectory. Under matched single-rollout budget, the SFT'd DiVE agent scores 46.2 on E-VQA (Table 11) — below the authors' own ProMSA reproduction at 49.0. App. B.4 concedes "the fixed-B analysis does not compare alternative allocations of the same total compute." No table reports total tokens/question or wall-clock/question including generation.
  • Modest, baseline-relative novelty. CTV's forest is a re-serialization of trajectories plus one LLM ranking call; the components (EVA-CLIP retrieval, cropping, direct corpus interaction per Li et al. 2026 / DR-DCI / A-RAG) are prior art. The only isolating evidence is Table 2's +3.9 over a flat listwise verifier, which is not run at matched verifier tokens — so structure and context compression are confounded.
  • Self-preference risk is untested. The CTV verifier is Qwen3-VL-8B for all four backbones, and is the generator itself for two of them. No verifier-family swap is reported, leaving open whether the gain is generic provenance-aware selection or in-family agreement.
  • The motivating audit is single-judge, LLM-only, and on a different agent. The 24.70% grounding-trap rate comes from GPT-5.6 Sol with gold entity/answer access, on a vanilla retrieval agent rather than DiVE itself, with no human or cross-judge validation (App. B.5 admits this). No audit of DiVE-CTV's residual errors is provided, so it is unknown whether the phenomenon persists after the fix — which is the claim that matters.
  • Related-work positioning is asserted, not tested. TTSP, ReflectiVA, and MACT are discussed (Sec. 2, App. G) but never run under matched conditions; the sole TTSP datapoint is the 13.63% number the authors themselves call non-causal. Leaving it in the paper risks being read as a refutation it cannot support.
  • Presentation of diagnostics is mildly favorable. Fig. 6's radar uses fixed, shared, post-hoc ranges (tokens 8k–35k inverted; Entity Match 65–85%) with out-of-range values clipped (App. H.3) — CTV's 80.4 vs 77.9 is inside the band but the "best on all five axes" framing is partly shaped by axis choice. Separately, Sec. 5.4/3 cites "38.90 absolute best result, which uses a 72B solver" for M3-VQA while Table 13's top row is 36.40 for a 78B model; and MC-SEARCH baselines are "reconstructed… weighted" totals (Table 14), not quoted values.
Questions for authors
  • Can you provide a compute-matched table — total tokens/query and wall-clock/query including rollout generation — comparing DiVE-CTV (N=8 and N=32) against ProMSA* at N=1 and against the strongest SFT/RL single-rollout policy on the same EVA-CLIP knowledge base?
  • Does CTV's advantage survive replacing the Qwen3-VL-8B verifier with a different model family (e.g., InternVL-3.5-8B or a text-only LLM over the serialized forest)? In-family agreement is a plausible confound for the +2.5 to +6.1 margins.
  • What fraction of DiVE-CTV's residual E-VQA errors are grounding traps under the App. B.5 protocol, and how sensitive is the 24.70% figure to judge choice (a second, independent auditor or human sample of, say, 200 cases)?
  • In Table 2, is the +3.9 over the flat listwise verifier attributable to entity grouping, to observation dedup (token savings), or to prompt length? A flat listwise baseline given the same deduplicated context and the same token budget would separate these.
  • App. A notes that independent rollouts repeat errors because no memory carries across stages. Given Oracle@N still climbs to N=32 (Fig. 7) while CTV saturates, what is the per-rollout budget B, and does raising B within a fixed total budget (N=2, B=60 vs N=8, B=15) change the accuracy–cost trade-off?
5 comments

Thank you for your careful review and constructive feedback. We address your comments and questions below.

W1&Q1: End-to-end cost comparison

We use the same measurement protocol for single- and multi-trajectory methods, reporting all model tokens for generation and selection and measured end-to-end latency. The measurements below use the same 200 queries per dataset, four H100 GPUs, an allocation of 32 CPU cores, four concurrent queries, at most eight parallel trajectories per query, and B=15.

DatasetMethodMean output tokens/queryMean total tokens/query (input + output)Median / P95 latency (s)
E-VQAProMSA, N=11,017.8414,346.3611.59 / 26.31
E-VQADiVE, N=11,074.8315,383.1012.24 / 30.31
E-VQADiVE + CTV, N=88,577.97131,636.3438.75 / 85.27
E-VQADiVE + CTV, N=1616,486.64254,777.0484.86 / 166.94
E-VQADiVE + CTV, N=3232,652.51509,963.79167.78 / 318.90
InfoSeekProMSA, N=1876.9512,613.8214.09 / 42.25
InfoSeekDiVE, N=1896.3013,983.0313.98 / 60.53
InfoSeekDiVE + CTV, N=87,206.41120,807.2729.20 / 110.59
InfoSeekDiVE + CTV, N=1613,913.89236,311.9067.12 / 203.74
InfoSeekDiVE + CTV, N=3227,067.44469,774.71140.46 / 369.59

These latencies are actual completion times under the stated hardware, concurrency, and cache settings; DiVE and ProMSA have similar median latencies in the single-rollout setting.

Cost summary. At N=1, DiVE and RL-trained ProMSA* have comparable token costs: mean total tokens (input + output) are 15.38k versus 14.35k on E-VQA, and 13.98k versus 12.61k on InfoSeek, respectively. Among TTS methods using the same rollout count and shared candidate pools, CTV reduces verifier-stage token usage relative to SC. Including the common rollout-generation cost therefore also yields a modest reduction in total tokens used for generation and selection. These two comparisons separately characterize the generation cost of a single-rollout agent and the efficiency of TTS selection under a given sampling budget.

Generation and verification breakdown. For N=8, we further separate the two stages. Generation latency is the completion time of the slowest of the eight parallel trajectories; verifier latency is measured for the first LLM request:

DatasetStageMean output tokens/queryMean latency (s/query)
E-VQAEight-rollout generation, including tool summaries8,289.2238.33
E-VQAVerifier first request288.762.57
InfoSeekEight-rollout generation, including tool summaries6,924.0915.44
InfoSeekVerifier first request282.332.52

The component token counts sum exactly to the totals at the original integer precision; displayed values may differ by 0.01 because of rounding. The generation-latency correction retains the original run's shared-service queuing and cache environment. Counting both input and output tokens, verification accounts for 8.79% and 8.31% of total model tokens in the two eight-rollout settings.

Manuscript revisions: We have added end-to-end token and latency measurements and the generation/verification breakdown to App. J.2 (Tables 25–26), while retaining the explicit verifier-stage efficiency scope in App. H.3 and the main text.

W2: Methodological novelty and prior work

We introduce a visual exploration + direct corpus interaction interface for knowledge-based visual question answering. DCI and DR-DCI study direct access to textual corpora, while A-RAG exposes hierarchical text-retrieval interfaces. In DiVE, visual entity exploration determines which entity pages are directly inspected. Compared with retrieval-based KB-VQA tools that return ranked passages, our interface enables exact/regex queries and adjustable context windows within candidate entity pages, interleaved with visual inspection. Table 14 (App. H.2) demonstrates the contribution of this interface: with the same policy checkpoint, replacing direct entity-page access with BGE retrieval reduces mean rollout BEM from 46.58% to 41.05% and Oracle@8 from 72.61% to 64.56%, supporting its role in improving candidate quality and coverage.

Manuscript revisions: We have clarified DiVE's relationship to DCI, DR-DCI, and A-RAG in Section 2 and App. G, and added the grouping-versus-compression controls to App. H.2.2 (Table 16; discussed in Q4 below).

W3&Q2: Verifier choice and self-preference

CTV can use verifiers from non-Qwen families; on fixed generation pools, a verifier from the same family as the generator is not always optimal. We fix each of the four generators' eight-rollout candidate pools and the corresponding serialized inputs, changing only the verifier to Ministral-3-8B, Llama-3.1-8B, InternVL3.5-8B, or Qwen3.6-35B-A3B. The table below reports accuracy and correct-answer counts.

Generation poolQueriesQwen3-VL-8BMinistral-3-8BLlama-3.1-8BInternVL3.5-8BQwen3.6-35B-A3B
evqa375054.13% (2030)54.21% (2033)51.23% (1921)52.93% (1985)56.16% (2106)
infoseek300054.23% (1627)53.13% (1594)51.00% (1530)52.37% (1571)56.47% (1694)
evqa_qwen2b375050.03% (1876)49.41% (1853)45.95% (1723)48.29% (1811)52.16% (1956)
infoseek_qwen2b300049.77% (1493)48.27% (1448)44.93% (1348)46.57% (1397)51.77% (1553)
evqa_internvl8b375051.12% (1917)51.47% (1930)47.79% (1792)49.04% (1839)53.71% (2014)
infoseek_internvl8b300049.03% (1471)47.27% (1418)42.63% (1279)45.37% (1361)52.27% (1568)
evqa_qwen35_9b375055.47% (2080)56.59% (2122)53.15% (1993)53.28% (1998)56.05% (2102)
infoseek_qwen35_9b300052.40% (1572)51.37% (1541)48.73% (1462)50.87% (1526)53.43% (1603)

Ranking relative to the baselines. On the fixed Qwen3-VL-8B E-VQA pool, for example, App. H.1 reports same-pool accuracies of 51.36%, 50.88%, and 48.37% for Self-consistency, FineVerify, and VisualPRM, respectively. With Ministral-3-8B or InternVL3.5-8B as the verifier, CTV reaches 54.21% and 52.93%, respectively, remaining above all three baselines. Llama-3.1-8B reaches 51.23%, close to but slightly below Self-consistency, indicating that relative rankings still depend on verifier choice. This comparison shows that CTV's selection advantage can be retained with the tested non-Qwen verifiers and does not depend solely on the generator and verifier belonging to the same family.

Results with the additional verifier. Qwen3.6-35B-A3B achieves the highest accuracy in seven of the eight generator–dataset settings and exceeds the original Qwen3-VL-8B verifier in all eight. On the Qwen3.5-9B E-VQA generation pool, Ministral-3-8B reaches 56.59%, whereas Qwen3.6-35B-A3B reaches 56.05%. These results show that changing the verifier can further improve answer selection on fixed candidate pools.

Manuscript revisions: We have added the cross-family and larger-verifier results, together with their scope, to App. H.2.3 (Table 17).

W4&Q3: Grounding-trap validation and residual errors

We conduct cross-model experiments on auditor choice and add a human review of sampled cases. We also report DiVE-CTV's residual trap rate and the sensitivity of grounding-trap identification to auditor choice.

Cross-auditor agreement and prevalence. On the same 2,251 errors examined in the original audit, we follow the original protocol in App. B.5 and Figure 15: the final answer is substantively incorrect; the agent explicitly commits to a wrong entity; observed tool evidence belongs to that entity and supports the answer; and, holding the queried attribute fixed, the answer would address the question about that wrong entity. The three audit results are:

AuditIdentified trapsShare of the same 2,251 errors
Original audit55624.70%
Qwen3-VL-235B67730.08%
Llama3-405B61727.41%

Pairwise positive agreement ranges from 78.83% to 84.85%, and all three auditors jointly identify 434 queries. These results support the existence of this failure pattern, with the expanded audits yielding proportions from 24.70% to 30.08%. We draw a fixed random sample of 200 cases for review by one human annotator. The current human-review summary records wrong-entity identification in all 200 cases, including 17 in which the predicted and reference answers partially overlap. Provisionally separating these 17 answer-boundary cases leaves a conservative count of 183/200, or 91.5%.

Selection accuracy when a correct candidate is available. Among the DiVE eight-rollout candidate pools for these 434 queries, 194 contain at least one correct answer. All methods use the same candidate pools and answer labels, with results as follows:

SelectorCorrect answers on the 434 historical trap queriesAnswer accuracy on the full setSelection accuracy when a correct candidate is available (denominator: 194)
DiVE +Self-consistency8319.12%42.78%
DiVE + Flat verifier9120.97%46.91%
DiVE + CTV10123.27%52.06%

CTV improves over the flat verifier and self-consistency by 5.15 and 9.28 percentage points, respectively. This flat control uses the same model, retained evidence, and verification rubric.

Where do the remaining errors occur? For the 333 queries that CTV answers incorrectly, we further distinguish candidate-coverage failures from selection failures:

Observed outcomeQueriesShare of all 434 queries
No gold-entity name detected, and all eight trajectories answer incorrectly15535.71%
Gold-entity name appears in candidates, tool-call targets, or evidence titles, but all eight trajectories answer incorrectly8018.43%
Only body-text mentions or related names appear, without confirming a correct entity hypothesis; all eight trajectories answer incorrectly51.15%
A correct candidate is available and included in the verifier's selectable set, but is not selected9321.43%

The first two categories indicate room for improvement in entity exploration and candidate-answer generation, while the last category reflects remaining verifier-stage errors. Our method mitigates grounding traps, but improving multimodal retrieval tools and addressing semantic ambiguity and knowledge alignment remain promising future directions for the field. In addition, among the 1720 current DiVE-CTV errors, 268 are identified as grounding traps with DiVE Oracle@8=1, accounting for 15.58% of all errors.

Manuscript revisions: We have added cross-model audits, human review, recovery and residual-error analyses, and illustrative cases to App. J.1 (Table 23; Figures 15–16), and updated the audit limitations in App. B.5.

W5: Related-work positioning and baselines

Table 1 now includes ReflectiVA’s published default-configuration results:

MethodE-VQAInfoSeek Unseen-QInfoSeek Unseen-E
ReflectiVA35.540.439.8

Our main comparison already includes recent KB-VQA systems such as OMGM and VLM-PRF. Adding ReflectiVA makes the coverage of reflection-based KB-VQA methods more explicit.

TTSP targets perceptual exploration, and MACT targets visual document understanding; neither was originally evaluated on E-VQA or InfoSeek. Our exploratory TTSP adaptation used two rounds of four trajectories and one Evidence Ledger update (8 trajectories in total per query), yielding 13.63% BEM on E-VQA (Appendix F.5). We identify this as an exploratory transfer result whose unmatched models, tools, and budgets preclude a controlled comparison.

Manuscript revisions: We have added ReflectiVA to Table 1 and clarified the relationship to TTSP and MACT and the scope of the exploratory TTSP result in Section 2 and Apps. F.5 and G.

Reviewer_Sz4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiVE addresses entity-centric knowledge-based visual question answering, where facts retrieved for a visually similar but incorrect entity can support a wrong answer. An agent trained on 2,000 teacher-generated trajectories combines image retrieval, cropping, reference-image inspection, and entity-scoped text access. Cross-Trajectory Verification organizes independently sampled trajectories into an entity-evidence forest and selects an existing answer using a textual verifier. Experiments on E-VQA and InfoSeek report improvements over the evaluated baselines; shared-pool comparisons support better selection accuracy and lower verifier-stage token usage.

Critical strengths
  1. The evaluation provides credible evidence of improved candidate selection. All TTS selectors reuse the same eight-rollout pools within each backbone. CTV improves E-VQA accuracy across all four backbones, with all 12 paired comparisons remaining significant after Holm correction (Appendix H.1). This isolates selection gains from differences in candidate generation.

  2. The evidence-access design improves the candidates available for verification. With the same policy checkpoint, replacing direct entity-page access with BGE retrieval reduces mean rollout BEM from 46.58% to 41.05% and Oracle@8 from 72.61% to 64.56% (Table 10). These results substantiate the practical value of the interface beyond the final selector.

  3. The transfer experiments broaden the empirical contribution. Without target-task weight updates, CTV improves ViQuAE EM from 53.38 to 59.82 and M3-VQA score from 25.74 to 30.67, with additional gains on MC-SEARCH (Appendix H.4). The disclosed prompt and output-format adaptations make these useful transfer results, although they do not establish superiority over alternative selectors on those tasks.

Critical weaknesses
  1. The grounding-trap explanation is less established than the accuracy improvement. Figure 10 explicitly excludes original and comparison images from CTV, making identity judgments dependent on recorded textual cues. Table 11 shows improved entity selection, but does not isolate recovery from evidence-supported wrong-entity trajectories. The exploratory audit in Appendix B.5 appropriately acknowledges its lack of independent validation. Consequently, the evidence does not establish how much improvement specifically addresses grounding traps rather than more general evidence-based answer selection.

  2. The contribution of entity-based organization is not sufficiently isolated. Table 2 reports 54.1% accuracy for CTV versus 50.2% for a flat listwise verifier, which is a meaningful comparison. However, the paper does not separately establish the effects of entity grouping, observation deduplication, and the verification rubric. The flat baseline’s prompt and input construction are also insufficiently specified to resolve this attribution. This matters because the entity-evidence forest is the central methodological contribution.

  3. The practical benefit of scaling remains incompletely quantified. Appendix H.3 correctly restricts its efficiency claim to verifier-stage tokens. For Qwen3-VL-8B, Table 11 reports approximately 11.2k tokens for CTV versus 12.3k for self-consistency, but omits the shared generation cost needed to interpret the overall saving. Figure 7 also shows CTV accuracy largely plateauing beyond 16 rollouts while token usage increases. The authors acknowledge diminishing returns; nevertheless, end-to-end cost measurements are needed to assess the practical value of additional sampling.

Questions for authors
  1. Additional analysis: On independently checked grounding-trap cases, how often does CTV recover the correct entity and answer relative to the flat verifier and self-consistency, conditional on a correct candidate being available?

  2. Clarification and targeted ablation: Does the flat verifier use the same model, verification rubric, and retained evidence as CTV? Would the improvement persist with observation deduplication retained but entity grouping removed?

  3. Cost reporting: What are the total generation-plus-selection token usage and measured latency at 8 and 16 rollouts, including model calls used by tool-side summarization?

2 comments

Thank you for your careful review and constructive feedback. We address your comments and questions below.

W1&Q1: Additional grounding-trap analysis

On grounding-trap queries jointly confirmed by independent LLM auditors, CTV selects a correct answer in 52.06% of cases when the candidate pool contains one, compared with 46.91% for the flat verifier and 42.78% for self-consistency. To also address the reviewer's question about correct-entity recovery, we report answer accuracy and joint entity–answer accuracy separately.

Multi-LLM Audit and human review. Following App. B.5, a grounding trap requires an incorrect final answer, an explicit wrong-entity commitment in the trajectory, and observed tool evidence from that entity supporting the predicted answer for the same queried attribute. App. J.1 reports the original LLM auditor and two additional LLM auditors' results on the same 2,251 errors.

LLM auditorQueries identified as trapsShare of the same 2,251 errors
Original LLM auditor55624.70%
Qwen3-VL-235B67730.08%
Llama3-405B61727.41%

All three LLM auditors jointly identify 434 queries. We draw a fixed random sample of 200 cases from this consensus-positive set for review by one human annotator. The current review summary records wrong-entity identification in all 200 cases, including 17 borderline cases in which the predicted and reference answers partially overlap; setting these aside leaves 183/200 (91.5%).

Selection performance when a correct candidate is available. Among the DiVE eight-rollout candidate pools for the 434 trap queries, 194 contain at least one correct answer. All three selectors use the same candidate pools:

SelectorCorrect answers on the 434 grounding-trap queriesAnswer accuracy on the full setSelection accuracy when at least one answer-correct candidate is available
Self-consistency8319.12%42.78%
flat ctv verifier9120.97%46.91%
CTV10123.27%52.06%

The flat CTV verifier uses the same LLM, retained evidence, and verification rubric. CTV improves conditional answer-selection accuracy by 5.15 and 9.28 percentage points over the flat verifier and self-consistency, respectively. Self-consistency and the flat CTV verifier also use DiVE's eight trajectories, so their recovery from grounding traps is partly attributable to DiVE's improved coverage of visual and textual evidence.

Joint entity–answer recovery: Among the 194 queries with a correct answer available, 143 contain at least one candidate with both the correct entity and the correct answer. Joint recovery rates are reported below:

MethodAnswer correctEntity and answer jointly correctJointly correct / 194Joint recovery rate / 143
SC835528.35%38.46%
Flat916432.99%44.76%
CTV1016734.54%46.85%

CTV therefore improves the joint recovery rate over SC and Flat by 8.39 and 2.09 percentage points, respectively.

Manuscript revisions: We have added cross-model audits, human review, and separate answer and joint entity–answer recovery results to App. J.1 (Table 23), included the audit prompt in Figure 15, and clarified the audit limitations in App. B.5.

W2: CTV component ablations

The flat verifier uses the same model and receives all eight complete trajectories. We also conduct more detailed CTV component ablations on all 3,750 E-VQA queries and 3,000 sampled InfoSeek queries.

Entity-based trajectory groupingObservation deduplicationVerification rubricE-VQA (all 3,750 queries)InfoSeek (3,000 sampled queries)
00050.21%53.43%
10051.55%54.67%
01053.33%54.00%
00152.93%54.50%
01153.44%54.23%
10153.07%54.33%
11054.08%54.57%
11154.35%55.30%

The results show that CTV achieves the highest accuracy when all three components are combined. Our methodological contribution has two complementary parts: direct visual exploration to improve candidate coverage, and CTV to improve evidence-based selection for test-time scaling in KB-VQA.

Manuscript revisions: We have added the grouping, deduplication, and verification-rubric ablations, together with verifier token usage, to App. H.2.2 (Table 16).

W3&Q3: End-to-end cost and latency

We report total generation and verification token usage and measured end-to-end latency at N=8 and N=16, including model calls for tool-side summarization. We use the same 200 queries per dataset, four H100 GPUs, an allocation of 32 CPU cores, four concurrent queries, at most eight simultaneous trajectories per query, and a per-rollout interaction limit of B=15.

DatasetRollout count NMean total tokens/query (input + output)Measured median latency (s/query)P95 (s/query)
E-VQA8131,636.3438.7585.27
E-VQA16254,777.0484.86166.94
InfoSeek8120,807.2729.20110.59
InfoSeek16236,311.9067.12203.74

Latencies are measured completion times under the stated hardware. Because at most eight trajectories per query run in parallel, the N=16 latency reflects this concurrency limit. At N=8, verifier tokens account for 8.79% and 8.31% of total model tokens on E-VQA and InfoSeek, respectively.

Cost reporting. At N=1, DiVE and RL-trained ProMSA* have comparable token costs: mean tokens are 15.38k versus 14.35k on E-VQA, and 13.98k versus 12.61k on InfoSeek, respectively. Among TTS methods using the same rollout count and shared candidate pools, CTV reduces verifier-stage token usage relative to SC. Including the common rollout-generation cost therefore also yields a modest reduction in total tokens used for generation and selection. These two comparisons separately characterize the generation cost of a single-rollout agent and the efficiency of TTS selection under a given sampling budget.

Manuscript revisions: We have added end-to-end token and latency measurements, the generation/verification breakdown, and depth–width allocation results to App. J.2 (Tables 25–27), with the efficiency scope clarified in App. H.3 and the main text.

We hope our responses and manuscript revisions have addressed all your concerns. We welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.