acceptodds
Under review as a conference paper at ICLR 2027

VeriSeek-VL: Seeking Visual Evidence and Verifying Hypotheses in Multimodal Deep Search

73 views

Abstract

Multimodal deep search agents solve long-horizon information-seeking tasks by iteratively retrieving and reasoning over textual and visual evidence. However, existing methods face two limitations: insufficient active acquisition of visual evidence and premature hypothesis lock-in. First, when key visual evidence is not explicitly provided, agents tend to rely on textual evidence and rarely actively acquire external visual evidence. Second, agents often treat an insufficiently supported hypothesis as fact in subsequent reasoning, so a wrong hypothesis leads the search away from the correct answer. To address these limitations, we propose VeriSeek-VL, a multimodal deep search framework that integrates data synthesis, training, and inference-time control. First, to strengthen active acquisition of visual evidence, we construct multimodal hypergraphs whose visual hyperedges link multiple entities through image-derived relations. Embedding these hyperedges into multi-hop reasoning chains yields tasks with implicit visual constraints, where key intermediate entities can be identified only through external visual evidence. Through supervised fine-tuning and GRPO on these tasks, the model learns to actively acquire external visual evidence. Second, to mitigate premature hypothesis lock-in, we identify a hypothesis lock-in axis through trajectory auditing and activation analysis. This axis is a linear direction in activation space that distinguishes lock-in steps from normal reasoning steps. During inference, we project each step's mean activation onto the axis and trigger verification of key hypotheses when the projection exceeds a threshold, reducing the risk of searching under a wrong hypothesis. Extensive experiments show that VeriSeek-VL achieves state-of-the-art performance among open-source methods on all five multimodal search benchmarks, surpassing the strongest baseline by 6.32% on average. Training improves the base model by 25.5%, and axis-guided verification adds a further 3.8%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Deep Search, Multimodal Agents, Synthetic Data Generation, Activation Analysis, Hypothesis Verification, Inference-Time Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PUPNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies two common failure modes in multimodal deep search agents: insufficient active acquisition of external visual evidence and premature lock-in on unverified intermediate hypotheses. First, the authors construct a multimodal entity hypergraph and rewrite entities into implicit visual constraints to synthesize multi-hop QA tasks requiring external image retrieval, which are combined with public datasets to train Qwen3-VL-32B-Thinking via SFT and GRPO. Second, using labels from complete-trajectory auditing, the authors extract a linear "Hypothesis Lock-in Axis" in the model's activation space and design an inference-time controller, HLI-Guard: when the projection score exceeds a threshold, an auxiliary model (GPT-5.6-Luna) is invoked to perform hypothesis extraction, verification prompt injection, context compression, and backtracking. Experiments across five multimodal search benchmarks demonstrate that both model training and inference-time control yield performance gains.

Critical strengths
  1. Decomposing failures in long-horizon multimodal search into two dimensions (evidence acquisition and hypothesis verification) is intuitive and well-targeted at key bottlenecks.
  2. The training pipeline yields substantial performance improvements over the base model.
Critical weaknesses
  1. VeriSeek-VL-FULL relies on a strong proprietary model as an auxiliary controller, leading to an unfair open-source comparison and unquantified overhead. Whenever HLI-Guard is triggered, evidence organization, hypothesis extraction, conflict assessment, and context compression/backtracking are all performed by GPT-5.6-Luna. Consequently, VeriSeek-VL-FULL-32B is in fact a hybrid system combining a 32B model with a proprietary frontier-model controller, making it unfair to directly compare it against purely open-source 7B–32B agents and claim open-source SOTA. It is recommended to include an evaluation where GPT-5.6-Luna is replaced by the 32B model itself.
  2. Lack of an isolated ablation on the core synthetic dataset. Although the paper presents "hypergraph data synthesis with implicit visual constraints" as a key contribution, there is no ablation comparing training on "public datasets alone (REDMM/LiveVQA/FVQA)" versus "incorporating the synthetic dataset," nor any statistics tracking changes in the frequency of active image-search tool calls before and after training (which I consider an essential behavioral pattern analysis for multimodal search). This ablation is critical for establishing the effectiveness of the proposed data synthesis pipeline.
Questions for authors

Please refer to the Critical Weaknesses above.

1 comment
AuthorsOpenReview ↗

Response: open-weight controller, data ablation, and visual search

Thank you for highlighting the controller dependency and the need to isolate our synthetic data's contribution. We have uploaded a revised PDF with an open-weight controller comparison, matched-supervision SFT ablations, and visual search analysis.

Scores below average four independent runs per question. Avg. is the equal-weight mean across the five benchmarks; MMSearch-Plus uses its 221-question single-image subset.

1. Open-weight controller

Following your suggestion, FULL now uses the same final RL 32B checkpoint as both search agent and auxiliary controller for evidence organization, hypothesis extraction, and conflict assessment. We retain Luna as a controller ablation. Both configurations use HLI-Guard at alpha=20% with a 100-turn limit.

ControllerMMBrowseCompBrowseComp-VLMMSearchMMSearch-PlusVisBrowseAvg.
GPT-5.6-Luna40.167.682.744.349.756.9
Same RL 32B checkpoint37.468.783.044.849.656.7

The average accuracies are close. In this evaluation, the 32B controller reduces mean end-to-end runtime from 33.05 to 25.81 minutes, saving 7.24 minutes per trajectory (21.9%). Mean trigger counts are also close: 1.194 for Luna and 1.230 for 32B. We therefore use the same 32B checkpoint as the default controller. (Sections 3.1-3.3, Tables 1-2; Appendix C.7.)

2. Contribution of synthetic data

Both SFT models start from Qwen3-VL-32B-Thinking and use the same optimization settings, with 13,919 training and 284 validation trajectories each. We match approximately 69.28M supervised tokens by adjusting the number of updates, counting loss-bearing assistant outputs across all turns. Public-only uses 4,671 REDMM, 6,140 LiveVQA, and 3,392 FVQA trajectories. The full corpus contains 6,669 trajectories from our QA pairs with implicit visual constraints before the training/validation split. Both models use the full toolset and a 40-turn limit.

SFT dataMMBrowseCompBrowseComp-VLMMSearchMMSearch-PlusVisBrowseAvg.
Public-only22.256.072.921.837.442.06
Full mixture33.962.280.335.945.651.58

Our data improves all five benchmarks at matched supervision, adding 9.52 points on average. The largest gains are on MMSearch-Plus (+14.1) and MMBrowseComp (+11.7), where fine-grained visual cues guide retrieval and evidence integration. (Section 3.2; Appendices A.2-A.3.)

3. Active visual search and its task benefit

We measure all 5,252 evaluation trajectories per model. Coverage is the fraction with at least one image-search call; image operations are zoom-in and OCR. Counts are per-trajectory means.

ModelImage-search coverage (%)Image-search callsImage operationsTotal tool calls
Base48.30.490.282.55
Public-only SFT66.10.720.359.76
Full SFT83.71.850.529.08

The full mixture raises image-search coverage from 66.1% to 83.7%, while reducing total tool calls from 9.76 to 9.08. Removing TextSearchImage and ImageSearchImage lowers average accuracy by 11.92 points for full SFT (51.58 to 39.66), versus 6.84 for Public-only (42.06 to 35.22). The gap between the models narrows from 9.52 to 4.44 points. Thus, our data improves active visual search and the task benefit obtained from image-search access, with fewer total tool calls. (Appendices A.3-A.4.)

Reviewer_nCxfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriSeek-VL trains a multimodal search agent using questions with implicit visual constraints synthesized from entity hypergraphs. SFT and GRPO develop its search policy. At inference, HLI-Guard uses an activation direction fitted to audited trajectories to trigger hypothesis verification, maintain evidence states, and backtrack. The full system combines Qwen3-VL-32B with GPT-5.6-Luna as its auxiliary controller.

Critical strengths
  • The problem is well motivated. Retrieving an image that the question does not supply and checking an intermediate entity binding are useful abilities for multimodal search.
  • The training gains are substantial. Under the same ReAct workflow, SFT raises average accuracy from 27.56 to 51.58; RL reaches 53.10. Both trained checkpoints use a 40-turn cap. FULL reports the highest scores among the listed specialized search agents on all five benchmarks.
  • Table 2 includes useful budget and trigger controls. At 100 turns, HLI-Guard reports 63.47 average accuracy on three benchmarks, versus 60.66 without Guard and 61.04 with periodic Guard. The Guard variants share verification, tracking, compression, and backtracking procedures.
Critical weaknesses
  1. The interpretation of the axis as hypothesis lock-in needs stronger validation. The label measures whether a hypothesis remains unverified at the end of a complete trajectory. Appendix D also labels deferred or abandoned hypotheses as positives, while a wrong hypothesis subsequently refuted contributes no positive label. This does not require premature commitment or continued search based on that hypothesis. The reported AUC of 0.9219 therefore supports prediction of eventual non-verification in the analyzed base-model trajectories; its specificity to lock-in remains unclear. The approximately 300 human-annotated trajectories are mentioned without agreement results, and quantitative held-out discrimination for the final RL checkpoint's separately fitted axis is missing.
  2. The Guard comparison is promising but insufficiently resolved. Table 2 uses single runs for the controls and four-run means for HLI-Guard, with no uncertainty estimates. It covers only three benchmarks, excluding VisBrowse, where FULL has one of its largest gains. Actual trigger frequencies and auxiliary-model calls are not reported. Selecting the top 20% of calibration scores does not ensure the same verification budget as random or periodic triggering during deployment.
  3. The contribution of implicit visual constraints is not isolated. The new data constitute 46.95% of SFT trajectories and 32.21% of RL questions, but there is no training ablation removing or replacing them at a fixed budget. Filtering questions that a tool-free model solves and retaining trajectories that invoke image search also leave text-search shortcuts untested. The overall training gains establish a strong recipe, while the specific benefit of the proposed data construction remains uncertain.
  4. The best configuration relies on GPT-5.6-Luna for runtime evidence organization, hypothesis extraction, and conflict assessment. The open-source SOTA claim should explicitly describe this dependency. Auxiliary token usage, calls, and latency are needed to assess the system's practical advantage; an open-weight controller comparison would also help. The 0.3-point lead on MMSearch is particularly difficult to assess without uncertainty estimates.
Questions for authors
  1. Can you report human–automatic annotation agreement and question-grouped held-out results for the final RL axis? How well does it distinguish premature commitment from tentative or abandoned hypotheses?
  2. Can all Guard variants be evaluated with four runs on all five benchmarks, reporting paired uncertainty, actual trigger counts, and auxiliary-model costs?
  3. At a fixed training budget, how much gain comes from the implicit-constraint data? What fraction of these questions can a web-enabled text-only solver answer?
2 comments
AuthorsOpenReview ↗

Annotation quality and synthetic data

Thank you for the specific questions about annotation, the final RL axis, verification budgets, and data contribution. We have uploaded the revised PDF. This part addresses annotation and data; Part 2 gives the full Guard comparisons, runtime statistics, and repeated-run results.

1. Human validation and final-checkpoint discrimination

Human annotators mark a turn positive when the agent treats an insufficiently verified hypothesis as an established fact and uses it to guide subsequent search. Complete-trajectory auditing uses the verification evidence to identify relevant hypotheses and trace their first occurrence. Our independent gold set contains 300 trajectories and 2,442 turns: 628 positive (25.72%) and 1,814 negative (74.28%). The audit achieves 93.86% accuracy, 82.21% precision, 97.13% recall, and 89.05% F1. We prioritize recall to retain lock-in events from the minority positive class. Of 742 audit-positive steps, 610 agree with the gold labels and 132 (17.79%) are false positives. Manual review finds that most involve hypotheses set aside or abandoned without sufficient verification. The revision states the behavioral definition, audit procedure, and these observed errors explicitly. (Section 2.2; Appendix B.2.)

For the final RL checkpoint, we evaluate the layer-42 axis with five-fold cross-validation grouped by question. All trajectories from a question stay in one fold. In each split, four folds estimate class means and the normalized axis, and the remaining fold supplies test projections. The five ROC-AUC values are 0.911, 0.898, 0.899, 0.845, and 0.916: 0.8938 +/- 0.0283 (mean and sample standard deviation). The final RL axis thus discriminates the two audited step classes on held-out questions.

We also examine the base Thinking model using 12 role prompts: four wordings for each of searcher-like, default, and answer-like roles. All use the same 1,000 questions, with five trajectories per question. The 66 pairwise cosine similarities between independently estimated layer-42 axes range from 0.89 to 0.98, showing consistent directions across these prompt variations. (Appendices B.5.4-B.5.5.)

2. Data contribution at matched supervision

Both SFT models start from Qwen3-VL-32B-Thinking and use the same optimization settings, with 13,919 training and 284 validation trajectories each. We match approximately 69.28M supervised tokens by adjusting the number of updates, counting loss-bearing assistant outputs across all turns. Public-only uses 4,671 REDMM, 6,140 LiveVQA, and 3,392 FVQA trajectories. Both models use the same ReAct evaluation with the full toolset and a 40-turn limit. Scores average four independent runs; Avg. equally weights the five benchmarks. MMSearch-Plus uses its 221-question single-image subset.

SFT dataMMBrowseCompBrowseComp-VLMMSearchMMSearch-PlusVisBrowseAvg.
Public-only22.256.072.921.837.442.06
Full mixture33.962.280.335.945.651.58

Our synthetic data improves all five benchmarks, by 9.52 points on average. The full mixture raises image-search coverage from 66.1% to 83.7%, while reducing total tool calls from 9.76 to 9.08. Removing TextSearchImage and ImageSearchImage lowers average accuracy by 11.92 points for full SFT (51.58 to 39.66), versus 6.84 for Public-only (42.06 to 35.22). The gap between the models narrows from 9.52 to 4.44 points. Thus, our data improves active visual search and the task benefit obtained from image-search access, with fewer total tool calls. Image-search coverage and tool counts are measured over all 5,252 trajectories per model. (Section 3.2; Appendices A.2-A.4.)

3. Text-shortcut audit

Our existing filtering first removes questions answered correctly without tools, then retains only teacher trajectories with at least one image-search call and a correct final answer. We add an audit of 500 distinct synthetic questions whose trajectories entered SFT. Using the same teacher, Qwen3-VL-235B-A22B-Thinking, we remove image-related tools and sample four independent rollouts per question. At least one rollout answers correctly for 43 questions (8.6%), estimating the residual share solvable through text shortcuts. Given the cost of repeated teacher sampling, we use the two-stage filters for corpus-wide selection and repeated sampling for this sample audit. (Appendix A.1.5.)

AuthorsOpenReview ↗

Verification timing and runtime

4. All five benchmarks, four runs, and actual verification frequency

Every setting below uses the final RL checkpoint and four independent runs per question. All Guard variants share the 32B controller and control procedure, with a 100-turn limit. Random Guard uses a per-turn probability; HLI alpha denotes the upper fraction of calibration scores. Columns follow MMBrowseComp, BrowseComp-VL, MMSearch, MMSearch-Plus (221 single-image questions), and VisBrowse. Avg. equally weights the five benchmarks.

SettingMMBCBC-VLMMSMMS+VisAvg.
No Guard, 40 turns34.465.981.340.143.853.1
No Guard, 100 turns34.266.781.039.943.853.1
Random, 10%33.765.379.740.645.152.9
Random, 20%31.964.279.640.044.452.0
Periodic, every 1034.265.481.239.144.152.8
Periodic, every 534.765.681.839.945.453.5
HLI, alpha=10%37.168.782.843.448.456.1
HLI, alpha=20%37.468.783.044.849.656.7
HLI, alpha=30%36.668.482.944.147.956.0

HLI outperforms random and periodic triggering on every benchmark. Runtime statistics include all trajectories, including those with no trigger:

SettingCoverage (%)Mean triggersMean turnsRuntime (min)
No Guard, 40008.7719.66
No Guard, 100008.7419.27
Random, 10%45.960.77610.0024.13
Random, 20%67.171.56610.9528.18
Periodic, every 1024.380.3639.3822.18
Periodic, every 555.501.19510.2926.83
HLI, alpha=10%57.370.79610.2823.18
HLI, alpha=20%73.521.23011.3225.81
HLI, alpha=30%80.821.49611.7831.30

No Guard at 100 turns scores 53.1, with 8.74 mean turns and 19.27 minutes runtime. HLI 20% scores 56.7, with 11.32 turns and 25.81 minutes. Random and periodic policies also add interactions and time.

At similar mean trigger counts, HLI-Guard improves accuracy over both alternative policies:

  • HLI 10% vs. Random 10%: 0.796 vs. 0.776 triggers; accuracy 56.1 vs. 52.9; runtime 23.18 vs. 24.13 minutes. The paired gain is 3.18 points, with 95% CI [1.25, 5.09].
  • HLI 20% vs. Periodic every 5 turns: 1.230 vs. 1.195 triggers; accuracy 56.7 vs. 53.5; runtime 25.81 vs. 26.83 minutes. The paired gain is 3.18 points, with 95% CI [1.19, 5.12].

There is also a comparison with nearly identical interaction lengths: HLI 10% and Periodic every 5 turns average 10.28 and 10.29 turns, respectively. HLI achieves higher accuracy (56.1 vs. 53.5) and lower runtime (23.18 vs. 26.83 minutes). These comparisons support using the lock-in signal to select more effective verification steps.

Raising HLI alpha from 20% to 30% increases coverage and runtime but reduces accuracy from 56.7 to 56.0. The 20% setting gives a better accuracy/runtime balance. Relative to No Guard at 100 turns, HLI 20% has a paired gain of 3.58 points, with 95% CI [1.60, 5.53]. (Section 3.3, Table 2; Appendix C.7.)

5. Controller comparison and run-to-run variation

FULL now uses the same final RL 32B checkpoint as search agent and auxiliary controller. Luna remains a controller ablation. Under HLI 20%, Luna and 32B score 56.9 and 56.7; coverage is 69.95% and 73.52%, mean triggers 1.194 and 1.230, and mean turns 11.56 and 11.32. Runtime drops from 33.05 to 25.81 minutes with 32B, a 7.24-minute (21.9%) saving at similar accuracy. (Sections 3.1-3.3, Tables 1-2; Appendix C.7.)

FULL's four independent runs are:

BenchmarkRun 1Run 2Run 3Run 4Mean
MMBrowseComp38.435.336.239.737.4
BrowseComp-VL69.267.770.966.968.7
MMSearch81.782.384.783.383.0
MMSearch-Plus45.746.242.544.844.8
VisBrowse51.550.347.948.549.6

MMSearch averages 83.0, compared with 82.4 for the strongest competing search agent. These per-run results show benchmark-level variation; the paired intervals above quantify the Guard-policy comparisons. (Appendix D.2, Figure D1; Appendix C.7.)

Reviewer_xz23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeriSeek-VL constructs multimodal hypergraphs from Wikipedia and Wikimedia images to synthesize multi-hop questions with implicit visual constraints, then trains a Qwen3-VL-32B-Thinking agent using SFT and reinforcement learning. It additionally identifies a Hypothesis Lock-in Axis from audited trajectories and uses HLI-Guard to selectively initiate verification, maintain hypothesis states, and backtrack when evidence contradicts an earlier assumption.

Critical strengths
  1. Relevant problem formulation. The paper identifies two practical failure modes of long-horizon search agents that are not adequately addressed by final-answer supervision alone. In particular, distinguishing the need to actively locate external images from merely understanding images already provided in the prompt is a meaningful research direction.

  2. Coherent framework and implementation details. The multimodal hypergraph construction and implicit visual-constraint rewriting provide a reasonably systematic way to generate tasks that encourage cross-modal, multi-hop retrieval. The paper also presents a concrete activation-based controller with hypothesis tracking, context compression, and backtracking, rather than relying solely on an informal instruction to double-check answers.

  3. Broad empirical evaluation with promising improvements. The authors evaluate on five benchmarks and compare with a substantial set of general-purpose and specialized search agents. The main table shows a sizable improvement from the base ReAct model to the trained models, while Table 2 offers an initial attempt to compare axis-based triggering with random and periodic verification. The supplementary analysis of activation directions across model variants and prompt categories is also useful.

Critical weaknesses
  1. The contribution of the proposed visual-data synthesis is not isolated. The principal training result combines the authors' synthetic data with several existing datasets, but there is no corresponding ablation that removes the hypergraph-derived questions while keeping the training setup and data budget comparable. According to Table 4, the proposed questions account for 46.95% of the SFT corpus and 32.21% of the RL questions; consequently, the substantial SFT improvement in Table 1 cannot be attributed specifically to implicit visual constraints or hypergraph construction. Furthermore, filtering questions based on whether a tool-free model answers them correctly does not demonstrate that the remaining questions genuinely require visual evidence: some may still be solvable through text retrieval or other shortcuts. A controlled data ablation, together with a measurement of successful image retrieval and visual-evidence usage, would substantially strengthen the central seeking claim.

  2. The operational definition of hypothesis lock-in does not fully match the claimed failure mode. Section 2.2 and Eq. (6) label the first introduction of a search-critical hypothesis as a lock-in onset whenever that hypothesis remains unverified at the end of the complete trajectory. However, this label does not explicitly require the agent to have accepted the hypothesis as fact or to have based later search decisions on it, which are central to the stated definition of premature lock-in.

  3. The benefit and computational cost of activation-guided verification need clearer attribution. HLI-Guard invokes GPT-5.6-Luna for evidence organization, hypothesis extraction, and conflict assessment, and the full system also increases the interaction limit from 40 to 100 turns. Moreover, the paper does not report the actual number of verification triggers, auxiliary-model calls, additional tokens, latency, or total inference cost. It is therefore difficult to determine the practical advantage of the internal activation signal over a comparably budgeted verification policy. Matching trigger counts or auxiliary-model budgets and reporting the corresponding costs would make this conclusion more convincing.

  4. Some performance improvements are modest relative to likely evaluation variability: for example, the RL stage improves the five-benchmark average by 1.52 percentage points, and FULL exceeds the strongest specialized-agent result on MMSearch by only 0.3 points. Confidence intervals or per-run variation would help assess the robustness of these conclusions.

Questions for authors
  1. Could the authors provide an ablation removing the proposed hypergraph-based synthetic data while keeping the training budget comparable? How much of the improvement can be attributed specifically to implicit visual constraints rather than general SFT/RL training?

  2. Since HLI-Guard relies on GPT-5.6-Luna for hypothesis extraction and verification, could the authors clarify how much of the improvement comes from activation-based triggering versus the additional reasoning provided by the auxiliary model?

2 comments
AuthorsOpenReview ↗

Response (1/2): synthetic data, visual search, and annotation

Thank you for the detailed suggestions. The revised PDF adds matched-supervision data ablations, a text-shortcut audit, human validation, and four-run Guard comparisons with a 32B controller. We address data and annotation here, then verification timing and variability in Part 2.

1. Independent data contribution and visual search

Both SFT models start from Qwen3-VL-32B-Thinking and use the same optimization settings, with 13,919 training and 284 validation trajectories each. We match approximately 69.28M supervised tokens by adjusting the number of updates, counting loss-bearing assistant outputs across all turns. Public-only uses 4,671 REDMM, 6,140 LiveVQA, and 3,392 FVQA trajectories. Evaluation uses the full toolset, a 40-turn limit, and four independent runs per question. Avg. equally weights the five benchmarks; MMSearch-Plus uses 221 single-image questions.

SFT dataMMBrowseCompBrowseComp-VLMMSearchMMSearch-PlusVisBrowseAvg.
Public-only22.256.072.921.837.442.06
Full mixture33.962.280.335.945.651.58

The full mixture improves all five benchmarks, by 9.52 points on average. Gains are largest on MMSearch-Plus (+14.1) and MMBrowseComp (+11.7).

We also measure all 5,252 trajectories per model. Coverage means at least one image-search call; image operations include zoom-in and OCR. Counts are per-trajectory means.

ModelImage-search coverage (%)Image-search callsImage operationsTotal tool calls
Base48.30.490.282.55
Public-only SFT66.10.720.359.76
Full SFT83.71.850.529.08

The full mixture raises image-search coverage from 66.1% to 83.7%, while reducing total tool calls from 9.76 to 9.08. Removing TextSearchImage and ImageSearchImage lowers average accuracy by 11.92 points for full SFT (51.58 to 39.66), versus 6.84 for Public-only (42.06 to 35.22). The gap between the models narrows from 9.52 to 4.44 points. Thus, our data improves active visual search and the task benefit obtained from image-search access, with fewer total tool calls. (Section 3.2; Appendices A.2-A.4.)

2. Residual text shortcuts

Our existing filtering first removes questions answered correctly without tools, then retains only teacher trajectories with at least one image-search call and a correct final answer. We add an audit of 500 distinct synthetic questions whose trajectories entered SFT. Using the same teacher, Qwen3-VL-235B-A22B-Thinking, we remove image-related tools and sample four independent rollouts per question. At least one rollout answers correctly for 43 questions (8.6%), estimating the residual share solvable through text shortcuts. Given the cost of repeated teacher sampling, we use the two-stage filters for corpus-wide selection and repeated sampling for this sample audit. (Appendix A.1.5.)

3. Complete-trajectory auditing and the behavioral definition

Human annotators mark a turn positive when the agent treats an insufficiently verified hypothesis as an established fact and uses it to guide subsequent search. Complete-trajectory auditing uses the verification evidence to identify relevant hypotheses and trace their first occurrence. Our independent gold set contains 300 trajectories and 2,442 turns: 628 positive (25.72%) and 1,814 negative (74.28%). The audit achieves 93.86% accuracy, 82.21% precision, 97.13% recall, and 89.05% F1. We prioritize recall to retain lock-in events from the minority positive class. Of 742 audit-positive steps, 610 agree with the gold labels and 132 (17.79%) are false positives. Manual review finds that most involve hypotheses set aside or abandoned without sufficient verification. The revision states the behavioral definition, audit procedure, and these observed errors explicitly. (Section 2.2; Appendix B.2.)

AuthorsOpenReview ↗

Response (2/2): verification timing and evaluation variability

4. Verification timing versus extra reasoning

We now use the same final RL 32B checkpoint as search agent and controller. Guard variants share verification, hypothesis tracking, compression, and backtracking, with a 100-turn limit and four independent runs on all five benchmarks. No Guard is the ReAct baseline without Guard.

SettingAvg. accuracyMean triggersMean turnsRuntime (min)
No Guard, 100 turns53.108.7419.27
Random Guard, 10%52.90.77610.0024.13
HLI-Guard, alpha=10%56.10.79610.2823.18
Periodic Guard, every 553.51.19510.2926.83
HLI-Guard, alpha=20%56.71.23011.3225.81

Relative to No Guard, HLI 20% raises accuracy from 53.1 to 56.7, turns from 8.74 to 11.32, and runtime from 19.27 to 25.81 minutes. Random and periodic verification also increase turns and runtime, but score 52.9 and 53.5, respectively.

At similar mean trigger counts, HLI-Guard improves accuracy over both alternative policies:

  • HLI 10% vs. Random 10%: 0.796 vs. 0.776 triggers; accuracy 56.1 vs. 52.9; runtime 23.18 vs. 24.13 minutes. The paired gain is 3.18 points, with 95% CI [1.25, 5.09].
  • HLI 20% vs. Periodic every 5 turns: 1.230 vs. 1.195 triggers; accuracy 56.7 vs. 53.5; runtime 25.81 vs. 26.83 minutes. The paired gain is 3.18 points, with 95% CI [1.19, 5.12].

There is also a comparison with nearly identical interaction lengths: HLI 10% and Periodic every 5 turns average 10.28 and 10.29 turns, respectively. HLI achieves higher accuracy (56.1 vs. 53.5) and lower runtime (23.18 vs. 26.83 minutes). These comparisons support using the lock-in signal to select more effective verification steps.

For the controller comparison, Luna and 32B score 56.9 and 56.7 on average, with mean runtimes of 33.05 and 25.81 minutes. The 32B configuration saves 7.24 minutes per trajectory (21.9%) while retaining similar accuracy; it is now the main configuration, with Luna retained as an ablation. (Section 3.3, Table 2; Appendix C.7.)

5. Evaluation variability

We report FULL's four independent runs on every benchmark:

BenchmarkRun 1Run 2Run 3Run 4Mean
MMBrowseComp38.435.336.239.737.4
BrowseComp-VL69.267.770.966.968.7
MMSearch81.782.384.783.383.0
MMSearch-Plus45.746.242.544.844.8
VisBrowse51.550.347.948.549.6

The revised MMSearch mean is 83.0, compared with 82.4 for the strongest competing search agent. We separately report the main SFT gain, the smaller benchmark-dependent RL gain, and the effect of Guard. The paired intervals above quantify the improvements over random and periodic triggering, and both lie above zero. (Sections 3.2-3.3; Appendix D.2, Figure D1; Appendix C.7.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.