acceptodds
Under review as a conference paper at ICLR 2027

BrowseComp-V³: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents

Abstract

Multimodal large language models (MLLMs) are evolving into browsing agents capable of multimodal deep search, supported by advances in autonomous planning and tool use. However, existing benchmarks remain limited in task complexity, information searchability, and evaluation dimensions, hindering comprehensive assessment in open-world settings. To address these limitations, we introduce BrowseComp-V³, a benchmark comprising 300 challenging, manually constructed questions across diverse domains. The tasks emphasize multi-hop, cross-modal reasoning at multiple levels of complexity and are designed to require evidence gathering through web browsing. We verify the public searchability of key evidence and provide expert-validated subgoals for process evaluation, enabling fine-grained analysis of search behavior and model capabilities. We also provide OmniSeeker, a general multimodal browsing agent framework, and evaluate a broad range of MLLMs and browsing systems. Even OpenAI Deep Research, the best-performing proprietary system in our evaluation, achieves a success rate of only 56% on BrowseComp-V³. Further analysis reveals bottlenecks in multimodal information integration and fine-grained perception, suggesting that effectively using visual evidence in multi-step search and reasoning remains challenging.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Browsing Agents, Multimodal DeepResearch, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_imYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BrowseComp-V3, a benchmark of 300 manually constructed questions and 383 images for evaluating multimodal web-browsing agents. Its questions span five broad domains and require multi-step search and reasoning across textual and visual evidence. In addition to final answers, the dataset provides annotated evidence chains and intermediate subgoals intended to measure an agent’s progress. The authors also introduce OmniSeeker, a model-agnostic browsing framework with text search, image search, reverse-image search, webpage access, and image cropping. They evaluate humans, proprietary research systems, tool-free models, open-source agents, and OmniSeeker-based agents using final-answer success rate and a subgoal-based Process Score.

Critical strengths

Evaluating the capabilities of browser search in multimodal scenarios is both interesting and important.

Critical weaknesses
  1. Fig.2 right table shows the task level and difficulty distribution. How are these two metrics defined? Especially for question difficulty? In Fig 15, one example in the right column "When was this photo taken?" Why is this sample defined as an easy case?
  2. The proposed dataset includes a process-level evaluation. For each question, is this process predefined and unique? Could there be multiple search paths to arrive at the correct answer? Is it possible for a search path to differ from the annotated process in the dataset, yet still yield the correct final result?
  3. In Fig. 5, Visual Grounding Error is the main failure mode for multiple models. How to assess whether an error is caused by incorrect visual grounding? Is it necessary to annotate grounding on browser pages retrieved through real-time search?
  4. typo: In Fig 15, bottom-right example, the second image does not align with the given question.
Questions for authors

Please see the weakness section

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer imYZ

Dear Reviewer imYZ,

We sincerely appreciate your careful review and helpful suggestions. We have carefully addressed the concerns you raised, including the definitions of task level and question difficulty (W1), whether the annotated process is unique and path-dependent (W2), how Visual Grounding Errors are assessed (W3), and the image–question mismatch in Figure 15 (W4). We have also outlined the corresponding revisions we will make to the manuscript.

As the discussion period is drawing to a close, we would be grateful if you could take another look at our response and let us know whether it adequately addresses your concerns. We would also be happy to clarify any remaining points.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Kind Reminder to Reviewer imYZ

Dear Reviewer imYZ,

We understand that this is a busy period, and we sincerely appreciate your time. As the discussion period will end shortly, we wanted to kindly check whether our response has addressed your concerns. If any questions remain, we would be glad to clarify them before the discussion closes.

Best regards,

The Authors

Dear Reviewer imYZ,

We apologize for following up again. As the discussion period will close in a few hours, we would be very grateful if you could briefly let us know whether our response has addressed your concerns. If anything remains unclear, we are happy to respond right away.

Thank you again for your time and review.

Best regards,

The Authors

Reviewer_uVp7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

BrowseComp-V3 offers a benchmark of 300 expert-crafted questions spanning 24 domains to evaluate multimodal browsing agents. Each task demands multi-hop reasoning that combines text and images from web searches. Three design principles guide the benchmark: all evidence must be findable through standard search engines, expert-validated subgoals enable process evaluation beyond just checking final answers, and task complexity varies along dimensions like search depth and cross-modal interaction. The authors built OmniSeeker, a ReAct-style agent framework with unified interfaces for web search, image search, and visual cropping. They tested deep research systems, tool-augmented MLLMs, tool-free MLLMs, and open-source agents. OpenAI Deep Research tops the automated leaderboard at 56% success versus 68% for humans. Tool augmentation makes a huge difference—GPT-5.2 jumps from 6% to 36% when given OmniSeeker tools. Interestingly, process scores consistently run higher than success rates, meaning agents often collect relevant evidence along the way without nailing the final answer.

Critical strengths
  • The public searchability requirement distinguishes this from earlier benchmarks where key evidence lived in videos or documents that normal tools can't handle, which makes evaluation more reproducible and realistic for systems people actually deploy.

  • Process evaluation through expert-validated subgoals lets us see where agents stumble along multi-step paths, which beats just looking at whether they got the final answer right or wrong.

  • The construction methodology looks solid—pairs of experts annotate independently, multiple verification stages, adversarial filtering with strong models, explicit protocols for link rot and content drift.

Critical weaknesses
  • At 300 questions, I'm not confident this captures the full range of scenarios where multimodal search matters. With 24 sub-domains, that's roughly a dozen examples each on average, which seems thin.

  • The failure analysis buckets errors into seven types but doesn't really explain why failures happen or point toward solutions.

  • OmniSeeker is essentially a standard ReAct loop with off-the-shelf tools. There's minimal architectural novelty beyond creating uniform evaluation conditions.

  • Every task depends on expert annotations for subgoals and reference paths. This supervised framing might not reflect how agents perform when genuinely exploring open-ended information needs without guidance.

Questions for authors
  1. How sensitive are these results to the specific search backends and vision models in OmniSeeker? Would switching search engines or vision components significantly change the rankings?

  2. What's the correlation between process score and success rate for individual questions? Does a high process score reliably tell us which questions an agent is close to cracking?

  3. Have you looked at computational cost and latency breakdowns across tools and models? This matters for whether these systems could work in interactive applications.

  4. Could the expert-annotated subgoals be used to train agents to decompose complex queries better? Have you tried this?

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer uVp7

Dear Reviewer uVp7,

We sincerely appreciate your thoughtful feedback and constructive suggestions. We have carefully addressed the concerns you raised, including benchmark scale and statistical power (W1), bottleneck localization and actionable remedies beyond the failure taxonomy (W2), a same-backbone comparison of OmniSeeker with existing agent frameworks (W3), and alternative-route compatibility (W4). We have also added new results on sensitivity to vision components and search backends (Q1), question-level PS–SR correlation (Q2), cost and latency (Q3), and sub-goal-guided query decomposition (Q4), and outlined the corresponding revisions we will make to the manuscript.

As the discussion period is drawing to a close, we would be grateful if you could take another look at these new results and clarifications and let us know whether they adequately address your concerns. We would also be happy to clarify any remaining points.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_uVp7OpenReview ↗

Reply to authors

The supplemental materials resolve my concern, I decide to increase my score.

Dear Reviewer uVp7,

Thank you very much for your reply and for increasing your score. We are glad that the supplementary materials have resolved your concerns, and we will incorporate the new results and corresponding revisions into the final version.

Best regards,

The Authors

Reviewer_YqV4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BrowseComp-V³, a benchmark of 300 manually constructed questions for multimodal web browsing, covering 24 sub-domains and three levels of cross-modal interaction. It combines publicly retrievable evidence, annotated reference trajectories, and intermediate sub-goals for process-level evaluation. The authors also provide OmniSeeker, a unified browsing framework, and evaluate a broad range of models and agent systems. Additional experiments examine visual-tool dependence, staged oracle assistance, alternative trajectories, and human performance under extended protocols.

The benchmark addresses a relevant problem and includes substantial annotation and diagnostic work. However, inconsistencies in the reported results and insufficient validation of the process metric currently weaken confidence in several central conclusions.

Critical strengths
  1. Useful benchmark design and substantial annotation effort. The combination of multi-step browsing, visual evidence distributed across sources, and intermediate annotations offers a useful evaluation setting. Independent annotation, expert reconciliation, and evidence archiving are valuable measures for improving dataset reliability.

  2. Informative controlled experiments. The cross-benchmark tool ablations and staged oracle experiments go beyond reporting aggregate accuracy. In particular, the matched crop/caption interventions provide useful evidence that locating relevant images and extracting visual facts remain challenging even when retrieval is substantially simplified.

  3. Broad evaluation with thoughtful supplementary analyses. The paper covers proprietary systems, open-source agents, and multiple backbones within a shared framework. The extended human evaluation and alternative-route analysis address legitimate concerns about time constraints and reference-trajectory dependence, although their conclusions remain limited to the evaluated subsets.

Critical weaknesses

W1. The main results contain numerical inconsistencies that require reconciliation.

In Table 2, OmniSeeker with Qwen3.6-Plus has domain-level success rates of 32.00, 20.00, 28.00, 29.33, and 24.00, yet its overall success rate is reported as 21.00. Using the domain sizes in Figure 2 gives approximately 27.02%, rather than 21.00%. Its domain-level Process Scores similarly imply approximately 38.66%, rather than 32.43%. Other entries also warrant checking: GPT-5.2-Thinking’s domain results imply approximately 39.04% rather than 39.13%. These discrepancies cannot be explained by the displayed rounding. Since the evaluation is described as covering all 300 questions with equal question weights, the authors should reconcile the aggregation and release per-question results.

W2. Process Score measures reported factual coverage more directly than verified reasoning progress.

Although the rubric checks for obvious fabrication and contradictions, routine Process Score computation matches gold key_info against emitted sub-goals without inspecting the underlying tool evidence or explicitly validating dependency satisfaction. Consequently, reporting a correct intermediate fact does not necessarily establish that the agent retrieved, grounded, or used it correctly. Conversely, Appendix A.3 explicitly identifies cases where evidence was acquired but omitted from the tags. This creates sensitivity to reporting behavior, particularly across proprietary systems and agents with different output interfaces.

The alternative-route analysis is helpful, but it does not validate judge accuracy or evidence faithfulness. A manually audited subset comparing reported coverage with evidence-supported completion would strengthen this central contribution. The paper should also clarify whether human anomaly adjudication changes the reported SR, since Appendix C.4 permits a final correctness verdict after inspecting the reasoning, whereas SR is otherwise defined by final-answer matching.

W3. Some diagnostic interpretations go beyond what the measurements establish.

Appendix A.6.3 interprets a decline in cumulative key-information coverage as evidence that additional content “dilutes” previously acquired information. However, coverage of a fixed gold set in accumulated text cannot decrease within an individual trajectory. If the plotted average changes because shorter trajectories leave the active sample, the decline may reflect changing sample composition rather than information dilution.

Similarly, the claim that models’ shallower performance decline with search depth demonstrates compensation through parametric knowledge is not directly tested. These explanations should be qualified as hypotheses or supported by analyses that rule out alternative explanations, including floor effects and differences in task composition.

W4. Evaluation reproducibility need stronger reporting.

The adversarial filtering procedure lacks operational details, including the initial candidate count, rejection rate, tool access, and criterion for being “trivially” solvable. Finally, proprietary systems are evaluated without the interaction limits imposed on OmniSeeker. This is reasonable for product-level comparisons, but exact evaluation dates, configurations, and resource usage are needed to interpret the rankings and reproduce the setting.

Questions for authors
  1. Can you reconcile Table 2 with the category sizes and provide per-question scores and aggregation code?
  2. How are intermediate findings collected consistently across all system types, and how well does Process Score agree with human judgments of evidence-supported completion? Note that a human audit on a representative subset would be sufficient, provided it supports agreement with evidence-based human judgments and addresses the anomaly-adjudication and reporting-sensitivity concerns raised in W2.
  3. Does Figure 13 average over a changing set of active trajectories? If so, does the information-dilution conclusion survive a fixed-cohort analysis?
  4. Can you provide filtering statistics, run-level uncertainty, and the number of outcomes changed by human adjudication?

Overall Assessment

This is a promising benchmark with useful diagnostic experiments and considerable curation effort. My current assessment is borderline, leaning toward rejection because the numerical inconsistencies and process-evaluation concerns affect the reliability of the central evidence. I would be willing to raise my assessment if the authors reconcile the results and provide convincing validation of the scoring protocol.

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer YqV4

Dear Reviewer YqV4,

We sincerely appreciate your detailed review and constructive suggestions. We have carefully addressed the concerns you raised, including the reconciliation of the Table 2 aggregates (W1 & Q1), the Process Score semantics and human evidence audit (W2 & Q2), the revised diagnostic interpretations (W3 & Q3), and further details on adversarial filtering, run-level variance, and proprietary system reproducibility (W4 & Q4). We have also outlined the corresponding revisions we will make to the manuscript.

As the discussion period is drawing to a close, we would be grateful if you could take another look at our response and let us know whether it adequately addresses your concerns. We would also be happy to clarify any remaining points.

Thank you again for your time and consideration.

Best regards,
The Authors

Thanks for the awesome reply. They have already addressed all my concerns with a detailed reply. I appreciate their work and I will raise my scores 3.

Dear Reviewer YqV4,

Thank you very much for your prompt reply and for raising your score. We are glad that our response has addressed your concerns, and we will incorporate the corresponding revisions into the final version.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary to the Area Chair and Reviewers

Dear Senior Area Chair, Area Chair, and Reviewers,

As the discussion period draws to a close, we would like to sincerely thank the Senior Area Chair and Area Chair for overseeing the review process and all three reviewers (YqV4, uVp7, and imYZ) for their thorough and constructive feedback. Your comments have substantially improved the rigor and clarity of our work.


Discussion Overview

  • Reviewer YqV4: Noted that our response "addressed all my concerns" and raised the rating to 3 (Weak acceptance).
  • Reviewer uVp7: Noted that our supplemental materials "resolve my concern" and raised the rating to 3 (Weak acceptance).
  • Reviewer imYZ: We have provided detailed point-by-point responses to all raised points (W1–W4) and remain happy to clarify any remaining questions should time permit.

Summary of Key Responses and Additions

  1. Result Verification, Robustness, and Reproducibility (YqV4 W1, W4; uVp7 W1, Q1, Q3). A row-by-row audit of Table 2 confirmed all per-domain scores are accurate; we replaced manual spreadsheet aggregation with an automated script recomputing overall averages from unrounded per-question scores across all 300 questions, leaving all group leaders and main conclusions unchanged. We also added statistical power analyses, search-backend sensitivity (Google/Bing/Yandex, confirming identical SR rankings), multi-run variance, per-question cost/latency breakdowns, the full adversarial filtering funnel, and proprietary evaluation windows and latencies.

  2. Process Score Validation and Path Invariance (YqV4 W2; uVp7 W4; imYZ W2). We clarified raw sub-goal coverage vs. main-table PS, documented collection protocols across all six system types, and verified that anomaly adjudication affects SR in only 2 of 1,500 rollouts (0.13%). A dual-annotator blind audit of 100 trajectories (520 pairs) shows 91.83% of judge-credited sub-goals are evidence-supported and strong correlation between automated raw coverage and human-verified evidence completion (r = 0.873). We further confirmed open-ended test-time exploration and order-/tool-invariant key_info matching via multi-route sampling (97.1% mean raw coverage on route-divergent correct trajectories).

  3. OmniSeeker Design and Controlled Comparison (uVp7 W3). We clarified OmniSeeker's continuous cross-modal action space over dynamically registered initial, retrieved, and cropped images, and added a controlled same-backbone (GPT-5.2) comparison across all 300 questions against MMSearch-R1, WebWatcher, and Vision-DeepResearch, outperforming the strongest baseline by +5.00 pp SR (36.00% vs. 31.00%).

  4. Diagnostic Actionability and Calibrated Interpretations (uVp7 W2, Q2, Q4; YqV4 W3). We connected our failure taxonomy to the staged oracle and matched single-factor analyses (Appendix A.2) to localize key bottlenecks (identifying the decisive image and extracting usable visual facts) and derive four concrete remedies. We also added a sub-goal-guided search-plan prompting experiment improving SR (+1.67 to +4.67 pp) and PS (+4.26 to +7.55 pp) across all three tested backbones, plus a question-level PS–SR and follow-up solvability analysis. Following YqV4's feedback, we reanalyzed Figure 13 with a fixed cohort (removing the "information dilution" claim while confirming early-step divergence) and removed the "parametric knowledge compensation" attribution.

  5. Task Definitions, Visual Error Diagnosis, and Figure 15 (imYZ W1, W3, W4). We clarified Task Level (L1–L3) vs. Question Difficulty (explicit retrieval-hop cutoffs from human reference trajectories) and the post-hoc trajectory-evidence criteria for diagnosing Visual Grounding Errors without dense live-webpage annotations. We confirmed the Figure 15 bottom-right mismatch is a layout copy-paste error—the released dataset and all metrics are unaffected—and will fix and cross-check the full figure in the revision.


Planned Revisions

We will faithfully incorporate all corrections, clarifications, and new analyses into the revised manuscript, including the updated Table 2 and figures, clarified Process Score protocols in Appendix C.4, fixed-cohort Figure 13, new comparisons and audits, and corrected Figure 15.


Thank you again to the Chairs and Reviewers for your time, thoughtful guidance, and service to the community.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.