- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces BrowseComp-V3, a benchmark of 300 manually constructed questions and 383 images for evaluating multimodal web-browsing agents. Its questions span five broad domains and require multi-step search and reasoning across textual and visual evidence. In addition to final answers, the dataset provides annotated evidence chains and intermediate subgoals intended to measure an agent’s progress. The authors also introduce OmniSeeker, a model-agnostic browsing framework with text search, image search, reverse-image search, webpage access, and image cropping. They evaluate humans, proprietary research systems, tool-free models, open-source agents, and OmniSeeker-based agents using final-answer success rate and a subgoal-based Process Score.
Critical strengths
Evaluating the capabilities of browser search in multimodal scenarios is both interesting and important.
Critical weaknesses
- Fig.2 right table shows the task level and difficulty distribution. How are these two metrics defined? Especially for question difficulty? In Fig 15, one example in the right column "When was this photo taken?" Why is this sample defined as an easy case?
- The proposed dataset includes a process-level evaluation. For each question, is this process predefined and unique? Could there be multiple search paths to arrive at the correct answer? Is it possible for a search path to differ from the annotated process in the dataset, yet still yield the correct final result?
- In Fig. 5, Visual Grounding Error is the main failure mode for multiple models. How to assess whether an error is caused by incorrect visual grounding? Is it necessary to annotate grounding on browser pages retrieved through real-time search?
- typo: In Fig 15, bottom-right example, the second image does not align with the given question.
Questions for authors
Please see the weakness section
3 comments
Kind Reminder to Reviewer imYZ
Dear Reviewer imYZ,
We understand that this is a busy period, and we sincerely appreciate your time. As the discussion period will end shortly, we wanted to kindly check whether our response has addressed your concerns. If any questions remain, we would be glad to clarify them before the discussion closes.
Best regards,
The Authors
Dear Reviewer imYZ,
We apologize for following up again. As the discussion period will close in a few hours, we would be very grateful if you could briefly let us know whether our response has addressed your concerns. If anything remains unclear, we are happy to respond right away.
Thank you again for your time and review.
Best regards,
The Authors
Follow-up to Reviewer imYZ
Dear Reviewer imYZ,
We sincerely appreciate your careful review and helpful suggestions. We have carefully addressed the concerns you raised, including the definitions of task level and question difficulty (W1), whether the annotated process is unique and path-dependent (W2), how Visual Grounding Errors are assessed (W3), and the image–question mismatch in Figure 15 (W4). We have also outlined the corresponding revisions we will make to the manuscript.
As the discussion period is drawing to a close, we would be grateful if you could take another look at our response and let us know whether it adequately addresses your concerns. We would also be happy to clarify any remaining points.
Thank you again for your time and consideration.
Best regards,
The Authors