- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
HyperBrowseComp introduces 423 manually authored and human-validated questions across 13 languages, emphasizing multi-hop discovery and integration of heterogeneous web evidence. The authors screen questions using models without internet access, evaluate five models across selected configurations using provider-native search, Exa, and OWL, and conduct a human evaluation. The strongest configuration achieves 31.68% accuracy, with substantial performance differences across retrieval harnesses.
Critical strengths
-
Substantive value as a data resource. Each question includes answer aliases, sources, precise evidence locations, and a solution trace. These annotations support answer verification and subsequent error analysis, making the resource more useful than a collection of question–answer pairs alone.
-
Informative system-level evaluation. Changing the retrieval harness substantially affects the same model’s performance. Reporting resource consumption and execution failures alongside accuracy provides useful context for assessing practical browsing systems.
Critical weaknesses
The additional evaluation value over existing benchmarks is insufficiently established.
Relative to multimodal browsing benchmarks such as MM-BrowseComp, the clearest distinction is natively authored multilingual questions. However, the paper does not sufficiently demonstrate what additional capabilities this extension diagnoses. Different languages contain different questions, so the descriptive results cannot disentangle language effects, intrinsic difficulty, and source-access issues. The distinctive challenges arising from combining multilinguality and multimodality also remain unclear. Consequently, I view the contribution primarily as a dataset extension within an established paradigm, whose significance has not yet been sufficiently demonstrated.
Questions for authors
Do the existing annotations or execution traces provide concrete evidence that the original language or local information environment materially changes how agents discover or use multimodal evidence? What capability gaps do these cases reveal that existing benchmarks such as MM-BrowseComp do not adequately cover? Clarifying this could change my assessment of the contribution’s significance.
3 comments
I think this reply had already demonstrated the necessity of the multilingual in multimodal environment because of the cultural background as a inherited underlying hint. I also encourage the Reviewer 4JAT to reconsider the contribution of this paper.
We thank the Reviewer for appreciating our contribution and raising their score
We thank the reviewer for this helpful question. During data construction, we encouraged annotators to write questions grounded in local information environments. Annotators built tasks around regional media, local institutions, and linguistic references, making these elements part of the process of finding evidence. This was a guideline, and annotators could also write broader questions. We see room to make local grounding more consistent.
The examples below show how tasks in the dataset require both finding information in the local language and interpreting multimodal evidence.
Example 1 (Shown in Figure 1)
Example 2