acceptodds
Under review as a conference paper at ICLR 2027

HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

Abstract

We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers. Questions are designed to be extremely challenging. Each question targets a concise, publicly verifiable answer whose discovery requires locating obscure evidence, following multi-step clue chains, or inspecting heterogeneous sources such as videos, scanned documents, images, or maps. Easier questions are filtered out by evaluating them with models without internet access to reduce the likelihood that they can be answered with parametric knowledge alone. We evaluate several models using provider-native search and a shared external retrieval harness under a common agent protocol. To contextualize model performance and effort, we also conduct a human evaluation on a sample of the questions. HyperBrowseComp provides a challenging testbed for persistent information seeking across languages and evidence modalities, with difficulty arising from discovering and connecting evidence on the open web.

Primary area:
datasets and benchmarks
Keywords:
multilinguality, multimodality, benchmarks, search agents, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g1QzOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HyperBrowseComp introduces 423 manually authored and human-validated questions across 13 languages, emphasizing multi-hop discovery and integration of heterogeneous web evidence. The authors screen questions using models without internet access, evaluate five models across selected configurations using provider-native search, Exa, and OWL, and conduct a human evaluation. The strongest configuration achieves 31.68% accuracy, with substantial performance differences across retrieval harnesses.

Critical strengths
  1. Substantive value as a data resource. Each question includes answer aliases, sources, precise evidence locations, and a solution trace. These annotations support answer verification and subsequent error analysis, making the resource more useful than a collection of question–answer pairs alone.

  2. Informative system-level evaluation. Changing the retrieval harness substantially affects the same model’s performance. Reporting resource consumption and execution failures alongside accuracy provides useful context for assessing practical browsing systems.

Critical weaknesses

The additional evaluation value over existing benchmarks is insufficiently established.

Relative to multimodal browsing benchmarks such as MM-BrowseComp, the clearest distinction is natively authored multilingual questions. However, the paper does not sufficiently demonstrate what additional capabilities this extension diagnoses. Different languages contain different questions, so the descriptive results cannot disentangle language effects, intrinsic difficulty, and source-access issues. The distinctive challenges arising from combining multilinguality and multimodality also remain unclear. Consequently, I view the contribution primarily as a dataset extension within an established paradigm, whose significance has not yet been sufficiently demonstrated.

Questions for authors

Do the existing annotations or execution traces provide concrete evidence that the original language or local information environment materially changes how agents discover or use multimodal evidence? What capability gaps do these cases reveal that existing benchmarks such as MM-BrowseComp do not adequately cover? Clarifying this could change my assessment of the contribution’s significance.

3 comments

We thank the reviewer for this helpful question. During data construction, we encouraged annotators to write questions grounded in local information environments. Annotators built tasks around regional media, local institutions, and linguistic references, making these elements part of the process of finding evidence. This was a guideline, and annotators could also write broader questions. We see room to make local grounding more consistent.

The examples below show how tasks in the dataset require both finding information in the local language and interpreting multimodal evidence.

Example 1 (Shown in Figure 1)

  • Task: Answer a German question about a bank near a real-world location shown in a German-language \emph{Euro Truck Simulator 2} video. Then identify which of the bank's continuing operations had the second-highest operating profit in its 2010 report.
  • Interaction: The task requires understanding the German audio in the video by a local YouTuber, using a map of Germany to identify the bank, and consulting a UBS Europe branch's company report.

Example 2

  • Task: Answer an Indonesian question about what NPCs shout in Sundanese in an indie game.
  • Interaction: The task first requires identifying the game from a clue in a local Indonesian news source. The game is available worldwide and includes an Easter egg in which an NPC speaks Sundanese, a regional language in Indonesia.

I think this reply had already demonstrated the necessity of the multilingual in multimodal environment because of the cultural background as a inherited underlying hint. I also encourage the Reviewer 4JAT to reconsider the contribution of this paper.

We thank the Reviewer for appreciating our contribution and raising their score

Reviewer_4JATOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A multilingual and multimodal benchmark for deep search agent.

Critical strengths
  1. well-written
  2. 423 manually authored and human-validated questions
Critical weaknesses
  1. the task is not novel now, multimodal multi-hop retrieval domain has been researched by MM-Browsecomp and Brosecomp-V3 and so on. multilingual is also not novel now, https://arxiv.org/abs/2606.15345. from my view, it is just a incremental work.

  2. for a multimodal tasks, the paper just gives texutal tools to the agent, for example, the exa only provides web search/web fetch, the return is texutal, only the OWL contains some visual tools. so maybe the lacks of the tools lead the difficulty.

  3. for the multilingual setting, you should do the comparison experiments. Without paired translations, difficulty matching, or analyses controlling for these confounding factors, the inclusion of 13 languages establishes multilingual coverage but does not support reliable comparisons of model capabilities across languages

  4. The benchmark largely operationalizes deep search as recovering a short answer through deliberately constructed, obscure clue chains. However, longer retrieval paths do not necessarily correspond to deeper user information needs. For example, Figure 1 links a video game playthrough to a real-world bank and then to a historical financial report, without explaining what practical information need motivates this chain. The paper provides no empirical evidence that these tasks reflect actual research workflows, which often require broad evidence collection, comparison, conflict resolution, and synthesis. Its single-answer evaluation favors narrow clue recovery while leaving these capabilities largely unmeasured.

Questions for authors
  1. What practical user information need motivates the chain in Figure 1, from a video game playthrough to a real-world bank and then to a historical financial report? Is there empirical evidence that comparable tasks occur in actual research workflows?

  2. How do you distinguish research depth from retrieval-path length, and validate that adding obscure clues and hops measures meaningful research capabilities?

  3. Does performance on HyperBrowseComp predict performance on realistic tasks requiring broad evidence collection, comparison, conflict resolution, and synthesis? How could the single-answer evaluation capture these capabilities?

6 comments

We are grateful to the reviewer for their valuable questions and concerns and are happy to respond to them and include this clarification in the paper. We invite the Reviewer to discuss weaknesses (W) and questions (Q) in more detail:

W1 On novelty

We would like to disagree with the Reviewer. Calling this work incremental simply because a multilingual benchmark already exists is dismissive of multilingual work. Which languages are covered, how the data are created, and which modalities are included all shape what a benchmark can measure.

As Table 1 shows (including the paper you mentioned), prior multilingual BrowseComp uses machine-translated text, and prior multimodal BrowseComp is monolingual. We wrote and annotated multimodal questions by hand, drawing on local linguistic and cultural knowledge and including languages absent from prior benchmarks. This requires significant effort beyond simpler machine-translated data. We ask the reviewer to address these differences in the novelty assessment.

W2 on Tool Capabilities

  • We agree that tool capabilities can affect performance. To investigate this, we already evaluated OWL, and reported its results alongside the native-search and Exa configurations.
  • We also attempted evaluations with OAgents (https://arxiv.org/abs/2506.15741) and AWorld (https://arxiv.org/abs/2508.20404), but encountered harness issues, including blocked source access and execution times that made the runs impractical to complete, giving very low performance. We will document these attempts and their limitations in the revised paper.
  • For some native search (e.g. gpt 5.6 sol, luna, terra, gemini 3.1 pro, and gemini 3.7 flash), the exposed interface does not give us sufficient visibility to establish which media types the internal tools can access or process. We therefore cannot assume that these configurations have equivalent modality access. Accordingly, our scores characterize the evaluated model–tool configurations. Limited evidence access may contribute to their difficulty, and the results should not be interpreted as isolating models’ multimodal understanding ability. We will make this distinction more explicit in the paper.

W3 on Multilinguality

We agree that differences in questions, difficulty, and source access prevent direct comparisons of language capabilities. We never claim such, and we already state this explicitly in the paper (page 9, lines 436-438):

However, this does not necessarily mean models underperform in these languages; rather, intrinsic question difficulty may vary across language splits.

This is NOT to be used to compare which language is harder vs the others. This is a known, standard trade-off of non-parallel multilingual benchmarks generally.

In any case, we translate our questions to English. We confirm this by running an experiment, showing English results are worse, as the model now does not have an implicit hint on the culture of the question from the language (Gemini 3.7 Flash with built-in Google search on English translations of all 423 questions, the performance dropped from 31.7% →\to 27.4%).

A possible explanation for the quality drop is that an English-speaking solver may understand a translated question while lacking the local context needed to interpret its clues or discover relevant sources. Translating the question alone does not supply that context or equalize source accessibility. Paired translations would be useful for studying the effect of question language, but would not fully control these other differences.

Our assumption is confirmed by the analyses, where questions grouped by how much the original run searched in the local language (The gap is larger where local search mattered)

Original run’s searchesQuestionsOriginal accuracyEnglish accuracyδ\delta
Mostly English (<20% local)11627.6% (32)25.0% (29)−2.6 pts
Partly or mostly local (≥20% local)30733.2% (102)28.3% (87)−4.9 pts (p = 0.049)
Our language results therefore describe performance on each subset, rather than rank language proficiency. We will make this point more prominent in the paper and add additional experiment results.

W4, Q1, and Q3 on practicality

HyperBrowseComp is designed as a stress test of browsing capabilities, as indicated in the title. Its questions deliberately combine difficult-to-find evidence and indirect clues to expose limitations in persistent search. They are not intended to reproduce the frequency or structure of everyday user requests. This follows the evaluation philosophy of the original BrowseComp dataset (https://arxiv.org/pdf/2504.12516). We already state this explicitly in the Limitations section, page 9 line 465:

HYPERBROWSECOMP is intentionally designed as a stress test, thus does not always represent the natural distribution of everyday search queries.

Section 3.1 on page 4 further explains the deliberate construction of challenging questions around obscure evidence and multi-step clue chains. We explicitly describe its coverage as an incomplete but useful assessment of browsing ability and state that generalization to all browsing tasks is not guaranteed (Section 2.4, page 4). Combining comparably challenging search with realistic research workflows, including broad evidence collection, conflict resolution, and synthesis, is an important direction for future work. We will clarify these boundaries once again in the revised paper.

Q2 on Research capabilities

We agree that a longer retrieval path does not automatically imply deeper research. However, details and hints constrain the answer and distinguish it from other plausible answers. Removing them as an experimental ablation can make the question ambiguous or non-solvable at all. Creating the data with additional clues and hops is a whole new task and is out of scope for our study.

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Reviewer_8tcbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a multilingual and multimodal stress test for web-browsing agents. Questions are authored natively across languages, require evidence from webpages and sources such as videos, images, and documents, and undergo independent validation and difficulty screening. The authors compare search and tool configurations and include a human study. The results show that current systems still struggle with questions requiring evidence across sources.

Critical strengths

-Native multilingual authoring is the contribution I find most compelling. It includes local information environments and cultural clues that benchmarks translated from English can easily miss. -The curation goes beyond final answers to include answer aliases, evidence locations, solution paths, and independent validation. This makes the resource useful for distinguishing failure to find evidence from failure to interpret it.

  • Failure handling and grading sensitivity are reported transparently. Terminal failures remain in the full denominator, and answer grading is checked with different judges. These choices make the reported system scores easier to interpret.
Critical weaknesses

1 A non-text source does not establish that non-text understanding is necessary. The current annotations rely mainly on questions and solution descriptions and do not sufficiently exclude captions, summaries, or alternative webpages that supply the answer as text. This distinction needs direct checking before making strong claims about multimodal requirements.

2 Configurations differ in media access, search interfaces, and stopping conditions, so performance gaps are difficult to attribute to a particular capability. The proposed explanation involving co-adaptation to native search does not yet rule out differences in evidence access and tool reliability.

3.Different languages use different questions, so content and difficulty are confounded with language. The human study is also better treated as descriptive context. Neither comparison supports broad language-capability rankings or claims of human-level performance.

Questions for authors
  1. Have non-text-labeled questions been checked for textual alternatives sufficient to answer them? How often do successful trajectories actually use the relevant media?
  2. Can existing failure traces distinguish retrieval, source access, content understanding, and termination problems? Do the main configuration gaps persist on matched questions with equivalent evidence access?
9 comments

We are grateful to the reviewer for their valuable questions and concerns and are happy to respond to them and include this clarification in the paper. We invite the Reviewer to discuss weaknesses (W) and questions (Q) in more detail:

W1 & Q1 On non-text sources

We thank the reviewer for highlighting the distinction between a non-text source and a task that requires non-text understanding. We agree that source format alone is insufficient. During data collection, we require annotators to provide proof to answer with traces of links that are used for assessment. We use LLMs to check whether non-text interpretation is required based on the question, answer, and proof to answer, separately from source-format labels. (page 6, lines 284-286) Across the 423 questions, 272 (64.30%) are annotated as requiring such interpretation. The proportions within each modality are:

ModalityTotal questionsRequiring non-text understandingPercentage
Video / YouTube16513783.03%
PDF / book / OCR1267257.14%
Image787798.72%
Tables433683.72%
Audio4545100.00%
Maps413892.68%
Plots / graphs8787.50%

Answer to Q1: Based on the table above, we can see that, for instance, in 16.97% of Video / YouTube questions, textual information could be enough to answer the question. However, given the statistics above, in the majority of cases, the use the relevant media is required.

We also did an additional check for the Video / YouTube modality: we searched for the ground-truth answer outside of the video, e.g., in the extracted page text, titles, descriptions, and selected metadata. This audit identifies 18 questions (10.91%) with potential textual shortcuts. However, we would like to specify that the answer mentioned does not necessarily supply sufficient context to solve the question.

We will do additional checks on the remaining modalities and will incorporate the findings and their limitations into the paper.

W2 & Q2 On configurations

We thank the reviewer for their valuable observation. We agree that performance differences between configurations cannot be attributed solely to model capabilities. Search coverage, media access, extraction quality, tool reliability, and stopping conditions can all contribute.

To address the issue, we automatically classified incorrect runs:

Failure classification

Failure categoryEstimated percentage
Retrieval57.23%
Source access / extraction7.99%
Content understanding / integration0.25%
Stopping conditions0.13%
Unclear34.39%

We use the saved execution traces: search queries, tool calls, returned content, final answers, and explicit execution errors. ChatGPT judges compare these with the reference answer and author-provided proof to assign categories. The results show that retrieval is the most frequently assigned failure category.

Do the main configuration gaps persist on matched questions with equivalent evidence access?

We compared configurations on the following question sets: all questions, 71 annotated as not requiring non-text understanding, and 272 annotated as requiring it. The remaining 80 have uncertain modality requirements and appear only in the full-benchmark results.

ModelQuestion subsetNative searchExaOWL
Gemini-ProAll42325.53%18.20% —
Gemini-ProText-based7140.85%36.62%
Gemini-ProNon-text27220.96%13.97%
GPT-SolAll42319.15%26.71%
GPT-SolText-based7128.17%43.66%
GPT-SolNon-text27217.65%22.79%
Gemini-FlashAll42331.68%22.22%
Gemini-FlashText-based7152.11%42.25%
Gemini-FlashNon-text27226.47%16.54%

The direction of the native-search versus Exa gap persists across all three sets: native search performs better for both Gemini models, whereas Exa performs better for GPT-Sol. For Gemini-Flash, native search also outperforms OWL in both modality subsets. The matched-question comparison shows that the main configuration trends persist when evaluation is restricted to questions annotated as answerable without non-text understanding. It does not establish equivalent evidence access: textual retrieval coverage, accessibility, extraction quality, and stopping conditions remain uncontrolled. We will include these analyses and limitations in the revised paper and distinguish observed system-level differences from hypotheses about their causes. All in all, based on the experiment above we can say that configuration gaps mostly persists on matched text-based questions

W3 On different languages

We agree that differences in content and intrinsic difficulty prevent interpreting the language-specific results as rankings of language capability. This limitation is already stated explicitly in the submitted paper (page 9, lines 436-438), which is a known, standard trade-off of non-parallel multilingual benchmarks generally:

By language, models fail most frequently on German and Indonesian queries. However, this does not necessarily mean models underperform in these languages; rather, intrinsic question difficulty may vary across language splits. Standardizing difficulty levels across languages remains an important consideration for future benchmark iterations.

The questions were authored natively to preserve local linguistic and cultural references and their associated information environments. Direct translation (e.g., as one of the solutions) may not be able to produce equivalent tasks, because most questions are embedded in local linguistic, cultural, and informational contexts.

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.