acceptodds
Under review as a conference paper at ICLR 2027

CocoonBench: Can Multilingual Search Agents Escape Language Information Cocoons?

Abstract

Multilingual search agents can access the global Web, yet may remain trapped in a language environment with insufficient evidence; we refer to this failure as a language information cocoon. However, existing search benchmarks largely operate within a single language, especially English, overlooking entity connections across multiple languages. We introduce CocoonBench, a benchmark that evaluates whether multilingual search agents can escape such language information cocoons. CocoonBench mines answer-centered relational paths from a frozen Wikidata-derived graph and selects clue sets that are individually necessary and jointly identify a unique answer. Using language-specific entity names, it records each task’s minimum language cover, removes direct identifiers, and renders each task as matched prompts in three languages together with its construction certificate and provenance. Across 10 state-of-the-art models, mean accuracy on CocoonBench ranges from 33.8% (DeepSeek-V4-Flash) to 69.7% (Claude-Opus-4.8), leaving even the strongest system below 70%. The same construction certificates support path-aligned trajectory analysis, enabling diagnosis of model failures in language routing, evidence-path recovery, and answer use. The result shows that escaping a language cocoon requires more than switching languages: successful cross-lingual search depends on both covering the correct languages and connecting the evidence found across them.

Primary area:
datasets and benchmarks
Keywords:
Multilingual NLP, Multilingual Large Language Models, Cross-Lingual Information Retrieval, Cross-Lingual Generalization, Multilingual Evaluation, Search Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sCwEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CocoonBench, a benchmark designed to evaluate whether multilingual search agents can retrieve and integrate complementary evidence distributed across different language environments. The authors identify a failure mode termed the language information cocoon, where agents remain confined to insufficient language-specific information despite having access to the multilingual Web. CocoonBench consists of 200 multi-hop search tasks constructed from a frozen Wikidata-derived knowledge graph, with structural guarantees concerning answer uniqueness, clue necessity, and cross-language lexical requirements. Each task is presented in three prompt languages: English, Chinese, and a task-specific designated language. The authors evaluate ten state-of-the-art language models and introduce a path-aligned failure analysis that distinguishes language-routing failures, evidence-path recovery failures, and failures after answer exposure. Experimental results show that model accuracy ranges from 33.8% to 69.7%, with most classifiable errors occurring during multilingual routing. The findings suggest that successful multilingual search requires not only switching languages but also identifying appropriate language routes and integrating evidence across them.

Critical strengths

S1. The paper addresses an important limitation of existing multilingual search-agent benchmarks by focusing on the need to retrieve complementary information across different languages, rather than merely evaluating performance under different prompt languages.

S2. The proposed generation framework incorporates explicit constraints on answer uniqueness, clue necessity, and cross-language lexical coverage. These constraints provide a systematic foundation for constructing challenging multi-hop search tasks.

S3. The path-aligned audit is a valuable contribution because it goes beyond final-answer accuracy and distinguishes failures in multilingual routing, evidence-path recovery, and answer exposure. This provides useful insights into where multilingual search agents encounter difficulties.

Critical weaknesses

W1. Although each task is evaluated under three prompt conditions, CocoonBench contains only 200 distinct semantic tasks. This relatively small dataset may limit the diversity of evaluated reasoning patterns, entity types, and multilingual search scenarios. Moreover, 85% of the tasks contain only two relational clues, potentially restricting the evaluation of more complex evidence-integration scenarios.

W2. The benchmark defines language requirements using the availability of entity labels and aliases in a frozen lexical inventory. However, the absence of an entity label in a particular language does not necessarily mean that the corresponding information cannot be discovered through that language using alternative descriptions, translated content, or other search strategies. Although the authors perform shortcut auditing, this cannot exhaustively establish that multilingual retrieval is necessary in the open Web.

W3. The related-work section discusses several relevant multilingual search benchmarks, including Cross-lingual BrowseComp-Plus, MARCA, and X-WebAgentBench. However, the experimental evaluation focuses exclusively on CocoonBench. Additional comparisons could better establish how the proposed benchmark differs from existing alternatives in terms of task difficulty, cross-language reasoning requirements, and diagnostic capabilities.

Questions for authors

Q1. The proposed failure analysis relies on predefined language covers, designated bridge entities, and deterministic matching of entity identifiers or surface forms. Consequently, valid alternative reasoning paths may not be fully captured by the classifier. Although sensitivity analyses are provided, the paper does not report a human-annotated evaluation of the correctness of the trajectory-level failure labels.

Q2. The paper provides a detailed diagnosis of multilingual search failures but does not evaluate concrete strategies for improving language routing or cross-language evidence integration. Experiments with explicit language planning, coverage tracking, or branch-aware retrieval would help establish the practical implications of the findings.

Q3. The minimum language cover is formally defined using lexical availability in the frozen knowledge graph. However, the paper sometimes interprets this requirement as evidence that agents must search particular languages to solve a task. These are not necessarily equivalent properties in an open-Web setting. The authors should clarify the extent to which the construction certificate guarantees lexical necessity versus practical retrieval necessity.

Q4. A language is considered entered when an agent issues a query or fetches a page in that language. Consequently, completing a minimum language cover does not necessarily imply that the agent has retrieved useful evidence from those languages. The paper distinguishes language covering from subsequent evidence recovery, but the interpretation of language-cover completion as successful routing could be clarified.

Q5. The main discussion suggests that designated-language prompts help agents progress further through the search process. However, the paired analysis in Table 9 shows relatively small answer-accuracy differences, with confidence intervals including zero. The authors should emphasize that improved process-stage progression does not necessarily translate into improved task-solving accuracy.

Q6. The reported failure distributions are calculated over incorrect, classifiable trajectories rather than all evaluation trajectories. Although this is specified in the paper, the distinction should be emphasized when interpreting statements such as the dominance of language-routing failures. Reporting corresponding proportions over all attempted tasks would provide additional context.

Q7. Evidence-path recovery is assessed using certificate-designated bridge entities. However, an agent may potentially identify the correct answer through a valid alternative evidence chain without encountering these particular entities. The authors should clarify how frequently such alternative paths occur and whether they affect the reliability of the failure analysis.

Q8. The audit identifies strong answer evidence primarily through answer QIDs or exact Wikipedia URLs. However, an answer may be explicitly stated in retrieved text without these identifiers. This creates a potential distinction between the operational definition of answer exposure and the information actually available to an agent. The authors should discuss how this restriction affects the interpretation of T3 and R failures.

8 comments
AuthorsOpenReview ↗

Response to Reviewer sCwE — Part 1/4

Thank you for recognizing the focus on complementary multilingual evidence, the verifiable generation constraints, and the value of process diagnosis. We have added an explicit language-planning intervention, human trajectory verification, and the requested process and population statistics, with the corresponding revisions highlighted in red in the manuscript.

1. Task scale and relational complexity (W1)

CocoonBench contains 200 independent semantic tasks. For each task, we compare English (EN), Chinese (ZH), and a designated language selected from one evidence branch (DL), assessing statistical uncertainty at the task level.

The fact that 85% of tasks have two relational clues reflects a deliberate design choice informed by inspecting generated cases. Adding relational constraints and requiring more languages makes satisfying entity combinations increasingly sparse, producing overly long-tail questions about rare entities and unusual relations. We control this complexity to retain natural, checkable questions whose difficulty centers on multilingual retrieval and evidence integration.

Clue count and language count capture different aspects of complexity. To measure the latter, we find the smallest language sets providing names for all hidden retrieval-key entities in the frozen inventory—the task’s minimum language covers. 39/200 tasks (19.5%) require at least three languages in a minimum cover, and even two clues can require multiple entity-recovery steps, searches, and cross-source checks. Each task also satisfies answer-uniqueness, clue-necessity, and frozen-lexicon coverage constraints. The released construction pipeline supports generating additional tasks and relation combinations; the current set provides a carefully constrained instance of that methodology. Thus, the clue-count distribution reflects controlled task construction, while the language-cover counts show that the set also tests broader multilingual retrieval.

2. Comparison with existing benchmarks (W3; W1)

Our contribution combines a reusable task-generation method, open-Web multilingual search, and diagnosis grounded in per-task construction certificates that record the answer, reference relations, and language covers. We generate questions from frozen knowledge-graph (KG) relations and jointly check answer uniqueness, clue necessity, and language coverage of hidden entity names. The 200 tasks instantiate this process.

BenchmarkTask construction and retrieval environmentMultilingual controlEvaluation and diagnosis
Cross-lingual BrowseComp-Plus (XBCP)Retains BrowseComp-Plus English questions and answers; retrieval uses a fixed evidence corpus and configured retrieverTranslates and redistributes evidence documents; cross-lingual evidence is assigned to one language per task, while the multilingual corpus is distributed across 12 languagesAnswer accuracy, evidence recall, citation quality, calibration, and a condition supplying reference evidence
MARCA52 manually written, multi-entity information-gathering questions on the open WebParallel English/Portuguese questions; compares direct tool use and subagent orchestrationHuman-verified checklists assess entity and attribute correctness and completeness
X-WebAgentBenchInteractive shopping tasks based on WebShopInstructions, products, and interfaces extended to 14 languagesTask scores and interaction performance under cross-lingual alignment strategies
CocoonBenchGenerates new tasks from frozen KG relations; agents search autonomously on the open WebJoint relational and hidden-name coverage constraints; matched EN/ZH/DL questions, with 33 designated languagesExecutable uniqueness and clue-deletion checks; retained relation, bridge, and language-cover certificates connect trajectories to language coverage, evidence recovery, and answer exposure

CocoonBench adds new task generation and in-the-wild multilingual retrieval. XBCP retains BrowseComp-Plus questions and answers, creating its multilingual corpus by translating and reassigning existing documents. Its evidence-language distribution is imposed by construction. CocoonBench generates new questions from KG relations and lets agents discover naturally occurring evidence on the live Web. This tests language selection and relation recovery under real Web conditions, beyond retrieval over a translated, fixed corpus.

These benchmarks also stress different capabilities: controlled corpus retrieval and integration, multi-entity answer completeness, or page interaction. CocoonBench focuses on recovering hidden entities and relations from clues while selecting useful search languages.

AuthorsOpenReview ↗

Response to Reviewer sCwE — Part 2/4

2. (continued)

The added intervention further demonstrates the usefulness of this design: supplying only candidate-language names raises accuracy by 8.5–12.5 percentage points (pp) across three models, making language selection an observable and actionable component of evaluation.

3. A concrete language-planning intervention (Q2; W2, Q3)

We retain the original settings and search budgets and use EN questions in all three conditions. The baseline reruns the original prompt. The oracle language hint supplies candidate evidence languages by adding You may find relevant evidence by searching in the following languages: {languages}. The candidate set contains the language cover, and the agent uses it to formulate searches. Coverage diagnosis uses the original certificate. EN-query-only adds Search instruction: write all search queries in English only.

Each condition evaluates the same 200 semantic tasks using final per-question judgments. Full accuracy confidence intervals (CIs) and paired effects appear in Appendix I, Tables 13–14 (pp. 30–31).

ConditionKimi-K3GLM-5.3-FlashQwen3.8-Max
Baseline55.5%45.0%45.0%
EN-query-only51.5%38.0%44.0%
Oracle language hint64.0%57.5%55.0%

Across the three models, the mean paired effect is −4.0 pp for EN-query-only (95% CI [−7.3, −0.7]) and +10.3 pp for the oracle [6.3, 14.5]. CIs resample the 200 semantic tasks jointly, retaining each task’s outcomes across models and conditions. Candidate-language guidance is therefore a tested way to improve these agents under the original budget, directly answering your request for an actionable strategy.

English-query shares rise from 60.75% to 91.46% for Kimi-K3, 37.32% to 90.15% for GLM-5.3-Flash, 48.69% to 84.48% for Qwen3.8-Max, as measured by the original language-identification (LID) pipeline (Appendix I, Table 15, p. 31). The query-language shift confirms that the prompt intervention changes how agents search, connecting the accuracy comparison to retrieval behavior.

To test whether guidance resolves the diagnosed language bottleneck, we followed baseline runs that answered incorrectly without completing a language cover (wrong-L). Across the three models, 169/180 (93.9%) baseline wrong-L runs complete the original certificate's language cover under the oracle, 44/180 (24.4%) answer correctly, and 41 achieve both (Appendix J, Table 17, p. 32). Another 128 complete coverage but remain incorrect, showing why language coverage and subsequent evidence recovery are separately useful diagnostic stages.

The same-task recoveries connect language planning to successful answers; the remaining errors explain why evidence recovery must still follow language coverage. The construction pipeline can also supply checkable language and relation-completion targets for future supervision or reinforcement-learning approaches, including coverage tracking and branch-aware retrieval.

Example 1: connecting the evidence for Santa Quiteria (Appendix J.3, p. 33). With Catalan and Spanish hints, Qwen identifies Elisa Núñez through Spanish search, obtains her birthplace Elche de la Sierra from Catalan results, and retrieves the Santa Quiteria church and organ-restoration records. It answers correctly, while baseline searches along an Italian-church hypothesis and answers incorrectly. Both runs complete language coverage while leaving some reference evidence checks unmet (T), but the answer changes from wrong to correct: a wrong-T → correct-T transition illustrating the relation-retrieval benefit of language planning.

Example 2: the missing museum relation in the gas-pipeline task (Appendix J.3, p. 33). The question identifies Ivatsevichy and Torzhok through museum clues and asks for a pipeline containing those cities in its name. EN-query-only uses 247/254 English queries (97.24%) and encounters the correct Torzhok–Minsk–Ivatsevichy candidate. However, its English query about the Ivatsevichy museum's origin returns no results; it does not verify the link to a commemorative people's museum and selects Urengoy–Pomary–Uzhhorod.

Baseline and oracle retrieve Russian and Belarusian evidence that the local-history museum developed from the Goishik commemorative museum and administers the Khavanshchyna forest memorial branch. These relations match the question; combined with the Vasilyovo wooden-architecture museum near Torzhok and pipeline-name records, both runs answer correctly. In this case, baseline and oracle finish normally and EN-query-only times out with the wrong answer. In the Santa Quiteria case, both controls time out with wrong answers and the oracle finishes normally. Both cases illustrate how language guidance helps connect the relations needed to verify an answer.

AuthorsOpenReview ↗

Response to Reviewer sCwE — Part 3/4

4. Open-Web validity and human verification (W2, Q1, Q3, Q7)

The open Web changes and can contain conflicting or incorrect claims. Our curated, frozen KG anchors gold answers and reference relations; its certificates provide verifiable reference paths for task construction and process diagnosis. Language-cover constraints are checked against the frozen lexical inventory. Agents search the live Web, where shortcut screening, trajectory audits, and the interventions above test actual retrieval behavior.

We checked the diagnosis across runs with incomplete coverage (L), incomplete reference evidence checks after coverage (T), and all checks passed (R), recording final-answer correctness separately. We audited 546 trajectories, sampling 100 each from correct-L, wrong-L, correct-T, wrong-T, and correct-R, and including all 46 wrong-R runs. GPT-6 Terra produced candidate judgments and rationales; human reviewers examined these together with the corresponding trajectory segments, checking actual support and consistency with the paper's L/T/R definitions. Further review and adjudication produced one final outcome for each trajectory.

The complete confusion matrix is in Appendix K, Table 19 (p. 34).

The agreement rates are 99.0% for L, 75.0% for T, and 85.6% for R; 86.6% overall. All samples remain in the denominator: 473 retain the category, 27 change category, and 46 have insufficient evidence to determine a category after adjudication. This supplies the requested human check of label accuracy, with especially strong support for identifying incomplete coverage.

Among the audited correct-L/T runs, 80.0% answer correctly without full visible support for the required relations (Appendix K, Table 20, p. 34). Most therefore appear to infer or guess the answer from partial clues before completing verification; supported alternative paths explain a smaller share.

For Q7, Appendix K.4 (p. 35) reports that sufficiently supported alternative paths occur in 58/546, and valid paths bypassing a designated bridge in 44/546, with these fields adjudicated per trajectory. These are frequencies within the stratified audit sample. Recording these paths alongside the process labels makes their frequency and supporting evidence explicit, directly addressing how alternative routes affect interpretation of the diagnosis.

5. Language entry, evidence recovery, and exposure (Q4, Q8; Q1)

Cover completion means that the agent has entered the languages of a valid cover through recorded queries or page accesses. Whether it retrieved useful relational evidence is assessed at the evidence-recovery stage. Answer exposure records a further observable event.

To check how answer detection reflects the evidence actually retrieved, we compared ordinary answer mentions with strong-identifier hits matching the answer’s Wikidata entity identifier (QID) or exact Wikipedia URL. In the audit (Appendix K, Table 21, p. 35), 84/546 (15.4%) trajectories contain answer text without a strong-identifier hit, while 229/546 (41.9%) have strong-identifier hits but incomplete support for key relations. The latter occurs in 41/46 wrong-R runs (89.1%).

This explains why a run can complete language, path, and branch coverage yet remain in T3: the strong answer-identifier check is unmet even though the answer may appear in ordinary text. Errors after all exposure checks pass (wrong-R) are residual errors after answer exposure, potentially involving evidence integration, relation verification, candidate selection, or output. The audit therefore supports treating exposure and relation support as distinct observations, answering how identifier-based detection affects the interpretation of these stages.

Reviewer_M2qgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CocoonBench contains 200 multi-hop questions constructed from a frozen Wikidata-derived graph. Each question is rendered in English, Chinese, and a task-specific designated language. The benchmark construction checks answer uniqueness and whether individual clues are necessary, and also derives a minimum set of languages whose recorded labels cover the hidden retrieval entities. Ten Web-search agents are evaluated using the same harness. For incorrect trajectories, a certificate-based audit assigns failures to language-cover failure (L), evidence-path failure (T), or residual failure after strong answer exposure (R). The main claim is that following the required language route is an important bottleneck that is distinct from recovering the evidence path once that route has been reached.

Critical strengths
  1. The benchmark construction is relatively careful. Each task includes executable checks for answer uniqueness and clue necessity, together with a certificate containing the intended paths, bridge entities, and minimum language cover. This provides a useful reference for relating agent trajectories back to the benchmark construction.

  2. The evaluation is well controlled in several respects. Every task is presented in EN, ZH, and DL, and all ten systems are run under the same harness, allowing prompt-language comparisons to be paired at the task level. The paper also reports task-cluster bootstrap intervals and applies a predefined single-language shortcut screen to remove tasks where the answer can already be found from one language alone.

Critical weaknesses
  1. I am not fully convinced that the current trajectory analysis establishes language routing as the main source of failure. Appendix D.1 appears to classify roughly 72% of correct classifiable runs as L based on the rounded values reported there. T also occurs in both incorrect and correct trajectories, covering 32.3% of incorrect runs and 20.5% of correct runs. This makes the interpretation of L and T less direct than the main discussion suggests.

  2. More generally, the minimum language cover is defined with respect to the frozen Wikidata-derived inventory. Appendix C.3 checks N + 1 template queries against search titles, snippets, and URLs, but this does not rule out other valid Web retrieval routes. A successful run may reach the answer through evidence that is not represented by the benchmark certificate and still be labelled as failing the intended language cover. I think the paper needs a closer analysis of successful trajectories that receive an L label before the 65.7% L rate among incorrect runs can be interpreted as evidence that language routing is the dominant cause of failure.

  3. The designated-language intervention is also somewhat entangled with the audit definition. DL is selected from an evidence branch, and Appendix C.2 reports that 80.5% of tasks have a two-language minimum cover. When DL itself belongs to such a cover, entering one additional language may already satisfy the certificate. This could contribute to the decrease in L4 and increase in L3 reported in Table 9, independently of whether the agent actually performs better multilingual retrieval.

  4. Relatedly, the audit counts issuing a query in a language as entering that language, while path hits are defined using designated entities from the benchmark construction. The final answer differences are quite small: −1.7 pp relative to ZH and +0.7 pp relative to EN, with confidence intervals crossing zero over the paired tasks. Given this, I think the process-level shifts are interesting, but it is difficult to tell how much reflects genuinely improved retrieval behavior and how much follows from the way route progress is defined.

Questions for authors
  1. Could you report the direct L/T/R counts separately for correct and incorrect classifiable trajectories, together with answer accuracy conditional on completing the minimum language cover? I would also find it useful to manually inspect a sample of correct trajectories labelled L and explain how those systems obtained the answer despite not satisfying the certificate.

  2. How does the Table 9 analysis change if route progress is defined in a way that does not depend on the prompt mode? In particular, could you stratify the results according to whether DL is itself part of a valid minimum cover? It would also help to validate the certificate paths against alternative evidence routes observed in successful trajectories.

  3. Could you report the number of EXCLUDED and UNDETERMINED runs for each model and prompt mode, together with their answer accuracies? I would also like to know how much of the 33-language set is covered by the translation audit and how the audited languages were selected.

7 comments
AuthorsOpenReview ↗

Response to Reviewer M2qg — Part 1/5

Thank you for highlighting how search progress, question language, and final-answer correctness relate to one another. We have added human verification of successful trajectories, the requested direct counts, a prompt-independent coverage analysis, and matched-task language interventions, highlighting the corresponding additions and clarifications in red in the revised manuscript.

We tested whether adding candidate evidence languages to the original prompt helps agents recover from unsuccessful searches. This oracle language hint improves all three models by 8.5–12.5 percentage points (pp). We tracked incorrect baseline runs that had not completed a minimum language cover (wrong-L), meaning a smallest set of languages providing names for all hidden retrieval-key entities in the frozen inventory. Across the three models, 169/180 (93.9%) baseline wrong-L runs complete the original certificate's language cover under the oracle, 44/180 (24.4%) answer correctly, and 41 achieve both. These matched-task gains connect the process diagnosis to improvements in final answers, directly addressing whether the language bottleneck has practical consequences.

1. Why correct answers can coexist with incomplete retrieval (W1, W2, Q1; Q2)

A stable reference is essential for evaluating open-Web search. The Web changes and can contain conflicting claims; our curated, frozen knowledge graph (KG) anchors the gold answers and reference relations. Its construction certificates retain each task’s answer, reference relations, and language covers, providing verifiable reference paths for task generation and process diagnosis. We inspect retrieved evidence against this explicit reference.

We inspected 100 runs that answered correctly without completing language coverage (correct-L), and 100 that answered correctly after coverage but before all reference-path, branch-coverage, and answer-exposure checks were met (correct-T). Human reviewers examined GPT-6 Terra's proposed judgments and rationales together with the corresponding trajectory segments. The dominant finding is that these runs answer correctly before their visible evidence fully supports the required relations.

The per-category evidence-support counts are in Appendix K, Table 20 (p. 34).

Partial support is the dominant pattern: 80.0% of the inspected correct-L/T runs give the right answer without fully verifying the required relations, consistent with inferring or guessing the answer from the available clues. This explains why a correct final answer can coexist with incomplete retrieval. A smaller share, 18.5%, is explained by sufficiently supported alternative paths. Both percentages use the full 200-trajectory denominator.

For example, one correct-L trajectory identifies Reni urban hromada through a painter, his birthplace, and the administrative relation, but does not identify the second person in the question. The model explicitly acknowledges this missing identification in its answer. It enters Romanian and English but not the certificate's required Russian, and nevertheless answers correctly from the partial evidence (sample_0182, events 19–26). The less common alternative-path pattern is illustrated by direct verification of Mir's current administrative affiliation through another supported route.

The dominant pattern explains why a correct answer can precede complete verification. L/T describe progress relative to the construction certificate; final-answer correctness is a separate outcome. Reporting actual evidence support alongside both distinguishes inference from partial clues from the less common successes supported by alternative paths.

AuthorsOpenReview ↗

Response to Reviewer M2qg — Part 2/5

2. Direct counts, conditional accuracy, and intervention evidence (W1, W2, Q1)

Our conclusion is that language routing is a major bottleneck and the most frequent certificate-relative diagnostic stage among incorrect, classifiable runs. Extending the same process checks to the original ten-model cohort, we also count runs that pass all evidence checks, including answer exposure (R). The direct L/T/R counts are 2,271/645/226 for correct runs and 1,565/770/46 for incorrect runs. The latter constitute 65.73%/32.34%/1.93% of incorrect, classifiable runs. Appendix L, Table 25 (p. 37) reports the counts and both denominators. The counts locate the most frequent unmet process stage and separately show how often each stage coexists with a correct answer.

Answer accuracy conditional on completing a minimum language cover is 871/1,687 = 51.63%, with a 95% confidence interval (CI) [44.95%, 58.30%], obtained by resampling whole tasks and retaining their model outcomes. Without completed coverage, it is 2,271/3,836 = 59.20%, CI [53.79%, 64.54%]. The difference is −7.57 pp, CI [−14.56, −0.62]. Coverage completion alone does not ensure a correct answer. These are behaviorally selected groups: difficulty, search length, and early answering can also affect their composition. The 65.73% figure therefore characterizes the diagnostic distribution. We test the practical role of language selection through interventions on the same tasks and models with matched budgets.

All new conditions use English (EN) questions and the original settings. EN-query-only appends Search instruction: write all search queries in English only. The oracle appends You may find relevant evidence by searching in the following languages: {languages}. The candidate set contains the language cover. Process classification uses the original certificate.

Each condition evaluates the same 200 semantic tasks using final per-question judgments. Full accuracy confidence intervals (CIs) and paired effects appear in Appendix I, Tables 13–14 (pp. 30–31).

ConditionKimi-K3GLM-5.3-FlashQwen3.8-Max
Baseline55.5%45.0%45.0%
EN-query-only51.5%38.0%44.0%
Oracle language hint64.0%57.5%55.0%

Across the three models, the mean paired effect is −4.0 pp for EN-query-only (95% CI [−7.3, −0.7]) and +10.3 pp for the oracle [6.3, 14.5]. CIs resample the 200 semantic tasks jointly, retaining each task’s outcomes across models and conditions. The interventions directly answer the causal question raised by the descriptive counts: changing search-language guidance changes accuracy on the same tasks, with gains under guidance and losses under English-only instructions.

The language hint improves accuracy by 8.5, 12.5, and 10.0 pp; the English-query instruction decreases it by 4.0, 7.0, and 1.0 pp. English-query shares rise from 60.75% to 91.46% for Kimi-K3, 37.32% to 90.15% for GLM-5.3-Flash, 48.69% to 84.48% for Qwen3.8-Max, as measured by the original language-identification (LID) pipeline (Appendix I, Table 15, p. 31). The query shares confirm that the prompts change search behavior, linking the accuracy effects to an observed change in language use.

AuthorsOpenReview ↗

Response to Reviewer M2qg — Part 3/5

2. Same-task wrong-L recovery and case studies (continued)

Can the original wrong-L runs improve? We applied the original L/T/R classifier and frozen certificates to all 1,800 trajectories across three models and three conditions, pairing by semantic task. Each model's baseline wrong-L subset is fixed before comparing oracle outcomes.

Across the three models, 169/180 (93.9%) baseline wrong-L runs complete coverage under the oracle, 44/180 (24.4%) answer correctly, and 41/180 (22.8%) achieve both. The model-wise results and complete stage counts are in Appendix J, Tables 17–18 (p. 32). This shows that guidance can move the original wrong-L runs past the language gate and recover correct answers on a subset of those same runs.

Coverage, correctness, and their intersection are reported separately; all 180 baseline wrong-L runs remain in the denominator. The pooled total counts model–task runs, with the same semantic task potentially represented under different models. Appendix J.1 reports the complete outcome breakdown.

Of the 90 model–task runs changed from incorrect to correct by the oracle, 44 (48.9%) were baseline wrong-L. There are also 28 correct-to-incorrect changes, giving a net gain of 62/600. On the same fixed baseline wrong-L subsets, EN-query-only answers 3/36 Kimi, 3/69 GLM, and 9/75 Qwen runs correctly (15/180, 8.3% in total), compared with 44/180 (24.4%) under the oracle. The recoveries show that baseline wrong-L runs contribute substantially to the oracle’s successful corrections, addressing whether the frequent L label identifies a useful intervention target. The 128 runs that complete coverage but remain incorrect show that relation recovery and integration remain necessary after reaching the required languages.

Example: Pantin, baseline wrong-L → oracle correct-T (Appendix J.3, p. 33). Both controls encounter Pantin as a candidate but do not establish its connection to the judicial official described in the question in the inspected tool returns. With French, Russian, and Ukrainian hints, Russian searches recover Myakinin's career and burial at Pantin, while French searches verify the cemetery name and tree-named avenues. Qwen completes the original language cover and answers correctly (rare100-graph075; Russian evidence at events 28, 30, and 32, French evidence at 31 and 103). Both controls time out with incorrect answers; the oracle finishes normally. The case makes the stage transition concrete: language guidance enables the missing relation checks and changes the final answer.

Reverse intervention: the gas-pipeline task. The question uses museum clues to identify Ivatsevichy and Torzhok, then asks for a pipeline containing these cities in its name. EN-query-only uses 247/254 English queries (97.24%) and encounters the correct Torzhok–Minsk–Ivatsevichy candidate, but its English query about the Ivatsevichy museum's origin returns no results. It fails to verify the commemorative-museum relation and selects Urengoy–Pomary–Uzhhorod. Baseline and oracle obtain Russian and Belarusian evidence linking the local-history museum to the Goishik commemorative museum and the Khavanshchyna forest memorial branch. Together with the Vasilyovo museum near Torzhok and pipeline-name records, both answer correctly. The multilingual runs finish normally; EN-query-only times out with the wrong answer. This illustrates the relation-retrieval mechanism underlying the aggregate comparison.

Reviewer_o552OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CocoonBench to evaluate whether multilingual search agents can escape “language information cocoons.” The benchmark contains 200 cross-lingual multi-hop tasks, with knowledge-graph constraints ensuring answer uniqueness, clue necessity, and cross-language lexical requirements under a frozen lexical inventory. Each task is presented in three matched prompt-language versions. The paper evaluates 10 models and uses construction certificates to categorize failures into language covering, evidence-path recovery, and residual failures after answer exposure.

Critical strengths
  1. Task construction incorporates explicit, verifiable constraints. Answer uniqueness and clue-deletion tests distinguish the benchmark from a simple collection of multilingual questions. Retaining construction certificates also supports auditing and reproducibility.

  2. Process-level diagnosis is more informative than accuracy alone. The paper distinguishes failures to reach the required languages, recover complete evidence paths, and use answer evidence already encountered, providing a useful perspective on multilingual search bottlenecks.

Critical weaknesses
  1. There is a gap between the guarantee of cross-language necessity and the actual evaluation environment. Equation (3) establishes that at least two languages are needed to cover the names of designated hidden entities within a frozen lexical inventory. However, evaluation uses open-web search, where these entities may still be found through pages in other languages, translations, transliterations, or alternative relations. Although the paper screens for single-language shortcuts using fixed queries, this only rules out the tested queries and does not sufficiently establish that cross-language search is necessary on the open Web. This affects whether the benchmark measures escape from language information cocoons or general difficulty in multi-hop search.

  2. Failure diagnosis may rely too heavily on predefined evidence routes. A construction certificate records a valid evidence route, but agents may find answers through alternative entities, sources, or relations. The absence of a designated bridge entity does not necessarily mean that the corresponding evidence was not recovered. Similarly, the appearance of an answer QID or Wikipedia URL does not necessarily indicate sufficient supporting evidence; their absence does not imply that answer evidence is unavailable. Validation against human annotations is needed to establish the accuracy of automatic classification and its ability to accommodate alternative paths.

  3. The benchmark’s scale and the instability of open-web search limit the reliability of the conclusions. The 200 semantic tasks yield 600 cases across three prompt-language conditions, but these are not 600 independent tasks. The DL condition covers 33 languages, potentially leaving few examples per language or language pair and limiting fine-grained generalization claims. Open-web search results may also change over time, further limiting the reliability of the conclusions.

Questions for authors
  1. Under matched search budgets, how does single-language-only search compare with unrestricted multilingual search? Does this confirm that cross-language search is necessary?

  2. How accurate is the automatic failure classification against human annotations, particularly for alternative evidence paths and answer evidence without QIDs or Wikipedia URLs?

  3. How stable are model rankings across repeated runs and search dates? Please report task-level confidence intervals and sample counts per language and language pair.

5 comments
AuthorsOpenReview ↗

Response to Reviewer o552 — Part 1/3

Thank you for highlighting the verifiable construction constraints and the value of process-level diagnosis. We address your three main concerns with matched-budget language interventions, human trajectory verification, and a contemporaneous rerun, now incorporated into the revised manuscript with the corresponding changes highlighted in red.

1. Frozen knowledge-graph guarantees and open-Web retrieval (W1)

The open Web cannot be exhaustively enumerated: pages and rankings change, and retrieved claims can be conflicting or incorrect. Evaluation needs a fixed, verifiable standard. We therefore use a curated, frozen knowledge graph (KG) to ground the gold answers and reference relations. This provides a stable basis for judging answers across agents and evaluation runs.

To make this reference inspectable, we retain each task’s answer and reference relations in a construction certificate. It also records the smallest language sets that provide names for all hidden retrieval-key entities in the frozen inventory—the valid minimum language covers. Together with executable checks of answer uniqueness and clue necessity, these records give task construction and process diagnosis a verifiable basis.

The frozen lexical inventory makes the language-cover constraints checkable. We then evaluate agents on the live Web, screen for direct monolingual shortcuts, and inspect the evidence they retrieve. The new interventions below directly test the value of language selection in this open-Web setting.

2. Matched-budget language interventions (Q1, W1)

All three conditions use the same English (EN) questions, search budgets, models, and original evaluation settings. The baseline is rerun without an intervention. EN-query-only adds an English-query instruction. The oracle language hint supplies the task's candidate evidence languages.

Each condition evaluates the same 200 semantic tasks using final per-question judgments. Full accuracy confidence intervals (CIs) and paired effects appear in Appendix I, Tables 13–14 (pp. 30–31).

ConditionKimi-K3GLM-5.3-FlashQwen3.8-Max
Baseline55.5%45.0%45.0%
EN-query-only51.5%38.0%44.0%
Oracle language hint64.0%57.5%55.0%

Across the three models, the mean paired effect is −4.0 percentage points (pp) for EN-query-only (95% CI [−7.3, −0.7]) and +10.3 pp for the oracle [6.3, 14.5]. CIs resample the 200 semantic tasks jointly, retaining each task’s outcomes across models and conditions.

Restricting searches to English lowers accuracy, while candidate-language guidance raises it across all three models. This directly addresses whether multilingual search matters on the actual open Web: language selection changes task-solving performance under matched budgets.

AuthorsOpenReview ↗

Response to Reviewer o552 — Part 2/3

2. Search instructions, process recovery, and case studies (continued)

The exact EN-query-only instruction is:

Search instruction: write all search queries in English only.

The oracle template is:

{question}

You may find relevant evidence by searching in the following languages: {languages}.

Here, {languages} specifies a language cover for the task, using full language names. The agent uses these languages to formulate searches. Process classification continues to use the original certificate's minimum covers.

The instruction changes actual search behavior. We applied the original language-identification (LID) pipeline to every issued nonempty WebSearch query.

EN-query-only English-query shares are 91.46%, 90.15%, and 84.48% for Kimi, GLM, and Qwen, respectively, compared with 60.75%, 37.32%, and 48.69% at baseline. Full counts and oracle shares are in Appendix I, Table 15 (p. 31).

These shares pool all issued nonempty queries, retaining unidentified queries in the denominator and using the same analysis pipeline across conditions. The observed shift confirms that the English-query prompt changes actual retrieval behavior, making it an informative test of the value of multilingual search.

The same-task comparison also shows recovery from incomplete coverage. We followed baseline runs that answered incorrectly without completing a valid language cover (wrong-L). Across the three models, 169/180 (93.9%) baseline wrong-L runs complete the original certificate's language cover under the oracle, 44/180 (24.4%) answer correctly, and 41 achieve both (Appendix J, Table 17, p. 32). Together with the three-model accuracy gains, this provides behavioral and outcome evidence that language routing is a major, improvable bottleneck.

Concrete example: verifying a gas-pipeline candidate (Appendix J.3, p. 33). The question first identifies two cities through museum clues, then asks for a cross-border gas pipeline whose name contains those cities and a third city. One clue describes a local-history museum originating from a commemorative people's museum and administering a wartime site in a forest. The other describes a wooden-architecture museum on a former estate, with relocated buildings and an arch bridge built without mortar. These identify Ivatsevichy and Torzhok; the answer is Torzhok–Minsk–Ivatsevichy.

Under EN-query-only, 247/254 queries are identified as English (97.24%). The agent encounters the correct pipeline candidate, but its query Ivatsevichi historical museum of local lore Belarus "people's museum named after" returns no results. It does not verify the museum-origin relation and ultimately selects Urengoy–Pomary–Uzhhorod, acknowledging that the museum details remain unverified.

Both baseline and oracle obtain Russian and Belarusian evidence that the Ivatsevichy museum developed from a museum commemorating Goishik and administers the Khavanshchyna forest memorial branch. Combining these relations with the Vasilyovo museum near Torzhok and the pipeline-name records, both answer correctly. In the recorded trajectories, the English-only museum query is event 170; baseline events 53–54 and oracle events 26 and 29 contain the relevant museum evidence. Baseline and oracle finish normally, whereas EN-query-only times out and returns the wrong answer. The case illustrates how multilingual retrieval supplies relations needed to verify a candidate.

Concrete example: connecting a scholar to Santa Quiteria (Appendix J.3, p. 33). With Catalan and Spanish language hints, Qwen identifies Elisa Núñez through Spanish search, recovers her birthplace, Elche de la Sierra, from Catalan results, and finds the Santa Quiteria church and organ-restoration records. It answers correctly, while baseline follows an Italian-church hypothesis and answers incorrectly. This task (rare100-graph037) changes from baseline wrong-T to oracle correct-T. In both runs, language coverage is complete but some reference evidence checks remain unmet (T); the prefixes record the change from an incorrect to a correct answer. This illustrates improved relation retrieval after language selection. The oracle finishes normally; both controls time out and return incorrect answers.

AuthorsOpenReview ↗

Response to Reviewer o552 — Part 3/3

3. Human verification of automatic labels (W2, Q2)

To check the labels across the full process, we included runs with incomplete language coverage (L), incomplete reference evidence checks after coverage (T), and all checks passed (R), recording answer correctness separately. We conducted a stratified audit of 546 trajectories: 100 each from correct-L, wrong-L, correct-T, wrong-T, and correct-R, plus all 46 wrong-R runs. GPT-6 Terra first produced a proposed judgment and rationale. Human reviewers then read these and inspected the corresponding queries, tool returns, and answer segments to verify the judgment against the paper's L/T/R definitions. Further review resolved disagreements into one final adjudication per trajectory.

The complete confusion matrix is in Appendix K, Table 19 (p. 34).

Agreement is 99.0% for L, 75.0% for T, and 85.6% for R, and 473/546 = 86.6% overall. All sampled trajectories remain in the denominator: 27 receive another category and 46 are adjudicated as having insufficient evidence for a category. This provides the requested human validation of automatic labels, with particularly strong agreement on incomplete language coverage.

To address answers appearing without identifiers, we checked retrieved text alongside the classifier’s strong-identifier test, which matches the answer’s Wikidata entity identifier (QID) or exact Wikipedia URL. The evidence audit (Appendix K, Table 21, p. 35) additionally finds answer text without a strong-identifier hit in 84/546 (15.4%) trajectories, and strong-identifier hits with incomplete support for key relations in 229/546 (41.9%). Percentages use all 546 reviewed trajectories; Table 21 gives the complete counts. These findings distinguish answer visibility from relational support and directly address your concern about answers appearing without identifiers: the audit checks the retrieved text as well as identifier matches.

For example, a trajectory that completed language, path, and branch coverage but missed the answer-identifier test (T3) nevertheless contains Minsk in both medical-institution and school-history returns and answers correctly, but neither the configured answer QID Q2280 nor an exact Minsk Wikipedia URL is matched. The text-exposure check identifies this distinction between the operational identifier test and the information available to the agent.

Among the audited correct-L/T runs, 80.0% answer correctly without full visible support for the required relations (Appendix K, Table 20, p. 34). Most therefore appear to infer or guess the answer from partial clues before completing verification; supported alternative paths explain a smaller share.

We also inspect actual retrieval paths: 58/546 trajectories contain a sufficiently supported alternative path, and 44/546 bypass a designated bridge. These frequencies describe the stratified audit sample. These checks directly address alternative evidence paths: we record whether the retrieved relations support an answer even when the agent does not follow the certificate’s designated bridge.

4. Rerun stability, independent tasks, and language counts (W3, Q3)

We reran three models under the original EN setting. Kimi changes from 55.1% to 55.5%, GLM from 48.0% to 45.0%, and Qwen from 48.0% to 45.0%: all differences are within 3 pp, and the ordering remains Kimi > GLM = Qwen. Appendix I, Table 16 (p. 31) reports the comparison, with rerun task-level CIs in Table 13 (p. 30). The similar scores and preserved ranking support consistency across these evaluation occasions, addressing the concern about time sensitivity while providing contemporaneous intervention controls.

We assess uncertainty over semantic tasks (n = 200), keeping each task’s English (EN), Chinese (ZH), and designated-language (DL) renderings together. The designated language is selected from one evidence branch, so these renderings remain paired versions of the same task. All new CIs resample tasks while retaining paired outcomes. They quantify task-sampling uncertainty; the original-versus-rerun comparison provides the evidence across evaluation occasions.

Appendix C.2 and Table 3 (pp. 15–16) report all 33 designated languages and their task shares, from which counts follow using n = 200. Appendix L.5 (p. 37) and Table 27 (pp. 39–40) now report all 132 canonical language combinations across 200 tasks: 94 two-language combinations spanning 161 tasks and 38 combinations of three or more languages spanning 39 tasks. Each task is counted once using its first configured minimum cover. The counts expose the actual support for each language and combination, addressing uneven language representation without counting translated versions as independent tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common response

We thank the reviewers for recognizing the value of CocoonBench's verifiable task construction and process-level diagnosis. We have added language interventions on three models, human verification of 546 trajectories, a contemporaneous baseline rerun, and the requested process and sample statistics, with the additions and revisions highlighted in red in the revised manuscript.

The new experiments test whether choosing useful search languages—language routing—improves task solving:

  • Consistent intervention effects across three models. Under English questions and the original experimental settings, instructing agents to use English-only queries (EN-query-only) reduces accuracy by 4.0, 7.0, and 1.0 percentage points (pp) for Kimi-K3, GLM-5.3-Flash, and Qwen3.8-Max. Providing the task's candidate evidence languages as a search hint (the oracle condition) increases accuracy by 8.5, 12.5, and 10.0 pp, respectively. The three-model mean paired effects are −4.0 pp (95% confidence interval (CI): [−7.3, −0.7]) for EN-query-only and +10.3 pp [6.3, 14.5] for the oracle. Search-language choice therefore has a measurable effect on task accuracy, and useful language guidance improves all three models.
  • Progress and answer recovery on the same tasks. We tracked initially incorrect runs whose retrieval had not covered a valid smallest language set providing names for all hidden retrieval-key entities in the frozen inventory (the task’s minimum language cover). These original-prompt runs form the baseline wrong-L group. Across the three models, 169/180 (93.9%) baseline wrong-L runs complete the original certificate's language cover under the oracle, 44/180 (24.4%) answer correctly, and 41 achieve both. The same initially failed runs show both coverage progress and answer recovery, linking the diagnosed bottleneck to a practical improvement.
  • Human verification of the diagnosis. In the 546-trajectory stratified audit, agreement between automatic and adjudicated labels is 86.6% overall and 99.0% for the incomplete-cover category L. We also inspect answer exposure and support for the required relations. Human review thus supports the operational diagnosis, especially the identification of incomplete language coverage.
  • Consistency with the original evaluation. The three rerun baseline accuracies differ from the submitted English-question (EN) results by at most 3 pp, with the same relative ordering. The rerun reproduces the broad performance pattern and provides a contemporaneous baseline for the interventions.

Together, these results support language routing as a major, improvable bottleneck in CocoonBench: guiding agents toward relevant languages advances retrieval and improves final answers, while restricting search to English reduces accuracy. Human verification supports the reliability of the process labels, and the baseline reruns show consistency with the original evaluation. CocoonBench thus identifies a concrete weakness that targeted search guidance can improve.

The open Web changes and can contain conflicting or incorrect information. We therefore ground evaluation in a curated, frozen knowledge graph (KG), providing stable gold answers and verifiable reference relations. Each task’s construction certificate records its answer, reference relations, and language covers, making task generation and process diagnosis inspectable while agents search the live Web. The interventions and trajectory audits test how language selection affects retrieval and final answers. We address each review below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.