- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CocoonBench, a benchmark designed to evaluate whether multilingual search agents can retrieve and integrate complementary evidence distributed across different language environments. The authors identify a failure mode termed the language information cocoon, where agents remain confined to insufficient language-specific information despite having access to the multilingual Web. CocoonBench consists of 200 multi-hop search tasks constructed from a frozen Wikidata-derived knowledge graph, with structural guarantees concerning answer uniqueness, clue necessity, and cross-language lexical requirements. Each task is presented in three prompt languages: English, Chinese, and a task-specific designated language. The authors evaluate ten state-of-the-art language models and introduce a path-aligned failure analysis that distinguishes language-routing failures, evidence-path recovery failures, and failures after answer exposure. Experimental results show that model accuracy ranges from 33.8% to 69.7%, with most classifiable errors occurring during multilingual routing. The findings suggest that successful multilingual search requires not only switching languages but also identifying appropriate language routes and integrating evidence across them.
Critical strengths
S1. The paper addresses an important limitation of existing multilingual search-agent benchmarks by focusing on the need to retrieve complementary information across different languages, rather than merely evaluating performance under different prompt languages.
S2. The proposed generation framework incorporates explicit constraints on answer uniqueness, clue necessity, and cross-language lexical coverage. These constraints provide a systematic foundation for constructing challenging multi-hop search tasks.
S3. The path-aligned audit is a valuable contribution because it goes beyond final-answer accuracy and distinguishes failures in multilingual routing, evidence-path recovery, and answer exposure. This provides useful insights into where multilingual search agents encounter difficulties.
Critical weaknesses
W1. Although each task is evaluated under three prompt conditions, CocoonBench contains only 200 distinct semantic tasks. This relatively small dataset may limit the diversity of evaluated reasoning patterns, entity types, and multilingual search scenarios. Moreover, 85% of the tasks contain only two relational clues, potentially restricting the evaluation of more complex evidence-integration scenarios.
W2. The benchmark defines language requirements using the availability of entity labels and aliases in a frozen lexical inventory. However, the absence of an entity label in a particular language does not necessarily mean that the corresponding information cannot be discovered through that language using alternative descriptions, translated content, or other search strategies. Although the authors perform shortcut auditing, this cannot exhaustively establish that multilingual retrieval is necessary in the open Web.
W3. The related-work section discusses several relevant multilingual search benchmarks, including Cross-lingual BrowseComp-Plus, MARCA, and X-WebAgentBench. However, the experimental evaluation focuses exclusively on CocoonBench. Additional comparisons could better establish how the proposed benchmark differs from existing alternatives in terms of task difficulty, cross-language reasoning requirements, and diagnostic capabilities.
Questions for authors
Q1. The proposed failure analysis relies on predefined language covers, designated bridge entities, and deterministic matching of entity identifiers or surface forms. Consequently, valid alternative reasoning paths may not be fully captured by the classifier. Although sensitivity analyses are provided, the paper does not report a human-annotated evaluation of the correctness of the trajectory-level failure labels.
Q2. The paper provides a detailed diagnosis of multilingual search failures but does not evaluate concrete strategies for improving language routing or cross-language evidence integration. Experiments with explicit language planning, coverage tracking, or branch-aware retrieval would help establish the practical implications of the findings.
Q3. The minimum language cover is formally defined using lexical availability in the frozen knowledge graph. However, the paper sometimes interprets this requirement as evidence that agents must search particular languages to solve a task. These are not necessarily equivalent properties in an open-Web setting. The authors should clarify the extent to which the construction certificate guarantees lexical necessity versus practical retrieval necessity.
Q4. A language is considered entered when an agent issues a query or fetches a page in that language. Consequently, completing a minimum language cover does not necessarily imply that the agent has retrieved useful evidence from those languages. The paper distinguishes language covering from subsequent evidence recovery, but the interpretation of language-cover completion as successful routing could be clarified.
Q5. The main discussion suggests that designated-language prompts help agents progress further through the search process. However, the paired analysis in Table 9 shows relatively small answer-accuracy differences, with confidence intervals including zero. The authors should emphasize that improved process-stage progression does not necessarily translate into improved task-solving accuracy.
Q6. The reported failure distributions are calculated over incorrect, classifiable trajectories rather than all evaluation trajectories. Although this is specified in the paper, the distinction should be emphasized when interpreting statements such as the dominance of language-routing failures. Reporting corresponding proportions over all attempted tasks would provide additional context.
Q7. Evidence-path recovery is assessed using certificate-designated bridge entities. However, an agent may potentially identify the correct answer through a valid alternative evidence chain without encountering these particular entities. The authors should clarify how frequently such alternative paths occur and whether they affect the reliability of the failure analysis.
Q8. The audit identifies strong answer evidence primarily through answer QIDs or exact Wikipedia URLs. However, an answer may be explicitly stated in retrieved text without these identifiers. This creates a potential distinction between the operational definition of answer exposure and the information actually available to an agent. The authors should discuss how this restriction affects the interpretation of T3 and R failures.
8 comments
Response to Reviewer sCwE — Part 2/4
2. (continued)
The added intervention further demonstrates the usefulness of this design: supplying only candidate-language names raises accuracy by 8.5–12.5 percentage points (pp) across three models, making language selection an observable and actionable component of evaluation.
3. A concrete language-planning intervention (Q2; W2, Q3)
We retain the original settings and search budgets and use EN questions in all three conditions. The baseline reruns the original prompt. The oracle language hint supplies candidate evidence languages by adding You may find relevant evidence by searching in the following languages: {languages}. The candidate set contains the language cover, and the agent uses it to formulate searches. Coverage diagnosis uses the original certificate. EN-query-only adds Search instruction: write all search queries in English only.
Each condition evaluates the same 200 semantic tasks using final per-question judgments. Full accuracy confidence intervals (CIs) and paired effects appear in Appendix I, Tables 13–14 (pp. 30–31).
| Condition | Kimi-K3 | GLM-5.3-Flash | Qwen3.8-Max |
|---|---|---|---|
| Baseline | 55.5% | 45.0% | 45.0% |
| EN-query-only | 51.5% | 38.0% | 44.0% |
| Oracle language hint | 64.0% | 57.5% | 55.0% |
Across the three models, the mean paired effect is −4.0 pp for EN-query-only (95% CI [−7.3, −0.7]) and +10.3 pp for the oracle [6.3, 14.5]. CIs resample the 200 semantic tasks jointly, retaining each task’s outcomes across models and conditions. Candidate-language guidance is therefore a tested way to improve these agents under the original budget, directly answering your request for an actionable strategy.
English-query shares rise from 60.75% to 91.46% for Kimi-K3, 37.32% to 90.15% for GLM-5.3-Flash, 48.69% to 84.48% for Qwen3.8-Max, as measured by the original language-identification (LID) pipeline (Appendix I, Table 15, p. 31). The query-language shift confirms that the prompt intervention changes how agents search, connecting the accuracy comparison to retrieval behavior.
To test whether guidance resolves the diagnosed language bottleneck, we followed baseline runs that answered incorrectly without completing a language cover (wrong-L). Across the three models, 169/180 (93.9%) baseline wrong-L runs complete the original certificate's language cover under the oracle, 44/180 (24.4%) answer correctly, and 41 achieve both (Appendix J, Table 17, p. 32). Another 128 complete coverage but remain incorrect, showing why language coverage and subsequent evidence recovery are separately useful diagnostic stages.
The same-task recoveries connect language planning to successful answers; the remaining errors explain why evidence recovery must still follow language coverage. The construction pipeline can also supply checkable language and relation-completion targets for future supervision or reinforcement-learning approaches, including coverage tracking and branch-aware retrieval.
Example 1: connecting the evidence for Santa Quiteria (Appendix J.3, p. 33). With Catalan and Spanish hints, Qwen identifies Elisa Núñez through Spanish search, obtains her birthplace Elche de la Sierra from Catalan results, and retrieves the Santa Quiteria church and organ-restoration records. It answers correctly, while baseline searches along an Italian-church hypothesis and answers incorrectly. Both runs complete language coverage while leaving some reference evidence checks unmet (T), but the answer changes from wrong to correct: a wrong-T → correct-T transition illustrating the relation-retrieval benefit of language planning.
Example 2: the missing museum relation in the gas-pipeline task (Appendix J.3, p. 33). The question identifies Ivatsevichy and Torzhok through museum clues and asks for a pipeline containing those cities in its name. EN-query-only uses 247/254 English queries (97.24%) and encounters the correct Torzhok–Minsk–Ivatsevichy candidate. However, its English query about the Ivatsevichy museum's origin returns no results; it does not verify the link to a commemorative people's museum and selects Urengoy–Pomary–Uzhhorod.
Baseline and oracle retrieve Russian and Belarusian evidence that the local-history museum developed from the Goishik commemorative museum and administers the Khavanshchyna forest memorial branch. These relations match the question; combined with the Vasilyovo wooden-architecture museum near Torzhok and pipeline-name records, both runs answer correctly. In this case, baseline and oracle finish normally and EN-query-only times out with the wrong answer. In the Santa Quiteria case, both controls time out with wrong answers and the oracle finishes normally. Both cases illustrate how language guidance helps connect the relations needed to verify an answer.
Response to Reviewer sCwE — Part 3/4
4. Open-Web validity and human verification (W2, Q1, Q3, Q7)
The open Web changes and can contain conflicting or incorrect claims. Our curated, frozen KG anchors gold answers and reference relations; its certificates provide verifiable reference paths for task construction and process diagnosis. Language-cover constraints are checked against the frozen lexical inventory. Agents search the live Web, where shortcut screening, trajectory audits, and the interventions above test actual retrieval behavior.
We checked the diagnosis across runs with incomplete coverage (L), incomplete reference evidence checks after coverage (T), and all checks passed (R), recording final-answer correctness separately. We audited 546 trajectories, sampling 100 each from correct-L, wrong-L, correct-T, wrong-T, and correct-R, and including all 46 wrong-R runs. GPT-6 Terra produced candidate judgments and rationales; human reviewers examined these together with the corresponding trajectory segments, checking actual support and consistency with the paper's L/T/R definitions. Further review and adjudication produced one final outcome for each trajectory.
The complete confusion matrix is in Appendix K, Table 19 (p. 34).
The agreement rates are 99.0% for L, 75.0% for T, and 85.6% for R; 86.6% overall. All samples remain in the denominator: 473 retain the category, 27 change category, and 46 have insufficient evidence to determine a category after adjudication. This supplies the requested human check of label accuracy, with especially strong support for identifying incomplete coverage.
Among the audited correct-L/T runs, 80.0% answer correctly without full visible support for the required relations (Appendix K, Table 20, p. 34). Most therefore appear to infer or guess the answer from partial clues before completing verification; supported alternative paths explain a smaller share.
For Q7, Appendix K.4 (p. 35) reports that sufficiently supported alternative paths occur in 58/546, and valid paths bypassing a designated bridge in 44/546, with these fields adjudicated per trajectory. These are frequencies within the stratified audit sample. Recording these paths alongside the process labels makes their frequency and supporting evidence explicit, directly addressing how alternative routes affect interpretation of the diagnosis.
5. Language entry, evidence recovery, and exposure (Q4, Q8; Q1)
Cover completion means that the agent has entered the languages of a valid cover through recorded queries or page accesses. Whether it retrieved useful relational evidence is assessed at the evidence-recovery stage. Answer exposure records a further observable event.
To check how answer detection reflects the evidence actually retrieved, we compared ordinary answer mentions with strong-identifier hits matching the answer’s Wikidata entity identifier (QID) or exact Wikipedia URL. In the audit (Appendix K, Table 21, p. 35), 84/546 (15.4%) trajectories contain answer text without a strong-identifier hit, while 229/546 (41.9%) have strong-identifier hits but incomplete support for key relations. The latter occurs in 41/46 wrong-R runs (89.1%).
This explains why a run can complete language, path, and branch coverage yet remain in T3: the strong answer-identifier check is unmet even though the answer may appear in ordinary text. Errors after all exposure checks pass (wrong-R) are residual errors after answer exposure, potentially involving evidence integration, relation verification, candidate selection, or output. The audit therefore supports treating exposure and relation support as distinct observations, answering how identifier-based detection affects the interpretation of these stages.
Response to Reviewer sCwE — Part 1/4
Thank you for recognizing the focus on complementary multilingual evidence, the verifiable generation constraints, and the value of process diagnosis. We have added an explicit language-planning intervention, human trajectory verification, and the requested process and population statistics, with the corresponding revisions highlighted in red in the manuscript.
1. Task scale and relational complexity (W1)
CocoonBench contains 200 independent semantic tasks. For each task, we compare English (EN), Chinese (ZH), and a designated language selected from one evidence branch (DL), assessing statistical uncertainty at the task level.
The fact that 85% of tasks have two relational clues reflects a deliberate design choice informed by inspecting generated cases. Adding relational constraints and requiring more languages makes satisfying entity combinations increasingly sparse, producing overly long-tail questions about rare entities and unusual relations. We control this complexity to retain natural, checkable questions whose difficulty centers on multilingual retrieval and evidence integration.
Clue count and language count capture different aspects of complexity. To measure the latter, we find the smallest language sets providing names for all hidden retrieval-key entities in the frozen inventory—the task’s minimum language covers. 39/200 tasks (19.5%) require at least three languages in a minimum cover, and even two clues can require multiple entity-recovery steps, searches, and cross-source checks. Each task also satisfies answer-uniqueness, clue-necessity, and frozen-lexicon coverage constraints. The released construction pipeline supports generating additional tasks and relation combinations; the current set provides a carefully constrained instance of that methodology. Thus, the clue-count distribution reflects controlled task construction, while the language-cover counts show that the set also tests broader multilingual retrieval.
2. Comparison with existing benchmarks (W3; W1)
Our contribution combines a reusable task-generation method, open-Web multilingual search, and diagnosis grounded in per-task construction certificates that record the answer, reference relations, and language covers. We generate questions from frozen knowledge-graph (KG) relations and jointly check answer uniqueness, clue necessity, and language coverage of hidden entity names. The 200 tasks instantiate this process.
CocoonBench adds new task generation and in-the-wild multilingual retrieval. XBCP retains BrowseComp-Plus questions and answers, creating its multilingual corpus by translating and reassigning existing documents. Its evidence-language distribution is imposed by construction. CocoonBench generates new questions from KG relations and lets agents discover naturally occurring evidence on the live Web. This tests language selection and relation recovery under real Web conditions, beyond retrieval over a translated, fixed corpus.
These benchmarks also stress different capabilities: controlled corpus retrieval and integration, multi-entity answer completeness, or page interaction. CocoonBench focuses on recovering hidden entities and relations from clues while selecting useful search languages.