acceptodds
Under review as a conference paper at ICLR 2027

OMNILINGUALGAIA2: EVALUATING THE MULTILINGUAL GAP IN FRONTIER AI AGENTS

Abstract

Agentic benchmarks aim to measure how well AI agents plan, reason, and act in realistic environments, yet they are almost exclusively English-only. As AI agents are increasingly deployed to linguistically diverse users, whether agentic competence measured in English generalizes to other languages remains an open question. We introduce OMNILINGUALGAIA2, a machine-translated expansion of the GAIA2 agentic benchmark, with partial human-expert validation. OMNILINGUALGAIA2 covers ten target languages spanning five writing systems and includes a cross-lingually calibrated multilingual verifier. Evaluating seven frontier proprietary and open-weight models, we find a universal cross-lingual gap of 8.8–18.4 pass@3 points. The gap varies substantially across agents, is concentrated in tool orchestration rather than quantitative reasoning, and persists with increasing model scale. A stratified error attribution shows that at least 55% of the observed gap can be attributed to model-driven failures, which we further study through human-expert linguistic analysis. We identify morphological cue loss and amplified ambiguity as the primary failure mechanisms, especially in non-Latin-script languages. These findings demonstrate that strong English-language agentic performance does not reliably transfer across languages, and argue that multilingual agentic evaluation should become a standard component of reporting for globally deployed agents.

Primary area:
datasets and benchmarks
Keywords:
multilingual evaluation, agentic benchmarks, cross-lingual transfer, tool use and orchestration, machine translation (for benchmark localisation), LLM-as-a-judge / automatic verifiers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LKVTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper reconstruct a benchmark (GAIA2 -> OmnilingualGAIA2) to evaluate LLM Agents on multi-lingual environments, separating genuine agent failures from translation defects and evaluation artifacts.

Critical strengths
  1. Evaluation across ten target languages and seven agents enables systematic comparison of multilingual task execution.
  2. Cross-surface translation consistency, multilingual judge calibration, and human audits strengthen evaluation reliability.
Critical weaknesses
  1. Limited novelty. The contribution is largely incremental over existing environments and multilingual agent benchmarks.
  2. No English-pivot strategy is evaluated to establish whether straightforward translation-based mitigation substantially reduces the gap. (another external translator)
  3. Translation defects and penalties for harmless self-corrections remain insufficiently addressed through corrected-data reruns and verifier sensitivity analyses.
Questions for authors

See weaknesses.

Reviewer_pJSiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Omnilingual GAIA2, a multilingual extension of the GAIA2 agentic benchmark covering ten target languages across five writing systems. The authors develop a machine-translation pipeline that preserves consistency between task instructions, environment states, and expected agent actions, alongside a multilingual LLM-based verifier calibrated against human annotations. The benchmark is evaluated using seven proprietary and open-weight AI agents across four capabilities: execution, search, adaptability, and ambiguity. The results reveal a consistent multilingual performance gap of 8.8–18.4 percentage points in pass@3 relative to English, with particularly pronounced degradation in some non-Latin-script languages. The authors further investigate changes in agent tool-use behavior and introduce an error-attribution methodology distinguishing translation defects, verifier artifacts, and genuine agent failures. Their analysis estimates that approximately 55% of the diagnosed multilingual regressions are agent-driven, complemented by human validation and qualitative linguistic analysis.

Critical strengths
  1. Relevant and practically important evaluation problem The paper addresses an important limitation of current agentic evaluation methodologies: the reliance on English-language benchmarks despite the growing deployment of AI agents in multilingual environments. Extending GAIA2 beyond English while preserving consistency across task descriptions, environment states, and verification targets is a meaningful technical contribution. Unlike conventional multilingual question-answering benchmarks, this work evaluates agent behavior involving multiple tool interactions and state-changing operations.

  2. Comprehensive empirical evaluation and detailed failure analysis The evaluation spans seven heterogeneous agents, ten target languages, and four agentic capabilities, providing a useful empirical perspective on the multilingual performance gap. Beyond reporting aggregate performance, the paper examines tool-use patterns, verifier behavior, translation defects, and linguistic failure mechanisms. The inclusion of controlled ablations and human validation strengthens the empirical contribution and provides insights into potential causes of multilingual degradation.

Critical weaknesses
  1. Unclear correspondence between the reported performance gap and its error attribution

A central claim of the paper is that approximately 55% of the multilingual performance gap originates from genuine agent failures. However, it is unclear whether the error-attribution methodology directly supports this conclusion for the headline pass@3 metric.

Specifically, pass@3 considers a scenario successful when at least one of three rollouts succeeds, whereas the attribution methodology operates at the level of individual failing rollouts and includes stochastic regressions in which the target-language agent successfully completes some attempts (Appendix I, Tables 17–19).

Such cases may contribute to differences in per-rollout success rates without contributing to the scenario-level pass@3 gap. Therefore, the resulting failure distribution does not necessarily represent a decomposition of the reported pass@3 performance degradation. Furthermore, the attribution study is conducted only for Claude 4.7 Opus, while the broader conclusions concern seven different agents.

This distinction is important because identifying whether the multilingual gap primarily reflects genuine agent limitations or benchmark-related artifacts is one of the central contributions of the paper.

  1. Remaining benchmark and verification artifacts limit the interpretation of the results

Although the authors make considerable efforts to validate the translated benchmark, their own analysis identifies substantial translation and evaluation artifacts. Translation defects account for approximately 35% of the attributed regressions, and an estimated 6.4% of scenario-language pairs become unsolvable due to translation issues.

Moreover, the paper acknowledges that the verifier can penalize agents for additional state-changing actions even when the final environment state is correct (Section 5.2). Such cases raise concerns about whether some reported failures reflect genuine agent inability or limitations of the evaluation mechanism.

The human validation remains relatively limited: 140 conclusive adjudications across seven languages, with notably lower agreement for Japanese (68.2%). The multilingual judge calibration also primarily relies on translated English-annotated trajectories rather than independently annotated target-language gold labels.

Consequently, while the existence of a multilingual gap is well supported, its precise magnitude and the relative contribution of different failure mechanisms remain less certain.

Questions for authors
  1. Consistency between pass@3 and failure attribution Could the authors clarify how the rollout-level error-attribution methodology corresponds to the reported scenario-level pass@3 gap? In particular, how are stochastic regressions treated when at least one target-language rollout succeeds, and how are cases where the target language outperforms English incorporated into the attribution? A decomposition explicitly aligned with pass@3 would substantially strengthen the main conclusion.

  2. Robustness of the results after correcting evaluation artifacts Can the authors report how the multilingual performance gap changes when known translation defects and verifier-related false failures are corrected or excluded? In particular, how does the gap change when additional write actions lead to a correct final environment state? Would the conclusion that tool-orchestration failures dominate remain valid?

  3. Generalizability of the error attribution Given that the quantitative failure attribution is performed for Claude 4.7 Opus, is there evidence that the same proportions of agent failures, translation defects, and verifier artifacts hold for other evaluated agent families? Even a smaller stratified analysis involving one additional proprietary or open-weight agent would help establish whether the reported decomposition reflects a general multilingual agent phenomenon rather than characteristics of one specific agent.

Reviewer_rQxXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces OMNILINGUALGAIA2, which is a machine-translated version of GAIA2 benchmark. OMNILINGUALGAIA2 can be used to evaluate and analyze the multilingual gap in frontier AI agents, which the authors perform in the paper. Along with OMNILINGUALGAIA2, the paper also presents cross-lingually calibrated verifier. The major contribution of the paper is the newly introduced multilingual dataset, and the multi-lingual gap analysis. For validation, the paper uses GPT-OSS-120B as the judge, which the authors find to perform comparably to LLAMA 3.3, the judge originally used in GAIA2.

Critical strengths
  1. The major strength of the paper is the vast analysis authors perform in the dataset. This shows that the dataset can be valuable to community for multiple analysis, once the issues are fixed (see weaknesses).
Critical weaknesses

While I commend the massive effort by the authors to convert the dataset to 10 different languages to bridge the gap in multilingual agentic analysis, I believe there are a lot of critical weaknesses in the paper that the authors must address before the dataset can be released to the other researchers.

  1. The dataset is machine-translated, and the translation are not systematically verified by native speakers. Authors' own analysis of gap attributes 35% of multilingual gap to translation defect. A machine-translated dataset for multilingual analysis, must be human validated, for each language.
  2. The paper is hard to read currently. Many tables that are referenced in main text are in appendix, same for figures. Readers must juggle between main body and appendix a lot.
  3. The paper says early on that multilingual agentic evaluation is largely unmeasured, however in the related work section, the authors write about some very close prior works in the field, with one of them also being a translated version of GAIA (GAIA-v2-LILT). GAIA-V2-LILT paper actually argues that machine translation with minimal post-editing breaks the validity of agent benchmarks, and they instead perform human audit. It is not clear where this paper stands in the field, as prior works have already explored machine-translation for multilingual agentic evaluation.

The major contribution of the paper is the dataset; however, the dataset itself is not human validated for translation, and it raises questions about if the dataset at its current form is even suitable for downstream analysis. The paper would also benefit from substantial rewriting.

Questions for authors

A question for authors:

  1. Will the known defects be corrected in the dataset release or not? These include translation defects that the authors discuss in line 368-375.
  2. Can the authors clarify further about the positioning of their paper with respect to current literature, especially GAIA-v2-LILT?
  3. What is the "reference translation" in the translation pipeline? Is this a human translated version, and will this be released?

Some suggestions:

  1. There are some typos in the paper, please correct them. For example: family-match appears twice in Line 292.
  2. Please push some figures and tables to main text, and many analysis texts can be cut from main body for better readability.
  3. Please give CIs on the pooled gaps in Table 1, we need these for cases like Gemini at 8.8 vs Claude at 9.1. Please also discuss robustness under pass@1.

Edit:

The authors have responded to my questions, and they have incorporated the suggestions. However, my major concerns still remain.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer rQxX

We thank the reviewer for the detailed review. We address validation, positioning and the requested robustness analyses below.

We have revised the manuscript to address the points below. Line references refer to the original submission.

Human validation and translation defects

Full-corpus human post-editing would indeed strengthen the benchmark, and its absence is acknowledged in the limitations (L474–478). Our validation includes five reviewers adjudicating 149 items in seven languages, with 91.4% agreement on the 140 conclusive cases (L352–358), and three linguists inspecting translated inputs and full agent trajectories for 16 scenarios across all four capabilities (L362–365). We have clarified that the linguistic analysis examines dataset quality as well as agent behaviour. These studies provide partial validation, rather than a substitute for a full translation audit.

The 35% translation-defect figure describes Claude's sampled, reweighted failing target-language rollouts on English-solved scenarios. Therefore, it is neither a defective-item rate nor a decomposition of the net pass@3 gap (L324–333, Table 18). We have clarified that distinction throughout the paper, including the abstract and conclusion. Separately, the analysis in the paper's Sec. 5.1 and Appendix I estimates translation-induced unsolvability in 6.4% of its 6,056 scenario–language pairs. Of the 6,056 analysed scenario–language pairs, 74.5% are solved at least once in the target language (L338–350). The both-fail cell has a sampled translation-defect rate of 9.9% (L1743–1744).

Positioning relative to GAIA-v2-LILT

GAIA-v2-LILT localises the original GAIA into five languages, adapting task semantics to each locale (L408–410). OmnilingualGAIA2 instead extends GAIA2, a distinct English-only dataset, with its stateful app environments, in ten target languages. Its translation pipeline aims to preserve the same underlying tasks and consistency across instructions, environment states and oracle events (L112–139). This enables paired, cross-lingual comparisons, subject to the residual defects discussed above. The contribution also includes a multilingual verifier, a seven-agent evaluation and diagnostic failure attribution. We have made this distinction clearer in the introduction and related work.

Confidence intervals and pass@1

We have added 95% CIs for the pooled gaps in Table 1, using 2,000 scenario-bootstrap resamples stratified by capability and paired across languages, and now report pass@1 alongside pass@3:

Agentpass@3 gap (pp) [95% CI]pass@1 gap (pp) [95% CI]
Claude 4.7 Opus9.1 [6.8, 11.4]14.0 [11.9, 16.1]
GPT-5.49.9 [7.1, 12.7]9.5 [7.3, 11.7]
Gemini 3.1 Pro8.8 [6.1, 11.4]7.7 [5.6, 9.7]
Kimi-2.612.3 [9.6, 15.2]11.1 [9.1, 13.1]
Gemma-4-31B11.9 [9.4, 14.4]10.1 [8.2, 12.0]
Qwen-3.6-27B12.0 [9.1, 15.0]11.0 [8.8, 13.2]
Qwen-3.6-35B-A3B18.4 [15.4, 21.5]17.0 [14.8, 19.2]

Every gap's CI excludes zero under both metrics. The Gemini–Claude pass@3 difference is not statistically resolved (−0.3 pp, 95% CI [−3.5, +2.9]), and we do not rank their gaps. At pass@1, Gemini's gap is smaller than Claude's (difference −6.3 pp, CI [−8.9, −3.8]). Thus the direction of the multilingual deficit seems to be robust to the metric, while the relative ordering can change: Claude benefits more from retries. We have clarified this metric-specific ordering in Sec. 4.1. Qwen-3.6-35B-A3B has the largest gap under both metrics.

Known defects and release

We will release the benchmark as evaluated, together with the error analysis. Known translation defects remain in that version, and the measured scores should be interpreted with this limitation. Our linguistic analysis also identifies translation-independent agent failures (L385–394), and the existence of these failures is distinct from estimating their share of the net gap.

Reference translations

The “reference translation” used in the translator evaluation is an internal machine-translated, high quality version produced using proprietary models. Human review covered stratified samples rather than the full reference set. It is not a human-translated gold standard and will not be released. We have clarified its provenance and the scope of human review in Appendix B.

Presentation

We have added gap CIs to Table 1, summarised the new diagnostics in the main text and removed the duplicated “family-match” at L292.

AuthorsOpenReview ↗

Response to Reviewer pJSi

We thank the reviewer for the careful reading and constructive questions. We agree that the relationship between failure attribution and pass@3 needed clarification.

We have revised the manuscript to address the points below. Line references refer to the original submission.

W1 / Q1. Attribution and pass@3

The 55% / 35% / 10% shares describe failing target-language rollouts on English-solved scenarios, not the net pass@3 or pass@1 gap (L324–333). When both languages succeed at least once, a pair contributes zero to the pass@3 gap but can affect pass@1 and passall@3. Agent failures account for 75% of attributed failures in this stochastic stratum (Table 17).

For Claude, we use Table 1's 6,326 pairs with at least one valid rollout per side. English-only successes are offset by target-only successes, which were not triaged. Applying Sec. 5.1's translation rates gives:

pass@3 accountingPairspp
Losses: English-only successes (448 + 468)91614.5
Gains: target-only successes (51 + 289)3405.4
Net gap5769.1
Translation: English 3/3 → target 0/3 (59% of 448)≈2644.2
Translation: English 1–2/3 → target 0/3 (16% of 468)≈751.2
Translation losses, total≈3395.4

Translation accounts for an estimated 37% of gross losses (339/916). Recovering those losses without changing other outcomes would leave 3.7 pp of the 9.1 pp net gap (237 pairs), still including verifier effects. This assumes recovery, rather than reporting a corrected-data rerun. Under the same assumption, pass@1 falls from 14.0 to 8.1 pp, including stochastic regressions with their 12% translation share.

Across 6,400 nominal pairs (640 scenarios × 10 target languages), 540 (8.4%) are solved in English by some agent but never in the target language by any evaluated agent. Excluding these pairs gives:

Agentpass@3 gap (pp)After excluding target-unsolved pairs (pp)
Claude 4.7 Opus9.12.9
GPT-5.49.96.0
Gemini 3.1 Pro8.85.6
Kimi-2.612.39.0
Gemma-4-31B11.99.3
Qwen-3.6-27B12.08.8
Qwen-3.6-35B-A3B18.416.8

This can exclude hard but solvable pairs, since no observed success does not prove unsolvability. All gaps remain positive, with over 60% remaining for each agent other than Claude. This is a sensitivity analysis, not performance on corrected data. We now distinguish conditional attribution from net gaps throughout the paper and state these populations and assumptions in Appendix I. Table 20's “agent gap” values likewise describe conditional agent-attributed failure rates, not residual net pass@3 gaps.

Q2. Verifier sensitivity and tool orchestration

Across seven agents, target-minus-English extra-write failure rates differ by −0.4 to +2.2 pp of rollouts, while missing-write failures rise by +3.4 to +12.0 pp. Counting every extra-write failure as successful reduces the pass@3 gap range from 8.8–18.4 to 5.5–12.3 pp. This also credits harmful writes and trajectories that could fail later checks, without checking final-state correctness. Extra-write penalties therefore do not fully explain the gap. First-failing checks concentrate in write-count mismatches and textual-reply rejections (L303–307), but do not establish agent-error dominance after all artefacts are corrected. We have added the sensitivity analysis to Appendix F and qualified the main-text interpretation.

Q3. Generalisability

The original attribution covers Claude, as clarified in the revised limitations. The check-level analysis covers Claude, GPT-5.4 and Kimi-2.6, and the exclusion above covers all seven agents.

We repeated attribution on GPT-5.4 in Japanese, Hindi and Spanish (132 scenarios, 263 failing rollouts), stratifying by language, capability and determinism, reweighting by stratum sizes and using scenario-clustered bootstrap CIs. We applied the same protocol to Claude in these languages. The agent/translation/verifier shares are:

  • Gemini 3.1 Pro triager: Claude 57/31/12, GPT-5.4 55/36/8 (95% CIs [48, 63], [28, 44], [4, 13]).
  • Claude 4.7 Opus triager: Claude 67/27/6, GPT-5.4 71/21/8.

GPT-5.4's shares fall within Claude's CIs under each triager. The patterns are similar in these languages, but exact shares depend on the triager. We have added this replication to Appendix I.

W2. Human validation and calibration

Reviewers confirmed 128/140 labels across seven languages (91.4%). Japanese agreement is 68%, with agent labels reassigned to translation. We have flagged possible underestimation of Japanese translation defects alongside Table 20. Judge calibration carries English human labels over to translated traces, without independent target-language gold labels. We have clarified this in the judge-reliability limitation.

AuthorsOpenReview ↗

Response to Reviewer LKVT

We thank the reviewer for their feedback and suggestions. We report new instruction-pivot and verifier-sensitivity analyses below.

We have revised the manuscript to address the points below. Line references refer to the original submission.

W1. Novelty

While our work is incremental over GAIA2 by design, its contribution relative to existing multilingual agentic benchmarks lies in combining the following:

  1. Paired stateful tasks in ten target languages across five writing systems. GAIA2 checks state-changing actions against oracle events. Our term-table pipeline aims to preserve tasks and consistency across translated instructions, app states and oracle events (L127–139, Fig. 2). GAIA-v2-LILT instead localises the original GAIA into five languages with locale-specific adaptation (L408–410). These designs address complementary questions.

  2. A multilingual verifier. We localise judge prompts and calibrate against GAIA2's human-labelled English traces and their translations into all ten target languages (L200–217, Appendix C.3).

  3. A seven-agent evaluation with diagnostic attribution and human analysis. We distinguish translation defects, verifier artefacts and agent failures, complemented by linguistic analysis and human re-adjudication (Sec. 5, Appendices I and K).

W2. English-pivot baseline

We thank the reviewer for suggesting an English-pivot baseline. The target language appears in the environment as well as the instructions. We test whether adding English instructions helps while the agent operates in the target-language environment, with the verifier unchanged. A full pivot would also translate tool observations and write actions, which we have not evaluated.

We append either GPT-5.4's back-translation or the original GAIA2 English instructions. Gemma-4-31B pass@3 results pool four capabilities with 160 scenarios each, at 131K context. English and target baselines were rerun alongside both pivot arms, pairing all four columns on the same scenarios. Table 12's Hindi includes full-context rescoring after truncation, whereas these controls use the fixed 131K rerun.

engtarget+ MT English+ original English
jpn64.141.242.746.4
hin64.135.040.641.9
cmn64.146.948.950.9
tur64.153.451.255.3
spa64.155.358.059.8
avg64.146.448.350.9

The external-translator pivot recovers 1.9 of the 17.7 pp average gap (11%, pooled paired McNemar p = 0.01, significant per-language gain only for hin). Original English instructions recover 4.5 pp (25%). Most of the gap remains under either intervention. Their difference, about 15% of the original gap, does not isolate translation defects. We have added this five-language instruction-level diagnostic in Appendix L.

W3. Translation defects and harmless self-corrections

Write-count sensitivity. GAIA2's write-count check is applied identically in English and target languages. Across seven agents, extra-write failure rates differ by −0.4 to +2.2 pp between target languages and English, while missing-write failures increase by +3.4 to +12.0 pp. Missing writes reflect omitted actions. We count every extra-write failure as successful, including harmful writes and trajectories that could fail later event matching:

AgentExtra-write failures, engExtra-write failures, targetpass@3 gap (pp)Gap if extra-write failures passed (pp)
Claude 4.7 Opus5.9%8.0%9.16.2
GPT-5.414.4%14.9%9.98.2
Gemini 3.1 Pro16.0%17.1%8.85.5
Kimi-2.615.5%16.5%12.38.7
Gemma-4-31B16.6%16.3%11.99.8
Qwen-3.6-27B14.7%16.9%12.06.5
Qwen-3.6-35B-A3B22.6%22.2%18.412.3

The gap remains positive for every agent, falling from 8.8–18.4 to 5.5–12.3 pp. This measures sensitivity to the extra-write penalty, without checking whether final states are correct. We have added both analyses to Appendix F.

Translation defects. Sec. 5.1 and Appendix I estimate translation-induced unsolvability in 6.4% of 6,056 analysed pairs. Of all 6,056 pairs, 74.5% are solved at least once in the target language. Separately, translation accounts for 35% of attributed failing target rollouts on English-solved scenarios.

For Claude, Table 1's 6,326 eligible pairs include 916 English-only and 340 target-only successes, a net English advantage of 576 pairs (9.1 pp). Recovering the estimated 339 translation-attributed losses without changing other outcomes would leave about 3.7 pp (237 pairs), still including verifier effects. This is an assumption-based estimate, not a corrected-data rerun. Table 20's conditional agent-attributed failure rates are also not residual net pass@3 gaps. We have clarified these distinctions throughout the paper, including the abstract and conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.