- Unclear correspondence between the reported performance gap and its error attribution
A central claim of the paper is that approximately 55% of the multilingual performance gap originates from genuine agent failures. However, it is unclear whether the error-attribution methodology directly supports this conclusion for the headline pass@3 metric.
Specifically, pass@3 considers a scenario successful when at least one of three rollouts succeeds, whereas the attribution methodology operates at the level of individual failing rollouts and includes stochastic regressions in which the target-language agent successfully completes some attempts (Appendix I, Tables 17–19).
Such cases may contribute to differences in per-rollout success rates without contributing to the scenario-level pass@3 gap. Therefore, the resulting failure distribution does not necessarily represent a decomposition of the reported pass@3 performance degradation. Furthermore, the attribution study is conducted only for Claude 4.7 Opus, while the broader conclusions concern seven different agents.
This distinction is important because identifying whether the multilingual gap primarily reflects genuine agent limitations or benchmark-related artifacts is one of the central contributions of the paper.
- Remaining benchmark and verification artifacts limit the interpretation of the results
Although the authors make considerable efforts to validate the translated benchmark, their own analysis identifies substantial translation and evaluation artifacts. Translation defects account for approximately 35% of the attributed regressions, and an estimated 6.4% of scenario-language pairs become unsolvable due to translation issues.
Moreover, the paper acknowledges that the verifier can penalize agents for additional state-changing actions even when the final environment state is correct (Section 5.2). Such cases raise concerns about whether some reported failures reflect genuine agent inability or limitations of the evaluation mechanism.
The human validation remains relatively limited: 140 conclusive adjudications across seven languages, with notably lower agreement for Japanese (68.2%). The multilingual judge calibration also primarily relies on translated English-annotated trajectories rather than independently annotated target-language gold labels.
Consequently, while the existence of a multilingual gap is well supported, its precise magnitude and the relative contribution of different failure mechanisms remain less certain.
Response to Reviewer rQxX
We thank the reviewer for the detailed review. We address validation, positioning and the requested robustness analyses below.
We have revised the manuscript to address the points below. Line references refer to the original submission.
Human validation and translation defects
Full-corpus human post-editing would indeed strengthen the benchmark, and its absence is acknowledged in the limitations (L474–478). Our validation includes five reviewers adjudicating 149 items in seven languages, with 91.4% agreement on the 140 conclusive cases (L352–358), and three linguists inspecting translated inputs and full agent trajectories for 16 scenarios across all four capabilities (L362–365). We have clarified that the linguistic analysis examines dataset quality as well as agent behaviour. These studies provide partial validation, rather than a substitute for a full translation audit.
The 35% translation-defect figure describes Claude's sampled, reweighted failing target-language rollouts on English-solved scenarios. Therefore, it is neither a defective-item rate nor a decomposition of the net pass@3 gap (L324–333, Table 18). We have clarified that distinction throughout the paper, including the abstract and conclusion. Separately, the analysis in the paper's Sec. 5.1 and Appendix I estimates translation-induced unsolvability in 6.4% of its 6,056 scenario–language pairs. Of the 6,056 analysed scenario–language pairs, 74.5% are solved at least once in the target language (L338–350). The both-fail cell has a sampled translation-defect rate of 9.9% (L1743–1744).
Positioning relative to GAIA-v2-LILT
GAIA-v2-LILT localises the original GAIA into five languages, adapting task semantics to each locale (L408–410). OmnilingualGAIA2 instead extends GAIA2, a distinct English-only dataset, with its stateful app environments, in ten target languages. Its translation pipeline aims to preserve the same underlying tasks and consistency across instructions, environment states and oracle events (L112–139). This enables paired, cross-lingual comparisons, subject to the residual defects discussed above. The contribution also includes a multilingual verifier, a seven-agent evaluation and diagnostic failure attribution. We have made this distinction clearer in the introduction and related work.
Confidence intervals and pass@1
We have added 95% CIs for the pooled gaps in Table 1, using 2,000 scenario-bootstrap resamples stratified by capability and paired across languages, and now report pass@1 alongside pass@3:
Every gap's CI excludes zero under both metrics. The Gemini–Claude pass@3 difference is not statistically resolved (−0.3 pp, 95% CI [−3.5, +2.9]), and we do not rank their gaps. At pass@1, Gemini's gap is smaller than Claude's (difference −6.3 pp, CI [−8.9, −3.8]). Thus the direction of the multilingual deficit seems to be robust to the metric, while the relative ordering can change: Claude benefits more from retries. We have clarified this metric-specific ordering in Sec. 4.1. Qwen-3.6-35B-A3B has the largest gap under both metrics.
Known defects and release
We will release the benchmark as evaluated, together with the error analysis. Known translation defects remain in that version, and the measured scores should be interpreted with this limitation. Our linguistic analysis also identifies translation-independent agent failures (L385–394), and the existence of these failures is distinct from estimating their share of the net gap.
Reference translations
The “reference translation” used in the translator evaluation is an internal machine-translated, high quality version produced using proprietary models. Human review covered stratified samples rather than the full reference set. It is not a human-translated gold standard and will not be released. We have clarified its provenance and the scope of human review in Appendix B.
Presentation
We have added gap CIs to Table 1, summarised the new diagnostics in the main text and removed the duplicated “family-match” at L292.