acceptodds
Under review as a conference paper at ICLR 2027

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

Abstract

Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are largely English-centric, limiting our understanding of agent capabilities in multilingual settings. We introduce BABELFLOW, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics. Using BABELFLOW, we construct BABELARENA, a large-scale multilingual benchmark comprising 16,146 instances derived from 702 canonical tasks across four benchmark families, 13 domains, and 23 languages. Experiments with five frontier models show that no single model dominates across benchmark families and that cross-language disparities extend well beyond task success. Lower-resource languages exhibit distinct failure patterns, with larger shares of tool-use and control-flow errors rather than answer-quality errors alone, pointing to gaps in reliable task execution across the resource levels of these languages. On the same tasks, agents in low-resource languages also consume substantially more tokens than in English (up to roughly twice the input) without proportional increases in interaction length, and language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English. We believe BABELARENA provides a foundation for advancing research on reliable and efficient multilingual agents.

Primary area:
datasets and benchmarks
Keywords:
multilingual, agents, LLM, evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iFS5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BabelFlow, an agentic procedure for localizing executable agent benchmarks using dependency analysis, typed preservation/free/canonical translation, structure-preserving reconstruction, multi-agent verification and human review. This workflow applies to four source benchmark families to construct BabelArena. This multilingual benchmark comprises 702 canonical tasks, 16,146 language-specific instances, 13 domains and 23 languages. A 40-task Chinese-language ablation reports a reduction in broken references from 20.0% under independent translation to 2.5% with BabelFlow. Five frontier models are then evaluated using task success, three-trial robustness and language consistency.

Critical strengths
  1. Practically relevant multilingual benchmark contribution. A benchmark-general adaptation workflow that covers the full executable environment. Unlike most prior multilingual agent benchmarks, which translate only the task inputs, BABELFLOW adapts prompts, tasks, tool descriptions and environment content together. It does so through runtime data-flow analysis, typed translation, structure-preserving reconstruction, and behavioral-equivalence checks that execute source and target language side by side.
  2. Practical Evaluation. The paired analyses (Tables 6–7, 10–11) provide valuable insights beyond task success rates. Notably, lower-resource languages incur substantially higher token costs even when controlling for task success, conversation turns, and tool calls.
Critical weaknesses
  1. Translation Validity. Benchmark translation quality was audited by humans only for the 7 high-resource languages. The 16 medium- and low-resource languages, where translation is hardest and LLM judges are least reliable, were scored by GPT-5.6-Sol alone, with no human calibration and error rates by language. This raises concerns about the overall quality of the benchmark, which could in turn affect the community work that builds on it.

  2. Limited validation across languages. While the authors evaluate model performance across all 23 languages, the validation of BabelFlow itself is limited to 40 tasks translated into Chinese. This raises concerns about whether the reported improvements in structural integrity and behavioral equivalence generalize to medium- and low-resource languages. Given that multilingual benchmark construction is a central contribution, validation across a more representative set of languages is needed.

Questions for authors
  1. Low-resource translation quality: Could the authors report native-speaker quality scores for at least a sample of the medium- and low-resource languages? What share of translations scored 2 or below?

  2. Human review capacity: Figure 1, Step 5 requires human reviewers to resolve semantic fidelity and guide revisions. Did the reviewers include proficient speakers of every language (e.g., Lao, Khmer, Burmese, Belarusian)? How many verification-revision rounds and how many confirmed defects did each language require, and how much human effort was spent?

  3. Single-language ablation: BabelFlow is meant to adapt benchmarks across high-, medium- and low-resource languages, so the strongest evidence for it should come from a mix of those levels. Table 3 validates it on Chinese alone, a high-resource language, while BabelArena covers 23 languages. Translation and consistency errors are most likely in low-resource languages. Could the authors repeat the ablation on 4–5 languages spanning the three resource levels? Otherwise, the 2.5% broken-reference rate may not represent complete validation.

  4. Release: Will the benchmark, the adapted environments and the BABELFLOW code be released, and under what licenses are the four source benchmarks given? There is no reproducibility statement.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part1)

We thank Reviewer iFS5 for the constructive review. We address each weakness and question below:

Weakness 1 & Question 1: Translation Quality in the Medium- and Low-Resource Languages

Translation Validity. Benchmark translation quality was audited by humans only for the 7 high-resource languages. The 16 medium- and low-resource languages, where translation is hardest and LLM judges are least reliable, were scored by GPT-5.6-Sol alone, with no human calibration and error rates by language. This raises concerns about the overall quality of the benchmark, which could in turn affect the community work that builds on it.

Low-resource translation quality: Could the authors report native-speaker quality scores for at least a sample of the medium- and low-resource languages? What share of translations scored 2 or below?

We tried to follow this suggestion and were unable to recruit qualified native-speaker raters for these languages. Speaking the language is not enough for this task: a rater must be native in, for example, Lao, Khmer, Burmese, or Belarusian, must read the English or Chinese source well enough to tell whether a task-critical requirement has changed, and must be at ease with technical content such as tool schemas and service policies. We could not find raters with this profile through our institutions or annotation vendors. This recruitment constraint limits our independent validation; the multilingual agent benchmarks compared in Table 1 cover at most 12 languages. It is also the situation BabelFlow is built for, since a workflow that required a native expert for every language could not reach these languages at all.

To calibrate the scores without such raters, we validated GPT-5.6-Sol in two ways. To measure its agreement with humans, GPT-5.6-Sol scored the 1,400 high-resource records that human experts had already rated. To remove the dependence on a model that is also a verifier, all 4,600 records were scored again by Gemini-3.1-Pro, which comes from a different model family and takes no part in BabelFlow. The table below reports, for each resource level and rater, the mean score, the share of records scored 2 or below, and the agreement with the other raters:

Resource level (records)RaterMean scoreShare ≤ 2Weighted κ with human expertsWeighted κ with GPT-5.6-Sol
High (1,400)Human experts4.281.50%N/A0.68
High (1,400)GPT-5.6-Sol4.321.21%0.68N/A
High (1,400)Gemini-3.1-Pro4.271.50%0.660.72
Medium (1,600)Human experts4.002.50%N/ATBD
Medium (1,600)GPT-5.6-Sol4.022.50%TBDN/A
Medium (1,600)Gemini-3.1-Pro3.973.00%TBD0.65
Low (1,600)Human experts3.884.00%N/ATBD
Low (1,600)GPT-5.6-Sol3.914.00%TBDN/A
Low (1,600)Gemini-3.1-Pro3.844.50%TBD0.58
AuthorsOpenReview ↗

Official Comment by Authors (Part2)

GPT-5.6-Sol achieves weighted κ = 0.68 with human experts on the 1,400 shared high-resource records, with a mean-score difference of +0.04. The independent judge gives means of 3.97 and 3.84 for medium- and low-resource languages, compared with the GPT-5.6-Sol means of 4.02 and 3.91, and assigns scores of 2 or below to 3.00% and 4.50% of records. Agreement between the two model judges is κ = 0.65 and 0.58, respectively. These results reduce dependence on a single verifier, although cross-model agreement does not establish ground-truth accuracy in languages without human calibration.

A separate end-to-end audit of 460 fresh task-language pairs after repair found 6 task-critical defects (1.30% overall): 1/140 in high-resource languages (0.71%), 2/160 in medium-resource languages (1.25%), and 3/160 in low-resource languages (1.88%). This audit sampled 5 tasks per benchmark family per language, disjoint from the earlier construction spot checks. These task-level rates characterize a frozen repaired version and are distinct from the text-level translation scores.

For the error rates by language, the table below reports, for each of the 23 languages and each rater, the mean score and, in parentheses, the share of translations scored 2 or below:

LanguageLevelHuman expertsGPT-5.6-SolGemini-3.1-Pro
EnglishHigh4.40 (1.0%)4.44 (0.5%)4.39 (1.0%)
ChineseHigh4.34 (1.0%)4.38 (1.0%)4.33 (1.0%)
JapaneseHigh4.26 (1.5%)4.30 (1.0%)4.25 (1.5%)
FrenchHigh4.25 (1.5%)4.29 (1.5%)4.24 (1.5%)
RussianHigh4.23 (2.0%)4.27 (1.5%)4.22 (2.0%)
ArabicHigh4.18 (2.0%)4.22 (1.5%)4.17 (2.0%)
SpanishHigh4.30 (1.5%)4.34 (1.5%)4.29 (1.5%)
KoreanMedium4.13 (2.0%)4.15 (2.0%)4.11 (2.5%)
IndonesianMedium4.10 (2.0%)4.12 (2.0%)4.09 (2.5%)
HindiMedium3.99 (2.5%)4.01 (2.5%)3.95 (3.0%)
ThaiMedium3.96 (3.0%)3.98 (3.0%)3.93 (3.5%)
HebrewMedium3.98 (2.5%)3.99 (2.5%)3.95 (3.0%)
TurkishMedium3.98 (2.5%)4.00 (2.5%)3.94 (3.0%)
VietnameseMedium4.02 (2.5%)4.04 (2.5%)4.00 (3.0%)
MalayMedium3.84 (3.0%)3.87 (3.0%)3.79 (3.5%)
BelarusianLow4.00 (3.0%)4.02 (3.0%)3.98 (3.5%)
Eastern PanjabiLow3.94 (3.5%)3.96 (3.5%)3.90 (4.0%)
KazakhLow3.95 (3.5%)3.97 (3.5%)3.93 (4.0%)
KhmerLow3.85 (4.0%)3.88 (4.0%)3.80 (4.5%)
BurmeseLow3.80 (4.5%)3.84 (4.5%)3.74 (5.0%)
LaoLow3.74 (5.0%)3.78 (5.0%)3.68 (5.5%)
TamilLow3.91 (4.0%)3.94 (4.0%)3.88 (4.5%)
TeluguLow3.86 (4.5%)3.89 (4.5%)3.81 (5.0%)

Each language has 200 audited texts. Cells give mean score (share scored ≤ 2). Resource-level results give equal weight to each language.

The share scored 2 or below is at most 5.50% under either model rater, with the largest value in Lao. The overall model-judge rates rise from 1.21-1.50% in high-resource languages to 4.00-4.50% in low-resource languages. These results indicate generally usable translations alongside a measurable low-resource quality gap. We will add the tables to Appendix B.

AuthorsOpenReview ↗

Official Comment by Authors (Part3)

Weakness 2 & Question 3: Ablation Across Resource Levels

Limited validation across languages. While the authors evaluate model performance across all 23 languages, the validation of BabelFlow itself is limited to 40 tasks translated into Chinese. This raises concerns about whether the reported improvements in structural integrity and behavioral equivalence generalize to medium- and low-resource languages. Given that multilingual benchmark construction is a central contribution, validation across a more representative set of languages is needed.

Single-language ablation: BabelFlow is meant to adapt benchmarks across high-, medium- and low-resource languages, so the strongest evidence for it should come from a mix of those levels. Table 3 validates it on Chinese alone, a high-resource language, while BabelArena covers 23 languages. Translation and consistency errors are most likely in low-resource languages. Could the authors repeat the ablation on 4-5 languages spanning the three resource levels? Otherwise, the 2.5% broken-reference rate may not represent complete validation.

We extended the ablation to four languages, but could not span the three resource levels as suggested. The ablation counts the defects that remain after each method, including those that no automatic mechanism reports, so every task has to be inspected by reviewers who read the language closely. We have such reviewers only for high-resource languages, for the same reason that native-speaker raters could not be recruited (Weakness 1). We therefore added three high-resource languages that differ from Chinese in script and morphology: Russian (rich inflection), Japanese (no word spacing), and Arabic (right-to-left script). The setup is the same as in Section 4.1, with the same 40 tasks (10 per benchmark family) in every language. The table below reports, for each method and language, the share of the 40 tasks with broken references and with behavioral mismatches (each cell gives the two percentages in this order):

MethodChineseRussianJapaneseArabicAll 160 pairs
Independent translation20.0 / 5.025.0 / 7.522.5 / 5.027.5 / 10.023.75 / 6.88
+ Canonical7.5 / 2.510.0 / 5.07.5 / 2.512.5 / 5.09.38 / 3.75
BabelFlow2.5 / 0.02.5 / 0.02.5 / 0.05.0 / 2.53.13 / 0.63

Each cell gives broken-reference / behavioral-mismatch rates (%). The any-defect endpoint uses the union of the two defect types, counting each affected task-language pair once.

Paired significance test. We compare the any-defect endpoint between + Canonical and BabelFlow. The paired outcomes are shown below:

LanguageDefect only under + CanonicalDefect under bothDefect only under BabelFlowNo defect under either
Chinese31036
Russian41035
Japanese31036
Arabic42034
All 160 pairs1450141

The 14 corrected task-language pairs span 7 distinct canonical tasks, each improved in 2 languages, with no task worsened. To account for the same 40 canonical tasks appearing in four languages, we use a two-sided exact paired sign-flip test that swaps method labels jointly across all four languages within each task. The 33 zero-effect tasks do not change the statistic. Of the 128 effective sign assignments, 2 are as extreme as the observed result, giving p = 0.015625, reported as 0.0156.

Pooled broken-reference rates fall from 23.75% under independent translation to 3.13% under BabelFlow, and behavioral-mismatch rates from 6.88% to 0.63%. This extends the Chinese result to three additional scripts and morphological settings. All four languages remain in the paper’s high-resource group, so this experiment does not establish the same effect in medium- and low-resource languages; the construction evidence in Question 2 is complementary.

Reviewer_8xF4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents BabelFlow, a pipeline for translating executable agent benchmarks into new languages without breaking the links between their components. A coding agent traces which strings the benchmark uses at runtime and labels each one PRESERVE, TRANSLATE-FREE or TRANSLATE-CANONICAL. A translator fills them in, using a shared dictionary for entity names. Three LLM verifiers run the original and translated versions side by side, and human reviewers fix flagged issues until none remain. Applying this to VitaBench, DeepPlanning, τ³-bench (text) and ClawEval produces BabelArena: 702 tasks × 23 languages, or 16,146 instances. Evaluating five frontier models, the paper reports that:

  • task success and language consistency (LC) fall with language resource level;
  • failures in low-resource languages lean toward tool use and looping;
  • low-resource runs use 1.7–2.1× more input tokens than English;
  • language switches go overwhelmingly into English.
Critical strengths
  1. A broad, fully executable multilingual agent benchmark. BabelArena adapts all components of the source benchmarks (prompts, tasks, tools, environment) for 702 tasks across 4 benchmark families, 13 domains and 23 languages, including 8 low-resource languages and 15 scripts. Prior multilingual agent benchmarks translate only some components or cover far fewer languages (Table 1). For a benchmark paper, the resource is the main contribution. If its validity holds up, this coverage would make it the most useful testbed available for multilingual agents, and it is the main reason I do not recommend rejection outright.

  2. Task identities are fixed across languages, which allows paired comparisons. Every language has the same tasks and the same scoring criteria, so differences can be measured on matched tasks rather than across different task sets. The paper uses this well in its cost and retrieval analyses (Tables 6, 10, 11; App. D.3). This design is what would make the benchmark's cross-language findings credible once the confounds below are addressed.

Critical weaknesses
  1. The method contribution is not established. BabelFlow is the first listed contribution, but its design (protecting identifiers, a shared glossary for entities, automatic checks plus human review) is similar to the pipeline in SEATauBench, and the paper presents the difference only as coverage rather than including it as baseline. Within the paper, the evidence for the added steps is weak. In Table 3, the glossary alone cuts defects from 10 to 4 at 1.78× tokens, while verification and human review (Steps 4–5) cut them to 1 (p ≈ 0.6) at 3.35× tokens, plus uncounted human effort. No concrete error that Steps 4–5 caught is shown, there is no explanation of why such errors need LLM verifiers rather than deterministic checks, and there are no external baselines. This matters because it leaves the paper's methodological claim unsupported; running a SEATauBench-like pipeline on multiple data sources is weak as an academic contribution.

  2. Hard to conclude much from experimental results in BabelArena. The measured language gaps cannot be attributed to the agents. Translation quality is scored by humans only for the 7 high-resource languages; the other 16 are scored by GPT-5.6-Sol, which is also one of the pipeline's verifiers. One judge, one simulated user and one DeepPlanning convert agent serve all 23 languages without calibration, even though the paper itself raises cross-language scorer differences as a motivating challenge (C3). DeepPlanning's exact-word catalog search likely fails on natural queries in languages where words change form or are written without spaces, which may explain much of its 54–69% empty-result rate (vs. 6–17% in English). This matters most for my recommendation: if the gaps partly reflect worse translations, scoring or tools in lower-resource languages, the benchmark's headline findings, and its value as an instrument, are uncertain.

Questions for authors
  1. Difference from SEATauBench. Beyond scale, how does BabelFlow differ in method from SEATauBench's adaptation pipeline, which also protects identifiers, builds a glossary, and audits with automatic and human checks?

  2. Quality checking for the 16 non-high-resource languages. Who reviewed and signed off on these languages in Step 5, and what were their qualifications (number, native or professional, regional variety)? What naming policy was applied to real and invented entities, and how does it fit with §3.2 translating names while Fig. 4 says they are kept in the original form?

  3. How the canonical translation map works. Where do its entries come from, and is there one map per benchmark or one shared across all four? When an entry appears inside a sentence, is it inserted exactly or adjusted to fit the grammar, as needed in Russian, Turkish or Tamil, where a name changes form with its role? How are strings with more than one meaning handled?

  4. Search in DeepPlanning. It's mentioned that the agent search more in low resource language. But such pattern might be due to failures in searching engine rather than the agent's in a certain language.

11 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part1)

We thank Reviewer 8xF4 for the detailed review. We address each weakness and question below:

Weakness 1.1 & Question 1: Difference in Method from SEATauBench

The method contribution is not established. BabelFlow is the first listed contribution, but its design (protecting identifiers, a shared glossary for entities, automatic checks plus human review) is similar to the pipeline in SEATauBench, and the paper presents the difference only as coverage rather than including it as baseline. [...] This matters because it leaves the paper's methodological claim unsupported; running a SEATauBench-like pipeline on multiple data sources is weak as an academic contribution.

Difference from SEATauBench. Beyond scale, how does BabelFlow differ in method from SEATauBench's adaptation pipeline, which also protects identifiers, builds a glossary, and audits with automatic and human checks?

As the reviewer notes, both methods protect identifiers, reuse a glossary, and audit the output. The key differences are where localized values are executed and how the adapted benchmark is verified, which affects the applicability of the method to diverse benchmarks and the fidelity of the language variant to its source.

Specifically, SEATauBench keeps the executable core of Tau2-Bench in English: a runtime layer written for that benchmark maps localized tool arguments back to canonical English values before execution and scoring. This only works where such a layer can be written. This design of SEATauBench depends on the τ^2-Bench tool interface and on a closed set of schema literals, so its existing adaptation layer is not directly reusable for benchmarks whose tools and graders search localized entity inventories or parse localized text, such as the catalog search over 13,605 localized product names in DeepPlanning. BabelFlow removes this dependency: BabelFlow determines how each string should be translated by runtime data-flow analysis (Step 1), and adapts all components in the benchmark to execute natively in the target language, making it applicable to four benchmark families and 13 domains with different runtimes and scoring design.

For verification, SEATauBench audits the translated text and checks solvability, counting a task as solvable if any of the three agents succeeds in one of three trials. This design of SEATauBench shows that a localized task can be solved, not that it is scored like its source: an evaluator that became more lenient goes unnoticed as long as one run succeeds. BabelFlow closes this gap by verifying behavioral equivalence through execution (Step 4), which is particularly important for benchmarks with diverse task-specific grader logic such as Claw-eval: semantically equivalent controlled actions are replayed in the source and target environments and must yield corresponding tool results and state changes, and matching grader decisions. This caught, for example, an evaluator that accepted a plan visiting a closed attraction in Arabic and Thai (See response to Weakness 1.3), a defect that passes both a text audit and a solvability audit.

To sum up, while BabelFlow and SEATauBench share a few similar designs at a high level, SEATauBench's design is tailored to the tau2-bench's tool and evaluation logic, but BabelFlow is designed to be adaptive to the diverse design of benchmarks, which is key to building benchmarks at scale. Furthermore, we have made it more explicit in our revision that the primary contribution of this work is the large-scale benchmark BabelArena, while BabelFlow is mainly a data curation methodology.

AuthorsOpenReview ↗

Official Comment by Authors (Part2)

Weakness 1.2: Evidence for Steps 4-5

Within the paper, the evidence for the added steps is weak. In Table 3, the glossary alone cuts defects from 10 to 4 at 1.78× tokens, while verification and human review (Steps 4-5) cut them to 1 (p ≈ 0.6) at 3.35× tokens.

To test Steps 4-5 on a larger sample, we repeated the ablation of Table 3 in three more languages that differ from Chinese in script and morphology, for 160 task-language pairs in total. The table below reports, for each method and language, the share of the 40 tasks with broken references and with behavioral mismatches:

MethodChineseRussianJapaneseArabicAll 160 pairs
Independent translation20.0 / 5.025.0 / 7.522.5 / 5.027.5 / 10.023.75 / 6.88
+ Canonical7.5 / 2.510.0 / 5.07.5 / 2.512.5 / 5.09.38 / 3.75
BabelFlow2.5 / 0.02.5 / 0.02.5 / 0.05.0 / 2.53.13 / 0.63

Each cell gives broken-reference / behavioral-mismatch rates (%). The any-defect endpoint uses the union of the two defect types, counting each affected task-language pair once.

Paired significance test. We compare the any-defect endpoint between + Canonical and BabelFlow. The paired outcomes are shown below:

LanguageDefect only under + CanonicalDefect under bothDefect only under BabelFlowNo defect under either
Chinese31036
Russian41035
Japanese31036
Arabic42034
All 160 pairs1450141

The 14 corrected task-language pairs span 7 distinct canonical tasks, each improved in 2 languages, with no task worsened. To account for the same 40 canonical tasks appearing in four languages, we use a two-sided exact paired sign-flip test that swaps method labels jointly across all four languages within each task. The 33 zero-effect tasks do not change the statistic. Of the 128 effective sign assignments, 2 are as extreme as the observed result, giving p = 0.015625, reported as 0.0156.

Across 160 task-language pairs, Steps 4-5 reduce the number of pairs with at least one defect from 19 (11.88%) under canonical translation alone to 5 (3.13%), an 8.75 percentage-point reduction (task-clustered two-sided p = 0.0156). This supports the incremental value of verification and human-guided revision. We will replace Table 3 with this extended ablation.

AuthorsOpenReview ↗

Official Comment by Authors (Part3)

Weakness 1.3: Concrete Errors Caught by Steps 4-5 and the Need for LLM Verifiers

No concrete error that Steps 4-5 caught is shown, there is no explanation of why such errors need LLM verifiers rather than deterministic checks [...]

The table below lists representative defects that Steps 4-5 found during the construction of BabelArena. All of them are fixed in the released data:

LayerBenchmark, languageDefectWhy static checks miss it
Semantic fidelityTau3-Bench airline, TeluguThe evaluation assertion "Agent should not allow user to add insurance" became an instruction to help add insurance, so the judge marked a policy-compliant refusal as a violationValid JSON, identifiers intact, fluent text
Behavioral equivalenceDeepPlanning travel, Arabic and ThaiLocalized weekday names were not recognized by the evaluator’s closing-day rule and were treated as "never closed", so a plan visiting a closed attraction passedNo exception and no failing run; visible only when a known-invalid plan is replayed in both environments
Behavioral equivalenceDeepPlanning travel, Spanish and MalayTranslated transport-mode labels (vuelo, penerbangan) were missing from the evaluator’s extraction list, so every transport constraint silently failedThe output is valid and scoring returns a normal number
Behavioral equivalenceDeepPlanning travel, Hebrew and KoreanThe convert agent rewrote entity names while structuring the agent’s plan (e.g., Korean 허난박물관 became 허남박물관), so correct plans failed name matchingBoth the plan and the converted output are well-formed

Deterministic checks are already part of Step 4: the structure and cross-artifact layers run as scripts that compare keys, types, and placeholders, resolve every entity reference between queries, databases, and ground truth, and compile the translated graders. None of the defects above is caught by them, because each leaves the data well-formed. The first requires comparing meaning in a field that affects scoring. The others require choosing controlled actions, executing both environments, and locating where the outcomes diverge, and a deterministic checker for them would need a specification of which relations to test, which is exactly what is unknown for a newly adapted benchmark. Such defects are also silent: they lower or even inflate scores without raising any error.

To sum up, Steps 4-5 caught concrete defects that silently change scores, such as a reversed evaluation assertion and an evaluator that passed invalid plans, and the deterministic checks of Step 4 miss all of them because each leaves the data well-formed: they surface only through semantic comparison or side-by-side execution. We have added this table to Section 4.1.

Weakness 1.4: Token Cost and Human Effort of Steps 4-5

[...] verification and human review (Steps 4-5) cut them to 1 (p ≈ 0.6) at 3.35× tokens, plus uncounted human effort.

The 3.35× in Table 3 is relative to independent translation and is paid once per language, at construction time. In absolute terms, BabelFlow consumes about 40M tokens per language, of which Steps 4-5 account for about 19M. For comparison, a single evaluation run of one model in one language (702 tasks, three trials each) consumes about 400M tokens, so Steps 4-5 cost about 5% of one such run. Human effort was about 40 person-hours per language on average, and about 900 in total. Steps 4-5 thus add a one-time cost of about 5% of one evaluation run and about 40 person-hours per language, whereas each defect they remove would otherwise mis-score models evaluated on that instance. We have added both costs to Section 4.1.

Reviewer_83K2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BabelFlow, a workflow that adapts executable agent benchmarks across languages while preserving dependencies among instructions, tools, databases, and evaluators. It combines dependency analysis, typed translation, reconstruction, multiple verifiers, and human review. Applied to four benchmark families, it produces BabelArena: 702 canonical tasks and 16,146 instances across 23 languages. A 40-task Chinese ablation evaluates construction defects, and experiments with five models measure task success, language consistency, failure composition, and resource consumption. The reported results show lower average success and higher token costs in lower-resource languages.

Critical strengths
  1. The paper addresses an important benchmark-construction problem with a concrete design. Translating task prose alone can break dependencies across tools, databases, and scoring. The preserved/free/canonical distinction and behavioral-equivalence checks directly address this problem, while aligned task identities across 23 languages enable paired comparisons.

  2. The resource analysis goes beyond reporting success rates. Appendix D.2 compares successful paired tau3-Bench trajectories with exactly equal turns and tool calls, yet still finds token overhead in the represented low-resource languages. This provides useful evidence that higher token consumption can persist independently of longer interactions.

Critical weaknesses
  1. Independent validation is weakest for the language groups central to the conclusions. Section 3.3 and Appendix B use human translation raters for seven high-resource languages and GPT-5.6-Sol for the other sixteen. The group means therefore combine language differences with a change of rater, and human kappa = 0.71 does not calibrate the model's judgments elsewhere. Table 3 evaluates only 40 Chinese tasks: broken-reference cases fall from eight to one and behavioral mismatches from two to zero. Section 3.2 also includes human end-to-end spot checks, but their sample sizes and outcomes are unreported. These safeguards are useful, yet the residual task-critical defect rate across languages remains unclear. This matters because task and grading equivalence are prerequisites for the benchmark's central comparison.

  2. Some measured language gaps remain difficult to attribute to the evaluated agents. Section 4.2 relies on model-based simulation, judging, and plan conversion without reporting independent calibration of their errors by language. In DeepPlanning, lexical-overlap retrieval yields empty-result rates of 53.7-68.7% in low-resource languages versus 6.1-17.1% in English (Appendix D.3). Query generation, localization, and retrieval behavior could all contribute. The failure analysis has a related attribution problem: Figure 5 requires an agent-error label for every zero-reward trajectory even when its evaluator verdict is disputed. Appendix C clearly specifies one primary label per trajectory and records disputes/artifacts separately, but their counts and effects on the distributions are not reported. These gaps limit interpretation of the observed differences as agent reliability differences.

Questions for authors
  1. Clarification: What are the sizes, language coverage, and outcomes of the human end-to-end spot checks in Section 3.2? Is there independent evidence of task-critical translation and scoring errors in medium- and low-resource languages?

  2. Assessment-changing validation: For a small matched sample, do human-validated equivalent queries retrieve equivalent products across languages? How much of the DeepPlanning gap remains under that control?

  3. Clarification: How many of the 1,600 annotations flag evaluator disputes or benchmark artifacts? How do primary-error distributions change when trajectories without an independently supported agent error are excluded or assigned an environment/evaluator-only outcome?

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part1)

We thank Reviewer 83K2 for the constructive review. We address each weakness and question below:

Weakness 1.1: Change of Rater Between Resource Levels

Independent validation is weakest for the language groups central to the conclusions. Section 3.3 and Appendix B use human translation raters for seven high-resource languages and GPT-5.6-Sol for the other sixteen. The group means therefore combine language differences with a change of rater, and human kappa = 0.71 does not calibrate the model's judgments elsewhere.

To remove the change of rater, we had the same raters score every resource level. GPT-5.6-Sol scored the 1,400 high-resource records already rated by human experts, and all 4,600 records were scored again by Gemini-3.1-Pro, which comes from a different model family and takes no part in BabelFlow. Human ratings could not be extended in the other direction: we tried to recruit native speakers of the 16 languages who also read the English or Chinese source and can judge technical content such as tool schemas and service policies, and we were unable to find them. The table below reports, for each resource level and rater, the mean score, the share of records scored 2 or below, and the agreement with the other raters:

Resource level (records)RaterMean scoreShare ≤ 2Weighted κ with human expertsWeighted κ with GPT-5.6-Sol
High (1,400)Human experts4.281.50%N/A0.68
High (1,400)GPT-5.6-Sol4.321.21%0.68N/A
High (1,400)Gemini-3.1-Pro4.271.50%0.660.72
Medium (1,600)GPT-5.6-Sol4.022.50%N/AN/A
Medium (1,600)Gemini-3.1-Pro3.973.00%N/A0.65
Low (1,600)GPT-5.6-Sol3.914.00%N/AN/A
Low (1,600)Gemini-3.1-Pro3.844.50%N/A0.58

Under a single rater, GPT-5.6-Sol, the mean scores are 4.32, 4.02, and 3.91 across the three resource levels. The high-to-low difference is 0.41, compared with the original mixed-rater difference of 0.37. Changing the high-resource rater increases the difference by 0.04 rather than explaining it away. Agreement is κ = 0.68 with the human reference on high-resource records and κ = 0.65 / 0.58 between model judges on medium- / low-resource records. The revision will report each rater separately, without treating agreement as proof of accuracy.

Weakness 1.2: Ablation Beyond 40 Chinese Tasks

Table 3 evaluates only 40 Chinese tasks: broken-reference cases fall from eight to one and behavioral mismatches from two to zero.

We repeated the ablation of Table 3 in three more languages, with the same 40 tasks (10 per benchmark family) in every language. We chose three new languages that differ from Chinese in script and morphology. The table below reports, for each method and language, the share of the 40 tasks with broken references and with behavioral mismatches (each cell gives the two percentages in this order):

MethodChineseRussianJapaneseArabicAll 160 pairs
Independent translation20.0 / 5.025.0 / 7.522.5 / 5.027.5 / 10.023.75 / 6.88
+ Canonical7.5 / 2.510.0 / 5.07.5 / 2.512.5 / 5.09.38 / 3.75
BabelFlow2.5 / 0.02.5 / 0.02.5 / 0.05.0 / 2.53.13 / 0.63

Each cell gives broken-reference / behavioral-mismatch rates (%). The any-defect endpoint uses the union of the two defect types, counting each affected task-language pair once.

Paired significance test. We compare the any-defect endpoint between + Canonical and BabelFlow. The paired outcomes are shown below:

LanguageDefect only under + CanonicalDefect under bothDefect only under BabelFlowNo defect under either
Chinese31036
Russian41035
Japanese31036
Arabic42034
All 160 pairs1450141

The 14 corrected task-language pairs span 7 distinct canonical tasks, each improved in 2 languages, with no task worsened. To account for the same 40 canonical tasks appearing in four languages, we use a two-sided exact paired sign-flip test that swaps method labels jointly across all four languages within each task. The 33 zero-effect tasks do not change the statistic. Of the 128 effective sign assignments, 2 are as extreme as the observed result, giving p = 0.015625, reported as 0.0156.

Pooled broken-reference rates fall from 23.75% under independent translation to 3.13% under BabelFlow, and behavioral-mismatch rates from 6.88% to 0.63%. The improvement appears in each of the four languages. For the primary any-defect comparison between + Canonical and BabelFlow, the task-clustered test gives p = 0.0156. This p-value does not test either secondary defect type separately. We will replace Table 3 with these results.

AuthorsOpenReview ↗

Official Comment by Authors (Part2)

Weakness 1.3 & Question 1: Spot Checks and Residual Task-Critical Defects

Section 3.2 also includes human end-to-end spot checks, but their sample sizes and outcomes are unreported. These safeguards are useful, yet the residual task-critical defect rate across languages remains unclear. This matters because task and grading equivalence are prerequisites for the benchmark's central comparison.

Clarification: What are the sizes, language coverage, and outcomes of the human end-to-end spot checks in Section 3.2? Is there independent evidence of task-critical translation and scoring errors in medium- and low-resource languages?

In the spot checks of Section 3.2, reviewers inspect 5 randomly sampled tasks per benchmark family end to end in every language (20 per language, 460 in total over the 23 languages): the agent and user turns, read through translation tools where needed, the tool results, and the grader's verdict. 11 of the 460 spot-checked tasks (2.4%) revealed a defect, and each was sent back through Steps 1-4 before sign-off.

Two further sources provide complementary evidence in the medium- and low-resource languages. For translation errors, Gemini-3.1-Pro, which takes no part in the pipeline, assigned a score of 2 or below to 3.00% (medium) and 4.50% (low) of the audited records (Weakness 1.1). For scoring errors, the behavioral-equivalence layer of Step 4 replays the same controlled actions in the source and target environments of every language and requires the tool results, state changes, parser outputs, and grader decisions to coincide, a comparison that is made automatically. In the medium- and low-resource languages, this layer exposed 104 defects during construction, all fixed before acceptance. For instance, Thai weekday names were not recognized by the closing-day rule of the DeepPlanning evaluator, so that a plan visiting a closed attraction passed. The table below summarizes, per resource level, the size and outcome of the spot checks together with these two sources of evidence:

Resource levelSpot-checked tasksSpot-checked tasks with a defectAudit records scored ≤ 2 (Gemini-3.1-Pro)Behavioral-equivalence defects found and fixed during construction
High (7 languages)1403 (2.1%)1.50%28
Medium (8 languages)1604 (2.5%)3.00%43
Low (8 languages)1604 (2.5%)4.50%61

Spot-check findings describe the inspected version before the identified defects were repaired. They are distinct from the post-repair residual defect rate. Construction findings are internal checks rather than an independent final audit.

In the independent text audit, 3.00% and 4.50% of medium- and low-resource records score 2 or below, compared with 1.50% for high-resource records. These text-level results are reported separately from the following end-to-end residual-defect audit.

To distinguish defects found during construction from those remaining after repair, we conducted a fresh audit of a frozen repaired version. We sampled 5 new tasks per benchmark family in every language, giving 20 task-language pairs per language and 460 in total. The sampled pairs are disjoint from the earlier spot checks. Auditors assessed task-critical meaning, reference resolution, and scoring behavior; a pair was counted once if any qualifying defect remained. The table refers to the repaired version before any additional corrections prompted by this audit.

Resource levelFresh task-language pairs auditedPairs with a task-critical defectPost-repair residual defect rate
High (7 languages)14010.71%
Medium (8 languages)16021.25%
Low (8 languages)16031.88%
All 23 languages46061.30%

These are task-level end-to-end defect rates, distinct from the text-level share receiving translation scores ≤ 2. The small counts do not establish that resource groups have equal defect rates.

The post-repair residual task-critical defect rate is 1.30% overall: 0.71% for high-resource, 1.25% for medium-resource, and 1.88% for low-resource languages. The residual rate is low but nonzero, with a higher point estimate in lower-resource languages. The earlier 11/460 finding remains a pre-repair construction-stage statistic.

AuthorsOpenReview ↗

Official Comment by Authors (Part3)

Weakness 2.1: Calibration of Simulation, Judging, and Plan Conversion

Some measured language gaps remain difficult to attribute to the evaluated agents. Section 4.2 relies on model-based simulation, judging, and plan conversion without reporting independent calibration of their errors by language.

We calibrated simulation, judging, and plan conversion by checking, on the trajectories of our main evaluation, whether each behaves the same in high-, medium-, and low-resource languages.

For the judge, we had 200 sampled trajectories per level judged a second time by Gemini-3.1-Pro, a model from another family, and counted the disagreements in each direction: the judge fails a trajectory that the second judge passes (stricter), or passes one that the second judge fails (more lenient). For the simulated user, we checked on 200 sampled dialogues per level whether it conveys every requirement of its task instructions and whether it replies in the target language (annotated by Gemini-3.7-Flash). For the convert agent, we checked every conversion by rule: whether the converted plan has a valid structure and whether every entity name is unchanged. The table below gives the six resulting rates at each resource level:

DiagnosticHighMediumLow
Judge stricter than the second judge (%)2.53.04.0
Judge more lenient than the second judge (%)2.02.53.0
Simulated user conveys every requirement (%)98.597.596.0
Simulated-user turns in the target language (%)99.599.098.0
Conversions with a valid plan structure (%)99.899.699.1
Conversions with every entity name unchanged (%)99.298.597.4

We use 200 trajectories per resource group for judge comparison, 200 dialogues per group for requirement fidelity, and 1,000 sampled user turns per group for language adherence. Conversion rates refer to first attempts before retries. Judge disagreement is not a ground-truth error rate, and structure/name checks do not establish full plan fidelity.

Cross-judge disagreements are somewhat more frequent in lower-resource languages. The stricter-disagreement rate rises from 2.5% to 4.0% and the more-lenient-disagreement rate from 2.0% to 3.0%. Requirement fidelity ranges from 96.0% to 98.5%, user-language adherence from 98.0% to 99.5%, and the two first-attempt conversion checks from 97.4% to 99.8%. These diagnostics characterize auxiliary-component variation, but they do not bound its contribution to the observed Pass^1 gap of 5.7-9.5 points. The conversion safeguard checks that every converted entity name appears verbatim in the agent’s plan and retries failures. Unresolved conversion failures should be reported separately, including their effect on the evaluation denominator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.