- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces BabelFlow, an agentic procedure for localizing executable agent benchmarks using dependency analysis, typed preservation/free/canonical translation, structure-preserving reconstruction, multi-agent verification and human review. This workflow applies to four source benchmark families to construct BabelArena. This multilingual benchmark comprises 702 canonical tasks, 16,146 language-specific instances, 13 domains and 23 languages. A 40-task Chinese-language ablation reports a reduction in broken references from 20.0% under independent translation to 2.5% with BabelFlow. Five frontier models are then evaluated using task success, three-trial robustness and language consistency.
Critical strengths
- Practically relevant multilingual benchmark contribution. A benchmark-general adaptation workflow that covers the full executable environment. Unlike most prior multilingual agent benchmarks, which translate only the task inputs, BABELFLOW adapts prompts, tasks, tool descriptions and environment content together. It does so through runtime data-flow analysis, typed translation, structure-preserving reconstruction, and behavioral-equivalence checks that execute source and target language side by side.
- Practical Evaluation. The paired analyses (Tables 6–7, 10–11) provide valuable insights beyond task success rates. Notably, lower-resource languages incur substantially higher token costs even when controlling for task success, conversation turns, and tool calls.
Critical weaknesses
-
Translation Validity. Benchmark translation quality was audited by humans only for the 7 high-resource languages. The 16 medium- and low-resource languages, where translation is hardest and LLM judges are least reliable, were scored by GPT-5.6-Sol alone, with no human calibration and error rates by language. This raises concerns about the overall quality of the benchmark, which could in turn affect the community work that builds on it.
-
Limited validation across languages. While the authors evaluate model performance across all 23 languages, the validation of BabelFlow itself is limited to 40 tasks translated into Chinese. This raises concerns about whether the reported improvements in structural integrity and behavioral equivalence generalize to medium- and low-resource languages. Given that multilingual benchmark construction is a central contribution, validation across a more representative set of languages is needed.
Questions for authors
-
Low-resource translation quality: Could the authors report native-speaker quality scores for at least a sample of the medium- and low-resource languages? What share of translations scored 2 or below?
-
Human review capacity: Figure 1, Step 5 requires human reviewers to resolve semantic fidelity and guide revisions. Did the reviewers include proficient speakers of every language (e.g., Lao, Khmer, Burmese, Belarusian)? How many verification-revision rounds and how many confirmed defects did each language require, and how much human effort was spent?
-
Single-language ablation: BabelFlow is meant to adapt benchmarks across high-, medium- and low-resource languages, so the strongest evidence for it should come from a mix of those levels. Table 3 validates it on Chinese alone, a high-resource language, while BabelArena covers 23 languages. Translation and consistency errors are most likely in low-resource languages. Could the authors repeat the ablation on 4–5 languages spanning the three resource levels? Otherwise, the 2.5% broken-reference rate may not represent complete validation.
-
Release: Will the benchmark, the adapted environments and the BABELFLOW code be released, and under what licenses are the four source benchmarks given? There is no reproducibility statement.
6 comments
Official Comment by Authors (Part2)
GPT-5.6-Sol achieves weighted κ = 0.68 with human experts on the 1,400 shared high-resource records, with a mean-score difference of +0.04. The independent judge gives means of 3.97 and 3.84 for medium- and low-resource languages, compared with the GPT-5.6-Sol means of 4.02 and 3.91, and assigns scores of 2 or below to 3.00% and 4.50% of records. Agreement between the two model judges is κ = 0.65 and 0.58, respectively. These results reduce dependence on a single verifier, although cross-model agreement does not establish ground-truth accuracy in languages without human calibration.
A separate end-to-end audit of 460 fresh task-language pairs after repair found 6 task-critical defects (1.30% overall): 1/140 in high-resource languages (0.71%), 2/160 in medium-resource languages (1.25%), and 3/160 in low-resource languages (1.88%). This audit sampled 5 tasks per benchmark family per language, disjoint from the earlier construction spot checks. These task-level rates characterize a frozen repaired version and are distinct from the text-level translation scores.
For the error rates by language, the table below reports, for each of the 23 languages and each rater, the mean score and, in parentheses, the share of translations scored 2 or below:
| Language | Level | Human experts | GPT-5.6-Sol | Gemini-3.1-Pro |
|---|---|---|---|---|
| English | High | 4.40 (1.0%) | 4.44 (0.5%) | 4.39 (1.0%) |
| Chinese | High | 4.34 (1.0%) | 4.38 (1.0%) | 4.33 (1.0%) |
| Japanese | High | 4.26 (1.5%) | 4.30 (1.0%) | 4.25 (1.5%) |
| French | High | 4.25 (1.5%) | 4.29 (1.5%) | 4.24 (1.5%) |
| Russian | High | 4.23 (2.0%) | 4.27 (1.5%) | 4.22 (2.0%) |
| Arabic | High | 4.18 (2.0%) | 4.22 (1.5%) | 4.17 (2.0%) |
| Spanish | High | 4.30 (1.5%) | 4.34 (1.5%) | 4.29 (1.5%) |
| Korean | Medium | 4.13 (2.0%) | 4.15 (2.0%) | 4.11 (2.5%) |
| Indonesian | Medium | 4.10 (2.0%) | 4.12 (2.0%) | 4.09 (2.5%) |
| Hindi | Medium | 3.99 (2.5%) | 4.01 (2.5%) | 3.95 (3.0%) |
| Thai | Medium | 3.96 (3.0%) | 3.98 (3.0%) | 3.93 (3.5%) |
| Hebrew | Medium | 3.98 (2.5%) | 3.99 (2.5%) | 3.95 (3.0%) |
| Turkish | Medium | 3.98 (2.5%) | 4.00 (2.5%) | 3.94 (3.0%) |
| Vietnamese | Medium | 4.02 (2.5%) | 4.04 (2.5%) | 4.00 (3.0%) |
| Malay | Medium | 3.84 (3.0%) | 3.87 (3.0%) | 3.79 (3.5%) |
| Belarusian | Low | 4.00 (3.0%) | 4.02 (3.0%) | 3.98 (3.5%) |
| Eastern Panjabi | Low | 3.94 (3.5%) | 3.96 (3.5%) | 3.90 (4.0%) |
| Kazakh | Low | 3.95 (3.5%) | 3.97 (3.5%) | 3.93 (4.0%) |
| Khmer | Low | 3.85 (4.0%) | 3.88 (4.0%) | 3.80 (4.5%) |
| Burmese | Low | 3.80 (4.5%) | 3.84 (4.5%) | 3.74 (5.0%) |
| Lao | Low | 3.74 (5.0%) | 3.78 (5.0%) | 3.68 (5.5%) |
| Tamil | Low | 3.91 (4.0%) | 3.94 (4.0%) | 3.88 (4.5%) |
| Telugu | Low | 3.86 (4.5%) | 3.89 (4.5%) | 3.81 (5.0%) |
Each language has 200 audited texts. Cells give mean score (share scored ≤ 2). Resource-level results give equal weight to each language.
The share scored 2 or below is at most 5.50% under either model rater, with the largest value in Lao. The overall model-judge rates rise from 1.21-1.50% in high-resource languages to 4.00-4.50% in low-resource languages. These results indicate generally usable translations alongside a measurable low-resource quality gap. We will add the tables to Appendix B.
Official Comment by Authors (Part3)
Weakness 2 & Question 3: Ablation Across Resource Levels
Limited validation across languages. While the authors evaluate model performance across all 23 languages, the validation of BabelFlow itself is limited to 40 tasks translated into Chinese. This raises concerns about whether the reported improvements in structural integrity and behavioral equivalence generalize to medium- and low-resource languages. Given that multilingual benchmark construction is a central contribution, validation across a more representative set of languages is needed.
Single-language ablation: BabelFlow is meant to adapt benchmarks across high-, medium- and low-resource languages, so the strongest evidence for it should come from a mix of those levels. Table 3 validates it on Chinese alone, a high-resource language, while BabelArena covers 23 languages. Translation and consistency errors are most likely in low-resource languages. Could the authors repeat the ablation on 4-5 languages spanning the three resource levels? Otherwise, the 2.5% broken-reference rate may not represent complete validation.
We extended the ablation to four languages, but could not span the three resource levels as suggested. The ablation counts the defects that remain after each method, including those that no automatic mechanism reports, so every task has to be inspected by reviewers who read the language closely. We have such reviewers only for high-resource languages, for the same reason that native-speaker raters could not be recruited (Weakness 1). We therefore added three high-resource languages that differ from Chinese in script and morphology: Russian (rich inflection), Japanese (no word spacing), and Arabic (right-to-left script). The setup is the same as in Section 4.1, with the same 40 tasks (10 per benchmark family) in every language. The table below reports, for each method and language, the share of the 40 tasks with broken references and with behavioral mismatches (each cell gives the two percentages in this order):
| Method | Chinese | Russian | Japanese | Arabic | All 160 pairs |
|---|---|---|---|---|---|
| Independent translation | 20.0 / 5.0 | 25.0 / 7.5 | 22.5 / 5.0 | 27.5 / 10.0 | 23.75 / 6.88 |
| + Canonical | 7.5 / 2.5 | 10.0 / 5.0 | 7.5 / 2.5 | 12.5 / 5.0 | 9.38 / 3.75 |
| BabelFlow | 2.5 / 0.0 | 2.5 / 0.0 | 2.5 / 0.0 | 5.0 / 2.5 | 3.13 / 0.63 |
Each cell gives broken-reference / behavioral-mismatch rates (%). The any-defect endpoint uses the union of the two defect types, counting each affected task-language pair once.
Paired significance test. We compare the any-defect endpoint between + Canonical and BabelFlow. The paired outcomes are shown below:
| Language | Defect only under + Canonical | Defect under both | Defect only under BabelFlow | No defect under either |
|---|---|---|---|---|
| Chinese | 3 | 1 | 0 | 36 |
| Russian | 4 | 1 | 0 | 35 |
| Japanese | 3 | 1 | 0 | 36 |
| Arabic | 4 | 2 | 0 | 34 |
| All 160 pairs | 14 | 5 | 0 | 141 |
The 14 corrected task-language pairs span 7 distinct canonical tasks, each improved in 2 languages, with no task worsened. To account for the same 40 canonical tasks appearing in four languages, we use a two-sided exact paired sign-flip test that swaps method labels jointly across all four languages within each task. The 33 zero-effect tasks do not change the statistic. Of the 128 effective sign assignments, 2 are as extreme as the observed result, giving p = 0.015625, reported as 0.0156.
Pooled broken-reference rates fall from 23.75% under independent translation to 3.13% under BabelFlow, and behavioral-mismatch rates from 6.88% to 0.63%. This extends the Chinese result to three additional scripts and morphological settings. All four languages remain in the paper’s high-resource group, so this experiment does not establish the same effect in medium- and low-resource languages; the construction evidence in Question 2 is complementary.
Official Comment by Authors (Part1)
We thank Reviewer iFS5 for the constructive review. We address each weakness and question below:
Weakness 1 & Question 1: Translation Quality in the Medium- and Low-Resource Languages
We tried to follow this suggestion and were unable to recruit qualified native-speaker raters for these languages. Speaking the language is not enough for this task: a rater must be native in, for example, Lao, Khmer, Burmese, or Belarusian, must read the English or Chinese source well enough to tell whether a task-critical requirement has changed, and must be at ease with technical content such as tool schemas and service policies. We could not find raters with this profile through our institutions or annotation vendors. This recruitment constraint limits our independent validation; the multilingual agent benchmarks compared in Table 1 cover at most 12 languages. It is also the situation BabelFlow is built for, since a workflow that required a native expert for every language could not reach these languages at all.
To calibrate the scores without such raters, we validated GPT-5.6-Sol in two ways. To measure its agreement with humans, GPT-5.6-Sol scored the 1,400 high-resource records that human experts had already rated. To remove the dependence on a model that is also a verifier, all 4,600 records were scored again by Gemini-3.1-Pro, which comes from a different model family and takes no part in BabelFlow. The table below reports, for each resource level and rater, the mean score, the share of records scored 2 or below, and the agreement with the other raters: