acceptodds
Under review as a conference paper at ICLR 2027

Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

Abstract

Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (ρ = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multilingual Large Language Models, Multilingual Translation, Multilingual Foundation Models, Large Language Models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_yVgLOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes evaluating multilingual capability through round-trip translation. Given a passage in a source language, round-trip translation involves translating it to a target language and back, comparing the result to the original passage. Semantic gaps between the original and back-translated text show whether a model can preserve meaning across languages. They argue that existing benchmarks measure factual recall and mathematical reasoning rather than multilingual proficiency. The paper computes Spearman rank-order correlations between benchmark scores and LMArena to show that these scores don’t correlate with user preference.

They introduce Lost in Translation (LiT), a round-trip translation benchmark of 1600 samples across 8 language sequences spanning high-, medium-, and low resource languages. They show that round-trip translation scores on LiT correlate almost perfectly (ρ = 0.94) with LMArena, suggesting closer alignment with real-world multilingual performance.

Critical strengths
  1. The paper analyzes widely used benchmarks (MT-AIME24 and INCLUDE) and the errors models make on them. It finds that errors mostly reflect gaps in reasoning or knowledge rather than misunderstanding of the language, and that models reason in English even when prompted in another language. These benchmarks therefore conflate reasoning and knowledge with multilingual understanding. This is an important finding, and it motivates benchmarks that can separate the two.

  2. . Round-trip translation is easy to apply at scale. It needs no human reference translations or per-sample annotation. Because the judge only compares two English texts, it also avoids the need for a judge model stronger than the tested models in each low-resource language. The benchmark covers over 30 languages across high, medium and low-resource levels.

Critical weaknesses
  1. The LMArena results in Section 3, Figure 2 are based on 6 models with several models scoring pretty close to each other. With only 6 data points, the spearman score of 0.94 would change noticeably with one swapper pair. There might also be a confounder where arena users may prefer instruct models over thinking models due to speed or style. Benchmarks that favor thinking will not correlate with LMArena scores in such cases. Plotting this correlation for more models would make this result more convincing.

  2. There is no intermediate evaluation in the target language. The benchmark only evaluates the back translated English text. The quality of text, fluency, naturalness in the target language is never evaluated. Adding additional metrics to verify the quality of the translated text would be useful.

Questions for authors
  1. Footnote 4 says Japanese is missing for both DeepSeek variants and French for Qwen3-235B (Thinking), so are modes averaged over different sets of languages?
  2. The paper evaluates 22 configurations against the new benchmark, why are these not used for Sec 3, LMArena ranking comparison?
1 comment

We thank the reviewer for the thoughtful and encouraging assessment. We appreciate that the reviewer recognizes both the benchmark analysis and the scalability of round-trip translation. We address the concerns below:

W1. On the small LMArena comparison and preference confounds.

We agree that the same LMArena correlation could be improved and extend the figure below. We will revise Appendix A.3 accordingly. However, we also provide results on judge sensitivity already in Section 4.2, Table 4 and Appendix G.1: Grok 4.1 Fast, DeepSeek-V4-Pro and Gemini-3.1-Flash-Lite agree across ten model configurations with high Pairwise Spearman correlations are 0.96-0.99, with Kendall's W = 0.987. However, we agree that this can be further strengthened by using human studies. We therefore add an independent comparison against WMT26 [1] human cESA translation annotations for documents, seen here:

https://imgur.com/a/MLVqHBf.

Across 11 models covering seven languages, MQM≥80 pass rates correlate with human ratings at Spearman ρ=0.952 (Pearson r=0.93). Therefore, we can see that the WMT26 human ratings highly correlate with the annotations used by our benchmark approach.

W2. On intermediate translation quality.

The reviewer is correct that final English translation does not certify the fluency, naturalness or register of every intermediate translation. Appendix C explicitly acknowledges this, and Appendix A.5 identifies target-language fluency and individual-language attribution as limitations. We will state this scope more prominently.

However, we would also like to highlight the existing 480-sample stress test of five classical translation challenges (Appendix C, Tables 8-9), whose rankings correlate strongly with LiT (average ρ = 0.946). Section 4.2 additionally checks three judges (Table 4) and chain length (Table 5; Appendix G.2). These support the robustness of our benchmark.

Q1. On missing languages in the LMArena averages.

We agree that this limitation affects results. Footnote 4 states that the submitted comparison averages available ratings: Japanese is missing for both DeepSeek variants and French for Qwen3-235B (Thinking). The means therefore use different language sets. We will make this limitation explicit and have generated a new, expanded figure with more models that can be seen here, which now use the same exact languages (DE, ES, KO, RU, ZH), available across all models: https://imgur.com/a/68ubc7I. We thank the reviewer for providing this feedback and will adjust our revision accordingly.

Q2. Why not use all 22 LiT configurations?

The 22 main evaluation uses multi-language translation chains, whereas Section 3 evaluates separate English→language→English round trips matched to language-specific Arena ratings. The chain scores therefore cannot simply be substituted into that comparison. The original six configurations were selected from a comparable capability tier, retaining thinking/non-thinking variants. To extend the number of models we use, we have now 14 models seen here: https://imgur.com/a/68ubc7I . The expanded comparison uses identical language coverage and a uniform DeepSeek V4 Pro judge.

Overall, we hope these clarifications address the concerns and are happy to answer further questions.

Reviewer_oChrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that existing multilingual benchmarks primarily measure reasoning and factual knowledge rather than multilingual generation ability. It proposes round-trip translation as an alternative evaluation method and introduces a benchmark for it.

Critical strengths

The idea for using round-trip translation as a proxy for evaluating model's multilingual capability is smart. It is interesting to see the high correlation between LMArena score and the benchmark proposed in the paper.

Critical weaknesses
  • I think there are a couple of correlations missing: existing translation benchmarks vs. LM Arena scores, existing translation benchmarks vs. multilingual benchmarks (INCLUDE, MT-AIME). Though the idea is cute, I think it's worth to understand these two dimensions to fully conclude that LiT is a reliable way to evaluate LLMs.
  • The claim about reasoning errors are way too strong and there is insufficient causal evidence. While the authors mention that the main errors in multilingual reasoning is non language-focused but rather logical/factual. It is yet to be proven that multilingual inputs do not induce these factual errors. And only by observing mostly English traces do not necessarily establish that multilingual reasoning benchmarks measure only English reasoning ability.
  • I think the paper could use better overall organization. While the body text is largely occupied by huge tables covering many different LLMs. The most interesting part on the specific design of the benchmarks are largely in the Appendix, which make is really hard to understand how the data is constructed and what exactly is the MQM score.
Questions for authors

I don't fully understand why computing the correlation between a specific multilingual benchmark and LM Arena. I think LM Arena covers many aspects of LLM response quality (or is the elo score specifically relate to multilingual?) and low correlation with a specific benchmark could be expected. I wonder if author could share more insights on why specifically LiT could inform model's overall capability.

2 comments

We thank the reviewer for the thoughtful feedback and for recognizing that our idea is cute. We appreciate that the reviewer finds the round-trip approach useful and its alignment with multilingual preferences interesting. We address the concerns below:

W1. On comparisons with existing translation benchmarks.

We agree that direct translation benchmarks are an important comparison. We additionally compare official WMT26 human cESA translation-quality ratings with language-specific LMArena ratings. We evaluate 5 models available in WMT26 and across the available languages DE, JA, KO, RU and ZH, which gives us a Spearman correlation of 1.0:

https://imgur.com/a/7WqAEkH

We also validate our round-trip procedure using official WMT26 translations with existing human cESA ratings for 11 models, which can be seen here:

https://imgur.com/a/MLVqHBf.

Across 11 models covering seven languages, MQM≥80 pass rates correlate with human ratings at Spearman ρ=0.952 (Pearson r=0.93). Therefore, we can see that the WMT26 human ratings highly correlate with the annotations used by our benchmark approach.

We will revise our paper accordingly with these additional results.

W2. On causal claims about reasoning and factual errors.

We agree that English reasoning traces alone cannot establish that multilingual inputs do not induce downstream factual or reasoning errors. Our evidence supports the narrower conclusion that these benchmark scores conflate language proficiency with reasoning and factual knowledge; improvements cannot be interpreted as isolated gains in multilingual generation.

This conclusion rests on complementary observations: MT-AIME24 correlates with English AIME25 (ρ = 0.94), INCLUDE correlates with MMLU-Pro (ρ = 0.83; Figure 2), and manual error analysis finds predominantly logical/factual rather than explicit comprehension failures (Section 3.2, Figure 3). For example, over 96% of the analyzed INCLUDE errors in Bengali and Chinese are factual. Appendices E-F extend the trace and error analyses across models. We believe this should be extensive evidence for our claims and will revise the claims to be more narrow.

W3. On organization and the MQM score.

We thank the reviewer for pointing this out. We will further improve the layout and presentation, however, we are surprised by this argument, considering that the reviewer, including all other reviewers, gave the highest score for presentation. We believe this shouldn’t be grounds for a rejection.

Q1. On what the LMArena comparison measures.

We thank the reviewer for this comment. We would like to clarify the reviewer’s confusion here and will make this clearer in the revision. LMArena specifically provides public results for multilingual capabilities under https://arena.ai/leaderboard/chat/text, under the language tab. We use language-specific LMArena ratings averaged over available Chinese, French, German, Japanese, Korean, Russian and Spanish leaderboards (Section 3.1, footnote 4). This compares benchmark rankings with preferences for multilingual outputs. We also provide the results in the figure above (https://imgur.com/a/MLVqHBf) for the correlation with WMT26 human annotations, showcasing that our setup strongly aligns with human preferences as well.

Overall, we hope these clarifications address the main concerns and are happy to discuss any remaining issues.

Thank you for your response and the additional results. I decided to keep my original score.

Reviewer_P3fkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether existing multilingual benchmarks adequately measure the multilingual generation capabilities of large language models. Through correlations between benchmarks and analyses of error types, the authors argue that multilingual mathematics and knowledge benchmarks jointly reflect language proficiency, reasoning ability, and factual knowledge, making it difficult to assess multilingual proficiency in isolation. To address this issue, the paper introduces LiT, a benchmark that uses round-trip translation to evaluate meaning preservation across languages without requiring human reference translations. LiT consists of 200 English passages covering academic abstracts, pragmatic phenomena, and informal language. Each passage is evaluated along eight translation chains, each passing through four intermediate languages before returning to English, yielding 1,600 evaluation samples. An LLM judge then compares the original and back-translated texts using the MQM scoring framework.

The experiments show that, among the models evaluated, LiT aligns more closely with multilingual user preferences than the mathematics and knowledge benchmarks used for comparison, revealing differences in generation ability that existing benchmarks struggle to capture. Models still exhibit substantial weaknesses in low-resource languages and informal expression, and enabling reasoning does not consistently improve translation quality. Further robustness analyses support the applicability of this evaluation approach across different judge models and evaluation settings. Overall, the work proposes a scalable method for evaluating multilingual generation, with a focus on semantic fidelity across languages.

Critical strengths
  1. The paper combines analyses of benchmark correlations, error types, and the languages used in reasoning to reveal how existing multilingual evaluations conflate language proficiency with reasoning ability and factual knowledge. These complementary analyses provide valuable evidence for designing evaluations that focus more directly on actual language generation capabilities.

  2. By using round-trip translation, the method turns the evaluation of meaning preservation across languages into a comparison between English texts, reducing reliance on human reference translations. The benchmark covers languages with different levels of resource availability and multiple text types, and includes sensitivity analyses of judge models and translation-chain lengths, making the evaluation design reasonably comprehensive.

Critical weaknesses
  1. The validation of the evaluation method is not yet sufficiently comprehensive. The paper primarily validates LiT's alignment with user preferences through simple round-trip translation involving languages with relatively abundant resources, whereas the main benchmark uses translation chains that include low-resource languages. Given this difference in setup, direct validation is still needed to establish whether scores from translation chains involving low-resource languages agree with human judgments of translation quality.

  2. Comparing the original English text with the final back-translation may miss unnatural phrasing, register mismatches, or errors in intermediate translations that are corrected in subsequent steps. Chaining multiple languages also makes it difficult to identify which language or translation step causes an error. Although the authors provide challenging examples and robustness analyses, consistency in model rankings does not directly rule out these issues. Bilingual human assessment or independent validation of the quality of intermediate translations is still needed.

Questions for authors
  1. The current validation of alignment with user preferences primarily uses simple round-trip translation involving languages with relatively abundant resources. Could the authors provide a small human evaluation of the main benchmark's translation chains involving low-resource languages to assess whether LiT scores remain consistent with human judgments of translation quality?

  2. Beyond the qualitative checks in the appendix, could the authors conduct bilingual human assessments of the intermediate translations at each step for a subset of examples, and quantify how often an intermediate translation contains errors despite an apparently correct final back-translation? To what extent do such corrections or masking of errors affect LiT scores?

  3. Is the drop in scores on translation chains involving low-resource languages primarily caused by limited proficiency in a particular language or by errors accumulating over successive translation steps? Could the authors distinguish these effects by evaluating individual translation steps separately or changing the order of languages?

1 comment

We thank the reviewer for the useful feedback and for acknowledging the usefulness, soundness and comprehensiveness of our benchmark. We appreciate that the reviewer recognizes the complementary benchmark analyses and the scalability of our approach through round-trip translation. We address the concerns and questions below:

W1. The validation of the evaluation method is not yet sufficiently comprehensive.

We thank the reviewer for providing this feedback. Section 3.1 adapts the round-trip approach to languages with public LMArena ratings. The full benchmark in Section 4 uses eight regional chains selected for diverse families, scripts and resource availability (Appendix B.2). We would like to highlight that several checks already cover the full benchmark: the three-judge comparison spans all eight sequences and both the main and robustness sets (Section 4.2, Table 4; Appendix G.1), with pairwise rank correlations of 0.96-0.99. Appendix D also checks alternative scoring metrics, showing that the setup is highly robust to the method that is used.

Finally, we also validate whether intermediate hops affect model rankings and show that the benchmark rankings of models is highly consistent regardless of the number of hops performed in Table 5 in the paper.

We additionally provide a comparison against Human annotations from WMT26 [1] human cESA translation annotations for documents, seen here:

https://imgur.com/a/MLVqHBf.

Across 11 models covering seven languages, MQM≥80 pass rates correlate with human ratings at Spearman ρ=0.952 (Pearson r=0.93). Therefore, we can see that the WMT26 human ratings highly correlate with the annotations used by our benchmark approach.

W2. Comparing the original English text with the final back-translation may miss unnatural phrasing, register mismatches, or errors in intermediate translations

The reviewer is correct that a faithful final English text does not certify every intermediate translation. Appendix C explicitly acknowledges this limitation and already provides a targeted 480-sample robustness test covering conceptual nuance, idioms, lexical ambiguity, register shifts and complex syntax. Tables 8-9 show strong agreement with LiT rankings (average ρ = 0.946; category correlations 0.78-0.96). Qualitative inspection also finds cases where models propagate awkwardness under faithful-translation instructions.

Q1. The current validation of alignment with user preferences primarily uses simple round-trip translation involving languages with relatively abundant resources.

We agree that the distinction between single-language round trips and the benchmark’s low-resource chains requires clearer qualification. We add independent validation using official WMT translations with existing human judgments. This provides external human evidence for round-trip evaluation, seen here:

https://imgur.com/a/MLVqHBf.

Q2. Intermediate Translations

We show that the model rankings for intermediate languages remains highly stable in the paper, in Section 4.2, Table 5 and Appendix G.2/Figure 10. Across six models and four sequences, two-language versus four-language rankings have Spearman correlations of 0.75-0.94, and three versus four have 0.83-1.00. Single-language round trips are often near ceiling and less stable (0.03-0.94). Thus, four languages provide a harder test with highly consistent rankings relative to two and three.

This shows that the intermediate languages increase the difficulty and scope of languages of the benchmark, without affecting the overall conclusions of the rankings.

Q.3 Drop in scores on translation chains.

We see that both can contribute to the drop in scores in the translation sequences by examining 3 results in the papers. First , all eight sequences use the same source texts, reducing content differences across comparisons. Secondly, Table 3 reports high-, medium- and low-resource sequence results separately, exposing much larger degradation in the latter and thirdly, Table 5 and Appendix G.2 vary chain length within four fixed sequences, as explained in the previous question. This shows Two- and three-language rankings broadly agree with four-language rankings, while pass rates generally decline with longer chains.

We hope these existing results and the clarified scope address the main concerns. Given the reviewer's positive assessment of the contribution and soundness, we hope we have clarified any remaining gaps for a recommendation.

Reviewer_t526OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that current multilingual benchmarks are misleading: models score well on translated English exams because they recognize the original English item, not because they reason multilingually. Their instrument, LiT, translates English into four intermediate languages and back to English, then measures how much the returning English drifted. Because the output is always English, the judge only compares two English paragraphs and needs no knowledge of the intermediate languages. LiT correlates ρ = 0.94 with LMArena human Elo, while MT-AIME24 gives −0.09 and INCLUDE −0.26; meanwhile MT-AIME24 correlates 0.94 with English AIME25 — the paper's evidence that those benchmarks track general capability.

Critical strengths

Trace analysis shows failures happen while the model reasons in English, which is mechanism rather than symptom. A manual taxonomy of all errors from one model across 11 languages shows that INCLUDE failures for bn and zh are >96% factual rather than linguistic.

Evaluating Quechua or Amharic has always been blocked by "no judge is more capable than the model in that language"; because LiT returns to English, the judge compares two English paragraphs and needs no human references. That is a methodological advance.

Critical weaknesses

Appendix A.3 says judge quality is validated by its high correlation with LMArena human ratings — the identical ρ = 0.94 used in the main results to argue that LiT is a good benchmark. If the judge is biased, both uses inherit the same bias and their agreement is guaranteed. The one thing that would break the circle is human MQM annotation.

Six models from three families × two variants is an effective n ≈ 3, and p = 0.008 does not account for that. The plot's y-axis spans roughly 96.0–98.5% and uses only the 7 high-resource LMArena languages. It seems that the central correlation is computed where the benchmark has the least resolution.

Questions for authors

Why is the headline correlation restricted to 7 high-resource languages in a 96–98.5% band? What is ρ over the full language set, and over the low-resource subset alone?

What is the measured untranslated-span / source-leakage rate per hop, especially for the languages scoring 59.2/32.0/0.2/0.0?

At what chain length does the metric lose rank stability, and is four languages the point of maximum discrimination?

1 comment

We thank the reviewer for the thoughtful and useful feedback. We appreciate that the reviewer recognizes the contirbutions of our work. We address the concerns below:

W1. LMArena Correlation

We agree that the same LMArena correlation could be improved and extend the figure below. We will revise Appendix A.3 accordingly. However, we also provide results on judge sensitivity already in Section 4.2, Table 4 and Appendix G.1: Grok 4.1 Fast, DeepSeek-V4-Pro and Gemini-3.1-Flash-Lite agree across ten model configurations with high Pairwise Spearman correlations are 0.96-0.99, with Kendall's W = 0.987. However, we agree that this can be further strengthened by using human studies. We therefore add an independent comparison against WMT26 [1] human cESA translation annotations for documents, seen here:

https://imgur.com/a/MLVqHBf.

Across 11 models covering seven languages, MQM≥80 pass rates correlate highly with human ratings at Spearman ρ=0.952 (Pearson r=0.93).

W2. Model families and narrow score range.

Good point. The six configurations represent three related families with thinking and non-thinking variants. We specifically choose both variants of each model to show the effect of thinking on the rankings. Section 3.1 and Appendix A deliberately compare models in the same capability tier to reduce confounding due to model scale. Therefore they score in a similar range. We have improved the figure and extended to 14 models, with thinking and non-thinking variants. We use the 5 languages that are supported among all evaluated models on LMArena, DE, ES, KO, RU, ZH. We correlate against the score directly given by the judge, which is: Deepseek V4 Pro, Nonthinking here: https://imgur.com/a/68ubc7I

This shows that for extended evaluations, the judge scores for an extended number of models highly correlate (Spearman=0.74) with LMArena rankings once more and can discriminate that thinking models often do not outperform non-thinking variants, for example, Claude Sonnet 4.5, whereas MT-AIME24 and Include show the opposite.

Q1. Why is the headline correlation restricted to 7 high-resource languages in a 96–98.5% band?

We use the 7 languages publicly provided by LMArena. The 0.94 result is based on single-language round trips on those languages. In Appendix B.2, we add more coverage across families, scripts and resource levels. We will make this clearer in the revision.

Q2. Measured untranslated-span / source-leakage hop rate.

We thank the reviewer for this valuable comment and suggestion. We will add this to the revision. We take two low resource sequences, Africa (Swahili → Arabic → Hausa → Amharic) and South America (Portuguese → Quechua → Spanish → Guarani), and show that the percentage of cases where the model output contains 8 or more consecutive words copied from the input.

Table 1: Africa - Percent of inputs that match outputs

ModelSwahiliArabicHausaAmharic
Gemini-3-Flash0.00.00.00.0
Gemma-4-31B Instruct0.00.00.00.0
Qwen3-235B Instruct0.50.53.70.0
Qwen3-30B Instruct0.00.50.51.0

Table 2: South America - Percent of inputs that match outputs

ModelPortugueseQuechuaSpanishGuarani
Gemini-3-Flash0.00.50.02.0
Gemma-4-31B Instruct0.00.50.03.5
Qwen3-235B Instruct0.00.51.15.3
Qwen3-30B Instruct0.00.014.09.9

Overall, in Tables 1 and 2, we see that the percentage of cases where the output is identical to the input throughout the hops is very low and for models such as Gemini-3-Flash it is basically 0. For example, Qwen3-30B Instruct copies quite a lot of input into the output verbatim for the South America chain because the model’s output degenerates into meaningless text, corresponding to a low score on our benchmark, showing the effectiveness of our benchmark to distinguish language capabilities across models.

Q3. Chain length and metric rank stability

This ablation is already provided in Section 4.2, Table 5 and Appendix G.2/Figure 10. Across six models and four sequences, two-language versus four-language rankings have Spearman correlations of 0.75-0.94, and three versus four have 0.83-1.00. Single-language round trips are often near ceiling and less stable (0.03-0.94). Thus, four languages provide a harder test with highly consistent rankings.

Overall, we hope these clarifications address the main concerns and are happy to answer further questions.

[1] Kocmi, Tom, et al. "Findings of the WMT26 General Machine Translation Shared Task: Contrastive Dynamic Human Evaluation at Scale." WMT. (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.