- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes evaluating multilingual capability through round-trip translation. Given a passage in a source language, round-trip translation involves translating it to a target language and back, comparing the result to the original passage. Semantic gaps between the original and back-translated text show whether a model can preserve meaning across languages. They argue that existing benchmarks measure factual recall and mathematical reasoning rather than multilingual proficiency. The paper computes Spearman rank-order correlations between benchmark scores and LMArena to show that these scores don’t correlate with user preference.
They introduce Lost in Translation (LiT), a round-trip translation benchmark of 1600 samples across 8 language sequences spanning high-, medium-, and low resource languages. They show that round-trip translation scores on LiT correlate almost perfectly (ρ = 0.94) with LMArena, suggesting closer alignment with real-world multilingual performance.
Critical strengths
-
The paper analyzes widely used benchmarks (MT-AIME24 and INCLUDE) and the errors models make on them. It finds that errors mostly reflect gaps in reasoning or knowledge rather than misunderstanding of the language, and that models reason in English even when prompted in another language. These benchmarks therefore conflate reasoning and knowledge with multilingual understanding. This is an important finding, and it motivates benchmarks that can separate the two.
-
. Round-trip translation is easy to apply at scale. It needs no human reference translations or per-sample annotation. Because the judge only compares two English texts, it also avoids the need for a judge model stronger than the tested models in each low-resource language. The benchmark covers over 30 languages across high, medium and low-resource levels.
Critical weaknesses
-
The LMArena results in Section 3, Figure 2 are based on 6 models with several models scoring pretty close to each other. With only 6 data points, the spearman score of 0.94 would change noticeably with one swapper pair. There might also be a confounder where arena users may prefer instruct models over thinking models due to speed or style. Benchmarks that favor thinking will not correlate with LMArena scores in such cases. Plotting this correlation for more models would make this result more convincing.
-
There is no intermediate evaluation in the target language. The benchmark only evaluates the back translated English text. The quality of text, fluency, naturalness in the target language is never evaluated. Adding additional metrics to verify the quality of the translated text would be useful.
Questions for authors
- Footnote 4 says Japanese is missing for both DeepSeek variants and French for Qwen3-235B (Thinking), so are modes averaged over different sets of languages?
- The paper evaluates 22 configurations against the new benchmark, why are these not used for Sec 3, LMArena ranking comparison?
We thank the reviewer for the thoughtful and encouraging assessment. We appreciate that the reviewer recognizes both the benchmark analysis and the scalability of round-trip translation. We address the concerns below:
W1. On the small LMArena comparison and preference confounds.
We agree that the same LMArena correlation could be improved and extend the figure below. We will revise Appendix A.3 accordingly. However, we also provide results on judge sensitivity already in Section 4.2, Table 4 and Appendix G.1: Grok 4.1 Fast, DeepSeek-V4-Pro and Gemini-3.1-Flash-Lite agree across ten model configurations with high Pairwise Spearman correlations are 0.96-0.99, with Kendall's W = 0.987. However, we agree that this can be further strengthened by using human studies. We therefore add an independent comparison against WMT26 [1] human cESA translation annotations for documents, seen here:
https://imgur.com/a/MLVqHBf.
Across 11 models covering seven languages, MQM≥80 pass rates correlate with human ratings at Spearman ρ=0.952 (Pearson r=0.93). Therefore, we can see that the WMT26 human ratings highly correlate with the annotations used by our benchmark approach.
W2. On intermediate translation quality.
The reviewer is correct that final English translation does not certify the fluency, naturalness or register of every intermediate translation. Appendix C explicitly acknowledges this, and Appendix A.5 identifies target-language fluency and individual-language attribution as limitations. We will state this scope more prominently.
However, we would also like to highlight the existing 480-sample stress test of five classical translation challenges (Appendix C, Tables 8-9), whose rankings correlate strongly with LiT (average ρ = 0.946). Section 4.2 additionally checks three judges (Table 4) and chain length (Table 5; Appendix G.2). These support the robustness of our benchmark.
Q1. On missing languages in the LMArena averages.
We agree that this limitation affects results. Footnote 4 states that the submitted comparison averages available ratings: Japanese is missing for both DeepSeek variants and French for Qwen3-235B (Thinking). The means therefore use different language sets. We will make this limitation explicit and have generated a new, expanded figure with more models that can be seen here, which now use the same exact languages (DE, ES, KO, RU, ZH), available across all models: https://imgur.com/a/68ubc7I. We thank the reviewer for providing this feedback and will adjust our revision accordingly.
Q2. Why not use all 22 LiT configurations?
The 22 main evaluation uses multi-language translation chains, whereas Section 3 evaluates separate English→language→English round trips matched to language-specific Arena ratings. The chain scores therefore cannot simply be substituted into that comparison. The original six configurations were selected from a comparable capability tier, retaining thinking/non-thinking variants. To extend the number of models we use, we have now 14 models seen here: https://imgur.com/a/68ubc7I . The expanded comparison uses identical language coverage and a uniform DeepSeek V4 Pro judge.
Overall, we hope these clarifications address the concerns and are happy to answer further questions.