- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how target-language pre-training exposure shapes cross-lingual transfer from reinforcement learning (RL). Using a controlled English–Korean arithmetic reasoning setting, the authors investigate the effects of pre-training data amount, difficulty coverage, mid-training, and RL configurations. The results show that target-language difficulty coverage matters more than simply increasing data amount, and mid-training can help compensate for limited pre-training coverage.
Critical strengths
- The paper uses a simple and well-controlled experimental setup to study how pre-training affects cross-lingual RL transfer, making it easier to understand the role of different pre-training factors.
- The paper suggests that difficulty coverage matters more than simply increasing data amount, which offers useful insights into cross-lingual RL transfer.
Critical weaknesses
- The experiments are limited to relatively small models (mainly 100M parameters) with a very small tokenizer vocabulary (around 2k tokens), just two languages (English and Korean), and only synthetic arithmetic data. This raises concerns for me about whether the findings can generalize to larger multilingual LLMs trained on more diverse real-world data.
- Although the pre-training framework could in principle treat English and Korean equally, the experiments largely assume an English-dominant pre-training setting. For example, Figure 2(d) only considers Korean corpus ratios up to 40%. The lack of balanced or Korean-dominant settings and corresponding Korean-to-English transfer experiments across these ratios makes the evaluation incomplete. Therefore, it remains unclear whether the findings reflect general properties of cross-lingual RL transfer or are also influenced by language-specific factors, such as tokenization differences and so on.
Questions for authors
- Figure 2(d) only considers Korean pre-training ratios up to 40% and uses English-only RL. Could the authors extend the ratios to 50%, 69.9%, and 93.7%, and run Korean-only RL across these pre-training ratios to see whether similar results hold for Korean-to-English transfer? This would help check whether the findings are consistent across both languages or affected by factors like tokenization differences.
- In Appendix C.2, the 1% mid-training setting already reaches 87.9% Korean Pass@128 before English RL, and only improves to 90.2% after RL. Does this mean most of the improvement comes from mid-training itself rather than cross-lingual transfer from English RL?
6 comments
W2 & Q1. Korean-to-English transfer under Korean-dominant pre-training
We thank the reviewer for suggesting this comparison. To address the reviewer's concern about the English-dominant pre-training setting, we extend our experiments to Korean-dominant pre-training and examine Korean-to-English transfer. We use Korean corpus ratios of 60%, 70%, 80%, and 94%, which mirror the ratios in Figure 2(d), followed by Korean-only RL on op11–14. All other settings follow Sections 2 and 3, and results are means over three seeds.
At every ratio, English Mid-OOD gains from Korean-only RL remain small under unbalanced coverage (English op2–6) and are substantially larger under shared coverage (Table 2). Under shared coverage, English Mid-OOD pass@128 after Korean-only RL remains similar across these ratios. Section 3 shows a related pattern: increasing the Korean corpus ratio does not consistently improve Korean Mid-OOD pass@128 after English-only RL.
Table 3 compares both transfer directions under unbalanced and shared coverage. Under unbalanced coverage, Korean, pre-trained on op2–10, benefits from English-only RL, while English, pre-trained only on op2–6, gains little from Korean-only RL. With shared coverage, English gains increase and Korean continues to benefit from English-only RL. Together with Section 3, these results show that shared difficulty coverage supports cross-lingual RL transfer when either English or Korean dominates pre-training. We will include these comparisons in the revision.
Table 2. English pass@128 (%) before and after Korean-only RL under Korean-dominant pre-training. Korean pre-training covers op2–10. Means over three seeds.
| Korean corpus ratio | English coverage | Mid-OOD (op11–14) Pre-training | Mid-OOD Korean RL | Mid-OOD Δ (pp) | Hard-OOD (op15–20) Pre-training | Hard-OOD Korean RL | Hard-OOD Δ (pp) |
|---|---|---|---|---|---|---|---|
| 60% | Unbalanced (op2–6) | 42.4 | 42.8 | +0.5 | 35.6 | 35.8 | +0.1 |
| 60% | Shared (op2–10) | 75.5 | 86.9 | +11.4 | 40.7 | 42.3 | +1.6 |
| 70% | Unbalanced (op2–6) | 39.2 | 40.0 | +0.7 | 34.8 | 33.0 | -1.8 |
| 70% | Shared (op2–10) | 74.6 | 85.6 | +11.0 | 39.6 | 44.7 | +5.1 |
| 80% | Unbalanced (op2–6) | 42.6 | 43.4 | +0.8 | 38.5 | 38.2 | -0.3 |
| 80% | Shared (op2–10) | 78.1 | 89.9 | +11.8 | 40.0 | 51.4 | +11.3 |
| 94% | Unbalanced (op2–6) | 43.1 | 45.6 | +2.4 | 38.5 | 38.4 | 0.0 |
| 94% | Shared (op2–10) | 78.0 | 85.1 | +7.2 | 45.1 | 50.8 | +5.7 |
Table 3. Bidirectional pass@128 (%) on op11–14 under shared op2–10 pre-training. Each RL run uses op11–14 in one language.
| Korean corpus ratio | KO: before RL | KO: English RL | KO: gain (pp) | EN: before RL | EN: Korean RL | EN: gain (pp) |
|---|---|---|---|---|---|---|
| 60% | 73.6 | 90.9 | +17.4 | 75.5 | 86.9 | +11.4 |
| 70% | 72.7 | 89.7 | +17.0 | 74.6 | 85.6 | +11.0 |
| 80% | 76.1 | 90.2 | +14.1 | 78.1 | 89.9 | +11.8 |
| 94% | 85.0 | 94.8 | +9.8 | 78.0 | 85.1 | +7.2 |
Q2. Gains from mid-training and English-only RL
We thank the reviewer for raising this distinction. We agree that most of the Korean Mid-OOD improvement in the 1% setting occurs during mid-training. To distinguish the contributions of mid-training and subsequent English-only RL, we report both absolute and normalized RL gains (Table 4). In the 1% mid-training, Korean Mid-OOD pass@128 rose from 36.5% to 87.9% during mid-training. English-only RL then added 2.3 percentage points, bringing the final score to 90.2%. The normalized gain measures the fraction of the remaining gap to 100% closed during RL.
On Korean Hard-OOD, both mid-trained models started RL with less remaining headroom than the model without mid-training, yet both gained more during English-only RL. Their normalized RL gains are also higher (11.3% and 13.6%) than without mid-training (0.8%). These results show that mid-training primarily improves the starting model in the 1% Mid-OOD setting, while subsequent English-only RL can provide larger additional gains on harder problems. We will add the pre-RL scores and normalized gains to the Korean results in Appendix Table C1 and revise Section 4 to distinguish the improvement during mid-training from the gain during subsequent RL.
Table 4. Korean pass@128 (%) before and after English-only RL, averaged over three seeds. Pre is measured after mid-training where applied. Δ = Post − Pre (pp); normalized gain (%) = 100 × Δ / (100 − Pre).
| Condition | Mid-OOD: Pre / Post | Mid-OOD: Δ (pp) | Mid-OOD: Norm. gain (%) | Hard-OOD: Pre / Post | Hard-OOD: Δ (pp) | Hard-OOD: Norm. gain (%) |
|---|---|---|---|---|---|---|
| Unbalanced coverage (op2-7) | 36.5 / 40.0 | +3.6 | 5.6 | 43.9 / 44.4 | +0.5 | 0.8 |
| Mid-train 1% | 87.9 / 90.2 | +2.3 | 18.8 | 66.7 / 70.4 | +3.8 | 11.3 |
| Mid-train 5% | 84.9 / 93.8 | +8.9 | 59.1 | 53.8 / 60.1 | +6.3 | 13.6 |
| Shared coverage (op2-10) | 75.0 / 88.1 | +13.1 | 52.3 | 48.8 / 54.1 | +5.3 | 10.3 |
Concluding remarks
We sincerely thank the reviewer for the constructive and insightful feedback. The additional experiments have strengthened our study by extending the evidence for our coverage findings across languages and transfer directions, and by clarifying the distinct contributions of mid-training and subsequent RL across difficulty ranges. We will incorporate these results and clarifications into the revised manuscript to provide a more comprehensive and precise account of our findings and their limitations.
We thank the reviewer for recognizing the value of our controlled experiments and for the suggestions on broader evaluation. Below, we report experiments in additional languages and in the reverse transfer direction. We then examine the gains from mid-training separately from those added by subsequent RL.
W1a. Generalization across model scales and languages
To address the reviewer's concerns about model scale and language coverage, we extend our experiments across model sizes and target languages. Across 100M, 300M, and 500M models, English-only RL yields larger Korean gains under shared coverage than under unbalanced coverage (Appendix D, Figure D1). We also examine whether the coverage finding generalizes to German, French, and Russian (Table 1).
German and French, which share the Latin script with English, already receive substantial Mid-OOD gains under unbalanced coverage. Shared coverage mainly increases their Hard-OOD gains. For Russian, as for Korean in our manuscript, it also substantially increases Mid-OOD gains. The language-pair differences are consistent with evidence that shared subwords can support cross-lingual transfer ([1],[2] and [3]). The mid-training finding of Section 4 also generalizes to these four languages, where 1% mid-training increases post-RL accuracy on both Mid-OOD and Hard-OOD. We kindly refer the reviewer to Table 2 in our response to W1/Q1 of Reviewer 49SV for the corresponding results. These results support the robustness of our findings across language pairs with separately trained bilingual tokenizers.
Synthetic arithmetic allows us to control language-specific pre-training exposure when training models from scratch. While these experiments extend our findings across model scales and languages, generalization to larger multilingual LLMs with more diverse pre-training data and tokenizers remains to be evaluated. We will include these results in the revised manuscript.
Table 1. Target-language pass@128 (%) before and after English-only RL. The comparison follows Figure 2(b), with Hard-OOD added. Mid-OOD covers op11–14 and Hard-OOD covers op15–20. Δ is the RL gain in percentage points, calculated before rounding. English pre-training covers op2–10 and the target corpus ratio is approximately 20%.
[1] Wu et al. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. EMNLP 2019.
[2] Deshpande et al. When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer. NAACL 2022.
[3] Kallini et al. False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models. Findings of EMNLP 2025.
W1b. Practical value of our work
As discussed in Sections 1 and 6, prior studies observed that cross-lingual RL gains vary across languages and models, but the role of language-specific pre-training conditions remained difficult to isolate. Section 3 examines how target-language pre-training difficulty coverage and data amount affects these gains under controlled conditions. For an existing model, mid-training on the missing target-language difficulties with about 1% of the pre-training budget brings Mid-OOD performance after English-only RL close to the shared-coverage reference. (Section 4). When target-language RL data are available, joint RL improves target-language performance on harder problems while retaining similar English performance (Section 5). These findings suggest that difficulty coverage should be considered alongside data amount when selecting pre-training and mid-training data. We will discuss these implications for data selection and their limitations in the conclusion.