acceptodds
Under review as a conference paper at ICLR 2027

How Pretraining Shapes Cross-Lingual Transfer from Reinforcement Learning

Abstract

Multilingual language models are pre-trained on uneven amounts of data across languages and can therefore enter reinforcement learning (RL) with different levels of language-specific exposure. RL can further improve their reasoning beyond its training language, but the magnitude of these gains varies across languages. How does target-language pre-training exposure shape cross-lingual transfer from RL? We study this question in a controlled English–Korean arithmetic reasoning setting by varying the amount and difficulty coverage of Korean pre-training data, then applying RL in English, Korean, or both. Cross-lingual transfer is more sensitive to target-language difficulty coverage than to additional data within an already covered range. At similar Korean corpus sizes, broader difficulty coverage yields larger Korean gains from English RL, whereas increasing the Korean corpus ratio with the fixed coverage yields limited additional improvement. Mid-training on previously unseen Korean difficulties improves Korean performance after English-only RL. With a 1% additional budget of mid-training, pass@128 within the RL difficulty range approaches that of a reference pre-trained from scratch with the same difficulty range in both languages. Finally, given the resulting target-language coverage, RL language and difficulty shape where later reasoning gains generalize. Direct Korean RL generalizes more strongly to harder problems beyond the RL range than English-only transfer, while different Korean RL ranges favor different evaluation difficulties. These results show that target-language coverage established through pre-training and mid-training shapes how effectively later reasoning gains transfer across languages, beyond what is reflected in accuracy at the start of RL alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, cross-lingual transfer, multilignuality, large reasoning models, Pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vdm1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how target-language pre-training exposure shapes cross-lingual transfer from reinforcement learning (RL). Using a controlled English–Korean arithmetic reasoning setting, the authors investigate the effects of pre-training data amount, difficulty coverage, mid-training, and RL configurations. The results show that target-language difficulty coverage matters more than simply increasing data amount, and mid-training can help compensate for limited pre-training coverage.

Critical strengths
  • The paper uses a simple and well-controlled experimental setup to study how pre-training affects cross-lingual RL transfer, making it easier to understand the role of different pre-training factors.
  • The paper suggests that difficulty coverage matters more than simply increasing data amount, which offers useful insights into cross-lingual RL transfer.
Critical weaknesses
  • The experiments are limited to relatively small models (mainly 100M parameters) with a very small tokenizer vocabulary (around 2k tokens), just two languages (English and Korean), and only synthetic arithmetic data. This raises concerns for me about whether the findings can generalize to larger multilingual LLMs trained on more diverse real-world data.
  • Although the pre-training framework could in principle treat English and Korean equally, the experiments largely assume an English-dominant pre-training setting. For example, Figure 2(d) only considers Korean corpus ratios up to 40%. The lack of balanced or Korean-dominant settings and corresponding Korean-to-English transfer experiments across these ratios makes the evaluation incomplete. Therefore, it remains unclear whether the findings reflect general properties of cross-lingual RL transfer or are also influenced by language-specific factors, such as tokenization differences and so on.
Questions for authors
  • Figure 2(d) only considers Korean pre-training ratios up to 40% and uses English-only RL. Could the authors extend the ratios to 50%, 69.9%, and 93.7%, and run Korean-only RL across these pre-training ratios to see whether similar results hold for Korean-to-English transfer? This would help check whether the findings are consistent across both languages or affected by factors like tokenization differences.
  • In Appendix C.2, the 1% mid-training setting already reaches 87.9% Korean Pass@128 before English RL, and only improves to 90.2% after RL. Does this mean most of the improvement comes from mid-training itself rather than cross-lingual transfer from English RL?
6 comments

We thank the reviewer for recognizing the value of our controlled experiments and for the suggestions on broader evaluation. Below, we report experiments in additional languages and in the reverse transfer direction. We then examine the gains from mid-training separately from those added by subsequent RL.


W1a. Generalization across model scales and languages

To address the reviewer's concerns about model scale and language coverage, we extend our experiments across model sizes and target languages. Across 100M, 300M, and 500M models, English-only RL yields larger Korean gains under shared coverage than under unbalanced coverage (Appendix D, Figure D1). We also examine whether the coverage finding generalizes to German, French, and Russian (Table 1).

German and French, which share the Latin script with English, already receive substantial Mid-OOD gains under unbalanced coverage. Shared coverage mainly increases their Hard-OOD gains. For Russian, as for Korean in our manuscript, it also substantially increases Mid-OOD gains. The language-pair differences are consistent with evidence that shared subwords can support cross-lingual transfer ([1],[2] and [3]). The mid-training finding of Section 4 also generalizes to these four languages, where 1% mid-training increases post-RL accuracy on both Mid-OOD and Hard-OOD. We kindly refer the reviewer to Table 2 in our response to W1/Q1 of Reviewer 49SV for the corresponding results. These results support the robustness of our findings across language pairs with separately trained bilingual tokenizers.

Synthetic arithmetic allows us to control language-specific pre-training exposure when training models from scratch. While these experiments extend our findings across model scales and languages, generalization to larger multilingual LLMs with more diverse pre-training data and tokenizers remains to be evaluated. We will include these results in the revised manuscript.

Table 1. Target-language pass@128 (%) before and after English-only RL. The comparison follows Figure 2(b), with Hard-OOD added. Mid-OOD covers op11–14 and Hard-OOD covers op15–20. Δ is the RL gain in percentage points, calculated before rounding. English pre-training covers op2–10 and the target corpus ratio is approximately 20%.

Target languageCoverageMid-OOD Pre-trainingMid-OOD English RLMid-OOD Δ (pp)Hard-OOD Pre-trainingHard-OOD English RLHard-OOD Δ (pp)
GermanUnbalanced (op2–6)47.465.9+18.446.150.2+4.1
GermanShared (op2–10)75.796.3+20.648.065.7+17.7
FrenchUnbalanced (op2–6)47.666.7+19.144.749.1+4.4
FrenchShared (op2–10)76.696.2+19.547.662.5+14.9
RussianUnbalanced (op2–6)42.143.2+1.043.243.7+0.6
RussianShared (op2–10)76.886.0+9.250.753.0+2.3
KoreanUnbalanced (op2–6)36.736.6-0.142.042.3+0.3
KoreanShared (op2–10)75.088.1+13.148.854.1+5.3

[1] Wu et al. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. EMNLP 2019.

[2] Deshpande et al. When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer. NAACL 2022.

[3] Kallini et al. False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models. Findings of EMNLP 2025.


W1b. Practical value of our work

As discussed in Sections 1 and 6, prior studies observed that cross-lingual RL gains vary across languages and models, but the role of language-specific pre-training conditions remained difficult to isolate. Section 3 examines how target-language pre-training difficulty coverage and data amount affects these gains under controlled conditions. For an existing model, mid-training on the missing target-language difficulties with about 1% of the pre-training budget brings Mid-OOD performance after English-only RL close to the shared-coverage reference. (Section 4). When target-language RL data are available, joint RL improves target-language performance on harder problems while retaining similar English performance (Section 5). These findings suggest that difficulty coverage should be considered alongside data amount when selecting pre-training and mid-training data. We will discuss these implications for data selection and their limitations in the conclusion.

W2 & Q1. Korean-to-English transfer under Korean-dominant pre-training

We thank the reviewer for suggesting this comparison. To address the reviewer's concern about the English-dominant pre-training setting, we extend our experiments to Korean-dominant pre-training and examine Korean-to-English transfer. We use Korean corpus ratios of 60%, 70%, 80%, and 94%, which mirror the ratios in Figure 2(d), followed by Korean-only RL on op11–14. All other settings follow Sections 2 and 3, and results are means over three seeds.

At every ratio, English Mid-OOD gains from Korean-only RL remain small under unbalanced coverage (English op2–6) and are substantially larger under shared coverage (Table 2). Under shared coverage, English Mid-OOD pass@128 after Korean-only RL remains similar across these ratios. Section 3 shows a related pattern: increasing the Korean corpus ratio does not consistently improve Korean Mid-OOD pass@128 after English-only RL.

Table 3 compares both transfer directions under unbalanced and shared coverage. Under unbalanced coverage, Korean, pre-trained on op2–10, benefits from English-only RL, while English, pre-trained only on op2–6, gains little from Korean-only RL. With shared coverage, English gains increase and Korean continues to benefit from English-only RL. Together with Section 3, these results show that shared difficulty coverage supports cross-lingual RL transfer when either English or Korean dominates pre-training. We will include these comparisons in the revision.

Table 2. English pass@128 (%) before and after Korean-only RL under Korean-dominant pre-training. Korean pre-training covers op2–10. Means over three seeds.

Korean corpus ratioEnglish coverageMid-OOD (op11–14) Pre-trainingMid-OOD Korean RLMid-OOD Δ (pp)Hard-OOD (op15–20) Pre-trainingHard-OOD Korean RLHard-OOD Δ (pp)
60%Unbalanced (op2–6)42.442.8+0.535.635.8+0.1
60%Shared (op2–10)75.586.9+11.440.742.3+1.6
70%Unbalanced (op2–6)39.240.0+0.734.833.0-1.8
70%Shared (op2–10)74.685.6+11.039.644.7+5.1
80%Unbalanced (op2–6)42.643.4+0.838.538.2-0.3
80%Shared (op2–10)78.189.9+11.840.051.4+11.3
94%Unbalanced (op2–6)43.145.6+2.438.538.40.0
94%Shared (op2–10)78.085.1+7.245.150.8+5.7

Table 3. Bidirectional pass@128 (%) on op11–14 under shared op2–10 pre-training. Each RL run uses op11–14 in one language.

Korean corpus ratioKO: before RLKO: English RLKO: gain (pp)EN: before RLEN: Korean RLEN: gain (pp)
60%73.690.9+17.475.586.9+11.4
70%72.789.7+17.074.685.6+11.0
80%76.190.2+14.178.189.9+11.8
94%85.094.8+9.878.085.1+7.2

Q2. Gains from mid-training and English-only RL

We thank the reviewer for raising this distinction. We agree that most of the Korean Mid-OOD improvement in the 1% setting occurs during mid-training. To distinguish the contributions of mid-training and subsequent English-only RL, we report both absolute and normalized RL gains (Table 4). In the 1% mid-training, Korean Mid-OOD pass@128 rose from 36.5% to 87.9% during mid-training. English-only RL then added 2.3 percentage points, bringing the final score to 90.2%. The normalized gain measures the fraction of the remaining gap to 100% closed during RL.

On Korean Hard-OOD, both mid-trained models started RL with less remaining headroom than the model without mid-training, yet both gained more during English-only RL. Their normalized RL gains are also higher (11.3% and 13.6%) than without mid-training (0.8%). These results show that mid-training primarily improves the starting model in the 1% Mid-OOD setting, while subsequent English-only RL can provide larger additional gains on harder problems. We will add the pre-RL scores and normalized gains to the Korean results in Appendix Table C1 and revise Section 4 to distinguish the improvement during mid-training from the gain during subsequent RL.

Table 4. Korean pass@128 (%) before and after English-only RL, averaged over three seeds. Pre is measured after mid-training where applied. Δ = Post − Pre (pp); normalized gain (%) = 100 × Δ / (100 − Pre).

ConditionMid-OOD: Pre / PostMid-OOD: Δ (pp)Mid-OOD: Norm. gain (%)Hard-OOD: Pre / PostHard-OOD: Δ (pp)Hard-OOD: Norm. gain (%)
Unbalanced coverage (op2-7)36.5 / 40.0+3.65.643.9 / 44.4+0.50.8
Mid-train 1%87.9 / 90.2+2.318.866.7 / 70.4+3.811.3
Mid-train 5%84.9 / 93.8+8.959.153.8 / 60.1+6.313.6
Shared coverage (op2-10)75.0 / 88.1+13.152.348.8 / 54.1+5.310.3

Concluding remarks

We sincerely thank the reviewer for the constructive and insightful feedback. The additional experiments have strengthened our study by extending the evidence for our coverage findings across languages and transfer directions, and by clarifying the distinct contributions of mid-training and subsequent RL across difficulty ranges. We will incorporate these results and clarifications into the revised manuscript to provide a more comprehensive and precise account of our findings and their limitations.

Reviewer_49SVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates bilingual (English & Korean) language models in different pre-training and RL setups. They mainly evaluate 100M parameter decoder-only models trained from scratch with arithmetic word problems (using GSM-Infinite).

They show that (1) difficulty does not transfer between languages, when pre-training excludes some difficulty levels in Korean, but English only RL is still helpful when all levels are included in the dataset, (2) more Korean data in the already included levels is not helpful, (3) including missing difficulty levels in Korean for mid-training helps, (4) Korean RL helps more with harder problems than English-only, gap is smaller with easier problems. Taken together, the work highlights the importance of difficulty level coverage before RL for multilingual transfer.

Critical strengths
  1. The results are interesting, especially Figure 2 showing how leaving out more difficult samples in pre-training for Korean significantly hurts the performance for only Korean.
  2. The study is well motivated and covers many aspects.
Critical weaknesses
  1. The title, problem setup in the introduction, and main findings are more general than the experiments support. The general language-agnostic claims would be better supported with more language tested. For example, this would ask the question if we see the same patterns with languages that share more tokens. The other question is if the task is representative of language understanding as the GSM-Infinite benchmark is linguistically simple, and the results for English and Korean are very close even when only 6.3% of the pre-training corpus is in Korean, it is unclear how little data is actually needed to achieve the same performance, and if the patterns transfer to tasks requiring more language understanding. In addition, the claims are very general but the experimental setup artificial, it is unclear how the findings transfer to more realistic multilingual scenarios.
  2. The paper does not discuss limitations.

Minor points:

  1. There are no monolingual models as comparison to isolate the transfer, would only Korean model trained with maximum 6 operation do worse than when it sees samples in English with higher number of operations, is some cross-lingual transfer still happening?
  2. The relationship between OOD for language and difficulty is intertwined, could the model perform well in 2-5 range when trained on 6-10 or odd numbers when trained on even numbers?

For presentation: The term 'difficult coverage' could use an explanation before mentioning it in the introduction. Language difficulty could have different meanings, and creates confusion before it is mentioned that we are dealing with number of arithmetic operations . Figure 1 is crowded and is difficult to understand even after reading the paper. However, the idea is good, especially since understanding the differences between experiments requires careful focus.

Questions for authors
  1. How could the findings be generalized to other setups in cross-lingual transfer, and is there additional empirical evidence supporting the patterns highlighted by these experiments?
5 comments

We thank the reviewer for recognizing the motivation of our study and for the suggestions on broader evaluation. Below, we report additional language experiments and the requested controls for monolingual pre-training and generalization to omitted operation counts. We also outline revisions to the limitations and presentation.


W1 & Q1 : Generalization across languages

Following the reviewer's suggestion, we added coverage comparisons in German, French, and Russian. Shared coverage gives higher target-language accuracy after English-only RL in every completed coverage pair (Table 1). German, and French which share the Latin script with English, show larger Hard-OOD gains under shared coverage. Russian and Korean show larger Mid-OOD gains. The manuscript also reports coverage comparisons at 100M, 300M, and 500M parameters in Appendix D.

We also added 1% mid-training on the missing target-language op8-10 difficulties before English-only RL. Across these four languages, this treatment increases post-RL accuracy on both Mid-OOD and Hard-OOD and brings Mid-OOD accuracy closer to the shared-coverage reference (Table 2).

Table 1. Target-language pass@128 (%) before and after English-only RL. The comparison follows Figure 2(b), with Hard-OOD added. Mid-OOD covers op11–14 and Hard-OOD covers op15–20. Δ is the RL gain in percentage points, calculated before rounding. English pre-training covers op2–10 and the target corpus ratio is approximately 20%.

Target languageCoverageMid-OOD Pre-trainingMid-OOD English RLMid-OOD Δ (pp)Hard-OOD Pre-trainingHard-OOD English RLHard-OOD Δ (pp)
GermanUnbalanced (op2–6)47.465.9+18.446.150.2+4.1
GermanShared (op2–10)75.796.3+20.648.065.7+17.7
FrenchUnbalanced (op2–6)47.666.7+19.144.749.1+4.4
FrenchShared (op2–10)76.696.2+19.547.662.5+14.9
RussianUnbalanced (op2–6)42.143.2+1.043.243.7+0.6
RussianShared (op2–10)76.886.0+9.250.753.0+2.3
KoreanUnbalanced (op2–6)36.736.6-0.142.042.3+0.3
KoreanShared (op2–10)75.088.1+13.148.854.1+5.3

Table 2. Target-language pass@128 (%) after English-only RL. Mid-training adds op8–10 to the unbalanced-coverage model (op2–7). Budgets are relative to the original pre-training budget. Scores are means over three seeds.

Target languageBandUnbalanced (op2–7)Mid-train 1%Mid-train 5%Shared (op2–10)
GermanMid-OOD75.794.696.496.3
GermanHard-OOD48.362.967.565.7
FrenchMid-OOD73.693.994.996.2
FrenchHard-OOD45.760.560.662.5
RussianMid-OOD52.585.987.286.0
RussianHard-OOD48.459.154.353.0
KoreanMid-OOD40.090.293.888.1
KoreanHard-OOD44.470.460.154.1

W2. Discussion and limitation section

We will add a discussion and limitations section. We will state that our experiments use synthetic arithmetic problems and models up to 500M parameters, and discuss the variation across languages and transfer directions. Generalization to tasks requiring substantial language understanding and larger multilingual models remains to be evaluated.


Minor point 1. Monolingual baseline and transfer during pre-training

We added a Korean-only op2-6 control with the same amount of Korean pre-training exposure as the bilingual model. Adding English op2-10 data improves Korean performance on op7-10 and op11-14 before RL (Table 3). The table also includes models with Korean op2-10 coverage.

Table 3. Korean pass@128 (%) after pre-training. The op2–6 comparison matches Korean exposure across conditions and averages three runs. The op2–10 comparison uses one run per condition on a separate evaluation set. (Δ = bilingual − Korean-only.)

Korean PT rangeEvaluation rangeKorean onlyKorean + English op2-10Δ (pp)
op2-6op2-699.8100.0+0.2
op2-6op7-1060.569.8+9.3
op2-6Mid-OOD (op11-14)30.836.3+5.5
op2-10Mid-OOD (op11-14)76.077.1+1.1

Minor point 2. Language and difficulty OOD

To examine whether omitting particular operation counts necessarily impairs target-language performance, we ran both suggested controls with three seeds. Models pre-trained on Korean op6-10 or on even operation counts already solve the omitted lower or intermediate counts after pre-training (Table 4). These counts are at or below the maximum seen during pre-training. Our main analysis examines whether source-language RL improves target-language performance beyond its maximum pre-training difficulty. Because pre-training coverage differs across languages, the same operation count can fall within English coverage and beyond Korean coverage. We appreciate the reviewer's suggestion, which helps clarify the interpretation of difficulty coverage. If we have misunderstood the reviewer's concern, we would welcome further discussion.

We appreciate the reviewer's suggestion, which helps clarify the interpretation of difficulty coverage. If we have misunderstood the reviewer's concern, we would welcome further discussion.

Korean PT operation countsOmitted evaluation countsPT pass@128 (%)
6-102-597.3
2, 4, 6, 8, 103, 5, 7, 9100.0

Presentation. Definition of difficulty coverage and Figure 1

Thank you for these helpful suggestions. We will define difficulty coverage in the task setup by the arithmetic operation counts included in each language's pre-training data, and introduce this meaning when the term first appears.

Figure 1 was intended to illustrate the experiments and their purposes. We agree that it contains too much text and will simplify the labels and layout to make the comparisons easier to follow.


Concluding remarks

We sincerely thank the reviewer for the constructive feedback. The additional language experiments broaden the evidence for our coverage finding, and the requested controls clarify transfer during pre-training and generalization to omitted operation counts. We will incorporate these results and clarifications into the revised manuscript during the discussion period, including an explicit discussion of limitations.

Thank you for including additional languages, which helps strengthen your claims. However, concerns remain regarding the representativeness of the experimental setup for evaluating language understanding. Furthermore, the response promises substantial additions that would significantly change the paper, and these cannot be verified at this stage. Therefore, I will maintain my original scores and my assessment that, while the paper has some merit, it has room for improvement.

Reviewer_8mU8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates how target-language pre-training exposure affects cross-lingual transfer from RL. It uses synthetic English-Korean arithmetic problems genrated from shared reasoning structures to vary Korean pre-training data difficulty coverage. The results show that broadening the difficulty coverage of Korean pre-training is more beneficial for cross-lingual transfer than adding more Korean data within a fixed difficulty range. Even a small amount of mid-training can substantially improve Korean performance after English-only RL. Direct RL in Korean yields stronger performance on Korean tasks than transfer from English-only RL, particularly on problems beyond the difficulty range used for RL.

Critical strengths

S1. The experimental settings from pre-training conditions to mid-training and RL configuration is coherent. The figures and appendices make the experimental comparisons reasonably easy to trace.

S2. This paper uses a carefully controlled experimental design. The experiments vary Korean data quantity and difficulty coverage while keeping English pre-training data and English-only RL fixed. Comparing different coverage ranges at similar Korean corpus sizes helps isolate the benefit of broader coverage.

Critical weaknesses

W1. This paper presents many findings from controlled experiments, but I find it difficult to identify a clear and compelling central insight. The paper builds on [1] that expand to multilingual. Their Appendix A.6.1 and A.6.2 already show that source-only RL transfers well when the target context has broad pre-training coverage, but poorly when exposure is limited to atomic operations. The bilingual experiments, coverage comparisons, and scale checks provide useful evidence, but it remains unclear what new insight they establish beyond these findings. Evaluation on one language pair and one synthetic task family also limits the breadth of the conclusions.

W2. The role of mid-training in improving subsequent RL transfer is unclear. In Appendix C.2, Korean op11–14 pass@128 reaches 87.9% after 1% mid-training and rises to 90.2% after English-only RL. Most of the improvement in this setting therefore occurs before RL. Although Appendix D.2 reports further RL gains at larger scales, matching a shared-coverage reference in final performance does not imply comparable gains from RL. The paper should clarify whether mid-training mainly improves the starting model or also increases the benefit of subsequent English-only RL, accounting for differences in pre-RL accuracy and remaining headroom.

Questions for authors

Q1. What new insight does the bilingual setting provide beyond the cross-context transfer findings in Zhang et al.?

Q2. How much of the improvement comes from mid-training itself versus subsequent English-only RL? Does difficulty coverage explain RL gains beyond pre-RL accuracy and remaining headroom?

Q3. Are generated solutions to Korean prompts required to be in Korean? How does English-only RL affect the proportions of Korean, English, and mixed-language solutions?

5 comments

We thank the reviewer for recognizing the coherent experimental setup and controlled comparisons. We address the comments and questions below.


W1 & Q1. Contribution and generalization

Thank you for pointing out this connection. We agree that Zhang et al. [1] establish the importance of pre-training coverage for cross-context RL generalization. Building on their findings, we investigate how language-specific pre-training conditions shape cross-lingual transfer, including the relative roles of difficulty coverage, data amount, and RL languages. To disentangle these factors, we train models from scratch under controlled conditions.

Our contribution is a more fine-grained analysis of these dependencies in a multilingual setting. While Zhang et al. compare broad target-context coverage with atomic-only exposure, we progressively vary target-language difficulty coverage at approximately matched corpus sizes and independently vary corpus ratio with fixed coverage. We find that broader Korean difficulty coverage yields larger gains from English-only RL, whereas increasing the Korean corpus ratio with fixed coverage provides limited additional benefit.

To address the reviewer's concerns about generalization, we extend the coverage comparison to multiple target languages (Table 1) and examine transfer in the reverse direction (Table 3). German and French show substantial Mid-OOD gains even under narrow coverage, whereas Russian and Korean benefit more from broader coverage. Hard-OOD gains also increase with broader coverage across all four languages. In the reverse direction, Korean-only RL also yields larger English gains with broader English pre-training coverage across all tested corpus ratios.

To examine whether missing target-language coverage can be established after pre-training, we introduce the missing difficulties through mid-training (Table 2). With a 1% additional budget, post-RL Mid-OOD performance approaches the shared-coverage reference across all four languages. Section 5 further shows that even with shared coverage, direct Korean RL generalizes more strongly beyond the RL difficulty range than English-only transfer.


Table 1. Target-language pass@128 (%) before and after English-only RL. English pre-training covers op2–10. Mid-OOD covers the RL range (op11–14), and Hard-OOD lies beyond it (op15–20). Scores are three-seed means. Gains are in percentage points and calculated before rounding.

Target languageCoverageMid-OOD PTMid-OOD EN RLMid-OOD Δ (pp)Hard-OOD PTHard-OOD EN RLHard-OOD Δ (pp)
GermanUnbalanced (op2–6)47.465.9+18.446.150.2+4.1
GermanShared (op2–10)75.796.3+20.648.065.7+17.7
FrenchUnbalanced (op2–6)47.666.7+19.144.749.1+4.4
FrenchShared (op2–10)76.696.2+19.547.662.5+14.9
RussianUnbalanced (op2–6)42.143.2+1.043.243.7+0.6
RussianShared (op2–10)76.886.0+9.250.753.0+2.3
KoreanUnbalanced (op2–6)36.736.6-0.142.042.3+0.3
KoreanShared (op2–10)75.088.1+13.148.854.1+5.3

Table 2. Target-language pass@128 (%) after English-only RL. Mid-training adds op8–10 to the unbalanced-coverage model (op2–7). Budgets are relative to the original pre-training budget. Scores are means over three seeds.

Target languageBandUnbalanced (op2–7)Mid-train 1%Mid-train 5%Shared (op2–10)
GermanMid-OOD75.794.696.496.3
GermanHard-OOD48.362.967.565.7
FrenchMid-OOD73.693.994.996.2
FrenchHard-OOD45.760.560.662.5
RussianMid-OOD52.585.987.286.0
RussianHard-OOD48.459.154.353.0
KoreanMid-OOD40.090.293.888.1
KoreanHard-OOD44.470.460.154.1

Table 3. English Mid-OOD pass@128 (%) before and after Korean-only RL under Korean-dominant pre-training. Korean pre-training covers op2–10, while English coverage varies. RL uses Korean op11–14, and evaluation uses English op11–14. Scores are three-seed means. Gains are in percentage points and calculated before rounding.

Korean corpus ratioEnglish coveragePre-trainingKorean RLΔ (pp)
60%Unbalanced (op2–6)42.442.8+0.5
60%Shared (op2–10)75.586.9+11.4
70%Unbalanced (op2–6)39.240.0+0.7
70%Shared (op2–10)74.685.6+11.0
80%Unbalanced (op2–6)42.643.4+0.8
80%Shared (op2–10)78.189.9+11.8
94%Unbalanced (op2–6)43.145.6+2.4
94%Shared (op2–10)78.085.1+7.2

[1] Zhang et al. On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models. ICML 2026.

W2 & Q2. Mid-training and subsequent RL gains

Thank you for raising this distinction. To compare RL gains across models with different starting accuracies, Table 4 reports both the gain in percentage points and the fraction of the remaining gap to 100% closed by English-only RL. As shown in Table 4, most of the Korean Mid-OOD improvement in the 1% condition occurred during mid-training. Korean pass@128 was already close to the post-RL score of the shared-coverage reference before English-only RL began, and RL added a further 2.3 percentage points. With 5% mid-training, the Mid-OOD gain during RL was larger than it was without mid-training.

On Hard-OOD, both mid-trained models started RL with higher accuracy and less remaining headroom than the model without mid-training, yet both gained more during English-only RL. RL closed about 11–14% of the remaining gap to 100% in the two mid-training conditions, compared with less than 1% without mid-training. Because the checkpoints differed in pre-RL accuracy, this comparison leaves open how much of the gain difference would persist at the same starting accuracy. We will add the pre-RL scores and normalized gains shown in Table 4 to Appendix Table C1 and revise Section 4 to distinguish the improvement during mid-training from the gain during subsequent English-only RL.

We will add the pre-RL scores and normalized gains shown in Table 4 to Appendix Table C1 and revise Section 4 to distinguish the improvement during mid-training from the gain during subsequent English-only RL.

Table 4. Korean pass@128 (%) before and after English-only RL, averaged over three seeds. Pre is measured after mid-training where applied. Δ = Post − Pre (pp); normalized gain (%) = 100 × Δ / (100 − Pre).

ConditionMid-OOD: Pre / PostMid-OOD: Δ (pp)Mid-OOD: Norm. gain (%)Hard-OOD: Pre / PostHard-OOD: Δ (pp)Hard-OOD: Norm. gain (%)
Unbalanced coverage (op2-7)36.5 / 40.0+3.65.643.9 / 44.4+0.50.8
Mid-train 1%87.9 / 90.2+2.318.866.7 / 70.4+3.811.3
Mid-train 5%84.9 / 93.8+8.959.153.8 / 60.1+6.313.6
Shared coverage (op2-10)75.0 / 88.1+13.152.348.8 / 54.1+5.310.3

Q3. Language of generated solutions

We do not impose any language constraints on generated solutions. Both the RL reward and evaluation are based on final-answer correctness, without requiring the solution to match the prompt language. Both before and after English-only RL, over 98% of outputs were Korean-dominant, fewer than 1% were mixed, and none were English-dominant. These proportions were unchanged after RL.


Concluding remarks

We thank the reviewer for the constructive feedback. The additional experiments extend our analysis across languages and transfer directions, while the stage-wise results distinguish improvements from mid-training and subsequent RL. We will incorporate these findings into the revised manuscript and clarify our contributions and limitations.

Thanks for the new experiments. I would like to raise the soundness score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Reviewers and Area Chairs,

We thank the reviewers for their constructive feedback. We have updated the manuscript with additional experiments. The main findings are:

  • Coverage across languages, Section 3, 4 and 5 (8mU8, 49SV, vdm1). Across German, French, Russian, and Korean, shared pre-training coverage yielded larger mean gains from English-only RL. The difference was larger on Hard-OOD for German and French and on Mid-OOD for Russian and Korean. In all four languages, 1% mid-training brought post-RL Mid-OOD scores close to the shared-coverage reference, target-language RL outperformed English-only RL on Hard-OOD.

  • Reverse transfer, Appendix B.4 (vdm1). Korean-to-English transfer also benefits from shared coverage. Under Korean-dominant pre-training, English Mid-OOD gains are larger with shared English coverage at all four Korean corpus ratios (60–94%).

  • Mid-training, Section 4 and Appendix C (8mU8, vdm1). With 1% mid-training, post-RL Mid-OOD performance approaches the shared-coverage reference across all four target languages. For Korean, Mid-OOD performance largely recovers before RL, while subsequent English-only RL produces larger Hard-OOD gains after mid-training.

  • Gap-based data selection, Section 7 and Appendix F (8mU8, 49SV, vdm1). To validate our findings on an existing multilingual model, we mid-trained Qwen2.5-3B- on NuminaMath problems that it solved correctly in English but rarely in the target language, then applied English-only RL. On Korean PolyMath Low, targeting these gaps improved post-RL accuracy over both random selection and no mid-training, suggesting that our findings extend beyond the controlled from-scratch setting.

We now report pre-RL scores and normalized gains, clarify the contribution in the Introduction, and include a Discussion and Limitations section. The revised PDF incorporates these updates, with changes marked in blue. We thank the reviewers for helping us strengthen the evaluation and clarify our findings.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.