How Pretraining Shapes Cross-Lingual Transfer from Reinforcement Learning

Multilingual language models are pre-trained on uneven amounts of data across languages and can therefore enter reinforcement learning (RL) with different levels of language-specific exposure. RL can further improve their reasoning beyond its training language, but the magnitude of these gains varies across languages. How does target-language pre-training exposure shape cross-lingual transfer from RL? We study this question in a controlled English–Korean arithmetic reasoning setting by varying the amount and difficulty coverage of Korean pre-training data, then applying RL in English, Korean, or both. Cross-lingual transfer is more sensitive to target-language difficulty coverage than to additional data within an already covered range. At similar Korean corpus sizes, broader difficulty coverage yields larger Korean gains from English RL, whereas increasing the Korean corpus ratio with the fixed coverage yields limited additional improvement. Mid-training on previously unseen Korean difficulties improves Korean performance after English-only RL. With a 1% additional budget of mid-training, pass@128 within the RL difficulty range approaches that of a reference pre-trained from scratch with the same difficulty range in both languages. Finally, given the resulting target-language coverage, RL language and difficulty shape where later reasoning gains generalize. Direct Korean RL generalizes more strongly to harder problems beyond the RL range than English-only transfer, while different Korean RL ranges favor different evaluation difficulties. These results show that target-language coverage established through pre-training and mid-training shapes how effectively later reasoning gains transfer across languages, beyond what is reflected in accuracy at the start of RL alone.