acceptodds
Under review as a conference paper at ICLR 2027

Crosslingual On-Policy Self-Distillation for Multilingual Reasoning

Abstract

Large language models (LLMs) have achieved remarkable progress in mathematical reasoning, but this ability is not equally accessible across languages. Especially low-resource languages exhibit much lower performance. To address this, we propose Crosslingual On-Policy Self-Distillation (COPSD), which transfers a model’s own high-resource reasoning behavior to low-resource languages. COPSD uses the same model as student and teacher: the student sees only the low-resource problem, while the teacher receives privileged crosslingual context, including the problem translation and reference solution in English. Training minimizes full-distribution token-level divergence on the student’s own rollouts, providing dense supervision while avoiding the sparsity and instability of outcome-only reinforcement learning (RL). Experiments on 17 low-resource African languages show that COPSD consistently improves mathematical reasoning across model sizes and substantially outperforms alternative post-training baselines, including Group Relative Policy Optimization (GRPO) and supervised fine-tuning (SFT). Further analysis shows that COPSD improves answer-format adherence, strengthens test-time scaling, mitigates repetitive degeneration, and generalizes to harder multilingual reasoning benchmarks, with especially large gains for lower-resource languages.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
less-resourced languages, cross-lingual transfer, distillation, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RL2hOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Crosslingual On-Policy Self-Distillation (COPSD) to improve mathematical reasoning in low-resource languages. The same model serves as both student and teacher: the student sees only the low-resource question, while the teacher additionally receives the English question and reference solution as privileged information. COPSD distills the teacher’s token-level distributions over student-generated trajectories. Experiments across Qwen3-1.7B/4B/8B, 17 African languages on AfriMGSM, and eight PolyMath languages show that COPSD consistently outperforms Base, GRPO, Trace-SFT, and Translation-SFT, with faster convergence and larger gains on harder tasks and lower-resource languages.

Critical strengths

The work addresses a well-motivated problem in low-resource language reasoning and the proposed method is simple and practical, requiring neither target-language rationales nor external teachers. The experiments are extensive across languages, model scales, and benchmarks, providing evidence of its effectiveness and generalization.

Critical weaknesses

1.The comparison with Translation-SFT may not be fully fair. Both methods rely on English reference solutions, while Translation-SFT additionally depends on translation quality and uses hard targets rather than distributional supervision. The authors should better control translation-related factors and further analyze whether COPSD’s gains come from distributional supervision or differences in target quality.

2.The test-time scaling claim should be stated more carefully. COPSD achieves larger absolute gains with increased generation budgets, but its relative gains are not consistently higher than the baselines. For example, Base shows a slightly larger relative improvement than COPSD on the 8B model. On the 1.7B model, COPSD continues to improve at 2,048 tokens while GRPO and Trace-SFT degrade, suggesting better scaling robustness, but the reason for this behavior remains unclear.

3.The paper lacks a direct evaluation of generation quality in low-resource languages. Although it attributes the challenge to weak low-resource generation and imperfect teacher signals, it does not evaluate generation fluency and grammaticality, how COPSD gains correlate with language-specific generation capability. Such analyses would provide stronger evidence for the proposed knowledge-transfer mechanism beyond final Pass@12 results.

Questions for authors

1.Why does COPSD plateau on the 1.7B model but exhibit performance degradation after early peaks on the 4B and 8B models?

4 comments

Dear Reviewer RL2h,

Thank you for your thoughtful review and positive feedback. We appreciate your recognition of the paper's motivation, clarity, novelty, and soundness. We want to address your concerns below.

1.The comparison with Translation-SFT may not be fully fair...

Thank you for raising this point. Translation quality can indeed affect SFT performance, and this is precisely one of our motivations for proposing COPSD: avoiding the costly and potentially unreliable translation of reasoning steps, especially for extremely low-resource languages. Translation is therefore an integral part of the Translation-SFT setup. In other words, Translation-SFT is a baseline where we allow for imperfect automatic translations, following a common translate-training approach in the community [1, 2, 3]. And our comparison evaluates the effectiveness of these complete training approaches under practical conditions.

Additionally, Trace-SFT provides a complementary baseline that does not require translating reference solutions: it trains on target-language traces generated using the same privileged teacher context. COPSD also outperforms both Translation-SFT and Trace-SFT, supporting COPSD as a better option when a reliable translator cannot be guaranteed, especially for extremely low-resource languages as in our setup Nevertheless, it would be quite interesting (but also expensive and beyond the scope of this paper) to investigate how translation quality affects Translation-SFT when translations are produced by (1) humans, (2) dedicated translation systems, (3) open-weight LLMs, and (4) closed-source LLMs.

2.The test-time scaling claim should be stated more carefully...

This is a good observation. We agree that COPSD's relative gains are not consistently higher than those of the baselines. However, relative improvement also depends on performance at the lower generation budget. For example, on the 8B model, COPSD achieves a larger absolute gain than Base (5.44 vs. 4.68 percentage points), despite a slightly smaller relative improvement (30.0% vs. 31.8%). We will revise the discussion in the test-time scaling part accordingly.

We also appreciate your observation regarding COPSD's better scaling robustness on the 1.7B model. One possible contributing factor is its more concise generation (see Table below): under the same 4,096-token budget, COPSD generates 1,680 tokens on average, compared with 2,991 for GRPO and 2,498 for Trace-SFT. Prior work has identified a length bias in the original GRPO objective that can encourage excessively long responses [4]. Trace-SFT may also retain verbosity from its model-generated training traces. These observations are consistent with our hypothesis that COPSD uses the available budget more effectively.

MethodQwen3-1.7BQwen3-4BQwen3-8B
Base2,987.331,884.342,905.19
GRPO2,990.991,903.922,915.17
Trace-SFT2,497.851,666.962,651.86
Translation-SFT2,121.601,481.272,335.03
COPSD1,680.381,556.622,213.75

3.The paper lacks a direct evaluation of generation quality in low-resource languages...

Thank you for this suggestion. We agree that target-language generation capability may influence how much a model benefits from COPSD. To examine this, we evaluated models on English-to-target-language translation using FLORES+, with 200 examples per language across AfriMGSM languages, providing an assessment of conditioned target-language generation. We then computed correlations between per-language chrF++ scores and COPSD's absolute Pass@12 gains over Base, as shown below (p-values are from two-sided permutation tests with 9,999 shuffles).

ModelAvg. chrF++Pearson rp-valueSpearman ρp-value
Qwen3-1.7B2.360.4190.10850.4120.1148
Qwen3-4B3.97−0.2380.3493−0.1970.4577
Qwen3-8B5.92−0.4570.0893−0.3640.1677

The low translation scores suggest that generation in these languages remains challenging. In addition, the correlations are not consistent across model sizes, and none is statistically significant. These results indicate that there is no simple relationship between generation capability and COPSD gains and that other factors may also play a role.

We also want to mention that the gains may depend on how well the model can reason in English, as COPSD aims to transfer reasoning capability from English to other languages. As you can see in our PolyMath evaluation, we observe larger improvements for Swahili and Telugu, both low-resource languages. However, this does not mean that the model has better generation capability in these two languages than in Japanese or Spanish (with fewer improvements). In other words, COPSD gains may depend on both target-language generation capability and English reasoning capability, among other factors. We will include this discussion in our camera-ready version.

Response to questions:

1.Why does COPSD plateau on the 1.7B model but exhibit performance degradation after early peaks on the 4B and 8B models?

Indeed, training instability appears to be a broader challenge for OPSD-like methods. Related work reports degradation during continued self-distillation [1], and we observed similar issues when reproducing OPSD using its original codebase [2]. The difference across model sizes may reflect model-specific responses to continued updates. The 1.7B model may reach a point where further teacher guidance brings limited benefit, whereas larger models may adapt more strongly to both useful and imperfect teacher signals, producing early gains followed by degradation. Nevertheless, with validation-based checkpoint selection, we observe average improvements over the baselines at all three scales. We believe that further stabilizing training to sustain these gains is an important direction for future work.

[1] Lai and Nissim (2024). mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models.
[2] Hwang et al. (2025), Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning.
[3] Sutawika et al. (2026), Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning.
[4] Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective.
[5] Kim et al. (2026). Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
[6] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.

Your rebuttal has adequately addressed my concerns. Considering the scope of this work, I maintain my original score.

Thank you very much for engaging in the discussion and for confirming that our rebuttal has addressed your concerns. We appreciate your positive assessment and will incorporate the additional analyses and clarifications into the camera-ready version.

Reviewer_scUqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Crosslingual On-Policy Self-Distillation (COPSD), an on-policy self-distillation framework which transfers a model’s own high-resource reasoning behavior to low-resource languages. COPSD uses the same LLM as student and teacher, giving the teacher privileged English problem and solution context for distillation along rollouts of the student. Experiments across 17 African languages and three model sizes of Qwen3 show gains over the base model and improvements over GRPO and SFT baselines.

Critical strengths
  • The paper extends on-policy self-distillation to crosslingual reasoning by giving the teacher access to English problems and reference solutions. This is a practical way to improve reasoning in low-resource languages without translating full solutions or using a separate larger teacher model.
Critical weaknesses
  • The paper does not explain how the quality of translated reference solutions was checked. This matters because translation quality could substantially affect the baseline of Translation-SFT’s performance.
  • COPSD's performance drops noticeably in later training stages, especially for larger models, raising concerns about its training stability. Also, the paper only reports Pass@12 without Pass@1, so it's unclear whether COPSD actually improves the accuracy of individual responses.
Questions for authors
  • How about SFT using the original English reference solutions, paired with either English or low-resource-language prompts? This could help clarify what COPSD adds over a simpler SFT baseline when language consistency is relaxed.
  • Beyond accuracy, could the authors provide average response lengths for COPSD and all baselines? This would help assess whether COPSD's performance gains are associated with longer reasoning traces.
  • Could the authors clarify the reason for using Pass@12? How about the performance of Pass@1? Moreover, what's the rason for setting the response budget to 1024 in Fig.2 but 4096 in Tab.1?
  • In Section 5.1 (line 222), the authors claim that Qwen3's post-training data is dominated by high-resource languages such as English. However, the post-training section of the Qwen3 Technical Report states, "To enhance the performance on tasks with low-resource languages, we particularly increase the proportion of translation tasks," without providing explicit statistics on the language distribution of post-training data. Could the authors provide supporting evidence for this claim or further clarify this statement?
4 comments

Dear Reviewer scUq,

Thank you for your careful review and constructive comments. We would like to address your concerns below.

The paper does not explain how the quality of translated reference solutions was checked. This matters because translation quality could substantially affect the baseline of Translation-SFT’s performance.

Intuitively, translation quality can affect SFT performance, and this is exactly why we aim to avoid SFT that requires translating reasoning steps (cf. Lines 041–046) and propose COPSD, which does not require this costly and potentially unreliable process. In other words, Translation-SFT is a baseline where we allow for imperfect automatic translations, following a common translate-training approach in the community [1, 2, 3]. From this perspective, extensively evaluating translation quality would not substantially strengthen the main motivation of our work. In fact, our experiments show that COPSD generally outperforms Translation-SFT, supporting COPSD as a better option when a reliable translator cannot be guaranteed, especially for extremely low-resource languages as in our setup. Nevertheless, it would be interesting (but also expensive and beyond the scope of this paper) to investigate how translation quality affects Translation-SFT when translations are produced by (1) humans, (2) dedicated translation systems, (3) open-weight LLMs, and (4) closed-source LLMs.

COPSD's performance drops noticeably in later training stages, especially for larger models, raising concerns about its training stability.

Indeed, training instability appears to be a broader challenge for OPSD-like methods. Related work has also reported performance degradation as self-distillation training progresses [4]. In fact, when reproducing OPSD using its original codebase [5], we observed similar issues. The rapid early gains suggest that models can quickly benefit from the teacher's dense distillation signal. In our multilingual setup, this instability might be further exacerbated by limited target-language generation capability (cf. Lines 362–367). Nevertheless, with validation-based checkpoint selection (a standard practice in model training), we observe improvements over the baselines on average across model scales. Although developing a general solution to training instability is beyond the scope of this work, we would welcome future research building on our method to further stabilize training and enable COPSD-like methods to benefit from longer training.

Also, the paper only reports Pass@12 without Pass@1, so it's unclear whether COPSD actually improves the accuracy of individual responses.

We use Pass@12 as the main metric to assess the model's potential. AfriMGSM covers very low-resource languages, where single-response accuracy (Pass@1) can remain low even after training. As suggested by the reviewer, we report both Pass@1 and Pass@12 under a 4,096-token generation budget below for all three model scales. The results show that COPSD achieves the highest Pass@1 as well as Pass@12 at every scale, confirming that it also improves the accuracy of individual responses. We will include the Pass@1 results in the camera-ready version.

Method1.7B Pass@11.7B Pass@124B Pass@14B Pass@128B Pass@18B Pass@12
Base1.469.114.2019.204.9719.41
GRPO1.499.184.2519.364.8819.22
Trace-SFT1.5710.123.8919.114.7120.42
Translation-SFT1.9312.564.4020.125.0721.11
COPSD2.6515.534.8820.616.0723.55

[1] Lai and Nissim (2024). mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models.
[2] Hwang et al. (2025), Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning.
[3] Sutawika et al. (2026), Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning.
[4] Kim et al. (2026). Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
[5] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.

Response to questions:

How about SFT using the original English reference solutions, paired with either English or low-resource-language prompts? This could help clarify what COPSD adds over a simpler SFT baseline when language consistency is relaxed.

This comparison would also be interesting, particularly when language consistency is relaxed. However, SFT using the original English reference solutions, paired with either English or low-resource-language prompts, would explicitly encourage the model to reason in English. This differs from our motivation: improving reasoning while keeping the reasoning language consistent with the input language. We therefore focus on baselines that pursue the same target-language reasoning objective.

Beyond accuracy, could the authors provide average response lengths for COPSD and all baselines? This would help assess whether COPSD's performance gains are associated with longer reasoning traces.

Thank you for this question. We report the average response lengths across the 17 AfriMGSM languages under a 4,096-token generation budget below. COPSD generates shorter responses than Base at all three model scales, while GRPO produces slightly longer responses than Base. Trace-SFT produces longer responses than Translation-SFT, which may reflect the difference between training on model-generated traces and translated, relatively concise English reference solutions. Overall, COPSD achieves higher accuracy than Base while generating shorter responses, suggesting that its gains are not associated with longer reasoning traces, but due to more concise and effective target-language reasoning.

MethodQwen3-1.7BQwen3-4BQwen3-8B
Base2,987.331,884.342,905.19
GRPO2,990.991,903.922,915.17
Trace-SFT2,497.851,666.962,651.86
Translation-SFT2,121.601,481.272,335.03
COPSD1,680.381,556.622,213.75

Could the authors clarify the reason for using Pass@12? How about the performance of Pass@1? Moreover, what's the rason for setting the response budget to 1024 in Fig.2 but 4096 in Tab.1?

Please refer to our response to the major concern above for the motivation behind using Pass@12 and the additional Pass@1 results. We use a 1,024-token generation budget in Figure 2 for computational efficiency when analyzing training dynamics; the overall trends remain similar under a 4,096-token budget. For Table 1, which presents our main results, we use the larger budget to better assess the models' reasoning performance within our available computational resources and compare COPSD with the baselines. Furthermore, Figure 4 shows that COPSD's improvements over Base persist across all evaluated generation budgets.

In Section 5.1 (line 222), the authors claim that Qwen3's post-training data is dominated by high-resource languages such as English. However, the post-training section of the Qwen3 Technical Report states, "To enhance the performance on tasks with low-resource languages, we particularly increase the proportion of translation tasks," without providing explicit statistics on the language distribution of post-training data. Could the authors provide supporting evidence for this claim or further clarify this statement?

Thank you for pointing this out. Indeed, we double-checked that the Qwen3 Technical Report does not disclose the language distribution of its post-training data, so our original wording was a bit strong. On the other hand, the report describes increasing translation tasks for low-resource languages, but does not specify the language coverage of its long-CoT training data, which is most relevant to us. We will revise this statement accordingly in our camera-ready version.

I appreciate the authors’ detailed response and the additional results provided. While my concerns regarding Pass@12 and response lengths have been addressed, I remain unconvinced that evaluating the translation quality of Translation-SFT is unnecessary, as unverified translation quality could undermine the validity of this baseline comparison. Moreover, the newly reported PolyMath language-compliance results show that COPSD reduces language compliance compared to Base, suggesting that it does not consistently promote target-language reasoning, which highlights the need for a comparison with English-based SFT to isolate gains in reasoning performance. Finally, COPSD closely follows the existing OPSD framework [1], and the late-stage performance degradation remains unresolved, limiting the novelty of this work.

Given that these core concerns remain unaddressed, I will maintain my original rating.

[1] Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

We thank the reviewer for engaging further in the discussion, and we appreciate that the reviewer considers several of the original concerns to have been addressed. Regarding the concerns that the reviewer views as unresolved, however, we would respectfully disagree that they undermine the validity or novelty of our work.

First, Translation-SFT should be evaluated as a complete baseline setup. Our objective is not to establish whether better or worse translations affect SFT performance; intuitively, they do. Instead, we compare COPSD against the standard translate-and-train setting, in which reasoning traces are automatically translated into the target language. Our results show that COPSD outperforms this practical baseline without requiring translated reasoning traces, which directly supports our motivation. A systematic evaluation involving human translations or multiple translation systems would answer a different research question and is beyond the scope of this paper.

Second, language compliance alone provides an incomplete assessment of target-language reasoning. Our objective is to improve correct reasoning in the target language, which requires considering language control and reasoning performance jointly. The compliant Pass@1 and Pass@12 metrics capture precisely this combination. COPSD consistently outperforms Base and other baselines on these metrics, showing that it produces more correct solutions whose reasoning is expressed in the target language. Therefore, the slight decrease in language compliance alone does not support the conclusion that COPSD fails to improve target-language reasoning.

Third, English-based SFT does not share our target-language reasoning objective. Training directly on English solutions would naturally encourage the model to reason in English and would therefore change the task rather than provide a controlled baseline for our setting. Models may indeed reason more accurately in English, but our goal is specifically to improve reasoning that remains accessible in the user’s target language. While English-based SFT could be relevant to work with a different objective, we do not consider it essential for evaluating the stated contribution of this paper.

Finally, late-stage degradation is an optimization challenge rather than evidence against the method’s contribution. Although eliminating this degradation would be desirable, validation-based checkpoint selection is standard practice and consistently yields improvements over the baselines across model scales. The existence of later degradation does not negate the clear gains achieved during training. Moreover, to the best of our knowledge, our work is the first to use an OPSD-style method to transfer a model’s own high-resource-language reasoning capability to low-resource languages and to demonstrate its effectiveness across a broad multilingual evaluation. We understand that novelty can be assessed differently across research communities, but we believe this constitutes a substantial contribution to multilingual NLP.

We respectfully hope that the reviewer will assess the validity and novelty of the paper in light of its stated objective and the additional evidence provided in our rebuttal.

Reviewer_WsSAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Crosslingual On-Policy Self-Distillation (COPSD) for mathematical reasoning in low-resource languages. The training data consists of a 0.5K subset of OpenThoughts with English problems and full English reference solutions, only the problems are translated into each target language. The same language model is instantiated as a student policy and a teacher policy. The student sees only the target-language problem and generates an on-policy reasoning trajectory, whereas the fixed teacher additionally receives the English problem and English reference solution as privileged cross-lingual context. Both policies evaluate the same student-generated prefixes, and training minimizes the full-vocabulary token-level forward KL divergence from the teacher to the student, with gradients flowing only through the student policy. A separate model is trained for each target language and model size. The training and inference prompts use a language-specific instruction with a target-language prefix immediately after <think> to encourage target-language reasoning; the student does not access the English reference at inference time.

The main AfriMGSM evaluation covers 17 African languages and three Qwen3 sizes, samples 12 responses per problem under a 4,096-token generation budget, selects checkpoints on 50 validation problems, and reports Pass@12 on 200 held-out problems over 10 random splits. Table 1 reports average Pass@12 increases from 9.2%, 19.2%, and 19.3% to 15.6%, 20.5%, and 23.4% for Qwen3-1.7B, 4B, and 8B, respectively. Additional experiments train a separate Qwen3-4B model for each of eight PolyMath languages and include a small-scale DeepSeek-R1-Distill-Llama-8B extension on three languages; these probe applicability across benchmarks and model families, rather than zero-shot transfer from one model to eight unseen languages.

Critical strengths
  1. The main experiment includes useful privileged-information controls. Trace-SFT uses the teacher prompt to generate target-language reasoning traces, whereas Translation-SFT translates the English reference solutions into the target language. Thus, COPSD is compared with alternatives that also receive privileged information when constructing supervision, although the forms of supervision are not identical. COPSD has the highest average Pass@12 at all three model sizes, although its margin over Translation-SFT is small at 4B. For low-resource languages that lack high-quality target-language reasoning traces, this training setup has potential practical value.

  2. The paper includes useful analyses of training dynamics, including negative results. The results show later Pass@12 degradation for the larger models; the paper does not present additional training as uniformly beneficial.

Critical weaknesses
  1. The core claim about target-language reasoning is not reliably measured. Section 5.2 frames the goal as enabling users without English proficiency to inspect intermediate steps. However, Appendix B marks traces classified as English or French, as well as failed detections, as non-compliant; every other detected language is counted as compliant. Because most target languages are absent from langdetect's default language profiles, this permissive proxy cannot distinguish the target language from another language, mixed-language text, or a misclassification. Table 4 shows that the gain survives this permissive filter ( Compliant Pass@12 of 20.40% vs. 16.21% for Qwen3-8B), but this does not establish that the correct reasoning is in the target language or understandable to target-language users. For Qwen3-1.7B, the compliance rate even decreases from 80.32% for Base to 76.25% for COPSD. Direct target-language identification and human evaluation of whether the reasoning is understandable in the target language are needed before the Pass@12 gains can be interpreted as improved target-language reasoning.

  2. The incremental contribution over prior privileged-information training is not yet sufficiently established. [1] already instantiates the same model as teacher and student, uses reference solutions and student-generated trajectories, and applies token-level distillation. [2] transfers high-resource context to low-resource languages through on-policy and off-policy self-distillation, although its task is safety alignment rather than mathematical reasoning. The claimed increment therefore, depends on showing what is new beyond this mechanism in the mathematical setting and in controlled empirical comparisons. However, the direct RL comparator for the claimed advantage over sparse outcome rewards is only binary-outcome GRPO. [3] (SP3F) uses English reference responses as privileged information for pairwise feedback and can provide relative preferences even when none of the candidate responses is fully correct, but the submission only groups it with related work in Section 2 and does not distinguish or compare against it. The existing SFT controls are useful, but they do not determine whether dense token-level distillation is better than privileged pairwise feedback under comparable information and budget. This comparison would materially affect the assessment of COPSD's incremental contribution.

[1] Siyan Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.

[2] Ruiyang Qin et al. Multilingual Safety Alignment via Self-Distillation. arXiv:2605.02971, 2026.

[3] Lintang Sutawika et al. Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning. arXiv:2601.18722, 2026.

Questions for authors
  1. Could you conduct reliable target-language identification and human evaluation of whether the reasoning is understandable in the target language on a stratified sample covering different languages and methods, and report the fraction of responses that are correct and whose reasoning is expressed in the target language? Please include the languages for which the current filter is least reliable and the Qwen3-1.7B setting where compliance decreases.

  2. Under the same English reference solution, target-language problem, and comparable budget, how does COPSD compare with a privileged pairwise-feedback method? At least on a set of representative languages, this would clarify the value of dense token-level distillation relative to an existing privileged-feedback alternative.

  3. Using the existing 12 samples, could you report Pass@1 or per-sample accuracy? Figure 3 shows that most newly solved problem-language pairs already had a format-valid Base response, but this does not fully rule out gains from producing more validly formatted completions. Could you also report accuracy stratified by completion/truncation status or with a more permissive answer extractor?

  4. Please report student-generated-token counts, teacher forward-pass token counts, training-token counts, GPU time, and variation across independent training runs for each method. Table 6 uses eight generations per prompt with a maximum completion length of 16,000 tokens for GRPO, versus one generation with a maximum completion length of 2,048 tokens for COPSD; equal numbers of optimization steps do not establish comparable compute. Measuring whether teacher feedback on incorrect or repetitive student prefixes improves later correctness would also help explain Figure 2's late-training degradation.

2 comments

Dear Reviewer WsSA,

Thank you very much for your insightful feedback. We are encouraged that you found our paper clear and easy to follow. Your two main concerns focus on additional analysis and discussion of recent related work. We address both concerns below.

The core claim about target-language reasoning is not reliably measured.

Indeed, target-language reasoning, i.e., reasoning in the same language as the input, is our goal. However, reliable automatic language identification remains challenging for many of the very low-resource languages covered in AfriMGSM. While human evaluation could provide additional evidence, comprehensive coverage of these languages is beyond our available resources. We would therefore like to present two pieces of evidence regarding language control and improvements in target-language reasoning.

  1. As pointed out by the reviewer, our AfriMGSM analysis uses a permissive proxy for language control by distinguishing English, French, and other detected languages. This is motivated by the concern that models may switch to English during reasoning, due to English-centric training. The proxy checks for such language drift. If the reasoning language is not detected as English (the dominant language) or French (the language that had significant influence on African languages due to historical reasons), the reasoning language is most likely the target language. But we acknowledge that this strategy does not, by itself, establish that the reasoning is in the requested language. We therefore complement this analysis with direct target-language identification on PolyMath.

  2. For PolyMath, fortunately, langdetect supports all eight languages covered in our evaluation: Swahili (SWA), Telugu (TEL), Thai (THA), Russian (RUS), Bengali (BEN), Japanese (JPN), Chinese (ZHO), and Spanish (SPA). Following the reviewer's suggestion, we directly check whether the detected language of each reasoning trace matches the input language. The results are shown below, where we report the average compliance rate (LC), Pass@1 (P@1) and Pass@12 (P@12), and their compliant counterparts (CP@1 and CP@12) for each difficulty level (Low, Medium, and High).

MethodLow: LCLow: P@1Low: CP@1Low: P@12Low: CP@12Medium: LCMedium: P@1Medium: CP@1Medium: P@12Medium: CP@12High: LCHigh: P@1High: CP@1High: P@12High: CP@12
Base95.1070.7266.7285.8085.0095.4032.7830.5351.2049.5097.3614.0213.4127.7026.60
COPSD92.3871.0866.3989.2087.4085.8137.0732.1563.3059.7086.9915.9413.8635.8031.40

The results support our claim: COPSD maintains average language compliance of 85.81% to 92.38%, although slightly lower than Base, while improving Compliant Pass@12 at all three difficulty levels. Compliant Pass@1 also improves at medium and high difficulty. Thus, performance gains remain when correct responses are required to match the input language, providing evidence that our goal of improving target-language reasoning is generally achieved.

The incremental contribution over prior privileged-information training is not yet sufficiently established.

Thank you for mentioning these three papers, all of which are already cited and discussed in our Related Work (Lines 098–110). We respectfully disagree that COPSD's contribution is insufficiently distinct from these works. Our contribution lies in developing and systematically validating crosslingual on-policy self-distillation for low-resource mathematical reasoning, with the goal of making a model's reasoning capabilities more accessible across languages.

[1] shares the underlying on-policy self-distillation technique, but does not address our crosslingual reasoning problem. We investigate a completely distinct and important question: whether a model can use English reference solutions to supervise its own reasoning trajectories in low-resource languages, without requiring translated target-language reasoning traces. Therefore, we believe the proposed COPSD method constitutes a substantive contribution to multilingual reasoning for the multilingual community.

[2] is concurrent work with a different objective. As discussed in our paper, both studies explore self-distillation in multilingual settings. However, [2] transfers safety behavior using multilingual queries, whereas COPSD uses privileged English reference solutions to guide mathematical reasoning in the target language. These works address distinct capabilities and provide complementary evidence about crosslingual self-distillation.

[3] (SP3F) uses a substantially different training framework. It applies RL using verifiable rewards and privileged pairwise feedback. During RL, an external judge, GPT-4o-mini, receives the English reference solution and compares candidate responses; their pairwise win rates are then used as rewards. Reliance on an external API-based judge introduces additional cost and is precisely what we aim to avoid. Our motivation is to make reasoning more accessible across languages by drawing on the model's own capabilities. In COPSD, the same model serves as teacher and student, with privileged English information directly conditioning token-level supervision on the student’s own rollouts. COPSD therefore requires no separate judge during training. This is a concrete distinction in both the supervision mechanism and the required training resources. A direct comparison with SP3F would therefore add limited value to our paper's main contribution, which is to establish that a model can improve its own target-language reasoning through self-distillation without relying on external judge supervision.

We therefore maintain that COPSD makes a distinct and substantive contribution to the multilingual community. We will make these distinctions more explicit in the camera-ready version.

Response to questions:

  1. Please see our response to the first main concern. We now report direct target-language identification on all eight PolyMath languages, together with language compliance, Compliant Pass@1, and Compliant Pass@12. These results provide additional evidence that performance gains persist when correct responses are required to match the input language.

  2. Please see our response to the second main concern, where we explain the differences in supervision mechanisms and training resources. COPSD improves target-language reasoning through the model's own capabilities without an external judge, which is central to our contribution.

  3. We now report Pass@1 estimated from the existing 12 responses, together with Compliant Pass@1, on PolyMath (see our response to the first concern). For AfriMGSM, the table below reports overall accuracy (equivalent to estimated Pass@1) and accuracy restricted to responses with extractable final answers, under the same 4,096-token budget. COPSD achieves the highest overall accuracy across all three model sizes. Accuracy among extractable responses varies across methods, suggesting that improved answer-format adherence should also be considered when interpreting the overall gains (recall that COPSD also improves the answer-format adherence as discussed in Section 6.1).

Method1.7B Accuracy1.7B Accuracy among extractable responses4B Accuracy4B Accuracy among extractable responses8B Accuracy8B Accuracy among extractable responses
Base1.463.704.205.854.9710.24
GRPO1.493.604.255.954.8810.10
Trace-SFT1.573.183.895.424.719.10
Translation-SFT1.933.024.405.295.078.38
COPSD2.653.334.885.896.079.32
  1. In the spirit of ICLR 2027's guidance on conciseness, we will include the training details and generation statistics in the Appendix. This keeps the rebuttal focused on the main concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their thoughtful feedback. WsSA, scUq, and RL2h all described the paper as clear and easy to follow. scUq highlighted the practical value of improving low-resource reasoning without translating full solutions or relying on a separate larger teacher, while RL2h emphasized the extensive evaluation across languages, model scales, and benchmarks, as well as the method's effectiveness and generalizability. RL2h also specifically recognized the novelty of the method. To the best of our knowledge, this is among the first works to use on-policy self-distillation (OPSD) to transfer a model's own reasoning capability from a high-resource language to low-resource languages, which we believe is a substantial contribution to the multilingual research community.

During the rebuttal period, we have carefully addressed all concerns raised in the reviews, with detailed responses and additional results on four key points:

  • Effectiveness of target-language reasoning: For AfriMGSM, our language-compliance proxy was designed to detect the most likely form of language drift, i.e., switching to English or French, because reliable language identification is unavailable for most of its low-resource languages. To further strengthen the evidence of effective language control, we added direct target-language identification for all eight PolyMath languages, which are fully supported by the language detector. We report language compliance together with Pass@1 and Pass@12, as well as compliant Pass@1 and Pass@12 restricted to reasoning traces identified as being in the requested language. These results show that COPSD maintains high target-language compliance while consistently improving compliant reasoning accuracy, directly supporting its effectiveness for target-language reasoning.

  • Comparison with related work: We clarified that COPSD is among the first approaches to use OPSD for transferring a model's own high-resource reasoning capability to low-resource languages. Unlike offline SFT, COPSD distills dense token-level feedback on the student’s own trajectories, reducing the distribution mismatch between training targets and the student policy. It also differs from outcome-based GRPO and privileged pairwise-feedback methods by providing supervision at every generated token, even when the model cannot yet produce a fully correct solution. We will expand the related-work discussion to make these distinctions and COPSD's contribution to multilingual mathematical reasoning more explicit.

  • Translation quality in the SFT baseline: Translation-SFT represents the established translate-and-train setting, in which the quality of automatically translated reasoning traces is an inherent practical limitation, especially for very low-resource languages. Avoiding this potentially costly and unreliable translation process is precisely one of COPSD's motivations: it transfers supervision from the English reference solution through the teacher's token distributions without translating the full reasoning trace. COPSD's consistent improvements over the Translation-SFT baseline therefore demonstrate its practical advantage when high-quality reasoning translations cannot be guaranteed.

  • Pass@1, generation length, and additional analyses: We added Pass@1 results for all three model sizes and further reported language-compliant counterparts of both Pass@1 and Pass@12. COPSD consistently improves Pass@1, Pass@12, and their compliant counterparts, providing stronger evidence that it improves both individual-response accuracy and target-language reasoning. We also found that COPSD generates shorter responses than the base model at all three model scales while achieving higher accuracy. These results confirm that its gains come from more accurate and efficient reasoning trajectories, rather than longer generations or additional sampling opportunities.

We hope that the reviewers, ACs, SACs, and PCs will take these clarifications and additional empirical results into account when assessing this submission. We welcome further discussion and invite reviewers to update their scores if their concerns have been resolved. We believe that doing so will support a fair, evidence-based evaluation of this submission and a constructive review process.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.