acceptodds
Under review as a conference paper at ICLR 2027

APL-OPSD: Instance-Adaptive Privileged Language Selection for Multilingual On-Policy Self-Distillation

Abstract

Large language models acquire knowledge from multilingual corpora, but this knowledge is not equally accessible through every language. The most informative language can vary across questions, domains, and cultural contexts, making fixed English supervision suboptimal. We propose Adaptive Privileged Language On Policy Self Distillation (APL-OPSD), a framework that selects the language of privileged context separately for each training instance. Given parallel question and answer pairs in multiple languages, APL-OPSD performs a one time offline probe that measures how each candidate context changes the likelihood of the target language reference answer and selects the most informative non target language. During training, the student observes only the target language question and generates an on policy response, while a frozen teacher additionally observes the selected crosslingual context and provides dense token level supervision over the same trajectory. Experiments across multiple languages, model scales, and tasks show that APL-OPSD achieves the strongest aggregate performance among the compared methods. Further ablations suggest that informative language selection contributes beyond merely adding privileged context. APL-OPSD jointly trains a single multilingual model and requires no additional routing, translation, or reference information at inference time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, On-Policy Self-Distillation, Multilingual

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_383GOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method for multilingual on-policy self-distillation that selects a language to encode privileged information for the teacher separately for each training example. During distillation, the student receives each question in the target language, while the teacher has access to privileged information (e.g., the answer) in a language that is selected according to the proposed method. The privileged information language is selected by measuring which candidate context minimizes the perplexity of the target-language reference answer under the model. Overall, I think this is a promising approach.

Critical strengths

I am generally positive about the paper. The experiments show consistent improvements, with encouraging results on CARE. The comparisons against fixed-English, random-language, and target-language baselines show that the proposed selection procedure is effective.

Critical weaknesses

My main concern is the lack of a baseline using the source language of each dataset example as the teacher’s privileged context. As this seems like the most natural way to do OPSD in this scenario, rather than translating the input which is what the paper is proposing.

The paper shows improved performance over English and the target language, but doesn't compare to just using the original language of the question. It would also be interesting to see how often the proposed selection method chooses the original source language.

I think the paper could also do a better job of explaining why the proposed method should be expected to work better than the english-only or original language baselines. The paper offers an explanation based on making knowledge accessible through different linguistic pathways, which seems plausible, but the description of the mechanism for how this might happen is left a bit vague. It would be nice to have a more specific hypothesis for how this works, along with some evidence to support it.

Questions for authors

see above

Reviewer_dZvCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In multilingual LLMs, certain cultural questions may be answered correctly in one language but incorrectly in another. The paper argues that knowledge is therefore not equally accessible across languages. Existing cross-lingual on-policy self distillation methods, however, use English as the privileged supervision language, which assumes English is optimal for every instance of OPSD. To address this limitation, the paper proposes APL-OPSD which selects the most informative candidate language (one that reduces the perplexity the most) for each training instance in OPSD.

For each training example, the paper evaluates 14 non-target languages using a frozen model and selects the language whose translated question–answer pair reduces perplexity on the target-language reference answer. During training, the student generates responses using only the target-language question, while a frozen teacher receives the selected privileged context and provides token-level KL-divergence supervision on the student's trajectories. An anchoring loss limits policy drift. The trained model requires no additional information or language routing at inference.

Experiments use Qwen3-1.7B, 4B, and 8B, with 3,936 training examples spanning 15 languages. Evaluation covers CARE, CARE-Pro, mGSM-v2, Global-MMLU-Lite, and INCLUDE-Lite, comprising 22,274 evaluation examples across 54 languages. The paper measures against SFT, GRPO, LRPO, COPSD, and the original model. APL-OPSD shows improvement of about 1.5-3 points over fixed-English COPSD in overall performance. The paper provides ablations on language selection strategy and privileged context content to support the proposed method. The largest improvements are on culturally grounded knowledge.

Critical strengths

A key strength of APL-OPSD is that it does not have any additional inference-time overhead, because the privileged-language selection is performed offline. Additionally, the method can be applied across different target languages within a single multilingual model, without requiring additional language routing or translation at inference time. Also, the formulation is simple and straightforward, which means that it is relatively easy to integrate into existing OPSD frameworks.

Critical weaknesses
  1. The improvement over other methods is not significant and the variance is not reported. APL-OPSD advantages over the base are only <2 points overall (Table 1). Therefore, it is hard to access without standard deviation or confidence intervals. Also, in the benchmark table (Table 1), several baseline cells are seen to be better and are not discussed properly. For example, CARE-Pro Seen at 8B drops from 35.01 (base) to 32.99; mGSM-v2 Seen at 4B drops from 93.84 to 92.38; SFT beats APL-OPSD on both multiple-choice benchmarks at 1.7B. The largest claimed gain, about 2 points on CARE, comes from 450 examples scored by an LLM judge on a 1 to 10 scale rescaled to 100. For these types of example, reporting per-seed variation, paired significance tests, and aggregation sensitivity would help strengthen the empirical conclusions.

  2. The paper has limited validation of the language-selection signal and the quality of translated training data. The main motivation is that certain languages provide better access to specific knowledge. However, the central assumption that reducing reference-answer perplexity finds a more informative privileged language may not always be true. The reduction could also be because of translation quality, lexical overlap, or prompt-specific effects. Appendix A.2 describes automated translation and script-level validation but doesn’t include a semantic-quality audit, and does not clearly explain how reference answers are selected from the source preference datasets. Since the teacher receives the reference answer as privileged information, I believe stronger controls are needed to distinguish genuine language-dependent knowledge transfer from differences in how effectively translated answers can be exploited.

Questions for authors
  1. Could the authors report per-seed standard deviations or confidence intervals for Table 1? Are the observed improvements statistically significant across model scales and benchmarks?
  2. Why does LRPO lose about 10 points on Global-MMLU-Lite and Include-Lite at 1.7B relative to base? Did you run its warm-start SFT, and in which thinking mode were LRPO and COPSD trained and evaluated? Was COPSD evaluated under the same On/On configuration used for your main results?
  3. Table 5, Case 1: adding the exact Italian gold answer as context raises the perplexity of that same Italian answer by 67 percent. A model given the exact answer should find it easier, not harder, to predict. What explains this? Sensitivity to prompt-formatting? Does perplexity-based selection actually capture genuine language-specific information rather than translation or tokenization artifacts?
  4. How are reference answers selected from CARE and HelpSteer3 and translated into the 15 languages?

Suggestions: Report the compute cost of the 15-way translation plus 14 probe passes per instance relative to training.

Reviewer_FvuVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the choice of privileged language in multilingual on-policy self-distillation. Existing methods such as COPSD always give the teacher English context. APL-OPSD uses the frozen initial model to measure how much each non-target-language QA pair lowers the perplexity of the target-language reference answer. It picks the best language for each instance and then performs token-level self-distillation. Experiments cover three Qwen3 sizes and five multilingual benchmarks.

Critical strengths
  1. The paper points out a design choice that prior work overlooked: the privileged language is always English.
  2. All results use three seeds, and the comparisons with Fixed English, Target Language, and Random Language are informative.
  3. No language selection or extra input is needed at inference.
Critical weaknesses
  1. APLS scores a language using a QA pair that already contains the translated answer. This mainly reflects how easily the answer can be copied or transliterated, as in the Table 5 cases, rather than knowledge access.
  2. Overall gains are 1.1 to 1.3 points, Table 1 has no confidence intervals, and mGSM-v2 shows almost no gain. It is hard to tell which differences are significant.
  3. Most gains come from CARE, and about 72% of the training data comes from the CARE training split. Benefits for other task types remain unclear.
Questions for authors
  1. If selection uses only the question, without the translated answer, does APLS still help?
  2. How often does the selected language share a script with the target answer, or allow direct transliteration of it?
  3. Can you give confidence intervals or significance tests for the gains over COPSD?
  4. Can you report results separately for CARE-derived and HelpSteer3-derived data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.