acceptodds
Under review as a conference paper at ICLR 2027

Context-consistent Teacher Signals for On-policy Distillation

Abstract

On-policy distillation (OPD) provides dense teacher supervision on student- generated trajectories, but the teacher signal is typically evaluated under a single context. Such single-context supervision can be sensitive to contextual changes, making it difficult to distinguish stable learning signals from context-specific teacher preferences. Can we identify teacher supervision that remains consistent across task-relevant contexts and use it to improve on-policy distillation? We introduce Factorial On-Policy Distillation (FOPD), which evaluates identical student actions across teacher checkpoints and contexts to identify cross-context consistent supervision. FOPD centers the resulting teacher signals over the student’s action distribution, retains directions consistently supported across contexts with a conservative shared magnitude, and anchors the update to the standard OPD direction. On Qwen3-1.7B in non-thinking mode across 14 mathematics, code, and science benchmarks, FOPD achieves a Macro score of 43.35, corresponding to a 5.9% relative improvement over the strongest baseline, and improves performance on 13 of the 14 tasks. Ablation studies show that removing or replacing any defining component reduces Macro performance by 1.82–5.85 points. Controls matched for reward sparsity or magnitude remain 2.35–2.65 points below FOPD, while token-level evaluation shows that FOPD achieves the highest direction precision throughout the evaluated 10–70% coverage range. These results demonstrate that cross-context consistency provides an effective principle for identifying reliable token-level supervision in on-policy distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
on policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CzFfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Factorial On-Policy Distillation (FOPD), which identifies token-level supervision that remains consistent across plain and privileged teacher contexts. The method compares base and final teacher predictions under both contexts and retains checkpoint-induced preference directions that agree across contexts and align with standard OPD.

Critical strengths
  1. The paper addresses an important problem of identifying reliable token-level supervision in on-policy distillation.

  2. The proposed method is relatively simple and can be incorporated into existing OPD frameworks without changing student rollout generation.

  3. The experiments demonstrate consistent improvements across multiple benchmarks and model configurations.

Critical weaknesses
  1. The central motivation for cross-context consistency is insufficiently justified.

    1. The paper assumes that teacher preferences that vary across contexts are less reliable. However, privileged information is intended to provide additional knowledge unavailable in the plain context. Therefore, context-dependent changes in teacher preferences may represent useful supervision rather than noise. It is unclear why agreement between plain and privileged contexts should be preferred over the privileged teacher's judgment.
  2. A critical privileged-only distillation baseline is missing.

    1. Since FOPD introduces both privileged information and cross-context consistency filtering, the current comparisons cannot isolate the contribution of the proposed consistency mechanism.
  3. Computational efficiency and statistical reliability require further evaluation.

    1. FOPD evaluates four teacher-scoring configurations, potentially increasing training overhead relative to OPD2. However, compute-matched comparisons and detailed wall-clock measurements are missing.
  4. The presentation quality of the figures needs substantial improvement.

    1. Figures 1 and 2 are overcrowded with text, mathematical notation, and visual elements. Their small font sizes make the main ideas difficult to follow.
Questions for authors

Please check the weaknesses above.

Reviewer_15BBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a method to incorporate several different teacher signals in an OPD setup. The authors use both a post-trained and base teacher, and evaluate both under standard context, and recently popular privileged contexts, and use these four signals to supervise a student model during on policy distillation training.

Critical strengths
  • The paper shows reliable gains on its aggregated benchmarks compared to the methods it lists.
  • The motivation is pretty reasonable - separating out signal from privileged context, domain specificity, etc.
Critical weaknesses
  • Figure 2 is completely unreasonable. Squeezing in smaller and smaller fonts expecting a reviewer to zoom in 500% just to read a figure is not good exposition or science.
  • Feeding in privileged context into a base model seems counterintuitive at the least, and awkward.
  • Result reporting for thinking models is heavily aggregated, and important comparisons to things like OPSD, etc.
  • Confidence intervals are not reported, except possibly in Figure 3, where it is unexplained and quite strange looking.
Questions for authors
  • How do you handle training a thinking student? How does the base teacher even score the thinking trace?
  • No raw thinking model performance listed on your aggregate comparisons?
Reviewer_bmmSOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers OPD using a modified context teacher signal to improve performance. The advantage-like quantities A∗(a)A_{*}(a) are computed using various methods including with a standard teacher and with teacher using a context with hints. The paper then computes sts_t which has a magnitude that is the minimum of the AA's computed with and without hints. This sts_t is gated and used as a weighting term per token in the standard policy gradient in (11). The authors then show that FOPD works well across different benchmarks (Table 3). Table 5 shows informative ablations across design choices. Table 7 shows modification of OPD2OPD^2 with some of FOPD's characteristics does not make OPD2OPD^2 match FOPDFOPD's performance. Overall, a satisfying paper with good results and justification.

Critical strengths

Overall, the paper:

  • Has good theoretical motivation and a clear derivation of the weighting method.
  • Good results showing significant improvement using the new FOPD method.
  • Extensive ablations which explore why the new method works.
  • Exploration of some of the characteristics of FOPD applied to OPD2OPD^2 with FOPD still being the clear winner
Critical weaknesses

Two weaknesses of the paper are:

  • Computationally this approach is expensive -- we need to compute the teacher on both the original context and the context with hints
  • The method is dependent on high-quality hints (from a "golden" source) and access to that type of information may be limited in some problems (but the authors do consider the effect of context quality). Both of these weaknesses are really a property of the approach and would be difficult to address.
Questions for authors

No major questions -- good paper.

1 comment

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.