acceptodds
Under review as a conference paper at ICLR 2027

Posterior-Weighted On-Policy Distillation: Learning When to Trust the Teacher

Abstract

On-policy distillation (OPD) trains a smaller student on its own rollouts, using token-level feedback from a stronger teacher. Yet it trusts the teacher equally on every rollout, and the task reward, when used, enters only as a hand-tuned additive term that competes with distillation. We ask when teacher guidance should be trusted, and answer with a derivation rather than a heuristic: under a Boltzmann model of trajectory optimality, the principled per-trajectory weight is the group-normalized posterior over group-relative advantages. Posterior-Weighted OPD (PW-OPD) thereby demotes the reward from a competing objective to a modulation of a fixed distillation budget—one new temperature, used untuned—and recovers standard OPD exactly when the reward carries no within-group information. We verify the premise directly: student–teacher divergence on high-advantage trajectories is that on low-advantage ones, in both the short- and long-context regimes. Distilling a 27B teacher into a 9B student, we find that in the long-context thinking regime the additive objective drives AIME accuracy from the untrained student's 0.90 down to 0.75–0.82; a reward-free control attributes the damage to the additive term. Posterior weighting preserves and marginally improves the student (0.90–0.92), and the same ordering—posterior above additive above uniform—replicates in a routed multi-teacher extension with a different student and five domain benchmarks.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Large Language Models, Group-Relative Advantage, Knowledge Distillation, Reward-aware Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_MqhqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies OPD, where a student generates its own answers and a stronger teacher gives token-level feedback on them. Usually, the student trusts the teacher equally on each answer, and the task reward is integrated in an additive manner to the loss. The authors instead propose a new method which uses the task reward to weight each answer's distillation loss. Answers that score better than the other answers to the same question get more weight, through a softmax over group-relative advantages. The motivation behind the weighting is a Boltzmann model of trajectory optimality. The authors perform experiments for a Qwen3.6-27B teacher distilled into a Qwen3.5-9B student on math (AIME 2026 and HMMT). The main reported findings are that adding the reward as a separate loss term (as usually done) hurts the student in the long thinking regime, while their method preserves or slightly improves it.

Critical strengths
  1. The proposed method is simple and easy to implement. It is a small change to standard OPD and it introduces no generation cost. There is only one hyperparameter and whenever the reward carries no information within a group the method reduces to standard OPD. This makes the method easy to adopt and implement.

  2. The experiments are organized and structured well. The authors treat the design as three switches - posterior weighting, the additive reward term, and position decay. Then, the experiments are performed in combinations of them, including standard OPD with none of them. This type of comparison makes it possible to see what each one contributes. Specifically, this is how the authors show that the drop in performance in the long thinking regime comes from the additive reward term and not from long-context training itself, which is useful.

Critical weaknesses
  1. The probabilistic view in Section 2, which justifies the method, is assumed by the authors and is not derived. The method assumes that low-reward trajectories drift into states the teacher rarely visits, so the teacher should be trusted more on high-reward trajectories. The Boltzmann model does not derive this. It only gives the probability that a trajectory is good, and the authors simply declare that probability to be the reliability weight. Jumping from “this trajectory is good” to “the teacher is reliable here” is the assumption itself, so Proposition 1 (which has no proof in the main body or appendix) only repeats the assumption in mathematical form. Moreover, the results in the paper contradict this assumption. Drifted trajectories (by this assumption, the low-reward ones) should show higher divergence from the teacher, yet Figure 3 shows that high-advantage trajectories diverge more. The paper then explains that successful trajectories follow “reasoning paths the teacher would not have taken”, which is itself a description of drift, and switches to a new rationale for the same weights (“the teacher still has something to teach there”). Also, divergence measures disagreement and not whether the teacher is correct. As a result, teacher reliability is never actually tested, even though it is in the title ("learning when to trust the teacher").

  2. There are significant issues with the evaluation protocol. The results seem to be single runs, with no seeds, standard deviations or error bars. Moreover, the reported accuracy of each method is the best test accuracy across its saved checkpoints, which means selecting on the test set, while the untrained student is evaluated only once .The differences between methods are small and their reported ranges overlap, so without repeated runs and with selection on the test set, it is unclear whether the reported gains come from the method itself or from noise and selection bias. Also, the standard baseline, which adds the reward as a separate loss term, is only tested with a single coefficient value, so its poor performance could be due to this specific choice rather than to the additive method in general. Finally, the authors cite other reward-aware methods (RLAD, RG-OPD, RWOPD), as well as simple alternatives such as distilling only on correct answers, but none are used as baselines.

Questions for authors
  1. How many seeds were performed per configuration in the experiments? Please report mean and standard deviation over at least three seeds. Also, please use a fair evaluation protocol, i.e. saving the last checkpoint or a checkpoint selected on validation.

  2. Regarding the observation in weakness 1: if low-reward trajectories are the ones that drift away from the teacher, why do high-advantage trajectories show higher divergence in Figure 3?

  3. The Boltzmann model gives the probability that a trajectory is good, but the method uses it as the probability that the teacher is reliable on that trajectory. Can you justify this step, either theoretically or empirically, rather than assuming it?

  4. Can you directly test whether the teacher is less reliable on low-reward answers than on high-reward ones?

  5. Can you re-run the additive baseline with additional coefficient values? Can you add additional baselines from your related works section such as RLAD, RG-OPD, RWOPD, and also simple alternatives?

Reviewer_zZ38OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PW-OPD weights each student trajectory's distillation loss by a group-normalized softmax of its outcome-reward advantage. It reduces to uniform weighting when the group rewards agree and can be combined with token-position decay. Experiments compare pure OPD, posterior weighting, and an additive task-reward objective in short- and long-context math settings and a routed multi-teacher setting. Weighting avoids the large degradation of the tested additive objective in the long-context experiments, while gains over pure OPD are smaller.

Critical strengths
  1. The change is simple and uses trajectories and outcome scores that are already available. The uniform-weight limit is easy to understand, and no extra rollouts are needed to compute the weights. It is a practical way to use task rewards in distillation.

  2. The experiments include pure OPD and separate trajectory weighting from token-position modulation (Tables 1-2). The multi-teacher experiment uses another student and also favors posterior weighting in mean score (Table 3). These comparisons provide useful evidence that weighting can work better than the particular additive objective tested here.

Critical weaknesses
  1. The evaluation makes the size and reliability of the improvement difficult to judge. External benchmark scores use the best accuracy across saved checkpoints (Section 4.1 and Appendix B). Applying this rule to all methods is consistent, but can still favor methods differently when their checkpoint variability differs. The additive reward coefficient is fixed at 1.0, so its failure does not establish an advantage over tuned additive objectives. This matters because the gains over pure OPD are modest, and the multi-teacher mean exceeds the additive baseline by 0.40 points. These evaluation choices are the main reason I am not yet convinced by the broader superiority claim.

  2. The fixed-budget interpretation needs a qualification. Equation 4 fixes the sum of trajectory weights, but Equation 5 divides the weighted token losses by the unweighted token count. With unequal response lengths, the total weighted token mass need not stay fixed. Thus weighting may change the overall loss scale as well as its allocation. This does not imply that the method performs poorly; it means the budget-preservation argument does not directly carry over to the stated token aggregation.

Questions for authors
  1. What do the ranges in Table 2 represent, and how do the methods compare at checkpoints selected without using the final test benchmarks? Are there existing results with another additive reward coefficient that would show whether its degradation is specific to 1.0?

  2. Was Equation 5 used as written? Please clarify what quantity is kept fixed when response lengths differ, and whether the weighted/unweighted token-mass ratio changes appreciably in the reported runs.

Reviewer_ZU8POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies two limitations of existing On-Policy Distillation (OPD) methods: standard OPD applies uniform distillation across all rollouts, while approaches combining OPD with reinforcement learning introduce task rewards as an independent objective added to the distillation loss, requiring a manually tuned balancing coefficient and potentially creating conflicts between the two objectives.

To address these issues, the authors propose Posterior-Weighted On-Policy Distillation (PW-OPD). Building on the control-as-inference framework, they approximate the posterior probability of trajectory optimality using a softmax over group-normalized advantages and use the resulting weights to scale each trajectory's distillation loss. In this way, task rewards no longer serve as an independent optimization objective but instead determine how teacher supervision is distributed across trajectories. The authors argue that, since the weights sum to the group size, no additional balancing coefficient is required.

The experiments use Qwen3.6-27B as the teacher and Qwen3.5-9B as the student, evaluating performance on AIME 2026 and HMMT 2026 under both non-thinking and thinking modes. Additional multi-teacher distillation experiments are also conducted. The authors report that PW-OPD outperforms uniform OPD in these setting.

Critical strengths
  1. The method is simple, clear, and easy to integrate into existing OPD pipelines. The softmax weights are derived from a control-as-inference view rather than chosen heuristically, they remove the balancing coefficient needed when combining OPD with an RL objective, and the method reduces to standard OPD when all rewards in a group are identical. Importance: high. This is the main reason for my positive assessment.

  2. The experiments cover both non-thinking (8K) and thinking (32K) modes and extend to multi-teacher distillation across several domains, suggesting some degree of generality. Importance: moderate. The results support the main claim, but their strength depends on the evaluation-variance questions raised below.

Critical weaknesses

OPD+TR uses a single coefficient of 1.0 (§3.1), and it degrades the student only in the 32K regime; in the 8K regime (Table 1) and the multi-teacher setting (Table 3), it outperforms uniform OPD. Results with a few coefficient values would show whether the degradation comes from the additive structure or from the coefficient choice.

Questions for authors

Coefficient of the additive term. Did the authors try coefficients other than 1.0 for OPD+TR? If so, does the 32K degradation persist with smaller values?

Reviewer_gHRuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new AI training approach called PW-OPD. Normally, when a "student" model learns from a "teacher" model, all of the student's attempts are treated equally. PW-OPD changes this by focusing the teacher's guidance on the student's most successful (highest-reward) responses. Rather than treating the reward and the teacher's feedback as two separate goals to be added together, this method uses the reward score to decide exactly where to spend a fixed budget of teacher feedback. Tests with Qwen models show this technique beats traditional methods, especially for complex, long-context reasoning and when learning from multiple teachers at once.

Critical strengths

The key strength is that the paper considers a very relevant topic and the proposed approach is simple and practical.

  1. The paper addresses an important question in OPD: whether teacher supervision should be applied uniformly across student-generated trajectories, regardless of their quality. The proposed group-normalized weighting is straightforward to implement and practical.

  2. The authors compare posterior weighting against pure OPD, additive reward training, and position-aware weighting, providing some insight into the interaction between these components.

  3. The reported 2.8× difference in student–teacher divergence between high- and low-advantage trajectories is noteworthy.

Critical weaknesses

Here are some of the key weakness of the paper. TL;DR is needs more detialed and careful evaluation to validate the claims.

  1. Conceptual justification is incomplete: The method assumes high-reward trajectories are those where teacher guidance is most valuable. However, higher student reward does not imply greater teacher reliability. The reported 2.8× student–teacher divergence shows disagreement, not that teacher guidance is more useful on these trajectories.

  2. Incremental gains are modest: PW-OPD offers only small improvements over pure OPD, with overlapping performance ranges and no statistical significance analysis. The multi-teacher improvement is similarly marginal.

  3. Lacks empirical rigor: The paper lacks clear multi-seed results, uncertainty estimates, and significance tests. Sensitivity to the weighting temperature, group size, and reward distribution is also insufficiently explored.

Budget normalization is not fully established: Normalizing sequence weights to sum to the group size does not necessarily preserve the effective token-level loss when response lengths vary. Likewise, the position-decay normalization over the full horizon does not guarantee a unit mean over valid tokens.

Questions for authors

Beside the issues pointed in the Weakness section, here are few additional questions:

  1. What are the measured end-to-end training-time and GPU-hour savings compared with standard OPD?
  2. Do the reported accuracy improvements remain statistically significant across more random seeds, with confidence intervals computed at the question level?
  3. What is the individual contribution of randomized continuation, history compensation, and centering?
  4. How does the method compare with existing horizon-reduction approaches at matched computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.