- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies OPD, where a student generates its own answers and a stronger teacher gives token-level feedback on them. Usually, the student trusts the teacher equally on each answer, and the task reward is integrated in an additive manner to the loss. The authors instead propose a new method which uses the task reward to weight each answer's distillation loss. Answers that score better than the other answers to the same question get more weight, through a softmax over group-relative advantages. The motivation behind the weighting is a Boltzmann model of trajectory optimality. The authors perform experiments for a Qwen3.6-27B teacher distilled into a Qwen3.5-9B student on math (AIME 2026 and HMMT). The main reported findings are that adding the reward as a separate loss term (as usually done) hurts the student in the long thinking regime, while their method preserves or slightly improves it.
Critical strengths
-
The proposed method is simple and easy to implement. It is a small change to standard OPD and it introduces no generation cost. There is only one hyperparameter and whenever the reward carries no information within a group the method reduces to standard OPD. This makes the method easy to adopt and implement.
-
The experiments are organized and structured well. The authors treat the design as three switches - posterior weighting, the additive reward term, and position decay. Then, the experiments are performed in combinations of them, including standard OPD with none of them. This type of comparison makes it possible to see what each one contributes. Specifically, this is how the authors show that the drop in performance in the long thinking regime comes from the additive reward term and not from long-context training itself, which is useful.
Critical weaknesses
-
The probabilistic view in Section 2, which justifies the method, is assumed by the authors and is not derived. The method assumes that low-reward trajectories drift into states the teacher rarely visits, so the teacher should be trusted more on high-reward trajectories. The Boltzmann model does not derive this. It only gives the probability that a trajectory is good, and the authors simply declare that probability to be the reliability weight. Jumping from “this trajectory is good” to “the teacher is reliable here” is the assumption itself, so Proposition 1 (which has no proof in the main body or appendix) only repeats the assumption in mathematical form. Moreover, the results in the paper contradict this assumption. Drifted trajectories (by this assumption, the low-reward ones) should show higher divergence from the teacher, yet Figure 3 shows that high-advantage trajectories diverge more. The paper then explains that successful trajectories follow “reasoning paths the teacher would not have taken”, which is itself a description of drift, and switches to a new rationale for the same weights (“the teacher still has something to teach there”). Also, divergence measures disagreement and not whether the teacher is correct. As a result, teacher reliability is never actually tested, even though it is in the title ("learning when to trust the teacher").
-
There are significant issues with the evaluation protocol. The results seem to be single runs, with no seeds, standard deviations or error bars. Moreover, the reported accuracy of each method is the best test accuracy across its saved checkpoints, which means selecting on the test set, while the untrained student is evaluated only once .The differences between methods are small and their reported ranges overlap, so without repeated runs and with selection on the test set, it is unclear whether the reported gains come from the method itself or from noise and selection bias. Also, the standard baseline, which adds the reward as a separate loss term, is only tested with a single coefficient value, so its poor performance could be due to this specific choice rather than to the additive method in general. Finally, the authors cite other reward-aware methods (RLAD, RG-OPD, RWOPD), as well as simple alternatives such as distilling only on correct answers, but none are used as baselines.
Questions for authors
-
How many seeds were performed per configuration in the experiments? Please report mean and standard deviation over at least three seeds. Also, please use a fair evaluation protocol, i.e. saving the last checkpoint or a checkpoint selected on validation.
-
Regarding the observation in weakness 1: if low-reward trajectories are the ones that drift away from the teacher, why do high-advantage trajectories show higher divergence in Figure 3?
-
The Boltzmann model gives the probability that a trajectory is good, but the method uses it as the probability that the teacher is reliable on that trajectory. Can you justify this step, either theoretically or empirically, rather than assuming it?
-
Can you directly test whether the teacher is less reliable on low-reward answers than on high-reward ones?
-
Can you re-run the additive baseline with additional coefficient values? Can you add additional baselines from your related works section such as RLAD, RG-OPD, RWOPD, and also simple alternatives?