acceptodds
Under review as a conference paper at ICLR 2027

Privileged Likelihood is not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

Abstract

On-policy self-distillation aims to improve upon reinforcement learning from verifiable rewards (RLVR) by providing token-level scores derived from privileged information, such as reference solutions or critic feedback available only during training. These scores are implicitly treated as estimates of token-level action values, yet they answer a fundamentally different question: how the model's prediction changes when its input context is enriched, rather than how the expected outcome changes when a token is changed. We examine this gap along three dimensions: (i) whether the token-level score tracks task success; (ii) whether feedback generated from the same rollout causes the score to reflect agreement with its own description, and whether using feedback from other rollouts in the group mitigates this self-referential effect; and (iii) what behavior the resulting training objective actually reinforces. In experiments with gpt-oss-20b on AIME 2025, the implemented two-sided score distinguishes correct from incorrect trajectories at approximately chance level (AUC=0.505); using feedback from a different rollout does not consistently improve this discrimination; and all five token-score training configurations achieve only 24.2–33.9% Avg@4, compared with 64.2% for outcome-only GRPO. Moreover, the highest-entropy token decile accounts for 57–71% of the total absolute token-advantage mass, despite the score being least informative about reasoning quality in this regime. By contrast, similar experiments with Qwen3-8B on SciKnowEval Biology improves held-out Avg@8 by 26.8–28.0%, while its corresponding trajectory scores achieve AUCs of 0.813–0.924. Together, these results suggest that dense credit assignment through distillation can be effective when its likelihood-based scores are empirically validated as meaningful proxies for outcome-relevant credit. When this alignment does not hold, however, the resulting supervision can fail to generalize and may substantially underperform outcome-based RL.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, self-distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rD4zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether changes in token probabilities after a model sees reference answers or feedback can tell us how much each token contributes to solving a problem. The authors propose three checks: whether the scores relate to correctness, whether the feedback comes from the solution being scored, and how the training objective uses the scores. On AIME 2025, the implemented score barely distinguishes correct from incorrect solutions. All five training configurations that add token scores perform substantially worse than GRPO using only final-answer rewards. The authors also report a successful case on biology questions, showing that the effectiveness of self-distillation depends on the setting.

Critical strengths
  1. The failure case has practical value. In Table 3, the five configurations that add token scores achieve only 24.2%–33.9% Avg@4, compared with 64.2% for outcome-only GRPO. The authors also examine completion and truncation rates. Models trained with token scores finish their answers more often but answer correctly less often. This shows that the performance drop cannot simply be attributed to unfinished answers.
  2. The three checks are useful for analyzing self-distillation methods. Beyond comparing final accuracy, the paper examines how scores relate to correctness, whether feedback depends on the solution being scored, and how training rules use the scores. The positive biology result also makes the discussion more balanced. These analyses can help researchers identify where problems arise.
Critical weaknesses
  1. The relationship between trajectory AUC and token credit is not sufficiently established. Sections 3.1 and 5.1 argue that if the average score of a whole solution cannot distinguish correctness, then token-level credit must be problematic. However, Proposition 1 concerns the relationship between token scores and action values at the same state. It does not directly establish that averaging processed scores across different positions must distinguish correct from incorrect solutions. In Table 1, the same correct-fact feedback has a pooled AUC of 0.416 but a within-prompt AUC of 0.587, illustrating that the comparison used can affect the conclusion. Also, a negative one-sided score on correct solutions does not by itself show that the teacher reduces the overall probability of success; Appendix A.10.1 already provides a relevant qualification. I see these results as warning signs, but the main text interprets them too strongly.
  2. The experiments do not fully establish what causes the performance drop. Positive feedback masks some numerical values and more often includes formatting requirements, while negative feedback provides explicit incorrect values. Score differences may therefore reflect both mathematical content and presentation. The five training configurations also combine several processing choices, leaving their individual effects unclear. In addition, each configuration uses only one training seed. The biology experiment changes the model, task, feedback, and training method together, and lacks an outcome-only baseline under the same setting. These results support the conclusion that the tested configurations perform poorly, but do not yet establish when likelihood-based scores will be effective.
Questions for authors
  1. [Clarification of the conclusion] Why does a failure of whole-solution average scores to distinguish correctness imply a problem with token-level credit? Please provide support for this connection. If the inference requires additional conditions, please state them explicitly and narrow the conclusions in Sections 3.1 and 5.1 accordingly.
  2. [Experimental controls] Positive and negative feedback differ in value masking, formatting requirements, and other details. Do you have control results using consistent templates? Also, does performance still drop substantially when the training weight of the token score is reduced? These results would help distinguish problems with the score itself from problems with how it is used.
  3. [Interpretation of the results] For the biology experiment, do you have outcome-only results under the same setting? Was score AUC measured before or after training? This affects whether AUC can be interpreted as predicting training gains. Please also include results from different training seeds if available.
Reviewer_HS7yOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study a genuine problem in on-policy self-distillation: whether token-level scores from a privileged teacher reflect task success, or only how the model's predictions shift when given enriched context. They test this through trajectory-level AUC against correctness, a cross-fitting control for self-dependence, and a comparison of training configurations against outcome-only GRPO, supported by theory showing that likelihood shifts need not equal action value.

Critical strengths

Right now much of the active research on OPSD (On Policy Self Distillation) treat the teacher's likelihood shift as token-level action value and is an ongoing research on how to make this learning better. This paper questions this assumption and verifies the research dependency on having good teacher models with access to privileged information.

This paper questions the entire concept of teacher driven action validation. I like theoretical foundation of good/not-so-good rollouts, privileged vs. Partially accurate information to the teacher, AUC with cross fitting etc. Three way decomposition of score/feedback/loss is also quite robust and gives practitioners a clear way in which to diagnose problems. Finally the authors are careful not to reject the approach: its Biology result suggests dense self-distillation works when the score is validated.

Critical weaknesses

Paper’s claim that privileged score is not automatically a value to lean has not been thoroughly established – results are on one benchmark gpt-oss-20b and one dataset AIME 2025, which is difficult to generalize. Moreover, the entire substantiation uses only one seed per configuration and one custom feedback construction. For instance, can we correlate inconclusive AUC on poor training? Authors tend to suggest this in the case of the Biology workload where good AUC is being attributed to training success. Paper starts off with token-level credit/importance but all the empirical results are at a trajectory level, there seems to be only a logical level association of trajectory failure to token prediction failure.

Questions for authors
  1. How many seeds did you run, do you have run-to-run variance for outcome-only GRPO (with seed changes) and at least one dense configuration?
  2. The Biology contrast differs from AIME in model, task, privileged information, and loss. Is there a way to show that low AUC causes the training failure?
  3. Have you validated token-level credit exclusively?
Reviewer_gNwCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether token-level likelihood differences induced by privileged context can serve as effective credit-assignment signals, examining outcome alignment, feedback self-dependence, and training objectives. Through theoretical discussion and observations across different tasks, the authors emphasize that changes in likelihood should not be directly equated with a token’s contribution to task success.

Critical strengths
  1. The paper addresses a practically relevant question. Distinguishing probability changes induced by privileged information from task value helps clarify the role and limitations of self-distillation signals.
  2. The main text presents a clear analytical framework, discussing scoring, feedback construction, and optimization separately, while covering both successful and unsuccessful settings.
Critical weaknesses
  1. The rationale for using trajectory-level AUC to evaluate token-level credit could be explained further. This metric measures how well aggregated scores distinguish correct from incorrect final answers, but its connection to individual tokens’ contributions remains somewhat unclear. The inference from low AUC to ineffective credit assignment would therefore benefit from further explanation.
  2. The interpretation of negative likelihood differences on correct trajectories in the one-sided teacher–policy residual analysis in Section 5.1 could be more precise. A decrease in the probability of one correct path may coincide with an increase in the probabilities of other correct paths. The relationship between this observation and task performance or the direction of the credit signal would benefit from clarification.
Questions for authors
  1. Could the authors further explain the rationale for using trajectory-level AUC to evaluate token-level credit? Specifically, how does the relationship between aggregated token scores and final-answer correctness support an assessment of local credit-assignment effectiveness?
  2. For negative likelihood differences on correct trajectories, how do the authors distinguish an inappropriate direction of the credit signal from a redistribution of probability among different correct paths? Further clarification of the interpretation in the main text would be helpful.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.