- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether changes in token probabilities after a model sees reference answers or feedback can tell us how much each token contributes to solving a problem. The authors propose three checks: whether the scores relate to correctness, whether the feedback comes from the solution being scored, and how the training objective uses the scores. On AIME 2025, the implemented score barely distinguishes correct from incorrect solutions. All five training configurations that add token scores perform substantially worse than GRPO using only final-answer rewards. The authors also report a successful case on biology questions, showing that the effectiveness of self-distillation depends on the setting.
Critical strengths
- The failure case has practical value. In Table 3, the five configurations that add token scores achieve only 24.2%–33.9% Avg@4, compared with 64.2% for outcome-only GRPO. The authors also examine completion and truncation rates. Models trained with token scores finish their answers more often but answer correctly less often. This shows that the performance drop cannot simply be attributed to unfinished answers.
- The three checks are useful for analyzing self-distillation methods. Beyond comparing final accuracy, the paper examines how scores relate to correctness, whether feedback depends on the solution being scored, and how training rules use the scores. The positive biology result also makes the discussion more balanced. These analyses can help researchers identify where problems arise.
Critical weaknesses
- The relationship between trajectory AUC and token credit is not sufficiently established. Sections 3.1 and 5.1 argue that if the average score of a whole solution cannot distinguish correctness, then token-level credit must be problematic. However, Proposition 1 concerns the relationship between token scores and action values at the same state. It does not directly establish that averaging processed scores across different positions must distinguish correct from incorrect solutions. In Table 1, the same correct-fact feedback has a pooled AUC of 0.416 but a within-prompt AUC of 0.587, illustrating that the comparison used can affect the conclusion. Also, a negative one-sided score on correct solutions does not by itself show that the teacher reduces the overall probability of success; Appendix A.10.1 already provides a relevant qualification. I see these results as warning signs, but the main text interprets them too strongly.
- The experiments do not fully establish what causes the performance drop. Positive feedback masks some numerical values and more often includes formatting requirements, while negative feedback provides explicit incorrect values. Score differences may therefore reflect both mathematical content and presentation. The five training configurations also combine several processing choices, leaving their individual effects unclear. In addition, each configuration uses only one training seed. The biology experiment changes the model, task, feedback, and training method together, and lacks an outcome-only baseline under the same setting. These results support the conclusion that the tested configurations perform poorly, but do not yet establish when likelihood-based scores will be effective.
Questions for authors
- [Clarification of the conclusion] Why does a failure of whole-solution average scores to distinguish correctness imply a problem with token-level credit? Please provide support for this connection. If the inference requires additional conditions, please state them explicitly and narrow the conclusions in Sections 3.1 and 5.1 accordingly.
- [Experimental controls] Positive and negative feedback differ in value masking, formatting requirements, and other details. Do you have control results using consistent templates? Also, does performance still drop substantially when the training weight of the token score is reduced? These results would help distinguish problems with the score itself from problems with how it is used.
- [Interpretation of the results] For the biology experiment, do you have outcome-only results under the same setting? Was score AUC measured before or after training? This affects whether AUC can be interpreted as predicting training gains. Please also include results from different training seeds if available.
Dear Reviewers,
Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.
Thank you.
Best,
AC