- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Standard outcome-based RLVR assigns the same advantage to every token in a rollout, potentially penalizing correct reasoning within failed responses. The paper proposes verbalized self-distillation (VSD) to provide token-level guidance. A frozen writer compares a failed rollout with a correct sibling and generates failure-specific instructions. A frozen teacher conditioned on these instructions scores the original failed rollout. The teacher–student forward KL divergence is approximated using top- tokens and a tail bucket, then mapped to bounded, sign-preserving multipliers for negative advantages. On four out-of-distribution mathematical reasoning benchmarks, VSD improves final mean pass@1 over GRPO by 3.59 percentage points on Qwen3-4B and 1.61 points on Qwen3-1.7B. Additional experiments compare teacher contexts and uniform attenuation, examine weight distributions, and evaluate whether instructions improve the frozen model’s accuracy.
Critical strengths
-
VSD provides a simple way to use self-generated, failure-specific instructions for advantage shaping rather than directly minimizing a distillation loss. The bounded multipliers preserve the advantage’s sign, and neither the writer nor the teacher is needed at inference time.
-
VSD consistently improves four-benchmark mean pass@1 over GRPO at matched checkpoints on both model sizes. Under the same weighting rule, instructions outperform raw correct sibling solutions and no privileged context. The separate inference experiment also shows that instructions improve the frozen model’s accuracy on a filtered subset of DAPO-Math problems.
Critical weaknesses
-
The paper lacks discussion and comparison with closely related work. [c1] already motivates feedback aligned to the student’s errors rather than raw reference solutions, while [c2] generates natural-language hindsight skills, including corrective guidance, to construct token-level training signals. Neither is cited. VSD differs in using instruction-conditioned forward KL to reweight negative advantages, but the significance and benefit of this distinction remain insufficiently established.
-
The advantage-shaping rule remains largely heuristic. Forward KL measures distributional disagreement, but does not establish whether the sampled token is correct or whether the teacher assigns it lower probability. Consequently, the rule does not ensure that stronger penalties correspond to erroneous tokens and weaker penalties to correct ones. The paper also provides limited justification for the clipped linear mapping, weight bounds , and interpolation parameter . For Qwen3-1.7B, is reduced from 1 to 0.5 because more covered tokens remain at the minimum multiplier, without a performance comparison validating this adjustment. The weight distributions describe the rule’s behavior but do not establish the appropriateness or robustness of these choices.
-
The additional computational cost is insufficiently examined. Writer generation and teacher scoring introduce extra model inference during training, but the paper does not analyze the resulting computational complexity or time overhead. It remains unclear whether the gains justify these additional operations or persist when GRPO receives comparable computational resources.
[c1] The Role of Feedback Alignment in Self-Distillation.
[c2] SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning.
Questions for authors
-
What evidence shows that larger forward KL identifies tokens requiring stronger negative updates? Can you relate KL and the resulting multipliers to actual reasoning errors, distinguishing cases where the teacher increases versus decreases the sampled token’s probability?
-
What motivates the clipped linear mapping followed by interpolation with 1? How were , the weight bounds, and selected, and how sensitive is performance to these choices? In particular, does reducing from 1 to 0.5 on Qwen3-1.7B improve benchmark performance, beyond moving more tokens away from the minimum multiplier?
-
Uniform 0.75 modifies every negative advantage, whereas VSD modifies only covered rollouts. Can you apply uniform attenuation to exactly the same covered rollouts, leaving other rollouts unchanged? Matching VSD’s average multiplier would further help distinguish token-level weighting from overall penalty reduction.
-
How does VSD differ from [c1] and [c2] in its use of corrective guidance? What evidence supports its forward-KL advantage shaping over the feedback-conditioned training approaches in these works?
-
How much additional computation and time do writer generation and teacher scoring require? How does this overhead depend on rollout length and the fraction of covered rollouts, and do VSD’s gains persist under comparable computational resources?
3 comments
We sincerely thank the reviewer for engaging with our response and for raising their score. We will add the intervention study and the other new results to the revised paper.
Dear Reviewer wt3f,
Thank you for reading the rebuttal and responding to it. Your efforts in reviewing this paper are very much appreciated!
Best,
AC
Thank you for the additional experiments and clarifications. The intervention study provides useful evidence that high-KL positions contain actionable information. I will therefore raise my score to weak acceptance.