acceptodds
Under review as a conference paper at ICLR 2027

Simple Self-distillation through Error Verbalization

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the de facto approach to enhancing model performance on tasks such as coding and mathematical reasoning. A common feature among popular RL algorithms, such as GRPO, is to assign every token of a failed rollout the same negative advantage, penalizing sound and unsound reasoning steps alike. *Self-distillation* prompts a teacher model with access to privileged information, such as a correct solution drawn from the model's own successful rollouts, to produce token-level learning signals. Inspired by this, we propose *verbalized self-distillation* (VSD), a novel method for reweighting negative advantages in GRPO. Rather than conditioning the teacher on a raw correct solution, a *writer model* is given a successful rollout and the failed attempt, and is tasked with identifying the first point of divergence in the attempt and distilling this diagnosis into hindsight instructions. The teacher is then conditioned on these instructions instead of the raw rollout. The per-token divergence between the teacher and the current policy is computed over the policy's failed rollout, and subsequently used to reweight the advantages. Across four mathematical reasoning benchmarks, VSD consistently outperforms GRPO and other self-distillation baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, GRPO, Credit Assignment, Self Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wyowOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Standard outcome-based RLVR assigns the same advantage to every token in a rollout, potentially penalizing correct reasoning within failed responses. The paper proposes verbalized self-distillation (VSD) to provide token-level guidance. A frozen writer compares a failed rollout with a correct sibling and generates failure-specific instructions. A frozen teacher conditioned on these instructions scores the original failed rollout. The teacher–student forward KL divergence is approximated using top-kk tokens and a tail bucket, then mapped to bounded, sign-preserving multipliers for negative advantages. On four out-of-distribution mathematical reasoning benchmarks, VSD improves final mean pass@1 over GRPO by 3.59 percentage points on Qwen3-4B and 1.61 points on Qwen3-1.7B. Additional experiments compare teacher contexts and uniform attenuation, examine weight distributions, and evaluate whether instructions improve the frozen model’s accuracy.

Critical strengths
  1. VSD provides a simple way to use self-generated, failure-specific instructions for advantage shaping rather than directly minimizing a distillation loss. The bounded multipliers preserve the advantage’s sign, and neither the writer nor the teacher is needed at inference time.

  2. VSD consistently improves four-benchmark mean pass@1 over GRPO at matched checkpoints on both model sizes. Under the same weighting rule, instructions outperform raw correct sibling solutions and no privileged context. The separate inference experiment also shows that instructions improve the frozen model’s accuracy on a filtered subset of DAPO-Math problems.

Critical weaknesses
  1. The paper lacks discussion and comparison with closely related work. [c1] already motivates feedback aligned to the student’s errors rather than raw reference solutions, while [c2] generates natural-language hindsight skills, including corrective guidance, to construct token-level training signals. Neither is cited. VSD differs in using instruction-conditioned forward KL to reweight negative advantages, but the significance and benefit of this distinction remain insufficiently established.

  2. The advantage-shaping rule remains largely heuristic. Forward KL measures distributional disagreement, but does not establish whether the sampled token is correct or whether the teacher assigns it lower probability. Consequently, the rule does not ensure that stronger penalties correspond to erroneous tokens and weaker penalties to correct ones. The paper also provides limited justification for the clipped linear mapping, weight bounds [0.5,1.5][0.5,1.5], and interpolation parameter λ=0.5\lambda=0.5. For Qwen3-1.7B, τ\tau is reduced from 1 to 0.5 because more covered tokens remain at the minimum multiplier, without a performance comparison validating this adjustment. The weight distributions describe the rule’s behavior but do not establish the appropriateness or robustness of these choices.

  3. The additional computational cost is insufficiently examined. Writer generation and teacher scoring introduce extra model inference during training, but the paper does not analyze the resulting computational complexity or time overhead. It remains unclear whether the gains justify these additional operations or persist when GRPO receives comparable computational resources.

[c1] The Role of Feedback Alignment in Self-Distillation.

[c2] SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning.

Questions for authors
  1. What evidence shows that larger forward KL identifies tokens requiring stronger negative updates? Can you relate KL and the resulting multipliers to actual reasoning errors, distinguishing cases where the teacher increases versus decreases the sampled token’s probability?

  2. What motivates the clipped linear mapping followed by interpolation with 1? How were λ\lambda, the weight bounds, and τ\tau selected, and how sensitive is performance to these choices? In particular, does reducing τ\tau from 1 to 0.5 on Qwen3-1.7B improve benchmark performance, beyond moving more tokens away from the minimum multiplier?

  3. Uniform 0.75 modifies every negative advantage, whereas VSD modifies only covered rollouts. Can you apply uniform attenuation to exactly the same covered rollouts, leaving other rollouts unchanged? Matching VSD’s average multiplier would further help distinguish token-level weighting from overall penalty reduction.

  4. How does VSD differ from [c1] and [c2] in its use of corrective guidance? What evidence supports its forward-KL advantage shaping over the feedback-conditioned training approaches in these works?

  5. How much additional computation and time do writer generation and teacher scoring require? How does this overhead depend on rollout length and the fraction of covered rollouts, and do VSD’s gains persist under comparable computational resources?

3 comments

Thank you for the additional experiments and clarifications. The intervention study provides useful evidence that high-KL positions contain actionable information. I will therefore raise my score to weak acceptance.

We sincerely thank the reviewer for engaging with our response and for raising their score. We will add the intervention study and the other new results to the revised paper.

Dear Reviewer wt3f,

Thank you for reading the rebuttal and responding to it. Your efforts in reviewing this paper are very much appreciated!

Best,

AC

Reviewer_wt3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces verbalized self-distillation (VSD) to reweight negative advantages in GRPO for mathematical reasoning. When a sampled group contains both a failed rollout and a correct sibling, a writer model compares them and generates instructions on how to avoid the error. A teacher model uses those instructions to score the failed rollout, and the teacher-to-student KL divergence between their next-token distributions rescales each token's negative advantage. On Qwen3-4B and Qwen3-1.7B trained on DAPO-Math-17K, VSD improves mean pass@1 at step 250 over GRPO by 3.59 percentage points and by 1.61 percentage points, respectively, across four benchmarks.

Critical strengths
  • Conditioning the teacher on failure-specific instructions increases the step 250 mean pass@1 over the Raw privilege control by 3.35 percentage points on Qwen3-4B and 1.40 percentage points on Qwen3-1.7B.
  • Rescaling negative token advantages with stop-gradient teacher-student KL avoids the out-of-distribution collapse of SDPO.
Critical weaknesses
  • The ablations do not show how much of VSD's improvement over GRPO on Qwen3-4B comes from scaling down negative advantages by 0.75 compared to token-level teacher reweighting. Section 4.3 shows that even late in training, roughly 80% of covered tokens on Qwen3-4B stay clamped at the 0.75 floor, and the teacher-free Uniform 0.75 control in Section 4.2 is run only on Qwen3-1.7B.
  • The paper compares VSD against GRPO only at equal training steps without showing wall-clock time or FLOPs, even though each VSD step adds writer generation and a teacher forward pass for every covered rollout.
Questions for authors
  • How does the teacher-free Uniform 0.75 control perform on Qwen3-4B across steps 100 to 250?
  • What are the wall-clock time and total FLOPs for VSD and GRPO per training step, and does VSD outperform GRPO when GRPO is given an equal compute budget?
2 comments

Dear Reviewer wt3f,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Thank you for the additional experiments.

W1: Largely addressed. Based on the results, it looks like Uniform 0.75 performance for pass@1 is still increasing even at step 250. I'm curious to see if scores plateau as we train for more steps. Also, do you have any idea why performance starts to drop for VSD after 200 steps? I think it'll be good to run training for multiple seeds. W2: Largely addressed.

The new control shows that the teacher reweighing has little gains over Uniform 0.75 at step 250 and it raises new concerns. I'll be maintaining my rating at 2.

Reviewer_eUJnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes verbalized self-distillation (VSD), a method for shaping negative token advantages in RLVR. A frozen writer compares a failed rollout with a successful sibling and produces instructions describing the first genuine error. A frozen teacher conditions on these instructions and scores the failed trajectory. A bounded, detached approximation to the forward teacher–student KL divergence then rescales the failed rollout's token advantages. Experiments on two Qwen3 model sizes report improvements over a common GRPO-style baseline on four mathematical reasoning benchmarks.

Critical strengths
  • Simple and intuitive approach. The method provides a practical way to introduce fine-grained learning signals into outcome-based RL without training a separate process reward model.

  • Reasonable experimental comparisons. The paper compares against GRPO, SDPO, RLRT and several ablations.

  • Promising empirical results. VSD improves mean pass@1 over GRPO by 3.59 percentage points on Qwen3-4B and 1.61 points on Qwen3-1.7B at step 250.

Critical weaknesses

The main concern is training efficiency and generalizability. VSD introduces additional writer generation and teacher scoring, but the performance gains over GRPO are relatively modest. It is unclear whether these improvements justify the additional computational cost.

Questions for authors

refer to weakness

2 comments

Dear Reviewer eUJn,

Thank you again for your review. As the discussion period closes on November 18, we wanted to follow up on our earlier comments. In our first note, we pointed out that parts of the Weaknesses section do not appear to correspond to our submission.

We would be grateful if you could confirm whether those comments were intended for our paper, and we would be glad to answer any remaining questions before the discussion period ends.

Dear Reviewer eUJn,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.