acceptodds
Under review as a conference paper at ICLR 2027

Self-Distillation with Verbal Regularization

Abstract

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher’s next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model–task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy self-distillation, Large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_fMGPOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose a way to improve the shortcutting/overconfidence that often appears with on-policy self-distillation, where access to privileged information in the teacher-copy of the model leads to shortcutted responses (directly to answer rather than reasoning), which the student then learns to mimic via the distillation loss. Authors propose avoiding this by conditioning the teacher-copy on additional context in the prompt that directs it not to shortcut. In a certain sense, this seems to be a prompting paper, through the lens of OPSD. Authors achieve an improvement of 4 points on average on various benchmarks.

Critical strengths

Authors correctly identify an existing issue with OPSD, supported by entropy-based analysis that seems solid. There is existing work that does this already, however [1].

Authors support that their method increases entropy in the student with measurements, giving some evidence that their method partially mitigates the problem they identified.

[1] Nicolicioiu, A. L., Pezeshki, M., & Courville, A. “On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity.” 2026.

Critical weaknesses

I believe the effect of the prompts is somewhat entangled in the way the experiments are presented. It seems to me reasonably possible that using these prompts for the student directly would already improve performance on the task, and I am surprised that this is not provided as a natural baseline. The entire work to me seems almost like context distillation but through the OPSD process? Why not just do context distillation directly on the student? Or, even easier, simply test the student conditioned under these prompts?

I am not convinced that the student model is actually learning more diverse reasoning paths. It seems plausible to me that the student is learning to increase probability for hedging tokens, like the ones given in the paper as examples, without necessarily producing more diverse reasoning chains. This could increase surface-level entropy without doing much else, this would effectively be stylistic distillation. Authors providing pass@1 and maj@16 also confuses me. Why not pass@1 and pass@k as well? Maj@16 could also be masking collapse. Seeing both of these in a graph over k would be quite helpful.

I am a bit confused as to how the multi-prompt thing works. Is the model not just being pulled in the direction of multiple different styles simultaneously, in a way that could increase its entropy but not in a way that is reasonable? The model does not see which style it should assume, so is it not just being pulled towards an average of all styles effectively? How do the authors understand what is happening here mechanistically?

Questions for authors

The gains seem to decrease with scale? 4B vs 7B vs 8B. Could this potentially limit the applicability?

4 comments

We thank the reviewer for the constructive feedback. We address the raised concerns as follows.

W1: "Using these prompts for the student directly would already improve performance … Why not just do context distillation directly on the student?"

We now report both baselines. First, prompting the student directly does not help (new App. H.1): the full prompt changes Pass@1 by −2.2 on average over 12 settings, and the best single prompt per setting adds 2.3, whereas VRSD is 15.4–18.8 points above the prompted student. Second, our existing no-PI control is exactly on-policy context distillation of the prompts (App. H.2): it uses the same loss, rollouts and mask as VRSD, and only removes cc from the teacher. It gains only 1–3 points over the untrained model and is 10–12 points below OPSD on three of four tasks. The prompts therefore act on how the teacher uses cc, not as a source of task knowledge.

W2: "I am not convinced that the student model is actually learning more diverse reasoning paths. … Why not pass@1 and pass@k as well?"

We added three analyses in the new App. M (Qwen3-4B).

Coverage. We sample 64 responses per problem (App. M, Table C). On all four tasks, VRSD's pass@k is above OPSD's at every kk, i.e., VRSD solves more problems across samples. On grading, chemistry and SQL, OPSD's coverage even falls below the untrained model at large kk, while VRSD stays above it.

Table C: Qwen3-4B, pass@64

GradingChemistrySQLPython
Untrained84.889.883.544.3
OPSD82.085.078.545.8
VRSD (single)86.891.587.049.6
VRSD (multi)88.490.490.049.6

Not hedging. Excluding hedging words and their neighboring tokens changes the mean student entropy by at most 0.003 nats, and suppressing these words at inference with a logit bias changes Pass@1 by at most 0.7 points.

Semantic diversity. Following Nicolicioiu et al. (2026), an LLM judge counts the unique concepts per problem across 16 responses. VRSD (multi) has more than OPSD on grading, chemistry and SQL (6.5 / 6.8 / 7.9 vs. 4.5 / 3.2 / 3.9), and a second, stronger judge agrees per problem (Spearman 0.69–0.89). On Python the order is reversed (9.9 vs. 11.9), but the measure is less informative there, since the responses are mostly code: a second judge (Claude Opus 5.5) and the first agree on which method uses more concepts for only half of the problems. OPSD also does not collapse on Python (student entropy 0.32, the highest of the four tasks).

We originally reported Maj@16 as a self-consistency measure next to Pass@1 and Worst@16. For the trained Qwen3-4B models it stays within about one point of Pass@1, so it does not hide a collapse; we now add pass@k and maj@k curves for every task (App. M).

W3: "Is it not just being pulled towards an average of all styles effectively? How do the authors understand what is happening here mechanistically?"

Each rollout is distilled toward one prompted teacher, so the student is fitted to a mixture of KK teachers rather than to a blend of instructions (new paragraph in Sec. 5.4). The outputs agree with this. Within one response, the prompt-specific patterns co-occur at the rate expected if they were independent (ratio 0.97–1.01 on the text tasks), so responses are not averaged styles. Across the 16 samples of a problem, VRSD (multi) varies more in which routines it follows than VRSD (single): Jaccard distance 0.15 / 0.27 / 0.19 vs. 0.04 / 0.23 / 0.13 on grading / chemistry / SQL.

Q1: "The gains seem to decrease with scale? … Could this potentially limit the applicability?"

The mean gain over OPSD is +7.2 (Qwen3-4B), +2.8 (OLMo-3-7B) and +1.6 (Qwen3-8B). The three models differ in family as well as size, so we cannot attribute this trend to scale alone. What the gain does track is how much OPSD itself collapses: the largest gain, Qwen3-4B SQL, is where OPSD falls below the untrained model, and Qwen3-8B is the model on which OPSD is strongest. Even there, VRSD (multi) beats OPSD on 3 of 4 tasks.

Related work: Nicolicioiu et al. (2026).

We now cite this work in Related Work. It shows that OPSD reduces output diversity; VRSD addresses this collapse with teacher-side prompts, and we use its unique-concepts measure to evaluate VRSD (W2 above).

Again, we thank the reviewer for the constructive feedback. We are happy to discuss further should you have any additional questions.

Thank you for the follow-up, and for noting that some of your original concerns have been addressed. We respond to your two remaining points, and add a result on your earlier question about scale.

"Task-specific prompts in combination with the larger teacher model." To clarify one point: the teacher in OPSD is not a larger model. It is an EMA copy of the student itself (Sec. 3), which additionally sees the verified solution cc and the prompt. Read as "task-specific prompts combined with a teacher that sees cc", your point is our claim: the prompts help by changing how such a teacher uses cc.
Task-specific content is not what makes this work. The prompt dd for chemistry and Python contains only reflection instructions; with it, VRSD (single) improves over OPSD on Python (+2.6, p=0.002p=0.002, Qwen3-4B, App. K), and the same prompt applied unchanged to SQL gains 9.2 points (App. G.1). Grading is the task where the rubric procedure is needed. The prompts are not sufficient on their own either: giving them to the student does not help (App. H.1), and giving them to a teacher that does not see cc gains only 1–3 points (App. H.2). As for the prompts, all candidates are in the anonymous repository (see our reply to Reviewer iZnR).

Relation to the entropy claims. We measured the attenuation coefficient γ^\widehat\gamma of App. A for each part of the prompt (Qwen3-4B; lower means the privileged information is attenuated more):

Teacher promptGradingSQL
Full dd0.690.68
Procedure only0.680.70
Reflection only0.890.86

The task procedure is not an alternative to our mechanism: it attenuates the privileged information as much as the full prompt and more than the reflection instructions, and it is also the part that yields the larger gain when used alone (App. G.1).
In our framework (Sec. 4), attenuation is the mechanism through which the prompts act, and entropy preservation is one of its effects: it holds for the teacher, and carries over to the student where OPSD collapses most. On SQL, each of the three teacher prompts of App. G.1 raises the student's entropy from 0.03 to 0.25–0.49, and for VRSD coverage and semantic diversity increase as well (App. M).

Scale. You asked whether the gains decrease with model size. To test this beyond 8B, we trained Qwen3-14B on grading with the Qwen3-8B prompts unchanged. OPSD never exceeds the untrained model (63.1; its best checkpoint reaches 59.7), while VRSD (multi) reaches 78.8. On grading, the gain of VRSD (multi) over OPSD is thus +11.1 (Qwen3-4B), +8.0 (OLMo-3-7B) and +5.4 (Qwen3-8B) in Table 1, and +19.1 on Qwen3-14B (+15.7 over the untrained model). The gain does not vanish at larger scale; on grading it is largest on the largest model.

We hope this clarifies the remaining points, and we are happy to discuss further.

I thank the authors for their response, and have raised my score accordingly.

Thank you for reconsidering your assessment and for the constructive discussion.

Reviewer_HAyDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is trying to solve an limitation of OPSD, where the teacher with privileged information tends to suppress the probabilities of certain tokens related to reconsideration, such as "Wait", 'but, "However". This can reduce reflective reasoning patterns in the student and lead to lower token entropy. To address this issue, the paper proposes a very simple method based on prompting. By adding reflective prompts to the teacher, the method encourages the teacher to preserve thinking behaviors, such as verification, reconsideration, and exploration of alternative reasoning paths, depending on the task. The method shows good improvements on four tasks across three models. The paper also includes several ablation studies, including comparisons between single prompts, multiple prompts, and prompt-only distillation without privileged information.

Critical strengths
  1. The paper tries to address an important problem in OPSD, where the teacher's reasoning can collapse due to privileged information and suppress the student's reflective thinking. The proposed solution is simple to implement. Although the prompts might be customized per task and need human judgement (as already stated in the limitation), this work might be providing beneficial analysis for future works that can be automatically searching for these prompts in a RSI loop.

  2. The paper includes relatively comprehensive ablation studies on different prompt designs, including single prompts, multiple prompts, and concatenating prompts. The results show that prompt-only distillation can indeed bring some improvements, although they are relatively small.

  3. Regardless of test-time evaluation accuracy, the prompt is also interesting and useful for encouraging reasoning patterns desired in rollout, and table 2 does show that VRSD has improved the reasoning pattern occurances.

Critical weaknesses
  1. one concern i have is that the paper does not clearly specify whether Qwen3 uses thinking mode or non-thinking mode during the evaluation, or the maximum response length used. This is important because Qwen3 already supports thinking mode, and the proposed teacher prompts seem to encourage behaviors that are closely related to thinking mode. Therefore, it is unclear to what extent the improvements come from the proposed regularization or from activating the model's existing thinking behaviors. The prompt-only ablation shows only small improvements, which somewhat reduces this concern, but I still think these implementation details should be clearly reported.
Questions for authors

please see weakness.

did you do any token or point-wise clipping as done in OPSD paper? which is also trying clipping the outlier token such as the large difference between these tokens such as 'however', 'wait'. If not, i think a comparison with this as baseline would be interesting.

2 comments

We thank the reviewer for the positive assessment and the constructive feedback. We address the raised concerns as follows.

W1: "The paper does not clearly specify whether Qwen3 uses thinking mode or non-thinking mode during the evaluation, or the maximum response length used."

All Qwen3 runs use non-thinking mode in both training and evaluation; OLMo-3-7B-Instruct has no thinking mode. The maximum response length is 2,048 tokens on the text tasks and 8,192 on Python, and evaluation uses the training length (Table 6). We now state both in Sec. 5.1. The gains therefore do not come from switching on Qwen3's thinking mode. As the reviewer notes, training with the prompts but without privileged information gives only small gains (App. H.2), which further supports this.

Q1: "Did you do any token or point-wise clipping as done in OPSD paper? … a comparison with this as baseline would be interesting."

No, neither OPSD nor VRSD in the paper uses clipping. Following the suggestion, we added the point-wise clipping of the OPSD paper with its default threshold of 0.05 on Qwen3-4B (new App. L). OPSD + clip reaches 57.3 / 71.5 on grading / SQL (71.9 / 69.8 without it), and VRSD (multi) + clip reaches 57.5 / 75.9 (83.0 / 81.0 without it). At this threshold, clipping removes 45–73% of the divergence mass but does not prevent the collapse: student entropy falls below 0.02 in every clipped run, whereas VRSD (multi) without clipping keeps it at 0.10 (grading) and 0.54 (SQL).

Again, we thank the reviewer for the constructive feedback. We are happy to discuss further should you have any additional questions.

Dear reviewer HAyD,

Thank you again for your positive assessment. In our reply, we clarified that all Qwen3 runs use non-thinking mode and stated the maximum response lengths (Sec. 5.1), and we added the point-wise clipping baseline you suggested (App. L). We also posted a summary comment on the role of the teacher prompts.

If you have any further concerns, we would be glad to address them in the remaining time.

Reviewer_ds6TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper addresses the problem of shortcut reasoning and entropy collapse in on-policy self-distillation (OPSD), which arise when the teacher has access to privileged information (PI). To mitigate this issue, the papers introduce various system prompts that encourage the teacher to engage in verification, reconsideration, and exploration of alternative solutions. By using the resulting conditioned teacher distributions, the proposed method preserves entropy and ultimately improves pass@1 performance across various models and tasks. Furthermore, the authors demonstrate that the trained student's entropy remains higher than that of a student trained with standard OPSD, supporting their claim that teacher-only prompts can influence the student's learned distribution.

Critical strengths

By simply modifying the teacher prefix, the proposed method achieves substantial performance improvements, particularly on Qwen3-4B SQL and Qwen3-8B Grading.

Critical weaknesses
  1. (Missing implementation detail) According to the implementation of the original OPSD paper [1], the teacher appears to be fixed as the initial model. However, the formulation in the VRSD seems to suggest a shared-parameter OPSD setup (Section 3). Since the training dynamics can differ depending on whether the teacher uses the current policy or a fixed policy, the authors should clearly clarify which setup is used.

  2. (Missing baseline) The authors explicitly state that the system prompts were manually designed (line 479). After examining Appendix I, I found that the prompts for each task incorporate fairly detailed human prior knowledge, seemingly reflecting an understanding of task-specific error patterns. To make the paper's explanation that PMI attenuation and entropy preservation are the primary mechanisms behind the performance improvements more convincing, the authors should evaluate the initial inference performance of the student when given these carefully designed prompts. This is important because the current experimental setup also allows for an alternative explanation: task-specific prompts -> better reasoning heuristics -> improved accuracy, rather than the improvements necessarily arising from PMI attenuation and entropy preservation.

[1] https://arxiv.org/abs/2601.18734

Questions for authors
  1. What is the inference performance of the initial, untrained student when using the curated system prompts?
  2. I would also like to see the performance of general (reflection?) system prompts on the Grading and SQL tasks.
4 comments

We thank the reviewer for the constructive feedback. We address the raised concerns as follows, grouping W2 with Q1.

W1: "The authors should clearly clarify which setup is used."

Both OPSD and VRSD use an EMA teacher, following the SDPO implementation. It is initialized to the initial student and updated after every step as θˉ←(1−β)θˉ+βθ\bar\theta\leftarrow(1-\beta)\bar\theta+\beta\theta with β=0.01\beta=0.01. We corrected the formulation in Sec. 3 accordingly. The OPSD baseline and VRSD share this teacher and differ only in the teacher prompt, so the comparison is controlled.

W2 & Q1: "What is the inference performance of the initial, untrained student when using the curated system prompts?"

We added this baseline in the new App. H.1 (3 models × 4 tasks). Giving the teacher prompts to the untrained student at inference barely helps: the full prompt changes Pass@1 by −2.2 on average and is below no prompt in 6 of 12 settings, and even the best single prompt per setting, picked on the test set, adds only 2.3. VRSD (single) is 18.8 points above the untrained student prompted with the same prompt. Training with the prompts but without privileged information also gains only 1–3 points (App. H.2). The improvement therefore does not come from reasoning heuristics that the prompts would give the student directly.

Q2: "I would also like to see the performance of general (reflection?) system prompts on the Grading and SQL tasks."

We added this in the new App. G.1 (Qwen3-4B). Using the reflection-only prompt of chemistry and Python unchanged, VRSD reaches 72.0 on grading and 79.0 on SQL, against 71.9 / 69.8 for OPSD and 78.7 / 80.0 for VRSD with the full prompt. It thus recovers most of the gain on SQL, but not on grading, where the rubric-specific procedure matters: the procedure alone reaches 76.8.

Again, we thank the reviewer for the constructive feedback. We are happy to discuss further should you have any additional questions.

Thank you for your follow-up question.

"An additional hyperparameter rather than a new insight." The insight of the paper does not depend on a prompt: privileged information makes the OPSD teacher over-commit, and attenuating its influence on the teacher counteracts this. App. L tests this without any prompt: interpolating the teacher with and without cc in log space, which is the attenuation of Sec. 4 with a constant γ\gamma, recovers 5.1 of the 6.8 points that VRSD (single) gains on grading and 8.1 of 10.2 on SQL (Qwen3-4B). A prompt is one way to realize this attenuation, changing neither the loss nor the model.
The prompts also do not act as task knowledge: given to the student, they change Pass@1 by −2.2 on average over 12 settings (App. H.1), and used for training without cc they gain only 1–3 points (Qwen3-4B, App. H.2). Nor does the prompt need re-tuning across models: the same prompts are used unchanged on OLMo-3-7B, Qwen3-8B and Qwen3-14B.

"An automatic way to identify effective distillation prompts." First, a gain does not require a task-specific prompt: on Qwen3-4B, the reflection-only prompt you asked about in Q2, which contains no task knowledge, gains 9.2 points on SQL (App. G.1) and 2.6 on Python (p=0.002p=0.002, App. K); grading is the one task where the rubric procedure is needed. Second, the two aspect prompts that hurt when used alone are identifiable from inference alone: with them, the teacher's continuation accuracy with cc in context drops from 95–96% to 74–76% (App. G). Third, the prompts transfer to a larger model as they are: a new Qwen3-14B run on grading reaches 78.8 with VRSD (multi), while OPSD never exceeds the untrained model (63.1). Automating prompt selection is the natural next step and is stated as future work (Sec. 6); the results above do not rely on it.

We are happy to discuss further.

Thank you for your clarification.

I think I had some misunderstanding about contribution of this paper.

Accordingly, I'll raised my score.

Thank you for reconsidering your assessment and for the constructive discussion.

Reviewer_T1m1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VRSD. Building on OPSD without changing the architecture, objective, or PI, it only adds teacher-only system prompts on the teacher side (requiring the model to verify intermediate steps, consider alternative solutions, and re-examine before answering). It models prompting as an affine attenuation of the PI-induced log-probability shift, and derives the exact conditions for entropy increase (Thm. 4.1, Cor. 4.2). Across three models and four tasks, VRSD improves Pass@1 in 11 out of 12 model-task settings, with an average gain of +3.9. It generally maintains higher token entropy, and through analysis and causal interventions, it is shown to mitigate the over-concentration of the output distribution and improve the faithfulness of reasoning paths.

Critical strengths
  1. That's a very interesting idea—simple and effective. In self-distillation, by designing system prompts for the teacher model to preserve the entropy of the teacher model's distribution given a PI, the student model is able to learn better during distillation.
  2. Theoretically, VRSD models the teacher-only system prompt as an affine attenuation of the log-probability shift induced by PI, derives the exact condition for entropy increase (Thm. 4.1), and shows that under mild conditions, prompting can mitigate the entropy collapse of the teacher distribution (Cor. 4.2). This mechanism corresponds precisely to guiding instructions in the prompt such as "verify intermediate steps, consider alternatives, and re-examine"—they weaken PI's tendency to concentrate probability mass on shortcut tokens.
  3. The effectiveness of the method is consistent across multiple models and tasks, and still holds when extended from single-prompt to multi-prompt.
Critical weaknesses
  1. The paper mentioned that the proposed system prompt helps preserve the teacher's entropy under mild conditions. However, the paper does not provide an upper bound for this entropy. This may affect the completeness of the retained entropy theory.
  2. 4.1 provides the gradient formula where At(u)A_t(u) is strictly increasing with respect to uu. When replaced with qtdq^d_t, the loss becomes Ja(sg[qtd]∥pt)Ja(sg[q^d_t]∥p_t). However, the excerpt fails to clarify what the gradient looks like after this replacement, nor does it clearly explain how this substituted gradient relates to the subsequent discussion.
  3. The conclusion is based on 3 models × 4 tasks = 12 settings. 11 out of 12 exceed OPSD, with an average of +3.9 percentage points. However, the excerpt does not provide variance, multiple seeds, or a significance test.
  4. The word "generally" in "generally maintaining higher token entropy" implies the existence of counterexamples. The authors need to specify which setting did not improve and what the reason is.
Questions for authors
  1. Can the authors provide an upper bound for the teacher model's entropy or for the entropy difference H(qtd)−H(qt)H(q^d_t) − H(q_t), as well as its relationship with the student model's entropy? Will it be greater than the student model's entropy? Or will it only approach it indefinitely?
  2. After substituting qtdq^d_t, does the weight in the gradient become At(utd)A_t(u^d_t)? Is affine attenuation equivalent to performing gradient reweighting? Furthermore, a closed-form expression after the substitution is needed to substantiate the claim that "attenuation alters the shape of the gradient."
  3. Which is the only model–task setting that did not exceed OPSD? Is the +3.9 average being driven by a few individual tasks? Are the standard deviations across multiple runs and a paired test reported?
3 comments
AuthorsOpenReview ↗

Reply (1/2)

We thank the reviewer for the positive assessment and the constructive feedback. We address the raised concerns as follows, grouping each weakness with its question. This comment covers the theory; the next covers the experiments.

W1 & Q1: "Can the authors provide an upper bound for the teacher model's entropy … as well as its relationship with the student model's entropy?"

We added these results in the new App. B.5. Let qλ∝peλuq_\lambda\propto p e^{\lambda u} be the attenuation path, where pp is the student at a fixed prefix, so that q1=qq_1=q, qγ=qdq_\gamma=q^d and q0=pq_0=p. In training, the distribution without cc comes from the EMA teacher (Sec. 3), which lags the student slightly, so q0q_0 is close to, but not exactly, the current student.

Upper bound. By Thm. 4.1 and Cauchy–Schwarz, H(qd)−H(q)=∫γ1C(λ)dλ≤(1−γ)sup⁡γ≤λ≤1S(λ),H(q^d)-H(q)=\int_\gamma^1 C(\lambda) d\lambda \le (1-\gamma)\sup_{\gamma\le\lambda\le1} S(\lambda), where C(λ)C(\lambda) is the covariance of uu and log⁡qλ\log q_\lambda under qλq_\lambda, and S(λ)S(\lambda) is the product of their standard deviations. The entropy increase is therefore at most linear in the attenuation strength 1−γ1-\gamma. For γ\gamma close to 1, its leading term is (1−γ)C(1)(1-\gamma)C(1), the covariance under qq that we measure in App. C. The trivial bound H(qd)≤log⁡∣V∣H(q^d)\le\log|\mathcal V| is also tight: App. B.5 gives a two-token example that attains it.

Relation to the student. The same example shows that, in general, H(qd)H(q^d) can exceed both H(q)H(q) and H(p)H(p). However, if the covariance is non-negative along the whole path, i.e., when the privileged information reinforces tokens the student already prefers, then H(q)≤H(qd)≤H(p),H(q) \le H(q^d) \le H(p), with H(qd)→H(p)H(q^d)\to H(p) as γ→0\gamma\to0. To answer the question directly: under this condition, the teacher's entropy approaches the student's entropy but does not exceed it. App. C supports the condition at λ=1\lambda=1 at prefixes with large PMI. Empirically, the logged teacher entropy stays below the student's at 94–100% of training steps (Qwen3-4B, grading and SQL), which agrees with this ordering, although it does not test the path condition directly.

W2 & Q2: "Is affine attenuation equivalent to performing gradient reweighting? … a closed-form expression after the substitution is needed."

We added the closed form in the new App. B.2, with a pointer in Sec. 4. Substituting ud=γu+κu^d=\gamma u+\kappa gives −∇θJα(sg[qd]∥p)=Ev∼p[A(γu(v)+κ)∇θlog⁡p(v)],κ=−log⁡Ep[eγu]≥0,-\nabla_\theta\mathcal J_\alpha(\mathrm{sg}[q^d] \Vert p)=\mathbb E_{v\sim p}\big[A(\gamma u(v)+\kappa) \nabla_\theta\log p(v)\big],\qquad \kappa=-\log\mathbb E_p\big[e^{\gamma u}\big]\ge0, and per logit, −∂Jα/∂zv=p(v)[A(γu(v)+κ)−Ew∼pA(γu(w)+κ)]-\partial\mathcal J_\alpha/\partial z_v=p(v)\big[A(\gamma u(v)+\kappa)-\mathbb E_{w\sim p}A(\gamma u(w)+\kappa)\big]. For γ∈(0,1)\gamma\in(0,1), the coefficient keeps the ranking of tokens, but its spread strictly shrinks: A(γumax⁡+κ)−A(γumin⁡+κ)<A(umax⁡)−A(umin⁡)A(\gamma u_{\max}+\kappa)-A(\gamma u_{\min}+\kappa)<A(u_{\max})-A(u_{\min}).

Whether this is a reweighting depends on the divergence. For the reverse KL used on Python (α=1\alpha=1, A(u)=uA(u)=u), it is: κ\kappa drops out, and the gradient at each prefix is exactly rescaled by γ\gamma. For the JSD used on the text tasks, it is not. The local factor dA(γu+κ)dA(u)=γα+(1−α)e−uα+(1−α)e−(γu+κ)\frac{dA(\gamma u+\kappa)}{dA(u)}=\gamma \frac{\alpha+(1-\alpha)e^{-u}}{\alpha+(1-\alpha)e^{-(\gamma u+\kappa)}} depends on uu: it tends to γ\gamma for tokens that the privileged information favors and diverges for tokens it disfavors. Attenuation therefore changes the relative weights of tokens, which is what we mean by changing the shape of the gradient.

AuthorsOpenReview ↗

Reply (2/2)

W3 & Q3: "The excerpt does not provide variance, multiple seeds, or a significance test. … Is the +3.9 average being driven by a few individual tasks?"

We now report all 12 settings over three training seeds (Table B). Averaged over seeds, VRSD (multi) outperforms OPSD in 11 of 12 settings, by +3.0 on average. The gains on grading (all three models) and on Qwen3-4B SQL and Python exceed the run-to-run variation.

Table B: Pass@1, mean ± std over three training seeds

ModelMethodGradingChemistrySQLPython
Qwen3-4BOPSD73.1 ± 2.482.3 ± 0.671.1 ± 1.261.3 ± 0.3
VRSD (single)78.8 ± 0.784.6 ± 1.377.0 ± 4.664.0 ± 0.9
VRSD (multi)81.3 ± 3.284.4 ± 1.479.6 ± 1.463.6 ± 0.7
OLMo-3-7BOPSD73.8 ± 4.482.8 ± 1.174.3 ± 1.357.8 ± 0.4
VRSD (single)85.2 ± 1.183.7 ± 0.272.4 ± 1.657.6 ± 1.0
VRSD (multi)83.0 ± 2.083.2 ± 0.774.9 ± 0.658.0 ± 0.5
Qwen3-8BOPSD75.7 ± 0.183.8 ± 0.278.8 ± 0.465.4 ± 0.3
VRSD (single)82.7 ± 4.783.4 ± 0.779.7 ± 0.664.5 ± 0.8
VRSD (multi)79.0 ± 1.983.1 ± 0.979.8 ± 0.965.8 ± 0.9

We also added paired per-problem tests on Qwen3-4B (new App. K). VRSD (multi) improves over OPSD by +6.7 on grading, +7.2 on SQL and +2.3 on Python (p<0.01p<0.01 for all three); on chemistry, where the gain is about one point, the difference is not significant.

Regarding the +3.9 average in Table 1, the mean gain per task is +8.2 (grading), +2.1 (chemistry), +4.2 (SQL) and +0.8 (Python). Without the two largest settings (Qwen3-4B SQL and grading), the mean gain is +2.4, and VRSD (multi) still leads in 9 of the remaining 10 settings. The gain is largest where OPSD collapses most (see W4).

W4 & Q3: "Which is the only model–task setting that did not exceed OPSD?" and "the existence of counterexamples" to higher entropy.

In Table 1, VRSD (multi) is below OPSD only on Qwen3-8B Python (65.4 vs. 65.6). Averaged over seeds (Table B), it is above OPSD there (65.8 vs. 65.4), and the one setting where it falls behind is instead Qwen3-8B chemistry (83.1 vs. 83.8). That the losing setting changes with the seeds suggests these small differences are within run-to-run variation. VRSD (single) is below OPSD on OLMo SQL and Python and on Qwen3-8B chemistry and Python. Student entropy (App. E) is below OPSD in 3 of 24 cells (Qwen3-4B grading single, Qwen3-4B Python multi, OLMo Python single) and equal in 2.

Both exceptions have a clear reason. On Python, OPSD does not collapse: on Qwen3-4B and Qwen3-8B, its student entropy is the highest of the four tasks, so VRSD has little to restore, and its gains are smallest there. On Qwen3-4B grading, VRSD gains 6.8 (single) and 11.1 (multi) points without higher student entropy; as App. G.1 shows, the task procedure can improve the teacher's targets without raising entropy. Entropy is thus one route to the gain, not the only one.

Again, we thank the reviewer for the constructive feedback. We are happy to discuss further should you have any additional questions.

Thank you for confirming that your previous concerns are addressed. On your remaining question, the paper separates the prompt from the fine-tuning as follows:

  • Prompt without fine-tuning. Given to the untrained student at inference, the prompts change Pass@1 by −2.2 on average over 12 settings (App. H.1).
  • Fine-tuning without the prompt. This is the OPSD baseline (Table 1).
  • Prompt and fine-tuning, without privileged information. This gains only 1–3 points over the untrained model and stays 10–12 points below OPSD on three of four tasks (Qwen3-4B, App. H.2).
  • Prompt and fine-tuning, with privileged information (VRSD). This is +3.9 over OPSD on average (Table 1), and 15.4–18.8 points above the prompted student (App. H.1).

The gain therefore comes from neither factor alone. As you put it in your review, the prompts "weaken PI's tendency to concentrate probability mass on shortcut tokens": this effect exists only in a teacher that sees the verified solution, and it reaches the student only through distillation. More training does not close the gap either: taking each run's last checkpoint, the gain of VRSD (multi) over OPSD grows from +6.7 to +10.0 on grading and from +7.2 to +10.2 on SQL (Qwen3-4B; see our reply to Reviewer iZnR), because OPSD degrades after its peak. Thank you for your suggestion, we will state this decomposition explicitly in the final version.

Reviewer_iZnROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes self-distillation with verbal regularization (VRSD), which adds teacher prompts encouraging verification, reconsideration, and alternative reasoning to on-policy self-distillation. The teacher retains access to verified solutions, while the student receives neither the solutions nor the additional prompts. An affine attenuation model is used to characterize conditions for increased teacher entropy. Experiments across three models and four tasks report an average Pass@1 improvement over OPSD for the multiple-prompt variant.

Critical strengths
  • VRSD modifies teacher conditioning without introducing a new learned module or changing the core distillation objective. The additional prompts are absent at student inference time, making the approach straightforward to integrate into existing OPSD pipelines.
  • Comparisons between individual prompts, concatenated prompts, and prompts distributed across rollouts help characterize the design. The control without privileged solutions also helps distinguish the benefit of prompting alone from its interaction with privileged supervision.
Critical weaknesses
  • Performance varies substantially with prompt choice, and some individual prompts degrade results. The prompt construction and selection process is insufficiently documented, with some wording also changed across models. This makes it difficult to assess robustness to alternative formulations and the effort required to extend VRSD to new tasks.
  • The prompts contain detailed task procedures as well as instructions to reconsider answers, so improvements may reflect better procedural guidance or more diverse supervision rather than attenuation of privileged information specifically. Comparisons with teacher temperature adjustment, direct distribution interpolation, and prompts providing task guidance without reflection would help distinguish these explanations.
Questions for authors
  • How were the teacher prompts constructed and selected, and what data informed their revisions, including the changes across models? Please clarify the training, validation, and test partitions and sample counts for each task, particularly Spider, and whether test results influenced prompt design.
  • What evidence distinguishes attenuation of privileged information from the benefits of detailed task guidance or increased teacher entropy alone? Comparisons with teacher temperature adjustment or prompts providing the same task procedures without reflection would materially clarify the mechanism.
4 comments

We thank the reviewer for the detailed feedback. We address the raised concerns as follows, grouping W1 with Q1 and W2 with Q2.

W1 & Q1: "The prompt construction and selection process is insufficiently documented … whether test results influenced prompt design."

The prompts were written by hand. We screened them only on training problems, by the untrained teacher's accuracy and entropy with the verified solution cc in context. Student accuracy was never used on any split, so test results did not shape the prompts. Prompts we used are documented in the paper and appendix (App. I).

Regarding the data, App. D.1 now lists the source and split sizes of every task. For Spider, we train on 3,000 problems from its training split and test on 200 problems from its development split. There is no separate validation split, so every method, OPSD included, selects its checkpoint on the test set (App. D.1). To check whether this favors VRSD, we took each run's last checkpoint instead, using the additional Qwen3-4B seeds of App. K. The gain of VRSD (multi) over OPSD then grows from +6.7 to +10.0 on grading, from +1.1 to +1.7 on chemistry, and from +7.2 to +10.2 on SQL, mainly because OPSD degrades after its peak.

Regarding sensitivity to single prompts, the two prompts that hurt when used alone, chemistry d3d_3 and grading d4d_4, ask the teacher to argue against its own answer. With cc in context, the teacher then often talks itself out of the correct answer: its continuation accuracy drops from 95% to 74% on chemistry and from 96% to 76% on grading (App. G). In VRSD (multi), each prompt is used for only 1/K1/K of the rollouts, and VRSD (multi) outperforms every individual aspect prompt dkd_k on all four tasks.

Regarding the effort to extend VRSD, a task procedure is not required: the reflection-only prompt of chemistry and Python, applied unchanged to SQL, reaches 79.0 Pass@1, against 69.8 for OPSD (App. G.1).

W2 & Q2: "What evidence distinguishes attenuation of privileged information from the benefits of detailed task guidance or increased teacher entropy alone?"

Task guidance. We split each prompt into its task procedure and its reflection instructions (new App. G.1). On Qwen3-4B, the procedure alone reaches 76.8 on grading and 82.5 on SQL, and the reflection instructions alone reach 72.0 and 79.0, against 71.9 / 69.8 for OPSD and 78.7 / 80.0 for the full prompt. The procedure helps, but only through a teacher that also sees cc. Given directly to the untrained student, the procedure alone lowers Pass@1 (48.2 vs. 57.2 on grading, 70.8 vs. 71.3 on SQL; App. G.1), and training with the prompts but without cc gains only 1–3 points (App. H.2).

Teacher entropy. Following the suggestion, we changed only the OPSD teacher, with no prompt: a higher temperature τ\tau, or an interpolation between the teacher with and without cc (new App. L, Table A). Temperature helps little on grading (at most +0.3), and for τ≥1.5\tau\ge1.5 the student's entropy diverges. Log-space interpolation, q(⋅∣x,c,y<t)λ q(⋅∣x,y<t)1−λq(\cdot\mid x,c,y_{<t})^{\lambda}\,q(\cdot\mid x,y_{<t})^{1-\lambda}, is the attenuation of Sec. 4 with a constant γ=λ\gamma=\lambda and no prompt, since the teacher without cc is the EMA copy of the student. On grading, it recovers 5.1 of the 6.8 points that VRSD (single) gains over OPSD, with the student's entropy staying near the OPSD level. We read this as evidence that attenuating the privileged information is one mechanism behind the gain, and that higher entropy by itself is not enough. SQL is less clear-cut: probability-space interpolation (79.0) does slightly better than log-space interpolation (77.9), and the procedure-only prompt (82.5) does better than the full prompt (80.0), so task guidance also contributes there. Among the teachers in Table A, VRSD (multi) is the best on both tasks.

Table A: Qwen3-4B, Pass@1 and student entropy H(p)H(p) at the selected checkpoint

OPSD teacherGradingH(p)H(p)SQLH(p)H(p)
Unchanged (OPSD)71.90.1069.80.03
Temperature τ=1.2\tau=1.272.20.2376.50.03
Temperature τ=1.5\tau=1.5 / 2.02.070.3 / 65.2diverges78.0 / 75.8diverges
Probability-space interpolation, λ=0.5\lambda=0.574.40.2979.00.20
Log-space interpolation, λ=0.5\lambda=0.577.00.1277.90.12
VRSD (single)78.70.0980.00.49
VRSD (multi)83.00.1081.00.54

Again, we thank the reviewer for the constructive feedback, which helped us sharpen the mechanism. We are happy to discuss further should you have any additional questions.

Thank you for the clarifications and additional experiments. These address some of my concerns, particularly regarding the data used for prompt screening and the contribution of alternative teacher modifications.

However, the method still relies substantially on manually constructed and selected prompts. The number of candidates, revision process, and associated selection costs remain insufficiently documented. The transfer of a generic reflection prompt to SQL is encouraging, but does not yet establish robust generalization to new tasks without further manual tuning.

The additional controls also suggest a more task-dependent explanation than the paper’s emphasis on mitigating PI-induced concentration and preserving entropy and exploration. In particular, procedure-only prompting reaches 82.5 on SQL, exceeding both VRSD variants. While the interpolation results support attenuation as a contributing mechanism, procedural guidance appears to explain a substantial part of the gains, and the evidence does not convincingly establish entropy preservation as the primary explanation.

Overall, the rebuttal resolves part of my concerns but does not fully address my reservations about the method’s generality and mechanistic interpretation. Nevertheless, in recognition of the useful additional evidence, I am willing to modestly increase my score.

Thank you for the follow-up and for reconsidering your score.

Prompt construction. All candidate prompts are now in the anonymous repository linked in the paper (prompts/candidates/). Two aspect prompts (grading d4d_4, chemistry d3d_3) were revised after the Qwen3-4B runs, because they led the teacher to argue itself out of the correct answer (App. G); the second versions are used for the other models (App. I). Checking a candidate prompt requires only inference with the untrained model, not a training run. Automating this selection is left to future work (Sec. 6).

The Qwen3-8B prompts transfer unchanged to a larger model: on Qwen3-14B (grading), OPSD never exceeds the untrained model (63.163.1), while VRSD (multi) reaches 78.878.8. On Qwen3-4B, VRSD (multi) also outperforms every individual aspect prompt dkd_k on all four tasks (App. G), so the outcome does not depend on picking the best of them.

Mechanism. Procedural guidance and attenuation are not competing explanations. Our model characterizes a prompt by its effect on the teacher distribution, not by its content, so we measured the attenuation coefficient γ^\widehat\gamma of App. A for each part of the prompt (Qwen3-4B; lower means stronger attenuation):

Teacher promptGradingSQL
Full dd0.690.68
Procedure only0.680.70
Reflection only0.890.86

The procedure alone attenuates the privileged information as much as the full prompt, and more than the reflection instructions; it is also the part that yields the larger gain when used alone (App. G.1). The procedure is one way of attenuating. Procedure-only prompting is therefore VRSD with a different teacher prompt, and its 82.5 on SQL is a result for the method: the difference from 80.0 / 81.0 is within the seed variation there (± 4.6 for VRSD (single) over three seeds). The procedure does not help as task knowledge: given to the untrained student it lowers Pass@1 (App. G.1), and training with the prompts but without cc gains only 1–3 points (App. H.2). In our framework (Sec. 4), attenuation is the mechanism through which the prompts act, and entropy preservation is one of its effects rather than a separate explanation.

We hope this addresses your remaining concerns. If anything is still unclear, please let us know.

Dear reviewer iZnR,

Thank you again for your follow-up and for raising your score. In our latest reply, we added the full set of candidate prompts to the anonymous repository, a new Qwen3-14B result using the same prompts without changes, and the attenuation coefficient measured for each part of the teacher prompt, which shows that the task procedure and attenuation are not competing explanations.

If you have any further concerns, we would be glad to address them in the remaining time.

Comments on the paper

2 comments

We thank all reviewers for their careful reading and constructive feedback. The revised PDF marks every change in color. The main additions are:

  • Mechanism. A closed-form gradient under attenuation (Sec. 4, App. B.2) and upper bounds on the teacher's entropy, with its relation to the student's (App. B.5) (T1m1). Prompt-free baselines that raise the OPSD teacher's entropy (new App. L): temperature helps little, while attenuating the teacher without a prompt recovers most of the gain on grading, which supports attenuation as one mechanism (iZnR, HAyD). An ablation of the task procedure versus the reflection instructions in the prompts (new App. G.1) (iZnR, ds6T).
  • Controls. The untrained student prompted with the teacher prompts (new App. H.1), and training with the prompts but without privileged information, i.e., on-policy context distillation (App. H.2). Both give little gain, so the prompts do not act by supplying task knowledge (ds6T, fMGP, HAyD). OPSD's point-wise clipping as a baseline (App. L) (HAyD).
  • Reliability and diversity. Paired per-problem significance tests on all four tasks (new App. K), and mean ± std over training seeds in our reply to T1m1 (T1m1). pass@k up to k=64k=64, a hedging analysis, and LLM-judged semantic diversity (new App. M) (fMGP).
  • Clarity. The EMA teacher (Sec. 3), non-thinking mode and response lengths (Sec. 5.1), data sources and splits (App. D.1), how multi-prompt training works (Sec. 5.4), and related work on diversity in OPSD (Sec. 2).
AuthorsOpenReview ↗

Summary of the discussion on the role of the teacher prompts

We thank the reviewers for the discussion. One question came up repeatedly in the discussion is whether the gains come from hand-crafted, task-specific prompts and not from the mechanism we propose. We summarize the evidence in one place, including two results added during the discussion.

1. The prompts do not act as task knowledge. Given to the student at inference, they change Pass@1 by −2.2 on average over 12 settings (App. H.1). Used for training without the privileged information cc, they gain only 1–3 points and stay 10–12 points below OPSD on three of four tasks (Qwen3-4B, App. H.2). The gain requires a teacher that sees cc. A task-specific prompt is not required either: on Qwen3-4B, a reflection-only prompt with no task knowledge gains 9.2 points on SQL (App. G.1) and 2.6 on Python (p=0.002p=0.002, App. K).

2. Attenuation works without any prompt. Interpolating the OPSD teacher with and without cc in log space is the attenuation of Sec. 4 with a constant γ\gamma. It recovers 5.1 of the 6.8 points that VRSD (single) gains on grading and 8.1 of 10.2 on SQL (Qwen3-4B, App. L).

3. Task procedure and attenuation are not competing explanations. We measured the attenuation coefficient γ^\widehat\gamma of App. A for each part of the prompt (new, Qwen3-4B; lower means stronger attenuation):

Teacher promptGradingSQL
Full dd0.690.68
Procedure only0.680.70
Reflection only0.890.86

The task procedure attenuates the privileged information as much as the full prompt and more than the reflection instructions, and it is also the part that gains more when used alone (App. G.1).

4. The prompts are not re-tuned across models. The same prompts are used unchanged on OLMo-3-7B, Qwen3-8B and Qwen3-14B. On Qwen3-14B, a new run on grading reaches 78.8 with VRSD (multi), while OPSD never exceeds the untrained model (63.1). On Qwen3-4B, VRSD (multi) also outperforms each of its aspect prompts in Pass@1 on all four tasks (App. G), so the outcome does not depend on picking the best one.

All candidate prompts are in the anonymous repository (prompts/candidates/), and the two aspect prompts that hurt when used alone are identifiable without training, by the teacher's accuracy with cc in context (App. G). The prompts are written by hand; the paper states this as a limitation and leaves automating their selection to future work (Sec. 6). The contribution of the paper is the diagnosis that privileged information makes the OPSD teacher over-commit, the attenuation model of how teacher conditioning counteracts it, and a teacher-only method that improves over OPSD in 11 of 12 settings; the controls above isolate the effect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.