- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose a way to improve the shortcutting/overconfidence that often appears with on-policy self-distillation, where access to privileged information in the teacher-copy of the model leads to shortcutted responses (directly to answer rather than reasoning), which the student then learns to mimic via the distillation loss. Authors propose avoiding this by conditioning the teacher-copy on additional context in the prompt that directs it not to shortcut. In a certain sense, this seems to be a prompting paper, through the lens of OPSD. Authors achieve an improvement of 4 points on average on various benchmarks.
Critical strengths
Authors correctly identify an existing issue with OPSD, supported by entropy-based analysis that seems solid. There is existing work that does this already, however [1].
Authors support that their method increases entropy in the student with measurements, giving some evidence that their method partially mitigates the problem they identified.
[1] Nicolicioiu, A. L., Pezeshki, M., & Courville, A. “On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity.” 2026.
Critical weaknesses
I believe the effect of the prompts is somewhat entangled in the way the experiments are presented. It seems to me reasonably possible that using these prompts for the student directly would already improve performance on the task, and I am surprised that this is not provided as a natural baseline. The entire work to me seems almost like context distillation but through the OPSD process? Why not just do context distillation directly on the student? Or, even easier, simply test the student conditioned under these prompts?
I am not convinced that the student model is actually learning more diverse reasoning paths. It seems plausible to me that the student is learning to increase probability for hedging tokens, like the ones given in the paper as examples, without necessarily producing more diverse reasoning chains. This could increase surface-level entropy without doing much else, this would effectively be stylistic distillation. Authors providing pass@1 and maj@16 also confuses me. Why not pass@1 and pass@k as well? Maj@16 could also be masking collapse. Seeing both of these in a graph over k would be quite helpful.
I am a bit confused as to how the multi-prompt thing works. Is the model not just being pulled in the direction of multiple different styles simultaneously, in a way that could increase its entropy but not in a way that is reasonable? The model does not see which style it should assume, so is it not just being pulled towards an average of all styles effectively? How do the authors understand what is happening here mechanistically?
Questions for authors
The gains seem to decrease with scale? 4B vs 7B vs 8B. Could this potentially limit the applicability?
4 comments
Thank you for the follow-up, and for noting that some of your original concerns have been addressed. We respond to your two remaining points, and add a result on your earlier question about scale.
"Task-specific prompts in combination with the larger teacher model." To clarify one point: the teacher in OPSD is not a larger model. It is an EMA copy of the student itself (Sec. 3), which additionally sees the verified solution and the prompt. Read as "task-specific prompts combined with a teacher that sees ", your point is our claim: the prompts help by changing how such a teacher uses .
Task-specific content is not what makes this work. The prompt for chemistry and Python contains only reflection instructions; with it, VRSD (single) improves over OPSD on Python (+2.6, , Qwen3-4B, App. K), and the same prompt applied unchanged to SQL gains 9.2 points (App. G.1). Grading is the task where the rubric procedure is needed. The prompts are not sufficient on their own either: giving them to the student does not help (App. H.1), and giving them to a teacher that does not see gains only 1–3 points (App. H.2). As for the prompts, all candidates are in the anonymous repository (see our reply to Reviewer iZnR).
Relation to the entropy claims. We measured the attenuation coefficient of App. A for each part of the prompt (Qwen3-4B; lower means the privileged information is attenuated more):
| Teacher prompt | Grading | SQL |
|---|---|---|
| Full | 0.69 | 0.68 |
| Procedure only | 0.68 | 0.70 |
| Reflection only | 0.89 | 0.86 |
The task procedure is not an alternative to our mechanism: it attenuates the privileged information as much as the full prompt and more than the reflection instructions, and it is also the part that yields the larger gain when used alone (App. G.1).
In our framework (Sec. 4), attenuation is the mechanism through which the prompts act, and entropy preservation is one of its effects: it holds for the teacher, and carries over to the student where OPSD collapses most. On SQL, each of the three teacher prompts of App. G.1 raises the student's entropy from 0.03 to 0.25–0.49, and for VRSD coverage and semantic diversity increase as well (App. M).
Scale. You asked whether the gains decrease with model size. To test this beyond 8B, we trained Qwen3-14B on grading with the Qwen3-8B prompts unchanged. OPSD never exceeds the untrained model (63.1; its best checkpoint reaches 59.7), while VRSD (multi) reaches 78.8. On grading, the gain of VRSD (multi) over OPSD is thus +11.1 (Qwen3-4B), +8.0 (OLMo-3-7B) and +5.4 (Qwen3-8B) in Table 1, and +19.1 on Qwen3-14B (+15.7 over the untrained model). The gain does not vanish at larger scale; on grading it is largest on the largest model.
We hope this clarifies the remaining points, and we are happy to discuss further.
I thank the authors for their response, and have raised my score accordingly.
Thank you for reconsidering your assessment and for the constructive discussion.
We thank the reviewer for the constructive feedback. We address the raised concerns as follows.
W1: "Using these prompts for the student directly would already improve performance … Why not just do context distillation directly on the student?"
We now report both baselines. First, prompting the student directly does not help (new App. H.1): the full prompt changes Pass@1 by −2.2 on average over 12 settings, and the best single prompt per setting adds 2.3, whereas VRSD is 15.4–18.8 points above the prompted student. Second, our existing no-PI control is exactly on-policy context distillation of the prompts (App. H.2): it uses the same loss, rollouts and mask as VRSD, and only removes c from the teacher. It gains only 1–3 points over the untrained model and is 10–12 points below OPSD on three of four tasks. The prompts therefore act on how the teacher uses c, not as a source of task knowledge.
W2: "I am not convinced that the student model is actually learning more diverse reasoning paths. … Why not pass@1 and pass@k as well?"
We added three analyses in the new App. M (Qwen3-4B).
Coverage. We sample 64 responses per problem (App. M, Table C). On all four tasks, VRSD's pass@k is above OPSD's at every k, i.e., VRSD solves more problems across samples. On grading, chemistry and SQL, OPSD's coverage even falls below the untrained model at large k, while VRSD stays above it.
Table C: Qwen3-4B, pass@64
Not hedging. Excluding hedging words and their neighboring tokens changes the mean student entropy by at most 0.003 nats, and suppressing these words at inference with a logit bias changes Pass@1 by at most 0.7 points.
Semantic diversity. Following Nicolicioiu et al. (2026), an LLM judge counts the unique concepts per problem across 16 responses. VRSD (multi) has more than OPSD on grading, chemistry and SQL (6.5 / 6.8 / 7.9 vs. 4.5 / 3.2 / 3.9), and a second, stronger judge agrees per problem (Spearman 0.69–0.89). On Python the order is reversed (9.9 vs. 11.9), but the measure is less informative there, since the responses are mostly code: a second judge (Claude Opus 5.5) and the first agree on which method uses more concepts for only half of the problems. OPSD also does not collapse on Python (student entropy 0.32, the highest of the four tasks).
We originally reported Maj@16 as a self-consistency measure next to Pass@1 and Worst@16. For the trained Qwen3-4B models it stays within about one point of Pass@1, so it does not hide a collapse; we now add pass@k and maj@k curves for every task (App. M).
W3: "Is it not just being pulled towards an average of all styles effectively? How do the authors understand what is happening here mechanistically?"
Each rollout is distilled toward one prompted teacher, so the student is fitted to a mixture of K teachers rather than to a blend of instructions (new paragraph in Sec. 5.4). The outputs agree with this. Within one response, the prompt-specific patterns co-occur at the rate expected if they were independent (ratio 0.97–1.01 on the text tasks), so responses are not averaged styles. Across the 16 samples of a problem, VRSD (multi) varies more in which routines it follows than VRSD (single): Jaccard distance 0.15 / 0.27 / 0.19 vs. 0.04 / 0.23 / 0.13 on grading / chemistry / SQL.
Q1: "The gains seem to decrease with scale? … Could this potentially limit the applicability?"
The mean gain over OPSD is +7.2 (Qwen3-4B), +2.8 (OLMo-3-7B) and +1.6 (Qwen3-8B). The three models differ in family as well as size, so we cannot attribute this trend to scale alone. What the gain does track is how much OPSD itself collapses: the largest gain, Qwen3-4B SQL, is where OPSD falls below the untrained model, and Qwen3-8B is the model on which OPSD is strongest. Even there, VRSD (multi) beats OPSD on 3 of 4 tasks.
Related work: Nicolicioiu et al. (2026).
We now cite this work in Related Work. It shows that OPSD reduces output diversity; VRSD addresses this collapse with teacher-side prompts, and we use its unique-concepts measure to evaluate VRSD (W2 above).
Again, we thank the reviewer for the constructive feedback. We are happy to discuss further should you have any additional questions.