acceptodds
Under review as a conference paper at ICLR 2027

Distilling Structure, Not Solutions: Structural Bottlenecks for On-Policy Self-Distillation

11 views

Abstract

On-policy self-distillation (OPSD) aligns supervision with student-generated trajectories but retains an information asymmetry: the teacher receives a verified solution that is unavailable to the student at inference. We investigate whether restricting this privileged context to reasoning structure improves distillation. **Structural-Bottleneck OPSD (SB-OPSD)** replaces full solutions with compact, result-free skeletons that preserve ordered reasoning operations while excluding intermediate results and final answers. A frozen, skeleton-conditioned teacher supplies next-token targets at student-generated prefixes, while the student uses a question-only prompt throughout training and inference. Across Qwen3-1.7B, 4B, and 8B on three competition-mathematics benchmarks, SB-OPSD achieves the highest mean **Average@12** among the compared methods, improving over the respective base models by **5.70, 3.31, and 3.22 percentage points**. Structural controls favor compact, ordered, question-matched skeletons. Relative to full-solution OPSD, SB-OPSD also reduces teacher-context and output lengths and exhibits a smaller gap to its conditioned teacher during training. These results support compact reasoning structure as an effective form of privileged supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Privileged Information, Mathematical Reasoning, Structural Reasoning, Large Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3zfaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at on-policy self-distillation (OPSD), where the frozen teacher is conditioned on the verified solution but the student only sees the question. The authors call this gap "privilege mismatch". Their fix is to replace the full solution in the teacher's context with a compact structural skeleton: an ordered sequence of tagged reasoning operations with intermediate and final results stripped out, enforced by a validator that rejects answer strings, result-bearing equations, and any numbers not already in the question. Teacher and student start from the same checkpoint and only the student's LoRA parameters are updated, using forward KL at student-generated prefixes. On Qwen3-1.7B/4B/8B over AIME24, AIME25 and HMMT25, SB-OPSD gets the best three-benchmark mean Average@12, +5.70/+3.31/+3.22 over base, ahead of full-solution OPSD. A bandwidth sweep from functional tags up to full solutions peaks at the short skeleton, and ordering and cross-sample controls suggest both operation order and question-skeleton matching matter.

Critical strengths
  1. The bandwidth hierarchy (B1 to B4) plus the ordering and cross-sample controls form a well-designed experimental spine. "More privileged information is not better" is a clean and non-obvious result. The fact that shuffled and reversed skeletons land below base makes the claim that structure matters (rather than just the removal of results) something the experiments actually test, not just assert.

  2. The intervention is small and deployable. It changes only what goes into the teacher's context, keeps the student question-only, and adds nothing at inference time. That makes it easy for others to adopt or to refute.

Critical weaknesses
  1. The training and evaluation lengths do not match, which undercuts the on-policy premise. Max training completion is 1,024 tokens (Table 5) while evaluation generates up to 38,912, with median outputs around 19.7K. Over 90% of each deployed trajectory is therefore never distilled on. For competition math with Qwen3 thinking models this means the method only ever shapes the opening of the reasoning, and I cannot tell whether the gains reflect better long-chain reasoning or something specific to the truncated training regime.

  2. The baselines look undertrained, which makes the headline deltas hard to read. At 4B, OPSD reaches 61.70 against a base of 61.76, so zero gain, and SFT lands at 61.61, below base. In Table 3 majority-vote SFT drops from 72.22 to 52.22 on AIME24. Most of the claimed +3.37 over OPSD at 4B comes from OPSD not working at all here. With LoRA only, 100 steps and 1 rollout per prompt, my worry is that every method is undertrained and the ranking between them is not informative.

  3. The skeleton extractor is never identified. Appendix B.3 says only "a fixed offline extractor" and names no model. If it is a stronger external LLM then this is not self-distillation any more, and the comparison against OPSD (which uses the dataset solution as-is) is confounded by another model's reasoning entering the teacher's context. It also blocks reproduction.

Questions for authors
  • Which model generates the skeletons, and what fraction of examples pass the validator inside the three-attempt budget?
  • In Table 2, shuffled (36.17) and reversed (34.01) fall below base (36.67). What do the training curves look like there? Is the student actively degrading or just not improving?
  • When the 1,024-token cap truncates a rollout mid-reasoning, is the KL still computed over the truncated tail, and did you apply any length correction?
Reviewer_JTy2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structural-Bottleneck On-Policy Self-Distillation. The method offline extracts a skeleton, composed of a small number of ordered reasoning operations, from validated mathematical solutions, filtering out explicit intermediate results and the final answer; During training, a frozen teacher provides next-token distributions based on the question, the skeleton, and the prefix generated by the student. The paper trains Qwen3-1.7B, 4B, and 8B on the OpenThoughts math dataset and evaluates them on AIME 2024, AIME 2025, and HMMT February 2025. SB-OPSD achieved the highest Average@12 among the compared methods across all three model scales.

Critical strengths
  1. The method simply modifies the context received by the teacher; the student requires no additional input during inference. Across three model scales, SB-OPSD’s Average@12 scores on all three benchmarks were higher than those of the full-solution OPSD, demonstrating the practical value of this simple modification.
  2. Shuffling or reversing the steps in the skeleton significantly degrades performance, indicating that the order of reasoning operations plays a role in training. The method also shortens the teacher context, reducing the teacher-to-student prompt length ratio from 7.9× to 2.2×.
Critical weaknesses
  1. The incremental improvement over prior work is unclear. The paper cites PS-OPSD [1], which has already converted complete solutions into structured guidance for the teacher to use within the same OPSD framework. This paper further emphasizes the removal of intermediate results, but does not directly compare it with PS-OPSD.
  2. The ablation analysis does not clarify whether the benefits actually stem from removing intermediate results. The short skeleton reduces both context length and reasoning details, making it impossible to distinguish these factors in a comparison with the full answer; experiments involving shuffled steps only demonstrate that order matters. There is a lack of experiments that retain the same reasoning operations, control context length, and compare whether intermediate results are included.

References [1] Xuyang Zhao et al. Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning. arXiv:2608.01589v1, 2026.

Questions for authors
  1. How do the performance and training overhead of SB-OPSD compare to those of PS-OPSD?
  2. Is it possible to retain the same reasoning operations and control the context length, then compare the results with and without intermediate results to confirm that the gains stem from removing result information?
  3. What model and prompt does Skeleton use for extraction? Have the filtered skeletons been manually inspected to confirm that they do not contain any implicit intermediate results or answer information?
Reviewer_qmvgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privilege mismatch in on-policy self-distillation: the teacher still sees a verified solution while the student is question-only at train and test. SB-OPSD replaces the full solution with a compact, result-free structural skeleton (ordered tagged operations, no intermediate results or final answer). A frozen teacher conditioned on the question, the skeleton, and the student prefix provides forward-KL next-token targets; the student never sees the skeleton. Main eval is Qwen3 LoRA on math (AIME / HMMT style benches), plus a privilege-bandwidth sweep and ordering / cross-sample controls. They report gains over Base and full-solution OPSD across a few scales. HMMT looks more mixed.

Critical strengths
  • The intervention relative to full-solution OPSD is clean and easy to understand. Same frozen teacher, same on-policy prefixes, same forward KL; only teacher context changes from the full answer to a validated result-free skeleton. I do not need a new training objective to get the idea.
  • The bandwidth sweep is useful. Short ordered skeletons beat tags-only and also beat fuller privilege (more detailed skeletons / full solutions) in their setup. That "more privilege is not better" finding is interesting, and the order / reverse / cross-sample controls go in the right direction for the story.
  • Gains over Base look real on the main Average metric across the reported scales, not just noise on one bench. Efficiency / shorter context claims are a nice side bonus if they hold.
Critical weaknesses
  • Scope is narrow. Everything is competition math on Qwen3 with a short LoRA recipe. HMMT already flips on some columns. I am not ready to treat the bandwidth sweet spot as settled beyond this setup.

  • The 4B OPSD baseline looks soft to me. Full-solution OPSD lands at or below Base there, which inflates the SB vs OPSD gap. If OPSD is just a weak run at that scale, the headline comparison is less convincing than the authors suggest.

  • "Distilling structure not solutions" is only partly proven. The skeleton is result-free but still highly plan-specific. Cross-sample skeletons still help a bit, so some of the gain may be generic structural scaffolding rather than matched structure. Leakage / answer-correlated branching cues are not fully closed.

  • Overall improve looks significant vs Base in places, but more marginal / messy once you look at OPSD, single-bench flips, and the soft 4B baseline. For me that is not enough for accept as written.

Questions for authors
  • On Qwen3-4B, why does full-solution OPSD land at or below Base while SB-OPSD jumps? Seed-level traces or a longer-horizon rerun would change how much weight I put on the SB vs OPSD claim at that scale.

  • If you ablate the extractor (weaker model, or skeletons written without seeing the full solution), do the short-skeleton gains survive?

  • For cross-sample shuffle: which problems still improve, and do those skeletons share coarse templates? A short breakdown would tell me how much is question-matched vs generic regularization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.