acceptodds
Under review as a conference paper at ICLR 2027

SAPD: Step-Aligned Privileged Distillation

Abstract

On-policy post-training can improve large language models by learning from their own trajectories, but requires costly rollout generation. We ask whether fixed demonstrations can support competitive off-policy learning through better supervision. Our premise is that their usefulness depends not only on the training trajectories, but also on whether supervision provides informative preferences among continuations and connects this guidance to the reasoning decision being learned. We introduce Step-Aligned Privileged Distillation (SAPD), a rollout-free self-distillation method that turns demonstrations into step-aligned distributional supervision. Its key insight is to use the known progression of a reference solution to associate each reasoning transition with targeted privileged guidance, rather than treating the solution as undifferentiated context. On mathematical reasoning benchmarks, SAPD outperforms supervised fine-tuning and label smoothing on average while remaining competitive with on-policy reinforcement learning and self-distillation. Analyses support both the value of context-dependent distributional guidance and the benefit of aligning privileged information with the current step. SAPD also largely preserves out-of-domain coding performance and achieves approximately training-loop speedups over the on-policy baselines in our efficiency evaluation. These findings suggest that carefully constructed supervision can make fully off-policy post-training a competitive and computationally efficient alternative in the evaluated setting. Our code is available at https://anonymous.4open.science/r/SPD_ICLR-9EE7.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training, self-distillation, off-policy learning, llms reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mL8sOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAPD, a method that combines the advantages of SFT and OPSD. SAPD trains on fixed reference trajectories, as in SFT, while leveraging the distributional supervision of OPSD, thereby eliminating the computational cost of autoregressive rollout generation required for on-policy training. Specifically, the method divides ground-truth reasoning data into segments and distills the teacher's distribution, conditioned on each segment as privileged input, into the student. As a result, SAPD achieves performance comparable to existing OPSD baselines while substantially reducing training time.

Critical strengths

The paper is well structured and generally easy to understand. Furthermore, it makes a meaningful contribution to computational efficiency by eliminating the rollout generation cost associated with conventional on-policy self-distillation while maintaining competitive performance.

Critical weaknesses

Minor 1. Since SAPD uses fixed reference trajectories and a frozen teacher, it appears possible to precompute and cache the teacher distributions generated using the ground-truth segment of each reasoning step as privileged context. This approach could further improve training efficiency by eliminating repeated teacher forward passes, although it may require substantial storage space. While the authors do not explicitly propose such a caching strategy, it would be valuable to provide further discussion on this possibility, considering the scalability of the method and the trade-off between computational and storage costs.

Major 1. It is unclear whether the teacher distribution generated by providing the ground-truth segment of the current reasoning step as privileged context offers meaningful supervision that encourages better reasoning, or whether it simply encourages the model to reproduce the given ground-truth segment. In Section 4.3, the authors present a case study showing that SAPD assigns relatively high probabilities to alternative tokens beyond the ground-truth token. However, it remains unclear whether this behavior consistently occurs across diverse reasoning contexts and how much it actually contributes to improved reasoning performance. A dataset-level quantitative analysis would help clarify these points. Furthermore, the authors should discuss whether assigning high probabilities to semantically similar alternative tokens necessarily provides a beneficial learning signal.

Major 2. The paper lacks sufficient ablation studies on the granularity of reasoning step segmentation. What criteria and level of granularity are most effective for dividing ground-truth reasoning data into steps? If performance varies significantly depending on segmentation granularity, this may raise concerns about the generalizability of the method to other datasets or benchmarks. Therefore, additional experiments examining performance across different segmentation granularities, along with a sensitivity analysis, would be valuable.

Major 3. The current performance comparison is conducted using only a single 4B LLM. However, while OPSD learns from trajectories generated by the student itself, SAPD learns from fixed ground-truth trajectories. Therefore, the relative effectiveness of the two approaches may vary depending on the performance and size of the base model. In particular, larger models may generate higher-quality student trajectories and make better use of privileged context, making it unclear whether SAPD's performance advantage would be maintained. Thus, the authors should compare OPSD and SAPD across different model sizes within the same LLM family to validate the scalability of the proposed method.

Questions for authors

The questions are summarized below. Please refer to the Critical Weaknesses section for detailed explanations.

Minor 1: Could precomputing and caching teacher distributions further improve training efficiency, and what are the computational and storage trade-offs? Major 1: Does privileged-context teacher supervision genuinely improve reasoning beyond reproducing ground-truth segments, and are alternative token probabilities beneficial learning signals? Major 2: How sensitive is SAPD's performance to reasoning step segmentation granularity? Major 3: Does SAPD maintain its performance advantage over OPSD across different model sizes within the same LLM family?

Reviewer_qYxLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Step-Aligned Privileged Distillation (SAPD), an off-policy self-distillation method for improving LLM reasoning without generating student rollouts. Unlike conventional supervised fine-tuning (SFT), SAPD uses a frozen teacher conditioned on the corresponding reference reasoning step to provide token-level probability distributions. The student learns from these distributions using forward KL divergence while training on fixed reference prefixes.

Experiments with Qwen3-4B-Instruct-2507 on three mathematical reasoning benchmarks show that SAPD achieves an average accuracy of 47.87%, compared with 44.54% for SFT, 46.76% for GRPO, and 47.50% for on-policy self-distillation.

Critical strengths
  1. Intuitive step-aligned supervision. Conditioning the teacher on the current reasoning step rather than the complete solution is a simple and potentially useful design choice. Table 2 supports its effectiveness over full-solution and deliberately misaligned privileged information.

  2. Useful analysis of distributional supervision. Figure 1 illustrates how the proposed targets distinguish valid alternatives from incorrect continuations. Figure 2 further examines non-reference token preferences, providing insight beyond final benchmark accuracy.

Critical weaknesses
  1. Modest performance gains without statistical validation. In Table 1, SAPD improves over OPSD by only 0.37 percentage points and GRPO by 1.11 points on average. It outperforms all baselines on AIME24 but not on AIME25 or HMMT25. Given the small evaluation sets (30 problems per benchmark), the absence of multiple training seeds, uncertainty estimates, or significance testing makes it difficult to establish whether these improvements are reliable. Furthermore, selecting checkpoints based on performance on the same benchmarks used for final reporting introduces potential selection bias.

  2. Insufficient comparison against standard off-policy distributional distillation. The proposed method combines established distributional knowledge distillation with step-specific privileged information. Although Table 2 demonstrates the importance of alignment, the experiments lack a conventional off-policy KL-distillation baseline without privileged information. Notably, Generalized Knowledge Distillation (GKD) already supports distillation on fixed reference trajectories. Comparing against its off-policy configuration would better isolate the contribution of step-aligned privileged supervision.

  3. Limited validation of step alignment. Table 2 compares SAPD against full-solution and deliberately misaligned supervision. However, the full-solution variant changes the amount and positioning of privileged context, while the misaligned variant deliberately introduces irrelevant information. These comparisons do not completely isolate whether improvements arise from meaningful reasoning-step alignment or simply providing shorter, more relevant context. Sensitivity to segmentation granularity would strengthen this claim.

  4. Limited generalizability and incomplete efficiency analysis. The experiments involve one 4B model, one mathematical reasoning training dataset, and a relatively narrow evaluation. It remains unclear whether the benefits extend to larger models, other reasoning domains, or different demonstration structures.

Questions for authors
  1. How does SAPD compare against standard off-policy KL distillation without privileged information? How sensitive are its improvements to reasoning-step segmentation granularity?
Reviewer_Wck8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Step-Aligned Privileged Distillation (SAPD), a method that learns from fixed demonstrations without generating student rollouts. A frozen teacher receives the current reference step as privileged information, and the student learns from its token distributions. SAPD improves average mathematical reasoning performance over supervised baselines, remains competitive with the evaluated on-policy methods, and reduces training time while largely preserving coding performance.

Critical strengths
  1. The paper addresses a practical problem: improving learning from existing demonstrations while avoiding repeated rollout generation. The method is straightforward and requires neither a larger teacher nor additional annotations.
  2. The full-solution and misaligned-guidance ablations provide useful evidence for step alignment while keeping the student’s training contexts unchanged.
  3. The results demonstrate a promising balance between accuracy and efficiency. SAPD improves over supervised baselines without increasing response length, largely preserves coding capabilities.
Critical weaknesses
  1. The comparison with OPSD uses substantially different training lengths. OPSD rollouts are limited to 1K tokens, whereas SAPD allows teacher-forced inputs up to 16K tokens. Given that evaluation responses average around 10K tokens, this may restrict OPSD’s supervision on later reasoning steps. Reporting accuracy and runtime for one longer-rollout OPSD setting would strengthen the comparison.
  2. The contribution of preferences among alternative tokens remains unclear. Figure 2 shows that SAPD preserves these preferences, but SAPD also changes the confidence assigned to the reference token. The fixed label-smoothing baseline does not separate these effects. An adaptive label-smoothing control that matches the teacher’s reference-token probability would help establish the additional value of structured preferences.
Questions for authors

Please answer the weakness points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.