- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SAPD, a method that combines the advantages of SFT and OPSD. SAPD trains on fixed reference trajectories, as in SFT, while leveraging the distributional supervision of OPSD, thereby eliminating the computational cost of autoregressive rollout generation required for on-policy training. Specifically, the method divides ground-truth reasoning data into segments and distills the teacher's distribution, conditioned on each segment as privileged input, into the student. As a result, SAPD achieves performance comparable to existing OPSD baselines while substantially reducing training time.
Critical strengths
The paper is well structured and generally easy to understand. Furthermore, it makes a meaningful contribution to computational efficiency by eliminating the rollout generation cost associated with conventional on-policy self-distillation while maintaining competitive performance.
Critical weaknesses
Minor 1. Since SAPD uses fixed reference trajectories and a frozen teacher, it appears possible to precompute and cache the teacher distributions generated using the ground-truth segment of each reasoning step as privileged context. This approach could further improve training efficiency by eliminating repeated teacher forward passes, although it may require substantial storage space. While the authors do not explicitly propose such a caching strategy, it would be valuable to provide further discussion on this possibility, considering the scalability of the method and the trade-off between computational and storage costs.
Major 1. It is unclear whether the teacher distribution generated by providing the ground-truth segment of the current reasoning step as privileged context offers meaningful supervision that encourages better reasoning, or whether it simply encourages the model to reproduce the given ground-truth segment. In Section 4.3, the authors present a case study showing that SAPD assigns relatively high probabilities to alternative tokens beyond the ground-truth token. However, it remains unclear whether this behavior consistently occurs across diverse reasoning contexts and how much it actually contributes to improved reasoning performance. A dataset-level quantitative analysis would help clarify these points. Furthermore, the authors should discuss whether assigning high probabilities to semantically similar alternative tokens necessarily provides a beneficial learning signal.
Major 2. The paper lacks sufficient ablation studies on the granularity of reasoning step segmentation. What criteria and level of granularity are most effective for dividing ground-truth reasoning data into steps? If performance varies significantly depending on segmentation granularity, this may raise concerns about the generalizability of the method to other datasets or benchmarks. Therefore, additional experiments examining performance across different segmentation granularities, along with a sensitivity analysis, would be valuable.
Major 3. The current performance comparison is conducted using only a single 4B LLM. However, while OPSD learns from trajectories generated by the student itself, SAPD learns from fixed ground-truth trajectories. Therefore, the relative effectiveness of the two approaches may vary depending on the performance and size of the base model. In particular, larger models may generate higher-quality student trajectories and make better use of privileged context, making it unclear whether SAPD's performance advantage would be maintained. Thus, the authors should compare OPSD and SAPD across different model sizes within the same LLM family to validate the scalability of the proposed method.
Questions for authors
The questions are summarized below. Please refer to the Critical Weaknesses section for detailed explanations.
Minor 1: Could precomputing and caching teacher distributions further improve training efficiency, and what are the computational and storage trade-offs? Major 1: Does privileged-context teacher supervision genuinely improve reasoning beyond reproducing ground-truth segments, and are alternative token probabilities beneficial learning signals? Major 2: How sensitive is SAPD's performance to reasoning step segmentation granularity? Major 3: Does SAPD maintain its performance advantage over OPSD across different model sizes within the same LLM family?