- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to improve language models on difficult problems while preserving their existing capabilities. It proposes Projected Distillation, which uses a stronger teacher to minimally correct the student's failed responses and dynamically refreshes these corrections as training progresses. Experiments on two student models (Qwen3-8B and OLMo-2-7B) across five benchmarks demonstrate improvements in the specialization–retention trade-off over several post-training baselines.
Critical strengths
- A well-motivated framework for learning from difficult problems. The paper addresses the limited learning signal available from on-policy reinforcement learning when student-generated solutions consistently fail. By transforming failed attempts into minimally corrected solutions, the proposed framework provides useful supervision while attempting to reduce distribution shift.
- Comprehensive evaluations of the specialization–retention trade-off. Experiments across two model families and five benchmarks compare the proposed method with GRPO, OPD, SPOT, and SODA. Table 1 reports the highest AUPF in 8 of 9 same-family teacher settings and 7 of 9 black-box projector settings, demonstrating promising effectiveness across diverse tasks.
Critical weaknesses
- The theoretical guarantees are not directly applicable to the implemented method. Theorem 1 analyzes an expert-disagreement surrogate under an idealized projector and a strong prefix-sensitivity assumption. However, Appendix I.2.9 explicitly acknowledges that the deployed diff-localized DPO objective is not covered. Moreover, the linear-in-horizon regret interpretation requires an edit size of O(1), which is not established empirically as the generation horizon increases. These limitations weaken the theoretical justification for the proposed training framework.
- Experimental reliability claims require stronger evidence. Table 6 reports a held-out accuracy of 0.760 ± 0.006 for black-box projected distillation on Qwen3-8B/MATH, whereas Appendix D.5 reports MATH-500 scores of 0.749, 0.311, and 0.688 across three math-trained seeds. These figures appear difficult to reconcile without additional protocol details.
- Dependence on privileged supervision and unmatched projector costs. Appendix D.3 indicates that the projector receives ground-truth answers or reference outputs for several tasks. Although this information is also provided to relevant baselines, the method's effectiveness without such supervision remains unclear. Additionally, the comparisons match student generation and optimization budgets but do not establish equivalent projector-query or computational costs. This limits conclusions about practical efficiency and the benefits attributable specifically to online correction.
Questions for authors
- Clarification of experimental consistency: How can the Qwen3-8B/MATH retention results in Table 6 be reconciled with the three training-seed MATH-500 scores reported in Appendix D.5? Do these results use different checkpoints, decoding configurations, or evaluation protocols?
- Clarification of theoretical applicability: Can the authors provide a theoretical connection between the deployed diff-localized DPO objective and Theorem 1, or additional evidence supporting the required assumptions? How does the edit-size-to-horizon ratio behave across different benchmarks?
- Clarification of projector dependence and computational cost: What are the total projector-query counts, correction success rates, and computational or API costs across different benchmarks? Would the performance advantage persist under matched teacher-query budgets and without providing gold answers directly to the projector?
9 comments
Response by Authors (2/2)
Additional projector details and cost analysis (W3, Q3)
Query counts and correction rates.
Each 250-update run generates 2,000 student attempts and requests two corrections per failed attempt, requiring at most 4,000 projector calls. Correct responses require no calls. The table reports request ranges across training seeds and the percentage of failed attempts successfully corrected.
| Benchmark | Qwen3-8B requests | Corrected | OLMo-2-7B requests | Corrected |
|---|---|---|---|---|
| GSM8K | 3,292–3,442 | 99.6% | 2,478–2,982 | 99.9% |
| MATH | 3,870–3,914 | 69.0% | 3,922–3,934 | 92.2% |
| LiveCodeBench | 3,060–3,164 | 13.4% | — | — |
| SciKnowEval | 2,720–2,976 | 94.2% | 1,780–2,956 | 95.4% |
| ToolAlpaca | 1,764–1,874 | 96.8% | 1,264–1,666 | 96.5% |
We have added this accounting and the projector configuration to Appendix D.4 (Table 6).
Across the five Qwen3-8B benchmarks, we estimate an average of approximately 7.8M teacher tokens (input + output) per 250-update training run.
Computational cost.
To assess the computational cost of projections for effective learning, we compare against concurrent work [4], first submitted to arXiv on 1 October 2026 after our ICLR submission, that uses MCMC sampling to construct correct targets with high student likelihood. Our teacher-generated minimal corrections achieve better performance with substantially less student computation: SFT reaches 0.545 versus 0.534 with 30× fewer estimated student FLOPs, while our full objective reaches 0.557 with 16× fewer estimated student FLOPs (Appendix D.4, Table 7).
This replaces repeated student sampling with teacher queries. Even allowing for a frozen 32B teacher with 1–2 refinement rounds, we estimate about teacher-plus-student FLOPs for this workload, compared with for MCMC-based Sampling SFT. This demonstrates that our approach leads to significant cost-effectiveness when learning from high likelihood correct targets. We have added the comparison setup, results and refinement-budget estimate to Appendix D.4.
We thank the reviewer for the feedback that helped us strengthen the manuscript. We hope the clarified theoretical scope, evaluation protocols and additional experiments address their concerns. We would appreciate the reviewer reconsidering their rating and would be happy to discuss any remaining questions.
References
[1] Ross, S. et al. (2011) ‘A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning’, AISTATS.
[2] Le, H. et al. (2016) ‘Smooth Imitation Learning for Online Sequence Prediction’, ICML.
[3] Pfrommer, D. et al. (2022) ‘TaSIL: Taylor Series Imitation Learning’, NeurIPS.
[4] Karan, A. et al. (2026) ‘Finetuning with Sampling: SFT Learns Better Than You Think’, arXiv:2610.02140.
[5] Penaloza, E. et al. (2026) ‘Privileged Information Distillation for Language Models’, ICML.
Requesting reviewer's thoughts on our rebuttal
Dear reviewer 3esT,
Thanks once again for your review. We have responded to your queries about the scope of the theory, the consistency of the MATH results, the ablation with no privileged information and the details of the projector. We have also modified the manuscript to incorporate your suggestions. We hope these responses and revisions address your concerns, and we would really appreciate it if you would consider increasing your rating of our paper in light of these updates. We would be happy to engage in further discussion during the remaining rebuttal period to address any further queries.
Best, Authors
Response by Authors (1/2)
We thank the reviewer for recognizing the motivation of our framework, the comprehensiveness of our evaluation of the specialization–retention trade-off and the clarity in our presentation.
In this rebuttal, we address their concerns about the theory and its applicability, the experimental protocols on MATH, the ablation without privileged information and the cost analysis of projections in projected distillation.
Theory and its applicability (W1, Q2)
First of all, we want to clarify that we have been transparent about the assumptions, scope and limitations of our analysis from the outset. Section 5 explicitly states the idealized projector setting and the prefix-sensitivity assumption. Section 6’s limitations paragraph discusses the strength of this assumption, while Appendix I.2.9 explicitly distinguishes the analyzed disagreement loss from the deployed contrastive objective.
Secondly, we deliberately chose a DAgger-style analysis because we want to compare our proposed approach of learning using close, correct targets from the teacher with the very well-established guarantees for DAgger and behavior cloning [1]. For this analysis, we use smoothness assumptions on prefix-sensitivity. Such smoothness assumptions have many precedents in imitation learning theory (e.g. SIMILE (ICML 2016)[2], TaSIL (NeurIPS 2022)[3]).
Thirdly, regarding the loss used in the analysis, we use 0–1 disagreement following the DAgger analysis [1, Theorem 2.2] to isolate the effect of edits on overall learning. Note that the original DAgger work itself uses surrogate losses that are different from the 0-1 loss used in the theory (please refer to section 5 of [1]).
Minimal-edit interpretation
Our analysis considers a finite generation horizon H. Here, the term “minimal edits” refers to localized corrections: the m=O(1) case describes a bounded edit length, while full-response replacement has m=Θ(H). Longer edits lie between these cases. The bound retains m explicitly through the drift term ηmH, giving linear horizon dependence in the bounded-edit case and quadratic dependence for full rewrites.
Consistency of the results on MATH (W2, Q1)
Thank you for requesting this clarification. Table 6 of the submitted manuscript (Table 8 in the revision) reports accuracy on the in-loop held-out MATH slice, averaged across three training seeds: the individual scores are 0.760, 0.770 and 0.750, yielding 0.760 ± 0.006. Appendix D.5 separately reports full MATH-500 evaluations, where the three training seeds score 0.749, 0.311 and 0.688. These evaluations use different problem sets and therefore need not yield matching accuracies. We have clarified these evaluation sets and protocols in Appendix D.5 and the relevant table captions, including the greater training-seed variation on full MATH-500 and the distinction between training-seed and evaluation-seed error bars.
We also conducted new cross-benchmark evaluations. After training on 501 hard MATH problems, our method improves pass@1 by 0.009 and pass@16 by 0.040 over the base model on 2,885 unseen hard problems from OlympiadBench, Omni-MATH, AIME and DAPO-Math, exceeding the pooled gains of GRPO, SPOT and SODA. We additionally repeat training on Omni-MATH and observe transfer back to held-out hard MATH problems, where pass@16 reaches 0.362. These results support generalization across training benchmarks. Each experiment uses one training seed and reports the best evaluated checkpoints. We have added the MATH-trained transfer experiment to Section 4.1, with the full experimental setup and results in Appendix E.5 (Tables 13–14).
Additional experiments with no privileged information (W3, Q3)
Our experiments establish the relative benefits of learning from close, correct targets against other baselines when the teacher is provided with the same information. Providing privileged information to the teacher during training is an established practice in distillation (please see Privileged Information Distillation for Language Models, ICML 2026 [5]). We are therefore not sure that withholding reference answers would add substantial insights into this relative edge beyond the existing information-matched comparisons.
However, to address the reviewer's suggestion directly, we performed an ablation where we compare projected distillation with OPD using a Qwen3-8B student and a Qwen3-32B teacher on the GSM8K dataset in a single-seed run.
This control experiment supports the same conclusion even without reference answers supplied to the teacher. We have added the experimental setup and results to Appendix E.6 (Table 15).