acceptodds
Under review as a conference paper at ICLR 2027

Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Abstract

On-policy learning is central to post-training: training on a model’s own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail. On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake. Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from model’s generations and, as we show, may erode existing capabilities. We therefore treat on-policyness as a continuum and propose **projected distillation**, which minimally edits the student’s failed attempts into correct targets. The method uses contrastive updates to favor corrections over failed attempts and refreshes these corrections as the student evolves, enabling near-on-policy learning even when all student attempts fail. Theoretically, we prove that the student’s learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning. Empirically, across two students (`Qwen3-8B` and `OLMo-2-7B`) and five domains spanning math, code, science, and tool use, projected distillation gives the highest area under the specialization–retention frontier (best on 8 of 10 model–task pairs). Our ablations highlight the importance of online corrections, contrastive loss, and, on-policyness: moving targets farther off-policy steadily worsens the specialization–retention trade-off, with retention falling from 95% to 75% when minimal corrections are replaced by student-agnostic gold solutions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Distillation, Post-training, Continual Learning, Online Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3esTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to improve language models on difficult problems while preserving their existing capabilities. It proposes Projected Distillation, which uses a stronger teacher to minimally correct the student's failed responses and dynamically refreshes these corrections as training progresses. Experiments on two student models (Qwen3-8B and OLMo-2-7B) across five benchmarks demonstrate improvements in the specialization–retention trade-off over several post-training baselines.

Critical strengths
  1. A well-motivated framework for learning from difficult problems. The paper addresses the limited learning signal available from on-policy reinforcement learning when student-generated solutions consistently fail. By transforming failed attempts into minimally corrected solutions, the proposed framework provides useful supervision while attempting to reduce distribution shift.
  2. Comprehensive evaluations of the specialization–retention trade-off. Experiments across two model families and five benchmarks compare the proposed method with GRPO, OPD, SPOT, and SODA. Table 1 reports the highest AUPF in 8 of 9 same-family teacher settings and 7 of 9 black-box projector settings, demonstrating promising effectiveness across diverse tasks.
Critical weaknesses
  1. The theoretical guarantees are not directly applicable to the implemented method. Theorem 1 analyzes an expert-disagreement surrogate under an idealized projector and a strong prefix-sensitivity assumption. However, Appendix I.2.9 explicitly acknowledges that the deployed diff-localized DPO objective is not covered. Moreover, the linear-in-horizon regret interpretation requires an edit size of O(1), which is not established empirically as the generation horizon increases. These limitations weaken the theoretical justification for the proposed training framework.
  2. Experimental reliability claims require stronger evidence. Table 6 reports a held-out accuracy of 0.760 ± 0.006 for black-box projected distillation on Qwen3-8B/MATH, whereas Appendix D.5 reports MATH-500 scores of 0.749, 0.311, and 0.688 across three math-trained seeds. These figures appear difficult to reconcile without additional protocol details.
  3. Dependence on privileged supervision and unmatched projector costs. Appendix D.3 indicates that the projector receives ground-truth answers or reference outputs for several tasks. Although this information is also provided to relevant baselines, the method's effectiveness without such supervision remains unclear. Additionally, the comparisons match student generation and optimization budgets but do not establish equivalent projector-query or computational costs. This limits conclusions about practical efficiency and the benefits attributable specifically to online correction.
Questions for authors
  1. Clarification of experimental consistency: How can the Qwen3-8B/MATH retention results in Table 6 be reconciled with the three training-seed MATH-500 scores reported in Appendix D.5? Do these results use different checkpoints, decoding configurations, or evaluation protocols?
  2. Clarification of theoretical applicability: Can the authors provide a theoretical connection between the deployed diff-localized DPO objective and Theorem 1, or additional evidence supporting the required assumptions? How does the edit-size-to-horizon ratio behave across different benchmarks?
  3. Clarification of projector dependence and computational cost: What are the total projector-query counts, correction success rates, and computational or API costs across different benchmarks? Would the performance advantage persist under matched teacher-query budgets and without providing gold answers directly to the projector?
9 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

We thank the reviewer for recognizing the motivation of our framework, the comprehensiveness of our evaluation of the specialization–retention trade-off and the clarity in our presentation.

In this rebuttal, we address their concerns about the theory and its applicability, the experimental protocols on MATH, the ablation without privileged information and the cost analysis of projections in projected distillation.

Note: We have provided a revised manuscript in accordance with ICLR guidelines, with changes highlighted in purple.


Theory and its applicability (W1, Q2)

First of all, we want to clarify that we have been transparent about the assumptions, scope and limitations of our analysis from the outset. Section 5 explicitly states the idealized projector setting and the prefix-sensitivity assumption. Section 6’s limitations paragraph discusses the strength of this assumption, while Appendix I.2.9 explicitly distinguishes the analyzed disagreement loss from the deployed contrastive objective.

Secondly, we deliberately chose a DAgger-style analysis because we want to compare our proposed approach of learning using close, correct targets from the teacher with the very well-established guarantees for DAgger and behavior cloning [1]. For this analysis, we use smoothness assumptions on prefix-sensitivity. Such smoothness assumptions have many precedents in imitation learning theory (e.g. SIMILE (ICML 2016)[2], TaSIL (NeurIPS 2022)[3]).

Thirdly, regarding the loss used in the analysis, we use 0–1 disagreement following the DAgger analysis [1, Theorem 2.2] to isolate the effect of edits on overall learning. Note that the original DAgger work itself uses surrogate losses that are different from the 0-1 loss used in the theory (please refer to section 5 of [1]).

Minimal-edit interpretation

Our analysis considers a finite generation horizon HH. Here, the term “minimal edits” refers to localized corrections: the m=O(1)m=O(1) case describes a bounded edit length, while full-response replacement has m=Θ(H)m=\Theta(H). Longer edits lie between these cases. The bound retains mm explicitly through the drift term ηmH\eta mH, giving linear horizon dependence in the bounded-edit case and quadratic dependence for full rewrites.


Consistency of the results on MATH (W2, Q1)

Thank you for requesting this clarification. Table 6 of the submitted manuscript (Table 8 in the revision) reports accuracy on the in-loop held-out MATH slice, averaged across three training seeds: the individual scores are 0.760, 0.770 and 0.750, yielding 0.760 ± 0.006. Appendix D.5 separately reports full MATH-500 evaluations, where the three training seeds score 0.749, 0.311 and 0.688. These evaluations use different problem sets and therefore need not yield matching accuracies. We have clarified these evaluation sets and protocols in Appendix D.5 and the relevant table captions, including the greater training-seed variation on full MATH-500 and the distinction between training-seed and evaluation-seed error bars.

We also conducted new cross-benchmark evaluations. After training on 501 hard MATH problems, our method improves pass@1 by 0.009 and pass@16 by 0.040 over the base model on 2,885 unseen hard problems from OlympiadBench, Omni-MATH, AIME and DAPO-Math, exceeding the pooled gains of GRPO, SPOT and SODA. We additionally repeat training on Omni-MATH and observe transfer back to held-out hard MATH problems, where pass@16 reaches 0.362. These results support generalization across training benchmarks. Each experiment uses one training seed and reports the best evaluated checkpoints. We have added the MATH-trained transfer experiment to Section 4.1, with the full experimental setup and results in Appendix E.5 (Tables 13–14).


Additional experiments with no privileged information (W3, Q3)

Our experiments establish the relative benefits of learning from close, correct targets against other baselines when the teacher is provided with the same information. Providing privileged information to the teacher during training is an established practice in distillation (please see Privileged Information Distillation for Language Models, ICML 2026 [5]). We are therefore not sure that withholding reference answers would add substantial insights into this relative edge beyond the existing information-matched comparisons.

However, to address the reviewer's suggestion directly, we performed an ablation where we compare projected distillation with OPD using a Qwen3-8B student and a Qwen3-32B teacher on the GSM8K dataset in a single-seed run.

MetricOursOPD
Final hard-set accuracy0.1800.057
Final held-out accuracy0.9300.960
AUPF0.1950.087

This control experiment supports the same conclusion even without reference answers supplied to the teacher. We have added the experimental setup and results to Appendix E.6 (Table 15).

AuthorsOpenReview ↗

Response by Authors (2/2)

Additional projector details and cost analysis (W3, Q3)

Query counts and correction rates.

Each 250-update run generates 2,000 student attempts and requests two corrections per failed attempt, requiring at most 4,000 projector calls. Correct responses require no calls. The table reports request ranges across training seeds and the percentage of failed attempts successfully corrected.

BenchmarkQwen3-8B requestsCorrectedOLMo-2-7B requestsCorrected
GSM8K3,292–3,44299.6%2,478–2,98299.9%
MATH3,870–3,91469.0%3,922–3,93492.2%
LiveCodeBench3,060–3,16413.4%——
SciKnowEval2,720–2,97694.2%1,780–2,95695.4%
ToolAlpaca1,764–1,87496.8%1,264–1,66696.5%

We have added this accounting and the projector configuration to Appendix D.4 (Table 6).

Across the five Qwen3-8B benchmarks, we estimate an average of approximately 7.8M teacher tokens (input + output) per 250-update training run.

Computational cost.

To assess the computational cost of projections for effective learning, we compare against concurrent work [4], first submitted to arXiv on 1 October 2026 after our ICLR submission, that uses MCMC sampling to construct correct targets with high student likelihood. Our teacher-generated minimal corrections achieve better performance with substantially less student computation: SFT reaches 0.545 versus 0.534 with 30× fewer estimated student FLOPs, while our full objective reaches 0.557 with 16× fewer estimated student FLOPs (Appendix D.4, Table 7).

This replaces repeated student sampling with teacher queries. Even allowing for a frozen 32B teacher with 1–2 refinement rounds, we estimate about 2.4–3.6×10182.4\text{–}3.6\times10^{18} teacher-plus-student FLOPs for this workload, compared with 8.4×10188.4\times10^{18} for MCMC-based Sampling SFT. This demonstrates that our approach leads to significant cost-effectiveness when learning from high likelihood correct targets. We have added the comparison setup, results and refinement-budget estimate to Appendix D.4.


We thank the reviewer for the feedback that helped us strengthen the manuscript. We hope the clarified theoretical scope, evaluation protocols and additional experiments address their concerns. We would appreciate the reviewer reconsidering their rating and would be happy to discuss any remaining questions.

References

[1] Ross, S. et al. (2011) ‘A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning’, AISTATS.

[2] Le, H. et al. (2016) ‘Smooth Imitation Learning for Online Sequence Prediction’, ICML.

[3] Pfrommer, D. et al. (2022) ‘TaSIL: Taylor Series Imitation Learning’, NeurIPS.

[4] Karan, A. et al. (2026) ‘Finetuning with Sampling: SFT Learns Better Than You Think’, arXiv:2610.02140.

[5] Penaloza, E. et al. (2026) ‘Privileged Information Distillation for Language Models’, ICML.

AuthorsOpenReview ↗

Requesting reviewer's thoughts on our rebuttal

Dear reviewer 3esT,

Thanks once again for your review. We have responded to your queries about the scope of the theory, the consistency of the MATH results, the ablation with no privileged information and the details of the projector. We have also modified the manuscript to incorporate your suggestions. We hope these responses and revisions address your concerns, and we would really appreciate it if you would consider increasing your rating of our paper in light of these updates. We would be happy to engage in further discussion during the remaining rebuttal period to address any further queries.

Best, Authors

Reviewer_GSGtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to help small language models learn to solve problems they previously struggled with while preserving their existing capabilities. The authors propose projected distillation: the student first generates its own solution, and if the solution is incorrect, a teacher makes small edits to the student’s rollout, with the aim of making the corrected solution easier for the student to learn from. The corrections are refreshed as the student is updated. However, the default approach relies primarily on prompting the teacher to keep edits small, without strictly enforcing this constraint. The authors evaluate the method through comparative experiments and ablation studies.

Critical strengths

The motivation is clearly presented, and the comparative experiments and ablation studies are well designed.

Critical weaknesses
  1. Relying on prompts to obtain minimally edited corrections seems highly dependent on the teacher’s capabilities. This makes it difficult to anticipate how reliably the method will work across different domains and teacher models.

  2. Table 2 is not presented clearly enough. I found the metrics and their interpretation difficult to follow.

Questions for authors

How can the method ensure that the teacher makes only small edits? How can it ensure that the resulting corrections are easier for the student to learn from? Would directly prompting a strong teacher to “revise the solution into a reference answer that is easier for the student to learn from” work better than emphasizing minimal edits?

5 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

We thank the reviewer for recognizing the motivation and the design of our comparative experiments and ablation studies. We address the questions about minimal corrections and their learnability, followed by the clarification on Table 2.

Note: We have provided a revised manuscript in accordance with ICLR guidelines, with changes highlighted in purple.


Minimality of corrections (W1, Q1)

Minimality need not depend on prompting alone: we can check correctness, measure edit size, and use the student's likelihood to select among correction proposals. Our experiments demonstrate two complementary ways to obtain close, correct targets.

(1) A strong teacher (e.g. Opus) can directly produce corrections that preserve most of the student's attempt. Appendix C provides examples of minimal edits in math, and we have added a LiveCodeBench code-correction excerpt in Appendix C.3 showing targeted fixes while preserving the student's variable names and loop structure. There are prominent papers in the literature (e.g. CanItEdit [1] (COLM 2024)) that also provide evidence of strong models' ability to follow code-editing instructions. Optionally, we can gather kk teacher projections and select the verified-correct candidate most likely under the student, similar to what we do in RQ5 discussed below.

(2) For mid-capacity teachers (e.g. Qwen3-32B), RQ5 (Section 4.5) demonstrates how iterative refinement improves projection quality. We re-prompt the teacher with its previous correction and the spans it edited. Across rounds, we retain the valid correction with the highest student log-likelihood on the edited tokens. Combining iterative prompting with selection under the student provides projections that are easier to learn from, even with a low-capacity teacher. Empirically, two rounds raise the share of correctable failures from 9.2%9.2\% to 16.7%16.7\% and reduce the mean edit per correction from 256256 to 130130 tokens. Hard-set pass@1 rises from 0.370.37 to 0.440.44, with retention between 0.910.91 and 0.930.93.

Q. Would directly prompting for "easier to learn from" help?

We thank the reviewer for raising this important clarifying question. Asking the teacher directly for an “easier revision to learn from” is an open-loop instruction: the teacher receives no counterfactual feedback about which revisions actually improve student learning, unless we actually train the model across different revisions and preserve the most improved branch of training.

We instead build our projection approach on the well-established principle of providing a corrective learning signal on-policy. Many important prior works (e.g. DAgger [2], On-policy (self-)distillation [3, 4]) have already established the benefits of such learning. Our idea is to preserve the parts of the model's response that are benign to the incorrectness and rather target the editable parts. This allows for a near-on-policy, correct signal that can be traced on a continuum of edits to the most off-policy targets.

Concurrent work [5], first submitted to arXiv on 1 October 2026 after our ICLR submission, also seeks targets that are easier for the student to learn from. It uses repetitive sampling with a reference target through MCMC to construct correct responses closer to the student’s distribution. Our teacher-generated minimal corrections achieve a math average of 0.545 versus 0.534 with approximately 30× fewer estimated student FLOPs; our full objective reaches 0.557 with approximately 16× fewer student FLOPs (Appendix D.4). This provides further evidence for using teacher provided close, correct targets for learning.

AuthorsOpenReview ↗

Response by Authors (2/2)

Clarity of Table 2 (W2)

Thanks for pointing out the clarity issue on the metrics used in Table 2. We had to keep the description in the submitted draft minimal and shift detailed formal definitions to Appendix E.1.

In the revised manuscript, we have added these explanations to Section 4.1 and the Table 2 caption, with clearer column labels. The caption also makes the normalization and retention thresholds explicit; formal definitions remain in Appendix E.1. We provide a summary below as well:

  • xˉ\bar{x} denotes the hard-set accuracy averaged over training.
  • yˉnorm\bar{y}_{\mathrm{norm}} denotes the held-out accuracy averaged over training, relative to the untrained model; 1 means matching its original accuracy.
  • BWT defines the final change in held-out accuracy relative to the untrained model; negative values indicate a decline.
  • FF denotes the drop from the highest held-out accuracy reached during training to the accuracy at the end of training. Smaller FF is better.
  • SafeSpec_{2%} / SafeSpec_{5%} define the highest hard-set accuracy achieved while preserving at least 98%98\% / 95%95\% of the untrained model's held-out accuracy.

We hope the additional evidence and clarifications address the reviewer’s concerns about minimal corrections and their learnability. We would appreciate the reviewer reconsidering their rating in light of these responses, and we welcome further discussion.

References

[1] Cassano, F. et al. (2024) ‘Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions’, COLM.

[2] Ross, S. et al. (2011) ‘A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning’, AISTATS.

[3] Agarwal, R. et al. (2024) ‘On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes’, ICLR.

[4] Zhao, S. et al. (2026) ‘Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models’, arXiv:2601.18734.

[5] Karan, A. et al. (2026) ‘Finetuning with Sampling: SFT Learns Better Than You Think’, arXiv:2610.02140.

AuthorsOpenReview ↗

Requesting the reviewer's thoughts on our rebuttal

Dear reviewer GSGt,

Thank you once again for your review. In our rebuttal response, we have clarified the efficacy of teacher-prompted minimal corrections and why directly asking a teacher LLM to provide easy-to-learn projections is hard. We have also simplified the description of the metrics in Table 2. In addition, we have updated the manuscript to incorporate these discussions. We hope our response answers your queries, and we would appreciate it if you would consider increasing your rating of our work given these updates. We would be glad to engage in further discussion during the remaining time.

Best, Authors

Reviewer_k4ZPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training on problems that a student repeatedly fails to solve. A teacher edits the student's failed attempts into correct responses while preserving as much of the original text as possible. The student learns from these pairs through DPO restricted to the changed spans. An auxiliary SFT loss reinforces its own correct responses. Corrections are refreshed as training proceeds. Experiments with Qwen3-8B and OLMo-2-7B examine the tradeoff between fitting difficult problems and retaining existing performance across five benchmarks. The paper also derives a regret bound that depends on edit size under an idealized projection model.

Critical strengths

The method addresses a concrete weakness of training on difficult problems. Outcome rewards provide little signal when attempts consistently fail. A minimally corrected attempt supplies a usable target while retaining much of the student's original response. The paper builds on closely related work, including SPOT, and evaluates against it directly. Its contribution is clearest in the combination of refreshed corrections, a localized preference loss, and the study of edit size.

The experiments examine several parts of the method rather than relying on one aggregate result. They vary correction size, online versus static targets, the training objective, and projector refinement. The retention measurements are useful, and Appendix E.3 extends the evaluation beyond the training domain. Table 1 also makes the exceptions visible. The method does not win on every task, and the discussion of those cases is helpful.

Critical weaknesses

The main specialization metric appears to measure performance on the same hard problems used for training. Sections 4.1 and D.2 define a fixed mined hard set and evaluate specialization on that set. This demonstrates that the model can learn the corrected examples. It leaves open how well the method transfers to unseen problems that the base model also cannot solve. The separate retention split does not answer this question because it is not matched for difficulty. Appendix E.3 provides useful held-out results for math-trained Qwen3-8B, but the central hard-problem claim needs a clearer generalization test. Please reserve an unseen hard subset and report its performance separately from training-set specialization.

The edit-size ablation does not fully isolate proximity to the student's policy. Larger edits also change the number of tokens in the DPO objective. Appendix D.4 states that these log-probabilities are summed without token normalization. Keeping the learning rate and beta fixed therefore does not keep the loss scale or update magnitude fixed. The results could partly reflect this optimization difference. A normalized-loss or matched-update comparison would help. Appendix F.3 also says that unchanged tokens cancel because their text is identical. After an earlier edit, their conditioning prefixes differ, so this cancellation is not exact. The reported correlation is useful empirical evidence, but the explanation should acknowledge that distinction.

The black-box projector is not identified precisely enough to reproduce a central set of results. Appendix D.4 describes it only as a frontier model accessed through an API. Please give its model name and version. Matching student rollouts and update steps also leaves the cost of online correction unclear. Reporting teacher calls and generated tokens would make the comparison with static correction methods easier to assess.

There are a few presentation issues. The abstract reports the best result on 8 of 10 pairs, while Table 1 reports 8 of 9 for the same-family teacher and 7 of 9 for the black-box teacher. These should agree. The abstract should also describe the theory as an upper bound under stated assumptions. An upper bound that grows with edit size does not establish that actual regret must grow at that rate. Figure 3's axis labels and legend could be enlarged.

Questions for authors
  1. Are any problems held out from the mined hard set when computing specialization? If not, how does the method perform on unseen problems selected by the same difficulty criterion?
  2. Does the advantage of minimal edits persist with token-normalized DPO or comparable update magnitudes across edit-size conditions?
  3. Which model and version provide the black-box corrections? How many teacher calls and tokens does each method consume over a complete run?
3 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

We thank the reviewer for acknowledging the importance of our work in addressing a major weakness of currently popular post-training paradigms in the hard regime, for highlighting the clarity of our contributions with respect to the literature, for appreciating the comprehensiveness of our analysis across multiple domains and with diverse metrics, and for finding our work very informative.

In our response below, we address (1) generalization to unseen hard problems, (2) the length-normalized DPO variation and prefix drift, (3) projector usage details, and (4) the stated presentation concerns about our empirical and theoretical results.

Note: We have provided a revised manuscript in accordance with ICLR guidelines, with changes highlighted in purple.


Generalization to unseen hard problems (W1, Q1)

We thank the reviewer for providing this helpful experiment suggestion. We have run additional experiments in which we evaluate the model on a separate set of unseen hard problems.

Regarding specialization on unseen problems, we have added a separate evaluation of generalization on hard problems held out from training, as the reviewer suggested.

In the new RQ1 experiment with Qwen3-8B, we split the mined MATH hard set (pass@16 = 0) into 501 training problems and 127 held-out problems. We also evaluate on 2,885 hard problems (pass@16 = 0) from OlympiadBench, Omni-MATH, AIME 1983–2023 and DAPO-Math, used only for evaluation after removing any problems that duplicate those in the training MATH dataset. In the tables below, we show gains in performance relative to the base Qwen3-8B model. We picked the best checkpoint of each baseline for reporting these numbers.

Unseen hard MATH (127 problems).

MethodΔ\Delta pass@1 ↑Δ\Delta pass@16 ↑
GRPO+0.006+0.055
SPOT+0.007+0.047
SODA+0.018+0.142
Ours+0.043+0.205

Unseen hard problems from other benchmarks: OlympiadBench (150), Omni-MATH (2,178), AIME 1983–2023 (199), DAPO-Math (358). (Individual columns report Δ\Delta pass@1; pooled columns aggregate all 2,885 problems.)

MethodOlympiadB. ↑Omni-MATH ↑AIME ↑DAPO-Math ↑Pooled @1 ↑Pooled @16 ↑
GRPO+0.005-0.001-0.002+0.005-0.001-0.006
SPOT+0.002+0.003-0.005+0.001+0.0010.000
SODA+0.003+0.006-0.007+0.008+0.005+0.013
Ours+0.006+0.009-0.003+0.016+0.009+0.040

Our method achieves the largest gains on held-out hard MATH, the pooled other-benchmark hard set, and three of the four individual hard benchmarks, while marginally lagging behind GRPO on AIME. Therefore, our method does help the model generalize to hard problems beyond the training hard set. We have added these results to Section 4.1 and the full experimental setup and results to Appendix E.5 (Tables 13–14).

AuthorsOpenReview ↗

Response by Authors (2/2)

Effect of token-normalization (W2,Q2)

We thank the reviewer for bringing up this important nuance in optimization.

To isolate the effect of edits when gradient steps are of similar magnitude, we repeated the minimal-edit and larger-edit conditions using token-normalized full-sequence DPO, keeping the learning rate, β\beta, and other training settings unchanged. We trained Qwen3-8B on the GSM8K hard set for 250 updates, with one seed per condition, and measured the actual parameter updates. Both conditions have the same median update norm (0.0584); the results are below.

MetricMinimal editsLarger edits
Final hard-set accuracy0.2460.221
Final held-out accuracy0.9270.907
Hard-set accuracy averaged over training0.2110.154
Held-out accuracy averaged over training0.9250.903
Median parameter-update norm0.05840.0584

Minimal edits achieve higher hard-set accuracy at all 10 evaluation checkpoints, with comparable update magnitudes. Thus, the advantage persists even with token normalization. We have added the experimental setup and results to Appendix F.5 (Table 17), which also clarifies that the RQ2 ablation uses full-sequence DPO with self-SFT.

W2, Q2 (token-drift clarification)

The reviewer is right in pointing out that the exact cancellation applies before the first edit and later shared tokens can differ because their conditioning prefixes change. In Appendix F.3, we isolate where the likelihood difference between the incorrect and correct answers lies. We show that across 2,249 steps, localized and full-response likelihood shifts correlate at r=0.9999r=0.9999, with localization retaining 94–95% of the magnitude. Thus, the edited spans capture almost all of the observed shift despite downstream prefix drift. This motivates the use of the localized-DPO objective. We have revised Appendix F.3 to explicitly distinguish exact cancellation before the first edit from downstream prefix drift.


Projector identity and cost analysis (W3, Q3)

Thanks for raising this point. For our empirical analysis, we use Claude Opus 4.8 (claude-opus-4-8) as the black-box projector.

Across the five Qwen3-8B benchmarks, our 250-update runs average approximately 3,010 teacher requests. For a single-candidate static pool, SPOT/SODA’s initial construction budget averages up to 737 requests before verification retries. We estimate an average of approximately 7.8M teacher tokens (input + output) per training run. We have added the projector configuration, per-benchmark request counts and correction rates to Appendix D.4 (Table 6).

We also want to highlight that although black-box projectors demonstrate effectiveness in providing high-quality learning targets, they are not essential. In our experiments (please see RQ5), we find that we can use even a 32B teacher with appropriate test-time scaling to close the gap with the black-box generator.

Most importantly, we argue that online refreshing pays off in effective continual learning, unlike static approaches that overfit to a dataset and possibly lead to catastrophic forgetting of pre-trained skills. Recovering those skills may incur much higher GPU training costs than the additional cost of querying a black-box model. The cost of querying capable models is also declining fast across recent model generations [1, 2].


Presentation issues around empirical and theoretical claims (W4)

(1) Thanks for pointing out the imprecision in the "8 out of 10" claim. We have corrected the abstract to state "best on 8 of 9 settings with the same-family teacher and 7 of 9 with the black-box projector."

(2) Regarding the theoretical regret, we have revised the abstract and introduction to explicitly scope the claim to the smoothness and recoverability assumptions. We have also clarified in Section 5 that the horizon dependence concerns a worst-case regret upper bound under the stated assumptions, with the remaining bound parameters held fixed.

(3) We have enlarged Figure 3 to improve legibility and increased the font size in Table 1.


We thank the reviewer again for the insightful review, and we hope these additional results and clarifications address the reviewer's questions. Please let us know if anything remains unclear; we would be happy to clarify further.

References

[1] Anthropic (2026) ‘Introducing Claude Opus 5.5’.

[2] OpenAI (2026) ‘Introducing GPT-6.1 Terra’.

AuthorsOpenReview ↗

Requesting reviewer's thoughts on our rebuttal

Dear reviewer k4ZP,

Thank you for your thoughtful review. Our rebuttal provides generalization results on unseen hard problems, results on token-normalized DPO and additional details on the projector. It also addresses some of the presentation issues you pointed out. We have incorporated these items into the manuscript, too. Please let us know if you have any additional thoughts on our rebuttal. We would like to engage in further discussion during the remaining time.

Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Projected Distillation: Requesting comments from the reviewers given18th Nov deadline for discussion

Dear Reviewers,

Thank you again for your valuable time and feedback. As the rebuttal period ends on EOD 18 November, 2026, we would appreciate it if you could take a look at our responses, additional results and revised manuscript, and reconsider your ratings in light of these updates. We would be available to discuss any outstanding concerns or provide further clarification during the remaining time.

Best,

Authors of "Projected Distillation"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.