acceptodds
Under review as a conference paper at ICLR 2027

BREAD: Transfer-Aware On-Policy Distillation from Boundary Experience

Abstract

Repeated sampling reveals correct programs for tasks that a language model fails to solve greedily. These boundary tasks contrast failed decisions with verified repairs, providing experience for cross-task learning. We introduce BREAD (Boundary Repair Experience for Adaptive Distillation) to connect repair experience with cross-task update utility. BREAD extracts textual repair cards from boundary tasks and uses them to condition a frozen teacher on task-only student rollouts. Temporary updates measure changes in correct-program likelihood across source clusters, and cluster-level utility scores route experience to training receivers. During distillation, candidate AdamW updates start from a shared snapshot and are selected by correctness-probe improvement under retention and displacement constraints. The resulting student generates programs from task-only inputs. Across four model families and four programming benchmarks, BREAD achieves average gains of 7.88 percentage points in macro-average greedy accuracy over the initial models and 1.82 points over the strongest baseline selected separately for each model. Paired analysis separates newly solved tasks from lost solutions, while component and carrier diagnostics examine how repair extraction, utility routing, and update selection contribute to reusable task-only student behavior.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-Policy Distillation, Cross-Task Experience Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T5EaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BREAD, a transfer-aware on-policy distillation framework for improving LLM code generation. The method first extracts reusable repair experiences from tasks where greedy decoding fails but repeated sampling discovers a correct solution. These experiences are then routed to training tasks based on cross-task utility, measured through the change in reference-program negative log-likelihood (NLL) after temporary student updates. Finally, BREAD performs experience-conditioned reverse-KL distillation and selects AdamW updates based on correctness-probe improvement, retention constraints, and bounded parameter displacement.

Critical strengths
  1. The paper distinguishes between improvements in an experience-conditioned teacher and improvements actually transferred to the student. This is an important consideration for context-conditioned on-policy distillation.

  2. The paper evaluates four model families and four programming benchmarks, with component ablations and analyses of acquired and lost solutions.

Critical weaknesses
  1. Reference-program NLL is an imperfect proxy for actual transfer, and its limitations require further investigation. The proposed routing and update-selection procedures rely heavily on reductions in the NLL of verified reference programs. However, increasing the likelihood of a particular correct program does not necessarily improve the probability of generating a correct solution. Programming tasks often admit multiple valid solutions, and reference-program likelihood may not reliably capture changes in the model's broader solution distribution.

  2. BREAD introduces considerable additional computation beyond conventional on-policy distillation, and the computational overhead is insufficiently evaluated.

  3. The main experiments use only 32 training iterations, which is unusually short for LLM post-training. The paper does not sufficiently justify this choice or investigate longer training schedules. The paper should report training curves and compare BREAD against relevant OPD baselines over longer schedules.

  4. The proposed framework is overly complex, and the necessity of its components is not sufficiently established. For example, although the paper includes ablations on experience sources and positive-only extraction, these experiments do not isolate the necessity of explicitly extracting textual repair cards.

  5. Figure 1 is overly dense, with numerous small labels, equations, and interconnected components. The overall pipeline is difficult to follow without repeatedly consulting the main text. A simpler overview focusing on the three principal stages would substantially improve readability.

Questions for authors

Please check the weakness above.

Reviewer_rSRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

BREAD extracts repair cards from programming tasks on which greedy generation fails but a sampled solution passes. It routes a card according to the effect of a temporary student update on other training tasks, then commits an update only if correctness probes improve while retention loss and parameter displacement remain bounded. The resulting student uses task-only prompts at inference. Across four model families and four coding benchmarks, BREAD reports a 7.88-point average gain over the initial models and a 1.82-point gain over each model's strongest baseline, leading in 13 of 16 model–benchmark settings.

Critical strengths
  1. The learner-side utility criterion is a substantive contribution. It evaluates whether experience improves the student on other tasks, rather than treating teacher-side contextual benefit as evidence of transfer. The carrier diagnostic directly shows that these two criteria can rank experiences differently.
  2. The empirical case is broader than a single favorable benchmark: four backbones, four test sets, three training seeds for the main results, gained-and-lost solve analysis, and targeted component ablations. The shared-context EOPD control and independent execution diagnostic make the proposed mechanism more credible.
Critical weaknesses
  1. The strongest-baseline advantage is not examined at the level used to interpret transfer. The 1.82-point average lead is measured against each backbone's strongest baseline, but the gained-and-lost solve analysis compares BREAD only with the initial model and OPCD. It therefore does not show whether BREAD acquires more solutions without sacrificing others relative to the competitive methods it actually needs to beat. The macro average also gives CodeContests, with 149 eligible tasks, the same weight as much larger test sets. The benchmark-level scores are reported, but the paired composition and task-weighting sensitivity of the incremental gain remain unclear.
  2. The training cost of that margin is unknown. BREAD evaluates candidate updates with optimizer trials and correctness and retention probes before deciding which state to keep. The training-rule controls share the experience bank, router, and probes, but the authors explicitly state that this diagnostic has neither a common compute cap nor end-to-end timing. Consequently, the paper establishes an accuracy gain under its training protocol, but not the amount of additional computation required relative to its closest controls or the resulting accuracy–cost tradeoff.
Questions for authors
  1. Using the existing task-level predictions, what are BREAD's gained and lost solves relative to each backbone's strongest baseline, broken out by benchmark? How does its overall advantage change if the 6,916 evaluation tasks are pooled rather than giving each benchmark equal weight?
  2. From existing training records, how many candidate optimizer updates, teacher forward passes, and probe evaluations does BREAD use relative to the shared-context EOPD control, and what are the corresponding GPU hours or wall-clock times? How much extra computation accompanies the reported accuracy gain?
1 comment

Thank you for the detailed reply and additional experiment results. The new analyses are helpful, but I still have two questions:

  • The rebuttal results reports lower GPU time than EOPD but performing more selector trials and probe evaluations. Could the authors explain what accounts for this efficiency gain?
  • The reply clarifies the net accuracy gains, but I am still unsure about solution retention. Do these gains come with substantial losses on previously solved tasks compared with the strongest baselines?
Reviewer_uC8XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes BREAD, a transfer-aware on-policy distillation framework that aims to improve code generation through reusable repair experiences. The method identifies boundary tasks where greedy decoding fails but repeated sampling produces verified correct programs. It then extracts repair cards from failed and successful programs to guide a frozen teacher in supervising student-generated trajectories. To identify transferable experiences, BREAD measures cross-task changes in correct-program negative log-likelihood (NLL) after temporary student updates and uses these measurements for experience routing. During distillation, the method further selects candidate AdamW updates based on correctness-probe improvements under retention and parameter displacement constraints.

Critical strengths
  • The paper addresses an important problem of transferring task-specific repair experiences into reusable student capabilities. The proposed framework combines experience extraction, cross-task utility routing, and retention-constrained updates, with a meaningful distinction between teacher benefits and student learning.

  • Experiments across four model families and four programming benchmarks demonstrate consistent improvements over existing baselines. The ablation studies and additional analyses provide evidence for the effectiveness of the proposed components.

Critical weaknesses
  • BREAD introduces additional computational overhead through temporary OPD updates for experience routing and repeated AdamW trials for update selection. However, the paper does not report end-to-end training costs or provide compute-matched comparisons with baselines. Given the average improvement of only 1.82 percentage points over the strongest baselines, it is unclear whether the performance gains justify the additional computational cost.

  • The paper does not clearly explain how training data are partitioned into source, receiver, and probe sets, including their selection criteria and sizes. More importantly, the update selection mechanism relies on very small probe pools. As shown in Table 5, both Qwen and Mistral use only 8 correctness and 8 retention probes, with Qwen evaluating just 4 of each per iteration. It remains unclear how these probes are selected and whether they adequately represent the broader task distribution. Repeatedly selecting updates based on such limited probes may lead to probe overfitting and unreliable generalization. I suggest reporting the detailed data partitioning procedure and conducting sensitivity experiments with different probe sizes and compositions.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A Brief Note Before the Discussion Closes

Dear Area Chair and Reviewers,

Thank you for taking the time to review our work and consider our rebuttal. We appreciate the follow-up questions, which gave us a chance to clarify our findings.

As the discussion period draws to a close, we would welcome any further feedback and would be happy to clarify any remaining concerns. We would also be grateful for the Area Chair’s guidance if any points would benefit from further discussion.

Thank you for considering the additional results and clarifications in our rebuttal as part of your assessment.

Best regards,

The Authors

Thank you, rSRs and uC8X, for getting back to us. We’re glad the additional results and clarifications helped address your concerns, and we appreciate your feedback on the paper’s soundness.

If the discussion has also changed how you see the paper overall, we’d appreciate your considering whether that change is reflected in your overall evaluation. Thank you again for taking the time to discuss our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.