- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BREAD, a transfer-aware on-policy distillation framework for improving LLM code generation. The method first extracts reusable repair experiences from tasks where greedy decoding fails but repeated sampling discovers a correct solution. These experiences are then routed to training tasks based on cross-task utility, measured through the change in reference-program negative log-likelihood (NLL) after temporary student updates. Finally, BREAD performs experience-conditioned reverse-KL distillation and selects AdamW updates based on correctness-probe improvement, retention constraints, and bounded parameter displacement.
Critical strengths
-
The paper distinguishes between improvements in an experience-conditioned teacher and improvements actually transferred to the student. This is an important consideration for context-conditioned on-policy distillation.
-
The paper evaluates four model families and four programming benchmarks, with component ablations and analyses of acquired and lost solutions.
Critical weaknesses
-
Reference-program NLL is an imperfect proxy for actual transfer, and its limitations require further investigation. The proposed routing and update-selection procedures rely heavily on reductions in the NLL of verified reference programs. However, increasing the likelihood of a particular correct program does not necessarily improve the probability of generating a correct solution. Programming tasks often admit multiple valid solutions, and reference-program likelihood may not reliably capture changes in the model's broader solution distribution.
-
BREAD introduces considerable additional computation beyond conventional on-policy distillation, and the computational overhead is insufficiently evaluated.
-
The main experiments use only 32 training iterations, which is unusually short for LLM post-training. The paper does not sufficiently justify this choice or investigate longer training schedules. The paper should report training curves and compare BREAD against relevant OPD baselines over longer schedules.
-
The proposed framework is overly complex, and the necessity of its components is not sufficiently established. For example, although the paper includes ablations on experience sources and positive-only extraction, these experiments do not isolate the necessity of explicitly extracting textual repair cards.
-
Figure 1 is overly dense, with numerous small labels, equations, and interconnected components. The overall pipeline is difficult to follow without repeatedly consulting the main text. A simpler overview focusing on the three principal stages would substantially improve readability.
Questions for authors
Please check the weakness above.
Thank you for the detailed reply and additional experiment results. The new analyses are helpful, but I still have two questions: