- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes alternating between learning in language and learning in weights. The model first writes natural language insights from contrasts between its failed and successful attempts (CORE), and then distills them into its weights by finetuning on solutions generated with the insights, using prompts without them. Across six reasoning tasks and two models, the distilled model retains nearly all of the in-context gain, outperforms rejection sampled finetuning and GRPO at matched rollout budgets in most cases, and improves further with additional rounds. Because the insights are readable, the authors also detect and remove a reward hack before it reaches the weights.
Critical strengths
- The paper proposes a very interesting and feasible solution for distillation, which leverages the strength of both language and weight updates.
- The experiments are comprehensive and well-controlled, which clear ablations to isolate different baselines and performance trend alone the training.
- The experiments on large models (27B and 120B) demonstrates superior performance over various baselines (near-complete retention of in-context gains and more efficient than GRPO).
Critical weaknesses
- The insight generation process is heuristic and not justified in this setting. How insights are produced (reflecting only on failures, contrasting with a correct attempt, the admission test, utility scoring, keeping a fixed top-25 set) is intuitive but neither explained nor ablated here. It's supported only by end-to-end gains and cast doubt on whether the major gains come from this specific design. Moreover, since the same 25 insights are placed in every prompt, the bank effectively acts as a fixed prompt. Ablations of these components and a comparison with simpler prompt-optimization baselines followed by distillation (e.g., GEPA) are needed to show that CORE’s specific mechanism matters.
- Limited novelty. CORE is an existing method, and insight distillation is essentially context distillation combined with rejection-sampling finetuning. Closely related methods that also alternate learning in language and in weights (e.g., Learning Fast and Slow, Online Experiential Learning, BetterTogether) are discussed but not compared empirically.
- No out-of-distribution evaluation. Training and held-out problems come from the same task distribution, and the same 25 task-specific insights are used throughout. Whether the insights generalize to harder or structurally different problems remains untested.
- The comparison with GRPO and the evaluation scale are limited. With only about 100 training problems per task, the setting favors in-context methods over RL, and the Qwen GRPO baseline uses a modified positive-only variant that learns more slowly. Combined with small held-out sets (about 100 problems) of mostly puzzle-like tasks, claims of much greater sample efficiency than conventional learning should be restricted to this low-data regime.
Questions for authors
please refer to the weaknesses
Dear reviewer,
Thank you for the thoughtful and detailed review! We appreciate your assessment that "The paper proposes a very interesting and feasible solution for distillation, which leverages the strength of both language and weight updates." From our general response above, the most relevant are E1, E2 and E4. Below, we address each of your points.
W1: Insight generation, the fixed prompt, and prompt optimization
Fixed prompt (Section 3.1). We agree: in the capability experiments, retrieval uses the whole training split, so the same 25 insights appear in every prompt and the bank acts as a fixed prompt.
Comparison with prompt optimization. In the CORE paper (Nasvytis et al., 2026), CORE outperformed GEPA in every condition on the two tasks that we reuse here (MathGAP and Matchstick). For this reason, we had not included GEPA as a comparison. However, beyond accuracy, the revision shows a property that a single optimized prompt does not have: Because CORE's insights are separate and individually tested, the model can inspect and remove them individually while retaining the others. We show the benefits of this approach in removing a specific skill (E1) or a hack (E2) before distillation.
Does the content of CORE's insights matter? Following your suggestion, we added a new experiment, whose results suggest that generating training solutions with the model's own insights for an unrelated task gives almost no gain after distillation (E4; Appendix P). This suggests that the gain depends on the content of the insights.
CORE's components. We agree with you, though CORE's mechanism (contrastive reflection, admission test, utility scoring) were ablated in the CORE paper, which is why we chose to not repeat them here. In this paper, we add a new component ablation for the CORE Alignment (Appendix O) method, showing that only the full method, with both the judge and the learned principles, keeps hacking at or below 0.6% in all four cells.
W2: Novelty and other alternating methods
We agree that alternating between language and weights is not new on its own, and the revised paper does not present that as the contribution. Our method’s contribution is now centered around providing more auditability and control over what is consolidated into the model: because the insights are separate, tested pieces of text, the model can remove specific skills (E1) or hacks (E2) before distillation, and learn its own alignment principles (E3), with the effect measured in the weights. To our knowledge, none of these methods removes individual lessons before distillation and measures the effect.
W3: Out-of-distribution evaluation
We agree this is a great idea! In our setting, within each task we have no harder split. However, across tasks, each distilled model is evaluated on every other task (Appendix L): the majority of evaluations are within 3 pp. The revision also adds GPQA Diamond, which none of the models was trained on: the seven gpt-oss-120b distilled models score 72.4–76.3%, against 73.6% for the base model. Taken together, these results suggest that distillation does not hurt other tasks, but we do not claim that the insights transfer to structurally different problems.
W4: GRPO comparison and scale
We agree, and the revision now states this in the limitations (Section 5): our efficiency comparisons are made with about 100 training problems per task, and whether the advantage over RL holds with more training data remains open. On the positive-only variant: we agree with you, though it was the only one of nine changes that trained stably on Qwen at low reasoning effort (Appendix C). We tested nine other standard remedies, which include: lower learning rates, adding KL penalty, larger batches, DAPO and Dr. GRPO, and positive-only advantages. Only the last remedy trained stably. For gpt-oss-120b, standard GRPO led to stable training, and there, insight distillation still outperforms GRPO at matched rollouts on 5 of 6 tasks. This suggests the result does not depend on the GRPO variant itself.
We thank you again for such helpful feedback – we believe that the revised paper has been significantly improved by addressing the weaknesses you have outlined!