acceptodds
Under review as a conference paper at ICLR 2027

Learning in Language and Weights: When the Two Reinforce Each Other

Abstract

Language models are unique in that they can learn both from context and weight optimization, each with complementary strengths. Here, we study a method for combining them: the sample efficiency and interpretability of learning from natural language, and the permanence of weights learning. We show that CORE, a nonparametric learning algorithm, can produce natural language insights that can be directly distilled back into weights via insight distillation. Across six reasoning tasks and two models, we find that our method is more sample efficient and outperforms rejection-sampled fine-tuning on all 12 pairs, and tuned GRPO baselines on 10/12 pairs, and uses the distilled insights while solving problems. A further round of CORE learning improves on the previous round of insight distillation in 11/12 pairs, demonstrating a mutually reinforcing relationship between language and weights learning. Insight distillation is more interpretable than in-weights learning alone, and allows direct control on what a model learns: we are able to easily detect a reward hacking strategy occurring in the wild in a coding task, and remove the exploiting insights before distillation to prevent the behavior from forming. Finally, we show signs of life with an autonomous loop that can schedule repeated rounds of insight distillation for further learning efficiency. Overall, our results show that with powerful enough language models, learning in natural language and with weights can reinforce each other.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning, complementary learning systems, self-improvement, in-context learning, supervised fine-tuning, knowledge consolidation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_STPnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes alternating between learning in language and learning in weights. The model first writes natural language insights from contrasts between its failed and successful attempts (CORE), and then distills them into its weights by finetuning on solutions generated with the insights, using prompts without them. Across six reasoning tasks and two models, the distilled model retains nearly all of the in-context gain, outperforms rejection sampled finetuning and GRPO at matched rollout budgets in most cases, and improves further with additional rounds. Because the insights are readable, the authors also detect and remove a reward hack before it reaches the weights.

Critical strengths
  1. The paper proposes a very interesting and feasible solution for distillation, which leverages the strength of both language and weight updates.
  2. The experiments are comprehensive and well-controlled, which clear ablations to isolate different baselines and performance trend alone the training.
  3. The experiments on large models (27B and 120B) demonstrates superior performance over various baselines (near-complete retention of in-context gains and more efficient than GRPO).
Critical weaknesses
  1. The insight generation process is heuristic and not justified in this setting. How insights are produced (reflecting only on failures, contrasting with a correct attempt, the admission test, utility scoring, keeping a fixed top-25 set) is intuitive but neither explained nor ablated here. It's supported only by end-to-end gains and cast doubt on whether the major gains come from this specific design. Moreover, since the same 25 insights are placed in every prompt, the bank effectively acts as a fixed prompt. Ablations of these components and a comparison with simpler prompt-optimization baselines followed by distillation (e.g., GEPA) are needed to show that CORE’s specific mechanism matters.
  2. Limited novelty. CORE is an existing method, and insight distillation is essentially context distillation combined with rejection-sampling finetuning. Closely related methods that also alternate learning in language and in weights (e.g., Learning Fast and Slow, Online Experiential Learning, BetterTogether) are discussed but not compared empirically.
  3. No out-of-distribution evaluation. Training and held-out problems come from the same task distribution, and the same 25 task-specific insights are used throughout. Whether the insights generalize to harder or structurally different problems remains untested.
  4. The comparison with GRPO and the evaluation scale are limited. With only about 100 training problems per task, the setting favors in-context methods over RL, and the Qwen GRPO baseline uses a modified positive-only variant that learns more slowly. Combined with small held-out sets (about 100 problems) of mostly puzzle-like tasks, claims of much greater sample efficiency than conventional learning should be restricted to this low-data regime.
Questions for authors

please refer to the weaknesses

1 comment

Dear reviewer,

Thank you for the thoughtful and detailed review! We appreciate your assessment that "The paper proposes a very interesting and feasible solution for distillation, which leverages the strength of both language and weight updates." From our general response above, the most relevant are E1, E2 and E4. Below, we address each of your points.

W1: Insight generation, the fixed prompt, and prompt optimization

"...since the same 25 insights are placed in every prompt, the bank effectively acts as a fixed prompt."

"Ablations of these components and a comparison with simpler prompt-optimization baselines followed by distillation (e.g., GEPA) are needed to show that CORE’s specific mechanism matters."

  1. Fixed prompt (Section 3.1). We agree: in the capability experiments, retrieval uses the whole training split, so the same 25 insights appear in every prompt and the bank acts as a fixed prompt.

  2. Comparison with prompt optimization. In the CORE paper (Nasvytis et al., 2026), CORE outperformed GEPA in every condition on the two tasks that we reuse here (MathGAP and Matchstick). For this reason, we had not included GEPA as a comparison. However, beyond accuracy, the revision shows a property that a single optimized prompt does not have: Because CORE's insights are separate and individually tested, the model can inspect and remove them individually while retaining the others. We show the benefits of this approach in removing a specific skill (E1) or a hack (E2) before distillation.

  3. Does the content of CORE's insights matter? Following your suggestion, we added a new experiment, whose results suggest that generating training solutions with the model's own insights for an unrelated task gives almost no gain after distillation (E4; Appendix P). This suggests that the gain depends on the content of the insights.

  4. CORE's components. We agree with you, though CORE's mechanism (contrastive reflection, admission test, utility scoring) were ablated in the CORE paper, which is why we chose to not repeat them here. In this paper, we add a new component ablation for the CORE Alignment (Appendix O) method, showing that only the full method, with both the judge and the learned principles, keeps hacking at or below 0.6% in all four cells.

W2: Novelty and other alternating methods

"Closely related methods that also alternate learning in language and in weights (e.g., Learning Fast and Slow, Online Experiential Learning, BetterTogether) are discussed but not compared empirically."

We agree that alternating between language and weights is not new on its own, and the revised paper does not present that as the contribution. Our method’s contribution is now centered around providing more auditability and control over what is consolidated into the model: because the insights are separate, tested pieces of text, the model can remove specific skills (E1) or hacks (E2) before distillation, and learn its own alignment principles (E3), with the effect measured in the weights. To our knowledge, none of these methods removes individual lessons before distillation and measures the effect.

W3: Out-of-distribution evaluation

"Whether the insights generalize to harder or structurally different problems remains untested."

We agree this is a great idea! In our setting, within each task we have no harder split. However, across tasks, each distilled model is evaluated on every other task (Appendix L): the majority of evaluations are within 3 pp. The revision also adds GPQA Diamond, which none of the models was trained on: the seven gpt-oss-120b distilled models score 72.4–76.3%, against 73.6% for the base model. Taken together, these results suggest that distillation does not hurt other tasks, but we do not claim that the insights transfer to structurally different problems.

W4: GRPO comparison and scale

"...claims of much greater sample efficiency than conventional learning should be restricted to this low-data regime."

We agree, and the revision now states this in the limitations (Section 5): our efficiency comparisons are made with about 100 training problems per task, and whether the advantage over RL holds with more training data remains open. On the positive-only variant: we agree with you, though it was the only one of nine changes that trained stably on Qwen at low reasoning effort (Appendix C). We tested nine other standard remedies, which include: lower learning rates, adding KL penalty, larger batches, DAPO and Dr. GRPO, and positive-only advantages. Only the last remedy trained stably. For gpt-oss-120b, standard GRPO led to stable training, and there, insight distillation still outperforms GRPO at matched rollouts on 5 of 6 tasks. This suggests the result does not depend on the GRPO variant itself.

We thank you again for such helpful feedback – we believe that the revised paper has been significantly improved by addressing the weaknesses you have outlined!

Reviewer_7d2eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes insight distillation, a way to move learning done in natural language into model weights. The authors first build a bank of self-written insights (using a prior work named CORE) while the weights remain frozen. The model then solves training problems with those insights in its prompt, and it is fine-tuned on the correct solutions paired with the bare prompt. Across six reasoning tasks and two models, the distilled model recovers almost all of the in-context gain. It achieves relatively consistent improvements compared to rejection-sampled SFT and a GRPO baseline at matched rollouts. The authors also show that the insight bank can be read and edited: on a benchmark, they find a reward hack written down as an insight and remove it before it reaches the weights. Finally, they present a simple autonomous loop that decides when to distill and when to stop.

Critical strengths

1- The experiments are careful and honest. There is a large GRPO tuning effort, rollout and token accounting, and good controlled experiments and ablation studies are conducted. The paper reports exceptions and failures clearly.

2- The CrossPL case study (where the authors monitored the extracted insights and found reward hacking) is the most interesting part of the paper to me. Finding a bias or harmful insight inside the insight bank and editing it out before distillation is a concrete example of auditable learning. I haven’t seen this done in related work, and it matters for oversight.

3- The harvest-gap predictor is useful in practice. Knowing in advance whether distillation will pay off, based on how much the insights raise the training solve rate, is useful.

4- The paper is clearly written and organized around well-posed questions.

Critical weaknesses

1- Novelty and positioning.\textbf{Novelty and positioning.} Having a model write lessons from its own attempts and then distill them into weights with the lessons removed from the student’s context has been explored extensively this year. Procedural Memory Distillation [1] seems to be very close (it is a concurrent work accepted at NeurIPS 2026). Also [3] already shows that self-extracted experience can be distilled and iterated over multiple rounds. Both are cited, but the paper never explains how its method differs from them. Several closely related works are not cited at all: Experience Distillation [4], LEAFE [5], and OPSD [6]. So, it seems to me that the main claims that in-context gains survive distillation, that repeated rounds help, and that this approach beats GRPO in sample efficiency are already known in general. The paper adds stronger evidence, not a new idea. The contribution needs to be sharpened.

2- The method uses off-policy rejection-sampled SFT, while the closest prior works use on-policy KL distillation [1, 3], and [3] reports that on-policy distillation works better. Without an on-policy distillation baseline using the same CORE bank, it is hard to tell how much of the gain comes from CORE’s insights and how much comes from the training objective.

3- R-SFT gets fewer rollouts than insight distillation (1.6k vs. 2.8–4k) and fewer accepted samples with the same number of updates. The comparison therefore mixes better data with more data.

4- On Qwen, GRPO trains only with positive-only advantages, which is close to online rejection-sampling fine-tuning, and it is still unstable.

5- The paper’s own Appendix D shows that a second CORE round adds little over the first-round bank on distilled weights. So, “mutually reinforcing” mostly reflects the benefit of consolidation.

6- “What the model learns is written down before it enters the weights” is too strong. The training targets are full reasoning traces, so unstated behaviors or biases could also get distilled.

7- As shown, detecting and removing reward hacking required understanding how the exploit worked, so claims implying that the method provides a way to easily detect such behaviors should be softened.

[1] Y. Liu, et al. Procedural Memory Distillation: Online Reflection for Self-Improving Language Models. NeurIPS 2026.

[2] T. Ye, et al. Online Experiential Learning for Language Models. 2026

[3] T. Ye, et al. On-Policy Context Distillation for Language Models. 2026

[4] C. Gou, et al. Sample-Efficient Learning from Agent Experience. 2026.

[5] R. Ge, et al. Internalizing Agency from Reflective Experience. ICML 2026.

[6] Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. 2026

Questions for authors

1- How does insight distillation differ, conceptually and empirically, from [1] and the other methods mentioned in the Weaknesses section? Could you add an on-policy self-distillation baseline (as in [3]) that uses the same CORE bank? I think this experiment would improve the paper the most.

2- I also suggest an experiment to make the role of distillation more pronounced. For instance, distill from a bank with shuffled or irrelevant insights, or from a prompt optimized by GEPA. This would show that the content of CORE’s insights is what drives the gains, rather than just any prompt that raises the training solve rate.

3- Several “X of 12” wins come from differences within one standard error. Win counts based on paired bootstrap intervals over problems would be more convincing.

4- The strongest selling point of the paper seems to me to be auditability. I’d suggest reframing the contributions around auditing the bank, the harvest predictor, and scheduling, and strengthening the auditing part beyond one example. For instance, delete specific insights on other tasks and check whether the corresponding behavior disappears from the distilled model.

5- Have you tried distilling several tasks’ banks into one adapter? That would support the permanence and continual-learning motivation much better than having one LoRA per task.

2 comments

Dear reviewer,

Thank you for such a thorough and helpful review! We appreciate your assessment that the "The experiments are careful and honest" and your suggestion that "The strongest selling point of the paper seems to me to be auditability". In addition to addressing the weakness you have outlined, the revision also follows your suggestion: we have added new results centered around the model inspecting and controlling what it consolidates into its weights. Our general response above summarizes the new experiments, and all four (E1–E4) points there should be also considered for your review.

W1 / Q1: Novelty and relation to prior work

"...the main claims that in-context gains survive distillation, that repeated rounds help, and that this approach beats GRPO in sample efficiency are already known in general."

"How does insight distillation differ, conceptually and empirically, from [1] and the other methods mentioned in the Weaknesses section?"

We agree that these three findings are not new on their own, and we did not intend to present them as such. However, the revision does put more emphasis for what is new on one of the points made in your review: our method providing more auditability and control over what is consolidated. Because CORE's insights are separate, individually tested pieces of text, the model can inspect and edit what it is about to distill into its own weights, and we measure the effect in the weights. We now showcase these benefits in the form of selective skill editing (E1), self-auditing and hack editing (E2), and CORE Alignment, in which the model writes its own principles against reward hacking (E3). To our knowledge, none of the works you cite removes or edits individual lessons before distillation and measures the effect, and none studies the context of reward hacking. As an example, PMD curates memory automatically and ablates whole memory levels; OEL does not select among items, and OPCD selects whole contexts. We now cite all of the works you mentioned, and discuss them in Section 2.

W2 / Q1: On-policy distillation baseline

"Could you add an on-policy self-distillation baseline (as in [3]) that uses the same CORE bank?"

We have now added an on-policy self-distillation (OPSD) baseline on Qwen (E4; Appendix P), which uses the same CORE bank in the teacher's prompt, and the same number of rollouts as our harvest. OPSD with CORE insights outperforms the base model and the R-SFT, but it trails insight distillation on both tasks (MathGAP 83% compared to 90%; Matchstick 81% compared to 97–99%). We acknowledge these are single seed runs, so we read it as "not better in our setting", rather than as a generalized ranking of the two methods.

W3: R-SFT gets fewer rollouts and fewer accepted samples

"The comparison therefore mixes better data with more data."

We use R-SFT primarily as a data control and not as a budget control: with R-SFT, we match starting weights, the 16 attempts per training problem, and the fine-tuning schedule, so that the only difference is whether insights are in the prompt when the training solutions are generated. The rollout-matched comparisons instead are GRPO and CORE extended. We have now made that more explicit in our revision. To separate better data from simply having more data, two new experiments change only the content of the insights: deleting insights centered around a specific skill removes that skill's performance gain after distillation, while maintaining the gain on other problems (E1). Moreover, irrelevant insights provide almost no gain (E4).

W4: Positive-only GRPO on Qwen

"On Qwen, GRPO trains only with positive-only advantages, which is close to online rejection-sampling fine-tuning, and it is still unstable."

We agree that it is close to online rejection-sampling fine-tuning. However, it was the only one of nine changes that trained stably on Qwen at low reasoning effort (Appendix C). Standard GRPO collapses on Qwen at the low reasoning effort we use. Beyond that, we tested nine other standard remedies, which include: lower learning rates, adding KL penalty, larger batches, DAPO and Dr. GRPO, and positive-only advantages. Only the last remedy trained stably. For gpt-oss-120b, standard GRPO led to stable training, and there, insight distillation still outperforms GRPO at matched rollouts on 5 of 6 tasks. This suggests the result does not depend on the GRPO variant itself.

W5: "mutually reinforcing"

"So, “mutually reinforcing” mostly reflects the benefit of consolidation."

We agree with you, which is why repeated rounds and scheduling are now moved to Appendices D and F, and explicitly state that a second-round bank adds only 0.4 pp on average, so the gain from alternating comes mostly from consolidation.

continued in the comment below

comment continued below

W6: "written down" is too strong

“What the model learns is written down before it enters the weights” is too strong."

We agree with you and have now removed the sentence in the revised draft.

W7: "easily detect" should be softened

"...claims implying that the method provides a way to easily detect such behaviors should be softened."

We agree and have now removed these claims in the revised paper. The CrossPL edit, which we keep as a case study, required us to understand how the exploit worked. In addition, the revised paper also tests whether the model itself can audit its own bank under a general rubric, without being told what the exploit is (E2): after distillation it lowers hacking in every task-model pair compared to plain insight distillation, but it cannot undo hacking the base model already does (which is addressed by CORE alignment).

Q2: Shuffled or irrelevant insights

"...distill from a bank with shuffled or irrelevant insights..."

Thank you for the great suggestion! We now include a new experiment where we use insight distillation from irrelevant insights (E4; Appendix P). Specifically, we generate the Matchstick training solutions with the model's own Chess960 insights, and show that it results in almost no gain after distillation (Qwen 76-79% against 74.5% for the base model and 97–98% with its own insights; no gain on gpt-oss-120b). The irrelevant bank keeps the format and length of a real bank, so the gain depends on what the insights say. Moreover, consistent with harvest predictor described in the Results section, the distillation gain still tracks how much the insight guidance raises the training solve rate.

Q3: Paired intervals for win counts

"Several “X of 12” wins come from differences within one standard error."

We agree that win counts by themselves can overstate evidence, and in the revision we replace "significantly" wherever no statistical significance test was run. The new skill-editing results (Table 11) use paired per-problem differences.

Q4: Reframing around auditability

"...strengthening the auditing part beyond one example. For instance, delete specific insights on other tasks and check whether the corresponding behavior disappears from the distilled model."

This was a great suggestion, thank you! We ran exactly this experiment, and the new additions to our revised paper are organized around it. On three tasks and both models, deleting one skill's insights removes 68–100% of that skill's distillation gain, while problems that need other skills keep their gains (E1; Section 4.2). E2 extends auditing to three reward-hacking environments with the model's own audit, and E3 adds CORE Alignment.

Q5: One adapter for several tasks

"Have you tried distilling several tasks’ banks into one adapter?"

Not yet, but we have now added it as part of future work. However, as a step in this direction, each single-task adapter is evaluated on every other task (Appendix L): 46 of 60 changes are within 3 pp, and the largest drop is 5.2 pp. On GPQA Diamond (which the model was never trained on), the seven gpt-oss-120b distilled models score 72.4-76.3%, against 73.6% for the base model.

We wanted to thank you again for such helpful and thoughtful feedback!

Reviewer_KMQjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The question here is whether learning in natural language and learning in weights can feed each other. CORE first mines short insights by contrasting successful against failed attempts, keeping the ones that survive a test. Insight distillation then puts those insights in context, collects verified solutions, and fine-tunes the same model on those solutions with the insight block stripped out, so the guidance has to end up in the weights rather than in the prompt. Later rounds rebuild the bank on the updated model; one variant instead uses the distilled checkpoint to initialize GRPO. The experiments cover two models and six reasoning tasks, mostly with three training seeds. Distillation retains roughly 1.02x of the in-context improvement, beats rejection-sampled SFT on all 12 model-task pairs, and beats the chosen GRPO baselines on 10 of 12 at matched rollouts. Ablations look at prompt removal, repeated rounds and cross-task effects. Separately, a CrossPL intervention shows that editing the bank can stop a grader exploit before it reaches the weights, and an exploratory single-seed scheduler suggests rollout savings. The main contribution is empirical: consolidation works, and the guidance is still editable while it is text.

Critical strengths
  1. The consolidation result is convincing. Near-complete transfer holds across two models and six tasks, and it is backed by three-seed reporting, rejection-sampled SFT controls and the prompt-removal ablations (Sections 4.1, 4.4).

  2. The language-plus-weights combination is doing real work: alternating CORE with distillation beats simply running CORE longer on 11 of 12 pairs, and distillation beats the chosen GRPO baselines on 10 of 12 at matched rollouts.

  3. The editing story is practically useful. Deleting the offending insights keeps the grader exploit out of the weights, while the genuine task improvement survives the stricter evaluation (Table 9).

Critical weaknesses
  1. Causal attribution stays limited. Insight guidance changes both how much accepted data there is and what that data looks like, so verbatim recall plus bank editing do not establish that individual insights are learned faithfully and can be controlled one at a time. The matched downstream controls are missing.

  2. The comparison scope is narrow. The Qwen setup uses positive-only GRPO advantages with a single globally stable recipe, and matching rollouts is not the same as matching compute, nor does it establish an advantage over other alternating methods.

  3. The iterative evidence is preliminary. The scheduler has one seed per pair, and rebuilding the bank in the second round buys only about 0.4 points on average over reusing the old one (Table 2; Appendix D).

Questions for authors
  1. Is there an SFT comparison that controls for quantity and composition, for instance by balancing accepted rows per problem or subsampling the insight-guided pool, together with a guidance baseline that is similarly effective but not produced by CORE? That would separate the value of tested insights from simply having more, and better, successful training data.

  2. For the CrossPL intervention, could you add a downstream-distillation control that deletes unrelated insights at matched harvest and pool size, and state which bank and training seeds sit behind Table 9? Appendix K discusses one particular edited bank, while the caption aggregates two evaluation seeds, and for the causal and robustness claims the difference between evaluation repeats and independent learning runs matters.

  3. Would the matched-budget GRPO conclusion still hold with task-specific, validation-selected recipes on the tasks where standard Qwen advantages train stably? Please also separate rollout or sample efficiency from total training compute, and say whether the SFT and GRPO learning rates were picked on a separate validation split or on the reported held-out problems.

2 comments

Dear reviewer,

Thank you for the helpful comments, they were very useful in strengthening the paper! In particular, we appreciate your assessments that "The consolidation result is convincing" and "The editing story is practically useful." In our general response we summarized the new experiments, the most relevant for your review being E1, E2 and E4.

Below, we address each of your specific questions and listed weaknesses:

W1 / Q1: Is the gain due to the insights themselves, or to more or better training data?

"...verbatim recall plus bank editing do not establish that individual insights are learned faithfully and can be controlled one at a time."

"Is there an SFT comparison that controls for quantity and composition...?"

We agree that insights act through the solutions that are generated with them in context. But using two new experiments, we try to separate what the training data contains from how much of the data there is in total. First, deleting insights that together describe a specific skill removes 68–100% of that skill's distillation gain, while retaining a similar performance gain on problems that do not rely on that particular skill (E1; Section 4.2). Second, generating the training solutions with irrelevant insights gives almost no gain (E4; Appendix P). These results provide additional evidence that the performance gains come from what the insights say. As a limitation, this control was done at the groups of insights describing a skill, not individual insights. We did not run a per-problem-balanced or subsampled pool, because the insight-guided pool contains correct solutions to train problems that the model does not solve without insights (on Qwen MathGAP, 95/100 training problems have a correct solution with insights, compared to only 70/100 without insights). Balancing cannot add those solutions, and producing them is what the insights contribute.

W2 / Q3: Comparison with GRPO

"...matching rollouts is not the same as matching compute..."

"Would the matched-budget GRPO conclusion still hold with task-specific, validation-selected recipes...?"

  1. Positive-only GRPO on Qwen (Appendix C). Standard GRPO collapses on Qwen at the low reasoning effort we use. Beyond that, we tested nine other standard remedies, which include: lower learning rates, adding KL penalty, larger batches, DAPO and Dr. GRPO, and positive-only advantages. Only the last remedy trained stably. For gpt-oss-120b, standard GRPO led to stable training, and there, insight distillation still outperforms GRPO at matched rollouts on 5 of 6 tasks. This suggests the result does not depend on the GRPO variant itself.
  2. Task-specific GRPO recipes (Appendices C and H). We did not tune GRPO per task, but two results suggest that it would likely not change the conclusion. First, on Qwen MathGAP, where standard GRPO trains stably, it needs about 10k rollouts to match what one round of insight distillation reaches with 3-4k rollouts. And even when GRPO's best checkpoint is picked on the held-out problems, which favours GRPO, it still falls short of one round of insight distillation on 7 out of 12 task-model pairs.
  3. Rollouts vs. compute (Appendix I). Appendix I counts generated tokens separately from tokens that are processed during fine-tuning. Plain GRPO's full run generates 4 to 8 times the amount of tokens of one round of insight distillation.
  4. How learning rates were chosen (Appendices B and C). Neither learning rate (GRPO or R-SFT) was chosen on a separate validation split due to small task sizes, and the revised submission now says so explicitly. The R-SFT rate (tuned for each model) was chosen on held-out results for MathGAP and Matchstick, averaged over insight distillation and R-SFT. The GRPO rate (tuned for each model) was chosen to be the largest rate that trains without collapse on every task. Importantly, on the eight model-task pairs that were never used to choose the SFT rate (Countdown, Chess960, Enigmata and MathConstruct, on both models) , insight distillation still beats R-SFT on all eight and GRPO on six.

continued in the official comment below

comment continued below

Q2: CrossPL

"...could you add a downstream-distillation control that deletes unrelated insights at matched harvest and pool size, and state which bank and training seeds sit behind Table 9?"

  1. Bank and seeds (Appendix M; Table 10, formerly Table 9). The table comes from one bank (Qwen, training seed 2), one harvest and one fine-tune. Its "two seeds" were two evaluations of the same model, not training runs. We have corrected the caption.

  2. Unrelated-deletion control (Appendix M). Before distillation, deleting the same number of exploit-unrelated insights leaves the exploit in almost all held-out solutions, whereas deleting the exploit insights removes it. We did not distill this control, only tested it in-context. Moreover, the edited bank's training pool is smaller than the full bank's, which if anything works against it, yet the model distilled from it no longer exploits the grader, and importantly, still outperforms the base model under the strict (fair) grader.

  3. Independent runs without human edits (E2; Section 4.3). As described in the general comment, we added a new section 4.3, where the model audits its own insight bank and removes insights that it judges to be hacks. Across three environments designed for reward hacking, this method lowers hacking rates after distillation in every task-model pair. Moreover, the CORE Alignment (described E3) pushed the hack rates even lower, below the base model's rate in every task-model pair, while simultaneously improving honest performance.

W3: Iterative evidence is preliminary

"The scheduler has one seed per pair, and rebuilding the bank in the second round buys only about 0.4 points on average over reusing the old one..."

We agree. Repeated rounds and the scheduler are now preliminary evidence in Appendices D and F, which state plainly that rebuilding the bank in round two adds only 0.4 pp on average.

We want to thank you again for such thoughtful feedback, and we hope that these changes were able to address your concerns!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment for all reviewers

We thank all reviewers for their thoughtful feedback! Your comments have helped us substantially improve the paper, both by clarifying our existing work, and also by adding new experiments that you requested. We are glad that the reviewers found the reward hacking mitigation to be an interesting direction, and the feedback to expand it was well received. Below, we summarize the main revisions.

E1. Selective skill editing (new Section 4.2). We show that insight distillation lets a model control which skills it learns, which methods that learn directly in the weights, such as GRPO, cannot do. This is because CORE stores what the model learns as separate, individually tested insights: the model can inspect them and remove specific ones before they are distilled into its weights. On three tasks (Matchstick, MathGAP, Chess960) and both models, the model identifies the insights in its bank that describe one specific skill and deletes them, and we then apply the same insight distillation to the remaining insights. We find that deleting the insights all related to one skill removes 68-100% of the distillation gain on problems that use that skill, while retaining performance gains on problems that do not require it (changes of at most 2.5 pp). For example, on Qwen Matchstick, the gain on chain-equality problems falls from +93 pp to +18 pp once insights about chain equalities are removed, while the gain on all other puzzles stays about the same (+5.6 to +4.9 pp). This holds in every pair where distillation improves the skill (5 of 6; the exception is gpt-oss-120b on Chess960, where distillation does not improve that skill in the first place).

E2. Self-auditing and hack editing (new Section 4.3). We then demonstrate the same advantage on three reward-hacking environments: Countdown-hack, Sphinx-hack and Wordle-hack. Plain insight distillation with CORE transfers the hack into the weights: hacking rises above the base model's rate in all six task–model pairs, to nearly 100% in most of them. Plain GRPO on the same reward also increases hacking in 5 of 6 pairs, to 98-100% in three of them. When the model audits its own bank using a generic rubric and removes the insights it judges to be hacks, hacking after distillation drops in all six pairs, compared to distilling all insights without selectively removing them.

E3. CORE Alignment (new Section 4.3). We show a further advantage of insight distillation: the model can learn its own alignment principles and consolidate them into its weights together with its task skills. This addresses the limits of auditing and removing insights, which lowers hacking but leaves it above the base model's rate and cannot undo hacking the base model already does. In CORE Alignment, the model judges its own attempts for hacking, learns task insights only from successes it judges to be honest, and writes its own principles by contrasting honest and hacking attempts. Importantly, the rubric is seen only by the model acting as judge; the model never directly sees it when solving the task. We distill the solutions generated with both insights and principles, and evaluate the model with no bank or judge. In every task–model pair, the aligned model hacks less and succeeds honestly more often than the base model. For example, on Countdown-hack, gpt-oss-120b hacks in 91% of episodes as a base model and in 100% after plain insight distillation, but in only 1% after CORE Alignment, while its honest success rises from 2% to 34%.

E4. Distillation controls (new Appendix P). (1) Irrelevant insights: generating the Matchstick training solutions with the model's own Chess960 insights instead of its Matchstick insights gives almost no gain after distillation (Qwen 76–79% against 74.5% for the base model and 97–98% with its own insights; no gain on gpt-oss-120b). The gain comes from what the insights say, not from having insights in the prompt or from the distillation procedure. (2) On-policy self-distillation (OPSD) with the same bank in the teacher's prompt and the same number of rollouts outperforms the base model and R-SFT, but trails insight distillation on both tasks we tested (Qwen; one seed).

Clarifications and corrections. Repeated rounds and the autonomous scheduler moved from the main text to Appendices D and F. We now state explicitly how the learning rates were chosen, for fine-tuning (shared by insight distillation and R-SFT) and for GRPO (Appendices B, C). We corrected the CrossPL table caption (Table 10: two evaluations of one model, not two training seeds), and added GPQA Diamond evaluations for the gpt-oss-120b distilled models (Appendix L), a benchmark none of them was trained on.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.