acceptodds
Under review as a conference paper at ICLR 2027

Evolve to Learn: Gradient-Aligned Context Evolution for Model Self-Improvement

Abstract

Self-evolving language models can continuously acquire useful experience through test-time interaction, reflection, and prompt optimization, yet turning such experience into model itself to ensure self improvement remains challenging. On-policy self-distillation (OPSD) provides a natural mechanism for internalizing test-time information into model parameters, yet it largely assumes that the generated experience is worth learning. We show that this assumption does not always hold: experiments that improve test-time performance may provide limited or even harmful training signals. To better characterize which experience should be internalized, we systematically analyze diverse mutated experiments across multiple tasks and models, and find that gradient-based alignment is a more reliable indicator of downstream distillation gains than immediate test-time improvement. Based on this finding, we propose E2L, a self-evolution framework that iteratively generates, evaluates, and distills experiments according to their estimated training utility. E2L closes the loop between experience generation and parameter learning, enabling agents to autonomously produce experience suitable for persistent self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self Improvements, On Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hMMSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

Inference time methods can help models improve performance, but it's hard to incorporate those strength. Namely, during OPD, selecting which context to distill is challenging. The paper proposes E2L to solve this.

Critical strengths
  1. The core problem is timely and important. During OPD, selecting which context to distill can be challenging. Authors made contributions by providing insights into this.

  2. The authors tried to provide many experimental details and some theoretical motivations.

Critical weaknesses
  1. While the topic is interesting, I found the soundness to be very limited. There are many such evidence in the paper (such as number inconsistencies between tables). I just mention one of them: On AIME25, AIME26, LiveBench Math, and HotPotQA, it seems like the performance difference between GEPA-OPD and E2L is very small. Also it's an over-claim to use self-improvement, as the method need better external models such as Qwen3.7-Max.

  2. The presentation is unclear. Examples:

(1) The introduction section is without a title.

(2) The authors mixed the usage of \citep and \citet.

(3) Figures look unpolished. In Figure 1, there are many small grey words that should not be included in an acedemic paper.

(4) Conclusion should appear after related work.

Questions for authors
  1. Revise writing substantially.

  2. Revisit experiments to check if the performance gain is significant. Currently it's not statistically better than baselines.

Reviewer_j88LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which evolved prompts are worth distilling into model weights. Its main observation is that good performance with a prompt does not necessarily translate into improvement after distillation. E2L selects prompts using alignment between candidate-induced gradients and supervised gradients on examples the model gets wrong. It combines this signal with GEPA-style search, then distills the selected prompt.

Critical strengths

The distinction between a prompt being useful at inference and useful for training is interesting. The correlation study and accuracy-matched comparisons provide evidence that these are different objectives.

The fixed-pool selection experiment helps separate the value of alignment from the search procedure. Evaluating without the selected prompt also directly tests whether improvements survive distillation.

Critical weaknesses

It is unclear whether the measured alignment actually uses the distillation gradient. Equation 6 defines the candidate gradient through the OPSD loss, while Appendix B describes answer-token NLL gradients on frozen candidate responses. These are different quantities. This matters because the paper motivates selection through the direction of the actual training update.

The gains need stronger evidence of robustness. Most improvements over GEPA-OPD are around 1–2 percentage points, with one search/training run per candidate and small evaluation sets. Sampling several responses does not establish stability across training seeds. Alignment also loses to accuracy selection on AIME25, and the correlation is not significant within the GLM candidate group.

Questions for authors

Which candidate gradient is used in the reported results: the reverse-KL distillation gradient or an answer-token NLL gradient? If the latter, how well does it approximate the actual update?

Do the main gains persist across independent search and training seeds?

Does the SFT baseline receive the same reference answers and solution explanations used for alignment? What is the total search cost, including gradient computation, relative to the baselines?

Reviewer_VZ4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which evolved textual experiences (or prompts) are worth internalizing into a language model's parameters via on-policy self-distillation (OPSD). The authors observe that a candidate prompt's inference-time accuracy when provided in context correlates poorly with the model's post-distillation accuracy once the prompt is removed. To address this, they propose measuring gradient alignment, defined as the cosine similarity between the parameter gradient induced by distilling a candidate prompt and a reference supervised gradient computed on examples the base model currently gets wrong. Building on this metric, they introduce Evolve-to-Learn (E2L), which combines Pareto-based prompt evolution with a gradient-alignment selection channel, and distills the highest-alignment prompt into the student model. Experiments with Qwen3-8B on AIME, LiveBench Math, HotPotQA, and JailbreakBench show improvements over the base model and self-distillation baselines.

Critical strengths
  • The observation that experiences achieving high inference-time performance as prompts are not necessarily suitable for self-distillation is interesting, and Figure 1 demonstrates this point well.
Critical weaknesses
  • Empirically, on most evaluated benchmarks (AIME25, AIME26, LiveBench Math, and HotPotQA), the proposed E2L-Alignment method does not outperform the GEPA-OPD baseline by much, with actual margins all within 1% to 2%. The only benchmark with a large improvement is JailbreakBench, but the paper does not adequately explain why the gain is so pronounced on this specific task.
  • The method computes gradients only on the last four transformer layers, but the paper does not clearly explain why these four layers were chosen, and there are no ablation experiments justifying this choice.
  • The prompt selection procedure requires taking gradients with respect to ground-truth answers, which assumes access to ground-truth labels that match the model's actual deployment setting. In practice, such labels are often unavailable. For example, in many math and coding tasks we use thinking models, where ground-truth thinking trajectories are difficult to obtain. Because of this, the experiments in the paper (such as on math and HotPotQA) are conducted using a non-thinking model, which is not realistic. Directly appending the final answer after the prompt to compute gradients ignores the distribution shift induced by intermediate Chain-of-Thought tokens generated during actual inference. How to use gradient alignment to select prompts in a more realistic thinking-model setting remains an open problem.
  • There are numerous formatting and presentation issues throughout the paper. For instance, in-text citations consistently use \citet instead of \citep, there are broken cross-references, and the appendix contains many uncleaned AI-generated artifacts, such as directly outputting raw local filenames and git commit hashes in Appendix A.2 (lines 616 and 620).
Questions for authors
  • Why does E2L-Alignment achieve such a large improvement over GEPA-OPD on JailbreakBench, whereas the gains on all other benchmarks are within 1% to 2%?
  • Why were the last four transformer layers chosen for computing gradient alignment, and how sensitive is the method to this choice?
  • How can gradient alignment be applied in realistic settings with thinking models, where ground-truth thinking trajectories are unavailable and computing gradients directly on final answers ignores the intermediate Chain-of-Thought tokens during inference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.