acceptodds
Under review as a conference paper at ICLR 2027

Learning to Self-Evolve: Reinforcement Learning for Context Self-Editing

Abstract

We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (SuperGPQA), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, post-training, self-evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_spcdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Learning to Self-Evolve (LSE), which trains a context editor through single-step reinforcement learning while keeping the downstream action model frozen. At evaluation time, the editor uses labeled feedback to revise instructions, and UCB-guided search selects contexts using a development set. Experiments on BIRD and SuperGPQA show improvements over untrained editors and prompt optimization baselines, with an additional transfer experiment on another action model.

Critical strengths
  1. Keeping the action model and evolution protocol fixed provides useful evidence that explicitly training the editor improves context adaptation. The gains are particularly encouraging on BIRD.
  2. The trained editor improves a different action model without additional training, providing preliminary evidence that the learned editing policy can transfer across models.
Critical weaknesses
  1. Limited methodological novelty and insufficient validation of the framework’s advantages. Prior work such as PRewrite [PRewrite: Prompt Rewriting with Reinforcement Learning] already trains prompt rewriters through RL using downstream performance. LSE’s single-step policy-gradient formulation and pre-edit reward baseline do not, by themselves, establish a substantial methodological advance. Although the results support training the editor, comparisons with other trained editing approaches are insufficient to establish the advantages of this particular framework.
  2. The test-time search ablation uses only the untrained editor, leaving the benefit of UCB search within the full LSE system unverified.
  3. The evaluation depends on labeled target-domain supervision, and the data protocol is insufficiently specified. The editor receives ground-truth answers for adaptation batches, and search repeatedly uses labels from a fixed target-domain development set. This is closer to supervised target-domain prompt optimization than test-time adaptation without target-domain labels; the availability of this supervision and the resulting scope of the claims need clarification. Although dev/test disjointness is stated, the adaptation pool’s source, size, and separation from dev/test remain unclear, as does whether RL reward evaluation or checkpoint selection uses evaluation-domain data.
Questions for authors
  1. Please clarify the relationship between the adaptation problem pool, dev set, and test set and clarify whether RL reward evaluation or checkpoint selection accesses any evaluation-domain data.
  2. Please discuss prior methods that train models to generate or refine prompts, such as PRewrite, and provide broader experimental comparisons to substantiate the advantages of the proposed RL framework. The current comparison with a GRPO variant alone is insufficient to establish these advantages.
  3. Please report the total cost of the test-time evolution process per domain, including feedback collection, context editing, and repeated development-set evaluation. How does the performance–cost trade-off compare with the baselines, and how is this adaptation cost amortized over subsequent queries?
Reviewer_VFuoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LSE, which trains an LLM through reinforcement learning to revise instructions using feedback from previous problems. Training rewards edits by their downstream accuracy improvement, while test-time tree search composes edits across rounds. A trained 4B editor improves performance on BIRD and SuperGPQA and transfers to a different SQL action model.

Critical strengths
  1. LSE consistently outperforms the untrained editor across all 16 evaluation domains while keeping the action model fixed.
  2. Without additional training, it improves a specialized SQL model from 63.0% to 66.1%, suggesting benefits beyond its original action model.
Critical weaknesses
  1. Each domain uses only 50 or 100 test questions, and the paper reports no uncertainty estimates or results across independent evolution seeds.
  2. Evolution requires ground-truth feedback and repeated labeled development-set evaluations, but performance under limited feedback and total computational cost are not evaluated.
  3. The paper trains separate editors for SQL and QA and evaluates cross-model transfer only on SQL, leaving it unclear whether LSE learns a broadly transferable context-editing skill.
Questions for authors
  1. Do the gains persist across evolution seeds and larger held-out test sets?
  2. What are the total training and evolution costs, and how does performance change with fewer labeled evaluations?
Reviewer_YKVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Learning to Self-Evolve (LSE), an RL framework that trains a 4B LLM (Qwen3-4B-Instruct) as a prompt editor fψ(ct,St)f_\psi(c_t, S_t) for inter-episode context evolution. LSE simplifies multi-step evolution during training to a single-step contextual bandit over states sampled from pre-populated 20-round evolution trees G\mathcal{G}, using the dev-set accuracy delta directly as the policy-gradient advantage: ALSE=Rˉ(c1)−Rˉ(c0)A_{\text{LSE}} = \bar{R}(c_1) - \bar{R}(c_0) (Eq. 9–10). At test time, LSE uses a 25-round UCB tree search (Algorithm 1) over contexts scored on DdevD_{\text{dev}}. On BIRD (Text-to-SQL) and SuperGPQA, the 4B LSE editor outperforms untrained Qwen3-4B-Instruct, GPT-5, Claude Sonnet 4.5, GEPA, and TextGrad, and transfers zero-shot to guide Arctic-Text2SQL-R1-7B (+3.1%, Table 3).

Critical strengths
  • 4B Trained Editor Outperforms Frontier Models Under Identical Search: Holding the action policy (Qwen3-4B-Instruct), problem batch sequence, and 25-round UCB tree search fixed, RL training of the 4B editor improves accuracy across all 16 domains, beating GPT-5 (69.3% vs. 62.1% on BIRD) and Claude Sonnet 4.5 (64.8% on BIRD), and transferring zero-shot to Arctic-Text2SQL-R1-7B.
  • Tree-Guided Evolution Prevents Catastrophic Linear Degradation: Figure 2b and Figure 3 clearly demonstrate that linear-chain editing collapses on BIRD Card Games (from 56% to <30%) after a bad edit sequence, whereas UCB tree search backtracks to high-scoring ancestors.
Critical weaknesses
  1. "Test-Time Self-Evolution" Relies on Ground-Truth Answers (y_i*), Training-Pool Sampling, and 25 Full D_dev Passes: Despite framing LSE as "test-time adaptation after deployment" (§1), Algorithm 1 and Appendix A reveal that at test time: (1) each round samples 10 problems from the labeled training pool and feeds ground-truth answers y_i* and rewards r_i in S_t = {(x_i, y_i, y_i*, r_i)}_{i=1..10} into f_ψ; and (2) every round evaluates the new context c_new across the entire labeled development set D_dev (50 × 8 = 400 generations per round on BIRD; 100 × 8 = 800 per round on SuperGPQA). This is supervised validation-set prompt optimization, not unsupervised test-time adaptation on unlabeled test inputs.

  2. Unaccounted Compute Discrepancy in A_LSE vs. A_GRPO Comparison (§3.3 & Appendix A): In Section 3.3 (Line 298), the authors claim A_LSE = R̄(c_1) - R̄(c_0) "requires neither multiple rollouts per prompt for group-based normalization nor a separate value network." Yet Appendix A (Line 666) states that RL training generates 4 rollouts per node! Furthermore, as proven in Line 256, when multiple edits share a starting state (c_0, S_0), subtracting R̄(c_0) cancels out identically in GRPO's mean-centering, leaving only GRPO's standard-deviation division. It is unclear why A_LSE beats A_GRPO by 4.4% in Figure 2a when both use 4 rollouts per node—unless A_GRPO was crippled by dividing by near-zero group standard deviations or did not use the same improvement-potential node curriculum (Appendix A, Line 663).

  3. Static Node Value R̄_n in UCB Tree Search Without Backpropagation (Eq. 5 & Algorithm 1): In Algorithm 1 (Line 10), when parent n* is expanded to create child c_new, v_{n*} is incremented by 1, but R̄_{n*} is never updated with the child's score R̄(c_new). Because R̄_n is a fixed deterministic score on D_dev rather than a running average of subtree returns, a node n* with high R̄_{n*} that only generates degraded children will still be repeatedly re-selected as ln(N) grows.

Questions for authors
  • Clarification Requests:
    1. Exact Difference Between ALSEA_{\text{LSE}} and AGRPOA_{\text{GRPO}} Given 4 Rollouts/Node: Appendix A states you generate 4 rollouts per node during RL training. Since subtracting Rˉ(c0)\bar{R}(c_0) is invariant under GRPO's group mean subtraction across those 4 rollouts (Eq. 8), is the 4.4% gap between ALSEA_{\text{LSE}} and AGRPOA_{\text{GRPO}} in Figure 2a solely caused by dividing by the 4-sample group standard deviation in AGRPOA_{\text{GRPO}}?
    2. In Algorithm 1 Line 7, how much does BIRD and SuperGPQA accuracy drop if ground-truth answers yi∗y_i^* are omitted from StS_t so that fψf_\psi only observes (xi,yi,ri)(x_i, y_i, r_i) (e.g., SQL execution error/binary pass signal without the ground-truth solution)?
  • Suggestions for Future Work:
    1. Subtree Value Backpropagation: Backpropagate child rewards Rˉ(cnew)\bar{R}(c_{\text{new}}) (e.g., max or mean subtree reward) into ancestor statistics in Algorithm 1 so UCB tracks evolutionary potential rather than static parent accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.