acceptodds
Under review as a conference paper at ICLR 2027

COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvement

Abstract

Recursive self-improvement (RSI) seeks to move large language models beyond static training pipelines toward systems that can participate in improving their own future behavior. Existing approaches largely follow two directions: updating model parameters through online learning, or improving the external context through search, reflection, and prompt optimization. Although both mechanisms can support continued improvement, they are typically studied independently. This separation overlooks an important interaction: the context shapes the experience from which a model learns, while an evolving model may interpret and utilize the same context differently over time. We therefore formulate RSI as a problem of parameter–context co-evolution, where model parameters and the learning context adapt within a shared feedback loop. We introduce COEVO, a framework that updates model parameters from on-policy experience while adapting contextual guidance according to the state of the evolving policy. Policy entropy and prompt-conditioned attention are used as complementary signals to guide this adaptation. Experiments show that COEVO consistently improves task performance over fixed-context reinforcement learning and produces policies that are more robust to changes in system prompts. More broadly, our results suggest that external context should be viewed not merely as a fixed interface to a language model, but as an adaptive component of recursive self-improvement.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Recursive Self-Improvement, Reinforcement Learning, Context Evolution, Policy Adaptation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y29aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces COEVO, a framework that jointly adapts model parameters and system-prompt strategy scaffolds during reinforcement learning. Parameters are updated through on-policy RL, while context revisions are periodically generated and evaluated using the current frozen checkpoint. Policy entropy and section-level attention guide the choice among four editing strategies: Expand, Relax, Clarify, and Consolidate. The main contribution is a policy-aware approach to context–parameter co-evolution, in which evolving guidance shapes subsequent training experience. Validation covers mathematical reasoning and code generation across multiple Qwen model settings, with comparisons against fixed-prompt RL, single-signal ablations, and a reward-driven prompt-evolution baseline. Cross-prompt evaluation further examines whether improvements persist under shared inference contexts.

Critical strengths
  1. A compelling formulation of policy-aware context–parameter co-evolution. The paper addresses a meaningful interaction: prompts shape the trajectories used for learning, while parameter updates change which guidance is useful. COEVO operationalizes this interaction through state-conditioned context revision within an RL loop (Secs. 2–3). This is consequential because it treats training context as an adaptive component and offers a promising approach to improving the learning process itself.

  2. Encouraging empirical gains with evidence of benefits beyond the evolved prompt. COEVO improves Average@12 across the reported mathematical reasoning and code-generation settings (Table 13). More importantly, shared-prompt evaluation on 4B mathematics shows stronger average and worst-prompt performance (Table 2), supporting the interpretation that context evolution benefits the learned parameters rather than merely producing a better inference-time prompt. This provides substantive preliminary support for the central idea.

Critical weaknesses
  1. Limited statistical support for the reported improvements. The mathematics evaluation uses only 30 AIME 2025 problems, and the experiments report a single training seed, without independent training repetitions or confidence intervals. On this evaluation set, a 6.67-percentage-point improvement in Pass@12 corresponds to only two additional solved problems. Consequently, the statistical reliability and training-run stability of the reported gains remain unclear.

  2. Insufficient validation of the heuristic intervention rules. The mapping from entropy–attention states to Expand, Relax, Clarify, and Consolidate is intuitively motivated but not directly validated. The single-signal ablations show that the combined method performs better in the reported setting, but do not establish whether, at the same policy checkpoint, the prescribed intervention yields better subsequent learning than the other three directions (Table 5). For example, whether Expand improves subsequent learning more than the alternative strategies when both entropy and attention are low.

  3. Additional computational expenditure. COEVO generates eight candidate edits every five RL steps and evaluates the candidates together with the incumbent using fresh rollouts (Appendix A.4; Algorithm 1). This introduces substantial additional sampling and evaluation expenditure relative to fixed-prompt RL. Without computational-cost reporting and matched-budget comparisons, it remains unclear how much of the improvement comes from the proposed policy-aware adaptation rather than the additional search budget.

Questions for authors
  1. Were any independent training runs conducted?

  2. What evidence supports the specific mapping from entropy–attention states to the four editing strategies?

  3. What is COEVO’s total computational overhead relative to fixed-prompt RL, including candidate generation, candidate evaluation, and attention measurement?

Reviewer_DBLnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

COEVO jointly adapts model parameters and system prompts during reinforcement learning. It uses policy entropy and attention to prompt sections to guide periodic changes to the prompt’s task strategy. Experiments on mathematical reasoning and code generation show gains over training with fixed prompts, with ablations examining the contribution of each signal.

Critical strengths
  • Treating the training prompt as an adaptive component of RL is a novel idea. The framework provides a practical way to study how contextual guidance interacts with parameter learning.
  • The results cover several model sizes and two task domains. Evaluating checkpoints under shared inference prompts is also a useful control for checking whether the benefits persist beyond the final evolved prompt.
Critical weaknesses
  • The paper decides whether current entropy and attention are high or low by comparing them against the cumulative historical means. However, prompt sections get rewritten during training. And the paper does not clearly explain how an updated section is matched to its past history, or why attention scores collected on the old text remain a fair baseline for the new text. Besides, as training goes on, new steps have less impact on a simple cumulative average. The reference value might become too slow to track recent changes in the policy. I suggest that add ablation study about reseting the statistics for a section after it gets edited, and compare the cumulative mean against a sliding window or an EMA baseline. It will make this part of the method much more convincing.
  • For prompt selection, COEVO uses a strictly greedy strategy along a single path. Each selected prompt becomes the starting point for later edits, and the method has no explicit way to rollback to earlier versions. A poor choice could therefore steer subsequent updates toward a less useful region of the search space. A version-control setup, similar to a Git worktree to allow model to branch out or roll back to strong earlier prompts may be a more reliable way to explore the exploration space.
  • The comparisons with fixed prompts do not establish whether COEVO’s gains hold under the same overall compute budget. Those baselines only perform the usual training, while COEVO also generates and evaluates multiple prompt candidates. Some gains could come from this extra search. Prompt variation itself may also enrich the training experience, even without accurate entropy and attention signals. More controlled ablations would help here.
  • The current ablations show that using entropy and attention to guide prompt edits improves task accuracy. However, since the paper frames this work around Recursive Self-Improvement (RSI), a key piece of evidence is missing: does the model actually become a better prompt editor (or harness improver) as its weights are updated? A useful control would keep the initial model frozen as the prompt editor while allowing the task model to continue training. Comparing this setup with COEVO would show whether updating the editor alongside the task model provides an additional benefit.
Questions for authors
  • Why is co-evolution run separately for Math and Code, with evaluation limited to the corresponding domain? Given the paper’s focus on RSI, it would be useful to understand whether these improvements carry over to other tasks or come at their expense. For example, how does a model’s performance on Code and general reasoning change after co-evolution on Math? Is there any evidence of performance degradation or catastrophic forgetting?
  • Even though the paper samples 12 responses per problem, each domain is only evaluated on a single benchmark. This feels a bit thin, especially for Math, where AIME 2025 only has 30 problems in total. On a 30-problem test set, solving just one or two more problems changes Pass@12 by 3.3% to 6.7%, which makes it hard to judge statistical significance. Could the authors provide uncertainty estimates and results on additional benchmarks (e.g. MATH-500) to show that the gains are consistent?
Reviewer_yfCQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces COEVO, a framework which, during reinforcement learning, simultaneously evolves the model's parameters and the system prompts. The policy entropy and the section-level attention are used to describe the current state of the policy and direct four prompt-editing actions: Expand, Relax, Clarify, and Consolidate. Before training continues, possible prompts are generated and compared using a frozen checkpoint. Experiments involving mathematical reasoning and code generation show improvements when compared to fixed-prompt RL, together with cross-prompt evaluation and single-signal ablations.

Critical strengths
  • There is a good reason for this. Since context influences the range of possible learning paths, and since adjustments to the parameters can affect how useful the contextual guidance is, it is a worthwhile area of study.
  • The empirical results are good, with improvements noted for various model sizes and in two different domains. The use of a shared inference-prompt evaluation allows one to determine if the benefits continue to hold beyond the evolved prompt.
  • Designing interventions that are interpretable. The editing operations are understandable, and the case studies show that adapting to a particular context can involve both providing guidance and loosening the existing constraints.
Critical weaknesses

1. The descriptions given in the main text and in the appendix are inconsistent or incomplete regarding the controller.

Several central execution rules are difficult to reconcile:

  • The text in Section 3.3 says that the current prompt is maintained if exploration stays within a calibrated reference range or if the utilization estimates are unreliable; however, Appendix A.4 and Algorithm 1 talk about comparisons with historical means without defining the reference range, the reliability criterion, or the corresponding conditions for skipping.

  • The main text states that the section with the greatest mismatch should be selected, while Appendix B.2 permits several sections that meet the criteria to be given TARGET instructions without specifying a step for selecting a single section.

  • The acceptance of a candidate is said to involve achieving a desired change in state while at the same time keeping the reward within a specified tolerance; yet the appendix assigns the task of selection to an undefined BestInDirection operation and provides no quantitative acceptance criteria, no reward tolerance, and no ranking rules.

  • The intervention determined by Algorithm 1 is based on measurements from θt\theta_t, but the candidates are evaluated using θt+1\theta_{t+1}. The connection between these diagnostic and comparative measurements should be made clear in relation to the main text's explanation of the "same frozen checkpoint" scenario.

Because of these discrepancies and omissions, it is not possible for readers to reconstruct the controller completely or to evaluate the proposed policy-aware mechanism.

2. The experiments do not adequately isolate the source of the gains.

Together with the controller, COEVO also includes additional candidate search, the computation of rollouts, and variation in the training prompts. Therefore, the fact that the same number of RL updates are carried out does not mean that the same amount of computation has been used. The fact that there are improvements over fixed-prompt RL does not show how much of the benefit is due specifically to matching interventions to the measured policy states.

In order to tell whether the advantage comes from targeted adaptation or from more general benefits resulting from search and prompt diversity, it would be useful to carry out compute-matched comparisons using: 1) random editing directions, 2) shuffled state signals, and 3) a fixed prompt mixture.

It would also be useful to include a fixed-prompt entropy-control baseline in order to check whether maintaining exploration requires prompt evolution.

3. The extent of the framing goes beyond the methodological contribution that has been demonstrated.

The joint evolution of the prompt and the parameters has already been investigated, for example by E-SPL. The main difference in this case is the entropy–attention controller and its four rules specifying how the states map to operations. Although these rules are intuitively understandable, they have not been adequately validated.

It is not necessary for token entropy to reflect meaningful diversity of strategy, and attention does not have to reliably show when an instruction is being used. Simply rearranging the rules while keeping the same editing and selection budgets would directly show whether the proposed correspondence is important.

4. The evaluation results and how they are compared need to be clearer.

It is not possible to consistently reconstruct the 4B-Math aggregate results from the cross-prompt matrix. For instance, Table 13 gives a figure of 40.55% Average@12 for the Medium category, while Table 2 shows 41.94% with the matched inference prompt and 41.67% when the results across prompts are averaged. There also needs to be an explanation for the difference between the approximately 44.3% COEVO figure in Figure 4/Table 5 and the 44.17% shown in Tables 2 and 13.

Furthermore, the math prompts vary in their specification of the tool interface as well as in the level of detail regarding the strategy, which may confound improvements in reasoning with compliance regarding format and tool use.

Questions for authors
  1. What controller specification agrees with the results of the actual experiments? Please specify the reference range, the reliability check, the procedure for selecting sections, and the BestInDirection acceptance rules.
  2. What role do the measurements from θt\theta_t play in guiding the selection of candidates under θt+1\theta_{t+1}, and how are the section histories kept up to date after the prompt has been restructured?
  3. When the rollout and token budgets are matched, does COEVO perform better than random intervention selection, shuffled state signals, fixed prompt mixtures, and fixed-prompt entropy control? What is the result when the four assignments from state to operation are permuted?
  4. Is it possible to associate each aggregate result with its checkpoint, inference prompt, and aggregation rule? Do the advantages remain when all the math prompts use the same tool interface?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.