acceptodds
Under review as a conference paper at ICLR 2027

Learning to Improve: Towards Learnable Recursive Self-Improvement for LLM Agents

Abstract

The self-improvement of large language model agents is expanding from single response correction to continuous updating of reusable skills and execution strategies. However, the improvement of task ability does not necessarily lead to the improvement of improvement ability : when the diagnosis and repair mechanism remains fixed, it is difficult for the system to use its own repair experience to optimize the subsequent improvement process. In order to bridge this gap, this paper proposes a recursive self-improvement framework based on executable feedback, so that agents can not only update task skills, but also modify the programs that generate these updates. Specifically, the framework generates and evaluates candidate fixes by combining execution trajectories, verification results, and multi-dimensional skill diagnosis through the collaboration of task execution agents and enhanced agents. On this basis, the framework compares the expected effect and actual effect of repair, transforms the repeated program defects with clear judgment basis into executable regression constraints, and guides the enhanced agent to modify its own diagnosis, repair generation and self-modifying programs accordingly. The independently verified new version replaces the parent version to perform subsequent improvements, so that the improvement mechanism is not only the object of evolution, but also the executive subject of the next round of evolution. In addition, we introduce cumulative regression testing, version isolation, and uniform budget constraints to test the effectiveness of code evolution and trace its cost. TRACE connects task feedback and mechanism update into a verifiable recursive closed-loop, which provides a specific path for studying the continuous accumulation of agent improvement capabilities and cross-task migration.

Primary area:
generative models
Keywords:
large language model agents, recursive self-improvement, executable feedback, skill diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JhCWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an agent framework that decides how to fix a failure based on what caused it. An Answer Agent runs the task. After a failure, an Improvement Agent reads the verifier feedback, the execution trace, and the existing skills, and chooses one of three fixes: guidance for the next run, a memory update, or a lasting change to a skill. A proposed change is adopted only after it passes replay and regression checks. The authors also propose using the results of past fixes to change the Improvement Agent itself. The experiments consist of results from archived runs, an eight-case check of the diagnostic rules, and one PDF-form case that tests update validation.

Critical strengths

Strength 1: The paper points out that different failure causes need different fixes

The same missing output can come from a missing skill, a skill that was not retrieved, or a skill that was not followed, and each cause needs a different fix. The paper maps these cases to execution guidance, memory updates, and skill changes, and keeps proposing a change separate from adopting it. This avoids changing stored skills after every failure.

Strength 2: The PDF case shows errors that a pass/fail reward hides

In Figure 4 and Appendix D, a faulty helper that skips checkbox and radio fields still gets reward 1 from the original verifier, although the underlying checks drop from 47 passes to 35 passes and 12 skips. Stricter checks tell the faulty and intact outputs apart. So a candidate update should be judged by its actual output and skipped checks, not only by the final reward.

Critical weaknesses

Weakness 1: The paper does not test its central claim

The abstract claims that an agent can use the results of past fixes to improve the way it chooses fixes. The paper never shows a model-generated change to the Improvement Agent that was accepted, put into use, and then improved later decisions. The algorithm on page 14 only records a meta_proposal. Table 6 lists the benefit of an accepted policy change as an evaluation target, not a reported result. The conclusion also leaves this test to future work. The authors are open about this, but it means the central claim in the abstract has no direct evidence. The paper should separate what is implemented, the diagnosis and validation steps, from what is not yet shown: outer-loop learning that improves the Improvement Agent itself.

Weakness 2: The comparisons cannot isolate the effect of outer-loop learning

For Qwen, the fixed-strategy and RSI-enabled runs cover 38 and 81 instances with different tasks, run histories, and implementations, so their success rates (65.8% and 71.6%) are not comparable. The GPT evolution result counts a task as solved if any of several archived attempts succeeded, so the gain may come from extra attempts. In the PDF case, a hand-written adapter and a fixed rule that always requires replay pick the same intact helper, so nothing shows that a learned policy beats the fixed rule. The authors acknowledge these limits, but they affect the main contribution. Fixing them needs a paired comparison, not more unpaired success rates.

Weakness 3: The "first to define" claim conflicts with prior work such as STOP

Page 2 claims to be the first to define the problem of improving both task skills and the improvement process. Yet page 3 cites STOP and related work for the idea that the improvement process itself can be changed. STOP uses a seed improver to improve itself and reports that the updated improver beats the original on a few downstream tasks. So improving how an agent improves is not new. The new part may be the combination of failure attribution, fix selection, and candidate validation. The paper should say what this adds over prior work and drop or narrow the "first" claim.

Questions for authors
  1. (Related to Weakness 1) Is there any model-generated change to the Improvement Agent that was accepted and put into use? If so, please show the change, why it was accepted, and how it affected fix decisions on later, independent tasks.

  2. (Related to Weakness 2) Is there a comparison on the same tasks, with the same implementation and total budget, that separates a learned Improvement Agent from fixed diagnosis, the replay rule, and extra retries? If not, which finished experiment supports outer-loop learning?

  3. (Related to Weakness 3) STOP already shows an improver modifying itself. Compared with STOP, what exactly is new in your two-level formulation?

5 comments
AuthorsOpenReview ↗

Response to JhCW (1/4)

Dear Reviewer JhCW,

Thank you for recognizing the value of cause-dependent interventions and the PDF audit’s finer-grained assessment of functionality. Below, we explain what each experiment establishes about routing, policy revision, and candidate validation.

W1&Q1: (W1) The paper does not test its central claim. The abstract claims that an agent can use the results of past fixes to improve the way it chooses fixes. The paper never shows a model-generated change to the Improvement Agent that was accepted, put into use, and then improved later decisions. The algorithm on page 14 only records a meta_proposal. Table 6 lists the benefit of an accepted policy change as an evaluation target, not a reported result. The conclusion also leaves this test to future work. The authors are open about this, but it means the central claim in the abstract has no direct evidence. The paper should separate what is implemented, the diagnosis and validation steps, from what is not yet shown: outer-loop learning that improves the Improvement Agent itself. (Q1) Is there any model-generated change to the Improvement Agent that was accepted and put into use? If so, please show the change, why it was accepted, and how it affected fix decisions on later, independent tasks.

Appendix E.3 documents a versioned policy proposal and its use in shadow evaluation. The proposal shifts improvement from task-Skill content to feedback strategy. The record distinguishes this execution from formal promotion and subsequent evaluation on independent tasks.

The revised rule governs the next intervention after Skill revision stalls. It retains the useful candidate knowledge and redirects feedback toward verification against source evidence. Previous outcomes motivated this shadow evaluation. In the dependency-auditing case, retrying with the original feedback achieves 60.00% checkpoint success. The original candidate Skill with that feedback reaches 80.00%, and a revised candidate with unchanged feedback also achieves 80.00%. Strategy-shift feedback with the revised candidate reaches 100.00%. The policy therefore redirects the intervention from further Skill rewriting toward source-evidence checks.

Local RSI policy revision and execution record (Appendix E.3, excerpt)

StageChange or statusRecord
Policy changeRedirect feedback toward evidence checksfeedback_shift_after_skill_stall_v1
Acceptance and activationShadow evaluation; not formally promotedcandidate_shadow_test_only
Subsequent executionOriginal feedback: 80.00%; strategy-shift feedback: 100.00%Additional strategy-shift run

The execution record identifies the policy change and its validation stage. Appendix E.4 then compares task outcomes using the revised candidate Skill under two feedback conditions. The reported configurations retain the revised candidate Skill while changing the feedback condition.

Checkpoint success under different candidate Skills and feedback conditions (Appendix E.4)

Dependency-auditing conditionCheckpoint success (CTRF)
Retry with original feedback60.00%
Original candidate Skill + original feedback80.00%
Revised candidate Skill + original feedback80.00%
Revised candidate Skill + strategy-shift feedback100.00%

The original candidate raises checkpoint success from 60.00% to 80.00%. Further Skill revision with unchanged feedback retains 80.00%, whereas strategy-shift feedback with the revised candidate reaches 100.00%, an increase of 20.00 percentage points. The sequence illustrates the intended role of local RSI. Previous revision outcomes motivate a shift from editing Skill content to improving feedback verification. The corresponding policy status is documented in Appendix E.3.

The case links a versioned policy proposal to a change in the next intervention. Useful candidate knowledge is retained while the strategy shifts toward feedback verification. The reported follow-up execution completes the remaining checkpoints.

AuthorsOpenReview ↗

Response to JhCW (2/4)

W2&Q2: (W2) The comparisons cannot isolate the effect of outer-loop learning. For Qwen, the fixed-strategy and RSI-enabled runs cover 38 and 81 instances with different tasks, run histories, and implementations, so their success rates (65.8% and 71.6%) are not comparable. The GPT evolution results count a task as solved if any of several archived attempts succeeded, so the gain may come from extra attempts. In the PDF case, a hand-written adapter and a fixed rule that always requires replay pick the same intact helper, so nothing shows that a learned policy beats the fixed rule. The authors acknowledge these limits, but they affect the main contribution. Fixing them needs a paired comparison, not more unpaired success rates. (Q2) Is there a comparison on the same tasks, with the same implementation and total budget, that separates a learned Improvement Agent from fixed diagnosis, the replay rule, and extra retries? If not, which finished experiment supports outer-loop learning?

The added experiments use DeepSeek and compare the Hyperagents adaptation, frozen-strategy routing, and a local RSI configuration on the same task identifiers. Each condition is evaluated over the original attempt followed by one execution after improvement. Appendix E.2 thus compares recovery within a common two-round window. Appendix E.1 provides the internal diagnosis-and-routing ablations, while Appendices E.3–E.4 document the strategy change discussed in our response to Q1.

Each comparison addresses a different mechanism. Fixed diagnosis versus dynamic routing examines intervention selection. Frozen-strategy routing versus the local RSI configuration compares recovery outcomes. The candidate audit examines functional completeness. Restricting the analysis to two rounds excludes gains from third-round retries. Full cost accounting must additionally include diagnosis, proposal generation, and replay. Policy-activation records establish which strategy changes were used during execution.

Round-by-round success under fixed diagnosis and dynamic routing (Appendix E.1, excerpt)

ConditionFirst roundSecond roundFinal
D: Fixed diagnosis55.56%37.50%72.22%
F: Dynamic routing55.56%62.50%83.33%

The internal ablations evaluate the same selected instances from each of 18 task families, using shared task identifiers and inputs. First-round and final success rates use the full cohort in each condition. Second-round success uses only tasks entering that round. Skills, Memory, and intervention mechanisms follow their respective ablation definitions.

D and F have identical first-round success rates of 55.56%, followed by final rates of 72.22% and 83.33%. The difference is 11.11 percentage points. Their second-round rates are 37.50% and 62.50%, respectively. The difference therefore emerges during recovery rather than in first-round performance. The independent contribution of outer-loop learning requires a separate comparison of activated policies under matched budgets.

Recovery across three baseline configurations

We select initially unsuccessful tasks from the original task set and compare the same task identifiers across methods. The tasks cover court-form filling, dependency auditing, earthquake calculations, and video object counting. The reporting window contains the original attempt and one execution after improvement. First-round task success is 0.00% in each condition, and checkpoint success is 63.64%. The following comparison reports outcomes after the second round.

Task recovery within two rounds (Appendix E.2)

ConditionTask successCheckpoint success
Hyperagents task-domain adaptation16.67%63.64%
F: Dynamic routing, frozen strategy30.00%78.79%
G: Dynamic routing, local RSI configuration50.00%90.91%

F achieves 30.00% task success and 78.79% checkpoint success; G achieves 50.00% and 90.91%; the Hyperagents adaptation achieves 16.67% and 63.64%. The differences between G and F are 20.00 and 12.12 percentage points. Task success reflects complete recovery, while checkpoint success captures partial functional completion. The contribution of policy revision must be assessed together with its activation record.

Checkpoint success aggregates all named checkpoints. Appendix E.2 specifies each condition’s initial trajectories, Memory configuration, and comparison protocol.

The internal ablations show how recovery varies with the intervention mechanism. The external comparison reports task success and checkpoint success within two rounds. Isolating policy learning additionally requires matched total model-call budgets and evidence that the revised policy was activated.

AuthorsOpenReview ↗

Response to JhCW (3/4)

W2 (PDF case): In the PDF case, a hand-written adapter and a fixed rule that always requires replay pick the same intact helper, so nothing shows that a learned policy beats the fixed rule.

In the PDF case, the fixed replay rule and our mechanism select the same intact helper. They therefore agree on candidate acceptance. Policy learning concerns the proposal and revision of interventions, rather than changes to an already correct acceptance decision. We retain fixed functional acceptance criteria while optimizing the procedure used to meet them. The PDF case examines the basis for adoption. The policy-revision case examines how historical outcomes influence the direction of improvement.

  1. The replay rule determines whether a candidate is acceptable; the improvement policy determines what to do next. Replaying the two supplied helpers identifies the intact version through functional checks. LRSI makes a separate decision about whether the current failure calls for guidance, Memory, or Skill revision. Previous intervention outcomes then inform the next action. Different policies can use the same acceptance rule. Selecting the same helper does not establish which policy is better at producing effective interventions.

  2. The PDF case establishes the need for functional checks beyond the final reward. Both helpers receive a reward of 1. The intact helper passes 47 checks without skips. The defective helper passes 35 checks and skips 12. The final reward cannot distinguish these candidates; detailed replay outcomes can. The agreement between the hand-written adapter and the fixed rule shows that candidate selection can follow explicit acceptance criteria. We retain these criteria outside the adaptive policy so that strategy changes remain subject to the same quality checks.

  3. We provide separate execution evidence for policy revision. In the dependency-auditing case discussed in Q1, the revised candidate Skill achieves 80.00% checkpoint success with the original feedback. Strategy-shift feedback raises the rate to 100.00%. The change redirects the intervention from continued Skill rewriting toward source-evidence checks. The acceptance criteria remain unchanged. This local strategy evaluation is reported at the shadow-validation stage. A comparison that isolates its contribution should share the acceptance rule and match the starting state and total model-call budget. Subsequent actions and task outcomes would then provide the evaluation targets.

Functional completeness of the PDF candidates (Appendix D.1)

CandidateOriginal rewardPassed checksSkipped checks
Intact helper1470
Defective helper13512

The PDF audit establishes a functional basis for accepting persistent Skill updates when final rewards are identical. Acceptance checks assess the update’s reliability; intervention strategies determine how to produce an effective correction. LRSI therefore separates candidate generation from candidate adoption. The PDF values are from the case reported in the submitted manuscript.

Reviewer_WLMgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors claim existing rsi methods, when performing self-improvement, they do not laverage previous self-improvement attempts and their outcome. Then they propose LRSI which makes the self-improvement process itself learnable, which introduces an improvement Agent that diagnoses task failures and selects appropriate improvement actions to guide the Answer Agent. Furthermore, lrsi laverages the outcomes of previous improvement decisions to refine how future failures are diagnosed and how improvement actions are selected and executed.

Critical strengths

None

Critical weaknesses
  • I have major novelty concerns for this paper. According to the authors, LRSI has two main "novelties": Improvement Agent + self-improvement-replay. 1) What the authors call "Improvement Agent" is exactly the "diagnosis step" in DGM [1]/HGM [2]/RQGM [3], where the agent is given a task failure case and asked to diagnose and propose a self-improvement scheme based on it. So I do not understand what is novel here. Furthermore, the authors fail to acknowledge this overlap (even thought they cite DGM). 2) In my opinion, the one different part here and novel part here compared to DGM/HGM/RGGM, is that LRSI records the outcomes of these improvement decisions as improvement experience and then use it for refining how future failures are diagnosed and how improvement actions are selected. However, that's exactly what hyperagent [4] (extension of DGM does). In hyperagent, they propose a meta-agent which has access to the whole tree of the self-improvement and has the ability to modify itself (improving the improvement process).

Overall, I think the authors should reconsider their position in the literature and discuss/compare against relevant prior works. Current empirical results do not compare against any of [1-4].

[1] Zhang, Jenny, et al. "Darwin gödel machine: Open-ended evolution of self-improving agents." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Wang, Wenyi, et al. "Huxley-g" odel machine: Human-level coding agent development by an approximation of the optimal self-improving machine." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Iacob, Alex, et al. "The Red Queen G" odel Machine: Co-Evolving Agents and Their Evaluators." arXiv preprint arXiv:2606.26294 (2026).

[4] Zhang, Jenny, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. "Hyperagents." arXiv preprint arXiv:2603.19461 (2026).

Questions for authors

1- What is the difference of the "Improvement Agent" to the diagnosis step in DGM [1]/HGM [2]/RQGM [3]?

2- Why not compare against hyperagent which also allows for learning from previous self-improvement attempts?

Suggestions for future iteration:

In future iteration, I invite the authors to do more thorough literature review and cite/discuss all relevant works. I see there is potential to frame the replay in LRSI as a competitor to how hyperagent allows learning from previous self-improvement attempts. However, this would require require a full rewriting of the paper and more importantly extensive expirementations comparing both methods to show superiority.

4 comments
AuthorsOpenReview ↗

Response to WLMg (1/3)

Dear Reviewer WLMg,

Thank you for asking us to clarify the relationship to DGM, HGM, RQGM, and Hyperagents. Below, we distinguish their improvement decisions from LRSI’s intervention routing. We also report the added external adaptation and explain its evaluation scope.

Q1: What is the difference of the "Improvement Agent" to the diagnosis step in DGM [1]/HGM [2]/RQGM [3]?

These methods and LRSI all use failure analysis to propose improvements. Our focus is the intervention selected after diagnosis. DGM, HGM, and RQGM incorporate diagnosis into agent-program evolution or coevolution. LRSI explicitly routes interventions to execution guidance, Memory, or task Skills. It manages transient assistance separately from persistent updates. Our contribution concerns this decision mechanism, rather than failure analysis in isolation.

  1. DGM: program evolution versus intervention selection. DGM modifies agent code and explores an archive of variants. LRSI explicitly selects guidance, Memory, or Skill revision according to the failure cause, then validates persistent updates. The distinction concerns how improvement decisions are organized, not what code modifications can express.

  2. HGM: search allocation versus failure-specific routing. HGM uses descendant performance to select branches for further evolution. LRSI instead selects the intervention required by a particular failure. These decisions concern where to allocate search effort and how to correct an observed error, respectively.

  3. RQGM: evaluator coevolution versus fixed acceptance criteria. RQGM coevolves agents and learned evaluators. LRSI retains a fixed task Verifier and validates candidates through replay and regression checks. Local RSI can revise diagnosis, routing, or feedback strategies without changing task answers or acceptance criteria.

Consider an output artifact with a missing field. If the execution trace shows that the existing Skill already contains the required step, the router can provide execution guidance. If the reusable procedure lacks that step, it can generate and validate a candidate Skill. Distinguishing these cases allows transient execution errors and persistent knowledge gaps to receive different interventions. This is the Improvement Agent’s concrete responsibility beyond producing a diagnosis.

The improver itself uses Skills to carry out this process. Routing Skills organize action selection; action Skills specify how interventions are executed. Local RSI uses previous outcomes to propose revisions to these instructions. Such revisions can redirect improvement from task-Skill content toward feedback verification. The framework thus connects failure causes, intervention scope, and candidate adoption.

First-round and final success in the internal ablations (Appendix E.1, excerpt)

ConditionFirst roundFinal
D: Fixed diagnosis55.56%72.22%
E: Memory61.11%77.78%
F: Dynamic routing55.56%83.33%

The internal ablations evaluate the same selected instances from each of 18 task families, with shared task identifiers and inputs. First-round and final rates use the full cohort in each condition. Skills, Memory, and intervention mechanisms follow their ablation definitions; E starts with an empty Memory store.

In Appendix E.1, F increases from 55.56% first-round success to 83.33% final success. This is a gain of 27.78 percentage points. E increases from 61.11% to 77.78%. The final rates of D and F are 72.22% and 83.33%, respectively. Their difference is 11.11 percentage points. These internal comparisons support an analysis of intervention routing. The Hyperagents adaptation is evaluated in our response to Q2; complete model-call costs are accounted for separately.

The Improvement Agent turns diagnosis into a specific intervention, then validates persistent changes before adoption. This links failure attribution to actionable correction and makes the proposed updates auditable.

AuthorsOpenReview ↗

Response to WLMg (2/3)

Q2: Why not compare against hyperagent which also allows for learning from previous self-improvement attempts?

We have added a task-domain adaptation of Hyperagents to compare recovery under different improvement approaches. Hyperagents is directly relevant because it uses improvement history and permits the meta-level program itself to evolve. LRSI selects the intervention target after failure, then validates candidate updates before adoption. Appendix E.2 compares the Hyperagents adaptation with frozen-strategy routing and a local RSI configuration. This external comparison complements the internal intervention ablations discussed in Q1.

  1. Evaluation target. Our tasks produce file artifacts assessed through Verifier checkpoints. LRSI determines whether to change execution guidance, Memory, or Skills; local RSI additionally considers revisions to the associated decision and feedback rules.

  2. Comparison protocol. The reported comparison aligns task identifiers and uses the first two rounds. Strict budget matching additionally requires accounting for proposal generation, candidate replay, and task execution. Model and environment settings must also be controlled.

  3. External adaptation. We implemented the adaptation separately using editable task-agent and meta-agent program interfaces informed by the public Hyperagents implementation. Appendix E.2 defines its scope. Full reproduction and complete budget matching require separate verification.

Hyperagents integrates the task agent and meta-agent into an editable program, allowing the procedure that generates future improvements to evolve. LRSI explicitly connects failure attribution to interventions with different scopes: guidance, Memory updates, and Skill revision. The comparison examines task recovery under these different approaches to improvement. LRSI additionally records the validation evidence used for persistent adoption.

Recovery across three baseline configurations

The added experiments use DeepSeek and compare the same initially unsuccessful task identifiers across methods. The tasks cover court-form filling, dependency auditing, earthquake calculations, and video object counting. The reporting window is the original attempt followed by one execution after improvement. Appendix E.2 reports outcomes after the second round.

Task recovery within two rounds (Appendix E.2)

ConditionTask successCheckpoint success
Hyperagents task-domain adaptation16.67%63.64%
F: Dynamic routing, frozen strategy30.00%78.79%
G: Dynamic routing, local RSI configuration50.00%90.91%

F achieves 30.00% task success and 78.79% checkpoint success; G achieves 50.00% and 90.91%, respectively. Relative to the Hyperagents adaptation, F gains 13.33 percentage points in task success and 15.15 points in checkpoint success. G gains 33.33 and 27.27 points, respectively. Task completion and checkpoint coverage provide complementary views of recovery. The external comparator is the Hyperagents task-domain adaptation implemented for this project.

Checkpoint success is aggregated over all named checkpoints. Appendix E.2 specifies each condition’s initial trajectories, Memory configuration, and comparison protocol.

Both routing configurations outperform the adaptation on task success and checkpoint success in this evaluation. The local RSI configuration achieves the highest observed rates. These results complement the conceptual comparison with measured recovery outcomes. They support further evaluation of structured intervention selection with candidate validation.

AuthorsOpenReview ↗

Response to WLMg (3/3)

Suggestions for future iteration: In future iteration, I invite the authors to do more thorough literature review and cite/discuss all relevant works. I see there is potential to frame the replay in LRSI as a competitor to how hyperagent allows learning from previous self-improvement attempts. However, this would require require a full rewriting of the paper and more importantly extensive experimentations comparing both methods to show superiority.

We organize related work around what is modified, how improvement history is used, and what justifies adoption. This organization identifies the shared foundations with prior work and clarifies the decisions studied by LRSI. The revised discussion follows.

Revised related work: agent improvement and policy evolution

Agent self-improvement studies how execution experience can improve subsequent task performance. DGM, HGM, and RQGM connect failure diagnosis to proposals for changing agent behavior. LRSI focuses on the target of those changes. The same visible failure may reflect a missing Skill, unused knowledge, or execution that departs from existing guidance. The framework jointly examines evaluation feedback, execution traces, and Skill use. It then maps the diagnosis to guidance, Memory, or Skill updates with different scopes. Candidate validation requires execution evidence before a proposed correction becomes a persistent update.

A second line of research makes the improvement procedure itself editable. STOP, Promptbreeder, and Gödel Agent explore self-modification of that procedure or its components. Hyperagents integrates the task agent and meta-agent into one editable program. The meta-level procedure that generates future improvements can itself be modified. Hyperagents also examines the role of previous improvement experience. Within this line of research, LRSI targets the rules for diagnosis, routing, and feedback. Historical outcomes inform both task execution and the next choice of intervention. Once a candidate Skill produces partial gains, the improver can consider further content revision or a change in feedback organization. The policy change is recorded as a versioned candidate for validation. In the additional dependency-auditing strategy-shift run, the revised candidate achieves 80.00% checkpoint success with the original feedback. Strategy-shift feedback raises this rate to 100.00%. This case provides a concrete within-task outcome for policy revision. The policy remains at the shadow-validation stage.

LRSI combines targeted correction with validated adoption of persistent knowledge. Execution guidance addresses transient errors, whereas persistent updates add reusable knowledge. Replay and regression checks assess whether candidate updates satisfy functional requirements. Local RSI extends revision to the intervention policy itself. This separates correction of the current task from revision of the rules governing future interventions. Task outcomes, action records, and policy versions provide corresponding evaluation evidence.

This positioning identifies mechanisms that can be examined through their execution records. The external comparison measures recovery performance. The strategy-shift case records how historical outcomes inform a different intervention.

Thank you for your suggestions on positioning and external comparisons. We will clarify the shared foundations with prior work, the specific decision problem studied here, and the scope of the Hyperagents adaptation. These clarifications and evaluation details will be retained in the final version if the paper is accepted.

Best regards, The Authors

Thank you for the detailed response and additional experiments. I appreciate the effort to clarify the contribution and address the concerns about related work.

However, my main concerns regarding novelty and experimental validation remain unresolved.

1. Novelty relative to prior work

The main issue is that the paper overclaims in this part. What you call "Improvement Agent" is fundamentally the "diagnosis step" in previous works. I agree that your implementation is slightly different from DGM/HGM, which primarily focus on evolving agent programs and improving the search process. However, I am less convinced by the distinction from Hyperagents. Hyperagents already allows the meta-agent to decide what to modify (prompts, memory, tools, control flow, etc.), learn from previous improvement outcomes, and modify its own improvement procedure. In fact, learning how to improve the improvement process itself is one of its central contributions. LRSI's explicit routing between execution guidance, Memory, and Skill updates seems to be a more structured or constrained implementation of this general capability rather than a fundamentally new self-improvement mechanism. Hence, I do not yet see what new learning principle LRSI introduces beyond making these intervention choices explicit. This could still be a meaningful contribution if the structured routing mechanism leads to more reliable, efficient, or effective improvement. However, such advantages need to be demonstrated experimentally.

2. Evidence for learnable recursive self-improvement

The central claim of LRSI is that the agent learns how to improve its improvement process from previous experiences. However, I do not see convincing experimental evidence demonstrating that this learned improvement ability generalizes beyond the cases used to develop it. For example, Hyperagents explicitly evaluates this using improvement@k (imp@k) and cross-domain transfer experiments, showing that an evolved meta-agent is better at generating improvements on unseen tasks than the original meta-agent. I think a similar evaluation is essential here. Specifically, can you demonstrate that an autonomously evolved improvement policy (I_1) performs better than the original policy (I_0) on held-out tasks, under the same initial conditions and improvement budget? This would directly test whether LRSI actually learns a better improvement mechanism, rather than simply improving task execution through additional feedback, memory, or retries.
Currently, the dependency-auditing example remains in shadow validation, and the additional recovery experiments do not isolate the benefit of an evolved improvement policy. Therefore, I do not think the central claim of learnable RSI is experimentally established yet.

3 Minor point: Comparison with Hyperagents: As acknowledged in Appendix E.2, the Hyperagents baseline is a task-domain adaptation rather than a full reproduction. More importantly, the experimental conditions are not fully matched in terms of initial trajectories, Memory configurations, and computational budgets. Therefore, I do not think the reported performance differences provide sufficient evidence that LRSI outperforms Hyperagents or that its proposed intervention-routing mechanism is superior, which is very important here as Hyperagent is a direct competing method.

For these reasons, I would maintain my original rating. I believe the paper would require more rethinking the novelty and the position in the literature. Furthermore, it needs substantial experimental validation, particularly of the meta-improvement mechanism.

Reviewer_5ge4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LRSI, which aims to improve both an agent's task-solving ability and its improvement process. The framework uses an Improvement Agent to diagnose failures, select appropriate actions, and validate skill updates. Experiments on SkillLearnBench report gains over fixed self-evolution baselines.

Critical strengths
  1. The paper addresses an important problem: deciding how an agent should improve based on previous failures.

  2. The failure attribution mechanism and replay-based validation provide useful safeguards against inappropriate skill updates.

Critical weaknesses
  • The experiments only compare against the authors' own baselines, without evaluating existing harness-improvement methods.

  • Their comparisons are not well controlled. For example, fixed evolution and RSI-enabled evolution use different numbers of Qwen instances (38 vs. 81), with different execution budgets and runtime settings.

  • The method relies heavily on prompts and skill instructions, it is also unclear whether the gains come from the proposed method or simply from better prompts. A strong fixed-prompt baseline would be particularly helpful here.

  • The paper claims that the Improvement Agent can learn to improve its own improvement policy, but I do not think the experiments establish this. Most results concern failure diagnosis and update validation.

  • I find it difficult to identify what is fundamentally new in LRSI compared to these approaches. The two-level self-improvement idea has already been explored in prior work, such as Promptbreeder (ICML 2024) and Gödel Agent (ACL 2025).

Questions for authors

Can the authors compare against existing self-improvement methods and stronger baselines under matched budgets?

4 comments
AuthorsOpenReview ↗

Response to 5ge4 (1/4)

Dear Reviewer 5ge4,

Thank you for recognizing the value of failure-informed intervention selection and replay-based safeguards. We address your concerns about baseline fairness, the contribution of routing, and evidence for policy revision below.

W1&W2&Q1: (W1) The experiments only compare against the authors' own baselines, without evaluating existing harness-improvement methods. (W2) Their comparisons are not well controlled. For example, fixed evolution and RSI-enabled evolution use different numbers of Qwen instances (38 vs. 81), with different execution budgets and runtime settings. (Q1) Can the authors compare against existing self-improvement methods and stronger baselines under matched budgets?

We have added a task-domain adaptation based on the public Hyperagents implementation, extending the comparison to an external self-improvement approach. We focus on tasks that fail on the first attempt. These tasks provide an opportunity to examine how each improvement mechanism supports recovery. We report both task success and checkpoint success to distinguish complete recovery from partial functional improvement.

To address budget fairness, we report internal ablations on a shared task cohort separately from recovery within a common two-round window. The added comparison aligns task identifiers and includes only the first two rounds. The A–F ablations are presented separately to keep the evaluation populations distinct. Total cost includes diagnosis, proposal generation, candidate replay, and task execution. We distinguish this accounting from the common limit on execution rounds. Appendix E.2 specifies the initial trajectories, Memory configurations, and comparison protocol.

Recovery across three baseline configurations

The added experiments use DeepSeek. We select initially unsuccessful tasks from the original task set and compare the same task identifiers across all three methods. The tasks cover court-form filling, dependency auditing, earthquake calculations, and video object counting. The reporting window consists of the original attempt followed by one execution after improvement. Appendix E.2 reports the outcomes after the second round.

Task recovery within two rounds (Appendix E.2)

ConditionTask successCheckpoint success
Hyperagents task-domain adaptation16.67%63.64%
F: Dynamic routing, frozen strategy30.00%78.79%
G: Dynamic routing, local RSI configuration50.00%90.91%

Both routing configurations achieve higher task success and checkpoint success than the Hyperagents adaptation. F reaches 30.00% task success and 78.79% checkpoint success. G reaches 50.00% and 90.91%, respectively. Checkpoint success captures partial repairs that task-level success alone would miss. The stronger fixed-prompt control will test gains beyond comprehensive instructions. Full model-call accounting will address the separate question of budget matching.

The external adaptation extends the comparison beyond our internal baselines. The ablations examine individual intervention mechanisms, while the recovery comparison evaluates their task-level outcomes.

AuthorsOpenReview ↗

Response to 5ge4 (2/4)

W3: The method relies heavily on prompts and skill instructions, it is also unclear whether the gains come from the proposed method or simply from better prompts. A strong fixed-prompt baseline would be particularly helpful here.

Our technical claim concerns intervention selection based on failure evidence. We examine it by comparing fixed guidance with dynamic routing. LRSI goes beyond supplying more detailed instructions. It selects whether to intervene in execution, stored experience, or reusable task Skills. It also validates persistent changes. These interventions have distinct scopes. Action logs record the selected intervention. Candidate contents and next-round loading records show which changes reach the Answer Agent.

  1. Fixed task support. Condition C provides a fixed task prompt, whereas B provides human-authored task Skills. These conditions distinguish generic instructions from structured task knowledge and establish reference levels of initial task performance.

  2. Fixed diagnosis versus dynamic routing. D and F both achieve 55.56% first-round success, but their final rates are 72.22% and 83.33%, respectively. The difference emerges after failure, directing attention to the recovery procedure rather than the initial prompt alone.

  3. We will further control prompt content and model-call budgets. This control will freeze comprehensive diagnostic and intervention rules before evaluation. It will match the information available to dynamic routing and its model-call budget. Condition C remains a separate fixed-task-prompt baseline.

Success rates across the A–F ablations (Appendix E.1)

ConditionFirst roundSecond roundFinal
A: No task Skill16.67%—16.67%
B: Human-authored task Skill61.11%—61.11%
C: Fixed task prompt27.78%—27.78%
D: Fixed diagnosis55.56%37.50%72.22%
E: Memory interface61.11%42.86%77.78%
F: Dynamic routing55.56%62.50%83.33%

The internal ablations evaluate the same selected instances from each of 18 task families. Task identifiers and inputs are shared across conditions. First-round and final rates use the full task cohort in each condition; second-round rates use the tasks entering that round. Skills, Memory, and intervention mechanisms follow the respective ablation definitions. A dash denotes an unreported second-round result.

Human-authored task Skills achieve 61.11% first-round success. The corresponding rates are 16.67% without task Skills and 27.78% with a fixed task prompt. These results show the value of reusable task guidance in this evaluation. Dynamic routing reaches 83.33% final success, gaining 27.78 percentage points over its first round. It exceeds fixed diagnosis by 11.11 percentage points at the final round.

These comparisons distinguish initial task support from recovery after failure. The proposed fixed-prompt control will test dynamic selection when both conditions receive comprehensive intervention rules.

AuthorsOpenReview ↗

Response to 5ge4 (3/4)

W4: The paper claims that the Improvement Agent can learn to improve its own improvement policy, but I do not think the experiments establish this. Most results concern failure diagnosis and update validation.

Local recursive self-improvement (RSI) revises the policy for choosing the next intervention based on previous intervention outcomes. This introduces a decision beyond repeatedly rewriting task Skills. The dependency-auditing case records a candidate policy that retains the useful Skill but redirects feedback toward verification against source evidence. Shadow execution records how this revised procedure operates.

  1. The revised object is the rule for selecting the next intervention. A task-level update adds procedures for checking vulnerabilities. A policy-level update instead determines when to revise those procedures or retain them while changing the feedback. It therefore changes how the improver chooses its next action.

  2. Previous intervention outcomes motivate the policy change. In the dependency-auditing development record, retrying with the original feedback reaches 60.00% checkpoint success. The original candidate Skill with that feedback reaches 80.00%. A revised candidate with the original feedback remains at 80.00%, whereas strategy-shift feedback with the revised candidate reaches 100.00%. This sequence motivates retaining the useful candidate while reconsidering the intervention strategy. We distinguish the proposal, shadow evaluation, and formal adoption in the execution record.

  3. A frozen strategy provides the control for policy revision. In this control design, the candidate Skill, starting artifacts, and total budget remain fixed. The frozen strategy retains its rules. The history-informed strategy proposes rule changes from previous outcomes, then validates them. Subsequent actions and checkpoint outcomes provide the measures of decision quality.

From revising task knowledge to revising the improvement procedure

Appendix E.3 records the change in the improver’s policy. Task-level Skill revision changes vulnerability-checking knowledge. Policy revision changes the allocation of the next improvement effort. When CSV-set consistency remains unresolved after Skill revision, the system triggers a candidate policy that redirects the intervention toward source-evidence checks in the feedback.

Local RSI policy revision and execution record (Appendix E.3)

StageRule or actionVersion and execution evidence
Identify the adjustment signalCheck residual CSV-set consistency after Skill revisiondecision.json: triggered=true
Change the interventionShift from Skill rewriting to source-evidence checksfeedback_shift_after_skill_stall_v1
Validate the candidateEnter shadow evaluationactivation=candidate_shadow_test_only
Record execution outcomesOriginal feedback: 80.00%; strategy-shift feedback: 100.00% (CTRF)Additional strategy-shift run

The record links the unresolved issue to a versioned policy proposal. It identifies the validation stage and reports the subsequent checkpoint outcomes. These entries make the strategy change traceable from its trigger to execution.

The checkpoint sequence also clarifies the practical motivation. The original candidate raises success from 60.00% to 80.00%, but further candidate revision with unchanged feedback remains at 80.00%. Using strategy-shift feedback with the revised candidate raises success to 100.00%, a gain of 20.00 percentage points. In this case, changing the feedback’s source-checking requirements offers a productive direction after Skill-content revision stalls.

This case documents a history-informed strategy change through a versioned proposal and shadow execution. It extends revision from task Skills to the rules governing their use. Formal adoption and cross-task evaluation are separate stages of this process.

AuthorsOpenReview ↗

Response to 5ge4 (4/4)

W5: I find it difficult to identify what is fundamentally new in LRSI compared to these approaches. The two-level self-improvement idea has already been explored in prior work, such as Promptbreeder (ICML 2024) and Gödel Agent (ACL 2025).

Our specific contribution is an executable mechanism that connects failure attribution to intervention selection and candidate validation. Its improvement policy is itself open to revision. Promptbreeder and Gödel Agent are important precedents for two-level improvement. Our contribution lies in the specific decision mechanism and its experimentally observable behavior, rather than priority for the general two-level idea.

The formulation separates task-level intervention from revision of the improvement policy. The task level determines which intervention the current failure requires. The policy level determines how future failures should be diagnosed and addressed. Each level can be examined through its inputs, decisions, and acceptance records. Action logs and subsequent loading records show how policy decisions affect task execution.

Three design choices make this formulation operational. First, failure attribution identifies the intervention target, distinguishing execution errors from gaps in reusable procedures. Second, routing accounts for the different scopes of transient guidance and persistent updates. Third, replay and acceptance checks govern subsequent use of candidate updates, while their observed effects inform later policy decisions.

The contribution is the link between failure-informed routing and validated adoption. Intervention history then provides input for revising the policy itself. This formulation specifies what changes at each level and how those changes can be evaluated.

Thank you for these suggestions. We will bring the external adaptation comparison and round-by-round ablations together in the revised manuscript. We will specify the evidence for prompt support, intervention routing, and policy evolution. We will also document the experimental configurations and budget accounting. If accepted, the final version will use consistent definitions and reporting conventions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.