acceptodds
Withdrawn from ICLR 2027

Self-Improving Skills from Failure: Learning to Recover, Remember, and Evolve

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent frameworks increasingly package procedural knowledge as reusable skills that are selected from a skill library and executed on demand. Yet even a correctly selected skill may fail in execution, and such failures are typically treated as terminal outcomes rather than reused for improvement. We propose a skill-augmented agent system that learns from failures through reflective recovery, reusable case memory, and persistent skill revision, enabling the model and skill library to co-evolve through interaction. Optimizing this system is challenging because outcome-level reinforcement learning cannot distinguish credit across skill selection, execution, and reflection. We therefore introduce FIRE (Failure-Informed Reinforcement and Evolution), which performs stage-aware credit assignment for these components. Across four in-distribution task families and three out-of-distribution benchmarks, FIRE achieves strong skill-routing and task performance across Qwen3-4B and Llama-3.1-8B. With Qwen3-4B, FIRE reaches Skill Hit, Task Success, and Joint Success in-distribution, together with average OOD Task Success.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill-Augmented Agents, Reinforcement Learning, Learning from Failure, Reflection, Skill Evolution, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Up1NOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FIRE, a framework that improves skill-based agents by applying reinforcement learning separately to three stages: skill selection, task execution, and reflection on failures. Experiments across multiple environments and base models demonstrate the effectiveness of the approach.

Critical strengths
  • The paper addresses an important problem and proposes a logically motivated solution.
  • The experimental evaluation covers a broad range of environments and base models.
Critical weaknesses
  • Limited novelty. I do not see a clear methodological advance over prior work. First, the formulation of skill-based agents is widely used in existing work on agent skills and memory, and does not appear novel. Second, the select–execute–reflect pipeline is also common in research on agentic systems and recursive self-improvement. Furthermore, separate advantage estimation and reinforcement learning for different stages have already been explored in works such as SkillMaster: Toward Autonomous Skill Mastery in LLM Agents and SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents. The main contribution therefore appears to be combining a three-stage select–execute–reflect formulation with the training approaches used in SkillMaster and SkillGate. In my view, this combination alone does not constitute a sufficiently substantial contribution.
  • Unclear experimental configurations and inconsistent results. My other major concern is the lack of clarity regarding the experimental settings and the consistency of the reported results. The method section primarily describes a three-channel formulation, whereas Table 2(a) reports results for a four-channel variant. It is unclear how this fourth channel is defined and which configuration is used in each experiment. Moreover, the four-channel results in Table 2(a) match those of “FIRE w/o M&R” in Table 1, rather than those of the full FIRE method. This correspondence requires explanation. Table 2(d), meanwhile, is described as reporting three-channel results. Its Skill Hit value matches that of FIRE in Table 1, but the other two metric values differ. These discrepancies make it difficult to determine which configurations are being compared and to assess the experimental evidence reliably. (If this concern can be fully resolved, I will raise my score to 2.)
Questions for authors
  • Could the authors report results for the “w Mem., w/o Rev.” configuration in Table 2(d)?
  • Is the Task Success value reported for the Qwen3 base model in Table 1 correct?
  • How is the fourth channel defined, and which channel configuration and other settings are used in each experiment?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Up1N: Clarifying FIRE’s Novelty and Experimental Consistency

We thank the reviewer for the careful assessment and for identifying ambiguities in our experimental reporting. We address the concerns below.

W1. Methodological Novelty Beyond Existing Skill-Based Agents

We agree that skill-augmented agents and the Select–Execute–Reflect pipeline are not novel by themselves. FIRE's contribution lies in failure-conditioned, stage-aware credit assignment. Unlike SkillGate's selection/execution separation or SkillMaster's task-solving/skill-editing advantages, FIRE explicitly optimizes failure-triggered recovery decisions: KEEP is rewarded by verified recovery success, while REPLACE is rewarded by improvement in skill-routing quality. FIRE further distinguishes initial and recovery execution through independently estimated advantages, while retaining failure evidence through reusable memory and persistent skill revision. In Table 2(a), separating execution advantages improves Task Success from 87.3% to 90.8% and Joint Success from 74.2% to 79.5%. We will clarify these methodological distinctions and strengthen the discussion of prior work.

W2 & Q3. Clarification of Channel Configurations and Table Consistency

We apologize for the insufficient explanation. FIRE has three semantic stages (Select, Execute, Reflect) but can use four independently estimated advantage channels by separating initial execution (Exec1) and recovery execution (Exec2). In the three-channel variant, both executions share one advantage; in the four-channel variant, they receive independently normalized advantages.

The configurations are as follows:

  • Table 1: FIRE and FIRE w/o M&R use four-channel optimization, with memory and revision enabled or disabled, respectively.
  • Table 2(a): Explicitly compares three- and four-channel variants without memory or revision.
  • Table 2(b–c): Use four-channel optimization without memory or revision.
  • Table 2(d): Uses three-channel optimization consistently across memory/revision ablations.

Consequently, Table 2(a)'s All (4-ch.) correctly matches Table 1's FIRE w/o M&R (86.9/90.8/79.5). Similarly, Table 2(d)'s full memory-and-revision configuration (95.1/89.0/85.9) differs from Table 1's full FIRE (95.1/92.2/88.0) because the optimization configurations differ. The identical Skill Hit values do not imply identical execution results.

We will explicitly document the channel assignments, component configurations, checkpoint identities, and evaluation settings to make these comparisons unambiguous.

Q1. Additional Memory-Only Ablation

Following the reviewer's suggestion, we additionally evaluated w Mem., w/o Rev. under Table 2(d)'s three-channel setting, obtaining 91.0% Skill Hit, 87.5% Task Success, and 81.0% Joint Success.

Adding skill revision improves these metrics to 95.1%, 89.0%, and 85.9%, respectively, yielding gains of +4.1, +1.5, and +4.9 percentage points. In contrast, revision without memory does not improve Joint Success (74.2% to 73.9%). These results suggest that persistent skill revision is more effective when supported by accumulated case experience. We will include the missing ablation in Table 2(d).

Q2. Correction of Qwen3 Base-Model Task Success

We thank the reviewer for identifying a reporting error in Table 1. The Qwen3 Base Model's ID Task Success of 76.0% is correct. However, its reported OOD aggregate Task Success of 4.0% is incorrect; 4.0% corresponds to the SkillEval result alone.

Following our stated evaluation protocol, the correct OOD Task Success is the macro-average of BigCodeBench (51.7%) and SkillEval (4.0%), giving 27.9%. We will correct this entry. This correction does not change FIRE's leading OOD Task Success (35.4%) among the compared methods.

Thank you for the reply. The experimental setup is now clearer, so I will raise my score to 2. The additional Reflect stage that FIRE introduces compared with SkillGate is indeed a novel contribution, although I still consider the overall novelty relatively limited.

Reviewer_h1hpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes FIRE, a skill-based agent system that learns from failures through reflection, case memory and skill revision. it also uses stage-aware RL to separately optimize skill selection, execution and recovery. experiments on two models show improvements in skill routing and task success, including some OOD gains.

Critical strengths
  1. separating the learning signals for skill selection, execution and reflection makes sense. the ablations on credit assignment are useful.
  2. the results are generally promising, especially for skill routing and OOD performance.
Critical weaknesses
  1. the method relies heavily on ground-truth skill labels during training, both for reward calculation and case memory. but in real agent tasks we usually dont know which skill is the correct one beforehand. this makes me question how practical the proposed learning method really is.
  2. a key ablation is missing. Table 2(d) compares no memory/no revision, revision only, and both together, but never tests memory only. interestingly, revision alone even slightly hurts performance. so its unclear whether skill revision actually contributes anything beyond case memory, which is a pretty central claim of this paper.
Questions for authors
  1. how would FIRE work without ground-truth skill annotations? can the same credit assignment be learned from task outcomes alone?
  2. can you add the memory-only ablation? i think this is necessary to show that persistent skill revision actually helps.
1 comment
AuthorsOpenReview ↗

Response to Reviewer h1hp: Skill Supervision and Additional Ablation Evidence

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We address the two main concerns below.

W1 & Q1. Clarification on Ground-Truth Skill Supervision and Outcome-Only Learning.

First, we acknowledge the dependence on ground-truth skill labels and clarify its scope. As discussed in our Limitations and Future Work section, the current FIRE formulation uses skill-level supervision to provide reliable credit assignment. Specifically, in Eq. (3), the selection reward and the REPLACE reflection reward depend on ground-truth skill sets, while execution rewards and KEEP reflection rewards are computed directly from task outcomes using deterministic verifiers. Skill annotations also inform case-memory construction and skill revision. Our primary goal is to investigate stage-aware credit assignment under reliable supervision, rather than claim that FIRE already supports fully annotation-free training.

Second, our OOD results demonstrate that the learned capabilities generalize beyond the supervised training tasks. Importantly, ground-truth skill identities for individual evaluation tasks are not provided to the policy during inference. As shown in Table 1, FIRE achieves the highest OOD Task Success among the compared methods on both Qwen3-4B (35.4%) and Llama-3.1-8B (19.4%).

More importantly, even without Case Memory and Skill Revision, FIRE w/o M&R consistently outperforms GRPO on both OOD Skill Hit and Task Success, as shown below.

BackboneOOD Skill Hit (%)OOD Task Success (%)
Qwen3-4B73.0 → 85.5 (+12.5)27.0 → 32.0 (+5.0)
Llama-3.1-8B77.8 → 82.2 (+4.4)13.2 → 16.7 (+3.5)

Values represent GRPO → FIRE w/o M&R, with improvements in percentage points.

These results indicate that FIRE's benefits are not solely attributable to external memory or skill library updates. The trained policy itself exhibits improved generalization in skill selection and task execution. While this does not eliminate the current training-time supervision requirement, it demonstrates the practical value of the learned policy beyond the annotated training distribution.

W2 & Q2. Additional Memory-Only Ablation and the Contribution of Skill Revision.

We agree that the memory-only ablation is essential for isolating the contribution of persistent skill revision. Following the reviewer's suggestion, we have conducted the missing experiment under the ablation setting of Table 2(d). The complete results are reported below.

Case MemorySkill RevisionSkill Hit (%)Task Success (%)Joint Success (%)
✗✗85.287.374.2
✗✓84.186.273.9
✓✗91.087.581.0
✓✓95.189.085.9
Additional gain from revision (with memory)+4.1+1.5+4.9

First, persistent skill revision provides clear additional benefits beyond case memory. Compared with the memory-only variant, enabling skill revision improves Skill Hit, Task Success, and Joint Success by 4.1, 1.5, and 4.9 percentage points, respectively. This directly addresses the reviewer's concern by demonstrating that the full system's improvements cannot be attributed to case memory alone.

Second, the results reveal a positive interaction between case memory and skill revision. We acknowledge that revision alone does not improve overall performance, yielding slightly lower Joint Success than the variant without either component (73.9% vs. 74.2%). However, when case memory is enabled, the effect of revision changes from −0.3 to +4.9 percentage points, corresponding to a positive interaction contrast of +5.2 percentage points.

One possible explanation is that skill revision converts recurring failures into persistent, general-purpose guidance, which may not always be sufficient when applied independently. Case memory provides relevant historical routing and execution experience that may help the agent apply these revised skills more appropriately. Although this mechanism requires further investigation, the complete ablation supports the complementary roles of the two components.

Therefore, our claim is not that skill revision independently improves performance in every setting, but that it provides measurable additional benefits when combined with case memory. Together, these mechanisms support more effective accumulation and reuse of failure-derived experience.

We thank the reviewer again for these suggestions. We will incorporate the complete ablation results, clarify the supervision requirements, and refine the discussion of the complementary roles of case memory and skill revision in the revised manuscript.

Reviewer_wfV5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FIRE (Failure-Informed Reinforcement and Evolution), which uses stage-aware credit assignment to provide separate learning signals for skill selection, execution, and reflection of the agent system. Substantial experiments are carried out, covering two model backbones (Qwen3-4B-Instruct and Llama-3.1-8B-Instruct) on four in-distribution (ID) and three out-of-distribution (OOD) benchmarks.

Critical strengths
  1. The studied problem is meaningful and the proposed method is technically sound. Since selecting appropriate skills does not guarantee successful execution, decoupling and assigning various learning signals to different decisions is intuitively and technically sound.
  2. The conducted experiments and the analysis are somewhat solid and promising. The experiments are conducted with two model backbones (Qwen3-4B-Instruct and Llama-3.1-8B-Instruct) on four in-distribution and three out-of-distribution benchmarks, demonstrating FIRE achieves better or comparable performance to the studied baselines. The reported ablations also provide promising evidence for fine-grained credit assignment and structured reflection.
Critical weaknesses
  1. The first concern I have is that the paper overstates the research gap, and the system-level novelty of this work is not sufficiently distinguished from existing failure-driven learning frameworks. The current motivation relies on the claim that recovery, experience reuse, and skill improvement remain ``largely disconnected''. However, the relevant prior work already connects these functions: (a). corrective attempts can produce persistent, reusable experience [1,2]. (b). failure evidence can drive lasting modifications to reusable skills [4,5]. (c). experience or skill evolution can be coupled with policy learning [3,5]. These are connections between components, not just isolated precedents for reflection, memory, or skill libraries. Besides, SkillGate[6] already separates selection and execution credit at the token level and balances their optimization weights. Therefore, the current discussion understates the overlap with existing approaches and makes the claimed research gap broader than the facts. The authors are encouraged to correctly discuss the connection and distinction with prior work to establish the contribution of FIRE. For instance, state concrete limitations of previous work and explain what new mechanism in FIRE overcomes them. In the current form, the novelty and contribution claims of this work should be narrowed and grounded in FIRE's specific coupling or optimization design.
  2. The contribution of the proposed skill revision is not sufficiently supported by the current empirical evidence. The performance of the memory-only variant is missing from Table 2 (d). Therefore, current Table 2 (d) demosntrates that adding memory to a skill revision system improves performance, but whether the skill revision helps with memroy-enabled system or skill revision and memory are complementary is unclear. Besides, the missing experiment also matters because the historical questions and their gold skill IDs are included in the prompts. Thus, higher Skill Hit could reflect retrieval of labeled examples rather than benefits from skill revision.
  3. Several important missing and incorrect experimental details should be elaborated and corrected to make the paper more convincing. First, it is unclear whether the comparisons is fair and preserve the baselines' mechanisms to isolate FIRE's recovery benefits. For instance, the paper mentions both SkillRL and adapted SkillRL with skill revision disabled, but Table 1 reports only an undefined SkillRL* row. The authors should clarify which version is evaluated. Outperforming a revision-disabled variant would not establish an advantage over the complete skill-evolution method. Similarly, SkillGate + Retry is described as adding a post-failure retry. However, with Qwen3-4B-Instruct, Skill Hit drops from SkillGate's 86.6% to the Retry variant's 70.7%. A retry alone should not change initial selection if the first-attempt outputs are fixed. More importantly, FIRE receives explicit correctness feedback during inference, not just additional computation. For valid yes/no answers in LogicBench, reliable negative feedback already identifies the correct alternative without further reasoning. This could make the comparison between FIRE and most baselines unfair. The authors should clarify these settings and report inference costs and verifier access. Otherwise, improvements cannot be attributed solely to FIRE’s recovery mechanism. Moreover, the performance of the Oracle-Skill row is also missing when using Llama-3.1-8B-Instruct as the backbone. The paper also has many minor issues and requires careful proofreading. For instance, the second-best Qwen OOD Skill Hit should be Base Model: 84.0, not FIRE: 83.4, the second-best Llama ID Task Success should be SkillGate + Retry: 70.32, not FIRE: 69.96, the † and * markers are also undefined, and so on.

References

  1. Zhao et al., ExpeL: LLM Agents Are Experiential Learners, Aug 2023
  2. Cao et al. Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution, Dec 2025
  3. Wu et al., EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle, Oct 2025
  4. Shi et al., Evolving Programmatic Skill Networks, Jan 2026
  5. Xia et al., SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning, Feb 2026
  6. Li et al., SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents, Aug 2026
Questions for authors

Please refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wfV5: Clarifying FIRE’s Contributions, Additional Ablation Evidence, and Experimental Settings

We thank the reviewer for the constructive feedback and careful evaluation of our work. We address the three concerns below.

W1 & Q1. Novelty and Contributions of FIRE.

We agree that prior work already connects failure recovery, experience reuse, skill evolution, and policy learning, and will revise the overly broad claim that these functions remain "largely disconnected." FIRE's contribution is not the first integration of these components, but a failure-driven self-improvement framework with stage-aware optimization.

First, FIRE connects reflective recovery, reusable case memory, and persistent skill revision, turning evaluated failures into both immediate recovery and reusable guidance for future interactions. Second, FIRE jointly improves the policy and skill library: differentiated rewards optimize skill selection, execution, and reflection, while accumulated routing and execution failures guide revisions to skill descriptions and contents. In particular, its KEEP/REPLACE recovery credit and separate advantages for initial and recovery execution extend beyond selection–execution credit alone. Third, experiments across ID/OOD tasks and two backbones, together with controlled ablations, support the effectiveness of these designs. We will clarify these specific contributions relative to SkillGate, SkillRL, and other related frameworks.

W2 & Q2. Additional Memory-Only Ablation and the Contribution of Skill Revision.

We agree that the missing memory-only ablation is essential. Following the reviewer's suggestion, we conducted this experiment under the Table 2(d) setting:

Case MemorySkill RevisionSkill Hit (%)Task Success (%)Joint Success (%)
✗✗85.287.374.2
✗✓84.186.273.9
✓✗91.087.581.0
✓✓95.189.085.9
Gain from revision with memory+4.1+1.5+4.9

Revision provides incremental benefits beyond memory. Compared with memory only, enabling revision raises Skill Hit, Task Success, and Joint Success by 4.1, 1.5, and 4.9 percentage points, respectively. The two components also show a positive interaction: revision alone changes Joint Success from 74.2% to 73.9% (−0.3 pp), whereas revision with memory improves it from 81.0% to 85.9% (+4.9 pp), an interaction contrast of +5.2 pp. Thus, we do not claim that revision is independently effective in every setting; the evidence supports its complementary benefit with case memory. We also acknowledge that historical gold skill IDs in retrieved training cases provide routing supervision, and will clarify this dependence rather than claim annotation-free improvement.

W3 & Q3. Clarification of Experimental Settings and Reporting.

We thank the reviewer for carefully identifying these issues. We acknowledge that several experimental settings and reporting details require further clarification. In particular, we will clarify the evaluated SkillRL variant and examine the discrepancy in the SkillGate + Retry results, since an additional retry should not alter initial skill selection under identical conditions.

We also acknowledge that FIRE uses explicit correctness feedback during recovery, which can provide particularly strong information for binary-answer tasks. We will clarify this verifier-assisted setting, its implications for baseline comparisons, and the limitations of attributing performance gains solely to the recovery mechanism.

Finally, we will correct the table annotations and highlighting errors, clarify the missing oracle results, and more carefully discuss inference-cost comparisons. We appreciate the reviewer's detailed suggestions and will improve the experimental reporting and discussion of limitations in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.