acceptodds
Under review as a conference paper at ICLR 2027

Textual Catastrophic Forgetting in Self-Evolving Agent Skill Libraries

Abstract

Self-evolving LLM agents externalize competence into textual skill libraries that grow with every solved task. We identify a previously uncharacterized failure mode—textual catastrophic forgetting (TCF): agents lose mastered competence not through weight drift but because the library itself degrades. We make three contributions. (i) We decompose TCF into three independently measurable channels—retrieval miss, semantic rot, and execution regression—operationalized in TCF-Bench, a four-phase leave-and-return protocol with coupled API and domain drift across five task domains. (ii) In accumulating libraries the dominant forgetting channel undergoes a drift-driven phase transition whose critical size shrinks with drift pressure (from ≈62 skills under mild drift to ≈33 under heavy drift; Pearson r = −0.85 over the 21 seed×severity points, Spearman ρ = −0.96 over the 7 severity means): under heavy drift a library is born already in the semantic-rot regime—the asymptotic bottleneck of uncurated growth. (iii) An ecological governance framework coordinating six load-bearing mechanisms (carrying-capacity admission control, stability shielding, niche-focused foraging, viability ranking, regeneration, and consolidation) plus prediction-error-gated reconsolidation—whose A/B-verified repair closes an autocatalytic contamination loop—reduces forgetting by 81% (TCF 0.036 vs. 0.186), attains the highest task success (0.667 vs. 0.140) and regression-probe success (0.755 vs. 0.159) against the strongest of the re-implemented published baselines (UtilityCull), and beats every baseline on all three metrics with Holm–Bonferroni-corrected paired tests (all p < 10⁻⁴). We also replicate the protocol on live inference on 5 served LLMs (6,901 API calls, 8.1M tokens, programmatic ground truth): bloat control transfers, the aggregate TCF advantage does not—a negative result reported in the external-validity section. A twenty-variant ablation and a carrying-capacity sweep confirm channel–mechanism correspondence as a graded dose-response. Our results reframe skill-library forgetting from an observed pathology into a measurable and governable process.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
textual catastrophic forgetting, self-evolving LLM agents, skill libraries, continual learning, TCF-Bench, ecological governance, external validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4Gu4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper names "textual catastrophic forgetting", where a self-evolving agent loses mastered skills because its text skill library degrades, and splits it into retrieval miss, semantic rot and execution regression. It introduces TCF-Bench, a four-phase leave-and-return protocol with injected API and domain drift, and EcoForge, a governance framework that caps skills per family, filters retrieval, shields key skills and repairs broken dependencies. With a simulated executor, EcoForge reports a TCF of 0.036 against 0.186 for the best baseline; in a live check on five LLMs it bounds same-family growth but does not reproduce the TCF advantage.

Critical strengths
  1. The problem is real and timely. Self-evolving agents that grow skill libraries are an active direction, and library degradation is a practical risk that the cited concurrent work also reports.
Critical weaknesses
  1. The novelty over existing work is limited. The cited concurrent work already identifies this failure mode: Yu et al. (2026) study capability degradation across evolution channels including skills, Library Drift addresses unbounded accumulation with retirement and a cap, and Fan et al. (2026) repair skills broken by API change. Further uncited work validates, repairs, merges and prunes libraries (SkillFoundry, https://arxiv.org/abs/2604.03964), gates and maintains skills by verification and score (Skill-Pro, https://arxiv.org/abs/2602.01869), and diagnoses library defects that harm retrieval, composition and execution (SkillOps, https://arxiv.org/abs/2605.13716), so the three channels and most EcoForge mechanisms overlap substantially with these ideas.
  2. The experiments do not yet support the main claims. The main results use a simulated executor with no LLM, so outcomes follow the paper's own formula: Section 7 attributes about 53 percent of the cap's effect to the same-family penalty the cap removes, and the simulator injects the broken dependencies that EcoForge's repair step patches. In the live check, with 2 to 6 paired cells and 20 tasks per phase, EcoForge forgets slightly more than plain accumulation on every model, which itself forgets little (TCF 0.025 to 0.046), and the five domains are not described in the main body.
  3. The presentation is dense. Eleven mechanisms with names borrowed from ecology, neuroscience and immunology are introduced in a few pages, five of them without a significant effect on TCF in the ablations, Table 1 has nine metric columns of abbreviations, and Figure 1 draws fourteen curves in one panel.
Questions for authors
  1. How do EcoForge and the baselines compare on an existing lifelong-skill benchmark, with an LLM that both writes and executes skills?
  2. Plain accumulation shows little forgetting in the live check. What evidence shows that textual catastrophic forgetting is large with real LLMs?
  3. How does EcoForge differ in mechanism from the cap and retirement of Library Drift, and from closed-loop validated maintenance such as SkillFoundry?
1 comment

Thanks the authors for detailed rebuttal, however, I still have concerns about the original paper and will hold my score.

Reviewer_uFj8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies a problem with self-evolving LLM agents losing performance due to degradation in the skill libraries (textual catastrophic forgetting, TCF). The authors introduce a three-factor channel decomposition of the problem: retrieval mass, semantic rot, and execution regression. The authors then develop TCF-bench, a benchmark that measures performance of LLM agents, introduce drifts, then simulate updated LLM agents behaviors with their executor. To mitigate these failures, the paper proposes EcoForge, a skill library harness framework. In the simulated experiment, their method achieved best result among others.

Critical strengths
  1. The paper addresses timely limitation of continual learning through external memory. It decomposes the problem is useful, particularly for autonomous agents that gradually accumulate knowledge.

  2. The framework provides a basis for diagnosing failure mechanism that might otherwise be confused with single task success rate.

Critical weaknesses
  1. The paper is clearly generative with several critical presentation issues, such as figures with overlapping labels (Figure 1), use of word ("The v2.1/v2.2 layers in one paragraph"), structurally chaotic figures (Figure 2, Figure 5), use of terms without definition ("same-family bloat"), use of syntax without clear mathematical definition (πov). It is strongly recommended to completely rewrite the paper and fix all issues in the figures.

  2. The construction of the TCF metric (the load bearing contribution of the paper) is also questionable, as the three channels are not casually independent as the execution regression already reflects issues in the other two channels. Meanwhile, retrieval of mass and semantic rot are measuring library quality instead of necessarily measuring forgetting.

  3. The simulated executor directly penalizes "same-family" skill accumulation, while EcoForge's strongest mechanism is explicitly limiting this quantity, which appears circulative. Removing this penalty eliminates approximately 53% of the carrying-capacity gate's measured benefit, raising concerns about the headline improvement.

  4. The authors use simulated executor to replace live LLM calling, which concerns the validity of the results. The authors directly admit that the result does not reproduce with actual LLM calls, thus the empirical evidence is very weak.

Questions for authors

The authors might consider to address the weaknesses listed above.

Reviewer_Fmd6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a previously uncharacterized failure mode—textual catastrophic forgetting (TCF), where the agents may lose previously acquired capabilities as their external skill libraries grow and degrade. And there are three proposed channels of degradation: 1) retrieval miss; 2) semantic rot; 3) excution regression. Then the authors test this TCF on a benchmark called TCF-Bench, where agent needs to first learn a task family T1 and then go to do task family T2, later come back and check if agent cna still do T1. From this benchmark, the authors can evaluate whether agent losses the ablity doing the previous work while it's working on new stuffs. To overcome these degradations, the authors proposed EcoForge, a management rules for the agent's skillsets. And in the simulation environment the authors created, it can help the agent anti-TCF.

Critical strengths
  1. The paper addresses an important problem in agent with growing external skill libraries by evaluating "whether the previous acquired capabilities gets preserved as agent learns new tasks". And they design a concrete leave-and-return protocal, which evaluates old task families against their acquisition phase performance using the fixed probes. This is very useful, because it helps to check one existing pattern: the overall success of tasks can hide degradation in the prior learned tasks.

  2. The paper provides systematic ablations and analyes in the simulated environment the authors setup. These experiments vary assumptions (bloat penalty, drift process, etc), and then examine individual mechanisms.

And I also appriciate that the author explicitly report the negative results from the live LLM experiments and acknowledge the limitation of their simulation-based findings.

Critical weaknesses
  1. The experiment relies on the transparent simulated executor a lot: the simulator sets up the rule for the same skill family, once the number of its skills exceeds a upper limit, the agent's success rate drops. Moreoverr, the most effective mechanic the proposed method EcoForge has is limiting the number of skills from the same family. Then I think this rises a question: how much of the improvement comes from addressing the skill library degradattion, rather than rermoving a penalty build into similation.

  2. The proposed measurements/metrics cannot sufficiently support the claim that the three forgetting channels are independently identified. Execution regression in this paper is defined as Fexec=max⁡(0,,1−PerfR(f)PerfA(f)).F_{\mathrm{exec}}=\max\left(0,,1-\frac{\mathrm{Perf}_R(f)}{\mathrm{Perf}_A(f)}\right)., this measures the relative decrease in returning-family success compared with acquisition-phase performance. But retrieval failures can also reduce task success (also outdated skill content), so this definition doesn't isolate excution regression from the other two channels. I think these three quantities can be very helpful as diagonostic metrics, but given the current definitions, they cannot establish three separately identified causal channels of forgetting.

Questions for authors
  1. (weakness 1) The control experiment without the bloat penalty actually partially addresses my concern. But the live LLM experiments cannot reproduce the forgetting advantage. Could you provide some empirical/experimentall results to support the simulator's assumption that library-wide same-family skill counts reduce execution success beyond their effects on retrieval?

  2. Since retrieval failures and outdated skill content can both reduce returning-family success, how does the current definition of FexecF_{\mathrm{exec}} distinguish execution regression from these other causes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.