- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper names "textual catastrophic forgetting", where a self-evolving agent loses mastered skills because its text skill library degrades, and splits it into retrieval miss, semantic rot and execution regression. It introduces TCF-Bench, a four-phase leave-and-return protocol with injected API and domain drift, and EcoForge, a governance framework that caps skills per family, filters retrieval, shields key skills and repairs broken dependencies. With a simulated executor, EcoForge reports a TCF of 0.036 against 0.186 for the best baseline; in a live check on five LLMs it bounds same-family growth but does not reproduce the TCF advantage.
Critical strengths
- The problem is real and timely. Self-evolving agents that grow skill libraries are an active direction, and library degradation is a practical risk that the cited concurrent work also reports.
Critical weaknesses
- The novelty over existing work is limited. The cited concurrent work already identifies this failure mode: Yu et al. (2026) study capability degradation across evolution channels including skills, Library Drift addresses unbounded accumulation with retirement and a cap, and Fan et al. (2026) repair skills broken by API change. Further uncited work validates, repairs, merges and prunes libraries (SkillFoundry, https://arxiv.org/abs/2604.03964), gates and maintains skills by verification and score (Skill-Pro, https://arxiv.org/abs/2602.01869), and diagnoses library defects that harm retrieval, composition and execution (SkillOps, https://arxiv.org/abs/2605.13716), so the three channels and most EcoForge mechanisms overlap substantially with these ideas.
- The experiments do not yet support the main claims. The main results use a simulated executor with no LLM, so outcomes follow the paper's own formula: Section 7 attributes about 53 percent of the cap's effect to the same-family penalty the cap removes, and the simulator injects the broken dependencies that EcoForge's repair step patches. In the live check, with 2 to 6 paired cells and 20 tasks per phase, EcoForge forgets slightly more than plain accumulation on every model, which itself forgets little (TCF 0.025 to 0.046), and the five domains are not described in the main body.
- The presentation is dense. Eleven mechanisms with names borrowed from ecology, neuroscience and immunology are introduced in a few pages, five of them without a significant effect on TCF in the ablations, Table 1 has nine metric columns of abbreviations, and Figure 1 draws fourteen curves in one panel.
Questions for authors
- How do EcoForge and the baselines compare on an existing lifelong-skill benchmark, with an LLM that both writes and executes skills?
- Plain accumulation shows little forgetting in the live check. What evidence shows that textual catastrophic forgetting is large with real LLMs?
- How does EcoForge differ in mechanism from the cap and retirement of Library Drift, and from closed-loop validated maintenance such as SkillFoundry?
Thanks the authors for detailed rebuttal, however, I still have concerns about the original paper and will hold my score.