- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether prolonged reliance on externally stored, reusable skills can impair an LLM agent's subsequent task-solving ability after those skills are withdrawn. The authors define non-parametric deskilling as a post-withdrawal performance gap between agents previously exposed to skills and matched agents that never received them. They propose a randomized longitudinal evaluation protocol with several control conditions and interventions on persistent agent state. The paper also introduces Interleaved Skill-Free Rehearsal, which periodically removes the skill library during the exposure phase to preserve skill-free trajectories.
Critical strengths
The paper investigates an important problem: whether prolonged reliance on external skills can undermine LLM agents' problem-solving capabilities after those skills are withdrawn. Understanding this potential form of non-parametric deskilling is relevant to the design of agents that rely on persistent memory and reusable skills, particularly for maintaining robust performance when external resources become unavailable.
Critical weaknesses
- Insufficient generalization across architectures. The self-evolution mechanism and agent architecture significantly influence outcomes. The paper fails to verify if the identified issue persists across diverse frameworks (e.g., Voyager, ExpeL, AutoSkill). Additional experiments on these baselines are required to rule out architecture-specific artifacts.
- Lack of reproducibility. Critical implementation details such as self-evolution framework, memory configuration, state updater, and skill extraction/retrieval are omitted. This undermines the reproducibility of the experiments.
- Outdated base models and questionable generalizability. The base models used (e.g., Llama-3.1-8B/70B, GPT-4o) are somewhat dated and do not reflect current frontier models for complex agent tasks. Given that ablation study shows the performance gap narrows with stronger models, does the reported "de-skilling" phenomenon still hold with today’s more capable base models?
Questions for authors
Please check the entire manuscript for writing issues. For example, in lines 321–322, the cited references are not displayed correctly.
Nice way of concealing pure claude output papers!