Open-Ended Skill Speciation: Evolving New Abstractions for Agent Skills

Large language model agents use reusable skills to encode procedural knowledge, tool interactions, and control logic. Existing skill-evolution methods optimize skill content within fixed representations or reward diversity without identifying behaviorally meaningful changes in abstraction. We formulate open-ended skill abstraction discovery as the evolution of an expanding repertoire that introduces temporal abstractions and intermediate representations while retaining verifiable utility. We propose Open-Ended Skill Speciation (OESS), a species-aware evolutionary framework that preserves and develops structural innovations when they address previously uncovered failure cases. OESS represents each skill using its input-output signature, temporal abstraction level, tool-dependency graph, failure-repair profile, and execution traces. A structured distance metric distinguishes changes in behavioral organization from superficial textual variation. To enable transitions between abstraction levels, OESS applies macro-mutations that convert repeated procedures into parameterized routines, encapsulate coordinated subskills, extract planners and verifiers, and transform linear workflows into state-dependent controllers. Candidates compete on quality within each species, while inter-species novelty and a lineage archive preserve intermediate innovations for subsequent composition. We evaluate OESS on task families spanning primitive, macro, monitor, and meta-level skills under matched execution budgets. We further test transfer across unseen tasks, models, agent harnesses, and environment perturbations. The evaluation measures verified failure coverage, species survival, abstraction depth, reuse, composition, and lineage-level transitions, yielding a systematic protocol for distinguishing functional abstraction discovery from inconsequential structural diversity.