- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
OESS proposes to evolve executable agent skills through LLM-generated macro-mutations, structured skill descriptors, species-aware selection, verified novelty, and lineage retention. A candidate can form a new species when it is structurally distant from existing representatives and reproducibly solves previously uncovered diagnostic failures. The five macro-mutations introduce procedural routines, encapsulated subskills, planners, monitors, and stateful controllers.
The method is evaluated on a custom benchmark with primitive, macro, monitor, and meta-level task families. The paper reports improvements in held-out success, species survival, cross-task reuse, hierarchy depth, composition, and transfer over several internally constructed baselines.
Critical strengths
I like that the paper tries to define abstraction discovery more carefully than simple textual novelty. It provides a clear high-level framework for structured descriptors, verified novelty, macro-mutations, species-aware selection, and lineage retention, although some operational details remain underspecified.
The evaluation design is also ambitious. The authors attempt to use matched search budgets, deterministic environment assertions, transfer tests, ablations, sensitivity studies, and multiple random seeds rather than relying only on free-form LLM judging.
Critical weaknesses
My main concern is that the reported main configuration is internally inconsistent. The implementation section gives one set of thresholds and archive capacity, while the main statistics exactly match different settings in the sensitivity tables. Without code, task instances, verifier implementations, or run logs, it is impossible to determine which configuration produced the reported results. The comparison is also limited to internal variants rather than published skill-evolution systems.
The paper also does not independently validate its central abstraction labels. Descriptor fields and abstraction levels are assigned by an LLM, with no agreement study, repeated extraction, or alternative model. Mutation selection is described both as uniform sampling and as an LLM decision. The results show that OESS finds useful skill artifacts, but they do not clearly establish that these artifacts represent genuinely new abstraction levels.
Questions for authors
Which thresholds and archive capacity produced the main results?
Can the authors release the code, tasks, verifiers, instantiated prompts, and per-seed traces and add a published-system baseline?
Which decisions are deterministic and which are produced by an LLM, and how stable are the abstraction labels across repeated extraction and alternative models?