- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SkillNEO, a framework for automatically improving LLM agent skills through non-elitist evolutionary optimization. The paper addresses the problem that conventional performance-driven skill refinement may prematurely discard intermediate skill versions whose current performance is poor but which contain useful partial improvements.
The proposed framework combines offspring-only parent selection, comparative analysis of execution trajectories, archive revisiting, and cross-branch procedure reuse. Unlike approaches that require revisions to improve upon their parents or enter a historical top-performing set, SkillNEO allows lower-scoring intermediate versions to remain eligible for further development. An LLM-based proposer and editor use execution feedback to revise skill instructions, scripts, and supporting resources.
The authors evaluate the framework on 86 SkillsBench tasks across eight domains, using GPT-5.4 Mini for task execution, proposal generation, and skill editing. The reported average reward improves from 49.1% to 66.4%. The authors further demonstrate that the evolved skills can benefit stronger models, including GPT-5.5 and GPT-5.6 Luna. Additional analyses of recorded search trajectories illustrate how lower-performing intermediate versions can lead to successful descendants.
Critical strengths
- An important and well-motivated problem.
The paper addresses a meaningful limitation of performance-driven agent skill optimization: current task reward may not adequately reflect the potential of an intermediate skill version for future improvement. The distinction between immediate performance and subsequent development potential is relevant to iterative agent optimization, particularly when improvements require multiple coordinated modifications.
- Encouraging empirical results and detailed analysis of evolutionary trajectories.
The reported improvements across 86 tasks are substantial, and the cross-model evaluation provides useful evidence that skills developed by a weaker model can benefit stronger executors. The paper also presents concrete examples of partial repairs, cross-branch procedure reuse, and archived-version revisiting. These analyses provide useful insights into the behavior of iterative skill optimization beyond aggregate performance metrics.
Critical weaknesses
- Limited methodological novelty and insufficient experimental evidence supporting the core contribution.
The main methodological contribution relies on offspring-only non-elitist selection, which is an established strategy in evolutionary optimization. Other components, including trajectory-based feedback, archive revisiting, and cross-branch reuse, are also closely related to mechanisms explored in prior skill optimization and evolutionary search frameworks.
Although their combination constitutes a particular system design, the paper does not sufficiently establish what new algorithmic capability this combination provides beyond existing approaches.
More importantly, the paper lacks controlled comparisons against alternative selection strategies under matched computational budgets. In particular, comparisons with elitist parent-offspring selection, historical top-k selection, and simpler iterative refinement approaches are necessary to establish whether the reported improvements are attributable to non-elitist selection rather than repeated LLM-based editing and evaluation.
The paper also lacks component-level ablations that isolate the effects of comparative execution evidence, archive revisiting, and cross-branch procedure reuse. The trajectory analyses demonstrate that lower-scoring candidates can produce successful descendants, but they do not establish that the proposed selection strategy is more effective or efficient than competing approaches.
This substantially weakens the central methodological claim and makes it difficult to assess the actual contribution of SkillNEO beyond its overall engineering implementation.
- Limited generalization evidence and potentially instance-specific optimization.
A significant limitation is that skills are independently optimized for individual benchmark tasks and evaluated on the same task instances used during development.
Although the paper conducts fresh executions after skill evolution and introduces safeguards against direct leakage of private evaluator answers, these procedures do not establish that the resulting skills generalize to unseen task inputs or variations.
Repeated optimization using feedback from a fixed task instance may encourage highly specialized instructions or procedures. Consequently, the reported performance improvements could partly reflect adaptation to specific benchmark instances rather than the acquisition of broadly reusable skills.
This distinction is particularly important because the paper emphasizes skill evolution and cross-model transfer. While cross-model reuse is demonstrated on the same tasks, the experiments do not establish whether the evolved skills remain beneficial under different task inputs.
The absence of held-out evaluation limits the strength of the paper's conclusions about reusable skill improvement.
Questions for authors
- How can the authors isolate the effectiveness of non-elitist selection from the benefits of repeated skill editing and evaluation?
Could the authors provide matched-budget comparisons with conventional elitist selection, historical top-k selection, and simpler iterative skill refinement? Ideally, these comparisons should use identical developer models, initial skill bundles, task execution budgets, and LLM-call budgets. Ablations of archive revisiting and comparative trajectory feedback would also help clarify which components are responsible for the observed gains.
This evidence would materially affect my assessment of the paper's methodological contribution.
- To what extent do the evolved skills generalize beyond the specific task instances used during development?
Could the authors evaluate the evolved skills on held-out inputs or task variations without further skill modification? Such an experiment would help distinguish reusable procedural improvement from instance-specific optimization. It would also clarify whether the observed cross-model benefits persist beyond the development instances.
This is important for assessing the significance of the claimed skill evolution and transferability.
General Response to All Reviewers and Area Chairs (1/2): What Reviewers Value and Two Shared Concerns
We thank all the reviewers for their careful and constructive feedback. SkillNEO starts from one observation: a skill version's current reward can miss its development value, because a partially repaired skill can score low until later edits finish the repair. SkillNEO keeps such a version editable through non-elitist parent selection, and uses execution comparisons to guide the next edit. We are encouraged that the reviewers value the problem, the idea, the analysis and the results.
Two shared concerns, both answered with new experiments
Our individual responses address every point raised and contain the full result tables. The reviews share two concerns:
T8ni W1,g1Y9 W1/Q1,ogPK W1/Q5T8ni W2,g1Y9 W2/Q2,ogPK Q5