- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EvoClawBench, a benchmark for evaluating whether LLM agents can create and reuse procedural skills from their own executions. The benchmark consists of 100 tasks and 502 sub-problems across multiple domains. It compares three execution modes: direct task execution without skills (Baseline), skill generation before execution (PreSkill), and skill generation from first-run evidence followed by a fresh execution of the same task (PostSkill). The authors evaluate five models under two agent runtimes, OpenClaw and nanobot, and report task scores, execution variability, and resource costs. The main findings are that self-generated skills generally provide limited or inconsistent performance improvements while introducing substantial token overhead, with a notable performance regression for DeepSeek-V4-Pro under nanobot.
Critical strengths
1. Careful experimental controls and execution auditing.
The paper includes skill-free reruns, paired bootstrap confidence intervals, and execution audits to account for variability and runtime failures (Tables 7 and 9). Skill mutation checks and scaffold ablations further help identify potential confounding factors (Sections 3.3 and 5.3). These controls improve the reliability and transparency of the evaluation.
2. Comprehensive resource accounting.
The paper distinguishes execution-only, reuse-only, and end-to-end costs. It shows that skill reuse incurs substantial overhead, requiring approximately 2.6× the baseline tokens even when excluding the initial execution (Section 5.2). This provides useful practical insights into the efficiency of skill-authoring workflows.
Critical weaknesses
1. Limited construct validity for reusable skill learning.
POSTSKILL reruns the identical task with identical fixtures after generating a skill from the first execution. PRESKILL also has access to task-specific fixtures, while POSTSKILL receives grading feedback, output summaries, and a transcript summary (Section 4.1, Appendix B). The protocol therefore cannot distinguish reusable procedural knowledge from task-specific memorization or feedback, making it closer to Reflexion-style same-task retry than transferable skill learning. Although the authors acknowledge this limitation, it weakens the benchmark's central positioning. The shared structure of generated task families (Appendix L) provides a natural opportunity to evaluate transfer to unseen tasks, which the paper does not explore.
2. Limited benchmark diversity, sensitivity, and empirical insight.
Of the 100 tasks, 79 are synthetically generated using a shared evidence-processing protocol (Appendices K–M), limiting procedural diversity. GPT-5.4 already achieves a baseline score of 99.08 on these tasks (Table 8). Without an oracle or manually authored skill baseline, it is unclear whether the benchmark can meaningfully measure the benefits of skill learning.
The results also raise concerns about attribution and reliability. An empty skill scaffold yields a larger gain (+19.84 points) than a generated PRESKILL skill (+4.05 points) in the four-task ablation (Table 6). Additionally, 11 of 15 model and execution mode combinations report identical generated task scores across OpenClaw and nanobot (Table 8), raising questions about the independence of the evaluations and the aggregation of results.
Finally, the largely negative results provide limited insight into when or why skill learning succeeds. The paper also does not adequately distinguish itself from related benchmarks such as SkillLearnBench (Zhong et al., 2026), SkillFlow (Zhang et al., 2026), and EvoAgentBench (Gao et al., 2026), which already evaluate skill generation and reuse on related or held-out tasks. The necessity of constructing a new task suite remains insufficiently justified.
Questions for authors
-
Result consistency: In Table 8, 11 of 15 model and execution mode combinations report identical generated task scores across OpenClaw and nanobot, while seed task scores differ. Table 9 also reports identical bootstrap confidence intervals despite differing point estimates. Were these results independently computed for each runtime? Could the authors provide per-task results or clarify the aggregation procedure?
-
Scope of “reusable”: The paper acknowledges that POSTSKILL measures within-task improvement rather than transfer to held-out sub-problems (Section 1), yet frames the benchmark around reusable skill learning. Since both executions use identical tasks and fixtures, and POSTSKILL receives grading criteria and first-run feedback (Table 5), what does this protocol measure beyond Reflexion-style retry? Could the authors either demonstrate transfer to related tasks or clarify the scope of their claims?
-
Skill sensitivity: Can the authors provide an oracle or manually authored skill baseline to demonstrate that these tasks can benefit from high-quality skills? This would help distinguish failed skill induction from tasks with little room for improvement, especially given the high baseline scores (Table 8).
-
Novelty and positioning: How does EvoClawBench differ from existing benchmarks such as SkillLearnBench, SkillFlow, and EvoAgentBench? What unique evaluation capability does the proposed benchmark provide?