acceptodds
Under review as a conference paper at ICLR 2027

EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?

Abstract

Existing agent benchmarks primarily test task completion, tool use, or skill utility, but do not isolate whether a runtime can convert evidence from its own runs into reusable skills that improve fresh executions after authoring overhead. We introduce EvoClawBench, a benchmark for this closed-loop skill-learning question on repeated, fixture-backed tasks. EvoClawBench compares direct execution without skills, PreSkill authoring before execution, and PostSkill summarization from first-run evidence followed by a fresh second execution of the same task. The suite contains 100 tasks and 502 sub-problems across coding, data, office, security, operations, and domain-document workflows, with support for multiple agent runtimes. We evaluate five models under OpenClaw and nanobot on the full 100-task suite, report percentage-point changes against a skill-free rerun inside each run, and log execution status for timeouts, empty transcripts, and fallback replies. Self-authored skills change mean scores by at most about 3.7 points on the four stable model rows and by about 5–6 points on Qwen3.6-Plus, comparable to the -3.3 to +3.1-point gap between two skill-free executions of the same model; DeepSeek-V4-Pro is the exception, with nanobot skill-mode scores near 19% while skill-free reruns stay near 82%. Reuse is not cheaper: excluding the first run, the summary and second execution still use about 2.6× the tokens of a direct run. These results indicate that learning reusable skills from an agent's own runs is selective and cost-sensitive, rather than an automatic benefit of adding skill authoring to an agent loop. We release the code at https://anonymous.4open.science/r/EvoClawBench-9380/.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Code4LLM, Code generation, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uvSNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EvoClawBench, a benchmark for evaluating whether LLM agents can create and reuse procedural skills from their own executions. The benchmark consists of 100 tasks and 502 sub-problems across multiple domains. It compares three execution modes: direct task execution without skills (Baseline), skill generation before execution (PreSkill), and skill generation from first-run evidence followed by a fresh execution of the same task (PostSkill). The authors evaluate five models under two agent runtimes, OpenClaw and nanobot, and report task scores, execution variability, and resource costs. The main findings are that self-generated skills generally provide limited or inconsistent performance improvements while introducing substantial token overhead, with a notable performance regression for DeepSeek-V4-Pro under nanobot.

Critical strengths

1. Careful experimental controls and execution auditing.

The paper includes skill-free reruns, paired bootstrap confidence intervals, and execution audits to account for variability and runtime failures (Tables 7 and 9). Skill mutation checks and scaffold ablations further help identify potential confounding factors (Sections 3.3 and 5.3). These controls improve the reliability and transparency of the evaluation.

2. Comprehensive resource accounting.

The paper distinguishes execution-only, reuse-only, and end-to-end costs. It shows that skill reuse incurs substantial overhead, requiring approximately 2.6× the baseline tokens even when excluding the initial execution (Section 5.2). This provides useful practical insights into the efficiency of skill-authoring workflows.

Critical weaknesses

1. Limited construct validity for reusable skill learning.

POSTSKILL reruns the identical task with identical fixtures after generating a skill from the first execution. PRESKILL also has access to task-specific fixtures, while POSTSKILL receives grading feedback, output summaries, and a transcript summary (Section 4.1, Appendix B). The protocol therefore cannot distinguish reusable procedural knowledge from task-specific memorization or feedback, making it closer to Reflexion-style same-task retry than transferable skill learning. Although the authors acknowledge this limitation, it weakens the benchmark's central positioning. The shared structure of generated task families (Appendix L) provides a natural opportunity to evaluate transfer to unseen tasks, which the paper does not explore.

2. Limited benchmark diversity, sensitivity, and empirical insight.

Of the 100 tasks, 79 are synthetically generated using a shared evidence-processing protocol (Appendices K–M), limiting procedural diversity. GPT-5.4 already achieves a baseline score of 99.08 on these tasks (Table 8). Without an oracle or manually authored skill baseline, it is unclear whether the benchmark can meaningfully measure the benefits of skill learning.

The results also raise concerns about attribution and reliability. An empty skill scaffold yields a larger gain (+19.84 points) than a generated PRESKILL skill (+4.05 points) in the four-task ablation (Table 6). Additionally, 11 of 15 model and execution mode combinations report identical generated task scores across OpenClaw and nanobot (Table 8), raising questions about the independence of the evaluations and the aggregation of results.

Finally, the largely negative results provide limited insight into when or why skill learning succeeds. The paper also does not adequately distinguish itself from related benchmarks such as SkillLearnBench (Zhong et al., 2026), SkillFlow (Zhang et al., 2026), and EvoAgentBench (Gao et al., 2026), which already evaluate skill generation and reuse on related or held-out tasks. The necessity of constructing a new task suite remains insufficiently justified.

Questions for authors
  1. Result consistency: In Table 8, 11 of 15 model and execution mode combinations report identical generated task scores across OpenClaw and nanobot, while seed task scores differ. Table 9 also reports identical bootstrap confidence intervals despite differing point estimates. Were these results independently computed for each runtime? Could the authors provide per-task results or clarify the aggregation procedure?

  2. Scope of “reusable”: The paper acknowledges that POSTSKILL measures within-task improvement rather than transfer to held-out sub-problems (Section 1), yet frames the benchmark around reusable skill learning. Since both executions use identical tasks and fixtures, and POSTSKILL receives grading criteria and first-run feedback (Table 5), what does this protocol measure beyond Reflexion-style retry? Could the authors either demonstrate transfer to related tasks or clarify the scope of their claims?

  3. Skill sensitivity: Can the authors provide an oracle or manually authored skill baseline to demonstrate that these tasks can benefit from high-quality skills? This would help distinguish failed skill induction from tasks with little room for improvement, especially given the high baseline scores (Table 8).

  4. Novelty and positioning: How does EvoClawBench differ from existing benchmarks such as SkillLearnBench, SkillFlow, and EvoAgentBench? What unique evaluation capability does the proposed benchmark provide?

Reviewer_RvqzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on the following problem: whether agents can convert evidence from their own previous runs into reusable skills that improve the following fresh execution (i.e. skill amortisation/learning). To this end, they introduce a benchmark called EvoClawBench, which has 3 settings: baseline (direct execution, no skills yet), preskill (skill authoring without execution) and postskill (skill evidence summarization + fresh skill execution). The benchmark suite contains a diverse set of tasks (100, 21 seed and 79 generated) and sub-problems. They evaluate 5 models powering OpenClaw and nanobot harnesses. The reported findings are that strong models score similarly across runtimes, that skill effects are small relative to rerun-to-rerun variation and of mixed sign, and that skill reuse does not pay for itself in tokens.

Critical strengths

The benchmark design is sound: 100 tasks with deterministic (non-LLM) verification, isolated workspaces per execution, and a skill-free first pass inside PostSkill that serves as a within-row rerun control

Critical weaknesses

I have some concerns about the validity of the reported scores. In Table 8, OpenClaw generated-task scores for four of five models equal the nanobot scores to two decimals in all three modes (GPT-5.4: 99.08 / 100.00 / 98.35 on both runtimes), and Table 9’s bootstrap intervals for OpenClaw GPT-5.4 equal nanobot’s. Its hard to imagine independent runs on different runtimes producing this. Upon inspecting the released artifact, its result files give OpenClaw means of 18–20% (79 of 100 tasks timed out), its own print_table2.py does not reproduce Table 2, and review/clean_rerun_results.md describes recomposing OpenClaw cells from nanobot scores and matches Table 2 exactly. Moreover, Table 7 reports ≤3 timeouts per OpenClaw cell against 76–79 in the logs. Finding 1 (runtime alignment) is a direct a consequence of this.

Questions for authors

Provide per-task OpenClaw result files for the runs reported in Tables 2 and 8, and explain how the OpenClaw generated-task scores in Table 8 came to equal nanobot’s?

Reviewer_2eRQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EvoClawBench, a benchmark for studying whether agents can turn their own execution experience into reusable skills. The authors compare direct execution, skill creation before execution, and skill generation from prior experience, evaluating both task performance and resource costs across several models and runtimes. The results suggest that self-generated skills do not consistently improve performance and often incur substantial overhead. I find the question interesting and relevant, although I have some concerns about the evaluation protocol and the strength of the conclusions.

Critical strengths

The paper studies a practical problem that is important for self-evolving agents.

Evaluating both task performance and end-to-end costs is useful.

The benchmark is reasonably well documented, and the authors are transparent about negative results and failure cases.

Critical weaknesses

Limited evidence of skill reuse. PostSkill reruns the same task after skill generation. It remains unclear whether the agent learns a reusable procedure or simply benefits from information specific to that instance. Evaluation on held-out tasks would make the claims more convincing.

The effect of skill content is unclear. In Table 6, an empty skill scaffold improves performance more than a normally generated skill. This suggests that prompting or execution-mode differences may explain part of the observed gains. Stronger, prompt-matched controls are needed.

Some experimental results need clarification. Tables 8 and 9 contain unusually similar results and confidence intervals across runtimes. There is also an inconsistency between the stated interpretation of the confidence intervals and the reported OpenClaw PostSkill interval for DeepSeek-V4-Pro.

The failure analysis is incomplete. The large performance drop for DeepSeek-V4-Pro under nanobot is striking, but the paper does not establish a convincing explanation. Changes in experimental configurations also make the repeated-run analysis difficult to interpret.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.