acceptodds
Under review as a conference paper at ICLR 2027

SKILLFLOW: Benchmarking Continual Learning through Self-Evolving Agent Skills

Abstract

What distinguishes an agent that merely completes tasks from one that improves by completing them? Existing benchmarks primarily evaluate task success across isolated episodes or assess the utilization of pre-supplied skills, leaving unaddressed whether agents can convert their own experience into reusable procedural knowledge. We introduce SkillFlow, a benchmark designed for continual learning via self-evolving agent skills. Across 20 workflow families, its 166 tasks share execution structures while varying domains, artifacts, and operational conditions. Within each family, an agent begins with an empty skill library and can create, refine, and reuse skills derived from execution trajectories and verifier feedback. Using a unified Terminus-2 evaluation harness, we evaluate vanilla execution against an evolving-library setting across ten models. This comparison demonstrates a pronounced capability gap: models exhibiting comparable vanilla performance can differ substantially in their capacity to benefit from skill evolution, and frequent skill access does not necessarily yield effective adaptation. Trajectory analysis indicates that successful skill transfer relies on portable procedures and objective-aligned verification, whereas overly specific instructions and self-confirming checks are associated with negative transfer. SkillFlow renders these learning dynamics observable and provides a controlled testbed to investigate how and when agent skills foster continual adaptation.

Primary area:
datasets and benchmarks
Keywords:
code agent, LLM, skills, self-evolving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_adrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SkillFlow introduces a benchmark for evaluating whether agents can turn task execution experience into reusable skills and improve their performance on subsequent tasks. It contains 166 tasks across 20 workflow families, with tasks within each family sharing an abstract procedural structure.

I consider this a very promising research direction, and the emphasis on learning from experience is valuable. However, the benchmark construction, task distribution, and metric definitions need clearer justification and deeper analysis. In particular, the current evaluation provides an aggregate measure of adaptation but offers limited diagnostic resolution into the different capabilities involved in skill evolution.

Critical strengths
  1. The benchmark moves beyond isolated task completion to examine whether agents can extract and reuse procedural knowledge from their own experience.
  2. Grouping tasks by shared execution structures creates opportunities to study transfer across changes in domain context, inputs, and operational requirements. It is a good point for the skill-learning scenarios.
Critical weaknesses
  1. The metrics are not sufficient for the task. Completion-rate improvement is a useful end-to-end metric, but a small gain can arise from poor skill content, failure to retrieve an appropriate skill, or incorrect execution after retrieval. The current skill-use indicator only establishes that a file was read (if I understand correctly). The benchmark may benefit from annotations of skill types and task–skill relevance, enabling retrieval precision and recall where appropriate. Content-attribution or procedural-coverage measures could additionally assess which skill-derived instructions are reflected in the subsequent execution trajectory.

  2. Table 3 reports lexical overlap and several dimensions of task heterogeneity, but word overlap does not establish how much the underlying tasks or solutions repeat. Tasks with different vocabulary may be solvable by the same script with substituted constants, while similarly worded tasks may require substantially different reasoning or procedures. I think the dataset distribution here is a very important factor for this benchmark, actually.

  3. Important: The difficulty distribution exhibits substantial floor and ceiling effects. Table 10 shows that four families (Sales Pivot Analysis, Financial Statement Rolling, DMAIC Quality Analysis, and Compensation Scenario Modeling) have zero success across all ten models in both conditions. These families contain 34 tasks, approximately 20.5% of the benchmark. Conversely, five families have pooled vanilla success rates of 96.25%–100%, covering 41 tasks, or 24.7%.

These extremes limit the diagnostic value of binary completion gains. The authors should investigate whether the all-zero results reflect genuinely difficult tasks, shared execution obstacles, or overly restrictive verifiers. Are those samples really good to be in the benchmark, or are there some bad cases?

  1. The within-family ordering is constructed and reviewed as a progression in difficulty, but the paper does not provide a sufficiently clear empirical characterization of that progression.
Questions for authors
  1. Can the evaluation distinguish failures of skill generation, retrieval, and application? Are skill categories and task–skill relevance annotations available?
  2. How much within-family variation requires genuine procedural adaptation rather than reusing a solution with changed names, paths, or constants?
  3. What explains the four universally unsuccessful families? Have their failures been audited for common environmental obstacles and verifier false negatives?
  4. What empirical evidence validates the intended difficulty ordering, and how sensitive are the reported gains to task order?
2 comments
AuthorsOpenReview ↗

Request for Further Discussion

Dear Reviewer adrf,

Thank you for taking the time to consider our responses. We would appreciate a brief reassessment of whether our clarifications and additional analyses address the decision-critical concerns raised in your original review.

For clarity, we summarize the key clarifications and additional evidence below:

  • Skill generation, retrieval, and application: We clarified how SKILLFLOW distinguishes these three stages through file-level patches recording skill creation, revision, and deletion; execution traces recording actual skill retrieval; and execution trajectories paired with verifier outcomes for examining skill application. We also provided concrete cases illustrating both successful procedural transfer (Weighted Risk Assessment) and negative transfer despite repeated skill retrieval (PowerPoint editing).

  • Procedural adaptation across tasks: We conducted a specification- and oracle-grounded audit of 40 systematically selected adjacent-task pairs across all 20 workflow families. The audit found that 27/40 pairs (67.5%) require substantive changes to artifact processing, implementation logic, or verification constraints rather than parameter substitution alone. These changes were confirmed in at least 16/20 families (80%). We further provided concrete examples, including nested JSON processing in Medical Data Standardization and the identification and modification of the correct embedded workbook in Embedded Data Repair.

  • Failure attribution and verifier reliability: We examined the four universally unsuccessful families and identified distinct failure patterns, including native PivotTable generation, multi-step spreadsheet dependencies, and restrictive schema or formatting checks. We also verified that 9/9 reference solutions passed their corresponding verifiers. Importantly, excluding the four all-zero and five near-ceiling families, the remaining 11 families (91 tasks) still exhibit a +6.37 percentage-point gain, compared with +3.01 percentage points across the full benchmark.

  • Task difficulty progression and ordering: We clarified that the within-family ordering was prespecified through the architect–critic construction process and expert review, rather than selected to maximize skill-evolution gains. The vanilla success rate generally decreases with task position, from 54.5% at position 1 to 41.5% at position 8. Furthermore, Vanilla and Skill Evolution are evaluated using the same fixed task order, providing a consistent basis for comparing skill accumulation under the same curriculum.

Taken together, we believe that all key concerns raised in the original review have been addressed and clarified through the additional evidence, case analyses, and empirical evaluations.

Could you let us know whether any decision-critical concerns remain unresolved after considering these clarifications? If the additional evidence addresses your original concerns, we would appreciate an updated assessment and rating reflecting the revised evaluation of the paper.

Thank you again for your time and consideration.

Thanks for the author's reply. Some of the discussion is helpful. But my primary concerns are still here: why did some tasks fail? The authors did not provide a detailed analysis during the rebuttal (it is important for a benchmark; this is the dataset problem). I will still hold a negative attitude towards this paper.

Reviewer_RrqBOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SKILLFLOW evaluates whether agents learn useful procedures across related tasks by creating and revising a file-based skill library. It contains 166 executable tasks in 20 workflow families across five domains. Within each family, tasks share an abstract workflow and run in a fixed difficulty order. Ten models are evaluated with and without skill evolution, giving an average completion gain of 3.01 percentage points. It also records skill edits and reads, and examines when reuse helps or hurts.

Critical strengths

1. The workflow families make reuse a concrete part of the evaluation. Agents must transfer procedures across changes in data, artifacts, and task context. Runnable environments, programmatic verifiers, and a documented construction process make this a useful resource for studying learning across tasks.

2. The file edits and execution traces make the results easier to inspect. The paper shows both helpful procedures and negative transfer, rather than treating skill retrieval as evidence of learning. Testing ten models under the same harness and auditing verifier decisions also gives the benchmark a useful starting point.

Critical weaknesses

1. The novelty needs a clearer comparison with the closest benchmarks. Table 2 leaves out SkillLearnBench, which already evaluates skill generation, feedback-based revision, reuse across task instances, and execution behavior. LifelongAgentBench is discussed in the text but is also absent from that comparison. SKILLFLOW adds workflow-defined families and online revision after each task, but the paper needs to show what these choices let us measure beyond those earlier settings.

2. The gains and model rankings need a repeatability check. Table 7 uses one sample per task, and four improving models gain only one or two successes out of 166. The reported interval resamples model-level gains, so it does not measure variation across reruns of the same learning sequences. This matters when the benchmark is used to compare models' ability to learn, rather than just describe this particular set of runs.

3. The vanilla comparison measures the benefit of the whole learning setup, but doesn't isolate what structured skills add. Evolution introduces feedback, post-task processing, and persistent files together. A trajectory or reflection memory with the same feedback and processing budget is missing. Appendix C.1 also excludes patch-generation calls from the reported trajectory token counts, leaving the total learning cost unresolved.

4. The claims about a separate capability and an acquisition threshold go beyond the evidence. Gain subtracts the baseline used in the correlation, so r=-0.56 can partly reflect that shared term and measurement noise. Meanwhile, the fixed sequence changes task identity and difficulty together with experience. These analyses show heterogeneous gains under this protocol, but don't establish an orthogonal capability or a learning threshold. The limitations acknowledge the order issue, while Section 4 still makes the stronger interpretation.

Questions for authors

1. How does SKILLFLOW differ from SkillLearnBench and LifelongAgentBench in what it can reliably measure? Please compare the protocols directly and explain what the DAEF families and task-by-task revision reveal that the earlier settings do not.

2. Do the gains and model rankings persist across independent runs of complete model-family sequences? A repeatability check covering both small-gain and larger-gain models, with uncertainty across runs, would directly affect my assessment.

3. With the same feedback and processing budget, how does a simple trajectory or reflection memory compare with the skill library? Please include patch generation in the total token/call cost so that the completion-cost comparison covers the learning process.

4. Does the threshold pattern persist with counterbalanced task orders or fixed probe tasks evaluated after different amounts of experience? Separately, does the baseline-gain relationship survive using an independent baseline estimate that is not reused in the gain calculation?

3 comments
AuthorsOpenReview ↗

Request for Further Discussion

Dear Reviewer RrqB,

Thank you for your thoughtful review and for the constructive suggestions. We appreciate the opportunity to further clarify our work and provide additional experimental evidence. We would welcome a brief reassessment of the paper in light of these responses.

For ease of reference, we highlight the key clarifications and additional evidence below:

  • Positioning and novelty: We clarified the distinction between SKILLFLOW and existing benchmarks, particularly SkillLearnBench and LifelongAgentBench. SKILLFLOW focuses on the longitudinal interaction of skill acquisition, revision, and reuse under controlled workflow structures. Across 1,660 paired evaluations, we identified 115 failure-to-success and 65 success-to-failure transitions, revealing bidirectional adaptation behaviors that aggregate completion rates alone cannot capture.

  • Repeatability across independent runs: We conducted two additional independent reruns of both Vanilla and Skill Evolution for Kimi K3 and GPT-5.5, covering both substantial and marginal improvement cases. Kimi K3 consistently achieved gains of 6.02 and 7.83 percentage points in the two reruns, with a mean gain of 7.43 ± 1.25 percentage points across three runs. GPT-5.5 showed smaller but consistent gains, averaging 1.00 ± 0.70 percentage points. All reruns used the same 166 tasks, task ordering, evaluation harness, and model configurations, with independently initialized skill libraries.

  • Evaluation scope and learning cost: We clarified that the Vanilla–Evolution comparison measures the effect of the complete skill-evolution process, including feedback-driven knowledge consolidation and persistent skill updates, rather than isolating the superiority of one memory representation. We also provided a more complete accounting of post-task skill-generation costs, including patch-generation calls: 25.72M tokens in total (19.95M input and 5.78M output), representing 11.60% of the measured overall token usage.

  • Controlled evaluation of early skill acquisition: To disentangle accumulated experience from changes in task identity and difficulty, we conducted an additional fixed-probe experiment using the same 20 downstream tasks across all conditions, with three independent executions per condition. Performance increased from 36.67% with an empty skill library to 45.00% after the first task and 50.00% after the second task, subsequently remaining stable at approximately 48–50%. This provides direct evidence that skills acquired from a relatively small amount of prior experience can improve performance on identical downstream tasks.

Taken together, these additional experiments and clarifications provide stronger empirical support for SKILLFLOW's distinct evaluation protocol, the repeatability of its observed gains, and the benefits of early skill acquisition. We believe they substantially address the main concerns raised in the original review.

We would appreciate your consideration of this additional evidence and, if these results address your remaining concerns, an updated overall assessment and rating reflecting the strengthened experimental support.

Thank you again for your time and consideration.

We sincerely thank the reviewer again for the constructive follow-up questions. For your convenience, we briefly summarize the additional analyses and clarifications provided above: we added a cross-fitted baseline–gain analysis to remove the direct mathematical coupling in the original correlation, reported the probe outcomes for all 20 families and clarified the observed early-gain/plateau pattern, and provided a detailed breakdown of API calls, token usage, and estimated costs.

We hope these additional results help clarify the three remaining questions. We would be grateful if the reviewer could take another look at our responses when time permits and consider whether the additional evidence addresses the concerns raised. We are of course happy to provide any further details if helpful.

Thanks for following up. Two task families account for most of the net gain, so I’m still unsure how well the overall score captures learning across workflows. I’ll keep my original rating.

Reviewer_RQiiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillFlow benchmarks continual learning through self-evolving agent skills across 20 workflow families. Agents maintain a persistent skill library within each family using JSON patches. Skill evolution improves performance in some families but reduces it in others, with almost no correlation between skill count and performance gains. Over-specific procedures, such as hard-coded values, can cause negative transfer. The benchmark includes a programmatic verifier with 91.07% agreement in a human audit, alongside detailed trajectories and patch event logs.

Critical strengths
  1. The patch format and linked task, trajectory, and change records support reproducibility and analysis of skill evolution.
  2. A case study links negative transfer to hard-coded purchase-order dates and self-confirming checks, while the near-zero correlation between library size and performance (r ~= 0.01) shows that accumulating more skills does not necessarily improve results.
Critical weaknesses
  1. The benchmark is intuitive and limited in novelty. The core insight "some agents improve, some don't, and more skills != better" is not particularly surprising. The paper would benefit from deeper analysis of why certain skill structures generalize while others do not.
  2. The 20 workflow families appear to be specific to certain business/document-processing domains. It is unclear how findings would transfer to other agent settings (e.g., code generation, web navigation, scientific reasoning). The paper's title claims to benchmark "continual learning through self-evolving agent skills" broadly, but the actual scope is more limited.
Questions for authors
  1. The correlation between skill count and performance is r ~= 0.01. Have you investigated what does predict performance gain?

  2. For the negative transfer case (hard-coded purchase-order dates), was this caused by the skill creation step or the skill application step?

  3. The verifier has 91.07% agreement with human judgment. What is the impact of these false negatives on the reported results?

  4. How sensitive are results to task ordering within a family? If tasks were presented in a different order, would the same families show improvement or decline?

2 comments
AuthorsOpenReview ↗

Request for Further Discussion

Dear Reviewer RQii,

Thank you for your thoughtful review and constructive suggestions. We particularly appreciate your recognition of SKILLFLOW's reproducibility, traceability, and analysis of both positive and negative skill transfer. Following your questions, we conducted additional analyses using our existing execution records and clarified several aspects of the evaluation. We would like to briefly summarize these updates.

  • Factors associated with performance gains: Beyond examining skill-library size, we analyzed 200 model–family sequences and found that neither skill count (Pearson r = 0.013) nor retrieval frequency (r = 0.166) strongly correlates with completion gains. We further audited 20 rescued cases and 20 regression cases. Among the 14 rescued cases with confirmed skill retrieval, 13/14 (92.9%) showed task-adapted application of the retrieved procedures. In contrast, only 6/12 (50.0%) regression cases with confirmed retrieval showed task-adapted application. These findings provide further insight into why successful transfer depends not simply on accumulating or retrieving skills, but also on adapting them to the current task.

  • Source of negative transfer: We traced the hard-coded purchase-order example through its execution trajectory and found that the failure originated primarily from skill application rather than skill creation. The retrieved skill contained a general-purpose procedure, but the agent failed to adapt it to the current output schema, resulting in hard-coded dates and quantities. Its subsequent self-check also failed to detect the mismatch.

  • Impact of verifier false negatives: Our manual audit of 56 outputs identified five false negatives and no false positives. All five cases involved outputs that human reviewers judged to satisfy the task requirements, indicating that the observed discrepancies lead to underestimation of success. Even if all five cases are corrected under the stated sensitivity assumptions, the overall macro-averaged Evolution Gain remains positive, at approximately +2.71 percentage points, compared with the original +3.01 percentage points.

  • Task ordering: We further clarified the rationale for the within-family ordering, which follows a progressively challenging curriculum established through the architect–critic construction process and expert review. Earlier tasks provide opportunities to acquire foundational procedures before agents encounter more demanding tasks. The Vanilla results are consistent with this design, with success rates generally decreasing from 54.5% at position 1 to 41.5% at position 8. We have also clarified that our findings characterize skill evolution under this prescribed curriculum, rather than claiming invariance to arbitrary task permutations.

Taken together, we believe that the additional analyses and clarifications address the key questions raised in your review and provide further support for the soundness and contributions of SKILLFLOW.

We sincerely hope these clarifications resolve your remaining concerns. If you find that the additional evidence adequately addresses the issues raised in your original review, we would greatly appreciate your consideration of an updated assessment and rating that reflects these clarifications.

Thank you again for your time, thoughtful feedback, and consideration.

Thank you for the response. It address my question on negative transfer and partly address the others, but given the unresolved task-order sensitivity and its limited domain scope, I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.