- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SkillFlow introduces a benchmark for evaluating whether agents can turn task execution experience into reusable skills and improve their performance on subsequent tasks. It contains 166 tasks across 20 workflow families, with tasks within each family sharing an abstract procedural structure.
I consider this a very promising research direction, and the emphasis on learning from experience is valuable. However, the benchmark construction, task distribution, and metric definitions need clearer justification and deeper analysis. In particular, the current evaluation provides an aggregate measure of adaptation but offers limited diagnostic resolution into the different capabilities involved in skill evolution.
Critical strengths
- The benchmark moves beyond isolated task completion to examine whether agents can extract and reuse procedural knowledge from their own experience.
- Grouping tasks by shared execution structures creates opportunities to study transfer across changes in domain context, inputs, and operational requirements. It is a good point for the skill-learning scenarios.
Critical weaknesses
-
The metrics are not sufficient for the task. Completion-rate improvement is a useful end-to-end metric, but a small gain can arise from poor skill content, failure to retrieve an appropriate skill, or incorrect execution after retrieval. The current skill-use indicator only establishes that a file was read (if I understand correctly). The benchmark may benefit from annotations of skill types and task–skill relevance, enabling retrieval precision and recall where appropriate. Content-attribution or procedural-coverage measures could additionally assess which skill-derived instructions are reflected in the subsequent execution trajectory.
-
Table 3 reports lexical overlap and several dimensions of task heterogeneity, but word overlap does not establish how much the underlying tasks or solutions repeat. Tasks with different vocabulary may be solvable by the same script with substituted constants, while similarly worded tasks may require substantially different reasoning or procedures. I think the dataset distribution here is a very important factor for this benchmark, actually.
-
Important: The difficulty distribution exhibits substantial floor and ceiling effects. Table 10 shows that four families (Sales Pivot Analysis, Financial Statement Rolling, DMAIC Quality Analysis, and Compensation Scenario Modeling) have zero success across all ten models in both conditions. These families contain 34 tasks, approximately 20.5% of the benchmark. Conversely, five families have pooled vanilla success rates of 96.25%–100%, covering 41 tasks, or 24.7%.
These extremes limit the diagnostic value of binary completion gains. The authors should investigate whether the all-zero results reflect genuinely difficult tasks, shared execution obstacles, or overly restrictive verifiers. Are those samples really good to be in the benchmark, or are there some bad cases?
- The within-family ordering is constructed and reviewed as a progression in difficulty, but the paper does not provide a sufficiently clear empirical characterization of that progression.
Questions for authors
- Can the evaluation distinguish failures of skill generation, retrieval, and application? Are skill categories and task–skill relevance annotations available?
- How much within-family variation requires genuine procedural adaptation rather than reusing a solution with changed names, paths, or constants?
- What explains the four universally unsuccessful families? Have their failures been audited for common environmental obstacles and verifier false negatives?
- What empirical evidence validates the intended difficulty ordering, and how sensitive are the reported gains to task order?
2 comments
Thanks for the author's reply. Some of the discussion is helpful. But my primary concerns are still here: why did some tasks fail? The authors did not provide a detailed analysis during the rebuttal (it is important for a benchmark; this is the dataset problem). I will still hold a negative attitude towards this paper.
Request for Further Discussion
Dear Reviewer adrf,
Thank you for taking the time to consider our responses. We would appreciate a brief reassessment of whether our clarifications and additional analyses address the decision-critical concerns raised in your original review.
For clarity, we summarize the key clarifications and additional evidence below:
Skill generation, retrieval, and application: We clarified how SKILLFLOW distinguishes these three stages through file-level patches recording skill creation, revision, and deletion; execution traces recording actual skill retrieval; and execution trajectories paired with verifier outcomes for examining skill application. We also provided concrete cases illustrating both successful procedural transfer (Weighted Risk Assessment) and negative transfer despite repeated skill retrieval (PowerPoint editing).
Procedural adaptation across tasks: We conducted a specification- and oracle-grounded audit of 40 systematically selected adjacent-task pairs across all 20 workflow families. The audit found that 27/40 pairs (67.5%) require substantive changes to artifact processing, implementation logic, or verification constraints rather than parameter substitution alone. These changes were confirmed in at least 16/20 families (80%). We further provided concrete examples, including nested JSON processing in Medical Data Standardization and the identification and modification of the correct embedded workbook in Embedded Data Repair.
Failure attribution and verifier reliability: We examined the four universally unsuccessful families and identified distinct failure patterns, including native PivotTable generation, multi-step spreadsheet dependencies, and restrictive schema or formatting checks. We also verified that 9/9 reference solutions passed their corresponding verifiers. Importantly, excluding the four all-zero and five near-ceiling families, the remaining 11 families (91 tasks) still exhibit a +6.37 percentage-point gain, compared with +3.01 percentage points across the full benchmark.
Task difficulty progression and ordering: We clarified that the within-family ordering was prespecified through the architect–critic construction process and expert review, rather than selected to maximize skill-evolution gains. The vanilla success rate generally decreases with task position, from 54.5% at position 1 to 41.5% at position 8. Furthermore, Vanilla and Skill Evolution are evaluated using the same fixed task order, providing a consistent basis for comparing skill accumulation under the same curriculum.
Taken together, we believe that all key concerns raised in the original review have been addressed and clarified through the additional evidence, case analyses, and empirical evaluations.
Could you let us know whether any decision-critical concerns remain unresolved after considering these clarifications? If the additional evidence addresses your original concerns, we would appreciate an updated assessment and rating reflecting the revised evaluation of the paper.
Thank you again for your time and consideration.