- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoSkills, a novel approach to the skill extraction task that focuses on obtaining more concise and broadly applicable skills from positive examples of performed tasks. The described method first obtains a complete skill from the reference implementation of a task and then attempts to remove each operation from it and retrain to see if it made any difference to the execution. After that, the skill is modified to make it more general and added to the skill library. The most applicable skills are then selected based on their past performance, similarity to the task at hand, and compatibility. The experiments were conducted on the domain of biomedical data analysis, with some results reported on mathematical problems and office question-answering tasks. Overall, the results suggest that leveraging skills significantly improves the performance, but it is unsure how much of these improvements are due to the method itself and how much comes from the intensive verification-based pruning as described in the paper.
Critical strengths
-
I agree with the problem statement that the paper addresses, that completing a task successfully once does not imply that every step taken toward completing it should be retained in future instances of the task. For agents that make several tool calls and process data, these operations that were found to be irrelevant to the task at hand could interfere with subsequent operations. I think that the approach of determining what can be discarded from the workflow by relying on the actual execution is an intuitive one.
-
The paper benchmarks realistic multi-step data analysis tasks and achieves significant improvements, such as an increase in accuracy from 77.36% to 94.97% in the larger BioDSBench evaluation. With additional clarification of the details of evaluation splits and skill selection, I think that these results could be leveraged to demonstrate the value of the approach well.
Critical weaknesses
-
It is unclear what test-task information the skill selector uses, which directly impacts how the results should be interpreted. The selector used in Section 3.5 uses both the current task's no-skill score and, preferentially, a skill's previously observed improvement on that same task. This suggests that selection already knows which skills work on the current problem. That is understandable in a context where multiple attempts at the same task are allowed, but different from selecting skills for an unseen task. The paper does not clearly report how many of the tasks in the main results use this direct feedback, nor does it clarify if skill construction, validation, selection and final testing are separated. The authors report that reference files are hidden from the solver, but this alone does not reveal if target-task information gets in the system through the content of skills or their selection. I am then unsure how much of the improvement comes from transfer of tasks.
-
The paper underlines the idea of "forgetting through verification," but the experiments mostly evaluate the value of skill selection, not the importance of pruning. Table 3 compares the performance using and not using the selector, but not the performance of full skills, skills shortened by an LLM only and skills pruned through execution-based verification. Even if the complete system performs well, it is unclear if verified pruning is more valuable than simply providing the reference workflow or using ordinary text-based simplification. The paper lacks before-and-after measures for skill length, number of retained operations and the extra cost of execution to judge if pruning is worth the overhead. There appears to be an upside-down sign in the core deletion criterion, too: Eq. (2) defines Δ as the full-skill score minus the score with deletion, but Eq. (4) requires Δ ≥ −zSE, which can be true even if deletion dramatically worsens the performance, contrary to the intended meaning that deletion should not worsen performance. Could the authors clarify whether that is a typo in the equation or if the implementation makes the same choice?
Questions for authors
-
Could the authors please clarify the splitting of tasks for skill construction, pruning, generalization validation, selector history, and final testing? How many test tasks in the main results make use of historical skill scores from that same task? What would happen if target task scores are excluded and skills are selected based on only information from other tasks?
-
Given the same reference to information and evaluation budget, how do full skills, skills shortened directly by an LLM, and skills pruned through execution-based verification compare? How many operations or tokens, on average, does pruning remove and how many additional executions does it require? This comparison would be critical to my assessment of the core contribution.
-
Is the inequality in Eq. (4) reversed? Could the authors please clarify the acceptance condition actually used in the code and how the standard error and threshold are found? If the implementation does follow the equation as written, would correcting it change the main results?
3 comments
Thank you for the clarifications and additional experiments. My concerns about target-task feedback and the isolated contribution of verified pruning remain unresolved, so I will maintain my original score.
We sincerely thank you for your valuable comments and constructive feedback.
Target-Task Feedback, Pruning Controls, and the Correction to Eq. (4)
We thank the reviewer for identifying the sign error and for distinguishing two questions: whether a skill transfers to another task, and what information the selector already has about that target. We address the information available at each stage before interpreting the pruning results.
W1 & Q1. Task splits, same-target history, and information available to the selector
The author constructs and prunes each skill on its source workflow. The generalizer receives the source skill without the target task; the rewritten skill is validated on a task group separate from the source training split. The final transfer comparison evaluates 37 held-out source–target pairs.
We audited two different routes by which task identity enters the evaluation:
With leave-one-target-out fitting, the selector attained 67.5% top-1 accuracy and 99.36% inject/skip accuracy at k=8. At runtime, Eq. (8) permits prior same-target gains in Phase 1 and uses the measured no-skill baseline bt in Phase 3. Excluding a target from the fit does not remove this input. The 0/50 count measures self-transfer, not Phase-1 usage. The published BioDSBench panels give neither per-route counts nor an end-task score with all target feedback disabled. They evaluate feedback-assisted reuse.
W2 & Q2. Full skills, direct LLM shortening, verified pruning, and cost
We evaluated three controls using DeepSeek-V4-Flash-0731 on 37 BioMNIBench transfer pairs. Each starts from a source-derived skill. The shortening prompt instructs the LLM to remove duplication while preserving the methods, thresholds, and order of retained operations; no solver execution decides which operations to remove in that control.
Direct shortening gained 0.0184 over retaining the full skill before generalization. The complete system has the highest reported reward. Its 0.917 includes runtime selection; the difference from a control is therefore a system-level comparison, not an estimate of pruning alone at equal search cost. The controls also differ in offline work: text-only shortening uses no deletion rollouts, whereas verified pruning explicitly pays for those tests.
In the 49-task pruning analysis, verification removed 87.0% of operations and reduced the rendered skill by 77.1% of tokens, at 11.06 additional evaluations per task. For one concrete source,
da-5-1, the original, LLM-shortened, and verified-pruned artifacts occupied 171, 55, and 18 lines, respectively. In a separate block-search experiment, grouping operations reduced evaluations from 10.50 to 6.12 per task on the 16-task common subset. These measurements quantify both the extra work during construction and the reduction in the artifact supplied at transfer.W2 & Q3. Sign of Eq. (4) and the implemented acceptance rule
Thank you for pointing out the reversed sign. We have corrected Eq. (4) in Section 3.3 of the revised manuscript. Eq. (2) defines Δ=U(S)−U(S′), where S′ is the skill after deletion, so a positive Δ denotes a loss in utility. The condition U(S′)≥U(S)−zSE is therefore equivalent to Δ≤zSE. The original manuscript incorrectly stated Δ≥−zSE. The reversed sign was confined to the manuscript, so correcting Eq. (4) does not affect any reported experimental result.