- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Skill2Task, a framework for synthesizing executable agent tasks from community skills. It builds a skill-state graph using embeddings and clustering, generates tasks through progressive validation and repair, and updates the synthesis harness based on failure feedback. The authors collect 10,880 successful trajectories for SFT on two Qwen3.5 models and report improvements across four agent benchmarks.
Critical strengths
-
The paper presents a reasonably complete pipeline for automated agent task synthesis, including skill composition, task validation, harness refinement, and trajectory collection.
-
The experimental results are encouraging, with consistent improvements across four agent benchmarks and two model sizes.
Critical weaknesses
-
The graph is constructed by clustering skill preconditions and postconditions using Qwen embeddings. However, semantic similarity does not necessarily imply logical compatibility. Two states may be close in embedding space without one satisfying the requirements of the other. The paper lacks direct evaluation of whether the resulting connections and sampled skill paths are logically valid.
-
During progressive validation, the framework can modify tasks to resolve compatibility issues. For example, the task repair case in Appendix A.15 replaces part of the original workflow with direct processing of an existing file. Such repairs may remove the dependencies between the original skills. The paper does not evaluate whether these dependencies are preserved in the final tasks.
-
The evaluation is limited to two Qwen models. Both models belong to the same family, making it difficult to assess how well the synthesized data generalizes. I would like to see the generated tasks evaluated by independent state-of-the-art API agents, such as GPT or Claude, to better assess their difficulty, quality, and actual skill requirements. The GLM-5.2 teacher rollouts alone are not sufficient for this purpose.
-
Table 2 only compares graph-based sampling against random skill combinations. Since random combinations are likely to produce incompatible workflows, this comparison is not particularly convincing. Stronger baselines, such as LLM-based skill planning or simple embedding-based retrieval without graph construction, would help establish whether the proposed graph is actually necessary.
-
Although the valid-bundle rate improves from 42.3% to 91.5%, the corresponding downstream performance gains are much smaller. In addition, the same 200 held-out paths are repeatedly used to select harness updates, raising concerns about overfitting to this validation set. Evaluation on a separate test set and comparison with a stronger static harness under a matched compute budget would make the results more convincing.
Questions for authors
-
How do the authors verify the logical compatibility of connected skills? Can they provide an evaluation of graph edge correctness and sampled path validity?
-
How much of the original skill dependency structure is preserved after task repair? Do the final tasks genuinely require all the sampled skills?
-
Can the authors evaluate the synthesized tasks using independent SOTA API agents and test whether the training gains transfer to non-Qwen models?
-
How does the proposed graph compare with LLM-based skill planning or simpler semantic retrieval methods?
3 comments
Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.
Thanks for the rebuttal and the additional experiments. Most of my concerns have been addressed, and I appreciate the effort.
However, I still have two concerns about the lack of comparisons with stronger alternatives to graph-based sampling, such as retrieval-assisted LLM planning, and the lack of evaluation on independent paths that were never used for diagnosis or update selection. I agree with your explanations, but these comparisons are still needed to establish the value of the graph and self-evolution components.
Also, my “Weaknesses” and “Questions for Authors” are not entirely repetitive, and some specific questions are still only partly answered. For example, the dependency-retention numbers do not fully answer whether the final tasks actually require all the sampled skills.
Given these remaining gaps, I don’t think the current evidence is sufficient for acceptance at ICLR, so I will maintain my score.
Thank you for your great efforts in reviewing this paper. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response as soon as possible.
[W1] Embedding similarity does not establish logical compatibility of graph connections or paths.
[A]: Thank you for asking about skill compatibility. An LLM first extracts each skill's pre/post-execution scenarios; embeddings cluster similar scenarios into shared nodes, with skills forming directed edges. Paths connect output to input scenarios, while downstream validation checks concrete execution requirements. We used gpt-5.6-sol to assess all 10,916 released tasks from their skill sequences, source documents, scenarios, and final instructions. Compatibility required the upstream output scenario to support the downstream requirements. 28,830/30,796 adjacent transitions (93.62%) were judged compatible and 10,414/10,916 workflows (95.40%) coherent, counting uncertain cases as non-passes.
[W2] Task repair may bypass original skill dependencies.
[A]: Thank you for highlighting dependency preservation. Appendix A.15 supplies an existing HDF5 artifact, bypassing upstream modeling/simulation; we will identify this dependency loss explicitly. Two annotators compared the sampled skills, pre-repair requirements, and final instructions/assets for 100 repaired tasks. A dependency counted only if the upstream contribution remained necessary downstream; pre-supplied results did not count. 294/320 dependencies (91.88%) and 84/100 complete chains were retained. Thus, preservation is not universal; we will distinguish sampled path length from retained dependency length.
[W3] Independent API-agent evaluation and transfer beyond Qwen are missing.
[A]: Thank you for suggesting these evaluations. Each model below made four attempts per task on the same 200 tasks, stratified by domain and path length without teacher-success filtering. Tools, skill access, and budgets were matched; references and verifiers were hidden. We report avg@4, the mean binary success rate over all 800 attempts per model. Qwen is the original, pre-SFT checkpoint.
Gemma-4-E4B-it improves on all four benchmarks after Skill2Task SFT, with a +5.18 mean gain under identical evaluation settings (avg@8 on PinchBench, avg@4 elsewhere). Training uses 10,240 retained trajectories and 251M supervised tokens per epoch, a different data budget from Qwen. This single run provides cross-family evidence without establishing training-seed significance.
[W4] Random composition is insufficient to establish the graph's value over stronger planners.
[A]: Thank you for suggesting these alternatives. (1) LLM planning: supplying all 84,728 skills in one prompt is impractical; retrieval-assisted planning reduces this burden but requires retrieval and planning for each task. (2) Simple embedding retrieval: similar skill descriptions do not establish execution order or output-to-input compatibility.
Our graph offers (1) scenario-based composition: embeddings cluster LLM-extracted pre/post scenarios into nodes, with skill edges connecting output to input scenarios; concrete compatibility is checked downstream. (2) Reusable, controllable sampling: the graph supports repeated sampling, with path length controlling the intended number of composed skills. These features motivate the design; Table 2 establishes its benefit over random composition, not empirical superiority to the proposed planners.
[W5] Repeated selection-set use risks overfitting; construction gains exceed downstream gains.
[A]: Thank you for asking about generalization and downstream impact. Section 4.3 and Appendix A.9 specify 300 held-in diagnostic paths and 200 disjoint held-out selection paths. Held-in feedback drives edits, while held-out yield rises 42.3% → 91.5%, supporting transfer beyond diagnostic paths. Repeated selection still makes this a validation set, so these results do not establish generalization to an untouched test set. Evolution primarily improves construction efficiency: repair iterations fall 7.42 → 3.91. Downstream gains are a separate measure: Figure 3b caps training-task counts yet improves 46.40 → 49.14. Downstream benchmarks provide neither update feedback nor acceptance criteria, avoiding direct optimization for those evaluations.