- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SKILLGRAPH organizes natural-language skills into a directed graph with prerequisite, enhancement, and co-occurrence relations. It retrieves an ordered collection of skills to guide an LLM agent, updates skill nodes and relations using trajectory feedback and teacher-model operations, and progressively unlocks skills while optimizing the policy with GRPO. Experiments use Qwen2.5-7B-Instruct on ALFWorld, WebShop, and seven search-augmented QA benchmarks. The paper reports improvements over SkillRL and skill-free GRPO, together with graph interventions and analyses of training dynamics and computational cost.
Critical strengths
-
A relevant and coherent approach to reusable agent knowledge. The paper addresses a meaningful limitation of independent skill retrieval: multi-step tasks can require both relevant advice and knowledge of how that advice composes. Sections 3.1–3.4 integrate retrieval, node maintenance, relation updates, and policy training in a coherent framework. The combination is practically useful, and the appendices provide skill schemas, prompts, hyperparameters, and maintenance details that make the proposal more concrete than a purely conceptual graph formulation.
-
Encouraging results with interventions relevant to the proposed mechanism. Table 1 reports ALFWorld success of 93.1% versus 89.9% for SkillRL, and WebShop success of 84.4% versus 72.7% (task scores: 91.5 versus 85.2). These gains, especially on WebShop, warrant attention. The evaluation spans household interaction, shopping, and search-based QA. Table 8 further reports that removing prerequisite edges lowers ALFWorld success to 87.5% and WebShop score to 82.5, supporting the utility of those edges within the full system. Flat-library and static-graph controls, together with the cost analysis in Appendix I.3, strengthen the empirical contribution.
Critical weaknesses
- The origin and validity of prerequisite structure are not sufficiently established. Appendix A.2 explicitly initializes no prerequisite edges. Section 3.3.2 reinforces existing edges and discovers co-occurrence links, while Appendix A.4 creates prerequisite edges when a teacher splits a skill into ordered sub-skills. This provides one concrete construction mechanism, but the paper does not specify a general operation that discovers a new prerequisite relation between existing skills. It is consequently unclear how much of the claimed dependency discovery comes from execution evidence versus teacher-proposed decomposition. The Figure 2 discussion also attributes growth in prerequisite and enhancement edge counts to path reinforcement, although increasing an existing edge weight cannot itself increase the edge count.
I acknowledge that Appendix A.1 defines the prerequisite/enhancement dependency subgraph and excludes co-occurrence from level computation, and Appendix I.5 distinguishes the relation types. My concern is therefore not that all edge types are indiscriminately ordered. Rather, Equation (2) should explicitly identify its ordering graph, and the update procedure should explain cycle handling. Representative learned prerequisite edges with their provenance and supporting trajectories would help establish that the graph captures valid procedural dependencies.
Teacher-proposed dependencies are not inherently a weakness. The distinction matters because the source and validation of these relations determine what the method learns and whether the claimed dependency-discovery mechanism is reproducible.
- The evidence supports the full system more strongly than the proposed explanation of its gains. Appendix I.1 provides useful controls, but retrieval without topology removes dependency expansion and ordering together. Its ALFWorld drop from 93.1% to 59.4% therefore does not isolate order from changes in retrieved content. Prerequisite removal also affects both candidate expansion and ordering. Holding the selected skills fixed and shuffling only their presentation order would clarify whether ordered guidance contributes beyond skill selection. Such an evaluation would test the trained policy’s use of order; a training ablation would additionally be needed to attribute improved learning to order.
Appendix A.3 credits every retrieved skill when an episode succeeds. The maintenance statistics therefore measure successful exposure, not independent utility. This is a plausible heuristic, and I do not require a new causal-credit algorithm. However, the rising node-success curve should not itself be interpreted as validation of skill quality; a targeted maintenance ablation would be useful supporting evidence.
The reported results lack seed-level uncertainty and clearly stated evaluation-set sizes. Appendix I.3 says the supplementary comparisons reuse public SkillRL SFT data, and Appendix I.5 fixes the base model; these are useful controls. However, the paper should consolidate which comparisons also match interaction, teacher, and prompt budgets. This is particularly
Reporting inconsistencies further limit interpretation. The seven SKILLGRAPH entries in Table 2 average to approximately 48.64, consistent with 48.6 in Section 4.2 and Appendix I.4, but not 48.9 in Tables 2 and 6. Table 3 is captioned as success rate and reports WebShop 84.4, matching Table 1 success, whereas Appendix C calls Table 3 task score. These are correctable issues, not evidence that the experiments are invalid, but the quantitative support should be made internally consistent.
Questions for authors
-
Which operations create prerequisite edges, and how many edges originate from each? If splitting is the only source, please state this explicitly and provide examples showing that the resulting directions reflect genuine prerequisites.
-
What graph is ordered in Equation (2), and how are cycles handled? How do depth-two backward expansion, the eight-skill cap, and active-skill filtering interact? Please clarify whether returned skills retain all required prerequisites.
-
With the trained policy and selected skills fixed, how does shuffling only skill order affect success? This is my highest-priority additional experiment, because it directly tests whether the policy uses the ordering. Separate maintenance or unlocking ablations would be useful supporting evidence, rather than prerequisites for answering this question.
-
Can you report repeated-seed results and evaluation counts for the primary SkillRL comparison, clarify matched budgets, and reconcile the QA averages and WebShop ablation metric? Appendix E also calls step 200 the best-average checkpoint: was it fixed in advance, or selected using held-out QA benchmark results?