- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a Stage-aware Skill Graph for retrieving reusable skills for multi-stage agent tasks. It augments complete skill nodes with internal stage nodes and typed relations, combines skill-seeded propagation on the full graph with stage-seeded propagation on the stage subgraph, and aggregates stage evidence back to complete skills. A planner additionally decomposes tasks into stage-aligned queries and organizes the retrieved skills into an execution-oriented summary.
Critical strengths
The paper addresses the important and practically relevant problem of skill retrieval for complex tasks.
Critical weaknesses
- Key analyses are missing or insufficient. Offline construction costs are not reported. Existing ablations do not adequately separate stage-graph benefits from task decomposition. Although downstream experiments are run twice, no standard deviations or significance analyses are provided, leaving the robustness of small gains unclear.
- The presentation and metrics require clarification. Figure 1 and 2 are not yet publication-ready. For example, text alignment and line breaks need refinement, while whitespace and panel space are poorly balanced. The figure requires substantial layout improvements to ensure readability. Moreover, Table 1 reports Ret@1=0.7% and MRR=0.4% for Vector Skills on ALFWorld. Standard MRR should be at least Hit@1. The paper describes Ret@K as "whether the correct skill appears among the top-K retrieved skills," suggesting Hit@K (or Recall@K ?).
Questions for authors
- What are the time, model-usage, and storage costs of stage extraction, embedding, and graph construction? And could the authors compare their methods against the baselines?
- With fixed plans, retrieval budgets, and context budgets, what retrieval and downstream gains come from stage text, structural edges, and S-PPR separately?
- Could the authors report means, standard deviations, and paired significance analyses over more independent downstream runs?
- Which experiments use the frozen recipe, and which main results enable filtering? Please compare accuracy, edge counts, and runtime with and without filtering. (Description in C.4 is confusing, like "dep_edge_filter=none")
- Is Ret@K Hit@K or Recall@K? How are multiple relevant skills, MRR, and averaging weights handled? Please provide the formulation of metrics.
- How can Ret@1=0.7% and MRR=0.4% be reconciled?
4 comments
Question 1 — What are the extraction, embedding, construction, and baseline costs?
Appendix F reports preparation usage for the 1,000-skill pool alongside GPT-5.2 execution:
| Method | Preparation tokens | Agent tokens | ALFWorld success |
|---|---|---|---|
| Vector Skills | 906,091 | 8,330,721 | 60.0% |
| SkillRouter | 906,091 | 24,040,021 | 29.3% |
| AgentSkillOS | 1,671,648 | ≥21,612,177 | 13.6% |
| GoS | 681,288 | 9,309,506 | 63.6% |
| SkillDAG | 16,635,181* | 13,771,733 | 82.0% |
| SSG | 6,818,038 | 10,440,060 | 86.1% |
SkillDAG's preparation entry includes 2,863,448 construction-window tokens and 13,771,733 tokens from one execution run.
SSG preparation comprises 978,683 node-indexing tokens, 4,165,787 SSL input tokens, and 1,673,568 serialized SSL output tokens. Its agent-token usage is lower than SkillRouter, AgentSkillOS, and SkillDAG, while its success rate is higher. The added cost table reports model usage; component-wise construction wall time and storage are not included.
Question 2 — What do stage text, structural edges, and S-PPR contribute under fixed queries and budgets?
Appendix D.1 fixes the queries and retrieval budget within each block:
| Benchmark / stage queries | Ret@1 | Ret@5 | MRR@5 |
|---|---|---|---|
| ALFWorld / Full | 49.3 | 71.4 | 57.9 |
| ALFWorld / w/o S-PPR | 48.9 | 70.6 | 57.7 |
| ALFWorld / w/o stage nodes | 48.3 | 69.3 | 56.6 |
| SkillsBench / Full | 69.2 | 86.8 | 76.8 |
| SkillsBench / w/o S-PPR | 68.5 | 84.2 | 75.1 |
| SkillsBench / w/o stage nodes | 65.8 | 82.7 | 72.8 |
Full SSG improves SkillsBench Ret@1/Ret@5/MRR@5 by 3.4/4.1/4.0 points over the skill-only projection. The stage-node removal measures stage text and incident edges together; their individual effects are not separately ablated. End-to-end SSG reaches 72.9% with conventional LLM-generated queries and 76.4% with the stage planner.
Question 3 — Can you add runs, means, standard deviations, and paired significance tests?
Appendix G reports both run outcomes. The run-level means and sample standard deviations, calculated from those success counts, are:
| Model | Run 1 | Run 2 | Mean success (%) | Sample SD (percentage points) |
|---|---|---|---|---|
| MiniMax-M2.7 | 107/140 | 107/140 | 76.43 | 0.00 |
| GPT-5.2 | 126/140 | 115/140 | 86.07 | 5.56 |
| GPT-5.6-Terra | 120/140 | 120/140 | 85.71 | 0.00 |
Rewrites and retrieval results are cached across repeats. MiniMax's successful task sets differ despite equal aggregate rates.
Question 4 — Which results use filtering, and how do accuracy, graph size, and runtime change?
The frozen recipe's dep_edge_filter=none disables additional query-time filtering. Build-time pruning is a separate operation, so a pruned graph can also be queried with this setting. Appendix D.5 identifies full, disabled, and DF-only graph configurations and compares them directly. The main query–structure ablations use the filtered graph; the historical frozen recipe has no additional query-time filter.
| Filtering | Edges | Hit@1 | MRR@10 | Query startup (s) | Materialization (s) |
|---|---|---|---|---|---|
| Full | 69,759 | 0.8289 | 0.8723 | 0.16 | 0.17 |
| Disabled | 217,395 | 0.8421 | 0.8801 | 0.65 | 0.80 |
| DF only | 117,734 | 0.8289 | 0.8723 | 0.18 | 0.54 |
The full-filter graph reduces stored edges by 67.9% relative to the disabled graph, while Hit@1 changes by 1.32 percentage points. It matches DF-only accuracy with fewer edges and shorter materialization time. The timings describe graph startup and node/edge materialization.
Question 5 — Is Ret@K Hit@K or Recall@K? How are multiple relevant skills and averaging handled?
Ret@K is Hit@K: an item receives one hit if any of its reference skills occurs in the top K. Multiple reference skills still produce one hit per item. MRR@5 uses the reciprocal rank of the first reference skill within five results, or zero if none appears. Query instances receive equal weight by default. Appendix A.6 now gives these definitions explicitly.
Explicitly, for reference set G_i and returned list L_i,
Ret@K = (1/N) sum_i 1[topK(L_i) intersects G_i].
MRR@K = (1/N) sum_i (1/r_i if r_i <= K, otherwise 0), where r_i is the first relevant rank. Shorter lists are evaluated at their actual length. These formulas specify both the multi-reference treatment and the averaging weights.
Question 6 — How is Ret@1 = 0.7% versus MRR = 0.4% reconciled?
We apologize for the transcription error. The corrected ALFWorld Vector Skills values are Ret@1 = 0.0%, Ret@5 = 0.7%, and MRR@5 = 0.4%. MRR is therefore greater than Ret@1, as required. The previous 0.7% belongs to Ret@5, not Ret@1, and the corrected row is now used in the manuscript.
Thank you for the rebuttal. However, the revised tables still contain internal inconsistencies. First, Tables 15 and 11 both correspond to the filtered graph and the 76 SkillsBench queries, yet Hit@1 is 63/76 in one and 66/76 in the other. Additionally, the MRR@10 in Table 15 is lower than the MRR@5 in Table 11, which is impossible for the same set of ranked results. Second, the revision also changes two baseline values in Table 1 without any explanation in the response (SkillDAG's AgentSkillOS Ret@5, 56.1 to 83.3. Vector Skills' ALFWorld Ret@5, 4.5 to 0.7). Third, the 82.0% reported in Table 2 for SkillDAG on GPT-5.2/ALFWorld is not attainable with 140 tasks, whether from one run or two, and this entry is the strongest baseline in that block. I therefore consider the main results of this paper unverifiable, and I maintain my rating.
Draft response to the follow-up on internal consistency
Thank you for checking these entries carefully. You are correct that the revised tables did not provide a consistent account of the evaluation protocol and denominators. We apologize for the reporting and evaluation errors. We have rerun the filtering comparison under the Table 11 protocol, audited the baseline rankings, and completed the missing SkillDAG episode. We address each concern below.
1. Tables 11 and 15: rankings and MRR.
We agree that, for the same ranked results, Hit@1 must agree and MRR@10 cannot be lower than MRR@5. Our earlier explanation based on S-PPR was insufficient: Table 11 also includes a w/o S-PPR condition. The two evaluation scripts additionally differed in candidate budget, input truncation, and skill-ID handling. Table 11 used complete instructions and a budget of five candidates before reranking; the earlier filtering table used a budget of 50 and instructions truncated to 2,000 characters. Because candidate selection precedes reranking, this budget changes the ranked results.
We reran all three graph artifacts using exactly the Table 11 Full, w/o LLM query rewriting protocol: the same 76 complete instructions, canonical directory IDs, cached query embeddings, separate retrieval with S-PPR, and a five-candidate budget. The corrected filtering results are:
| Filtering configuration | Stored edges | Hit@1 | Ret@5 | MRR@5 |
|---|---|---|---|---|
| Full | 69,759 | 66/76 = 86.8% | 73/76 = 96.1% | 90.9% |
| Disabled, historical artifact | 217,395 | 66/76 = 86.8% | 74/76 = 97.4% | 91.6% |
| DF only | 117,734 | 66/76 = 86.8% | 73/76 = 96.1% | 90.9% |
The full-filter row reproduces all 76 saved top-5 rankings of the corresponding Table 11 Full row. We also reran w/o S-PPR as a check: its full-filter result reproduces Table 11's 86.8/98.7/92.3 (%) and all 76 rankings. The filtering table now reports MRR@5.
2. Table 1: the two baseline corrections.
SkillDAG's AgentSkillOS value of 56.1% was transcribed from task-average set recall, mean(|top-5 ∩ GT| / |GT|). Our stated Ret@K is Hit@K, mean(1[top-K ∩ GT ≠ ∅]). Auditing the same 30 saved rankings gives 25/30 = 83.3% Hit@5. Ret@1 and MRR@5 remain 43.3% and 60.8%. Thus the correction is from an incorrectly selected metric field, not a new retrieval run. We should have explained this specific change in the earlier response.
The Vector Skills ALFWorld correction is a separate transcription error. The saved original-index rankings contain no rank-1 hit and one rank-2 hit across 140 tasks: Ret@1 = 0/140, Ret@5 = 1/140 = 0.7143%, and MRR@5 = (1/2)/140 = 0.3571%. The corrected row is 0.0/0.7/0.4 (%). The previous 4.5% was not Hit@5. These values describe the original distributed index, which failed a sampled index self-retrieval audit; the manuscript now states that limitation rather than presenting this row as a validated aligned-index estimate.
3. Table 2: SkillDAG's 82.0% on GPT-5.2/ALFWorld.
he previous discrepancies in the values were due to a missing record caused by an error in an earlier experiment, which resulted in the use of an incomplete denominator for the calculations. We acknowledge this oversight in failing to promptly identify and address the missing denominator. We have now successfully executed the missing episode (which completed smoothly after several steps, yielding an exact success rate of 82.1429%, rounded to 82.1% as per standard practice). We have verified the uniqueness of the task names, the consistency of the system prompt, and confirmed that the hashes for the original records, graph, and embedding cache remain unchanged. Additionally, the corresponding token accounting has been updated accordingly. The revised manuscript now includes the corrected values and provides comprehensive details on metric definitions, candidate budget, ID handling, denominators, and completion provenance.
Weakness 1 — Missing costs, component attribution, and repeated-run analysis.
Appendix F adds preparation and agent-token comparisons. SSG uses 6,818,038 preparation tokens and 10,440,060 GPT-5.2 agent tokens, with 86.1% success. SkillDAG uses 2,863,448 construction-window tokens and 13,771,733 agent tokens, with 82.0% success. Appendix D.1 fixes query source and compares full, unified-PPR, and skill-only structures: SkillsBench stage-query Ret@1/Ret@5/MRR@5 are 69.2/86.8/76.8, 68.5/84.2/75.1, and 65.8/82.7/72.8, respectively.
Appendix G reports both run outcomes. The run-level means and sample standard deviations, calculated from those success counts, are:
Rewrites and retrieval results are cached across repeats. MiniMax's successful task sets differ despite equal aggregate rates. The revision reports two runs per model; a paired significance test is not reported.
Weakness 2 — Presentation quality and inconsistent metric definitions.
We have standardized parenthetical citations and clarified metric definitions in Appendix A.6. Ret@K is Hit@K, and MRR@5 uses the first relevant rank. The Vector Skills ALFWorld row is corrected to 0.0/0.7/0.4 (%). We will refine Figures 1–2 for text alignment, line breaks, spacing, and balanced panel layout.