acceptodds
Under review as a conference paper at ICLR 2027

From Skills to Stages: Stage-Aware Graph Retrieval for Relevant Skills in Large-Scale Agent Skill Libraries

Abstract

Agents based on Large Language Models (LLMs) increasingly rely on reusable skills. As these agents become deeply integrated with personal applications, web browsers, and various interactive interfaces, the scale of skill libraries has expanded to thousands of entries. Currently, numerous methods have attempted to address this challenge through skill retrieval. However, existing approaches treat each skill as an opaque monolith, matching solely based on holistic descriptions or global features. Since a skill actually consists of multiple execution stages—each with its own independent goals, preconditions, and execution outcomes—compressing this internal structure into a single representation makes it impossible to distinguish between skills that have similar descriptions but different execution workflows. To address the aforementioned limitations, we propose the Stage-aware Skill Graph (SSG), which explicitly models two node hierarchies during graph construction: the skill level and the internal stage level. During inference, we introduce a stage-aware retrieval mechanism that combines stage seed selection, stage-aware PageRank, and stage-aware query rewriting to retrieve complementary skills for different stages of task execution. Experiments on ALFWorld, SkillsBench, and AgentSkillOS with three representative models demonstrate the effectiveness of our approach. SSG achieves a task success rate of 76.4% on ALFWorld, outperforming the strongest baseline by 8.5 percentage points, and achieves up to a 7.8 percentage-point improvement over the strongest baseline on SkillsBench. Across all three benchmarks and model settings, SSG consistently improves end-to-end task performance, highlighting the benefit of stage-aware skill retrieval for multi-stage task execution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agent, Skill, Graph Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XCeEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a Stage-aware Skill Graph for retrieving reusable skills for multi-stage agent tasks. It augments complete skill nodes with internal stage nodes and typed relations, combines skill-seeded propagation on the full graph with stage-seeded propagation on the stage subgraph, and aggregates stage evidence back to complete skills. A planner additionally decomposes tasks into stage-aligned queries and organizes the retrieved skills into an execution-oriented summary.

Critical strengths

The paper addresses the important and practically relevant problem of skill retrieval for complex tasks.

Critical weaknesses
  1. Key analyses are missing or insufficient. Offline construction costs are not reported. Existing ablations do not adequately separate stage-graph benefits from task decomposition. Although downstream experiments are run twice, no standard deviations or significance analyses are provided, leaving the robustness of small gains unclear.
  2. The presentation and metrics require clarification. Figure 1 and 2 are not yet publication-ready. For example, text alignment and line breaks need refinement, while whitespace and panel space are poorly balanced. The figure requires substantial layout improvements to ensure readability. Moreover, Table 1 reports Ret@1=0.7% and MRR=0.4% for Vector Skills on ALFWorld. Standard MRR should be at least Hit@1. The paper describes Ret@K as "whether the correct skill appears among the top-K retrieved skills," suggesting Hit@K (or Recall@K ?).
Questions for authors
  1. What are the time, model-usage, and storage costs of stage extraction, embedding, and graph construction? And could the authors compare their methods against the baselines?
  2. With fixed plans, retrieval budgets, and context budgets, what retrieval and downstream gains come from stage text, structural edges, and S-PPR separately?
  3. Could the authors report means, standard deviations, and paired significance analyses over more independent downstream runs?
  4. Which experiments use the frozen recipe, and which main results enable filtering? Please compare accuracy, edge counts, and runtime with and without filtering. (Description in C.4 is confusing, like "dep_edge_filter=none")
  5. Is Ret@K Hit@K or Recall@K? How are multiple relevant skills, MRR, and averaging weights handled? Please provide the formulation of metrics.
  6. How can Ret@1=0.7% and MRR=0.4% be reconciled?
4 comments

Weakness 1 — Missing costs, component attribution, and repeated-run analysis.

Appendix F adds preparation and agent-token comparisons. SSG uses 6,818,038 preparation tokens and 10,440,060 GPT-5.2 agent tokens, with 86.1% success. SkillDAG uses 2,863,448 construction-window tokens and 13,771,733 agent tokens, with 82.0% success. Appendix D.1 fixes query source and compares full, unified-PPR, and skill-only structures: SkillsBench stage-query Ret@1/Ret@5/MRR@5 are 69.2/86.8/76.8, 68.5/84.2/75.1, and 65.8/82.7/72.8, respectively.

Appendix G reports both run outcomes. The run-level means and sample standard deviations, calculated from those success counts, are:

ModelRun 1Run 2Mean success (%)Sample SD (percentage points)
MiniMax-M2.7107/140107/14076.430.00
GPT-5.2126/140115/14086.075.56
GPT-5.6-Terra120/140120/14085.710.00

Rewrites and retrieval results are cached across repeats. MiniMax's successful task sets differ despite equal aggregate rates. The revision reports two runs per model; a paired significance test is not reported.

Weakness 2 — Presentation quality and inconsistent metric definitions.

We have standardized parenthetical citations and clarified metric definitions in Appendix A.6. Ret@K is Hit@K, and MRR@5 uses the first relevant rank. The Vector Skills ALFWorld row is corrected to 0.0/0.7/0.4 (%). We will refine Figures 1–2 for text alignment, line breaks, spacing, and balanced panel layout.

Question 1 — What are the extraction, embedding, construction, and baseline costs?

Appendix F reports preparation usage for the 1,000-skill pool alongside GPT-5.2 execution:

MethodPreparation tokensAgent tokensALFWorld success
Vector Skills906,0918,330,72160.0%
SkillRouter906,09124,040,02129.3%
AgentSkillOS1,671,648≥21,612,17713.6%
GoS681,2889,309,50663.6%
SkillDAG16,635,181*13,771,73382.0%
SSG6,818,03810,440,06086.1%

SkillDAG's preparation entry includes 2,863,448 construction-window tokens and 13,771,733 tokens from one execution run.

SSG preparation comprises 978,683 node-indexing tokens, 4,165,787 SSL input tokens, and 1,673,568 serialized SSL output tokens. Its agent-token usage is lower than SkillRouter, AgentSkillOS, and SkillDAG, while its success rate is higher. The added cost table reports model usage; component-wise construction wall time and storage are not included.

Question 2 — What do stage text, structural edges, and S-PPR contribute under fixed queries and budgets?

Appendix D.1 fixes the queries and retrieval budget within each block:

Benchmark / stage queriesRet@1Ret@5MRR@5
ALFWorld / Full49.371.457.9
ALFWorld / w/o S-PPR48.970.657.7
ALFWorld / w/o stage nodes48.369.356.6
SkillsBench / Full69.286.876.8
SkillsBench / w/o S-PPR68.584.275.1
SkillsBench / w/o stage nodes65.882.772.8

Full SSG improves SkillsBench Ret@1/Ret@5/MRR@5 by 3.4/4.1/4.0 points over the skill-only projection. The stage-node removal measures stage text and incident edges together; their individual effects are not separately ablated. End-to-end SSG reaches 72.9% with conventional LLM-generated queries and 76.4% with the stage planner.

Question 3 — Can you add runs, means, standard deviations, and paired significance tests?

Appendix G reports both run outcomes. The run-level means and sample standard deviations, calculated from those success counts, are:

ModelRun 1Run 2Mean success (%)Sample SD (percentage points)
MiniMax-M2.7107/140107/14076.430.00
GPT-5.2126/140115/14086.075.56
GPT-5.6-Terra120/140120/14085.710.00

Rewrites and retrieval results are cached across repeats. MiniMax's successful task sets differ despite equal aggregate rates.

Question 4 — Which results use filtering, and how do accuracy, graph size, and runtime change?

The frozen recipe's dep_edge_filter=none disables additional query-time filtering. Build-time pruning is a separate operation, so a pruned graph can also be queried with this setting. Appendix D.5 identifies full, disabled, and DF-only graph configurations and compares them directly. The main query–structure ablations use the filtered graph; the historical frozen recipe has no additional query-time filter.

FilteringEdgesHit@1MRR@10Query startup (s)Materialization (s)
Full69,7590.82890.87230.160.17
Disabled217,3950.84210.88010.650.80
DF only117,7340.82890.87230.180.54

The full-filter graph reduces stored edges by 67.9% relative to the disabled graph, while Hit@1 changes by 1.32 percentage points. It matches DF-only accuracy with fewer edges and shorter materialization time. The timings describe graph startup and node/edge materialization.

Question 5 — Is Ret@K Hit@K or Recall@K? How are multiple relevant skills and averaging handled?

Ret@K is Hit@K: an item receives one hit if any of its reference skills occurs in the top K. Multiple reference skills still produce one hit per item. MRR@5 uses the reciprocal rank of the first reference skill within five results, or zero if none appears. Query instances receive equal weight by default. Appendix A.6 now gives these definitions explicitly.

Explicitly, for reference set G_i and returned list L_i,

Ret@K = (1/N) sum_i 1[topK(L_i) intersects G_i].

MRR@K = (1/N) sum_i (1/r_i if r_i <= K, otherwise 0), where r_i is the first relevant rank. Shorter lists are evaluated at their actual length. These formulas specify both the multi-reference treatment and the averaging weights.

Question 6 — How is Ret@1 = 0.7% versus MRR = 0.4% reconciled?

We apologize for the transcription error. The corrected ALFWorld Vector Skills values are Ret@1 = 0.0%, Ret@5 = 0.7%, and MRR@5 = 0.4%. MRR is therefore greater than Ret@1, as required. The previous 0.7% belongs to Ret@5, not Ret@1, and the corrected row is now used in the manuscript.

Thank you for the rebuttal. However, the revised tables still contain internal inconsistencies. First, Tables 15 and 11 both correspond to the filtered graph and the 76 SkillsBench queries, yet Hit@1 is 63/76 in one and 66/76 in the other. Additionally, the MRR@10 in Table 15 is lower than the MRR@5 in Table 11, which is impossible for the same set of ranked results. Second, the revision also changes two baseline values in Table 1 without any explanation in the response (SkillDAG's AgentSkillOS Ret@5, 56.1 to 83.3. Vector Skills' ALFWorld Ret@5, 4.5 to 0.7). Third, the 82.0% reported in Table 2 for SkillDAG on GPT-5.2/ALFWorld is not attainable with 140 tasks, whether from one run or two, and this entry is the strongest baseline in that block. I therefore consider the main results of this paper unverifiable, and I maintain my rating.

Draft response to the follow-up on internal consistency

Thank you for checking these entries carefully. You are correct that the revised tables did not provide a consistent account of the evaluation protocol and denominators. We apologize for the reporting and evaluation errors. We have rerun the filtering comparison under the Table 11 protocol, audited the baseline rankings, and completed the missing SkillDAG episode. We address each concern below.

1. Tables 11 and 15: rankings and MRR.

We agree that, for the same ranked results, Hit@1 must agree and MRR@10 cannot be lower than MRR@5. Our earlier explanation based on S-PPR was insufficient: Table 11 also includes a w/o S-PPR condition. The two evaluation scripts additionally differed in candidate budget, input truncation, and skill-ID handling. Table 11 used complete instructions and a budget of five candidates before reranking; the earlier filtering table used a budget of 50 and instructions truncated to 2,000 characters. Because candidate selection precedes reranking, this budget changes the ranked results.

We reran all three graph artifacts using exactly the Table 11 Full, w/o LLM query rewriting protocol: the same 76 complete instructions, canonical directory IDs, cached query embeddings, separate retrieval with S-PPR, and a five-candidate budget. The corrected filtering results are:

Filtering configurationStored edgesHit@1Ret@5MRR@5
Full69,75966/76 = 86.8%73/76 = 96.1%90.9%
Disabled, historical artifact217,39566/76 = 86.8%74/76 = 97.4%91.6%
DF only117,73466/76 = 86.8%73/76 = 96.1%90.9%

The full-filter row reproduces all 76 saved top-5 rankings of the corresponding Table 11 Full row. We also reran w/o S-PPR as a check: its full-filter result reproduces Table 11's 86.8/98.7/92.3 (%) and all 76 rankings. The filtering table now reports MRR@5.

2. Table 1: the two baseline corrections.

SkillDAG's AgentSkillOS value of 56.1% was transcribed from task-average set recall, mean(|top-5 ∩ GT| / |GT|). Our stated Ret@K is Hit@K, mean(1[top-K ∩ GT ≠ ∅]). Auditing the same 30 saved rankings gives 25/30 = 83.3% Hit@5. Ret@1 and MRR@5 remain 43.3% and 60.8%. Thus the correction is from an incorrectly selected metric field, not a new retrieval run. We should have explained this specific change in the earlier response.

The Vector Skills ALFWorld correction is a separate transcription error. The saved original-index rankings contain no rank-1 hit and one rank-2 hit across 140 tasks: Ret@1 = 0/140, Ret@5 = 1/140 = 0.7143%, and MRR@5 = (1/2)/140 = 0.3571%. The corrected row is 0.0/0.7/0.4 (%). The previous 4.5% was not Hit@5. These values describe the original distributed index, which failed a sampled index self-retrieval audit; the manuscript now states that limitation rather than presenting this row as a validated aligned-index estimate.

3. Table 2: SkillDAG's 82.0% on GPT-5.2/ALFWorld.

he previous discrepancies in the values were due to a missing record caused by an error in an earlier experiment, which resulted in the use of an incomplete denominator for the calculations. We acknowledge this oversight in failing to promptly identify and address the missing denominator. We have now successfully executed the missing episode (which completed smoothly after several steps, yielding an exact success rate of 82.1429%, rounded to 82.1% as per standard practice). We have verified the uniqueness of the task names, the consistency of the system prompt, and confirmed that the hashes for the original records, graph, and embedding cache remain unchanged. Additionally, the corresponding token accounting has been updated accordingly. The revised manuscript now includes the corrected values and provides comprehensive details on metric definitions, candidate budget, ID handling, denominators, and completion provenance.

Reviewer_V1tXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing skill retrievers treat each skill as a monolithic unit, although a skill consists of several execution stages with their own goals, preconditions and outcomes. It proposes the Stage-aware Skill Graph (SSG). The stage nodes that extracted from skill documents by LLM are added to a skill-level graph inherited from Graph-of-Skills (GoS). And the retrieval includes skill-channel PPR on the full graph and a stage-channel PPR on the stage subgraph. Also, stage-aware query-rewriting is applied by an LLM planner to decompose the task into ordered subtasks and the resulting summary of subtasks is injected into the agent context.

Critical strengths
  1. This paper provides a reasonable idea for a timely problem. As agent skill libraries grow to thousands of entries, retrieving the right skill becomes a bottleneck. Viewing a skill as a sequence of internal stages, rather than as a single unit, is a reasonable basis for finer-grained retrieval evidence.

  2. The method is presented in detail. The appendix specifies seed selection, the personalization weights, the transition matrix with reverse coefficients, the PPR stopping rule, the reranking coefficients and the planner output format, which makes the pipeline easier to reproduce.

  3. A transparent reporting and supplementary material are provided. The paper states that rewriting is the primary source of gains (Appendix D), discloses that the build-time filter is not used in the frozen configuration, and marks numbers taken from other papers. The released code allows the most numbers to be recomputed.

Critical weaknesses
  1. The paper's title and contributions emphasize the stage-aware graph, but the most of the improvement are gained from plan injection. In the ALFWorld retrieval ablation (Table 3), removing Stage-aware PPR changes Ret@1 from 49.3 to 48.9, and additionally removing stage nodes lowers it to 48.3. Removing query rewriting lowers it to 35.3. End to end, the variant without rewriting succeeds on 33/140 ALFWorld tasks (23.6%, Appendix E.2). This is far below the full system (76.4%), and also below GoS (54.3% in Table 2) and the no-retrieval Vanilla Skills setting (47.1%).

  2. The comparison with baselines is unfair. The baselines are not applied the same planner decomposition and plan injection. The planner prompt is tailored to each benchmark, with benchmark-specific few-shot examples, domain vocabulary and suggested decompositions (Appendix A.1).

  3. Some reported metrics are internally inconsistent, and several results look anomalous. For example, in Table 1, Vector Skills on ALFWorld reports Ret@1 = 0.7 and MRR = 0.4. On a common query set, MRR cannot be lower than Ret@1.

  4. The paper motivates SSG by the need to distinguish skills with similar descriptions but different workflows, and to find intermediate stages not stated in the task. Neither is evaluated directly.

Questions for authors
  1. The paper is motivated by the need to distinguish skills with similar descriptions but different workflows, and to find intermediate stages that are not stated in the task. Did you evaluate either case directly, for example on pairs of description-similar skills, or by splitting the results by whether the required intermediate stage is stated in the task description?

  2. What is the end-to-end performance of SSG retrieval without plan injection? Could you report a full 2×2 ablation of decomposition on ALFWorld? The variant without both components (GoS with the original task query) is needed to assess whether the stage graph is useful on its own.

  3. Section 3.2.2 presents build-time dependency-edge filtering as part of the method, and Figure 2 includes it in the pipeline. Why does the frozen configuration use dep_edge_filter=none? Section 3.2.2 also states that "experiments show that removing them usually does not reduce retrieval accuracy."

  4. Please clarify the definition of Ret@K and MRR to help me understand the Vector Skills row on ALFWorld (MRR below Ret@1).

3 comments

Weakness 1 & 2— Are the gains mainly due to planning rather than the stage graph?

Appendix D.1 now crosses three query strategies with three graph structures on both benchmarks. With queries fixed within each block, Ret@1 (%) is:

Benchmark / query sourceNFullw/o S-PPRw/o stage nodes
ALFWorld / stage queries53849.348.948.3
ALFWorld / LLM rewriting45935.332.234.0
ALFWorld / no LLM rewriting14017.119.316.4
SkillsBench / stage queries70169.268.565.8
SkillsBench / LLM rewriting8481.081.081.0
SkillsBench / no LLM rewriting7686.886.882.9

Stage nodes improve Ret@1 in five settings and tie in one. S-PPR adds improvements in several settings, including SkillsBench stage-query Ret@5/MRR@5 of 86.8/76.8 versus 84.2/75.1 without S-PPR. Query rewriting is benchmark-dependent: in our experiments on SkillsBench, we observed that disabling query rewriting could actually improve retrieval metrics. However, to ensure a fair comparison, we chose not to include this result in the main table. Ultimately, our performance gains stem primarily from the proposed stage node method.

We also corrected the earlier 33/140 (23.6%) result, which came from an action-interface compatibility problem and was incorrectly labeled as “without rewriting.” The valid configurations are:

ConfigurationSuccessesSuccess rate
SSG, original query89/14063.6%
SSG, conventional LLM-generated queries102/14072.9%
SSG, full stage-planner pipeline107/14076.4%
SkillDAG, LLM-generated queries95/14067.9%

SSG achieves 72.9% without the independent stage planner, versus 67.9% for SkillDAG; the full stage planner increases success to 76.4%.

Weakness 3 — The ALFWorld Vector Skills metrics are inconsistent.

We apologize for the transcription error. The corrected ALFWorld Vector Skills values are Ret@1 = 0.0%, Ret@5 = 0.7%, and MRR@5 = 0.4%. MRR is therefore greater than Ret@1, as required.

Weakness 4 & Question 1 — The two stated motivations are not directly evaluated.

Appendix E.3 examines all 41 cool/heat-and-place episodes. The skill alfworld-temperature-regulator has description cosine similarities of 0.725 and 0.694 to the canonical temperature skills, but mixes appliance-insertion instructions with examples using the correct cool/heat command. GoS ranks it first in 38/41 episodes; 27 of its 37 failures execute an appliance-insertion put action. Single-query SSG raises canonical Ret@1 from 0/41 to 9/41 and Ret@4 from 0/41 to 28/41. Full SSG succeeds on 26/41 tasks (63.4%).

Appendix E.2 also traces “put two newspaper in drawer” through LOCATE → ACQUIRE → ACQUIRE → PLACE. The planner makes the repeated acquisition step explicit and retrieves object-picker at rank 1, while the displayed GoS and SkillDAG lists omit it. This case shows how decomposition exposes an intermediate action that the original instruction does not enumerate as a separate stage.

Question 2 — What happens without plan injection, and can you provide a full 2×2 ablation? Appendix D.1 now crosses three query strategies with three graph structures on both benchmarks.

Question 3 — Why is filtering disabled in the frozen recipe, and what is its measured effect?

Filtering is disabled in the frozen recipe because we treat it as an optional ablation study, which we have detailed in our supplementary experiments. Appendix D.5 identifies full, disabled, and DF-only graph configurations and compares them directly.

FilteringEdgesHit@1MRR@10Query startup (s)Materialization (s)
Full69,7590.82890.87230.160.17
Disabled217,3950.84210.88010.650.80
DF only117,7340.82890.87230.180.54

Full filtering removes 67.9% of stored edges relative to the disabled graph and reduces startup/materialization times by 75.4%/78.8%, with Hit@1 changing from 84.21% to 82.89%. Against DF-only filtering, it preserves Hit@1 and MRR@10 while reducing edges from 117,734 to 69,759.

Question 4 — Please define Ret@K/MRR and explain the Vector Skills row.

Ret@K is Hit@K: an item receives one hit if any of its reference skills occurs in the top K. Multiple reference skills still produce one hit per item. MRR@5 uses the reciprocal rank of the first reference skill within five results, or zero if none appears. Query instances receive equal weight by default. Appendix A.6 now gives these definitions explicitly.

We apologize for the transcription error. The corrected ALFWorld Vector Skills values are Ret@1 = 0.0%, Ret@5 = 0.7%, and MRR@5 = 0.4%. MRR is therefore greater than Ret@1, as required.

Thank you for the detailed response. The corrected Vector Skills values, the metric definitions and the filtering comparison answer my questions about Table 1 and Section 3.2.2, and the Appendix E.3 analysis of the temperature-regulator skill is a useful direct test of the first motivation. Thank you also for correcting the 33/140 result: without LLM rewriting, SSG reaches 63.6%, so my statement that it falls below Vanilla Skills no longer holds.

My main concern about attribution remains. On ALFWorld, your new table shows that, with queries fixed, stage nodes and S-PPR together add about one point of Ret@1 (49.3 vs. 48.3 with stage queries; 17.1 vs. 16.4 without LLM rewriting, i.e., one of 140 tasks). This agrees with §4.3, where removing decomposition causes a degradation "markedly larger than that caused by removing other components", rather than with the response's statement that the gains "stem primarily from the proposed stage node method". End-to-end, 3.5 of the 8.5-point lead over SkillDAG come from the stage planner, which SkillDAG does not receive (76.4% vs. 72.9%), and without a run that removes stage nodes under the same harness, it is unclear how much of the remaining 5.0 points they account for.

Thank you for the careful distinction between the contribution of query construction and that of the retrieval structure. We agree that the fixed-query ALFWorld results show a modest Ret@1 contribution from stage nodes, and that our earlier statement that the gains “stem primarily from the proposed stage node method” was too strong. Our evidence supports two separate conclusions: the SSG retrieval backend improves end-to-end performance under SkillDAG’s query-generation protocol, and stage nodes provide additional retrieval benefits in controlled structural comparisons across multiple datasets.

The 72.9% versus 67.9% comparison uses SkillDAG’s query-generation protocol without our stage planner. Both configurations use the same SkillDAG execution harness, agent, query-generation prompt and protocol, task set, skill-pool size, execution budget, and embedding model. The intervention is to replace SkillDAG’s retrieval backend with SSG. The queries are produced by the SkillDAG agent using its existing rewriting mechanism; they are not specially constructed by our stage planner. SSG succeeds on 102/140 tasks, compared with 95/140 for SkillDAG, an observed improvement of seven tasks, or 5.0 percentage points. This comparison supports a benefit from the retrieval backend under a shared query-generation mechanism, independently of introducing our stage planner.

The contribution of stage nodes is tested separately with the query inputs fixed. In “Cross-Comparison of Query Strategies and Graph Structures” (Appendix D.1), adding stage nodes and their associated retrieval operations improves Ret@1 in five of the six query–benchmark settings and ties in the sixth. We acknowledge that the ALFWorld Ret@1 improvements are small. The benefits also appear in other ranking metrics and on SkillsBench: under stage queries, Full versus w/o stage nodes gives 69.2% versus 65.8% Ret@1, 86.8% versus 82.7% Ret@5, and 76.8% versus 72.8% MRR@5. These comparisons do not depend on giving the full model a different query set within a block.

Adding our stage layer to the existing GoS graph also improves retrieval on AgentSkillOS. The complementary “Scene Increment Controlled Experiment” (Appendix D.3) preserves GoS’s original skill nodes, their representations, and skill-to-skill edges, and adds our stage nodes and their associated edges. It uses the same task queries across the compared retrieval configurations:

ConfigurationHit@1
Original GoS skill subgraph73.33%
Same skill subgraph + stage layer, unified PPR80.00%
Same skill subgraph + stage layer, stage-subgraph PPR83.33%

Thus, enabling the added stage layer yields a 6.67-point improvement with unified PPR and a 10.00-point improvement with stage-subgraph PPR, without reconstructing the underlying skill subgraph or introducing stage-planner queries. This is a supplementary experiment. Since its embedding configuration and evaluation setup differ from those of the main experiment, its absolute scores vary from the previous ones.

Together, these results support the usefulness of stage nodes and their relations across multiple datasets, query strategies, and graph configurations. They also show that SSG’s observed end-to-end advantage is not solely due to introducing our stage planner. Nevertheless, we agree that current evidence cannot pinpoint the exact fraction of the 5.0 percentage point ALFWorld backend gain attributable to stage nodes. Addressing this requires an end-to-end stage-node ablation study under the same harness, which we will provide in our future revisions.

Reviewer_kNtyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates improving skill retrieval process for LLM agents. Compared with previous methods that treat full skill as a single unit, this paper proposes a stage-aware skill graph sturcutre, which splits each skill into differet execution stages and a stage-level retrieval that adaptively performs multiple retrieval on each stage during task exeution. The full execution process thus involves sub-task decomposition and sub-task retrieval query writing. The experiments on three benchmarks show that the proposed method outperform several baselines in both retrieval and end-to-end task performance.

Critical strengths
  • Practical problem: this paper studies a useful problem of LLM agent skill retrieval.

  • Intuitive method and clear design: using stages as extra retrieval decision point is simple and intuitive. The method is also clearly designed.

Critical weaknesses
  • Limited analysis on the stage retrieval result: I feel there are many intresting analysis that can be made to help better understand why the stage-aware retrieval improves performance, which are missing from the current paper. For example, the number of different skills retrieved for solving a task. This analysis would help understand how different skills composite to make the performance better and how the stage-aware retriegval helps finding missing information. Another analysis I think would be beneficial is how many stages can be decomposed for each task across different benchmarks and how the structure of each stage decomposition look like. I feel some of the skills may not be a simple sequntial stage-by-stage pipeline, which may bring more retrival challenges.

  • Missing cost analysis: The proposed method involves LLM extraction of stages for each skill and execution time retrieval at different steps, which brings more LLM calls during agent rollout. Involving a cost analysis would be beneficial for understanding the inference overhead.

  • Unclear experiment details: I feel confused about several details. It would be beneficial if authors can clarify them:

    • How ground-truth skill is constructed for retrieval experiment: For ALFWorld and AgentSkillOS, it's not clear how these ground-truth skills are constructed. Line 336 describes SkillsBench as having without skill/with curated skill/with self-generated skill. But it's not clear how the other two benchmarks provide ground-truth skills.
    • How the scoring for retrieval is designed: The retrieval process involves multiple different scoring, such as the p(skill), p(stage) and multiple different hyper-parameter for the score aggregation. However, it's not clear how they are defined.
  • Minor:

    • For citations that does not serve as a noun in a sentence. Please keep consistent and use \citep instead of \cite. For example, line 209, it's inapproprate to use \cite in independently of other stages Liang et al. (2026a).
    • How does the online skill injection work during task rollout? Is it injecting the skill content with only the next stage skill retrieved from the pool or the full skill?
Questions for authors

Please see weakness.

2 comments

1. Stage retrieval and skill composition. Appendix E.3 analyzes 140 ALFWorld trajectories: 20 tasks have two stages, 90 have three, and 30 have four, averaging 3.07 stages. The agent loads 1–4 distinct whole skills per task (mean 2.17; median 2). These are skills actually loaded, rather than all returned candidates. Appendix E.2 traces “put two newspaper in drawer” through LOCATE → ACQUIRE → ACQUIRE → PLACE, showing how repeated phases retrieve supporting skills. Our current analysis covers ordered plans with repeated phases

2. Construction and execution costs. Appendix F now reports preparation usage for the 1,000-skill pool and GPT-5.2 execution usage:

MethodPreparation tokensAgent tokensALFWorld success
Vector Skills906,0918,330,72160.0%
GoS681,2889,309,50663.6%
SkillDAG2,863,448*13,771,73382.0%
SSG6,818,03810,440,06086.1%

SSG requires more preparation than some baselines, but uses fewer agent tokens than SkillDAG while achieving higher success. Vector Skills remain cheaper in Preparation tokens columns, but use more in Agent tokens.

3. Ground truth and scoring. Appendix A.6 clarifies that we retain the baseline reference skills: SkillDAG’s protocol for ALFWorld and SkillsBench, and official annotations for AgentSkillOS. A hit requires membership in that reference set, so an unannotated but useful alternative can still be scored as a miss. The skill and stage scores are internal PPR signals. Their fusion is g(k|q) = p_skill(k) + 0.5 × max p_stage(s) over stages belonging to skill k, followed by field-aware reranking. Appendix C.3 specifies normalization, coefficients, and stopping criteria.

4. Injection and presentation. Stage matches contribute evidence for their parent whole skills. ALFWorld receives an ordered summary of subtasks, candidate skills, and descriptions; stages are not treated as standalone executable skills. AgentSkillOS makes the top-five whole-skill files available in each task workspace. We have also standardized parenthetical citations in the active manuscript.

Thanks for the additional results. I would like to keep my original positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.