acceptodds
Under review as a conference paper at ICLR 2027

Imagine Before Acting: Prospective Skill Identification for LLM Agents

Abstract

Effective skill identification is crucial for LLM agents to reuse task-solving experience for complex task execution, which however remains challenging as agents must accurately identify useful skills from large skill gallery and invoke them on demand. Existing studies have explored various strategies for skill retrieval and invocation, but largely overlook the natural discrepancy between task descriptions and skill documents. To be specific, our empirical study reveals that task descriptions mainly specify task objectives, whereas skill documents describe the required capabilities and procedural guidance for execution, leading to the so-called Task–Skill Misalignment (TSM). Such misalignment would cause desirable skills to be missed during retrieval or remain unselected during invocation, thus hindering task execution. As a remedy, inspired by human prospective cognition, we propose SkillDreamer, a prospective skill identification framework to mitigate TSM. In brief, SkillDreamer first infers execution-required capabilities and imagines their realizations as pseudo skills, thereby translating task objectives into execution-oriented guidance for skill retrieval. As execution proceeds, SkillDreamer reevaluates the capabilities required by emerging subtasks to determine which skills should be invoked, thereby aligning skill invocation with evolving execution needs. Extensive experiments on three representative agent skill benchmarks not only verify the effectiveness of SkillDreamer in mitigating TSM, but also demonstrate its generalizability across diverse retrievers and LLM agents. The code will be released upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM Agent, Skill Retrieval, Skill Invocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PpbuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies Task–Skill Misalignment (TSM), where task descriptions focus on objectives while skill documents provide procedural guidance. To address this issue, the authors propose SkillDreamer, which generates pseudo skills to improve retrieval and explicit reasons about capability requirements to guide skill invocation. Experiments on multiple benchmarks demonstrate improvements in skill retrieval and downstream task execution.

Critical strengths
  1. The paper identifies a practical problem with agent skill identification and provides a unified framework for both skill retrieval and invocation.
  2. Experiments across multiple benchmarks and model backbones demonstrate consistent retrieval improvements and gains in downstream task performance.
Critical weaknesses
  1. Limited novelty. The main contribution of SkillDreamer lies in integrating existing retrieval enhancement and agent prompting strategies into a unified skill identification pipeline, rather than introducing a fundamentally new skill retrieval or invocation mechanism.
  2. Insufficient evidence for the necessity of Prospective Skill Generation. The ablation results in Table 3 show that Pseudo-Skill Retrieval performs worse than Direct Retrieval. These results do not demonstrate that generating skill-like procedural guidance is necessary. The reported improvements may primarily come from other components rather than the proposed core mechanism.
  3. Insufficient evidence supporting Task–Skill Misalignment (TSM). The paper demonstrates differences between task descriptions and skill documents, as well as a correlation between explicit capability requirements and higher skill invocation rates. However, these observations are insufficient to establish that TSM directly causes performance degradation.
  4. Limited novelty of Capability-aware Skill Invocation (CSI). CSI primarily relies on additional prompting instructions and periodic capability checks. Its effectiveness may largely result from stronger prompting rather than a new capability-aware invocation mechanism.
Questions for authors
  1. Table 3(a) shows that Pseudo-Skill Retrieval alone performs worse than Direct Retrieval. Could the authors provide additional experiments to demonstrate that generating procedural guidance is necessary for the reported improvements?
  2. The proposed Prospective Skill Generation shares similarities with existing hypothetical document generation methods, such as HyDE and Query2doc. Could the authors include comparisons with these methods to better establish the novelty and effectiveness of their approach?
  3. CSI introduces capability reasoning, explicit skill-use instructions, and periodic runtime checkpoints. Could the authors conduct separate ablation studies on these three components to clarify their individual contributions?
  4. SkillDreamer introduces additional computational overhead from LLM-based capability inference, pseudo-skill generation, and capability checks during execution. Could the authors report the corresponding token consumption, inference latency, and computational costs?
8 comments

We sincerely thank the reviewer for recognizing the practical relevance of our work and for the constructive suggestions. In response, we have added comparisons with HyDE and Query2doc, controlled experiments on capability inference and pseudo-skill generation, and CSI ablations to clarify the contributions of our design. We also report token consumption and request latency to assess the additional computational overhead. The additional retrieval experiments are conducted on SkillUsage, while execution analyses use DeepSeek-V4.1-Flash-1017 on SkillNet-Gym. Below, we will address your questions and concerns point by point.

W1: Limited novelty. The main contribution of SkillDreamer lies in integrating existing retrieval enhancement and agent prompting strategies into a unified skill identification pipeline, rather than introducing a fundamentally new skill retrieval or invocation mechanism.

Thank you for the comment. We will clarify our contributions at both the problem and methodological levels. At the problem level, we identify and formulate Task-Skill Misalignment (TSM) as a fundamental bottleneck in agent skill identification. At the methodological level, our core contribution lies in formulating a principled, capability-driven reasoning structure across both retrieval and execution. Realizing this structure via prompting rather than parameter updates is an implementation choice made specifically to validate our framework directly on frontier LLMs. Below, we elaborate on these aspects with supporting controlled experiments.

1. Methodological Novelty in Skill Retrieval: Capability-Driven Proxy Generation

While SkillDreamer shares the high-level intuition of using LLM-generated texts to assist retrieval, it differs fundamentally from HyDE-style query expansion. Specifically, HyDE [A] and Query2doc [B] expand queries within a homogeneous semantic space by generating hypothetical answer passages to increase topical overlap. In contrast, agent skill retrieval faces an asymmetric gap between declarative task objectives ("what to achieve") and procedural skill documents ("how to execute"). Rather than rewriting or elaborating on the query, SkillDreamer introduces an explicit intermediate reasoning step (task → capability → pseudo-skill) that decouples latent execution-required capabilities from task-specific semantic anchors, translating each capability into an execution-oriented retrieval proxy.

To empirically verify this distinction, we conduct two sets of controlled comparisons on SkillUsage using Qwen3.6-27B as the generator:

  • Explicit capability inference vs. objective restatement: Prepending inferred capability requirements to the raw query consistently outperforms a length-matched restatement of the task objective across both Qwen3-Embedding-0.6B and E5-large-v2 (Recall@5 / Recall@10, %):
Query variantQwen3-Embedding-0.6BE5-large-v2
Original query52.68 / 62.2546.42 / 48.29
Objective restatement + original query52.81 / 62.6646.04 / 50.44
Capability requirements + original query55.07 / 64.3048.22 / 52.01
  • Capability-driven pseudo skills vs. HyDE-style proxies and step decomposition: Using Qwen3-Embedding-0.6B, SkillDreamer outperforms HyDE and Query2doc both in direct comparisons and when their generated documents replace PSG inside our HSR pipeline (w/ HSR), as well as outperforming sequential execution-step decomposition at lower token cost:
MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
HyDE — three documents59.2213.14
HyDE — three documents (w/ HSR)63.0113.14
Query2doc61.473.95
Query2doc (w/ HSR)60.873.95
Execution-step decomposition68.4715.82
SkillDreamer69.4712.50

2. Methodological Novelty in Agentic Execution: Capability-Aware Skill Invocation (CSI)

We respectfully argue that methodological novelty in agentic systems is not restricted to updating model weights or retraining policies. Much like foundational paradigms such as ReAct [C] and in-context learning [D], significant novelty lies in formulating a principled reasoning structure that governs how an agent decomposes decisions. In our method, unlike standard execution prompting that maps interaction history directly to actions or skills, CSI introduces an explicit subgoal → capability gap → skill matching bottleneck prior to action selection to resolve invocation-side TSM. We intentionally instantiate this mechanism via prompting so that it can be validated directly on frontier LLMs in a training-free manner, without modifying model weights or restricting applicability to fine-tuned backbones.

To verify that CSI’s gains stem from this capability-aware invocation mechanism rather than generic execution prompting, we compare full CSI against a controlled baseline retaining only generic execution guidance (e.g., output verification, failure avoidance, and periodic checkpoints) on SkillNet-Gym using DeepSeek-V4.1-Flash-1017 across three runs:

SettingPass (%)Total LLM tokens (k/task)
No-skill67.80 ± 1.69351.72
No-skill + skill-execution-agnostic guidance (generic guidance + checkpoints)68.36 ± 5.18399.55
GT-skill66.10 ± 3.39400.58
GT-skill + skill-execution-agnostic guidance (generic guidance + checkpoints)67.80 ± 3.39445.31
GT-skill + CSI (skill-execution-oriented mechanisms and skill-execution-agnostic guidance)71.19 ± 4.48420.40

While generic guidance yields only marginal gains (+0.56 pp under No-skill and +1.70 pp under GT-skill), full CSI further improves mean Pass by 3.39 percentage points (67.80% → 71.19%) with lower token usage, confirming the distinct contribution of capability-aware skill matching and invocation.

3. Problem-Level Contribution: Formulating Task-Skill Misalignment (TSM)

Beyond specific methodological designs, our problem-level contribution should not be overlooked. To be specific, prior studies largely focus on scaling query-skill similarity or training invocation policies, without diagnosing why strong retrievers and capable LLMs still miss or fail to invoke useful skills. In contrast, by formalizing TSM across both retrieval and execution stages, our work sheds light on two key directions for subsequent research: i) addressing the fundamental misalignment between tasks and skills, rather than simply adapting standard document retrieval paradigms to the skill domain; and ii) enabling explicit capability awareness in agentic execution, revealing that suboptimal task performance in frontier agents may stem from a lack of awareness of required capabilities rather than limited execution ability.

AuthorsOpenReview ↗

part three

W2: Insufficient evidence for the necessity of Prospective Skill Generation. The ablation results in Table 3 show that Pseudo-Skill Retrieval performs worse than Direct Retrieval. These results do not demonstrate that generating skill-like procedural guidance is necessary. The reported improvements may primarily come from other components rather than the proposed core mechanism.

Q1: Table 3(a) shows that Pseudo-Skill Retrieval alone performs worse than Direct Retrieval. Could the authors provide additional experiments to demonstrate that generating procedural guidance is necessary for the reported improvements?

Keypoints: The effectiveness of pseudo skill generation.

Effectiveness of pseudo-skill generation. We acknowledge that replacing the original query with pseudo skills reduces Recall@10 from 62.25% to 59.00% in Table 3(a), potentially because generated pseudo skills may omit some task-specific objectives or constraints. This suggests that pseudo skills alone cannot fully replace the original query as discussed in the manuscript. Therefore, SkillDreamer combines the original query’s task-specific information with pseudo skills’ procedural cues, thus improving Recall@10 from 62.25% to 69.47% in Table 3(a). To further assess the contribution of pseudo skill generation, we added two controlled comparisons:

  • Raw query + BM25: Rerank raw-query dense top-20 candidates using 2:1 dense–BM25 score fusion.
  • Capability-description substitution: Replace pseudo skills with their corresponding capability descriptions.

The results below support the effectiveness of pseudo-skill generation. On the one hand, reranking raw-query top-20 candidates with BM25 improves Recall@10 from 62.25% to 63.52%. The full SkillDreamer pipeline, which combines raw-query and pseudo-skill information, achieves 69.47%, outperforming this control by 5.95 percentage points. On the other hand, replacing pseudo skills with capability descriptions reduces Recall@10 from 69.47% to 67.99%. With all other HSR settings unchanged, this 1.48-point decrease supports the role of pseudo-skill generation in improving retrieval performance.

MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
Pseudo-skill-only retrieval59.0012.50
Raw query + BM25, top-20 reranking63.52—
Replacing pseudo skills with capability descriptions67.994.24
SkillDreamer69.4712.50
Reviewer_K7bDOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies a problem called Task–Skill Misalignment (TSM), where a discrepancy between task descriptions and skill descriptions degrades skill retrieval and invocation performance. To address this issue, the authors propose a skill identification framework called SkillDreamer. The proposed framework first infers the capabilities required to solve a task and generates corresponding pseudo skills for retrieval. During execution, it further determines which skills should be invoked based on the required capabilities. The authors demonstrate that SkillDreamer improves skill retrieval and task execution performance across different retrievers and LLM agents.

Critical strengths
  1. Well-motivated problem: I think the Task–Skill Misalignment (TSM) problem highlighted in this paper is practically important. I agree that the discrepancy between task objectives and the procedural knowledge described in skill documents can be an important issue for LLM agents.

  2. Plug-and-play applicability: The proposed method can be easily integrated with existing skill retrievers without additional training or architectural modifications, making it practical and broadly applicable.

Critical weaknesses

I have serious concerns about this paper.

1. Limited Novelty

In short, what SkillDreamer does is the following:

  1. An LLM translates a task into capability descriptions and then expands them into pseudo skill documents.
  2. The framework retrieves relevant skills using both the original query and the generated documents.

I think these steps are largely similar to, or fall under the umbrella of, existing approaches such as query expansion, Query2doc, LLM-based query rewriting, and task decomposition-based retrieval. Although the authors discuss the distinctions in Appendix B, I do not find the methodological differences sufficiently clear or convincing. For example, HyDE [1] also generates a hypothetical document and uses it for retrieval. While SkillDreamer introduces capability inference and hybrid retrieval, it remains unclear whether these modifications constitute substantial methodological novelty over existing approaches. I strongly encourage the authors to include comparisons with these existing methods in the experiments. In addition, please consider moving the related work discussion from the appendix to the main paper.

[1] Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels, 2022.

2. Missing Important Baselines

The baselines are primarily embedding-based retrievers, while the proposed method introduces additional LLM-based reasoning and query expansion. I think the paper is missing several important baselines, making it difficult to assess the relative effectiveness of the proposed method. In particular, comparisons against existing methods such as HyDE and Query2doc should be conducted using the same LLM backbone and comparable inference budgets. Without these comparisons, it is difficult to determine whether the reported improvements come from the proposed capability-aware framework or simply from additional LLM-based query expansion and reasoning.

Furthermore, the authors use different hyperparameter settings ((\alpha) and (\gamma)) for certain experiments, such as SkillRet-Embedding-0.6B and R3-Embedding on the SkillRet benchmark, without a clear explanation. Were these hyperparameters selected through a sweep to identify the best-performing configurations? If so, this could introduce an unfair advantage in favor of the proposed method, particularly if the hyperparameters were tuned on the test set. The authors should clarify how these hyperparameter values were determined and whether the same tuning procedure was applied to the baselines.

3. Minor: Missing Limitations Section

Please consider adding a limitations section to discuss the potential limitations of the proposed framework, such as the additional inference cost introduced by LLM-based capability reasoning and pseudo-skill generation.

Also, since the proposed method incurs additional LLM calls, it would be helpful to include an analysis of its inference cost. For example, the authors could report inference cost as an additional metric when comparing against the baseline methods suggested above, such as LLM-based query expansion methods.

Questions for authors

I would appreciate it if the authors could address each of the weaknesses raised above.

7 comments

We sincerely thank the reviewer for the thoughtful assessment and constructive suggestions. To clarify our methodological contribution, we added comparisons with HyDE and Query2doc and controlled ablations. We further clarify that the parameters were fixed heuristically before test-set evaluation. We also report retrieval-stage token usage to quantify the additional inference overhead. Below, we will address your questions and concerns point by point.

W1: In short, what SkillDreamer does is the following:

  1. An LLM translates a task into capability descriptions and then expands them into pseudo skill documents.
  2. The framework retrieves relevant skills using both the original query and the generated documents.

I think these steps are largely similar to, or fall under the umbrella of, existing approaches such as query expansion, Query2doc, LLM-based query rewriting, and task decomposition-based retrieval. Although the authors discuss the distinctions in Appendix B, I do not find the methodological differences sufficiently clear or convincing. For example, HyDE [1] also generates a hypothetical document and uses it for retrieval. While SkillDreamer introduces capability inference and hybrid retrieval, it remains unclear whether these modifications constitute substantial methodological novelty over existing approaches. I strongly encourage the authors to include comparisons with these existing methods in the experiments. In addition, please consider moving the related work discussion from the appendix to the main paper.

[1] Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels, 2022.

Keypoints: Methodological novelty over query-expansion and task-decomposition approaches, missing experimental comparisons, and moving related work to the main paper.

Thank you for the thoughtful comment. While SkillDreamer shares the high-level idea of leveraging LLM-generated texts to assist retrieval, it differs substantially from existing query-expansion (e.g., HyDE [A], Query2doc [B]) and task-decomposition retrieval methods [C] in both problem formulation and algorithmic design. Below, we clarify our methodological distinctions, provide empirical evidence for capability-driven reasoning, and report fair comparisons against HyDE and Query2doc.

1. Methodological Differences

  • Capability-mediated proxy generation (task → capability → pseudo-skill) vs. direct query expansion (query → doc): HyDE and Query2doc expand queries within a homogeneous semantic space by generating hypothetical answer passages to increase topical overlap. In contrast, agent skill identification suffers from Task-Skill Misalignment (TSM): task descriptions state declarative objectives ("what to achieve"), whereas skill documents specify procedural execution guidance ("how to execute"). Rather than rewriting or elaborating on the task objective, SkillDreamer introduces an explicit intermediate reasoning step (task → capability → pseudo-skill) that decouples latent execution-required capabilities from task-specific semantic anchors, translating each capability into a structured, execution-oriented pseudo skill.
  • Cross-stage unification from retrieval to runtime invocation: Existing query-expansion and task-decomposition retrieval methods focus exclusively on pre-execution retrieval. SkillDreamer uses the exact same explicit capability bottleneck to govern runtime skill invocation (CSI), dynamically reassessing capability gaps for emerging subtasks so retrieved skills are reliably invoked during execution.

2. Superior Retrieval Proxy Inference over HyDE-Style Approaches

In agent skill retrieval, our capability-driven pseudo-skill generation is specifically designed to resolve Task-Skill Misalignment (TSM) by inferring execution-oriented retrieval proxies rather than generic query expansions. To verify that this mechanism produces stronger retrieval proxies than HyDE-style approaches, we conduct two complementary evaluations on SkillUsage using the same generator (Qwen3.6-27B) and retriever (Qwen3-Embedding-0.6B):

  1. Direct comparison: Following their original retrieval protocols, HyDE averages the embeddings of the hypothetical skill documents with the original-query embedding, while Query2doc generates a pseudo document using four demonstrations from the SkillRet training split and encodes query [SEP] document.
  2. PSG-substitution comparison (w/ HSR): Replacing our capability-driven Prospective Skill Generation (PSG) module with HyDE- or Query2doc-generated documents within our Hybrid Skill Retrieval (HSR) pipeline, directly comparing their effectiveness as retrieval proxies against our capability-driven pseudo skills.

We also include a controlled ablation, i.e., replacing capability-based decomposition with execution-step decomposition, to further verify how our capability-oriented proxy design contributes to retrieval quality.

MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
HyDE — three documents59.2213.14
HyDE — three documents (w/ HSR)63.0113.14
Query2doc61.473.95
Query2doc (w/ HSR)60.873.95
Execution-step decomposition68.4715.82
SkillDreamer69.4712.50

As shown above, SkillDreamer consistently outperforms HyDE-style methods under both evaluation settings:

  • Direct comparison: SkillDreamer (69.47% Recall@10) surpasses standard three-document HyDE (59.22%) and Query2doc (61.47%) by 10.25 and 8.00 percentage points, respectively.
  • PSG-substitution comparison: Even when HyDE- or Query2doc-generated documents replace PSG inside our HSR pipeline, SkillDreamer still outperforms three-document HyDE (63.01%) by 6.46 percentage points at comparable token cost (12.50k vs. 13.14k tokens/task) and Query2doc (60.87%) by 8.60 percentage points.
  • Capability-based decomposition ablation: Decomposing tasks by independent capabilities rather than sequential execution steps improves Recall@10 by 1.00 percentage point (69.47% vs. 68.47%) while consuming fewer tokens (12.50k vs. 15.82k tokens/task).

Together, these direct and PSG-substitution comparisons confirm that capability-driven pseudo-skill generation infers substantially better retrieval proxies to bridge TSM, rather than relying merely on retrieval aggregation heuristics or extra LLM compute.

3. Empirical Analysis of Capability-Driven Reasoning

To further verify that explicitly inferring required capabilities provides retrieval signals beyond generic query rewriting or goal elaboration, we conducted a controlled prefix experiment on SkillUsage. Specifically, we prepend either the inferred capability requirements or a length-matched restatement of the task objective to the original query, and evaluate retrieval performance (Recall@5 / Recall@10, %) across two embedding models:

Query variantQwen3-Embedding-0.6BE5-large-v2
Original query52.68 / 62.2546.42 / 48.29
Objective restatement + original query52.81 / 62.6646.04 / 50.44
Capability requirements + original query55.07 / 64.3048.22 / 52.01

As the results show, length-matched objective restatements yield negligible gains (and slightly degrade Recall@5 on E5-large-v2), whereas prepending explicit capability requirements consistently improves both Recall@5 and Recall@10 across encoders (+2.26/+1.64 pp over objective restatement on Qwen3-Embedding-0.6B, and +2.18/+1.57 pp on E5-large-v2). This confirms that capability reasoning captures execution-oriented cues that generic query elaboration misses.

4. Manuscript Revision

As suggested, we will move the related work discussion from Appendix B into the main paper in the revised manuscript and expand the discussion to clearly position SkillDreamer relative to HyDE, Query2doc, and task-decomposition-based skill retrieval approaches.

AuthorsOpenReview ↗

part three

W2.1: The baselines are primarily embedding-based retrievers, while the proposed method introduces additional LLM-based reasoning and query expansion. I think the paper is missing several important baselines, making it difficult to assess the relative effectiveness of the proposed method. In particular, comparisons against existing methods such as HyDE and Query2doc should be conducted using the same LLM backbone and comparable inference budgets. Without these comparisons, it is difficult to determine whether the reported improvements come from the proposed capability-aware framework or simply from additional LLM-based query expansion and reasoning.

Keypoints: Comparisons against LLM-based query-expansion baselines under the same LLM backbone and comparable inference budgets.

Thank you for this suggestion. We have added HyDE and Query2doc adapted to skill retrieval on SkillUsage, reporting both retrieval performance and LLM token usage. For a strictly fair comparison, all LLM-based methods use the same generator (Qwen3.6-27B), retriever (Qwen3-Embedding-0.6B), gallery representation, and encoding settings. Specifically, we evaluate HyDE and Query2doc under two settings:

  1. Direct comparison: Following their original retrieval protocols, HyDE averages the embeddings of one, three, or five hypothetical skill documents with the original-query embedding, while Query2doc generates a pseudo document using four demonstrations from the SkillRet training split and encodes query [SEP] document.
  2. PSG-substitution comparison (w/ HSR): Replacing our capability-driven Prospective Skill Generation (PSG) module with HyDE- or Query2doc-generated documents within our Hybrid Skill Retrieval (HSR) pipeline, directly comparing their effectiveness as retrieval proxies against our capability-driven pseudo skills.
MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
HyDE — one document61.674.33
HyDE — three documents59.2213.14
HyDE — five documents60.3321.96
HyDE — three documents (w/ HSR)63.0113.14
Query2doc61.473.95
Query2doc (w/ HSR)60.873.95
SkillDreamer69.4712.50

The results yield two key findings:

  • Limited effectiveness of HyDE and Query2doc for skill retrieval: Under their original formulations, all HyDE variants (59.22%–61.67%) and Query2doc (61.47%) perform even worse than raw-query retrieval (62.25%). Moreover, even when used to replace PSG inside our HSR pipeline, three-document HyDE achieves only 63.01% (+0.76 pp over raw query) and Query2doc drops to 60.87%, indicating that generic hypothetical documents struggle to bridge Task-Skill Misalignment.
  • Superior performance–cost trade-off of SkillDreamer: SkillDreamer achieves 69.47% Recall@10 (+7.22 pp over raw-query retrieval and +6.46 pp over the strongest HyDE variant) at a cost of 12.50k tokens per task. Notably, it requires slightly fewer tokens than three-document HyDE (13.14k tokens) and 43.1% fewer tokens than five-document HyDE (21.96k tokens, which achieves only 60.33%), demonstrating that our gains stem from capability-driven pseudo-skill generation rather than extra LLM compute.
Reviewer_TY7AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies skill identification for LLM agents, where an agent must retrieve useful skills from a large gallery and invoke them during execution. The authors identify Task–Skill Misalignment (TSM): task descriptions state objectives, whereas skill documents describe capabilities and procedures, so needed skills may be ranked low during retrieval or left uninvoked during execution. They propose SkillDreamer, a training-free framework with three modules. Prospective Skill Generation uses an LLM to infer the capabilities a task requires and to write a pseudo skill (a hypothetical SKILL.md) for each. Hybrid Skill Retrieval combines task and pseudo-skill embedding similarities, reranks candidates with BM25 over task-specific anchors, and merges per-capability rankings. Capability-aware Skill Invocation prompts the agent to identify the needed capability before each action and decide whether to load a candidate skill. Experiments on SkillRet, SkillUsage, and SkillNet-Gym show consistent retrieval improvements across seven retrievers and end-to-end gains with GPT-5.5 and DeepSeek-V4-Flash.

Critical strengths
  1. A clear problem formulation that covers invocation, not just retrieval. The paper identifies two failure points in skill identification: a needed skill may be ranked too low during retrieval, or it may already be in the candidate set and still never be invoked during execution. The second point has direct support: even when DeepSeek-V4-Flash is given the ground-truth skills, it loads a skill in only 72.6% of trajectories on SkillUsage and 40.7% on SkillNet-Gym. Prior work largely treats retrieval and invocation separately, and connecting the two stages through an explicit “capability” variable is a coherent framing.

  2. Training-free and plug-and-play, with broad retrieval gains that are practically useful when no training data is available. R@5 improves in all 21 settings (7 retriever backbones × 3 benchmarks), with an additional +8.18 on SRA-Bench. Most notably, a general-purpose Qwen3-Embedding-0.6B combined with SkillDreamer outperforms retrievers trained specifically for skills on SkillUsage and SkillNet-Gym. Its R@5 is 60.42 vs. 57.88 for SkillRet-Embedding and 55.50 for R3 on SkillUsage, and 60.92 vs. 55.68 and 54.70 on SkillNet-Gym.

Critical weaknesses
  1. The source of the retrieval gains is unclear, and closely related baselines are missing.
  • When sim is cosine or inner product, the first-stage HSR score α·cos(z_q, z_i) + cos(z̃_j, z_i) is equivalent to retrieving with a single vector, α·ẑ_q + ẑ_j, where ẑ is the L2-normalized embedding. This is a HyDE-style mixture of the query embedding and a hypothetical-document embedding, applied once per capability. Yet the paper does not compare against HyDE (Gao et al., 2023), Query2doc (Wang et al., 2023), decomposition-based skill retrieval (Chen et al., 2026c; Gao, 2026), or LLM rerankers. Table 1 only compares each retriever with and without SkillDreamer.
  • Retrieving with pseudo skills alone performs worse than direct retrieval: R@5 is 48.85 vs. 52.68 on SkillUsage and 55.56 vs. 59.14 on SkillRet. SkillDreamer only wins once original-query weighting, BM25, and round-robin merging are added. Retrieval is also insensitive to shuffling the sentences within pseudo skills (Fig. 3a). Together, these results suggest the gains come from multi-query fusion and vocabulary rewriting, not from the claimed alignment with procedural guidance.
  1. The end-to-end results do not show that better skill identification is what improves task success.
  • Changing only retrieval, without CSI, raises R@5 on SkillUsage by 7.7 points but Pass by only 1.6 (GPT-5.5) and 1.4 (DeepSeek). That is roughly one additional task solved out of 84.
  • On SkillNet-Gym, ground-truth skills provide no benefit. GPT-5.5 scores 66.1 with them vs. 69.5 without any skills, and DeepSeek scores 61.0 in both settings. SkillDreamer nevertheless exceeds the ground-truth setting, at 72.9 and 66.1. These gains more plausibly come from skill-independent instructions in the CSI prompt (verify outputs before finishing, do not repeat failed approaches, track the time budget), or from noise: the benchmark has only 59 tasks, and no variance is reported.
Questions for authors
  1. What is encoded on the skill side? Is each gallery skill embedded as the full SKILL.md or only its name and description, and how is the 512-token limit of BGE and E5 handled? If full files are truncated, most of the procedural content is never encoded. Yet on SkillRet, BGE and E5 gain more than Qwen3: R@5 improves by +10.73 and +12.34, compared with +7.79 and +7.15 for Qwen3. Could the authors use Qwen3 to compare SkillDreamer’s gains when the skill side encodes only the name and description versus the full file?

  2. How does SkillDreamer compare with related methods at matched LLM cost? SkillDreamer calls a 27B model up to six times per query. How does it compare with HyDE and Query2doc under the same α fusion, and with listwise reranking of the base retriever’s top-k by Qwen3.6-27B? Please also include two ablations: one that replaces pseudo skills with the capability descriptions u_j, and one that decomposes by execution step instead of by capability.

  3. Where do the end-to-end gains come from? On SkillNet-Gym, SkillDreamer outperforms directly providing the ground-truth skills. Could the authors report a No Skills baseline that keeps only the skill-independent instructions of the CSI prompt, as well as Ground-Truth Skills + CSI? Please report means and variances over repeated runs for both.

If these questions are addressed convincingly, I would be willing to raise my score.

8 comments

We sincerely thank the reviewer for the detailed assessment and constructive suggestions. To assess our retrieval contribution, we added comparisons with HyDE, Query2doc, and LLM reranking, reporting both retrieval performance and LLM token usage. We also conducted more ablations and analyses regarding the truncation, document content, and sentence order to clarify the effects of pseudo-skill generation and skill representations. For end-to-end execution, we added repeated evaluations, failure analyses, and CSI ablations to examine performance differences and distinguish CSI’s contribution from generic execution guidance. Below, we will address your questions and concerns point by point.

W1.1: When sim is cosine or inner product, the first-stage HSR score α·cos(z_q, z_i) + cos(z̃_j, z_i) is equivalent to retrieving with a single vector, α·ẑ_q + ẑ_j, where ẑ is the L2-normalized embedding. This is a HyDE-style mixture of the query embedding and a hypothetical-document embedding, applied once per capability. Yet the paper does not compare against HyDE (Gao et al., 2023), Query2doc (Wang et al., 2023), decomposition-based skill retrieval (Chen et al., 2026c; Gao, 2026), or LLM rerankers. Table 1 only compares each retriever with and without SkillDreamer.

Q2: How does SkillDreamer compare with related methods at matched LLM cost? SkillDreamer calls a 27B model up to six times per query. How does it compare with HyDE and Query2doc under the same α fusion, and with listwise reranking of the base retriever’s top-k by Qwen3.6-27B? Please also include two ablations: one that replaces pseudo skills with the capability descriptions u_j, and one that decomposes by execution step instead of by capability.

Thank you for these constructive suggestions. We address these concerns by adding comparisons with the suggested baselines, clarifying our contribution relative to HyDE-style approaches, and conducting more ablations to assess our design choices. The details are as follows.

W1.1&Q2 (1): Comparisons with suggested baselines.

Thanks for the suggestion. We add three kinds of baselines: HyDE [A], Query2doc [B], and LLM reranking [C]. For HyDE, we evaluate variants generating one, three, and five hypothetical skill documents to examine performance across different generation budgets. For Query2doc, we follow its document-generation and query-expansion procedure, using four demonstrations from the SkillRet training split and encoding the concatenated query and generated document. For LLM reranking, we use Qwen3.6-27B, the same LLM used in SkillDreamer, to rerank the raw-query retriever’s top-20 candidates in a listwise manner.

For HyDE and Query2doc, we report both their adapted retrieval configurations and variants using the suggested 2:1 query–document fusion weighting as SkillDreamer, with round-robin merging when multiple documents are generated. All methods use the same Qwen3-Embedding-0.6B encoder and gallery representation on SkillUsage.

As shown in the following table, SkillDreamer achieves the highest Recall@10 among the evaluated methods at comparable or lower token costs. Specifically, it outperforms three-document HyDE with matched fusion by 6.46 percentage points at comparable token usage (12.50k vs. 13.14k tokens per task), and exceeds LLM reranking by 3.47 points with substantially fewer tokens (12.50k vs. 53.64k).

MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
HyDE — one document61.674.33
HyDE — three documents59.2213.14
HyDE — five documents60.3321.96
HyDE — one document, 2:1 fusion62.734.33
HyDE — three documents, 2:1 fusion63.0113.14
HyDE — five documents, 2:1 fusion61.9621.96
Query2doc61.473.95
Query2doc — 2:1 fusion60.873.95
LLM reranking — top 2066.0053.64
SkillDreamer69.4712.50

W1.1&Q2 (2): Additional ablations.

Thanks for the suggestion. We conduct the two suggested ablations: i) replacing pseudo skills with capability descriptions, and ii) replacing capability-based decomposition with execution-step decomposition [D]. Notably, all other SkillDreamer settings remain unchanged in each ablation. As shown in the following table, both design choices contribute to retrieval performance. More specifically, replacing pseudo skills with capability descriptions reduces token usage from 12.50k to 4.24k per task, but lowers Recall@10 from 69.47% to 67.99%, indicating a trade-off between retrieval accuracy and generation cost. Replacing capability-based decomposition with execution-step decomposition lowers Recall@10 to 68.47% while increasing token usage to 15.82k per task. These results support the complementary value of pseudo-skill generation and the greater effectiveness and token efficiency of capability-based decomposition.

MethodRecall@10 (%)Total LLM tokens (k/task)
Replacing pseudo skills with capability descriptions67.994.24
Execution-step decomposition68.4715.82
SkillDreamer69.4712.50

W1.1&Q2 (3): Novelty clarification.

While SkillDreamer uses an LLM to synthesize intermediate retrieval proxies, it differs fundamentally from HyDE-style query expansion in the following two respects:

i) Capability-guided proxy generation. HyDE-style approaches generate hypothetical documents directly from the original query. SkillDreamer follows a task → capability → pseudo-skill process, first inferring execution-required capabilities and then generating a pseudo skill for each. This intermediate reasoning step helps expose implicit execution needs that direct task-conditioned generation may overlook, while separate capability-specific retrieval branches cover complementary skill requirements. The above supplementary comparisons against HyDE-style approaches, together with our controlled ablations, verify the effectiveness of explicit capability factorization and procedural pseudo-skill generation.

ii) Joint consideration of retrieval and invocation. HyDE-style approaches primarily target document retrieval, whereas SkillDreamer addresses both skill retrieval and invocation during agent execution. Beyond retrieving candidate skills, CSI reassesses the capabilities required by emerging subtasks to determine whether and which skills to invoke. Thus, explicit capability reasoning connects task objectives with skill selection across both stages.

AuthorsOpenReview ↗

part three

W1.2: Retrieving with pseudo skills alone performs worse than direct retrieval: R@5 is 48.85 vs. 52.68 on SkillUsage and 55.56 vs. 59.14 on SkillRet. SkillDreamer only wins once original-query weighting, BM25, and round-robin merging are added. Retrieval is also insensitive to shuffling the sentences within pseudo skills (Fig. 3a). Together, these results suggest the gains come from multi-query fusion and vocabulary rewriting, not from the claimed alignment with procedural guidance.

W1.2 (1): The effectiveness of pseudo skill generation.

Effectiveness of pseudo-skill generation. We acknowledge that replacing the original query with pseudo skills reduces Recall@10 from 62.25% to 59.00% in Table 3(a), potentially because generated pseudo skills may omit some task-specific objectives or constraints. This suggests that pseudo skills alone cannot fully replace the original query as discussed in the manuscript. Therefore, SkillDreamer combines the original query’s task-specific information with pseudo skills’ procedural cues, thus improving Recall@10 from 62.25% to 69.47% in Table 3(a). To further assess the contribution of pseudo skill generation, we added two controlled comparisons:

  • Raw query + BM25: Rerank raw-query dense top-20 candidates using 2:1 dense–BM25 score fusion.
  • Capability-description substitution: Replace pseudo skills with their corresponding capability descriptions.

The results below support the effectiveness of pseudo-skill generation. On the one hand, reranking raw-query top-20 candidates with BM25 improves Recall@10 from 62.25% to 63.52%. The full SkillDreamer pipeline, which combines raw-query and pseudo-skill information, achieves 69.47%, outperforming this control by 5.95 percentage points. On the other hand, replacing pseudo skills with capability descriptions reduces Recall@10 from 69.47% to 67.99%. With all other HSR settings unchanged, this 1.48-point decrease supports the role of pseudo-skill generation in improving retrieval performance.

MethodRecall@10 (%)Total LLM tokens (k/task)
Raw-query retrieval62.25—
Pseudo-skill-only retrieval59.0012.50
Raw query + BM25, top-20 reranking63.52—
Replacing pseudo skills with capability descriptions67.994.24
SkillDreamer69.4712.50

W1.2 (2): Insensitivity to sentence order.

Thanks for the comment. Sentence shuffling changes the order of the text while preserving its operational descriptions and technical details. Thus, the encoder can still use this information to represent pseudo skills, potentially producing similar embeddings before and after shuffling. This provides a plausible explanation for the relatively stable retrieval performance in Fig. 3(a). To verify our claims, we measured the cosine similarity between original and shuffled pseudo-skill embeddings using Qwen3-Embedding-0.6B. As shown in the following table, the mean similarity remains 0.9792 even at 100% perturbation, indicating that the encoder is relatively insensitive to sentence reordering in this setting. This supports the explanation that preserved content enables similar representations despite reordering.

Perturbation ratioMean similarityMinimum
20%0.99160.9067
40%0.98680.9162
60%0.98380.9166
80%0.98130.8806
100%0.97920.8713

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.