- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SkillRet introduces a large-scale benchmark to evaluate skill retrieval in LLM agents. The authors build the benchmark by collecting public skills, generating corresponding queries conditioned on randomly sampled skills, and filtering invalid query-skill pairs. The benchmark contains both training and testing splits, enabling the training of skill retrievers. Experiments on the constructed evaluation set show that fine-tuned retrievers outperform existing baselines. The main contribution is the benchmark and training resources, supported by retrieval analyses.
Critical strengths
- The paper addresses a practical problem and provides a large resource and a scalable pipeline for training and evaluating skill retrievers.
- The benchmark construction includes functionally equivalent skill auditing and cross-split leakage checks, which make the evaluation more accurate.
- The fine-tuned models demonstrate improvements over the same-sized baselines. The authors provide useful analyses of multi-skill completeness and category-level performance.
Critical weaknesses
- The construction of multi-skill queries raises concerns about realism. Skills are sampled randomly for coverage rather than compatibility, and the pipeline requires generating a query that covers all sampled skills. This may lead to unrealistic queries that combine unrelated skills. Although some queries can be rejected, the paper does not report rejection rates or provide evidence on whether retained queries reflect real workflows. Consequently, the multi-skill gains may simply reflect adaptation to the synthetic composition process.
- Weak connection to downstream utility. Terminal-Bench success rate changes only from 65.5% to 65.8%, and comparisons against other retrievers are missing. More fundamentally, query-skill relevance does not establish usefulness to a particular agent: the model may already possess the retrieved knowledge, while its actual capability gaps may emerge only during execution. The paper therefore does not demonstrate that higher retrieval scores translate into meaningful downstream benefits.
- Missing adaptive retrieval baselines. The evaluation lacks agentic search and LLM-based query rewriting, which could formulate search queries based on the agent's capability gaps or execution feedback. Comparing these approaches with direct query-based retrieval, including their costs, would clarify whether fine-tuning alone adequately addresses the problem.
- Relevant prior work is missing. [1] already compares direct and agentic skill search and evaluates downstream utility in realistic settings. [2] also introduces SRA-Bench with separate evaluation for skill retrieval, incorporation, and execution.
[1] Liu et al. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings.
[2] Su et al. Skill Retrieval Augmentation for Agentic AI.
Questions for authors
Please see the weakness.
7 comments
Response to Reviewer y8qq (2/4)
[W3] Adaptive retrieval baselines
The evaluation lacks agentic search and LLM-based query rewriting, which could formulate search queries based on the agent's capability gaps or execution feedback.
We ran two experiments. The first compares adaptive retrieval with direct retrieval on our test set. The second forms the search query from the agent's capability gaps on SRA-Bench. Both use Qwen3.8-27B as the search LLM, which is larger than the 9B agent in W2, so that the adaptive baselines have a strong search model.
Adaptive retrieval from the request. We compared three ways of retrieving skills for the same request, on all 4,392 test queries.
-
Direct. The user request is given to the retriever as is, and the top 10 skills are returned. No LLM is used. This is the setting of the paper.
-
Query rewriting. The LLM rewrites the request into one search query that describes the needed capabilities (about 20 words on average). The retriever is called once with this query.
-
Agentic search. The LLM uses the retriever as a search tool. It writes a search query, reads the names and descriptions of the top 10 results, and may search again with a new query, up to four times. It then ranks the skills it has seen.
In the adaptive strategies the LLM only writes queries and selects results, and the search itself is still done by a retriever. We therefore ran all three strategies with each of five retrievers.
| Retriever | Strategy | nDCG@10 | Recall@10 | Completeness@10 | LLM calls / query | LLM tokens / query |
|---|---|---|---|---|---|---|
| BM25 | Direct | 51.7 | 59.4 | 44.6 | 0 | 0 |
| Rewriting | 53.9 | 61.9 | 46.9 | 1 | 323 | |
| Agentic | 75.8 | 79.0 | 68.3 | 4.3 | 5,907 | |
| Qwen3-Embedding-0.6B | Direct | 61.9 | 67.9 | 51.1 | 0 | 0 |
| Rewriting | 52.1 | 59.6 | 42.9 | 1 | 323 | |
| Agentic | 76.3 | 79.7 | 69.1 | 4.3 | 5,868 | |
| SkillRet-Embedding-0.6B | Direct | 81.1 | 87.7 | 78.9 | 0 | 0 |
| Rewriting | 66.8 | 75.3 | 61.9 | 1 | 323 | |
| Agentic | 80.0 | 84.3 | 75.5 | 4.3 | 5,549 | |
| Qwen3-Embedding-8B | Direct | 63.6 | 70.3 | 54.1 | 0 | 0 |
| Rewriting | 55.0 | 62.9 | 45.5 | 1 | 323 | |
| Agentic | 77.5 | 81.5 | 71.4 | 4.3 | 5,706 | |
| SkillRet-Embedding-8B | Direct | 86.4 | 93.2 | 88.1 | 0 | 0 |
| Rewriting | 70.4 | 78.7 | 66.7 | 1 | 323 | |
| Agentic | 81.1 | 85.4 | 76.9 | 4.3 | 5,413 |
Fine-tuning outperforms adaptive retrieval at no inference-time LLM cost. SkillRet-8B with direct retrieval exceeds the best adaptive configuration of any off-the-shelf retriever, Qwen3-8B with agentic search, by +9.0 [8.1, 9.8] nDCG@10 and +16.8 [15.5, 18.0] Completeness@10 (paired bootstrap 95% CIs). That configuration needs 4.3 calls to a 27B LLM per query. SkillRet-0.6B with direct retrieval also exceeds Qwen3-0.6B with agentic search (+4.8 [4.0, 5.7] nDCG@10).
Agentic search strongly helps weak retrievers (+14 to +24 nDCG@10) but does not help SkillRet, whose direct ranking of the full request is already better than what the LLM's reformulations recover. Query rewriting lowers every embedding retriever by 9 to 16 points because it compresses requests of about 170 words into about 20. Within agentic search, SkillRet remains the best search tool (+3.6 [2.9, 4.3] nDCG@10 for 8B over its base model). We have added this experiment to Appendix Q of the revised paper, marked in blue.
Response to Reviewer y8qq (3/4)
[W3] Adaptive retrieval baselines (continued)
Retrieval from the agent's capability gaps. We used the SRA-Bench setting of W2 (1,123 instances, Qwen3.5-9B agent, top-5 skills). The agent first attempts each problem without skills. The search LLM then reads this attempt as feedback and writes one search query for the knowledge or technique the agent lacked. The top-5 skills retrieved with this query are given to the agent, which solves the problem again. We compare this with direct retrieval, where the problem itself is the query, for the two 0.6B retrievers.
| Retriever | Search query | MedCalc | TheoremQA | LogicBench | CHAMP | Macro | LLM calls / problem | LLM tokens / problem |
|---|---|---|---|---|---|---|---|---|
| – | No skill | 66.0 | 70.7 | 74.7 | 74.9 | 71.6 | 1 | 8.5K |
| Qwen3-Embedding-0.6B | Problem (direct) | 84.7 | 73.7 | 36.7 | 79.8 | 68.7 | 1 | 16.5K |
| Capability-gap query | 83.3 | 75.0 | 55.7 | 75.8 | 72.4 | 3 | 24.2K | |
| SkillRet-Embedding-0.6B | Problem (direct) | 87.0 | 78.7 | 57.0 | 82.5 | 76.3 | 1 | 12.2K |
| Capability-gap query | 81.7 | 74.7 | 49.7 | 76.2 | 70.6 | 3 | 24.1K | |
| – | Gold skills (oracle) | 85.3 | 77.3 | 83.7 | 82.1 | 82.1 | 1 | 8.6K |
Accuracy is in %. Tokens are input plus output over all LLM calls for a problem, including the first attempt and the query written by the search LLM.
Capability-gap queries help the off-the-shelf retriever. Qwen3-0.6B improves by +3.7 [1.1, 6.4] in macro accuracy (paired bootstrap 95% CI), mostly on LogicBench, where direct retrieval returns unrelated skills. Fine-tuning helps more, and at lower cost. SkillRet-0.6B with direct retrieval is higher than Qwen3-0.6B with capability-gap queries on all four benchmarks, by +3.8 [1.3, 6.4] in macro accuracy, with one LLM call instead of three and about half the tokens.
Capability-gap queries do not help SkillRet (−5.7 [−8.3, −3.1]), and with them the two retrievers perform similarly (−1.9 [−4.5, 0.7]). These queries average 11 words, and as with query rewriting in the first experiment, SkillRet ranks better from the full problem than from a short reformulation. In both experiments, a fine-tuned retriever used directly outperforms an off-the-shelf retriever with an adaptive strategy, without the extra LLM calls.
Response to Reviewer y8qq (1/4)
We thank the reviewer for the thoughtful review and for recognizing the scale of the resource, the functional-equivalence audit, and the same-size gains. We answer each weakness in order below, with new experiments for W2 and W3.
[W1] Realism of multi-skill queries
We agree that randomly sampled combinations can be unrealistic. The pipeline therefore checks at three stages whether a query naturally requires all of its skills, and we report the rejection counts below.
Generation. The generator is asked to write a message that "must naturally require ALL of the following skills", and to output None "if this skill combination makes no sense together in any realistic scenario". No query is produced for those combinations.
LLM review from three perspectives. Each generated query is judged by three independent reviewer prompts. Skill Coherence asks, "If multiple skills are provided, does the query naturally require all of them?", and marks the query invalid if "the combination is forced". Query Quality asks whether the query "could plausibly come from a real user in a professional setting". Benchmark Discriminability asks whether the query can be answered without the skills. A query is removed when two or more perspectives mark it invalid.
Human validation. Annotators then review every remaining query against its skills, and reject it when the skills are unrelated to the request or the connection is forced.
Of the 10,000 generated queries, automatic filtering (leakage filter and LLM review) removes 4,086, the functional-equivalence audit removes 716, and annotators reject 806 of the remaining 5,198 (15.5%), leaving 4,392. Every retained multi-skill query has therefore passed three checks, the last by a human annotator, that it naturally requires all of its skills. We have added these counts to Section 3.2 and Appendix C.3 of the revised paper, marked in blue.
[W2] Connection to downstream utility
We agree that the Terminal-Bench run does not show that better retrieval leads to better outcomes. Because Terminal-Bench has no gold-skill labels, it cannot separate retrieval failures from agent failures. We therefore ran an end-to-end comparison on SRA-Bench [2], whose gold skills provide an oracle. A Qwen3.5-9B agent receives the full text of the top-5 skills retrieved from its 26,262-skill corpus in its prompt on four datasets (1,123 instances, with a random sample of 300 from each of the three larger ones). We use the top 5 because no instance has more than five gold skills, so the top 5 can always contain all of them.
The oracle row answers the concern that the agent may already possess the retrieved knowledge. The gold skills raise macro accuracy by 10.5 points over no skill, so they provide knowledge this agent lacks. SkillRet-8B recovers nearly all of that gain (+9.6 [6.9, 12.4] over no skill, within 0.9 [−1.2, 3.1] of the oracle). SkillRet-0.6B improves over its base model by +7.6 [5.1, 10.1]. SkillRet-8B's gain over its base model is not significant (+1.4 [−0.8, 3.7]).
LogicBench shows that relevance determines usefulness. Its gold skills are abstract inference rules while its questions are everyday stories, so weak retrievers return unrelated skills, which mislead the agent into reasoning until the token limit without answering. SkillRet-8B retrieves the gold skill for 75% of instances, against 13% for Qwen3-0.6B and 26% for BM25, and is the only retriever above the no-skill baseline. Across retrievers, LogicBench accuracy follows exactly the ranking of the gold-in-top-5 rate. We have added this experiment to Appendix P of the revised paper, with the setup, the table above, and a figure of this relation, all marked in blue.