acceptodds
Under review as a conference paper at ICLR 2027

SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

Abstract

As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 16,129 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,392 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved. Off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet improves NDCG@10 by 12.9 points over the strongest prior retriever and by 16.2 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems. We publicly release the benchmark, code, and model checkpoints for embeddings at 0.6B and 8B, and for reranking at 0.6B.

Primary area:
datasets and benchmarks
Keywords:
Agentic AI, Skill, Retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y8qqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillRet introduces a large-scale benchmark to evaluate skill retrieval in LLM agents. The authors build the benchmark by collecting public skills, generating corresponding queries conditioned on randomly sampled skills, and filtering invalid query-skill pairs. The benchmark contains both training and testing splits, enabling the training of skill retrievers. Experiments on the constructed evaluation set show that fine-tuned retrievers outperform existing baselines. The main contribution is the benchmark and training resources, supported by retrieval analyses.

Critical strengths
  1. The paper addresses a practical problem and provides a large resource and a scalable pipeline for training and evaluating skill retrievers.
  2. The benchmark construction includes functionally equivalent skill auditing and cross-split leakage checks, which make the evaluation more accurate.
  3. The fine-tuned models demonstrate improvements over the same-sized baselines. The authors provide useful analyses of multi-skill completeness and category-level performance.
Critical weaknesses
  1. The construction of multi-skill queries raises concerns about realism. Skills are sampled randomly for coverage rather than compatibility, and the pipeline requires generating a query that covers all sampled skills. This may lead to unrealistic queries that combine unrelated skills. Although some queries can be rejected, the paper does not report rejection rates or provide evidence on whether retained queries reflect real workflows. Consequently, the multi-skill gains may simply reflect adaptation to the synthetic composition process.
  2. Weak connection to downstream utility. Terminal-Bench success rate changes only from 65.5% to 65.8%, and comparisons against other retrievers are missing. More fundamentally, query-skill relevance does not establish usefulness to a particular agent: the model may already possess the retrieved knowledge, while its actual capability gaps may emerge only during execution. The paper therefore does not demonstrate that higher retrieval scores translate into meaningful downstream benefits.
  3. Missing adaptive retrieval baselines. The evaluation lacks agentic search and LLM-based query rewriting, which could formulate search queries based on the agent's capability gaps or execution feedback. Comparing these approaches with direct query-based retrieval, including their costs, would clarify whether fine-tuning alone adequately addresses the problem.
  4. Relevant prior work is missing. [1] already compares direct and agentic skill search and evaluates downstream utility in realistic settings. [2] also introduces SRA-Bench with separate evaluation for skill retrieval, incorporation, and execution.

[1] Liu et al. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings.

[2] Su et al. Skill Retrieval Augmentation for Agentic AI.

Questions for authors

Please see the weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer y8qq (1/4)

We thank the reviewer for the thoughtful review and for recognizing the scale of the resource, the functional-equivalence audit, and the same-size gains. We answer each weakness in order below, with new experiments for W2 and W3.

[W1] Realism of multi-skill queries

Skills are sampled randomly for coverage rather than compatibility ... the paper does not report rejection rates or provide evidence on whether retained queries reflect real workflows.

We agree that randomly sampled combinations can be unrealistic. The pipeline therefore checks at three stages whether a query naturally requires all of its skills, and we report the rejection counts below.

  • Generation. The generator is asked to write a message that "must naturally require ALL of the following skills", and to output None "if this skill combination makes no sense together in any realistic scenario". No query is produced for those combinations.

  • LLM review from three perspectives. Each generated query is judged by three independent reviewer prompts. Skill Coherence asks, "If multiple skills are provided, does the query naturally require all of them?", and marks the query invalid if "the combination is forced". Query Quality asks whether the query "could plausibly come from a real user in a professional setting". Benchmark Discriminability asks whether the query can be answered without the skills. A query is removed when two or more perspectives mark it invalid.

  • Human validation. Annotators then review every remaining query against its skills, and reject it when the skills are unrelated to the request or the connection is forced.

Of the 10,000 generated queries, automatic filtering (leakage filter and LLM review) removes 4,086, the functional-equivalence audit removes 716, and annotators reject 806 of the remaining 5,198 (15.5%), leaving 4,392. Every retained multi-skill query has therefore passed three checks, the last by a human annotator, that it naturally requires all of its skills. We have added these counts to Section 3.2 and Appendix C.3 of the revised paper, marked in blue.

[W2] Connection to downstream utility

Terminal-Bench success rate changes only from 65.5% to 65.8%, and comparisons against other retrievers are missing. ... the model may already possess the retrieved knowledge, while its actual capability gaps may emerge only during execution.

We agree that the Terminal-Bench run does not show that better retrieval leads to better outcomes. Because Terminal-Bench has no gold-skill labels, it cannot separate retrieval failures from agent failures. We therefore ran an end-to-end comparison on SRA-Bench [2], whose gold skills provide an oracle. A Qwen3.5-9B agent receives the full text of the top-5 skills retrieved from its 26,262-skill corpus in its prompt on four datasets (1,123 instances, with a random sample of 300 from each of the three larger ones). We use the top 5 because no instance has more than five gold skills, so the top 5 can always contain all of them.

Accuracy (%)MedCalcTheoremQALogicBenchCHAMPMacro
No skill66.070.774.774.971.6
BM2579.778.741.772.268.0
Qwen3-Embedding-0.6B84.773.736.779.868.7
SkillRet-Embedding-0.6B87.078.757.082.576.3
Qwen3-Embedding-8B88.380.768.381.679.7
SkillRet-Embedding-8B86.379.777.081.681.2
Oracle85.377.383.782.182.1

The oracle row answers the concern that the agent may already possess the retrieved knowledge. The gold skills raise macro accuracy by 10.5 points over no skill, so they provide knowledge this agent lacks. SkillRet-8B recovers nearly all of that gain (+9.6 [6.9, 12.4] over no skill, within 0.9 [−1.2, 3.1] of the oracle). SkillRet-0.6B improves over its base model by +7.6 [5.1, 10.1]. SkillRet-8B's gain over its base model is not significant (+1.4 [−0.8, 3.7]).

LogicBench shows that relevance determines usefulness. Its gold skills are abstract inference rules while its questions are everyday stories, so weak retrievers return unrelated skills, which mislead the agent into reasoning until the token limit without answering. SkillRet-8B retrieves the gold skill for 75% of instances, against 13% for Qwen3-0.6B and 26% for BM25, and is the only retriever above the no-skill baseline. Across retrievers, LogicBench accuracy follows exactly the ranking of the gold-in-top-5 rate. We have added this experiment to Appendix P of the revised paper, with the setup, the table above, and a figure of this relation, all marked in blue.

AuthorsOpenReview ↗

Response to Reviewer y8qq (2/4)

[W3] Adaptive retrieval baselines

The evaluation lacks agentic search and LLM-based query rewriting, which could formulate search queries based on the agent's capability gaps or execution feedback.

We ran two experiments. The first compares adaptive retrieval with direct retrieval on our test set. The second forms the search query from the agent's capability gaps on SRA-Bench. Both use Qwen3.8-27B as the search LLM, which is larger than the 9B agent in W2, so that the adaptive baselines have a strong search model.

Adaptive retrieval from the request. We compared three ways of retrieving skills for the same request, on all 4,392 test queries.

  • Direct. The user request is given to the retriever as is, and the top 10 skills are returned. No LLM is used. This is the setting of the paper.

  • Query rewriting. The LLM rewrites the request into one search query that describes the needed capabilities (about 20 words on average). The retriever is called once with this query.

  • Agentic search. The LLM uses the retriever as a search tool. It writes a search query, reads the names and descriptions of the top 10 results, and may search again with a new query, up to four times. It then ranks the skills it has seen.

In the adaptive strategies the LLM only writes queries and selects results, and the search itself is still done by a retriever. We therefore ran all three strategies with each of five retrievers.

RetrieverStrategynDCG@10Recall@10Completeness@10LLM calls / queryLLM tokens / query
BM25Direct51.759.444.600
Rewriting53.961.946.91323
Agentic75.879.068.34.35,907
Qwen3-Embedding-0.6BDirect61.967.951.100
Rewriting52.159.642.91323
Agentic76.379.769.14.35,868
SkillRet-Embedding-0.6BDirect81.187.778.900
Rewriting66.875.361.91323
Agentic80.084.375.54.35,549
Qwen3-Embedding-8BDirect63.670.354.100
Rewriting55.062.945.51323
Agentic77.581.571.44.35,706
SkillRet-Embedding-8BDirect86.493.288.100
Rewriting70.478.766.71323
Agentic81.185.476.94.35,413

Fine-tuning outperforms adaptive retrieval at no inference-time LLM cost. SkillRet-8B with direct retrieval exceeds the best adaptive configuration of any off-the-shelf retriever, Qwen3-8B with agentic search, by +9.0 [8.1, 9.8] nDCG@10 and +16.8 [15.5, 18.0] Completeness@10 (paired bootstrap 95% CIs). That configuration needs 4.3 calls to a 27B LLM per query. SkillRet-0.6B with direct retrieval also exceeds Qwen3-0.6B with agentic search (+4.8 [4.0, 5.7] nDCG@10).

Agentic search strongly helps weak retrievers (+14 to +24 nDCG@10) but does not help SkillRet, whose direct ranking of the full request is already better than what the LLM's reformulations recover. Query rewriting lowers every embedding retriever by 9 to 16 points because it compresses requests of about 170 words into about 20. Within agentic search, SkillRet remains the best search tool (+3.6 [2.9, 4.3] nDCG@10 for 8B over its base model). We have added this experiment to Appendix Q of the revised paper, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer y8qq (3/4)

[W3] Adaptive retrieval baselines (continued)

Retrieval from the agent's capability gaps. We used the SRA-Bench setting of W2 (1,123 instances, Qwen3.5-9B agent, top-5 skills). The agent first attempts each problem without skills. The search LLM then reads this attempt as feedback and writes one search query for the knowledge or technique the agent lacked. The top-5 skills retrieved with this query are given to the agent, which solves the problem again. We compare this with direct retrieval, where the problem itself is the query, for the two 0.6B retrievers.

RetrieverSearch queryMedCalcTheoremQALogicBenchCHAMPMacroLLM calls / problemLLM tokens / problem
–No skill66.070.774.774.971.618.5K
Qwen3-Embedding-0.6BProblem (direct)84.773.736.779.868.7116.5K
Capability-gap query83.375.055.775.872.4324.2K
SkillRet-Embedding-0.6BProblem (direct)87.078.757.082.576.3112.2K
Capability-gap query81.774.749.776.270.6324.1K
–Gold skills (oracle)85.377.383.782.182.118.6K

Accuracy is in %. Tokens are input plus output over all LLM calls for a problem, including the first attempt and the query written by the search LLM.

Capability-gap queries help the off-the-shelf retriever. Qwen3-0.6B improves by +3.7 [1.1, 6.4] in macro accuracy (paired bootstrap 95% CI), mostly on LogicBench, where direct retrieval returns unrelated skills. Fine-tuning helps more, and at lower cost. SkillRet-0.6B with direct retrieval is higher than Qwen3-0.6B with capability-gap queries on all four benchmarks, by +3.8 [1.3, 6.4] in macro accuracy, with one LLM call instead of three and about half the tokens.

Capability-gap queries do not help SkillRet (−5.7 [−8.3, −3.1]), and with them the two retrievers perform similarly (−1.9 [−4.5, 0.7]). These queries average 11 words, and as with query rewriting in the first experiment, SkillRet ranks better from the full problem than from a short reformulation. In both experiments, a fine-tuned retriever used directly outperforms an off-the-shelf retriever with an adaptive strategy, without the extra LLM calls.

Reviewer_tYacOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a skill-retrieval benchmark from 22,795 public skills crawled from claude-plugins.dev. Filtering for language, license and exact duplicates leaves 17,810. After query generation and an LLM-assisted functional-equivalence audit, 16,129 skills remain, in disjoint training (10,123) and evaluation (6,006) pools. Queries are synthesized from skill names and descriptions, with 1–3 skills per query and GAIA questions as style seeds. Claude Opus 4.6 generates the 4,392 evaluation queries and Qwen3.5-122B the 63,259 training queries. Queries pass n-gram leakage filtering, three LLM "reviewer persona" checks and human validation. The paper also gives a 6×18 taxonomy and evaluates 19 retrievers plus rerankers. Fine-tuned Qwen3-Embedding models reach 81.1 (0.6B) and 86.4 (8B) NDCG@10, against 70.3 for the best off-the-shelf model and 73.5 for SkillRouter. Further analyses cover multi-skill completeness, MTEB transfer, sentence erasure, latency, and a Terminal-Bench run with Claude Code.

Critical strengths
  1. The disjoint train/eval skill pools, the released training split and the released checkpoints make this a rare public skill-retrieval resource that supports both evaluating generalization to unseen skills and training retrievers. That fills a real gap compared with SkillRouter (75 evaluation queries, private training data).
  2. Evaluation hygiene goes beyond typical synthetic benchmarks. The functional-equivalence audit uses a model family different from the retrievers, removes cross-split capability twins, and shows rankings are unchanged across all 171 model pairs. A graded-relevance robustness check, an oracle-pool reranking analysis and per-k completeness all add to it. The multi-skill finding is informative on its own: completeness@10 of 1.8% vs 42.1% at k=3.
Critical weaknesses
  1. Both splits come from the same generation recipe, so the headline fine-tuning gains may partly reflect learning the pipeline. Queries are LLM-written scenarios conditioned only on skill name+description, and training queries are produced by the same recipe with a different generator. The +12.9/+16.2 NDCG@10 gains are therefore measured in-distribution with respect to the construction procedure. The paper has no evaluation of SkillRet models on an externally constructed skill-retrieval set (e.g., SkillRouter's expert queries) or on human-written, terse, real requests. The Terminal-Bench study does not close this gap: Pass@1 moves 65.5→65.8 (+0.3 pp, no variance reported), uses the pre-audit library, and has no comparison against other retrievers, so it does not show that better retrieval leads to better outcomes.
  2. Label completeness and quality-control statistics are under-reported. Only seed skills are gold among 6,006 candidates. The audit only checks skill–skill pairs among top-5 neighbors with cosine ≥0.80, yet it alone raised NDCG@10 by 3.3 points on average. So residual false negatives may be material at the 86+ level. Please estimate them by human-judging a pooled sample of top-ranked non-gold results. Also missing: counts at each QC stage (generated, leakage-rejected, LLM-rejected, routed to humans, human-rejected) and inter-annotator agreement on query validity, which is given only for the taxonomy. Note also that Section 3.2 describes "three expert annotators" while the ethics statement says validation was done by the three authors, and that the LLM review prompts instruct "when in doubt, mark VALID".
Questions for authors

Clarifications:

  1. How many retained skills have no declared license? Section 3.1 removes only skills declaring a non-MIT/Apache license (plus their undeclared near-duplicates), while the ethics statement says only MIT/Apache skills are retained. If many undeclared-license skill bodies are redistributed, that seems legally problematic.
  2. How do SkillRet-Embedding models perform on an external skill-retrieval set, and on held-out queries from a third generator family?
  3. Per-stage QC counts and the human rejection rate.
  4. Train queries are 72 words with uniform k; eval queries are 170 words with k skewed to 1–2. Does the style mismatch help or hurt? E.g., evaluate on held-out Qwen-generated queries. Suggestions: the sentence-erasure comparison masks each model's own top sentences, so it is suggestive rather than controlled. Consider masking a fixed, human-annotated "skill-relevant" sentence.
9 comments
AuthorsOpenReview ↗

Response to Reviewer tYac (1/6)

We thank the reviewer for the careful and constructive review. We appreciate the recognition of the disjoint train/evaluation pools, the released training split and checkpoints, and the evaluation hygiene of the functional-equivalence audit. We answer each question below.

[Q1] Skills without a declared license

How many retained skills have no declared license? ... If many undeclared-license skill bodies are redistributed, that seems legally problematic.

No released skill lacked a license at collection. During collection we checked the LICENSE file of every source repository and kept only skills from MIT or Apache-2.0 repositories. All 16,129 released skills (6,006 in the evaluation pool) are covered by one of these two licenses: MIT for 14,065 and Apache-2.0 for 2,064.

"Undeclared" in Section 3.1 refers to the optional per-skill license: field in the SKILL.md front matter, not to the repository license.

  • 1,394 skills set this field, all to MIT or Apache-2.0.

  • The remaining 14,735 do not set it and fall under their repository's license.

  • Skills whose front matter declared any other license were removed, together with their near-duplicates.

We have rewritten Section 3.1 and the ethics statement to describe both levels consistently, marked in blue in the revised paper.

Later license changes. The skills were collected in March 2026. MIT and Apache-2.0 grants cannot be revoked for copies obtained under them. The license at collection therefore governs the released copies, even if a repository is later relicensed or deleted. The released data records each skill's source repository, URL, and license at collection, so the license of every redistributed skill can be traced.

AuthorsOpenReview ↗

Response to Reviewer tYac (2/6)

[Q2] External skill-retrieval sets and a third generator family

How do SkillRet-Embedding models perform on an external skill-retrieval set, and on held-out queries from a third generator family?

(a) External sets. Both sets below were not built by our pipeline. Because SkillRouter's released model is 0.6B, we compare models of the same size, each run with its own published inference settings. Differences are computed before rounding.

SkillRouter benchmark. We run its 75 scored tasks against its two candidate pools and report the mean. The "no overlap" columns exclude tasks with a gold skill that also appears in our training pool (same name or at least 80% of its body text). All SkillRouter results use the publicly released checkpoint. With SkillRouter's own evaluation code, it obtains Hit@1 56.7, below the 65.3 reported in its paper, while the base Qwen3-Embedding-0.6B is close to its reported value. We could not determine the cause. SkillRet-0.6B obtains 56.0, so it does not exceed the reported value.

ModelAll 75 tasks: nDCG@10Recall@10Completeness@10No overlap, 53 tasks: nDCG@10Recall@10Completeness@10
SkillRouter-Embedding-0.6B51.457.140.060.967.652.8
Qwen3-Embedding-0.6B (base)55.262.945.364.673.058.5
SkillRet-Embedding-0.6B56.667.251.365.476.866.0

SkillRet-0.6B outperforms the released SkillRouter-0.6B by +5.2 [1.2, 9.9] in nDCG@10, +10.1 [4.5, 16.3] in Recall@10, and +11.3 [4.0, 19.3] in Completeness@10 (paired bootstrap 95% CIs over tasks), and all three gains remain significant without overlapping tasks. Against its base model, SkillRet-0.6B improves Completeness@10 by +6.0 [2.0, 11.3], while nDCG@10 is similar. Hit@1 is similar across models.

SRA-Bench retrieval. We also evaluated retrieval on the six SRA-Bench datasets. We use each instance's question as the query, as in SkillRet and SkillRouter, rather than SR-Agents' default query, which also concatenates the system prompt and few-shot examples.

nDCG@10TheoremQALogicBenchMedCalcBenchBigCodeBenchToolQACHAMPMacro
SkillRouter-Embedding-0.6B80.715.079.031.924.643.645.8
Qwen3-Embedding-0.6B (base)85.610.497.943.639.147.254.0
SkillRet-Embedding-0.6B86.032.199.360.263.841.763.8

SkillRet-0.6B improves macro nDCG@10 over its base model by +9.9 [9.1, 10.6] and over SkillRouter-0.6B by +18.1 [17.1, 19.0]. It scores higher than both models on five of the six datasets. The exception is CHAMP, which requires distinguishing closely related mathematical techniques, a domain nearly absent from our training pool (24 of 10,123 skills by keyword search). No gold skill has at least 80% of its body text in a training skill, and excluding the 11 of 5,400 instances whose gold skill shares a name with one leaves the macro gains unchanged. We have added this table to Appendix P of the revised paper, marked in blue.

(b) Third generator family (on the SkillRet evaluation set). We kept the SkillRet evaluation pool of 6,006 skills and its qrels (i.e., the relevance labels that record which skills are the gold skills of each query), and replaced only the queries. For 500 sampled evaluation queries, we generated new queries for the same skill combinations with Gemma 4 26B A4B (Google, google/gemma-4-26B-A4B-it), using the same prompt, GAIA seed examples, and leakage filter. Gemma is a family different from both our evaluation generator (Claude) and training generator (Qwen). 499 queries passed the leakage filter. We evaluate all models on these queries and, for reference, on the 499 paired original queries.

ModelOriginal (Claude): nDCG@10Recall@10Completeness@10Gemma: nDCG@10Recall@10Completeness@10
BM2551.257.544.345.652.640.3
Qwen3-Embedding-0.6B61.968.352.958.764.148.5
SkillRet-Embedding-0.6B80.385.978.280.086.377.6
Qwen3-Embedding-8B64.471.056.563.670.254.9
SkillRet-Embedding-8B85.792.488.086.592.487.4

On queries from a generator family used in neither training nor the original evaluation, both SkillRet models outperform their same-size base models on all three metrics. On Gemma-written queries, the nDCG@10 gain over the base model is +21.3 [18.7, 24.1] for 0.6B and +22.9 [20.1, 25.9] for 8B (paired bootstrap 95% CIs), as large as on the original queries (+18.3 [15.8, 21.0] and +21.4 [18.7, 24.0]). We have added this experiment to Appendix R of the revised paper, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer tYac (3/6)

[Q3] Per-stage QC counts and the human rejection rate

Also missing: counts at each QC stage (generated, leakage-rejected, LLM-rejected, routed to humans, human-rejected)

We have added the following per-stage counts to Section 3.2 and Appendix C.3 of the revised paper, marked in blue.

Stage (evaluation split)Queries inRemovedQueries out
Generation (Claude Opus 4.6)––10,000
Automatic filtering (3-gram leakage filter + three-perspective LLM review)10,0004,086 (40.9%)5,914
Functional-equivalence and capability-leakage audit5,914716 (12.1%)5,198
Human validation5,198806 (15.5%)4,392

The leakage filter and LLM review are reported together as automatic filtering. Every query that passes automatic filtering and the audit is then reviewed by annotators, who reject 806 of the 5,198 queries (15.5%).

The audit also enforces the separation between the training and evaluation splits. For each evaluation skill, we check its nearest neighbors in both the evaluation and training pools. Equivalents within the evaluation pool are consolidated and their qrels remapped, so those queries keep their answers. Evaluation skills with a functional counterpart in the training pool are removed, together with the queries that rely on them. The evaluation pool shrinks from 6,660 to 6,006 skills, and all detected capability-level overlap between the splits is removed on the evaluation side, which is why the 63,259 training queries are unchanged. The two splits share no skills. Human validation is applied to the evaluation split, and we have redrawn Figure 1 so that it appears only on the evaluation branch, matching the text and the table above.

Reviewer_cCWkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced a benchmark for retrieving long-form agent skills. They eval different retrievers and ranking modles and report results on the benchmark.

Critical strengths
  1. the benchmark is very large. and many experiments have been done

  2. many method in the area have been tested. comprehensiveness is high

Critical weaknesses

my main concern is on the impact - why do we need such a benchmark to improve skill retrieval? What are the existing gap of the current agentic system in finding/ranking/retrieving skills.

It is not like traditional IR area where ranking a relevant doc down can make users very upset/miss important imformation. Agents can now easily find the relevant skill even if the skill is ranked low. Therefore, I don't link the work to impact or real utility.

Questions for authors

Could you elaborate more on how the benchmark can improve exisitng agentic system? explaining what are the gaps

4 comments
AuthorsOpenReview ↗

Response to Reviewer cCWk (1/2)

Agents can now easily find the relevant skill even if the skill is ranked low. Therefore, I don't link the work to impact or real utility.

We thank the reviewer for raising this question about practical impact. We agree that agents can partly compensate for imperfect retrieval. To quantify this compensation and assess whether retrieval quality still matters for task success, we conducted three new experiments. (1) Retrieval-time compensation compares longer candidate lists, top 50 against top 10, and iterative agentic search on all 4,392 test queries. (2) Downstream task evaluation measures end-to-end accuracy on 1,123 instances from the external SRA-Bench with no skills, retrieved skills, or gold skills. (3) Attempt-guided retrieval lets the agent first attempt a problem, has a search LLM write a query from that attempt, and lets the agent solve the problem again with the retrieved skills. These experiments show that additional search only partly compensates for weak retrieval and incurs extra inference overhead, and that skills from a weak retriever can leave the agent worse off than using no skills. They also show that a retriever trained on SkillRet improves task accuracy on an external benchmark, most clearly at 0.6B, which connects the benchmark's training resources to practical agent performance.

1. Can the agent compensate for a weak retriever? We tested the two ways an agent can do so at retrieval time on all 4,392 test queries: reading a longer candidate list (top 50 instead of top 10), and agentic search, in which a search LLM (Qwen3.8-27B) issues up to four searches, reads the top 10 results each time, and selects the skills itself. The table reports Completeness, the share of requests for which every required skill reaches the agent.

Completeness (%)Top-10, directTop-50, directAgentic search
Cost per request1 retrieval, about 0.5K tokens to read1 retrieval, about 2.5K tokens to read3.3 retrievals, 4.3 LLM calls, about 5.7K LLM tokens
BM2544.658.568.3
Qwen3-Embedding-0.6B51.165.269.1
Qwen3-Embedding-8B54.168.971.4
SkillRet-Embedding-0.6B78.991.175.5
SkillRet-Embedding-8B88.196.476.9

The token counts for the two direct columns are the names and descriptions of the candidates that the agent must read.

The reviewer's intuition is partly right. Agentic search lifts off-the-shelf retrievers by 17 to 24 points. But it does not close the gap. With the best off-the-shelf retriever, about 29% of requests still lack a required skill after agentic search, and 31% lack one in a top-50 list. A skill that never reaches the agent cannot be used, however capable the agent is. SkillRet-8B reaches 88.1% with a single top-10 retrieval and no LLM call, +16.8 [15.5, 18.0] above Qwen3-8B with agentic search (paired bootstrap 95% CI). Even SkillRet-0.6B at top 10 exceeds every off-the-shelf retriever at top 50 and with agentic search, at about one fifth and one tenth of their token cost.

2. Does this matter for task success? A low-ranked skill is not harmless, because the skills ranked above it enter the agent's context. On SRA-Bench (Su et al.), an external benchmark with gold skills, a Qwen3.5-9B agent receives the full text of the top-5 skills retrieved from a 26,262-skill library in its prompt (1,123 instances from four datasets, with a random sample of 300 from each of the three larger ones). We use the top 5 because no instance has more than five gold skills, so the top 5 can always contain all of them.

Accuracy (%)MedCalcTheoremQALogicBenchCHAMPMacro
No skill66.070.774.774.971.6
BM2579.778.741.772.268.0
Qwen3-Embedding-0.6B84.773.736.779.868.7
SkillRet-Embedding-0.6B87.078.757.082.576.3
Qwen3-Embedding-8B88.380.768.381.679.7
SkillRet-Embedding-8B86.379.777.081.681.2
Oracle (gold skills)85.377.383.782.182.1

With weak retrievers, the agent does worse than with no skills at all (68.0 and 68.7 versus 71.6). On LogicBench, unrelated skills mislead the agent into reasoning until its token limit without answering. Across the five retrievers, LogicBench accuracy follows exactly the ranking of how often the gold skill is among the five injected skills, and Appendix P of the revised paper plots this relation. SkillRet-8B improves on no skill by +9.6 [6.9, 12.4] and is within 0.9 [−1.2, 3.1] of the gold skills. SkillRet-0.6B improves over its base model by +7.6 [5.1, 10.1], while the gain of SkillRet-8B over its base model is not significant (+1.4 [−0.8, 3.7]).

AuthorsOpenReview ↗

Response to Reviewer cCWk (2/2)

3. Can the agent compensate after a first attempt? We let the agent attempt each problem without skills, had the search LLM (Qwen3.8-27B) read the attempt and write a search query for the knowledge the agent lacked, and let the agent solve the problem again with the top-5 skills retrieved by that query. This raises macro accuracy with Qwen3-0.6B from 68.7 to 72.4 (+3.7 [1.1, 6.4]), which is about the level of using no skills (71.6), at three LLM calls per problem. SkillRet-0.6B with a single direct retrieval reaches 76.3, +3.8 [1.3, 6.4] higher, with one LLM call and about half the tokens.

4. How the benchmark improves existing systems.

  • It measures the gap. Most existing skill benchmarks evaluate end-to-end success, where a failure may come from retrieval, selection, or execution. SkillRet isolates retrieval and scores multi-skill requests with Completeness. It shows that Qwen3-Embedding-8B retrieves all skills within the top 10 for only 32% of two-skill requests and 7% of three-skill requests.

  • It closes the gap. Its 63,259 training queries raise these figures to 85% and 67% on the same 8B backbone, and a fine-tuned 0.6B model surpasses every off-the-shelf model we evaluated.

  • The result is a drop-in component. An agent that searches still calls a retriever. Inside agentic search, SkillRet remains the better search tool (+3.6 [2.9, 4.3] nDCG@10 for 8B over its base model), and used directly it removes the extra LLM calls.

  • It complements end-to-end benchmarks such as SRA-Bench. SRA-Bench evaluates off-the-shelf retrievers and provides no training data, so it shows that retrieval is a bottleneck but does not provide resources to improve the retriever. Its 636 gold skills are constructed for its six task benchmarks, with web-collected skills serving as distractors. In SkillRet, the targets are real public agent skills (5,640 of the 6,006 evaluation skills are the answer to at least one query), half of the requests need several skills, and the training split makes retrievers trainable. The SkillRet rows in the table above are retrievers trained on SkillRet data and applied to SRA-Bench, whose library and tasks were not used in training.

Experiments (1) and (2) are now in Appendices Q and P of the revised paper, with all changes marked in blue. The Terminal-Bench run in the paper shows a cost reduction at matched success and does not by itself link retrieval quality to task success, so Section 5 now points to Appendix P for that evidence.

Thank you for your review and for raising an important question about the practical impact of skill retrieval. In our rebuttal, we addressed this concern through three additional experiments: comparing direct retrieval with larger candidate pools and agentic search, evaluating downstream task performance on SRA-Bench, and testing retrieval guided by an agent's initial attempt. We found that agentic search only partially compensates for weak retrieval while requiring substantially more inference. More importantly, weak retrievers can reduce task accuracy below the no-skill baseline (68.0% vs. 71.6%), whereas SkillRet-8B achieves 81.2%, close to the gold-skill oracle at 82.1%. We hope these results help clarify why retrieval quality matters for real agent performance, and would appreciate hearing whether they address your main concern before the discussion period ends.

Dear Reviewer,

With the author–reviewer discussion deadline approaching (November 18, AoE), we wanted to follow up once more. We sincerely want to do everything we can to address any remaining concerns you may have about our work. If there are any issues you feel have not been fully resolved, we would greatly appreciate it if you could share them with us so that we can address them before the discussion closes.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.