acceptodds
Under review as a conference paper at ICLR 2027

SkillCraft: Skill-Set Retrieval Beyond Ranking for LLM Agents

Abstract

Large language model (LLM) agents increasingly rely on external procedural skills to solve complex tasks, yet existing skill retrieval predominantly follows a relevance ranking formulation. We show that this formulation overlooks a critical gap: candidate discovery does not imply set recoverability. In multi-skill queries, target skills are frequently interleaved with irrelevant distractors, destroying rank-prefix recoverability. On SkillRet, while top-50 retrieval achieves 89.6% candidate completeness, 46.5% of complete queries remain source-rank inseparable, admitting a median of seven unavoidable intruders under contiguous prefixes. To resolve this bottleneck, we propose SkillCraft, which reformulates post-retrieval selection as a dedicated set-resolution problem. Rather than treating the upstream ranking as an immutable boundary, SkillCraft constructs contextualized list-relative evidence and estimates non-monotonic membership probabilities, dynamically skipping high-ranked distractors to recover non-contiguous skill combinations. Candidate-level membership learning is further coupled with set-level calibration to govern variable-cardinality admission. On the 6600-skill SkillRet benchmark, SkillCraft achieves the strongest skill set-recovery performance and set F1, outperforming recent skill retrieval systems (e.g., SkillRouter, SkillReason, and R3-Skill. Beyond set recovery, its selected skill contexts yield higher downstream task accuracy across three external benchmarks in practice. Moreover, SkillCraft outperforms 17 competitive baselines across sparse, encoder-only, decoder-only, and two-stage retrieve-and-rerank pipelines in candidate retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent SKills, Skill Retrieval, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_LLEyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a gap between skill retrieval and skill-set recovery: even when all required skills are retrieved, they may be interleaved with distractors such that no ranked prefix recovers the correct set. It proposes SkillCraft, which predicts non-monotonic candidate membership scores using contextualized retrieval/reranking evidence and calibrates a variable-cardinality skill set. Experiments on SkillRet show improved ExactSet and Set F1, with additional zero-shot downstream gains on three external tasks.

Critical strengths
  1. The distinction between candidate completeness and set recoverability is well motivated and empirically substantial, especially for multi-skill queries where rank-prefix selection frequently fails.
  2. The evaluation seems olid with ablations and zero-shot downstream evaluation on a different skill library without retraining the resolver.
Critical weaknesses
  1. The method is evaluated mainly in a static pre-execution set-selection setting. SkillCraft assumes that the required skills can be resolved before execution as a small set, which makes the problem set up no 100% aligned with real world agentic scenarios and applications - whereas in more interactive agent settings tool requirements may depend on intermediate observations and evolve over time. It is therefore unclear how well the formulation transfers to sequential tool-use scenarios. (such as tau bench or MCP Atlas).
  2. The cardinality setting is relatively limited. The current implementation caps the predicted set at three skills, matching the maximum target-set size in SkillRet. Since the paper itself shows that cardinality estimation is a major bottleneck, it is unclear how the method behaves when tasks require larger skill sets.
Questions for authors
  1. How would SkillCraft extend to interactive and multiturn settings where the required tools cannot be determined before execution and instead depend on intermediate observations? Would the resolver need to be rerun at every step?
  2. How does the method behave when the required skill-set cardinality is substantially larger than three? Is the current performance partly dependent on the restricted 1–3 skill setting?
Reviewer_tSeGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important problem in skill retrieval: recovering the exact set of required skills from retrieved candidates. It trains a gradient-boosted classifier to predict each candidate’s membership in the target set, then applies a threshold calibrated on training data to select one to three skills. Experiments support the usefulness of contextual features, although recovering more complex skill sets remains challenging.

Critical strengths

"Even when all targets are retrieved, truncating the source ranking can admit distractors." is a useful problem formulation that better reflects the skill sets agents actually consume.

The pointwise-selector comparison is well controlled, sharing the candidate pool, underlying scoring signals, model class, and calibration procedure while varying contextual features.

Critical weaknesses

The implementation is a candidate-level classifier with contextual features and set-level threshold calibration. Skipping distractors in the source ranking is not unique to this approach. Using reranker model with a threshold is a similar method. But the paper didn't test the performance of a trained reranker with same training data. Why tree model? Why not llm with a classifier head?

Evidence for recovering complex skill sets is limited. ExactSet is only 21.49% for two-skill queries and 1.93% for three-skill queries.

Questions for authors

Does the advantage persist when generative selectors receive the same body snippets? What are the latency and computational costs of the complete pipeline?

What explains the limited performance in recovering complex skill sets? Are there any bottlenecks in the method?

1 comment

Thank you for the additional experiments. This addresses my main concern about the added value of list context. Three-skill recovery remains a limitation, but the new evidence moves me from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.