- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper identifies a gap between skill retrieval and skill-set recovery: even when all required skills are retrieved, they may be interleaved with distractors such that no ranked prefix recovers the correct set. It proposes SkillCraft, which predicts non-monotonic candidate membership scores using contextualized retrieval/reranking evidence and calibrates a variable-cardinality skill set. Experiments on SkillRet show improved ExactSet and Set F1, with additional zero-shot downstream gains on three external tasks.
Critical strengths
- The distinction between candidate completeness and set recoverability is well motivated and empirically substantial, especially for multi-skill queries where rank-prefix selection frequently fails.
- The evaluation seems olid with ablations and zero-shot downstream evaluation on a different skill library without retraining the resolver.
Critical weaknesses
- The method is evaluated mainly in a static pre-execution set-selection setting. SkillCraft assumes that the required skills can be resolved before execution as a small set, which makes the problem set up no 100% aligned with real world agentic scenarios and applications - whereas in more interactive agent settings tool requirements may depend on intermediate observations and evolve over time. It is therefore unclear how well the formulation transfers to sequential tool-use scenarios. (such as tau bench or MCP Atlas).
- The cardinality setting is relatively limited. The current implementation caps the predicted set at three skills, matching the maximum target-set size in SkillRet. Since the paper itself shows that cardinality estimation is a major bottleneck, it is unclear how the method behaves when tasks require larger skill sets.
Questions for authors
- How would SkillCraft extend to interactive and multiturn settings where the required tools cannot be determined before execution and instead depend on intermediate observations? Would the resolver need to be rerun at every step?
- How does the method behave when the required skill-set cardinality is substantially larger than three? Is the current performance partly dependent on the restricted 1–3 skill setting?
Thank you for the additional experiments. This addresses my main concern about the added value of list context. Three-skill recovery remains a limitation, but the new evidence moves me from 2 to 3.