- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies composite skill selection, where a user request may require a variable-size set of complementary skills rather than a single relevant item. The authors argue that conventional relevance ranking is insufficient because it does not directly determine prerequisite coverage, skill reuse across subtasks, or the number and composition of the selected skills. To address this, they propose SkillDES as a combined system of LLM-based DAG decomposition, dual-view global and node-level local retrieval, Long-Context Evidence Aggregation (LCEA) with dense/lexical/redundancy-aware block selection, supervised logistic reranking, cardinality capping, pairwise compatibility, and dependency-constrained beam search with anchor-beam routing. Evaluations on R3-Skill, SkillRet, ToolBench, ToolRet, and MCP are presented as tests of whether dependency-aware decoding and node-conditioned evidence improve complete set recovery (SS-F1/ESM) over ranking/reranking baselines.
Critical strengths
1. Useful formulation of composite retrieval as variable-cardinality skill-set selection
The paper identifies an important mismatch between individual relevance ranking and complete skill-set recovery. Its formulation explicitly considers prerequisite discovery, reuse of one skill across multiple subtasks, control of the number of distinct selected skills, and evidence distributed across long documentation.
This is consequential because the final objective is not merely to place relevant items near the top of a ranking, but to recover a compact set that jointly covers a composite request.
2. Broad empirical results, particularly for node-conditioned evidence
SkillDES reports the highest SS-F1 on all five benchmarks and increases average SS-F1 from 51.2 to 68.4. Removing the complete LCEA evidence branch produces large drops, suggesting node-conditioned evidence features are important in the long-document setting. The decomposition control also shows substantial recall improvement, supporting node-level retrieval views.
Critical weaknesses
1. Incomplete specification of the evaluated decoder and router
The reported system cannot be reconstructed from the paper alone. The final prediction depends on the anchor-beam router in Eq. (8), but , the size-difference rule, and the threshold are not defined. It is therefore unclear how the system chooses between and .
The full decoder in Eq. (36) depends on , , , and , whose exact forms are not provided. The associated weights are also missing. Appendix C reports only the four channel weights, which is insufficient to recover the full decoder and routing.
Appendix B.11 analyzes a reduced three-term objective, while the main results are stated to use the full decoder. As a result, the fully specified objective in B.11 is not the evaluated system, and the evaluated system is not fully specified. This matters because the missing terms determine beam ranking and the router determines the final reported output.
Questions for authors
Q1. What is the central contribution beyond ablations?
The paper ablates dependency edges, LCEA branch, reranker, cardinality, and beam search, and compares to no-decomposition. These show components help, but do not separate decomposition from dependency modeling. No-decomposition changes local retrieval, LCEA, and cardinality features at once, while w/o dependency edges is only 1.1-1.9pp.
Could you provide:
- Unordered subtask model: keep node-level retrieval + node-conditioned evidence, remove directed edges and DAG scheduling.
- Retrieval + LCEA reranking + adaptive cardinality, without compatibility, beam search, or router.
- Full SkillDES.
For LCEA, w/o LCEA branch removes the whole feature branch. Could you compare your block-selection rule vs matched-budget top-k dense and MMR, keeping downstream features and reranker fixed?
Also report DAG quality (node coverage/atomicity, edge P/R) or robustness to missing/spurious/reversed edges.
Q2. How reliable are cardinality and routing?
Which benchmarks use RF for vs anchor size? Please report exact accuracy, MAE, under/over-estimation rate, and , plus performance with oracle cardinality.
is a hard cap , so underestimation makes gold infeasible. How does hard compare to slack or soft penalty? Please separate hard cap, new-skill penalty, and size rule.
Please provide exact definitions of , size-difference rule, and , and what "calibrated" means and how it is calibrated.
Per benchmark, please report fraction routed to , plus always-anchor, always-beam, reported-router, and oracle-router performance.
Q3. Are budgets and cardinality rules matched?
MCP has 250 entries, but your budgets allow up to candidates. Please report mean/median/percentiles of realized after dedup for all benchmarks, and MCP results with matched budgets 10/20/40.
SS-F1/ESM are sensitive to output size. For each baseline in Table 1, please specify final set construction: fixed top-k, dev-selected k, threshold, native stopping rule, or cardinality predictor, plus candidate/reranking depth and whether all methods share the same pool and tuning budget. What does the strongest reranking baseline obtain with the same cardinality predictor/margin rule as SkillDES?