acceptodds
Under review as a conference paper at ICLR 2027

SkillDES: Dependency- and Evidence-Aware Skill-Set Selection for Composite Tasks

Abstract

Composite requests require complementary capabilities, yet ranking individual descriptions inherently fails to recover the required skill set. This mismatch is severely amplified by long documentation, uneven evidence distributions, and strict topological dependencies between subtasks. To address these challenges, we present SkillDES, a comprehensive retrieve-and-select framework that explicitly couples semantic task decomposition with dual-view candidate recall. To overcome semantic dilution in long documents, we introduce Long-Context Evidence Aggregation (LCEA), dynamically extracting bounded, node-specific complementary evidence for robust reranking. Furthermore, our dependency-aware beam search algorithm jointly balances skill compatibility, dynamic cardinality, and full capability coverage. Extensive evaluations across five diverse benchmarks demonstrate the effectiveness and robustness of our approach. Our system achieves a state-of-the-art complete-system SS-F1 of 68.4, representing a substantial improvement over the strongest baseline score of 51.2, while consistently outperforming existing retrieval, adaptation, and reranking methods and establishing a rigorous standard for composite capability selection. Our code is available at https://anonymous.4open.science/r/SkillDES.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill Retrieval, Composite Task Decomposition, Long-Context Evidence Aggregation, Dependency-Aware Skill Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1u3yOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies composite skill selection, where a user request may require a variable-size set of complementary skills rather than a single relevant item. The authors argue that conventional relevance ranking is insufficient because it does not directly determine prerequisite coverage, skill reuse across subtasks, or the number and composition of the selected skills. To address this, they propose SkillDES as a combined system of LLM-based DAG decomposition, dual-view global and node-level local retrieval, Long-Context Evidence Aggregation (LCEA) with dense/lexical/redundancy-aware block selection, supervised logistic reranking, cardinality capping, pairwise compatibility, and dependency-constrained beam search with anchor-beam routing. Evaluations on R3-Skill, SkillRet, ToolBench, ToolRet, and MCP are presented as tests of whether dependency-aware decoding and node-conditioned evidence improve complete set recovery (SS-F1/ESM) over ranking/reranking baselines.

Critical strengths

1. Useful formulation of composite retrieval as variable-cardinality skill-set selection

The paper identifies an important mismatch between individual relevance ranking and complete skill-set recovery. Its formulation explicitly considers prerequisite discovery, reuse of one skill across multiple subtasks, control of the number of distinct selected skills, and evidence distributed across long documentation.

This is consequential because the final objective is not merely to place relevant items near the top of a ranking, but to recover a compact set that jointly covers a composite request.

2. Broad empirical results, particularly for node-conditioned evidence

SkillDES reports the highest SS-F1 on all five benchmarks and increases average SS-F1 from 51.2 to 68.4. Removing the complete LCEA evidence branch produces large drops, suggesting node-conditioned evidence features are important in the long-document setting. The decomposition control also shows substantial recall improvement, supporting node-level retrieval views.

Critical weaknesses

1. Incomplete specification of the evaluated decoder and router

The reported system cannot be reconstructed from the paper alone. The final prediction depends on the anchor-beam router U(G)U(G) in Eq. (8), but U(G)U(G), the size-difference rule, and the threshold η\eta are not defined. It is therefore unclear how the system chooses between GaG_a and GbG_b.

The full decoder in Eq. (36) depends on Future⁡(t,s∣P)\operatorname{Future}(t,s\mid P), Dt(s∣P)D_t(s\mid P), Gain⁡(s∣P)\operatorname{Gain}(s\mid P), and Red⁡(s,S)\operatorname{Red}(s,S), whose exact forms are not provided. The associated weights wT,wF,wD,wv,wred,λnew,ηw_T,w_F,w_D,w_v,w_{\mathrm{red}},\lambda_{\mathrm{new}},\eta are also missing. Appendix C reports only the four channel weights, which is insufficient to recover the full decoder and routing.

Appendix B.11 analyzes a reduced three-term objective, while the main results are stated to use the full decoder. As a result, the fully specified objective in B.11 is not the evaluated system, and the evaluated system is not fully specified. This matters because the missing terms determine beam ranking and the router determines the final reported output.

Questions for authors

Q1. What is the central contribution beyond ablations?

The paper ablates dependency edges, LCEA branch, reranker, cardinality, and beam search, and compares to no-decomposition. These show components help, but do not separate decomposition from dependency modeling. No-decomposition changes local retrieval, LCEA, and cardinality features at once, while w/o dependency edges is only 1.1-1.9pp.

Could you provide:

  1. Unordered subtask model: keep node-level retrieval + node-conditioned evidence, remove directed edges and DAG scheduling.
  2. Retrieval + LCEA reranking + adaptive cardinality, without compatibility, beam search, or router.
  3. Full SkillDES.

For LCEA, w/o LCEA branch removes the whole feature branch. Could you compare your block-selection rule vs matched-budget top-k dense and MMR, keeping downstream features and reranker fixed?

Also report DAG quality (node coverage/atomicity, edge P/R) or robustness to missing/spurious/reversed edges.

Q2. How reliable are cardinality and routing?

Which benchmarks use RF for r^\hat r vs anchor size? Please report exact accuracy, MAE, under/over-estimation rate, and P(r^≥∣G∣)P(\hat r \geq |G|), plus performance with oracle cardinality.

r^\hat r is a hard cap ∣S∣≤r^|S| \leq \hat r, so underestimation makes gold infeasible. How does hard r^\hat r compare to r^+1\hat r+1 slack or soft penalty? Please separate hard cap, new-skill penalty, and size rule.

Please provide exact definitions of U(G)U(G), size-difference rule, and η\eta, and what "calibrated" means and how it is calibrated.

Per benchmark, please report fraction routed to GbG_b, plus always-anchor, always-beam, reported-router, and oracle-router performance.

Q3. Are budgets and cardinality rules matched?

MCP has 250 entries, but your budgets allow up to 64+6×16=16064+6\times16=160 candidates. Please report mean/median/percentiles of realized ∣C(q)∣|C(q)| after dedup for all benchmarks, and MCP results with matched budgets 10/20/40.

SS-F1/ESM are sensitive to output size. For each baseline in Table 1, please specify final set construction: fixed top-k, dev-selected k, threshold, native stopping rule, or cardinality predictor, plus candidate/reranking depth and whether all methods share the same pool and tuning budget. What does the strongest reranking baseline obtain with the same cardinality predictor/margin rule as SkillDES?

Reviewer_c3CQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SkillDES, a framework for selecting a variable-size set of skills for complex tool using tasks. It uses an LLM to decompose a user request into a task dependency DAG. It further combines global and subtask-level retrieval to form a candidate pool. The proposed LCEA selects relevant and non-redundant document chunks to support candidate reranking. The framework finally uses cardinality estimation and dependency-aware beam search to construct a skill set, with a final router choosing between the search result and a ranking-based anchor set. Experiments show that SkillDES outperformed several tool retrieval baselines across different benchmarks.

Critical strengths

S1: Problem Formulation. This paper targets at important topic in the agentic era, in which tool/skill selection is crucial to the overall agentic performance.

S2: Extensive experiments and analysis: The authors have done extensive experiments to support the arguments in the paper, showing that SkillDES has better performance over baselines. Analysis on the component and ablations are appreciate for the completeness of this work.

Critical weaknesses

W1: Innovation. This work combines a lot of techniques, including subtask decomposition, long-text chuncation, evidence scoring/reranking, etc. However, these are very well-studied topics in this area. The authors have to clearly state the reason why combining these techniques is important and difficult, especially the latter which I do not see from the paper. Simply combining these techniques is an engineering contribution instead of an academic one.

W2: For the long-context problem, a clear baseline is to use a long-context LLM. No such baselines have been introduced and no reason has been given.

W3: There is no explanation and justifications for component designs:

  • Evidence-scoring: any justification for the weights of nerual and lexical relevance? Experimental or heuristic?
  • LCEA and evidence-conditioned ranking: Why a budget is fixed? And is it reliable? Why not just use a reranker and choose top-k? Where do the benefits come from?

W4: Potential unfair comparisons. SkillDES uses a dataset specific MPNet but the baselines are general or the information is missing. SkillDES uses a relatively large candidiate size (max 160), but baselines' max candidate sizes are not sure.

W5: Readability. The method and experiments are relatively not well-presented. Equations often break the descriptions, which harms readability. Illustrative figures (2 & 3) are not easy to read, potentially too informative and abstractive.

Questions for authors

See weaknesses.

Reviewer_oGhXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a multi-stage pipeline for selecting a set of skills for composite tasks: an LLM decomposes the query into a task graph, a retriever pulls global and per-subtask candidates, chunk-level evidence is selected from long skill documents, a learned reranker and a cardinality predictor pick an anchor set, and a dependency-aware beam search produces the final skill set. Large gains are reported on five benchmarks.

Critical strengths

The problem is the right one for multi-step agents: what we need is a complete set of skills, not a ranked list of individually similar ones.

Extensive experiments: five datasets, many baselines, module ablations and a decomposition control.

Critical weaknesses

My main concern is whether the gain comes from the proposed ideas or simply from training many dataset-specific components. Every learnable part of the syste is fitted on each target dataset, while the paper never says how the baselines are run, or how a baseline's ranked list is turned into a set for the set-level metrics. Since the paper's own cardinality predictor alone is worth 12 points on one dataset, this unstated choice can explain much of the gap. Removing the dependency edges, which is the contribution in the title, only costs 1-2 points.

From a practical point of view, the pipeline needs an LLM call to decompose every query, plus chunk encoding and beam search. For a real agent this is far heavier than a single retrieval call, and the paper does not discuss latency or cost, so I cannot tell whether an agent system would actually want this over a strong retriever plus a simple set cutoff.

Questions for authors

How are the baseline sets obtained (fixed k? which k?), and what does a strong retriever look like when it gets the same per-dataset training and the same cardinality predictor as your system?

What is the per-query latency and cost of the full pipeline compared with plain retrieval, and when would an agent system prefer it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.