acceptodds
Under review as a conference paper at ICLR 2027

Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing

Abstract

Large language model agents increasingly rely on reusable skills, making skill retrieval a critical front-end component of agent systems. Skill retrieval, however, is not ordinary document retrieval: a useful top- result must contain individually relevant skills that also form an executable set for the current query. Existing benchmarks and training pipelines largely supervise pairwise relevance and discard the rejection decisions produced when a language model judges a sampled skill combination to be implausible. We introduce R3-Skill, a Chinese–English benchmark that retains these rejections as query-conditioned compatibility supervision. R3-Skill contains 10,246 deduplicated skills, 41,592 accepted queries, and 32,828 rejected annotations across four language directions; all multi-skill test labels were independently reviewed by nine experts, and 15,962 parseable rejections are organized into an eight-class taxonomy. We further propose a two-stage system composed of R3-Embedding, a multi-positive bi-encoder for large-pool recall, and R3-Reranker, a cross-encoder trained with graded ListNet supervision. Our analysis shows that this signal is stage-dependent, helping cross-encoder reranking while providing no benefit for the tested bi-encoder objective. On R3-Skill, the complete pipeline achieves Hit@1, NDCG@10, and Set-Compat, a relative gain over the strongest reranking baseline. It also obtains NDCG@10 on SkillRet, demonstrating transfer beyond R3-Skill.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Skill retrieval, LLM agents, set compatibility, rejection supervision

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_fEAGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that skill routing for LLM agents differs from document retrieval: a useful top-K result must contain skills that are individually relevant and also form an executable set for the given query. The authors argue that the rejection information (SKIP) produced when an LLM judges whether a skill combination can satisfy a task can serve as a supervision signal.

Contributions:

  1. R3-Skill, a Chinese–English benchmark that retains these rejections as query-conditioned compatibility supervision.
  2. R3, a two-stage system composed of R3-Embedding (multi-positive InfoNCE with a sibling term, no SKIP) and R3-Reranker (ListNet with graded labels GT=3 / SKIP partner=1 / other=0).

Experimental results show that SKIP supervision hurts when injected as bi-encoder negatives but helps the cross-encoder.

Critical strengths
  1. A valuable perspective. Skill routing should consider not only the relevance between the task and each individual skill, but also whether a combination of multiple skills can actually solve the task. The paper further proposes using the LLM's rejection judgments as a training signal to address this problem.
  2. An important benchmark for related research. The skill-level train/test split prevents capability memorization; expert review helps ensure label quality; and the bilingual directions (en2zh, zh2en, zh2zh) are a useful addition to existing English-only resources.
Critical weaknesses
  1. How the query-conditioned SKIP labels is obtained is not described clearly enough. According to Appendix G, the generator makes SKIP decisions on skill sets without any user query, and the released skip_train.jsonl is consistent with this. Query conditioning is therefore introduced only at training time. However, neither the paper nor the submitted code specifies how a WRITE query is matched to a SKIP set or how the shuffled control D is implemented. The released evaluation script is inconsistent with the cited metric definition.The paper states that Recall@K and Comp@K follow SkillRet, which defines Completeness@K as the fraction of queries with perfect recall, i.e., 𝟙[G ⊆ T_K]. In reproduce.py, line 55 computes Recall as |G∩T_K| / max(1,|G|) and line 56 computes Comp as |G∩T_K| / min(|G|,K). Every test query has |G| ≤ 3 and K ∈ {5,10,15}, so both denominators equal |G| and the script yields Recall@K = Comp@K for any ranking. The script's own reference dictionary (lines 14–29) indeed lists comp equal to recall. However, Table 4 reports different values.

  2. Part of the stated motivation is outside what the method can represent, and the benchmark does not directly test it.The R3-Reranker scores each (q, s) pair independently. It can learn query-dependent necessity but cannot represent within-query substitutability or redundancy, because when two near-duplicate skills are both relevant to q, f(q, s) cannot see whether the other is already selected, as in the redundant pair "Code Review + Systematic Code Review" showcased in Table 1. A construction that targets the title's thesis more directly would sample skills with near-identical descriptions but incompatible bodies.

Other issue (anonymity). The supplementary ZIP includes compiled Python bytecode files (pycache/*.pyc) that retain local source paths containing a personal-looking username, which may compromise double-blind anonymity. I have reported this to the AC and, following the AC's guidance, have evaluated the submission on its technical merits; this issue did not affect my rating.

Questions for authors
  1. Could you provide the exact method used to pair SKIP sets with WRITE queries? Please give one complete example tracing a WRITE query and its source SKIP record to the final 3/1/0 training list.
  2. Could you state the exact Comp@K definition used, and explain the discrepancy between the reranker reference values in reproduce.py and Table 4?
  3. Can the pointwise reranker address the compatibility between different skills under the same query, which is part of the motivation for constructing the benchmark?
Reviewer_qLLEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes a framework for skill routing with relevance and trains embedding and re-ranking models for optimal skill usage. The work collects multilingual skill training and test dataset; and trains models with multi-positive recall and listwise ranking objectives. The resulting framework demonstrates strong skill retrieval performance over existing embedding and reranker models in the curated test set and a separate benchmark.

Critical strengths

Strong methodology: The paper proposes intuitive framework focusing on set compatibility for training skill retrieval with well curated dataset. The paper reflected data diversity and quality though the rejection taxonomy and query distribution.The framework shows strong retrieval performance (87 vs 64 of SkillRet) and Set-Compatibility (21 vs 11 of SkillRet) over strong baselines.

Bi-lingual scope: The data covers two languages which demonstrates the cross-lingual generalization of the proposed method. Covering multiple languages are important dimension to ensure SoTA LLM technologies, like skills for agents, are being adopted across different regions.

Critical weaknesses

Expert Review: It is unclear how the experts have reviewed the test labels, including their number, agreement and accuracy. Therefore, I am not sure how much the used Test set can be trusted. As the train and test set results from same pipeline (though different LLMs are used), this step requires transparency for better reliability of reported results.

Questions for authors

How are the test set reviewed (protocols) and what is their agreement scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.