- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that skill routing for LLM agents differs from document retrieval: a useful top-K result must contain skills that are individually relevant and also form an executable set for the given query. The authors argue that the rejection information (SKIP) produced when an LLM judges whether a skill combination can satisfy a task can serve as a supervision signal.
Contributions:
- R3-Skill, a Chinese–English benchmark that retains these rejections as query-conditioned compatibility supervision.
- R3, a two-stage system composed of R3-Embedding (multi-positive InfoNCE with a sibling term, no SKIP) and R3-Reranker (ListNet with graded labels GT=3 / SKIP partner=1 / other=0).
Experimental results show that SKIP supervision hurts when injected as bi-encoder negatives but helps the cross-encoder.
Critical strengths
- A valuable perspective. Skill routing should consider not only the relevance between the task and each individual skill, but also whether a combination of multiple skills can actually solve the task. The paper further proposes using the LLM's rejection judgments as a training signal to address this problem.
- An important benchmark for related research. The skill-level train/test split prevents capability memorization; expert review helps ensure label quality; and the bilingual directions (en2zh, zh2en, zh2zh) are a useful addition to existing English-only resources.
Critical weaknesses
-
How the query-conditioned SKIP labels is obtained is not described clearly enough. According to Appendix G, the generator makes SKIP decisions on skill sets without any user query, and the released
skip_train.jsonlis consistent with this. Query conditioning is therefore introduced only at training time. However, neither the paper nor the submitted code specifies how a WRITE query is matched to a SKIP set or how the shuffled control D is implemented. The released evaluation script is inconsistent with the cited metric definition.The paper states that Recall@K and Comp@K follow SkillRet, which defines Completeness@K as the fraction of queries with perfect recall, i.e., 𝟙[G ⊆ T_K]. Inreproduce.py, line 55 computes Recall as |G∩T_K| / max(1,|G|) and line 56 computes Comp as |G∩T_K| / min(|G|,K). Every test query has |G| ≤ 3 and K ∈ {5,10,15}, so both denominators equal |G| and the script yields Recall@K = Comp@K for any ranking. The script's own reference dictionary (lines 14–29) indeed lists comp equal to recall. However, Table 4 reports different values. -
Part of the stated motivation is outside what the method can represent, and the benchmark does not directly test it.The R3-Reranker scores each (q, s) pair independently. It can learn query-dependent necessity but cannot represent within-query substitutability or redundancy, because when two near-duplicate skills are both relevant to q, f(q, s) cannot see whether the other is already selected, as in the redundant pair "Code Review + Systematic Code Review" showcased in Table 1. A construction that targets the title's thesis more directly would sample skills with near-identical descriptions but incompatible bodies.
Other issue (anonymity). The supplementary ZIP includes compiled Python bytecode files (pycache/*.pyc) that retain local source paths containing a personal-looking username, which may compromise double-blind anonymity. I have reported this to the AC and, following the AC's guidance, have evaluated the submission on its technical merits; this issue did not affect my rating.
Questions for authors
- Could you provide the exact method used to pair SKIP sets with WRITE queries? Please give one complete example tracing a WRITE query and its source SKIP record to the final 3/1/0 training list.
- Could you state the exact Comp@K definition used, and explain the discrepancy between the reranker reference values in
reproduce.pyand Table 4? - Can the pointwise reranker address the compatibility between different skills under the same query, which is part of the motivation for constructing the benchmark?