acceptodds
Under review as a conference paper at ICLR 2027

SkillComp: Internalizing Skill Relations in Dense Retrievers for Complete Skill-Set Retrieval from Large-Scale Libraries

Abstract

Reusable skills equip large language model agents with specialized capabilities for executing complex tasks, which often require multiple skills to work together. However, some skills needed in intermediate steps may be implicit in the task description, making it difficult for dense retrievers to retrieve the complete required skill set from a large skill library. To address this limitation, we propose SkillComp, which uses structural relations among required skills to construct additional training supervision for complete skill-set retrieval. Specifically, SkillComp selects related skills based on these relations and appends their descriptions to the query to form expanded queries, providing additional context for retrieving the remaining required skills. It then jointly trains the retriever on the plain and expanded queries using the complete required skill set as positive targets, thereby improving the alignment between the plain query and the complete required skill set. Experiments on SkillRet and SkillBench show that SkillComp improves skill retrieval, with particularly strong gains on multi-skill tasks. On the multi-skill subset of SkillRet, SkillComp improves Recall@10 and Completeness@10 over Qwen3-Embedding-0.6B by +33.08 and +48.54 points, respectively, and over the strongest baseline by +2.57 and +5.19 points. These results support that SkillComp improves complete-set retrieval, while retaining standard plain retrieval at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
skill retrieval, multiple positives retrieval, dense encoder

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iwYkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies complete skill-set retrieval for LLM agents. The authors observe that some required skills are only implicit in the task description, which makes them hard for dense retrievers to find.

They propose SkillComp:

  1. An LLM (GPT-5.4-mini) annotates pairwise relations (sequential or co-required) among the required skills of each training query.
  2. The description of a selected required skill is appended to the query to form expanded queries.
  3. The retriever is trained jointly on plain and expanded queries, with the complete required set as positives for both. Only the plain query is used at inference.
Critical strengths
  1. A relevant problem. Retrieving the complete required skill set is more import than a single best skill for multi-skill routing.
  2. A simple, inference-free design. Expanded queries and skill relations are used only in training-time, so inference remains standard dense retrieval with no extra latency and no skill graph to maintain.
Critical weaknesses
  1. The ablation provides limited support for attributing the gains to skill relations.The title and motivation attribute the improvement to internalizing structural relations among required skills, but Table 3 suggests that relation-guided multi-skill expansion without single-skill expansion performs almost identically to Base (C@20 83.47 vs. 83.49) and the gains appear only once the single-skill self-expanded query (q⊕s → s) is added. That auxiliary task does not involve skill relations or multi-skill completeness. In addition, the ablation is reported only on the full SkillRet test set, not on the multi-skill subset, which the paper claims the largest benefit from. The additional generalization benefit of relation-guided training has not been demonstrated. The relationships between skills are internalized as prior knowledge through parameter updates. Whether this injected local knowledge remains effective for unseen skills warrants further investigation. Table 6 only compares the original Qwen3-Embedding-0.6B with SkillComp-Embedding-0.6B and does not include results for the Base and Random Appended Skill variants. These results therefore cannot establish whether relation-guided training provides additional benefits on unseen skill combinations beyond domain adaptation and expanded-query supervision.

  2. The relation-annotation pipeline is inconsistent with the research motivation. The motivation (§1) concerns required skills that are "implied by other required skills rather than directly stated in the task description." However, the annotation (Appendix F.1) first restricts relations to the subset H ⊆ G whose relevance is directly supported by the task description. Implicit skills are therefore the ones most likely to fall outside H. They remain positives, but by construction the relation-guided expansion does not address them. Relations also determine the appended skill for only part of the training data. By Table 12, 14,300 of the 42,483 multi-skill queries (33.7%) contain at least one sequential dependency.

Questions for authors
  1. Could you report the Table 3 ablation on the SkillRet multi-skill subset, and add a variant with only single-skill self-expansion (no multi-skill expansion) ?

  2. How does the relation-guided expansion help required skills that are implicit in the task description, given that relations are annotated only among skills directly supported by the query?

  3. Appendix A.1 provides useful evidence of transfer to unseen skills. Could you also report the results of Base and Random Appended Skill on SRA-Bench, especially R@K and C@K for multi-skill tasks? This comparison would help clarify whether relation-guided training adds a generalization benefit beyond domain adaptation and expanded-query supervision.

7 comments
AuthorsOpenReview ↗

Response to Reviewer iwYk (1/2)

Thank you for your detailed feedback. We appreciate your recognition of the importance of complete skill retrieval and SkillComp's simple and efficient design. We have addressed your concerns below and would appreciate hearing whether our responses address them.


Q1. Could you report the Table 3 ablation on the SkillRet multi-skill subset, and add a variant with only single-skill self-expansion (no multi-skill expansion)?

Following your suggestion, we added w/o Multi-Skill Expansion (single-skill self-expansion only) and evaluated ablations on the multi-skill subset of the SkillRet test set (Table R3-1). Both w/o Multi-Skill Expansion and w/o Single-Skill Expansion perform close to Base, while SkillComp-Embedding-0.6B improves C@20 by +2.69% over Base and +2.01% over w/o Multi-Skill Expansion.

These results show that the gains are not driven by single-skill self-expansion pairs alone. Within our cross-view training framework (Section 4.2), single-skill self-expansion extends training to all queries under a unified objective, while multi-skill expansion helps the encoder learn the relevance between skills. Thus, single-skill self-expansion supports unified cross-view training rather than driving the improvements on its own.

Furthermore, under the same cross-view training framework, SkillComp-Embedding-0.6B consistently outperforms Random across three training seeds, improving multi-skill C@20 by +0.62% on average, providing further evidence for the benefit of relation-guided expansion.

Table R3-1: Ablation results on the multi-skill subset of the SkillRet test set (C@20, %). Random and SkillComp-Embedding-0.6B report mean ± std across three training seeds.

VariantCross-view CoverageRelationSkillRet (All)SkillRet (Multi)SkillBench (Multi)
Base–✓\checkmark83.4973.6749.02
w/o Single-Skill ExpansionMulti only✓\checkmark83.4773.6745.10
w/o Multi-Skill ExpansionSingle only×\times83.7774.3547.06
Random Appended SkillSingle + Multi×\times84.55 ± 0.2375.74 ± 0.4847.71 ± 1.13
SkillCompSingle + Multi✓\checkmark85.23 ± 0.0276.36 ± 0.1352.29 ± 2.26
AuthorsOpenReview ↗

Response to Reviewer iwYk (2/2)

Q2. How does the relation-guided expansion help the required skills that are implicit in the task description, given that relations are annotated only among skills directly supported by the query?

We retain implicitly required skills during relation annotation while excluding task-irrelevant skills to provide more informative expansions. Meanwhile, we preserve the original gold skill labels for retrieval training to ensure a fair comparison. We clarify these two points below.

(1) Implicitly required skills are retained during annotation. When constructing HH, the LLM evaluates whether a skill is needed to complete the task, rather than whether it is explicitly mentioned in the query. For example, query q-train-029706 asks the agent to analyze a codebase, create a module, and provide a spoken response. Although text-to-speech is not explicitly mentioned, it is required to deliver the spoken output and is therefore retained in HH for dependency annotation.

Conversely, query q-train-060513 requests Seoul weather and League of Legends match history, for which the gold skill go-error-checking is unnecessary. Excluding this skill from HH avoids noisy relation annotations and prevents it from being selected for relation-guided expansion. Our manual audit shows 98% annotation accuracy across 200 samples.

By providing additional information about related required skills, relation-guided expansion helps the encoder learn the relevance of implicitly required skills. As shown in Table R3-1, SkillComp-Embedding-0.6B outperforms Random in multi-skill C@20, supporting the benefit of relation-guided expansion.

(2) We preserve the original training data for a fair comparison. We retain the same training queries and complete gold skill sets, including skills excluded from HH. The 33.7% figure refers only to queries with annotated sequential dependencies. For the remaining multi-skill queries, we apply score-based selection within the gold set. Thus, all 42,483 multi-skill queries receive expansion and are trained to retrieve the complete gold skill set, including skills without annotated relations.

We will revise Appendix F.1 to clarify these distinctions.


Q3. Appendix A.1 provides useful evidence of transfer to unseen skills. Could you also report the results of Base and Random Appended Skill on SRA-Bench, especially R@K and C@K for multi-skill tasks? This comparison would help clarify whether relation-guided training adds a generalization benefit beyond domain adaptation and expanded-query supervision.

We extended our ablation study to SkillBench and SRA-Bench, reporting multi-skill results on both benchmarks across three training seeds. SkillComp-Embedding-0.6B improves multi-skill C@20 over Random by +4.58% on SkillBench and +0.68% on SRA-Bench. On SRA-Bench overall, SkillComp-Embedding-0.6B also improves C@20 by +11.28% over Base and +6.10% over Random. SkillBench includes more challenging queries requiring up to seven skills, where SkillComp-Embedding-0.6B shows larger gains in multi-skill retrieval.

These results show that relation-guided selection provides additional benefits beyond random expansion. We will clarify the evaluation details in the revision.

Table R3-2: Ablation results on SkillBench and SRA-Bench. Random and SkillComp-Embedding-0.6B report mean ± std across three training seeds; Base is a single run.

BenchmarkSubsetModelR@20C@20
SkillBenchMultiBase72.9749.02
Random72.42 ± 1.1147.71 ± 1.13
SkillComp73.66 ± 1.1052.29 ± 2.26
SRA-BenchAllBase64.2157.44
Random69.45 ± 10.2562.62 ± 10.24
SkillComp75.57 ± 0.3468.72 ± 0.34
MultiBase61.7332.96
Random63.62 ± 0.4934.31 ± 0.42
SkillComp64.19 ± 0.2134.99 ± 0.21

Thank you again for your feedback. We would greatly appreciate it if you could let us know whether our responses address your concerns.

AuthorsOpenReview ↗

Summary of revisions for Reviewer iwYk

Thank you again for your detailed feedback. We understand that the discussion period can be busy, so we briefly summarize our responses and revisions below.

1. Single-skill self-expansion ablation (W1, Q1). We extended our ablation study by adding the w/o Multi-Skill Expansion variant. Expanding only single-skill tasks yields little improvement in multi-skill C@20 over Base, whereas expanding both task types with randomly selected skills improves multi-skill C@20 by 2.07% on SkillRet. Relation-guided expansion provides further gains of 0.62% on SkillRet and 4.58% on SkillBench over Random. These results suggest that cross-view training helps the encoder learn from expanded queries across task types, while relation-guided expansion provides more informative training views. We revised Section 5.3 and added the multi-skill results to Appendix A.3.

2. Implicit skills and the scope of HH (W2, Q2). A skill is included in HH based on whether it is necessary for task completion, not whether it is explicitly mentioned in the query. Thus, implicitly required skills remain eligible for relation-guided expansion, while skills not needed for the task are excluded to avoid irrelevant relation annotations. Importantly, HH only determines the scope of relation annotation and expansion; all gold skills, including those outside HH, remain positive training targets to ensure a fair comparison. We clarified this in Appendix E.1 (Lines 1416–1421) and added examples in Appendix E.3.

3. SRA-Bench comparison (Q3). We extended our multi-skill evaluation by adding Base and Random to the SRA-Bench comparison. SkillComp achieves the highest C@20 among the three methods, outperforming Base by 2.03% and Random by 0.68%, suggesting that relation-guided expansion remains beneficial in this setting. We also added the multi-skill comparisons across all three benchmarks to Appendix A.3 (Table 9).

We would appreciate your feedback on whether our additional experiments and clarifications address your concerns, and we would be happy to discuss any remaining questions.

Reviewer_TgFsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a dense skill retriever with an extra supervision signal: the task query is expanded with the description of a related required skill, which is chosen from LLM-annotated skill relations, and both the plain and the expanded query are trained against the complete required skill set. Inference uses only the plain query. Gains in recall and completeness are reported on SkillRet and on the SkillRouter evaluation set.

Critical strengths

Simple training-only idea with zero inference overhead; this is the kind of change an agent system could adopt without changing its retrieval stack.

Clean ablations and honest reporting of where it does not help.

Critical weaknesses

My main concern is that the actual gain is small and does not come from the idea in the title. Against the authors' own baseline trained the same way without expanded queries, the full method gains about 1.6 points recall and 2.8 points completeness. Picking the appended skill by the annotated relations instead of at random adds only about 0.4 points, and applying expansion only to multi-skill tasks matches the baseline, so most of the improvement comes from the single-skill self-expansion, which has nothing to do with skill relations and does not need the LLM-annotated relation graph. For a practitioner, the message is then "append the skill description to the query during training", which is a small, known kind of data augmentation. Everything is also run with a single seed, so differences of this size are hard to trust, and the secondary benchmark has only 75 queries.

Questions for authors

If you simply add (skill description -> skill) training pairs to the baseline, without any relation annotation, how close do you get to SkillComp? What does the relation graph buy in practice?

Why is the official SkillRet 0.6B checkpoint so much lower here than in its own paper on the same split, and does the same setup reproduce the published 8B number?

One baseline number also needs explaining: the SkillRet 0.6B checkpoint is reported far below what the SkillRet arXiv paper reports on the same split, while the authors' own reproduction matches the published number, which suggests that checkpoint was not evaluated with its intended setup.

3 comments
AuthorsOpenReview ↗

Response to TgFs (1/2)

Q1. If you simply add (skill description -> skill) training pairs to the baseline, without any relation annotation, how close do you get to SkillComp? What does the relation graph buy in practice?

The gains are not simply due to adding self-expansion pairs, but reflect two complementary designs: cross-view training provides a unified learning framework, while the relation graph helps make query expansion more informative.

(1) Following your suggestion, we added w/o Multi-Skill Expansion in Table R2-1, which uses only single-skill self-expansion without relation annotations. This variant performs close to Base, while SkillComp-Embedding-0.6B improves C@20 over it by +1.46% on the SkillRet test set and +5.23% on SkillBench multi-skill queries. The reverse variant, w/o Single-Skill Expansion, also shows limited gains.

Our proposed cross-view training jointly supervises original and expanded queries to retrieve the complete required skill set. Single-skill self-expansion extends this framework to all training queries, while multi-skill expansion helps the encoder learn the relevance between skills. Thus, self-expansion helps maintain a unified training framework rather than acting as mere data augmentation.

(2) The relation graph provides additional benefits beyond random expansion. Within the same training framework, SkillComp-Embedding-0.6B improves C@20 over Random by +0.68% on the SkillRet test set and +4.58% on SkillBench multi-skill queries across three seeds. On these SkillBench queries, which require up to seven skills, SkillComp-Embedding-0.6B also outperforms Base by +3.27% in C@20, showing that the relation graph effectively improves complex multi-skill retrieval.

Table R2-1: Ablation results on the SkillRet test set and SkillBench (C@20, %). Random and SkillComp-Embedding-0.6B report mean ± std across three training seeds.

VariantCross-view CoverageRelation GraphSkillRet (All)SkillRet (Multi)SkillBench (Multi)
Base–✓\checkmark83.4973.6749.02
w/o Single-Skill ExpansionMulti only✓\checkmark83.4773.6745.10
w/o Multi-Skill ExpansionSingle only×\times83.7774.3547.06
Random Appended SkillSingle + Multi×\times84.55 ± 0.2375.74 ± 0.4847.71 ± 1.13
SkillComp-Embedding-0.6BSingle + Multi✓\checkmark85.23 ± 0.0276.36 ± 0.1352.29 ± 2.26
Improvement over Random––+0.68+0.62+4.58
AuthorsOpenReview ↗

Response to TgFs (2/2)

Q2. Why is the official SkillRet 0.6B checkpoint so much lower here than in its own paper on the same split, and does the same setup reproduce the published 8B number?

Yes, the same setup reproduces the published 8B result. We follow the same evaluation protocol as the SkillRet paper (arXiv v1), using 4,997 queries and 6,660 skills, and only the released 0.6B checkpoint deviates from its published numbers. Importantly, SkillComp-Embedding-0.6B still outperforms SkillRet-Embedding-0.6B even when compared with its published results. As shown in Table R2-2, SkillComp-Embedding-0.6B improves R@10/15 by +1.96%/+1.99% and C@10/15 by +3.20%/+3.40% over the published SkillRet-Embedding-0.6B results.

Table R2-2: Comparison with the reported results in the SkillRet paper (arXiv v1).

ModelR@10R@15C@10C@15
SkillRet-Embedding-0.6B (original)85.4288.0975.0979.03
SkillComp-Embedding-0.6B87.3890.0878.2982.43
Improvement+1.96+1.99+3.20+3.40

We also compared our reproduced results with Table 3 of the SkillRet paper (arXiv v1), as shown below. All models except SkillRet-Embedding-0.6B closely match the published results, including SkillRet-Embedding-8B from the same model family. For SkillRet-Embedding-0.6B, we evaluated the official Hugging Face checkpoint five times using the public evaluation code on NVIDIA A800 and H100 GPUs and consistently observed the same discrepancy. To ensure consistent comparisons, we report reproduced results for all baselines rather than mixing them with published numbers. We will clearly distinguish reproduced and originally reported results in the revision to avoid confusion.

Table R2-3: Performance differences between our reproduction and the SkillRet paper (arXiv v1).

ModelΔR@10ΔR@15ΔC@10ΔC@15
BM250.000.000.000.00
bge-small-en-v1.5−0.23−0.01+0.02−0.02
e5-large-v2−0.030.00−0.02−0.02
Qwen3-Embedding-0.6B+0.10+0.11+0.16+0.18
SkillRouter-Embedding-0.6B+0.06−0.02+0.08−0.12
SkillRet-Embedding-8B−0.64−0.68−0.92−1.20
SkillRet-Embedding-0.6B−14.83−14.69−19.16−19.67

Q3. Why does the authors' own reproduction match the published SkillRet-0.6B number, while the released checkpoint performs substantially worse?

Our Base model is independently trained on the same training set as SkillRet-Embedding-0.6B but uses a different contrastive loss. Although its performance is close to the published results of SkillRet-Embedding-0.6B, it is not a reproduction of SkillRet-Embedding-0.6B.

For our evaluation, we used the officially released SkillRet-Embedding-0.6B checkpoint, which consistently produced lower results across five runs on different GPUs, while other baselines, including SkillRet-Embedding-8B, closely matched their published results. We will clarify this distinction and attempt to retrain SkillRet-Embedding-0.6B.


Weakness: Everything is also run with a single seed, so differences of this size are hard to trust, and the secondary benchmark has only 75 queries.

We would also like to clarify these two remaining concerns: first, we extended our evaluation of SkillComp-Embedding-0.6B and Random Appended Skill across three random training seeds (Table R2-1). SkillComp-Embedding-0.6B outperforms Random Appended Skill on both the SkillRet test set and SkillBench, with low variance across runs on the SkillRet test set.

Second, although SkillBench contains only 75 queries, it is a widely used benchmark in the agentic skills domain (over 80 citations), designed to evaluate challenging, realistic scenarios. Its tasks span 55 application domains and require up to seven skills per query, often involving complex workflows and skill collaboration. Moreover, its candidate pool of ~80K skills offers a much larger-scale retrieval setting.

In contrast, the SkillRet test split provides broader evaluation across more queries, but with a smaller skill corpus (6,660 skills) and simpler setups (1–3 required skills). The two benchmarks, therefore, complement each other: SkillBench tests complex multi-skill retrieval from a large skill pool, while the SkillRet test set provides evaluation across more queries.


Thank you for your feedback. We would appreciate hearing whether our additional experiments and clarifications address your concerns.

AuthorsOpenReview ↗

Summary of revisions for Reviewer TgFs

Thank you again for your thoughtful feedback. We have further retrained SkillRet-Embedding-0.6B using its official training setup to better reproduce the published results. We understand that the discussion period can be busy, so we have also briefly summarized our responses and revisions below for your convenience.

1. Source of the improvements and role of relations (Q1). The improvements come primarily from cross-view training rather than self-expansion alone, with relation-guided expansion providing additional gains.

  • Cross-view training. Using only self-expanded (skill description → skill) pairs improves C@20 by just 0.28% over Base, whereas applying cross-view training to both single- and multi-skill tasks yields a 1.74% gain.
  • Relation-guided expansion. Across three training seeds, SkillComp consistently outperforms Random, improving C@20 by 0.68% on SkillRet and 4.58% on SkillBench multi-skill queries, which involve more complex tasks and skill combinations (Table R2-1).

These results suggest that cross-view training helps the encoder learn from expanded queries, while relation-guided expansion provides more informative training views. We revised Section 5.3 to better clarify these contributions of our method.

2. SkillRet-Embedding-0.6B baseline (Q2, Q3). SkillComp still outperforms SkillRet-Embedding-0.6B even after we retrained it. Our evaluations of all other baselines, including SkillRet-Embedding-8B from the same model family, closely match their published results (Table R2-3), so the gap is specific to the released 0.6B checkpoint.

To verify this, we retrained SkillRet-Embedding-0.6B with its official training data and configuration on 4 NVIDIA H100 GPUs and evaluated it under the same protocol (Table R2-4). The retrained model is within 2.26% R@10 and 3.35% C@10 of the published results, compared with gaps of 14.83% and 19.16% for the released checkpoint. Our Base is trained with our own objective and is not a reproduction of SkillRet-Embedding-0.6B. We now report the retrained model in Tables 1 and 2 and clarify the Base setup in Appendix A.4.

Table R2-4: SkillRet-Embedding-0.6B results on the SkillRet test set, comparing the originally reported results (arXiv v1), the released Hugging Face checkpoint, and our retrained model under the same protocol.

ModelR@10R@15C@10C@15
SkillRet-Embedding-0.6B (reported)85.4288.0975.0979.03
SkillRet-Embedding-0.6B (released)70.5973.4055.9359.36
SkillRet-Embedding-0.6B (retrained)83.1686.0871.7476.09
Ours87.3890.0878.2982.43

3. Training stability and SkillBench (W). We extended our experiments by training SkillComp and Random with three different random seeds and found that SkillComp consistently outperforms Random on SkillRet multi-skill queries, improving C@20 by 0.62% with a lower standard deviation across runs. Although SkillBench contains only 75 queries, it is a widely used benchmark with more complex skill combinations than SkillRet, making it a useful complement to SkillRet's larger test set. We added the three-seed results to Appendix A.3 and expanded the description of SkillBench in Appendix A.1.

We would greatly appreciate it if you could let us know whether these additional experiments and clarifications address your concerns, and we are happy to discuss any remaining questions.

Reviewer_GPa2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training method (SkillCOmp) for retrieving the complete set of skills an LLM agent needs for a task, including prerequisites and intermediate capabilities that are weakly expressed in the task description. The authors validate using SkillRet and SkillBench with Qwen3 embedding backbones of 0.6B and 8B parameters. The results show improved recall and complete-set retrieval, particularly for multi-skill tasks.

Critical strengths
  • The proposed method is efficient and scalable. SkillComp uses skill relations and expanded queries only during training; inference remains a single standard dense-retrieval pass over the original query. Thus it adds no iterative query expansion, LLM calls, or graph propagation at test time.

  • The idea is well motivated. SkillComp improves coverage without requiring iterative retrieval, graph traversal, or graph maintenance at deployment.

Critical weaknesses
  • The evaluation provides limited evidence of generalization and downstream utility. SkillBench contains only 75 queries, while the larger cross-domain evaluation shows that performance becomes less reliable as target skills diverge from the training distribution, including degradation on CHAMP.

  • The training data could come from costly, potentially brittle relation annotations. SkillComp assumes access to complete GOLD skill sets and uses an LLM-based, multi-stage process to annotate and validate relations for tens of thousands of training queries.

Questions for authors
  • Could the authors report results over multiple random seeds, including uncertainty or significance tests, for the comparison between relation-guided and random skill selection?

  • Is there evidence that higher Completeness@K produces higher end-to-end agent task-completion rates?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GPa2 (1/2)

Thank you for your valuable feedback. We appreciate your recognition of SkillComp's efficiency, scalability, and motivation. We have addressed each concern below and would appreciate hearing whether our responses address your concerns.


Q1. Could the authors report results over multiple random seeds, including uncertainty or significance tests, for the comparison between relation-guided and random skill selection?

Following your suggestion, we trained SkillComp-Embedding-0.6B and Random using three random seeds and conducted paired statistical tests. SkillComp-Embedding-0.6B consistently outperforms Random, with improvements of +0.53% in R@10 and +0.76% in C@10. All reported improvements are statistically significant (paired t-tests, p < 0.05). SkillComp-Embedding-0.6B also shows low variation across seeds on the SkillRet test set, further supporting the consistency of our results. We will include the multi-seed results and statistical analysis in the revised paper.

Table R1-1: Evaluation results on the SkillRet test set across three training seeds (mean ± std, %), with p-values from two-sided paired t-tests.

ModelR@10C@10
Random86.94 ± 0.0577.56 ± 0.18
SkillComp-Embedding-0.6B87.47 ± 0.0978.32 ± 0.03
Improvement+0.53%+0.76%
p-value0.0190.025

Q2. Is there evidence that higher Completeness@K produces higher end-to-end agent task-completion rates?

Yes. Prior studies provide empirical evidence that higher retrieval completeness is associated with better downstream agent performance [1,2,3].

In skill retrieval, Graph-of-Skills [1] shows a consistent trend across three retrieval variants on SkillsBench: bundle completeness increases from 31.1% to 42.1%, while end-to-end agent reward rises from 25.3% to 34.4%. Similarly, in tool retrieval, ToolQP [2] improves Completeness@10 by +8.8% on ToolRet (averaged across 35 datasets), alongside an improvement of up to +8.3% in Solvable Pass Rate on StableToolBench.

These findings support a positive association between retrieval completeness and downstream agent performance. We will add these references and clarify this claim in the revision.

References

[1] Liu et al. Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills. EMNLP 2026.

[2] Fang & Glass. Beyond Single-Shot: Multi-step Tool Retrieval via Query Planning. ACL 2026.

[3] Shi et al. Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models. ACL 2025.

AuthorsOpenReview ↗

Response to Reviewer GPa2 (2/2)

Weakness1-1. The evaluation provides limited evidence of generalization and downstream utility. SkillBench contains only 75 queries, while the larger cross-domain evaluation shows that performance becomes less reliable as target skills diverge from the training distribution, including degradation on CHAMP.

To evaluate generalization, we evaluated SkillComp-Embedding-0.6B on three benchmarks, where it achieves consistent improvements in retrieval performance. Compared with Base, it improves C@20 by +1.74 on SkillRet, +3.27 on SkillBench multi-skill queries, and +11.28 on SRA-Bench.

Although SkillBench contains only 75 queries, it is a widely used benchmark in the agentic skills domain (over 80 citations). Its tasks span 55 application domains, often involving complex workflows and skill collaboration. It also evaluates retrieval from a much larger pool of ~80K skills than our training data, providing evidence of generalization to a larger skill collection.

Thus, SkillBench and SkillRet provide complementary evidence: SkillRet evaluates retrieval performance across many more queries, while SkillBench tests complex multi-skill retrieval over a ~8× larger skill pool than the SkillRet training corpus.

Table R1-2: Dataset statistics.

Dataset# Queries# SkillsMulti-skill (%)Max. Required Skills
SkillRet (Train)63,25910,12367.23
SkillRet (Test)4,9976,66053.63
SkillBench75~80K68.07

Weakness 1-2. Clarification on performance degradation on CHAMP.

We further analyzed the performance degradation on CHAMP and found that just seven of the 89 gold skills appear in 74% of multi-skill queries. Some of these frequently used skills have functionally similar alternatives outside the gold sets, which are not counted by R@K or C@K, even when they help complete the task.

For example, in one example task (ID: champ_00031), SkillComp-Embedding-0.6B misses the gold skill Rule of Sum but retrieves Combinatorial Counting, which covers both the sum and product rules. The agent completes the task using this alternative skill but fails without any skills.

To further verify the usefulness of these alternative skills, we evaluated task success using the top five skills retrieved by each retriever, following the original SRA-Bench setup with the same agent model (Qwen3-4B). Despite lower C@5 on CHAMP, the agent equipped with skills retrieved by SkillComp-Embedding-0.6B achieves higher task success than with skills retrieved by Qwen3-Embedding-0.6B, by +2.7% overall and +4.6% on multi-skill queries.

The degradation on CHAMP therefore partly reflects limitations of its gold-skill annotations, which do not account for valid alternative skills, rather than weaker generalization. We will include this analysis in the revision.

Table R1-3: Completeness@5 and downstream task success rate (%) on CHAMP

SubsetSkills given to the agentC@5Success rate
All tasks (n=223)No skill–52.5
Oracle (gold skills)100.069.5
Qwen3-Embedding-0.6B31.865.5
SkillComp23.368.2
Multi-skill subset (n=109)No skill–50.5
Oracle (gold skills)100.074.3
Qwen3-Embedding-0.6B15.664.2
SkillComp11.068.8

Weakness2. The training data could come from costly, potentially brittle relation annotations. SkillComp assumes access to complete GOLD skill sets and uses an LLM-based, multi-stage process to annotate and validate relations for tens of thousands of training queries.

Yes, compared with vanilla contrastive learning, our method requires additional relation annotations. When only the required skill sets are available, as in the public SkillRet dataset, we need to further annotate dependencies among these skills. However, this process is relatively lightweight, as it mainly involves binary decisions about whether dependencies exist. In our experiments, annotating ~42K training queries cost around $350, and our manual audit of 200 samples showed 98% accuracy, supporting the low cost and reliability of the annotations.

Moreover, when agent execution traces or plans are available in real-world settings, these relations can be identified without additional LLM annotation. For example, sequential dependencies can be identified when one skill's output is used by another, while parallel relations can be identified when skills independently address different subgoals.

AuthorsOpenReview ↗

Summary of revisions for Reviewer GPa2

Thank you again for your helpful and encouraging feedback. We understand that the discussion period can be busy, so we briefly summarize our responses and revisions below.

1. Multiple seeds (Q1). We extended our experiments by training SkillComp and Random with three different random seeds, and SkillComp consistently outperforms Random across runs with low variance. On SkillRet, SkillComp improves C@10 by 0.76% over Random, and the difference is statistically significant (paired t-test, p < 0.05). We added the three-seed results to Appendix A.3.

2. Completeness and downstream success (Q2). Yes. Previous studies (GoS, ToolQP, and ToolRet) show a positive relationship between retrieval completeness and downstream task success. We added these references to the Introduction (Lines 40–43).

3. Generalization and CHAMP (W1). SkillComp outperforms Base on SkillRet, SkillBench, and SRA-Bench, providing further evidence of its effectiveness across different benchmark settings. The performance drop on CHAMP may stem from SkillComp retrieving valid alternatives to the gold skills, which receive no credit under retrieval metrics. Our additional downstream task evaluation provides further support for this explanation. We added this analysis to Appendix A.2.

4. Annotation cost and reliability (W2). Although relation annotation requires an additional LLM pass, the cost remains modest and the annotations are reliable. Annotating all 42K multi-skill training queries costs about $350, and a manual review of 200 samples shows 98% accuracy. We added these details to Appendix E.2.

We would greatly appreciate it if you could let us know whether these additional experiments and clarifications address your concerns, and we are happy to discuss any remaining questions.

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive feedback. We appreciate their recognition that SkillComp is well motivated (GPa2), addresses the relevant problem of complete skill-set retrieval (iwYk), and is simple and practical, with relations used only during training and no inference overhead (GPa2, TgFs, and iwYk). Below, we summarize the main revisions:

  1. Clarification of the role of relations (Reviewers TgFs and iwYk). We clarified that SkillComp internalizes relations among required skills through both of its components, rather than only through relation-guided expansion:

    • Cross-view training learns which skills are jointly required. It supervises the plain and expanded queries with the complete required skill set. Because each expanded query scores every required skill in the context of the appended one, the encoder learns the relevance between jointly required skills, rather than relating each required skill only to the plain query, and transfers it to the plain query (Section 4.2). Even with randomly appended skills, this improves SkillRet C@20 over Base by 1.16 points (Table 3). When a required skill is appended at test time, SkillComp recovers 17.8% of the skills it had missed while dropping only 2.4% of those it had found (Figure 4).
    • Relation-guided expansion further refines these relations. Annotated dependencies among required skills guide which skills are appended, providing targeted context for skills whose relevance is weakly expressed in the plain query. Refining the relations through these finer-grained dependencies further improves C@20 over random selection by 0.58 points on SkillRet and 6.67 points on SkillBench (Section 4.1; Tables 3 and 11).

    We added the single-skill-only variant to Table 3 and revised the Introduction and Ablation Study accordingly.

  2. Training stability and generalization (Reviewers GPa2, TgFs, and iwYk).

    • Adding three-seed training experiments and extending comparisons with Random Appended Skill across the multi-skill subsets of SkillRet, SkillBench, and SRA-Bench (Appendix A.3, Table 9).
    • Clarifying the advantage of SkillBench's evaluation setting (Appendix A.1, Table 5) and analyzing required-skill ranks in addition to coarse query-level results (Appendix A.3, Table 10).
  3. Baseline and downstream utility (Reviewers GPa2 and TgFs).

    • Retraining SkillRet-Embedding-0.6B using its official setup, updating Tables 1–2, and clarifying its distinction from our Base (Appendix A.4).
    • Adding a CHAMP agent task-success evaluation to assess the usefulness of retrieved skills (Appendix A.2, Table 8).
  4. Annotation scope and cost (Reviewers GPa2 and iwYk).

    • Clarifying how implicitly required skills can be retained for relation annotation, with illustrative cases (Appendices E.1 and E.3).
    • Reporting annotation cost and manual-audit accuracy (Appendix E.2).

We believe these revisions clarify that SkillComp internalizes relations among required skills during training, and strengthen the evidence that this improves complete skill-set retrieval while using only the plain query at inference. We hope we have addressed the reviewers’ major concerns. These changes have been incorporated into the revised manuscript.

We thank the reviewers again for their insightful feedback, and the AC, SAC, and PC for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.