acceptodds
Under review as a conference paper at ICLR 2027

SkillECTOR: Adaptive Contextual Lean Skill Selector Framework

Abstract

Reusable skills allow language-model agents to acquire specialized business or user-aligned behaviors and capabilities without modifying their underlying models. But, the real future of intelligent agents lies beyond just more skills; it lies in knowing exactly which skills to deploy, when, and why. While both language models and agent harnesses have become increasingly skill-aware, existing skill orchestrators remain mostly na\"ive and context-insensitive, with limited adaptation of retrieval breadth and ranking strategy to the task at hand. Such rigid orchestration will become progressively ineffective and inefficient (costly) as skill repositories expand, diversify, and accumulate overlapping capabilities. We present *SkillECTOR*, an evolving, adaptive, lean, and model-independent policy that learns how to construct a focused skill set for a given task/query as well as environment/user context with **0 LLM invocation** in its online inference path. Across multiple widely-used benchmarks ( s.a. SkillsBench, AgentSkillOS, and Dataverse) our framework significantly () improves complete-skill coverage over LLM-based selectors. We observe significant improvement in task success () with substantially lower token usage (%) and lower inference latency (%) when *SkillECTOR* is integrated with task agents (powered by advanced LLMs). Impacts become more pronounced as skill repositories grow larger. These findings establish adaptive skill orchestration as a compact control problem and offer a scalable yet more robust alternative to placing an additional language model before every agent task.

Primary area:
generative models
Keywords:
Agent Skills, Context, Agents, LLM, Adaptive Selection

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LjooOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SkillECTOR, that instead of using an LLM to pick which skills an agent should load, it uses a tiny RL policy that tunes a classic retrieval pipeline per query, with zero LLM calls at inference time. In this way, the controller’s purpose is adaptivity: deciding, per query, how to configure retrieval so the model gets the right skills without loading too many or too few.

Critical strengths
  • The paper's proposed method is novel. The authors use the policy output configuration parameters rather than skill identities makes the decision space independent of catalog size. New skills only require index updates, not retraining a head or resizing an output layer.

  • The paper also clearly address and acknowledge the loss of the model. The paper acknowledges that strong LLM selectors win on the small 16-skill Dataverse catalog, and that GPT-6 Astra beats it on precision and F1 in the combined setting. That candor makes the positive results more believable and scopes the claim sensibly.

Critical weaknesses
  • The paper misses an important ablation study. The paper never compares the learned RL controller against a fixed-K, fixed-weight (or simply tuned) fusion of the same three rankers.

  • The Train/test set separation is unclear. The learning curves are plotted over epochs on the same task sets (94 SkillsBench tasks, 30 AgentSkillOS tasks, 183 Dataverse tasks) used for reporting results, and the text does not describe a held-out split.

Questions for authors
  • Are the policies trained and evaluated on the same tasks?

  • Can a policy trained on one catalog transfer to another, or to the same catalog after skills are added without retraining?

3 comments
AuthorsOpenReview ↗

Response to review comments

We thank the reviewer for the positive assessment of the method's novelty, catalog-size-independent design, and candid reporting.

W1: Comparison against a fixed-K, fixed-weight fusion of the same rankers

We agree this is an important baseline. We evaluated every combination of BM25, Dense skill, and Instruction dense using fixed RRF (weight 1.0, k=60) at Top-K=3/5/10 across all benchmarks.

Setup

SettingAgentSkillOSSkillsBenchDataverse
Skills100022516
Top-K3 / 5 / 103 / 5 / 103 / 5 / 10
Family weight1.0 each1.0 each1.0 each
RRF kk60 each60 each60 each

Overall Summary: Best Static Configuration vs. SkillECTOR

BenchmarkTop-KBest static familiesStatic TSARSkillECTOR TSAR
AgentSkillOS3Dense skill + Instruction dense26.67%
AgentSkillOS5Dense skill + Instruction dense40.00%50.00%
AgentSkillOS10BM25 + Dense skill + Instruction dense56.67%
SkillsBench3BM25 + Instruction dense47.87%
SkillsBench5BM25 + Dense skill + Instruction dense63.83%67.02%
SkillsBench10BM25 + Dense skill + Instruction dense78.72%
Dataverse3Instruction dense40.44%
Dataverse5Instruction dense58.47%77.05%
Dataverse10Instruction dense (tied with all three)87.98%

Static retrieval exceeds SkillECTOR only with fixed K=10K=10, increasing context for every query. SkillECTOR adapts retrieval per query.

Detailed Static LLM-Free Retriever Results

Bold indicates the best static configuration at each depth.

AgentSkillOS
Static ranker configurationK=3K=5K=10
BM2513.33%13.33%13.33%
Dense skill16.67%33.33%40.00%
Instruction dense23.33%30.00%33.33%
BM25 + Dense skill16.67%20.00%43.33%
BM25 + Instruction dense13.33%16.67%36.67%
Dense skill + Instruction dense26.67%40.00%43.33%
BM25 + Dense skill + Instruction dense16.67%26.67%56.67%
SkillsBench
Static ranker configurationK=3K=5K=10
BM2542.55%50.00%62.77%
Dense skill38.30%50.00%71.28%
Instruction dense45.74%56.38%72.34%
BM25 + Dense skill45.74%58.51%73.40%
BM25 + Instruction dense47.87%57.45%76.60%
Dense skill + Instruction dense43.62%60.64%77.66%
BM25 + Dense skill + Instruction dense46.81%63.83%78.72%
Dataverse
Static ranker configurationK=3K=5K=10
BM2513.66%22.40%67.21%
Dense skill28.96%42.62%78.69%
Instruction dense40.44%58.47%87.98%
BM25 + Dense skill23.50%39.89%79.23%
BM25 + Instruction dense30.05%53.01%84.15%
Dense skill + Instruction dense35.52%53.01%85.25%
BM25 + Dense skill + Instruction dense34.97%56.83%87.98%

Three patterns emerge: (1) no static configuration is consistently optimal; (2) adding rankers can help or hurt, e.g., AgentSkillOS at K=5K=5: 40.00%→26.67% after adding BM25; (3) larger fixed KK improves TSAR but increases context. This motivates adaptive depth and fusion.

W2 / Q1: Are the policies trained and evaluated on the same tasks?

Yes. Reported policies are trained and evaluated on the same small task sets in SkillsBench and AgentSkillOS, so we report final-checkpoint performance after convergence, a valid continual-RL protocol. We will state this explicitly. We separately evaluate cross-catalog generalization, with training and testing on different catalogs and hence zero leakage.

Q2: Can a policy transfer to another catalog, or to the same catalog after skills are added, without retraining?

Yes. We trained on one catalog (e.g., AgentSkillOS) and transferred the checkpoint to another (e.g., SkillsBench) without retraining from scratch. It reaches optimal performance and beats baselines within ~5 adaptation steps (Appendix H, Figure 9).

This follows from SkillECTOR's design: the policy predicts subset-construction parameters, retrieval depth KK and rank-fusion weights WW, rather than skill identities. It is therefore transferable across catalogs.

AuthorsOpenReview ↗

Additional comparison with SkillRouter as highlighted by one of the reviews

SkillRouter vs SkillECTOR

[Appendix K - we will try to fit this into main paper during camera ready stage].

Skillector selects an adaptive set of skills per task (about 2.9 on the SkillRouter Hard pool) instead of a fixed top-10. That set is 46.7% precise, while any top-10 list on this benchmark is capped at 26.7%. SkillRouter's own agent study shows that top-10 adds almost no task success over top-1 (27.78% vs 27.56%), so a small, well-chosen set matters more than a long one. Skillector does this with about 44M parameters on a CPU laptop: training took 2.85 hours and needed no LLM-generated training data or fine-tuning. SkillRouter's 1.2B pipeline needed 38K synthetic training queries, fine-tuning on a 96 GB GPU, and about 19 GB of GPU memory to serve. Skillector's ranking sources are pluggable, and its per-query choices of K and fusion weights are explainable.

Results

SkillECTOR vs SkillRouter on AgentSkillOS (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)4.43 average101010 (Hit@1 uses 1)
TSAR / FC at that K50.00%53.33%56.67%36.67%
… single-skill100%100%100%80.00%
… multi-skill25.00%30.00%35.00%15.00%
Recall at that K70.56%75.00%75.56%51.39%
Precision at that K39.99%13.33%13.33%8.00%
Hit@146.67%46.67%33.33%30.00%
MRR0.593 (within predicted skills)0.593 (@10)0.493 (@10)0.432 (@10)

SkillECTOR vs SkillRouter on Dataverse (DV) (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)3.90 average101010 (Hit@1 uses 1)
TSAR / FC at that K74.32%90.16%87.98%65.03%
… single-skill (64 tasks)93.75%93.75%96.88%81.25%
… multi-skill (119 tasks)63.87%88.24%83.19%56.30%
Recall at that K88.41%95.65%96.16%81.32%
Precision at that K62.12%23.33%23.22%19.07%
Hit@159.02%59.02%54.10%61.75%
MRR0.742 (within predicted skills)0.745 (@10)0.708 (@10)0.715 (@10)

Our SkillECTOR framework significantly beats SkillRouter across two well-known benchmarks at ZERO GPU cost. We can train on a CPU. Our TSAR and SkillRouter's FC are equivalent metrics. We could not run the SkillRouter paper's modified benchmark with 88K+ skills in the time available. Building the initial Instruction-Dense index takes setup time. Instead, we tested both SkillRouter and SkillECTOR on our existing benchmarks.

SkillRouter's claim of generalizing to any benchmark or skill set is proven false (relative to SkillECTOR), despite its static retrieval depth of K=10 and our much lower predicted K.

AuthorsOpenReview ↗

Request to reviewer

We believe we have been able to satisfactorily address all the concerns highlighted in your review and beyond. If there are any other concerns please let us know. Otherwise, we request you to kindly consider revising the score. Thank you again for your valuable time and review.

Reviewer_Uwy8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SkillECTOR combines BM25, dense retrieval, and embeddings of synthetic skill-specific requests. A lightweight controller chooses retrieval depth and rank-fusion parameters, with rewards derived from offline LLM judgments. Selection requires no online LLM call. Experiments compare selection quality and cost across three skill ecosystems, followed by an 81-task deployment study comparing selective loading with full-catalog exposure.


Thank you for the detailed rebuttal and substantial additional experiments. I appreciate the new comparisons with static retrieval baselines and SkillRouter, as well as the additional downstream evaluations, which meaningfully strengthen the paper. While some concerns remain regarding held-out evaluation and budget-matched comparisons, I acknowledge these improvements and have raised my score accordingly.

Critical strengths
  1. Choosing retrieval depth and fusion parameters instead of skill identities gives the controller a fixed action space as the catalog grows. Keeping the LLM judge offline removes the generation call before every task.

  2. The deployment study measures actual task completion and resource use. With 1,209 skills, Table 5 reports a reduction from 76.66M to 53.56M tokens and from 17.39 to 12.79 minutes per task.

Critical weaknesses
  1. It is unclear whether the results measure generalization to unseen tasks. I could not find a train/validation/test split or a description of which tasks provide controller rewards. Learning curves are shown on the evaluation benchmarks, and Appendix E reports the best-F1 result along a training trajectory without specifying a separate checkpoint-selection set. Table 9 also changes reward weights across benchmarks without stating how or on which data the values were selected. This matters most for AgentSkillOS, which reuses the same 30 tasks across all three catalog sizes. The comparison with zero-shot selectors needs a clearly held-out evaluation.

  2. The benefit of the adaptive controller over ordinary retrieval is not established. A fixed weighted-RRF ranker with tuned K would share the main latency advantage over LLM selectors, but is not evaluated. Table 6 removes retrieval modes but keeps the controller, so it does not isolate the controller's contribution. The quality results show a coverage-precision trade-off: on AgentSkillOS-1000, TSAR exceeds GPT-6 Astra's (50.0% vs. 43.3%), while F1 is 0.460, below both GPT-5.4-nano (0.479) and Astra (0.637). On the combined catalog, F1 is 0.587 versus Astra's 0.798. The introduction's claim of consistent outperformance with up to 27% higher F1 should be qualified to distinguish coverage gains from these F1 losses. SkillECTOR also loads nearly twice Astra's skill context there (14.92K vs. 7.94K tokens), so comparisons should match coverage or context size.

Questions for authors
  1. Please specify the training, validation, and test tasks, reward sources, and checkpoint-selection procedure. Were the deployment tasks held out from controller training and tuning?

  2. How does SkillECTOR compare with fixed RRF and tuned K? What happens when only K or only fusion parameters are learned?

  3. In Table 5, the success-rate increase from 54.32% to 58.02% amounts to three additional successes among 81 tasks; at 209 skills, success rates are identical. Can you provide repeated-run or paired uncertainty estimates for the claimed task-success improvement?

3 comments
AuthorsOpenReview ↗

Response to review comments

We thank the reviewer for the comments.

Generalization:

We trained on AgentSkillOS and transferred the policy (not frozen, allowing adaptation) to SkillsBench. It beats baselines within 5 steps (Appendix H - Figure 9). The policy predicts subset parameters (K and W), not the subset.

W2 / Q2: Comparison with fixed RRF and tuned K:

Every BM25, Dense skill, and Instruction dense combination uses fixed RRF, weight 1.0, k=60, and Top-K=3/5/10.

Setup

SettingAgentSkillOSSkillsBenchDataverse
Skills100022516
Top-K3 / 5 / 103 / 5 / 103 / 5 / 10
Family weight1.0 each1.0 each1.0 each
RRF kk60 each60 each60 each

Overall Summary: Best Static Configuration vs. SkillECTOR

BenchmarkTop-KBest static familiesStatic TSARSkillECTOR TSAR
AgentSkillOS3Dense skill + Instruction dense26.67%
AgentSkillOS5Dense skill + Instruction dense40.00%50.00%
AgentSkillOS10BM25 + Dense skill + Instruction dense56.67%
SkillsBench3BM25 + Instruction dense47.87%
SkillsBench5BM25 + Dense skill + Instruction dense63.83%67.02%
SkillsBench10BM25 + Dense skill + Instruction dense78.72%
Dataverse3Instruction dense40.44%
Dataverse5Instruction dense58.47%77.05%
Dataverse10Instruction dense (tied with all three)87.98%

Static retrieval exceeds SkillECTOR only at fixed K=10K=10, increasing context. SkillECTOR adapts retrieval per query.

Detailed Static LLM-Free Retriever Results

All combinations at K∈{3,5,10}K\in\{3,5,10\}. Columns give TSAR at K=3/5/10.

RankerAgentSkillOSSkillsBenchDataverse
BM2513.33/13.33/13.3342.55/50.00/62.7713.66/22.40/67.21
Dense skill16.67/33.33/40.0038.30/50.00/71.2828.96/42.62/78.69
Instruction dense23.33/30.00/33.3345.74/56.38/72.3440.44/58.47/87.98
BM25 + Dense16.67/20.00/43.3345.74/58.51/73.4023.50/39.89/79.23
BM25 + Instr.13.33/16.67/36.6747.87/57.45/76.6030.05/53.01/84.15
Dense + Instr.26.67/40.00/43.3343.62/60.64/77.6635.52/53.01/85.25
All three16.67/26.67/56.6746.81/63.83/78.7234.97/56.83/87.98

Three patterns emerge: (1) no static configuration is always optimal; (2) adding rankers can help or hurt (AgentSkillOS K=5: 40.00% to 26.67% with BM25); (3) larger fixed K improves TSAR but increases context. This motivates adaptive depth and fusion.

Is reported advantage marginal?

No. With exact ground-truth skills, the strongest models reach ~69% TSAR on SkillsBench Leaderboard (https://www.skillsbench.ai/leaderboard); three additional solved tasks are meaningful. The margin is negative only on Dataverse, against high-cost GPT 6 Astra. Small catalogs let LLMs inspect nearly everything; this becomes costly at scale.

Visible advantage with other task models We tested SOTA OPUS 5.5 with agent memory, giving the agent more opportunity to infer skills. SkillECTOR still helps (Appendix J).

ExperimentTSRTotal Tool CallsTotal TokensTokens / Task P95Tokens / Task P99
No-skill41.38%3,436.50266,596,490.489,074,660.4012,975,703.22
Native-120957.47%3,807.77345,690,104.0812,303,870.3019,948,503.46
SkillECTOR-120962.07%3,920.06324,481,096.7811,208,019.4018,204,337.48

Benefit of the Adaptive Controller

Table 6 is a retrieval-mode ablation, not a controller ablation, and we will clarify this. Fixed-ranker/K results isolate the controller: no static configuration is always optimal; matching/exceeding SkillECTOR generally requires K=10, increasing context. SkillECTOR selects K and fusion per query.

We will qualify the Introduction's F1 claim: SkillECTOR is not designed to maximize F1 in isolation, but coverage for downstream completion. On AgentSkillOS-1000 it beats Astra despite lower F1 (50.0% vs. 43.3% TSAR; 0.460 vs. 0.637 F1), with the same pattern on the combined catalog. Loaded-skill context (14.92K vs. 7.94K) omits Astra's 90.86K selector tokens/task; SkillECTOR uses zero online LLM-selection tokens. The relevant comparison is the end-to-end coverage–precision–context trade-off.

AuthorsOpenReview ↗

Additional comparison with SkillRouter as highlighted by one of the reviews

SkillRouter vs SkillECTOR

[Appendix K - we will try to fit this into main paper during camera ready stage].

Skillector selects an adaptive set of skills per task (about 2.9 on the SkillRouter Hard pool) instead of a fixed top-10. That set is 46.7% precise, while any top-10 list on this benchmark is capped at 26.7%. SkillRouter's own agent study shows that top-10 adds almost no task success over top-1 (27.78% vs 27.56%), so a small, well-chosen set matters more than a long one. Skillector does this with about 44M parameters on a CPU laptop: training took 2.85 hours and needed no LLM-generated training data or fine-tuning. SkillRouter's 1.2B pipeline needed 38K synthetic training queries, fine-tuning on a 96 GB GPU, and about 19 GB of GPU memory to serve. Skillector's ranking sources are pluggable, and its per-query choices of K and fusion weights are explainable.

Results

SkillECTOR vs SkillRouter on AgentSkillOS (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)4.43 average101010 (Hit@1 uses 1)
TSAR / FC at that K50.00%53.33%56.67%36.67%
… single-skill100%100%100%80.00%
… multi-skill25.00%30.00%35.00%15.00%
Recall at that K70.56%75.00%75.56%51.39%
Precision at that K39.99%13.33%13.33%8.00%
Hit@146.67%46.67%33.33%30.00%
MRR0.593 (within predicted skills)0.593 (@10)0.493 (@10)0.432 (@10)

SkillECTOR vs SkillRouter on Dataverse (DV) (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)3.90 average101010 (Hit@1 uses 1)
TSAR / FC at that K74.32%90.16%87.98%65.03%
… single-skill (64 tasks)93.75%93.75%96.88%81.25%
… multi-skill (119 tasks)63.87%88.24%83.19%56.30%
Recall at that K88.41%95.65%96.16%81.32%
Precision at that K62.12%23.33%23.22%19.07%
Hit@159.02%59.02%54.10%61.75%
MRR0.742 (within predicted skills)0.745 (@10)0.708 (@10)0.715 (@10)

Our SkillECTOR framework significantly beats SkillRouter across two well-known benchmarks at ZERO GPU cost. We can train on a CPU. Our TSAR and SkillRouter's FC are equivalent metrics. We could not run the SkillRouter paper's modified benchmark with 88K+ skills in the time available. Building the initial Instruction-Dense index takes setup time. Instead, we tested both SkillRouter and SkillECTOR on our existing benchmarks.

SkillRouter's claim of generalizing to any benchmark or skill set is proven false (relative to SkillECTOR), despite its static retrieval depth of K=10 and our much lower predicted K.

AuthorsOpenReview ↗

Request to reviewer

We believe we have been able to satisfactorily address all the concerns highlighted in your review and beyond. If there are any other concerns please let us know. Otherwise, we request you to kindly consider revising the score. Thank you again for your valuable time and review.

Reviewer_W7vVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SkillECTOR, an LLM-free selector that can retrieve relevant skills from large skill libraries. Three ranking models are used here: BM25, Dense and Instruction-dense. These ranking models contribute to scoring the related skill sets. On top of that, a light Rainbow DQN policy is trained to output retrieval depth K, merging weights and RRF. Skills are retrieved based on the output of this DQN policy and the scores given by the above three ranking models. Experiments on SkillsBench, AgentSkillOS demonstrate the strong performance of SkillECTOR, beating GPT-5.4-nano, GPT-5.6-sol and GPT-6 Astra. Further results on Github Copilot harness also verify SkillECTOR’s effectiveness.

Critical strengths
  1. Important research question. With skill library scaling in the harness, how to retrieve related skills efficiently and effectively is an important question.
  2. Superior performance compared with LLMs. SkillECTOR shows its stronger performance on multiple benchmarks: SkillsBench, AgentSkillOS than other LLM baselines.
  3. Efficient retrieving. Without any LLMs being involved in the retrieving process, SkillECTOR uses BM25, Dense and Instruction-dense ranking models and a trained DQN to do the whole process. Therefore, there’s 0 LLM invocation and efficient.
Critical weaknesses
  1. Poor writing of math formula / figure / words / tables. This includes the abuse of annotation in Line 157, the writing of loss function in Line 156 and unreadable Figure 1 and Figure 2. Other typos or words like ‘qlty’ in the caption of Table 2 is hard to understand. Table 6 has two duplicated lines.
  2. Poor introduction. The introduction is largely redundant and does not concisely explain the motivation behind this work.
  3. Poor novelty. The combination of BM25, Dense, Instruction-Dense as rankers and a Rainbow DQN is not novel to the research community. This works is more like a combination of existing works instead of introducing a new algorithm or bringing any novel insights to other people.
  4. Unfair comparison. The paper’s proposed SkillECTOR is basically a trained selector. However, its compared baselines are zero-trained LLM-based selector. No other trained baselines like SkillRouter, SkillRet are compared with SkillECTOR.
  5. Marginal improvements over naive baseline. In table 5, SkillECTOR-1209 surpasses its naive counterpart only by 4%, which means SkillECTOR is better on only 3 more tasks. The improvements are marginal and may even subject to the selection of random seed in program.
  6. Unclear experimental setting. Information like which tasks are used to train the model is missing in this paper.
Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to review comments

We thank the reviewer for the constructive feedback.

Typos and Errors in Formulas

We corrected the identified typos/formula errors and reviewed the relevant sections for consistency.

Introduction and Motivation

Our motivation is deploying large, evolving skill corpora: selection must remain effective as catalogs grow under latency, cost, and context constraints. We revised the Introduction accordingly.

Novelty and Contribution

We respectfully disagree that novelty requires a new foundation model or retrieval primitive. SkillECTOR contributes at the system and decision-policy level: deciding how retrieval operates per query rather than using fixed configurations or another LLM in the selector's hot path.

SkillECTOR jointly adapts retrieval depth and rank-fusion configuration from query context while avoiding runtime LLM selection. Its novelty is adaptive skill selection under practical agent-serving constraints, evaluated across heterogeneous catalogs and task-agent settings. We revised the manuscript to clarify this contribution.

Evaluation and Baselines

We evaluated alternatives relevant to a lean, adaptive selector.

SkillRouter

We agree SkillRouter is relevant. However, its >1B-parameter model increases training/maintenance cost as catalogs evolve, contrary to our lean-selector objective. A pretrained selector also faces distribution shift when enterprise skills/APIs/vocabulary differ from its training distribution. We are running additional SkillRouter experiments.

SkillRet

SkillRet is a benchmark for skill retrieval and related capabilities, not a skill selector; we therefore do not consider it a directly comparable selector baseline and clarified this distinction.

System-One Decision Models

We also evaluated lightweight System-One models, specifically Jev 1.13. Jev marginally leads only on AgentSkillOS, while SkillECTOR substantially leads on Dataverse and SkillsBench. Serving metrics show substantial token, call, latency, and cost overheads on large catalogs (Appendix I).

DatasetSource armJev TSARSkillECTOR TSARToken P90 [2]Token P95 [2]Token P99 [2]Combined avg calls/taskCombined avg cost/taskCombined total costCombined avg time/task
AgentSkillOS [1]single53.33%50.00%108,128108,171108,4641,001.00$0.036715$1.10142877.481 s
AgentSkillOS [1]batch5053.33%50.00%108,229108,262108,55829.10$0.017174$0.5152504.755 s
Dataversesingle62.30%77.05%2,8482,8912,95617.00$0.000592$0.1084561.678 s
Dataversebatch5062.30%77.05%2,9462,9873,0522.00$0.000364$0.0667020.649 s
SkillsBenchsingle59.57%67.02%309,201323,664603,678226.00$0.009548$0.8975338.518 s
SkillsBenchbatch5057.45%67.02%89,67690,187102,4067.19$0.003627$0.3409050.923 s

Jev leads AgentSkillOS 53.33% vs. 50.00%, but SkillECTOR leads Dataverse 77.05% vs. 62.30% and SkillsBench 67.02% vs. 59.57%/57.45%. On AgentSkillOS, Jev single requires 1,001 calls/task and ~108K P90 tokens; batching cuts calls to 29.10 but not token volume. Thus, TSAR and scalability both matter.

Are the Reported Improvements Marginal?

We do not believe so. Even with exact ground-truth skills, the strongest foundation models achieve ~69% TSAR on SkillsBench. Against this ceiling, three additional solved tasks are meaningful.

SkillECTOR targets the performance-efficiency trade-off, not TSAR alone. On Dataverse it trails the strongest LLM baseline, GPT-6 Astra, an extremely high-cost model. LLMs can inspect much of a small catalog, but this becomes costly and less scalable as catalogs grow.

Visible advantage with other task models: We also changed the GHCP model to parallel SOTA model OPUS 5.5 and enabled agent memory, giving the agent more opportunity to identify skills itself. SkillECTOR still helps (Appendix J).

ExperimentTSRTotal Tool CallsTotal TokensTokens / Task P95Tokens / Task P99
No-skill41.38%3,436.50266,596,490.489,074,660.4012,975,703.22
Native-120957.47%3,807.77345,690,104.0812,303,870.3019,948,503.46
SkillECTOR-120962.07%3,920.06324,481,096.7811,208,019.4018,204,337.48

Clarity and Reproducibility of the Experiments

AgentSkillOS and SkillsBench tasks are available through their benchmark repositories. Dataverse uses synthetic tasks; we will release its skills/tasks upon acceptance. We clarified the experimental description and welcome pointers to any remaining unclear details.

Thanks for your response. For the errors in formulas, there're still some ambiguities existing. For example, there should be a clear math definition of what is 'Task Success' in Line 156. Some of these modifications are marginal, but are helpful to improve the rigor of this paper. Besides, the Figure 1 and Figure 2 should also be refined to make it more human-readable.

All other concerns have been addressed. I therefore decide to raise my score to 2.

AuthorsOpenReview ↗

Additional comparison with SkillRouter

SkillRouter vs SkillECTOR

[Appendix K - we will try to fit this into main paper during camera ready stage].

Skillector selects an adaptive set of skills per task (about 2.9 on the SkillRouter Hard pool) instead of a fixed top-10. That set is 46.7% precise, while any top-10 list on this benchmark is capped at 26.7%. SkillRouter's own agent study shows that top-10 adds almost no task success over top-1 (27.78% vs 27.56%), so a small, well-chosen set matters more than a long one. Skillector does this with about 44M parameters on a CPU laptop: training took 2.85 hours and needed no LLM-generated training data or fine-tuning. SkillRouter's 1.2B pipeline needed 38K synthetic training queries, fine-tuning on a 96 GB GPU, and about 19 GB of GPU memory to serve. Skillector's ranking sources are pluggable, and its per-query choices of K and fusion weights are explainable.

Results

SkillECTOR vs SkillRouter on AgentSkillOS (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)4.43 average101010 (Hit@1 uses 1)
TSAR / FC at that K50.00%53.33%56.67%36.67%
… single-skill100%100%100%80.00%
… multi-skill25.00%30.00%35.00%15.00%
Recall at that K70.56%75.00%75.56%51.39%
Precision at that K39.99%13.33%13.33%8.00%
Hit@146.67%46.67%33.33%30.00%
MRR0.593 (within predicted skills)0.593 (@10)0.493 (@10)0.432 (@10)

SkillECTOR vs SkillRouter on Dataverse (DV) (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)3.90 average101010 (Hit@1 uses 1)
TSAR / FC at that K74.32%90.16%87.98%65.03%
… single-skill (64 tasks)93.75%93.75%96.88%81.25%
… multi-skill (119 tasks)63.87%88.24%83.19%56.30%
Recall at that K88.41%95.65%96.16%81.32%
Precision at that K62.12%23.33%23.22%19.07%
Hit@159.02%59.02%54.10%61.75%
MRR0.742 (within predicted skills)0.745 (@10)0.708 (@10)0.715 (@10)

Our SkillECTOR framework significantly beats SkillRouter across two well-known benchmarks at ZERO GPU cost. We can train on a CPU. Our TSAR and SkillRouter's FC are equivalent metrics. We could not run the SkillRouter paper's modified benchmark with 88K+ skills in the time available. Building the initial Instruction-Dense index takes setup time. Instead, we tested both SkillRouter and SkillECTOR on our existing benchmarks.

SkillRouter's claim of generalizing to any benchmark or skill set is proven false (relative to SkillECTOR), despite its static retrieval depth of K=10 and our much lower predicted K.

Reviewer_kXgcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how an agent harness should decide which skills, and how many, to expose to an agent as skill libraries grow. It argues that loading the full catalog causes context bloat, while LLM-based selectors add a model call, tokens and latency before every task. The proposed SkillECTOR selects skills without any LLM call at inference time: ● Three rankers each rank the full catalog. ● A Rainbow-DQN controller takes the query embedding and ranking statistics as state, and predicts the subset size K, per-ranker fusion weights and RRF constants instead of skill identities. Weighted RRF then returns the top-K skills. ● The reward compares the selected subset with required skills predicted by an LLM judge, combining selection errors (F1, precision, recall, rank fidelity) with errors in K and in the fusion weights.

Critical strengths

● A practical problem. As skill libraries grow, deciding what to expose to the agent affects both task success and cost. Avoiding an extra LLM call before every task is a relevant goal for deployed agents. ● An identity-independent action space. Predicting the subset size and fusion parameters instead of skill identities is a reasonable design: the output space does not grow with the catalog. ● Cost is evaluated together with quality. The paper reports selection tokens, loaded-skill tokens and latency, and includes an end-to-end study in a real agent harness with No-skill and Oracle bounds.

Critical weaknesses
  1. Generalization to unseen tasks is not evaluated. The paper does not describe a split between training and evaluation tasks and reports no results on held-out tasks or catalogs. The controller is trained on each benchmark with benchmark-specific reward weights, while the LLM selectors are zero-shot. Without such results, the experiments do not show that the learned selector works on unseen tasks, which is what a deployed selector must do.
  2. The central claim, that a learned controller is needed, is not tested against the right alternatives. The paper motivates SkillECTOR by criticizing retrievers that use a fixed depth and a fixed ranking strategy, but never compares with such retrievers. The selection experiments compare only with zero-shot LLM selectors, and the end-to-end study compares only with Native loading of the full catalog. No experiment places SkillECTOR, an LLM selector and an LLM-free retriever side by side. As a result, the paper shows neither that the RL controller improves over the same rankers with a fixed or simply tuned configuration, nor that SkillECTOR can replace an LLM selector in an agent.
  3. The reported advantage is mainly a trade of precision for coverage, and the claims overstate it. SkillECTOR's higher TSAR comes with lower precision and F1 than the stronger LLM selectors in most settings and with more loaded skill tokens. TSAR is a coverage metric that favors larger subsets, and the reward weights are recall-oriented on the benchmarks where SkillECTOR leads. The margins are small or negative on several settings (e.g., 1–2 of 30 tasks on AgentSkillOS-50/1,000, and negative on Dataverse), and the end-to-end gain is 3 of 81 tasks in a single run, observed only at 1,209 skills.
  4. Key properties claimed for the framework are not demonstrated. The abstract describes SkillECTOR as evolving, context-aware and lean. None of these is supported. No experiment evaluates new skills added after training without retraining. The state contains only the query embedding and ranking statistics, with no environment, user or execution context.
Questions for authors
  1. Can you compare with LLM-free baselines: standard dense embedding retrieval with a fixed top-K , weighted RRF of the same rankers with a fixed K tuned on training tasks, each single ranker, a threshold-based adaptive K?
  2. The ranking of weighted RRF depends only on the relative weights, but E_W penalizes their absolute values. When no required skill is missed, all targets equal 1, which pushes every weight to 0.8, i.e., unweighted RRF. Is this intended?
  3. How are the reward weights in Table 9 chosen, and why do they differ across benchmarks? F1 already combines precision and recall; what is the role of including all three terms?
  4. How is TSAR defined in Table 5? Oracle, which exposes exactly the ground-truth skills, obtains 87.65%, and Native, which exposes the full catalog, obtains 14.81%; under the definition in Section 4.1, both should be 100%.
  5. On Dataverse, the catalog has 16 skills (about 1.7K tokens of descriptions) and K can be up to 10. Is subset selection meaningful at this scale, and what K does SkillECTOR choose there?
  6. How does the controller perform on skills added after training, without retraining? Does the state include any environment or user context beyond the query?
3 comments
AuthorsOpenReview ↗

Response to the review comments

We thank the reviewer for the insightful comments.

Generalization

We evaluated generalization to unseen tasks and skill catalogs by training on one catalog (e.g., AgentSkillOS) and transferring the policy checkpoint to another (e.g., SkillsBench) without retraining from scratch. The transferred policy reaches optimal performance and outperforms baselines within ~5 adaptation steps (Appendix H, Figure 9).

This follows from SkillECTOR's design: rather than predicting task-specific skill subsets, it predicts subset-construction parameters, namely retrieval depth KK and rank-fusion weights WW. Thus, the policy is not tied to specific skill identities and can transfer across catalogs.

SkillECTOR vs. LLM-Free Retrievers

Our GHCP experiments (Table 5) show that SkillECTOR can replace an LLM-based selector in an actual task agent.

We further tested a parallel SOTA task model, OPUS 5.5, with agent memory enabled, giving the agent more opportunity to identify relevant skills itself (Appendix J):

ExperimentTSRTool CallsTotal TokensP95 TokensP99 Tokens
No-skill41.38%3,436.50266.60M9.07M12.98M
Native-120957.47%3,807.77345.69M12.30M19.95M
SkillECTOR-120962.07%3,920.06324.48M11.21M18.20M

SkillECTOR still improves task performance while using fewer tokens than native skill selection.

We also compare against non-adaptive, LLM-free retrievers using fixed-weight rankers at K∈{3,5,10}K\in\{3,5,10\} (Appendix G). All families use weight 1.0 and RRF k=60k=60.

SkillECTOR vs. Best Static Configuration

BenchmarkKBest static familiesStatic TSARSkillECTOR TSAR
AgentSkillOS3Dense + Instr.26.67%
5Dense + Instr.40.00%50.00%
10BM25 + Dense + Instr.56.67%
SkillsBench3BM25 + Instr.47.87%
5BM25 + Dense + Instr.63.83%67.02%
10BM25 + Dense + Instr.78.72%
Dataverse3Instr.40.44%
5Instr.58.47%77.05%
10Instr. / All three87.98%

For completeness, the best static configurations from the full breakdown are:

BenchmarkK=3K=5K=10
AgentSkillOSD+I: 26.67D+I: 40.00B+D+I: 56.67
SkillsBenchB+I: 47.87B+D+I: 63.83B+D+I: 78.72
DataverseI: 40.44I: 58.47I / B+D+I: 87.98

The full static results are reported in Appendix G. They show three patterns: (1) no static ranker combination is consistently optimal across benchmarks or KK; (2) adding rankers can help or hurt, e.g., on AgentSkillOS at K=5K=5, adding BM25 to D+I reduces TSAR from 40.00% to 26.67%; and (3) larger fixed KK improves TSAR but increases context cost for every query. Static methods exceed SkillECTOR only at K=10K=10, motivating SkillECTOR's adaptive KK and ranker selection.

Is the Reported Advantage Marginal?

We do not believe so. SkillsBench is deliberately challenging: even with exact ground-truth skills, the strongest foundation models achieve ~69% TSAR according to the SkillsBench leaderboard. Thus, solving three additional tasks is meaningful in this regime.

On Dataverse, SkillECTOR's TSAR is below the strongest LLM-based baseline, GPT-6 Astra, an extremely high-cost model. With a small catalog, an LLM selector can inspect a large fraction of all skills; this becomes increasingly expensive and less scalable as catalogs grow. SkillECTOR's advantage should therefore be assessed jointly on performance, cost, and context efficiency.

Is SkillECTOR Context-Aware?

Yes. SkillECTOR is context-aware because its policy conditions on query context to dynamically determine the retrieval configuration. Context awareness need not require user- or tenant-level information: the query itself provides task-specific context determining the appropriate skills and retrieval strategy.

The current work focuses on query-level context. Incorporating additional signals, such as user- or tenant-level context, is a natural direction for future work.

Additional Questions

  • Reward weights: Hyperparameters selected by grid search, a standard ML practice. Small variations produce the same result.
  • Table 5 TSAR: Computed over skills the agent actually chooses to consider; often it does not consider even the full set of oracle-recommended skills.
  • Maximum KK: K=10K=10, irrespective of the original catalog size.
AuthorsOpenReview ↗

Additional comparison with SkillRouter as highlighted by one of the reviews

SkillRouter vs SkillECTOR

[Appendix K - we will try to fit this into main paper during camera ready stage].

Skillector selects an adaptive set of skills per task (about 2.9 on the SkillRouter Hard pool) instead of a fixed top-10. That set is 46.7% precise, while any top-10 list on this benchmark is capped at 26.7%. SkillRouter's own agent study shows that top-10 adds almost no task success over top-1 (27.78% vs 27.56%), so a small, well-chosen set matters more than a long one. Skillector does this with about 44M parameters on a CPU laptop: training took 2.85 hours and needed no LLM-generated training data or fine-tuning. SkillRouter's 1.2B pipeline needed 38K synthetic training queries, fine-tuning on a 96 GB GPU, and about 19 GB of GPU memory to serve. Skillector's ranking sources are pluggable, and its per-query choices of K and fusion weights are explainable.

Results

SkillECTOR vs SkillRouter on AgentSkillOS (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)4.43 average101010 (Hit@1 uses 1)
TSAR / FC at that K50.00%53.33%56.67%36.67%
… single-skill100%100%100%80.00%
… multi-skill25.00%30.00%35.00%15.00%
Recall at that K70.56%75.00%75.56%51.39%
Precision at that K39.99%13.33%13.33%8.00%
Hit@146.67%46.67%33.33%30.00%
MRR0.593 (within predicted skills)0.593 (@10)0.493 (@10)0.432 (@10)

SkillECTOR vs SkillRouter on Dataverse (DV) (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)

MetricSkillector DQN’s predicted KPredicted fused ranking, Static K=10Static fusion 1:1:1, Static K=10SkillRouter 1.2B
Skills loaded per task (K)3.90 average101010 (Hit@1 uses 1)
TSAR / FC at that K74.32%90.16%87.98%65.03%
… single-skill (64 tasks)93.75%93.75%96.88%81.25%
… multi-skill (119 tasks)63.87%88.24%83.19%56.30%
Recall at that K88.41%95.65%96.16%81.32%
Precision at that K62.12%23.33%23.22%19.07%
Hit@159.02%59.02%54.10%61.75%
MRR0.742 (within predicted skills)0.745 (@10)0.708 (@10)0.715 (@10)

Our SkillECTOR framework significantly beats SkillRouter across two well-known benchmarks at ZERO GPU cost. We can train on a CPU. Our TSAR and SkillRouter's FC are equivalent metrics. We could not run the SkillRouter paper's modified benchmark with 88K+ skills in the time available. Building the initial Instruction-Dense index takes setup time. Instead, we tested both SkillRouter and SkillECTOR on our existing benchmarks.

SkillRouter's claim of generalizing to any benchmark or skill set is proven false (relative to SkillECTOR), despite its static retrieval depth of K=10 and our much lower predicted K

AuthorsOpenReview ↗

Request to Reviewer

We believe we have been able to satisfactorily address all the concerns highlighted in your review and beyond. If there are any other concerns please let us know. Otherwise, we request you to kindly consider revising the score. Thank you again for your valuable time and review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.