- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SkillECTOR, that instead of using an LLM to pick which skills an agent should load, it uses a tiny RL policy that tunes a classic retrieval pipeline per query, with zero LLM calls at inference time. In this way, the controller’s purpose is adaptivity: deciding, per query, how to configure retrieval so the model gets the right skills without loading too many or too few.
Critical strengths
-
The paper's proposed method is novel. The authors use the policy output configuration parameters rather than skill identities makes the decision space independent of catalog size. New skills only require index updates, not retraining a head or resizing an output layer.
-
The paper also clearly address and acknowledge the loss of the model. The paper acknowledges that strong LLM selectors win on the small 16-skill Dataverse catalog, and that GPT-6 Astra beats it on precision and F1 in the combined setting. That candor makes the positive results more believable and scopes the claim sensibly.
Critical weaknesses
-
The paper misses an important ablation study. The paper never compares the learned RL controller against a fixed-K, fixed-weight (or simply tuned) fusion of the same three rankers.
-
The Train/test set separation is unclear. The learning curves are plotted over epochs on the same task sets (94 SkillsBench tasks, 30 AgentSkillOS tasks, 183 Dataverse tasks) used for reporting results, and the text does not describe a held-out split.
Questions for authors
-
Are the policies trained and evaluated on the same tasks?
-
Can a policy trained on one catalog transfer to another, or to the same catalog after skills are added without retraining?
3 comments
Additional comparison with SkillRouter as highlighted by one of the reviews
SkillRouter vs SkillECTOR
[Appendix K - we will try to fit this into main paper during camera ready stage].
Skillector selects an adaptive set of skills per task (about 2.9 on the SkillRouter Hard pool) instead of a fixed top-10. That set is 46.7% precise, while any top-10 list on this benchmark is capped at 26.7%. SkillRouter's own agent study shows that top-10 adds almost no task success over top-1 (27.78% vs 27.56%), so a small, well-chosen set matters more than a long one. Skillector does this with about 44M parameters on a CPU laptop: training took 2.85 hours and needed no LLM-generated training data or fine-tuning. SkillRouter's 1.2B pipeline needed 38K synthetic training queries, fine-tuning on a 96 GB GPU, and about 19 GB of GPU memory to serve. Skillector's ranking sources are pluggable, and its per-query choices of K and fusion weights are explainable.
Results
SkillECTOR vs SkillRouter on AgentSkillOS (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)
| Metric | Skillector DQN’s predicted K | Predicted fused ranking, Static K=10 | Static fusion 1:1:1, Static K=10 | SkillRouter 1.2B |
|---|---|---|---|---|
| Skills loaded per task (K) | 4.43 average | 10 | 10 | 10 (Hit@1 uses 1) |
| TSAR / FC at that K | 50.00% | 53.33% | 56.67% | 36.67% |
| … single-skill | 100% | 100% | 100% | 80.00% |
| … multi-skill | 25.00% | 30.00% | 35.00% | 15.00% |
| Recall at that K | 70.56% | 75.00% | 75.56% | 51.39% |
| Precision at that K | 39.99% | 13.33% | 13.33% | 8.00% |
| Hit@1 | 46.67% | 46.67% | 33.33% | 30.00% |
| MRR | 0.593 (within predicted skills) | 0.593 (@10) | 0.493 (@10) | 0.432 (@10) |
SkillECTOR vs SkillRouter on Dataverse (DV) (SkillECTOR trained on Skillsbench; SkillRouter checkpoint used directly from the repository)
| Metric | Skillector DQN’s predicted K | Predicted fused ranking, Static K=10 | Static fusion 1:1:1, Static K=10 | SkillRouter 1.2B |
|---|---|---|---|---|
| Skills loaded per task (K) | 3.90 average | 10 | 10 | 10 (Hit@1 uses 1) |
| TSAR / FC at that K | 74.32% | 90.16% | 87.98% | 65.03% |
| … single-skill (64 tasks) | 93.75% | 93.75% | 96.88% | 81.25% |
| … multi-skill (119 tasks) | 63.87% | 88.24% | 83.19% | 56.30% |
| Recall at that K | 88.41% | 95.65% | 96.16% | 81.32% |
| Precision at that K | 62.12% | 23.33% | 23.22% | 19.07% |
| Hit@1 | 59.02% | 59.02% | 54.10% | 61.75% |
| MRR | 0.742 (within predicted skills) | 0.745 (@10) | 0.708 (@10) | 0.715 (@10) |
Our SkillECTOR framework significantly beats SkillRouter across two well-known benchmarks at ZERO GPU cost. We can train on a CPU. Our TSAR and SkillRouter's FC are equivalent metrics. We could not run the SkillRouter paper's modified benchmark with 88K+ skills in the time available. Building the initial Instruction-Dense index takes setup time. Instead, we tested both SkillRouter and SkillECTOR on our existing benchmarks.
SkillRouter's claim of generalizing to any benchmark or skill set is proven false (relative to SkillECTOR), despite its static retrieval depth of K=10 and our much lower predicted K.
Request to reviewer
We believe we have been able to satisfactorily address all the concerns highlighted in your review and beyond. If there are any other concerns please let us know. Otherwise, we request you to kindly consider revising the score. Thank you again for your valuable time and review.
Response to review comments
We thank the reviewer for the positive assessment of the method's novelty, catalog-size-independent design, and candid reporting.
W1: Comparison against a fixed-K, fixed-weight fusion of the same rankers
We agree this is an important baseline. We evaluated every combination of BM25, Dense skill, and Instruction dense using fixed RRF (weight 1.0, k=60) at Top-K=3/5/10 across all benchmarks.
Setup
Overall Summary: Best Static Configuration vs. SkillECTOR
Static retrieval exceeds SkillECTOR only with fixed K=10, increasing context for every query. SkillECTOR adapts retrieval per query.
Detailed Static LLM-Free Retriever Results
Bold indicates the best static configuration at each depth.
AgentSkillOS
SkillsBench
Dataverse
Three patterns emerge: (1) no static configuration is consistently optimal; (2) adding rankers can help or hurt, e.g., AgentSkillOS at K=5: 40.00%→26.67% after adding BM25; (3) larger fixed K improves TSAR but increases context. This motivates adaptive depth and fusion.
W2 / Q1: Are the policies trained and evaluated on the same tasks?
Yes. Reported policies are trained and evaluated on the same small task sets in SkillsBench and AgentSkillOS, so we report final-checkpoint performance after convergence, a valid continual-RL protocol. We will state this explicitly. We separately evaluate cross-catalog generalization, with training and testing on different catalogs and hence zero leakage.
Q2: Can a policy transfer to another catalog, or to the same catalog after skills are added, without retraining?
Yes. We trained on one catalog (e.g., AgentSkillOS) and transferred the checkpoint to another (e.g., SkillsBench) without retraining from scratch. It reaches optimal performance and beats baselines within ~5 adaptation steps (Appendix H, Figure 9).
This follows from SkillECTOR's design: the policy predicts subset-construction parameters, retrieval depth K and rank-fusion weights W, rather than skill identities. It is therefore transferable across catalogs.