acceptodds
Under review as a conference paper at ICLR 2027

SkillCover: Fixed-Pool Skill Library Selection as Noisy Submodular Coverage

Abstract

When an agent can retain only \(k\) skills, ranking them by individual performance can waste slots on redundant capabilities. SkillCover greedily selects skills from a fixed pool to maximize baseline-relative task coverage on a calibration matrix. This classical submodular objective measures the best performance available in the retained library. We analyze selection under task sampling and rollout noise, and evaluate coverage and agent success separately. On real SkillsBench packages, greedy improves held-out matrix coverage over singleton ranking on GLM-5 (\(+0.083\) at \(k=4\)); results on three other models are mixed. A deliberately redundant MCPAgentBench pool and synthetic controls show when singleton ranking wastes the budget on overlap. Deployment on 14 new tasks does not establish a success advantage. A fresh-calibration diagnostic gives positive mean differences on two models at four repetitions, but intervals include zero and one comparison reverses at two repetitions. Whether coverage gains improve deployment remains unresolved.

Primary area:
optimization
Keywords:
agent skills, skill library selection, submodular optimization, noisy submodular maximization, LLM agents, model routing

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_EtDGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SkillCover, a method to greedily select skills from a pool to guarantee coverage given a limited budget of skills. The results show almost no improvement for most models.

Critical strengths

The idea of having a library of skills with efficient coverage that retains agent performance could be interesting, if done correctly.

Critical weaknesses

The writing is really poor (I suspect it’s mostly AI generated) and it reads more as a blog post than an academic paper. The motivation of the problem is also extremely poor. Why would we even want to reduce the number of skills? I’m not saying there’s no reason, but it has to be articulated instead of assumed to be a relevant problem. In the paper, the agents retained lots of skills, even when the candidate pool is reduced to k=4k=4. In comparison, I’m not sure why we’d even care about reducing that pool so much.

There’s almost no novelty. The proposed algorithm is standard submodular maximization with lots of proofs and lemmas that I think distract and inflate the appendix rather than providing anything meaningful.

Even if I bought into the idea of a limited skill budget, the approach seems inherently flawed. The method focuses on skill coverage given a set of tasks, but future tasks cannot be foreseen. You basically set an upper limit on the capabilities of these agents to perform out-of-distribution tasks in the future. Optimizing for the present won’t help the future if you need a rare but very important skill, such is the case in black swan events. This leads to another flaw, which is assuming that you can actually figure out the best skill for each task.

Even if all of these issues were fixed, the experiments are quite weak, and the results just display all these flaws. The baseline is just an obvious singleton ranking, so even if your method was better, it’d be better than a very simple approach. Finally, the scale of the experiments is too small to infer anything meaningful.

The results show basically no improvement in agent success at all. Improved coverage is not enough to show higher agent performance. You also don’t show any other benefits in terms of tokens, etc.

Questions for authors

I don’t have questions that can change my assessment.

Reviewer_dtfVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillCover, which retains a limited number of agent skills by greedily maximizing complementary task coverage rather than ranking skills individually. It analyzes how task sampling and rollout noise affect the greedy guarantee and evaluates on SkillsBench, MCPAgentBench, and synthetic studies. Coverage improves on GLM-5 and a deliberately redundant pool, but results on other models are mixed, and deployment shows no clear advantage.

Critical strengths
  1. Reasonable formulation of skill complementarity. The coverage objective accounts for overlapping capabilities and frames skill retention as a submodular selection problem. The main value is this problem framing: it gives a clear justification for selecting beyond individual skill rankings.

  2. Separation of coverage and deployment performance. The paper evaluates these separately and transparently reports mixed results and sensitivity checks. This makes its conclusions credible and shows that better library coverage cannot be assumed to translate into improved agent success.

Critical weaknesses
  1. Limited methodological novelty. SkillCover applies standard greedy optimization to a classical coverage objective, with guarantees derived from established approximation and concentration arguments. The paper offers limited new insight beyond adapting this framework to skill retention.

  2. Insufficient evidence of practical value. Coverage gains are inconsistent across models. At k=4 in Table 1, greedy falls below random selection on DeepSeek-V3.2 (0.133 vs. 0.184) and Qwen3-Coder (0.150 vs. 0.153). GLM-5's headline +0.083 advantage does not persist under exploratory task-disjoint refitting (-0.067), and the refit results across models (-0.017, 0, +0.050, -0.067) show no stable advantage in either direction. Deployment studies show no reliable improvement in success, and the selected libraries do not clearly improve on Base (Table 2: Sonnet Greedy 13/28 vs. Base 14/28; GLM-5 4/28 for both, while its All-15 arm reaches 8/28). The practical need for the budget is also not demonstrated, since all arms retain 64 background skills outside it and the cost of keeping all 15 candidates is not measured. Finally, the transfer study records no explicit skill invocations in Greedy/Singleton executions, and the 13 invocations in the fresh-calibration diagnostic do not occur on the invoice or OCR tasks behind Sonnet's advantage, leaving the mechanism unverified. Together, these results do not establish that SkillCover meaningfully improves agent skill retention.

Questions for authors
  1. What explains GLM’s advantage changing from +0.083 in the same-task rollout evaluation to −0.067 under leave-one-task-out refitting? Does the available evidence support generalization beyond calibration tasks?

  2. Given the absence of recorded explicit skill invocations in the greedy/singleton transfer executions, is there evidence that agents accessed or used the selected skill contents? This would affect how I interpret the deployment comparison.

  3. Theorem 2 appears to be vacuous at the sample sizes used in the real experiments. Does the noisy-calibration analysis provide any actionable guidance at these settings, such as for choosing n and q, beyond a standard combination of uniform concentration and the classical greedy approximation guarantee?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.