- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces SkillCover, a method to greedily select skills from a pool to guarantee coverage given a limited budget of skills. The results show almost no improvement for most models.
Critical strengths
The idea of having a library of skills with efficient coverage that retains agent performance could be interesting, if done correctly.
Critical weaknesses
The writing is really poor (I suspect it’s mostly AI generated) and it reads more as a blog post than an academic paper. The motivation of the problem is also extremely poor. Why would we even want to reduce the number of skills? I’m not saying there’s no reason, but it has to be articulated instead of assumed to be a relevant problem. In the paper, the agents retained lots of skills, even when the candidate pool is reduced to . In comparison, I’m not sure why we’d even care about reducing that pool so much.
There’s almost no novelty. The proposed algorithm is standard submodular maximization with lots of proofs and lemmas that I think distract and inflate the appendix rather than providing anything meaningful.
Even if I bought into the idea of a limited skill budget, the approach seems inherently flawed. The method focuses on skill coverage given a set of tasks, but future tasks cannot be foreseen. You basically set an upper limit on the capabilities of these agents to perform out-of-distribution tasks in the future. Optimizing for the present won’t help the future if you need a rare but very important skill, such is the case in black swan events. This leads to another flaw, which is assuming that you can actually figure out the best skill for each task.
Even if all of these issues were fixed, the experiments are quite weak, and the results just display all these flaws. The baseline is just an obvious singleton ranking, so even if your method was better, it’d be better than a very simple approach. Finally, the scale of the experiments is too small to infer anything meaningful.
The results show basically no improvement in agent success at all. Improved coverage is not enough to show higher agent performance. You also don’t show any other benefits in terms of tokens, etc.
Questions for authors
I don’t have questions that can change my assessment.