- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper audits SkillOpt, a skill self-evolution system, by running skill and no-skill conditions on the same task pools with five repetitions per task, and by categorizing the rules that evolution adds to the skill. The authors find that the paired gain changes sign across benchmarks, base models, and harnesses, e.g., +13.4 points on SearchQA but −6.2 on OfficeQA. They further show that the gains mainly come from tasks that were only sometimes solved, while the harm falls on tasks the no-skill executor always solves, and that the added rules are operational conventions rather than knowledge. Based on this, they propose to keep the skill off by default and retry once with the skill on failure, which reaches 59.1% on SearchQA versus 37.0% without the skill and 50.4% with full injection.
Critical strengths
- The evaluation protocol is careful and reports what happens at the task level. Every comparison pairs the same tasks under skill and no-skill conditions with five repetitions, reports how many tasks improved and how many were harmed, and uses McNemar tests and task-cluster bootstraps. For example, the +14.2 net gain of one SearchQA skill comes from 315 improved tasks against 116 harmed ones. This is useful because it shows that a single average score can hide large changes in both directions, which is a good practice for how self-evolved skills should be evaluated.
- The paper also checks what evolution actually writes into the skill. The 189 rules added by evolution are categorized with released, deterministic patterns rather than LLM judges, and the paper reports that only about one in ten proposed revisions is accepted (10.7%). This is useful because it gives a concrete and checkable picture of the evolved skill, instead of only reporting its final score.
Critical weaknesses
-
The major concern is that the technical contribution of this work seems limited. The proposed rule, i.e., keep the skill off by default and retry once with the skill after a failure, is quite intuitive and more like a simple engineering trick than a new method. It is also not surprising that a second attempt helps, since almost any retry strategy would. Besides, the main finding that skills mostly stabilize tasks that were already sometimes solved, rather than adding new capability, has already been reported by prior work, e.g., Jiang et al. (2026) find that a given skill mostly stabilizes existing behavior. The paper argues that its difference is studying the evolution process rather than given skills, but the analysis itself, i.e., paired skill vs. no-skill runs and categorizing the added rules, is largely a standard evaluation, and it only covers one system, SkillOpt. So it is hard to tell what new insight or technique the paper brings beyond a careful audit of one system.
-
Another concern is that the analysis behind the main findings may be misleading. The paper groups tasks into easy, mixed, and hard bands based on how the no-skill runs perform, and then compares them with new runs using the skill. The problem is that the grouping itself already decides the result: a task picked because it was correct in all five no-skill runs can only go down in new runs, and a task that was never correct can only go up. This would happen even if the skill does nothing. In fact, the paper's own human seed, which is just a short placeholder without any rules, shows the same pattern, i.e., −23.33 on the easy band and +6.18 on the hard band. So the claim that the skill helps hard tasks and hurts easy tasks is not well supported. In addition, the retry rule is not compared fairly. It decides whether to retry using the official evaluator, which needs the gold answer, and it gets two attempts while the baselines only get one. Since more than half of the SearchQA tasks are already sometimes solved without the skill, simply retrying without the skill may also give a large gain. The authors are encouraged to address these issues.
Questions for authors
- What is new in the findings compared with prior skill audits, e.g., Jiang et al. (2026)?
- Have the authors compared the retry rule with simply retrying once without the skill?