- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Skill², a framework that jointly evolves an agent policy and its skill library through alternating reinforcement learning. The actor uses Fork-GRPO to learn skill invocation decisions, while the extractor learns to maintain the library through counterfactual utility and delayed skill-level feedback. Experiments across seven terminal and web benchmarks show consistent improvements over existing baselines at two model scales. The evolved skill library also improves the performance of larger frozen models.
Critical strengths
-
Well-motivated co-evolution framework. Jointly training the actor and skill extractor addresses a meaningful limitation of existing skill-learning methods. The use of structured skill artifacts and native invocation makes the framework particularly relevant to practical agents.
-
Convincing empirical evaluation. The improvements across seven benchmarks and two model scales are encouraging. Beyond the main results, the ablations, frozen-model transfer experiments, and skill evolution case study provide useful support for the proposed approach.
Critical weaknesses
-
Limited counterfactual comparison in Fork-GRPO. The control branch in Section 2.2 compares invoking a selected skill against proceeding without one. Although useful for evaluating skill invocation, this comparison does not directly distinguish between alternative skills.
-
Ambiguity in delayed edit credit. Section 2.3 uses subsequent skill invocation gains to credit earlier edits. When a skill undergoes multiple revisions, the resulting feedback may not accurately reflect the contribution of each revision.
-
Unconditional release of skill revisions. Algorithm 1 releases the candidate library regardless of its counterfactual utility, although Figure 4 shows that some revisions have negative utility. The rationale for retaining these revisions is not sufficiently discussed.
Questions for authors
-
Fork-GRPO provides feedback on whether invoking a particular skill is beneficial. How does the actor learn to prefer one skill over another when multiple skills are applicable?
-
Suppose a skill receives several updates before producing a nonzero invocation gain. Would all earlier edits receive the same micro-level credit, even if some have been superseded by later revisions?
-
What motivates releasing a revised library when its measured counterfactual utility is negative? Would retaining the previous library in such cases interfere with the intended co-evolution process?
5 comments
Response to Reviewer Cb1A (2/3)
W2 / Q2."Suppose a skill receives several updates before producing a nonzero invocation gain. Would all earlier edits receive the same micro-level credit, even if some have been superseded by later revisions?"
Thank you for your careful reading. All edits of a skill that are still waiting when it is certified, including superseded ones, share the micro credit equally, and we are glad to explain this design below.
A superseded edit has left no evidence of its own. An edit is certified in the first round after it in which its skill's invocation gain is nonzero. An edit is therefore still waiting when superseded only if its version produced no disagreeing pair while in the library; the certification of a later version is then the only evidence about this lineage.
Earlier edits are ancestors of the certified version. An UPDATE rewrites the body it reads from the draft (Eq. 7). In Appendix G, the parameterized auditor of round 77 extends the two-file comparison of round 56, which in turn extends the one-key check of round 30.
Crediting only the latest edit would leave intermediate steps uncredited. This holds in particular for description edits, which change when a skill is invoked rather than what it does.
The shared credit is bounded. It is split equally among the edits of a unit, an edit waits at most rounds for certification, and edits older than 32 updates are dropped. Figure 5 shows that costs 1.0 point while adds nothing (36.1 vs. 36.3), suggesting that delayed credit is worth waiting for, while waiting beyond brings no further gain. Over all certified units of the 9B terminal run:
| Share of units | Units containing a superseded edit | |
|---|---|---|
| 1 | 63.2% | – |
| 2 | 24.0% | 17.0% |
| ≥ 3 | 12.8% | 84.4% |
Superseded edits account for 13.0% of all micro-credited edits and wait a median of 4 rounds for certification.
Response to Reviewer Cb1A (3/3)
W3 / Q3."Algorithm 1 releases the candidate library regardless of its counterfactual utility, although Figure 4 shows that some revisions have negative utility. … What motivates releasing a revised library when its measured counterfactual utility is negative? Would retaining the previous library in such cases interfere with the intended co-evolution process?"
Thank you for raising this point. We are glad to explain this design in more detail below.
A negative marks a negative-advantage action. We optimize the library through the extractor, a policy whose actions are edits of the library and whose reward is the gain these edits bring to the current actor (§2.1). measures this gain, so it is the extractor's reward rather than an acceptance test for the library, and releasing the candidate is the transition caused by the extractor's action. A negative yields and lowers the probability of similar revisions, just as policy gradient executes a negative-advantage action and penalizes it rather than vetoing it.
Noise averages out in a gradient but not in a gate. averages the paired differences over 128 probe tasks with one repetition per task, and it shrinks toward zero in the refinement phase. As a reward, its noise averages over updates; as a gate, each noisy estimate makes a hard binary decision, and near zero the library would follow the probe draw rather than learning.
Retaining the previous library would likely interfere with co-evolution. First, it could stall the process: the next maintenance sets come from the actor on the same library, so the extractor would likely propose a similar revision again. Second, it would discard micro credit where it matters most: the edits of a withheld candidate are not invoked and cannot be certified, so the helpful edits within a net-negative revision could not be rewarded, nor the harmful ones singled out.
The risk of unconditional release is bounded. Since is measured against the released library, a revision that repairs earlier damage earns positive credit; in the case study, the rewrite of round 77 extends the tool but introduces two regressions, and the edit of round 96 repairs both while keeping the extension. The actor's invocation decisions, trained by , act as a learned gate, with unused skills retired at capacity.
The training dynamics in Figure 4 are consistent with this. The utility is positive in only 82 of the 143 rounds and still negative in 16% of the last 43 rounds, yet the training reward rises from 78.6% to 88.5% and skill use from 2.3% to 44.7%, so co-evolution progresses steadily even though revisions with non-positive utility are released.
Thank you again for your thoughtful questions, which helped us explain these design choices more clearly. We hope these responses address your concerns, and we would be very happy to clarify any further points.
Response to Reviewer Cb1A (1/3)
Thank you very much for your careful review and thoughtful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.
Thank you for this insightful question. We are glad to explain how the actor learns to prefer one applicable skill over another.
Although each δ is measured against a branch that does not invoke at this turn, its effect on the skill choice compares skills with one another. The control branch takes an environment action at h=ht⋆ and thereafter follows the same policy, so E[δ∣h,n⋆]=Q(h,invoke(n⋆))−Venv(h), where Venv(h), the value of not invoking at this turn, does not depend on n⋆. Eq. 3 writes δ onto the decision tokens, including the skill name. We write the probability of invoking n at h as p(h)πinv(n∣h), the probability of invoking times that of choosing n, and let Vinv(h)=∑nπinv(n∣h)Q(h,invoke(n)). Since ∑nπinv(n∣h)∇logπinv(n∣h)=0, Venv(h) drops out of the part of the expected update that acts on the skill choice:
∑nπinv(n∣h)∇logπinv(n∣h)[Q(h,invoke(n))−Vinv(h)].
Every applicable skill is thus compared with the same baseline, the value of the actor's own mixture of skills, and since the probabilities are normalized, raising one skill lowers the others. The remaining part, ∇logp(h)[Vinv(h)−Venv(h)], acts only on whether to invoke.
The library also helps the actor tell applicable skills apart. The actor chooses from metadata alone, and the extractor, trained with per-skill micro credit, rewrites descriptions so that they state when each skill applies (the skill in Appendix G has its description rewritten twice).
Test-time interventions indicate that the actor has learned a meaningful preference among skills. Replacing each skill the final 9B actor invokes with an alternative sampled from its own distribution lowers TB 2.1 from 36.3 to
31.9, and replacing it with a random skill lowers it to27.6.