acceptodds
Under review as a conference paper at ICLR 2027

Skill: Fully Agentic Co-Evolution of Policy and Skill Library

11 views

Abstract

Skills have become the native carrier of procedural knowledge in deployed agents. However, most skill learning is misaligned with deployment in three respects, the artifact it produces, the pathway through which skills are invoked, and the tasks on which it is validated. The few fully agentic efforts evolve only one side, training the policy against a fixed skill library or evolving the skill library under a frozen model. Yet a skill matures on neither side alone. We present Skill², a fully agentic co-evolution framework in which a policy and its skill library advance together by alternating two separately trained roles around one skill library. The actor learns whether, when, and which skill to invoke through the native harness with Fork-GRPO, which forks a control branch at each skill decision so that credit lands on the tokens that made it. The extractor learns to write and maintain skills with counterfactual-utility policy optimization, which rewards each edit by the paired gain the current actor obtains from the revised skill library with dual-level credit assignment. Extensive experiments on seven real long-horizon agentic benchmarks demonstrate that Skill² is (1) effective, surpassing baselines with relative gains of up to 46.7% on terminal tasks and 59.8% on web tasks, (2) scale-consistent, with its average gain growing from Qwen3.5-4B to Qwen3.5-9B, and (3) transferable, lifting four larger frozen models from three families by up to 13.0% through the evolved skill library alone. Our code is available here.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agentic Skill Learning, Policy–Skill Co-evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Cb1AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Skill², a framework that jointly evolves an agent policy and its skill library through alternating reinforcement learning. The actor uses Fork-GRPO to learn skill invocation decisions, while the extractor learns to maintain the library through counterfactual utility and delayed skill-level feedback. Experiments across seven terminal and web benchmarks show consistent improvements over existing baselines at two model scales. The evolved skill library also improves the performance of larger frozen models.

Critical strengths
  1. Well-motivated co-evolution framework. Jointly training the actor and skill extractor addresses a meaningful limitation of existing skill-learning methods. The use of structured skill artifacts and native invocation makes the framework particularly relevant to practical agents.

  2. Convincing empirical evaluation. The improvements across seven benchmarks and two model scales are encouraging. Beyond the main results, the ablations, frozen-model transfer experiments, and skill evolution case study provide useful support for the proposed approach.

Critical weaknesses
  1. Limited counterfactual comparison in Fork-GRPO. The control branch in Section 2.2 compares invoking a selected skill against proceeding without one. Although useful for evaluating skill invocation, this comparison does not directly distinguish between alternative skills.

  2. Ambiguity in delayed edit credit. Section 2.3 uses subsequent skill invocation gains to credit earlier edits. When a skill undergoes multiple revisions, the resulting feedback may not accurately reflect the contribution of each revision.

  3. Unconditional release of skill revisions. Algorithm 1 releases the candidate library regardless of its counterfactual utility, although Figure 4 shows that some revisions have negative utility. The rationale for retaining these revisions is not sufficiently discussed.

Questions for authors
  1. Fork-GRPO provides feedback on whether invoking a particular skill is beneficial. How does the actor learn to prefer one skill over another when multiple skills are applicable?

  2. Suppose a skill receives several updates before producing a nonzero invocation gain. Would all earlier edits receive the same micro-level credit, even if some have been superseded by later revisions?

  3. What motivates releasing a revised library when its measured counterfactual utility is negative? Would retaining the previous library in such cases interfere with the intended co-evolution process?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Cb1A (1/3)

Thank you very much for your careful review and thoughtful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.

W1 / Q1. "The control branch in Section 2.2 compares invoking a selected skill against proceeding without one. Although useful for evaluating skill invocation, this comparison does not directly distinguish between alternative skills. … How does the actor learn to prefer one skill over another when multiple skills are applicable?"

Thank you for this insightful question. We are glad to explain how the actor learns to prefer one applicable skill over another.

Although each δ\delta is measured against a branch that does not invoke at this turn, its effect on the skill choice compares skills with one another. The control branch takes an environment action at h=ht⋆h=h_{t^\star} and thereafter follows the same policy, so E[δ∣h,n⋆]=Q(h,invoke(n⋆))−Venv(h)\mathbb E[\delta \mid h, n^\star] = Q(h, \mathrm{invoke}(n^\star)) - V_{\mathrm{env}}(h), where Venv(h)V_{\mathrm{env}}(h), the value of not invoking at this turn, does not depend on n⋆n^\star. Eq. 3 writes δ\delta onto the decision tokens, including the skill name. We write the probability of invoking nn at hh as p(h)πinv(n∣h)p(h) \pi_{\mathrm{inv}}(n\mid h), the probability of invoking times that of choosing nn, and let Vinv(h)=∑nπinv(n∣h)Q(h,invoke(n))V_{\mathrm{inv}}(h)=\sum_n\pi_{\mathrm{inv}}(n\mid h) Q(h,\mathrm{invoke}(n)). Since ∑nπinv(n∣h)∇log⁡πinv(n∣h)=0\sum_n\pi_{\mathrm{inv}}(n\mid h) \nabla\log\pi_{\mathrm{inv}}(n\mid h)=0, Venv(h)V_{\mathrm{env}}(h) drops out of the part of the expected update that acts on the skill choice:

∑nπinv(n∣h)∇log⁡πinv(n∣h)[Q(h,invoke(n))−Vinv(h)].\textstyle\sum_n\pi_{\mathrm{inv}}(n\mid h) \nabla\log\pi_{\mathrm{inv}}(n\mid h) \big[Q(h,\mathrm{invoke}(n))-V_{\mathrm{inv}}(h)\big].

Every applicable skill is thus compared with the same baseline, the value of the actor's own mixture of skills, and since the probabilities are normalized, raising one skill lowers the others. The remaining part, ∇log⁡p(h)[Vinv(h)−Venv(h)]\nabla\log p(h) [V_{\mathrm{inv}}(h)-V_{\mathrm{env}}(h)], acts only on whether to invoke.

The library also helps the actor tell applicable skills apart. The actor chooses from metadata alone, and the extractor, trained with per-skill micro credit, rewrites descriptions so that they state when each skill applies (the skill in Appendix G has its description rewritten twice).

Test-time interventions indicate that the actor has learned a meaningful preference among skills. Replacing each skill the final 9B actor invokes with an alternative sampled from its own distribution lowers TB 2.1 from 36.3 to 31.9, and replacing it with a random skill lowers it to 27.6.

AuthorsOpenReview ↗

Response to Reviewer Cb1A (2/3)

W2 / Q2. "Suppose a skill receives several updates before producing a nonzero invocation gain. Would all earlier edits receive the same micro-level credit, even if some have been superseded by later revisions?"

Thank you for your careful reading. All edits of a skill that are still waiting when it is certified, including superseded ones, share the micro credit equally, and we are glad to explain this design below.

A superseded edit has left no evidence of its own. An edit is certified in the first round after it in which its skill's invocation gain gk,ng_{k,n} is nonzero. An edit is therefore still waiting when superseded only if its version produced no disagreeing pair while in the library; the certification of a later version is then the only evidence about this lineage.

Earlier edits are ancestors of the certified version. An UPDATE rewrites the body it reads from the draft (Eq. 7). In Appendix G, the parameterized auditor of round 77 extends the two-file comparison of round 56, which in turn extends the one-key check of round 30.

Crediting only the latest edit would leave intermediate steps uncredited. This holds in particular for description edits, which change when a skill is invoked rather than what it does.

The shared credit is bounded. It is split equally among the edits of a unit, an edit waits at most W=16W=16 rounds for certification, and edits older than 32 updates are dropped. Figure 5 shows that W=8W=8 costs 1.0 point while W=32W=32 adds nothing (36.1 vs. 36.3), suggesting that delayed credit is worth waiting for, while waiting beyond W=16W=16 brings no further gain. Over all certified units En\mathcal E_n of the 9B terminal run:

∣En∣\lvert\mathcal E_n\rvertShare of unitsUnits containing a superseded edit
163.2%–
224.0%17.0%
≥ 312.8%84.4%

Superseded edits account for 13.0% of all micro-credited edits and wait a median of 4 rounds for certification.

AuthorsOpenReview ↗

Response to Reviewer Cb1A (3/3)

W3 / Q3. "Algorithm 1 releases the candidate library regardless of its counterfactual utility, although Figure 4 shows that some revisions have negative utility. … What motivates releasing a revised library when its measured counterfactual utility is negative? Would retaining the previous library in such cases interfere with the intended co-evolution process?"

Thank you for raising this point. We are glad to explain this design in more detail below.

A negative UkU_k marks a negative-advantage action. We optimize the library through the extractor, a policy whose actions are edits of the library and whose reward is the gain these edits bring to the current actor (§2.1). UkU_k measures this gain, so it is the extractor's reward rather than an acceptance test for the library, and releasing the candidate is the transition caused by the extractor's action. A negative UkU_k yields Akmac<0A_k^{\mathrm{mac}} < 0 and lowers the probability of similar revisions, just as policy gradient executes a negative-advantage action and penalizes it rather than vetoing it.

Noise averages out in a gradient but not in a gate. UkU_k averages the paired differences over 128 probe tasks with one repetition per task, and it shrinks toward zero in the refinement phase. As a reward, its noise averages over updates; as a gate, each noisy estimate makes a hard binary decision, and near zero the library would follow the probe draw rather than learning.

Retaining the previous library would likely interfere with co-evolution. First, it could stall the process: the next maintenance sets come from the actor on the same library, so the extractor would likely propose a similar revision again. Second, it would discard micro credit where it matters most: the edits of a withheld candidate are not invoked and cannot be certified, so the helpful edits within a net-negative revision could not be rewarded, nor the harmful ones singled out.

The risk of unconditional release is bounded. Since Uk+1U_{k+1} is measured against the released library, a revision that repairs earlier damage earns positive credit; in the case study, the rewrite of round 77 extends the tool but introduces two regressions, and the edit of round 96 repairs both while keeping the extension. The actor's invocation decisions, trained by δ\delta, act as a learned gate, with unused skills retired at capacity.

The training dynamics in Figure 4 are consistent with this. The utility is positive in only 82 of the 143 rounds and still negative in 16% of the last 43 rounds, yet the training reward rises from 78.6% to 88.5% and skill use from 2.3% to 44.7%, so co-evolution progresses steadily even though revisions with non-positive utility are released.

Thank you again for your thoughtful questions, which helped us explain these design choices more clearly. We hope these responses address your concerns, and we would be very happy to clarify any further points.

Reviewer_uSfgOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper studies agentic skill learning for long horizon agents. It argues the current methods misalign validation with deployment. Skills often appear as prompt text with a pipeline to injects them. And training and testing happen in text simulators.

The proposed method combines two policies with one skill library inside a deployment style harness. The actor is trained with Fork-GRPO. At each skill decision, the method forks a branch that proceeds without the skill. The reward difference is written to the decision tokens. Other tokens keep a group relative advantage.

The extractor other other hand is trained with counterfactual utility policy optimization. It first reads actor rollouts and produces edits. A candidate library is valued by runs with the current actor. Credit is assigned: 1. Macro credit covers the whole revision 2, Micro credit covers each skill through the actor control branches. The library is from the current actor. Then the actor trains on the new library.

The experiment covers seven terminal and web benchmarks with Qwen3.5.

Critical strengths

1.The paper is based on a real gap between skill learning and deployment. Skill methods often look strong in text simulators and then fail in real harnesses and the three misalignments are useful.

2.The claim is well supported. The role-level comparison and the ablation show the gains are based on specific mechanisms. All benchmarks run live environments with programmatic verifiers.

Critical weaknesses
  1. baselines compute concern. Skill2 use two trained models, forks a full control rollout at every skill decision, and executes two runs of every probe task in every round. If SkillGate received a comparable budget of policy-gradient updates on the same tasks, the gap between them might shrink substantially. The claim that co-evolution outperforms one-sided training is the central contribution. And it cannot be fully evaluated based on the current experiments.

  2. Single training run per configuration. The optimization involves two policies and a library rewritten each round. Every cell lacks a standard deviation over training runs. Several conclusions, including the scale consistency claim (4B,9B) and the transfer margins, rest on differences of one to two points.

Questions for authors
  1. What are the matched GPU hours, environment steps, and inference tokens for each trained baseline?
  2. What fraction of fork pairs is degraded to group-relative credit because their states do not match, in each domain? What is the empirical standard deviation of the paired differences?
  3. What is the state match rate for forked control branches, and how often does a pair keep only group relative credit?
  4. Were any results obtained with more than one training seed? If so, what is the seed variance?
6 comments
AuthorsOpenReview ↗

Response to Reviewer uSfg (1/4)

Thank you very much for your careful review and constructive comments, and for the time you have devoted to our paper. We greatly appreciate your feedback and are glad to address each point below in detail.

W1 / Q1. "Skill2 use two trained models, forks a full control rollout at every skill decision, and executes two runs of every probe task in every round. If SkillGate received a comparable budget of policy-gradient updates on the same tasks, the gap between them might shrink substantially. … What are the matched GPU hours, environment steps, and inference tokens for each trained baseline?"

Thank you for raising this important point. We are glad to provide a complete accounting of the compute used by every trained method.

Forking does not add rollouts. Each fork is a full control rollout, but it takes a slot of the group rather than adding to it, so the group size stays at G=8G=8 (§2.2). Appendix E specifies four natural rollouts per task, a fork at every skill decision in them, and further natural rollouts filling the rest of the group. Every trainable method thus uses eight rollouts per task over one epoch of the same 4,567 tasks, with 143 actor updates (4,000 tasks and 250 updates on the web), as all trained baselines use RL on the same task pools and budgets (Appendix E). The extra cost of a fork is only a container clone or a prefix replay.

Where the extra compute goes. Skill² devotes 12 of its 48 GPUs to the extractor and adds the probes (36,608 episodes, comparable to the 36,536 training episodes). Neither updates the actor: the extractor is a separate model, and no probe run enters any loss (§2.3). This compute is thus spent on learning the library. At deployment, Skill² runs one actor with one library, as SkillGate does.

Compute accounting, 9B terminal (tokens in millions; for SkillGate, the GPU hours include the construction of its fixed library, and the second token entry gives the tokens of that construction):

MethodGPU·hEnv. episodes (train / probe)Env. steps (turns)Input tokens (actor / extractor)Generated tokens (actor / extractor)Actor updates
GRPO73936,536 / –1,141k22,309 / –632 / –143
DAPO75436,536 / –1,147k22,345 / –624 / –143
Skill193136,536 / –1,193k23,690 / 419651 / 63143
SkillRL95036,536 / –1,229k24,192 / 527682 / 83143
SkillGate89836,536 / –1,176k23,568 / 9655 / 2143
Skill²1,17636,536 / 36,6082,002k36,320 / 5931,100 / 97143
SkillGate, 2 epochs1,73873,072 / –2,289k43,287 / 91,258 / 2286

Compute accounting, 9B web (same format):

MethodGPU·hEnv. episodes (train / probe)Env. steps (turns)Input tokens (actor / extractor)Generated tokens (actor / extractor)Actor updates
GRPO87832,000 / –458k5,976 / –229 / –250
DAPO90732,000 / –451k6,195 / –224 / –250
Skill11,07032,000 / –474k6,742 / 497239 / 79250
SkillRL1,09032,000 / –480k6,914 / 513243 / 83250
SkillGate1,04232,000 / –467k6,550 / 7235 / 2250
Skill²1,42132,000 / 32,000873k14,883 / 461441 / 89250
AuthorsOpenReview ↗

Response to Reviewer uSfg (2/4)

W1 (continued). "If SkillGate received a comparable budget of policy-gradient updates on the same tasks, the gap between them might shrink substantially. The claim that co-evolution outperforms one-sided training is the central contribution."

Thank you for this suggestion, which targets the central claim of our work. We are glad to test it directly.

Budget-matched baseline. To test whether a comparable policy-gradient budget on the same tasks closes the gap, we continue SkillGate for a second epoch over the same tasks with unchanged hyperparameters. It then receives twice the actor updates of Skill², nearly the same total number of environment episodes (73,072 vs. 73,144), and more GPU hours (1,738 vs. 1,176), environment turns (2,289k vs. 2,002k), and inference tokens, both input and generated:

9B terminalTB 2.1TBLiteTW-VerifiedIC-BashTerminal avg.
SkillGate (1 epoch)31.548.132.843.238.9
SkillGate (2 epochs)32.148.933.444.139.6
Skill²36.353.335.249.643.6

The second epoch adds only 0.7 points on the terminal average, and Skill² remains ahead by 4.0 points with half the actor updates.

The controlled comparison in the paper points the same way. Figure 1 (bottom right) trains the actor identically and varies only whether the extractor is trained: with a frozen extractor, the trained actor reaches 31.2 on TB 2.1, on par with SkillGate (31.5), and training the extractor as well raises it to 36.3. Together, these results indicate that the gain stems from co-evolution rather than from more updates of one side.

AuthorsOpenReview ↗

Response to Reviewer uSfg (3/4)

W2 / Q4. "Every cell lacks a standard deviation over training runs. Several conclusions, including the scale consistency claim (4B,9B) and the transfer margins, rest on differences of one to two points. … Were any results obtained with more than one training seed? If so, what is the seed variance?"

Thank you for this valuable suggestion. We have trained additional seeds for the main comparison and examined the scale and transfer claims against the resulting seed variation.

Training seeds. We have trained two additional seeds of both Skill² and SkillGate in the 9B terminal setting of Figure 1 (bottom right) and Figure 5, and evaluated each seed under the paper's protocol (seed 42 is the run reported in the paper):

9B terminalTB 2.1TBLiteTW-VerifiedIC-BashTerminal avg.
SkillGate, seed 4231.548.132.843.238.9
SkillGate, seed 4328.750.432.843.038.7
SkillGate, seed 4430.149.131.642.538.3
SkillGate, mean ± std30.1 ± 1.449.2 ± 1.232.4 ± 0.742.9 ± 0.438.7 ± 0.3
Skill², seed 4236.353.335.249.643.6
Skill², seed 4333.553.033.650.842.7
Skill², seed 4437.052.535.151.344.0
Skill², mean ± std35.6 ± 1.952.9 ± 0.434.6 ± 0.950.6 ± 0.943.4 ± 0.6

On the terminal average, the seed standard deviation is 0.3 points for SkillGate and 0.6 for Skill², whereas Skill² leads SkillGate by 4.7 points. Moreover, on each of the four benchmarks, the lowest Skill² seed remains above the highest SkillGate seed. Both deviations are also well below the seven-benchmark average lead of Skill² over SkillGate in Table 1 at either scale (3.3 points at 4B and 4.1 at 9B).

The scale consistency rests on margins far larger than this variation. It does not rely on the growth of the average gain from 4B to 9B (9.6 vs. 11.5 points). Instead, Skill² improves over the vanilla backbone by at least 6.7 points in each of the fourteen comparisons across backbones and benchmarks, well above the seed standard deviations reported above, and at 4B already surpasses vanilla 9B on the seven-benchmark average (37.5 vs. 33.5).

Transfer margins. To measure how much the transfer results depend on the training run, we give the libraries of the three Skill² seeds to the two frozen models of Figure 3 and evaluate them on the four terminal benchmarks (no library and + Skill1 library from Table 6):

Frozen model and libraryTB 2.1TBLiteTW-VerifiedIC-BashTerminal avg.
Qwen3.5-27B, no library36.356.333.953.545.0
Qwen3.5-27B, + Skill1 library37.457.634.954.246.0
Qwen3.5-27B, + Skill² library (seeds 42–44)41.1 ± 0.361.4 ± 1.038.4 ± 0.855.9 ± 0.949.2 ± 0.3
Muse Glimmer 30B, no library43.461.739.858.750.9
Muse Glimmer 30B, + Skill1 library44.662.840.659.852.0
Muse Glimmer 30B, + Skill² library (seeds 42–44)46.9 ± 0.465.7 ± 1.543.4 ± 0.561.8 ± 0.254.5 ± 0.5

On the terminal average, the standard deviation over the three libraries is 0.3 points on Qwen3.5-27B and 0.5 on Muse Glimmer 30B, whereas the Skill² library stays 3.2 and 2.5 points above Skill1's in the mean over seeds, and the library of every seed is at least 2.1 points above it.

The margin is also consistent across the settings of Table 6: over four frozen models, two of them from families other than the actor's, and all seven benchmarks, the library reported in the paper stays 1.9 to 3.5 points above Skill1's in each of the 28 cells, and its gain (3.0–4.6 points) exceeds that of Skill1's library (0.7–1.3 points) in every cell.

Reviewer_KaAKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Skill$^2$, a fully agentic framework that jointly evolves an LLM agent's policy and its external skill library. It alternately trains an actor to perform tasks and invoke skills, and an extractor to create and maintain reusable skills from execution trajectories. Both components use counterfactual credit assignment. Fork-GRPO trains the actor by comparing a skill invocation with an environment action from the same state. The extractor evaluates a revised library against the previous library using a fixed actor, combining revision-level credit with delayed credit from later skill use. Experiments on seven long-horizon terminal and web benchmarks show gains over conventional RL and skill-learning baselines at both 4B and 9B scales. Further results show that the evolved libraries can also benefit larger frozen models.

Critical strengths

The experimental results are encouraging. Skill$^2$ consistently outperforms the reported baselines across seven benchmarks and two model scales. Ablations support the value of counterfactual utility, skill-level credit assignment, and decision-point forking. Transfer results also suggest that some learned capability is captured in reusable skill artifacts. A key strength is the use of execution-based counterfactual feedback rather than relying primarily on LLM judgments of skill quality. Fork-GRPO compares skill invocations with alternative environment actions, while the extractor measures how library revisions affect a fixed actor’s task performance. This provides a promising way to connect policy optimization with skill evolution through measured execution outcomes.

Critical weaknesses

**Limitations of counterfactual credit assignment. **Fork-GRPO compares invoking a skill now with taking an environment action, while allowing later skill use in both branches. This provides feedback on the current invocation, but does not directly isolate skill selection from invocation timing. The paper would benefit from separate analyses of improvements in these two aspects. For the extractor, the authors acknowledge that macro and micro credit cannot identify the marginal contribution of individual edits. Although the ablation results demonstrate the effectiveness of these approximate signals, the paper does not sufficiently analyze how inaccurate credit assignment affects subsequent skill library evolution. A more detailed analysis would help clarify the scope and reliability of the proposed counterfactual credit assignment mechanisms.

Questions for authors

Q1: Could the authors clarify how the Fork-GRPO reward supports learning skill selection and invocation timing? Since the control branch may invoke the same skill later, what does the paired outcome difference measure? Separate analyses of skill selection and invocation timing would help clarify the mechanism.

**Q2: How does inaccurate edit-level credit assignment affect the long-term evolution of the skill library? **The authors acknowledge that macro and micro credit do not identify the marginal contribution of individual edits. While the ablation results demonstrate the usefulness of these signals, it remains unclear how often ineffective or harmful edits receive positive credit and how this affects subsequent library updates. Could the authors provide an analysis of such misattributed rewards and their impact on long-term skill evolution?

5 comments
AuthorsOpenReview ↗

Response to Reviewer KaAK (1/3)

Thank you very much for your careful review and insightful questions, and for the time you have devoted to our paper. We truly appreciate your feedback and are glad to address each point below.

Q1. "Could the authors clarify how the Fork-GRPO reward supports learning skill selection and invocation timing? Since the control branch may invoke the same skill later, what does the paired outcome difference measure? Separate analyses of skill selection and invocation timing would help clarify the mechanism."

Thank you for this thoughtful question. We are glad to clarify what the paired difference measures and how it trains both the timing and the choice of an invocation.

What δ\delta measures. The control branch takes an environment action a−∼πθ(⋅∣h;Aenv)a^-\sim\pi_\theta(\cdot\mid h;\mathcal A_{\mathrm{env}}) at h=ht⋆h=h_{t^\star} and thereafter follows the same policy to the end of the episode (§2.2), so it may invoke any skill later, including n⋆n^\star. Since R(τ−)R(\tau^-) depends on hh but not on n⋆n^\star,

E[δ∣h,n⋆]=Q(h,invoke(n⋆))−Venv(h),\mathbb E[\delta \mid h, n^\star] = Q\big(h, \mathrm{invoke}(n^\star)\big) - V_{\mathrm{env}}(h),

where Venv(h)V_{\mathrm{env}}(h) is the value of the policy's own continuation without invoking at this turn. Later invocation is thus part of the baseline, which is exactly what makes δ\delta a timing signal: if invoking later works equally well, δ\delta is zero in expectation and no net credit is written; if delaying is costly, δ>0\delta > 0 reinforces invoking now; and if invoking now hurts, δ<0\delta < 0. In other words, δ\delta measures how pivotal the decision is, rather than the absolute value of the skill.

Separating selection from timing. Let p(h)p(h) be the probability of invoking at hh and πinv(n∣h)\pi_{\mathrm{inv}}(n\mid h) the distribution over skills given an invocation, so that log⁡πθ(invoke(n)∣h)=log⁡p(h)+log⁡πinv(n∣h)\log\pi_\theta(\mathrm{invoke}(n)\mid h)=\log p(h)+\log\pi_{\mathrm{inv}}(n\mid h), and let Vinv(h)=∑nπinv(n∣h)Q(h,invoke(n))V_{\mathrm{inv}}(h)=\sum_n\pi_{\mathrm{inv}}(n\mid h) Q(h,\mathrm{invoke}(n)). In expectation over n⋆∼πinv(⋅∣h)n^\star\sim\pi_{\mathrm{inv}}(\cdot\mid h), the gradient that Eq. 3 places on the decision tokens of the invoking rollout decomposes as

E[∇log⁡πθ(invoke(n⋆)∣h)δ]=∇log⁡p(h)[Vinv(h)−Venv(h)]⏟whether to invoke now+∑nπinv(n∣h)∇log⁡πinv(n∣h)[Q(h,invoke(n))−Vinv(h)]⏟which skill to invoke.\mathbb E\big[\nabla\log\pi_\theta(\mathrm{invoke}(n^\star) \mid h) \delta\big] = \underset{\text{whether to invoke now}}{\underbrace{\nabla\log p(h) \big[V_{\mathrm{inv}}(h) - V_{\mathrm{env}}(h)\big]}} + \underset{\text{which skill to invoke}}{\underbrace{\textstyle\sum_n \pi_{\mathrm{inv}}(n \mid h) \nabla\log\pi_{\mathrm{inv}}(n \mid h) \big[Q(h, \mathrm{invoke}(n)) - V_{\mathrm{inv}}(h)\big]}}.

The first term compares invoking with not invoking at this turn (timing), and the second compares each skill with the policy's own mixture of skills (selection). Since the two terms use different baselines, they are separated in expectation, although a single δ\delta contributes to both. The −δ-\delta on the control branch falls on an environment action, which contains no skill name, and thus leaves this comparison among skills unchanged.

AuthorsOpenReview ↗

Response to Reviewer KaAK (2/3)

Q1 (continued). "Separate analyses of skill selection and invocation timing would help clarify the mechanism."

Thank you for this helpful suggestion. As suggested, we analyze the two aspects separately from an empirical perspective.

Fork pairs by what the control branch does next. We classify the 11,450 state-matched fork pairs of the 9B terminal run by the first later invocation in the control branch:

Control branch after t⋆t^\starShareMean δ\deltaδ=0\delta=0
Never invokes a skill56.6%0.09568.9%
Invokes n⋆n^\star at the next turn18.5%0.00993.9%
Invokes n⋆n^\star later18.0%0.05976.2%
First invokes another skill6.9%0.03971.9%

When the control branch never invokes a skill, δ\delta reflects the full value of the invocation, and its mean is the largest (0.095). When the control branch invokes n⋆n^\star at the next turn, it only delays the invocation by one turn, and δ=0\delta=0 in 93.9% of pairs, so no spurious credit is written; when it invokes n⋆n^\star only later, the mean δ\delta is larger (0.059 vs. 0.009), which is the timing signal. When the control branch first invokes another skill, the mean δ\delta remains positive (0.039), indicating that the actor's choice outperforms the alternative the policy would otherwise take.

Test-time interventions on the final 9B actor (TB 2.1, pass@1):

Test-time interventionpass@1
None (Skill²)36.3
Replace each invoked skill with an alternative sampled from the actor31.9
Replace each invoked skill with a random one27.6
Delay the first invocation by 1 / 2 / 4 turns35.4 / 33.5 / 30.1

Replacing the actor's choice with its own alternative costs 4.4 points, and with a random skill 8.7, indicating that the actor has learned which skill to invoke. The cost of delaying grows with the delay, from 0.9 points at one turn to 6.2 at four, indicating that it has also learned when to invoke.

Scope. Fork-GRPO forks only where the actor invokes. It does not fork states where an invocation was omitted, since every turn without an invocation would then become a fork point, nor a branch that invokes a different skill, which would double the forks per decision within the group of eight. Omitted invocations are still trained by the group-relative advantage, and the rise of skill use from 2.3% to 44.7% (Figure 4) is consistent with the paired credit generalizing across similar states.

AuthorsOpenReview ↗

Response to Reviewer KaAK (3/3)

Q2. "How does inaccurate edit-level credit assignment affect the long-term evolution of the skill library? … it remains unclear how often ineffective or harmful edits receive positive credit and how this affects subsequent library updates. Could the authors provide an analysis of such misattributed rewards and their impact on long-term skill evolution?"

Thank you for this important question. We are glad to analyze how the approximate edit-level credit behaves and how it affects the long-term evolution of the library.

Nature of the approximation. Within a round, the extractor's edits are applied in order to one draft (Eq. 7), the candidate is the end state of this sequence, and UkU_k is its terminal reward. Sharing AkmacA_k^{\mathrm{mac}} equally over Rk\mathcal R_k therefore treats the round as REINFORCE treats a multi-step episode with a terminal reward: it requires no estimate of each edit's contribution, at the cost of per-edit resolution and higher variance. Micro credit, collected from the invocation pairs of actor training at no extra cost, plays the same role for the lineage of edits behind a certified skill version.

How misattributed credit is corrected. A harmful edit that receives positive macro credit can still be corrected. If invoking its skill hurts the actor, the edit receives negative micro credit, and the actor learns to invoke the skill less until it is retired at capacity. The failed trajectories that load the skill also enter the next maintenance sets, and repairing it earns a positive reward, since Uk+1U_{k+1} is measured against the library that contains it.

Empirical analysis. For every UPDATE edit of the 9B terminal run whose old and new versions were both invoked, we take as its effect Δg\Delta g the change in its skill's invocation gain from the replaced version to the new one, and call it harmful or helpful when Δg\Delta g is distinguishable from zero. The pairs of the round in which an edit received its micro credit are excluded from Δg\Delta g, so that the effect and the credit are measured on different pairs. Since shifting all credit by a constant leaves the expected update unchanged, we report how often an edit is credited above the average edit:

Effect of the editEditsMacro credit above averageMicro credit above average
Harmful24947.0%7.2%
Neutral73448.4%29.0%
Helpful37746.7%61.5%

At the macro level, harmful edits are credited above average about as often as helpful ones (47.0% vs. 46.7%), as expected for credit shared across a revision. At the skill level, only 7.2% of harmful edits are credited above average, against 61.5% of helpful ones, and this credit is concentrated: 63.2% of certified units hold a single edit. Over time, among edits with a measurable effect, the share of harmful ones falls from 41.9% in the adoption phase to 36.0% in the refinement phase, indicating that misattributed credit does not accumulate.

These findings are consistent with the training dynamics reported in the paper. Figure 4 shows no cumulative degradation: the training reward rises from 78.6% to 88.5%, and the utility is non-negative in 84% of the last 43 rounds. Table 7 shows how much the precision of credit matters: broadcasting credit to uncertified edits costs 3.0 points and settling it within the round 1.9, yet even the weakest variant remains above SkillGate on average (42.0 vs. 40.9).

Thank you again for your thoughtful and constructive questions, which helped us analyze the credit assignment more precisely. We hope these responses address your concerns, and we would be very happy to discuss any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for handling our submission, and the three reviewers (KaAK, uSfg, Cb1A) for their time, careful reading, and insightful, constructive feedback. We have answered each reviewer with the complete new results and incorporated the key ones into the revised paper, where all changes are highlighted in blue.


Strengths recognized by the reviewers

We are deeply encouraged that the reviewers recognize:

  • A real and well-motivated problem. The paper "is based on a real gap between skill learning and deployment" (uSfg), and the framework "addresses a meaningful limitation of existing skill-learning methods" (Cb1A).
  • A principled learning signal. "A key strength is the use of execution-based counterfactual feedback rather than relying primarily on LLM judgments of skill quality" (KaAK).
  • Strong empirical evidence. Skill² "consistently outperforms the reported baselines across seven benchmarks and two model scales" (KaAK), "the claim is well supported" (uSfg), and the evaluation is "convincing" (Cb1A).

Main concerns and how we have addressed them

1. Compute fairness and training variance · uSfg

Concern. Does the gain over SkillGate stem from a larger budget, and does it hold beyond one training run?

  • Forks add no rollouts. In the terminal domain, all trained methods use the same 36,536 training episodes and 143 actor updates (32,000 and 250 on the web), and neither the extractor nor the probes update the actor.
  • Budget-matched baseline. SkillGate trained for a second epoch, with twice the actor updates and more GPU hours, environment turns, and tokens than Skill², reaches 39.6 on the terminal average and still trails Skill² by 4.0 points, indicating that the gain stems from co-evolution rather than from more updates.
  • Training seeds. Over three seeds, Skill² leads SkillGate by 4.7 points with seed standard deviations of at most 0.6, and on every benchmark its lowest seed exceeds the highest SkillGate seed. Each seed's library also exceeds Skill1's by at least 2.1 points on two frozen models (terminal average).

In the revised paper: §3.1 (training budget); new Appendix J (budget-matched baseline and training seeds).

2. What the decision credit measures · KaAK Cb1A uSfg

Concern. What does the paired difference measure, and how does the actor choose among skills?

  • Derivation. The paired difference estimates the advantage of invoking a skill now over the policy's own continuation, which may invoke later, and its expected gradient separates whether to invoke from which skill, comparing each skill with the actor's own mixture.
  • Test-time interventions. Replacing the chosen skill costs 4.4 points (the actor's own alternative) or 8.7 (a random skill), and delaying the first invocation by one to four turns costs 0.9 to 6.2, indicating that the actor has learned both which skill to invoke and when.
  • Fork diagnostics (uSfg). 94.6% (terminal) and 92.1% (web) of fork pairs are state-matched and receive the paired credit.

In the revised paper: §2.2; new Appendix K (derivation and test-time interventions).

3. Reliability of the extractor's edit-level credit · KaAK Cb1A

Concern. How often do harmful edits receive positive credit, and should superseded edits share micro credit?

  • Skill-level credit separates harmful from helpful edits. Only 7.2% of harmful edits are credited above average, against 61.5% of helpful ones.
  • Misattributed credit does not accumulate. The share of harmful edits falls from 41.9% (adoption) to 36.0% (refinement).
  • Superseded edits (13.0% of micro-credited edits, median wait of 4 rounds) share the credit of the version they led to, whose certification is the only evidence about their lineage.

In the revised paper: §2.3; Appendix K (edit-level credit).

4. Unconditional release of the candidate library · Cb1A

Concern. Why release a revision whose utility is negative?

  • UkU_k is the extractor's reward rather than an acceptance test: a negative value penalizes similar revisions through the policy gradient, and withholding the candidate would keep its edits from being certified. In Figure 4, the training reward still rises from 78.6% to 88.5%.

In the revised paper: §2.4.


We are sincerely grateful to the Area Chair and the reviewers for their time and effort, which have helped strengthen the paper, and we would be very happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.