acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

Abstract

Large language model agents increasingly rely on external skills to solve complex tasks, where skills act as modular units that extend their capabilities beyond what parametric memory alone supports. Existing skill-based RL methods assume external skills either accumulate as persistent guidance or internalized into the policy, eventually leading to zero-skill inference. We argue this assumption is overly restrictive, since with limited parametric capacity and uneven marginal contribution across skills, the optimal active skill set is non-monotonic, task- and stage-dependent. In this work, we propose SLIM, a framework of dynamic **S**kill **LI**fecycle **M**anagement for agentic reinforcement learning (RL), which dynamically updates the active external skill set as the policy evolves. Specifically, SLIM estimates each active skill’s marginal external contribution through leave-one-skill-out validation, then applies three lifecycle operations: retaining high-value skills, retiring skills whose contribution becomes negligible after sufficient exposure, and expanding the skill bank when persistent failures reveal missing capability coverage. Experiments show that SLIM outperforms the best baselines by an average of 7.1% points across ALFWorld and SearchQA. Results further indicate that policy learning and external skill retention are not mutually exclusive: some skills are absorbed into the policy, while others continue to provide external value, supporting SLIM as a more general paradigm for skill-based agentic RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent Skill, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JLUoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SLIM treats an agent's external skill set as something to manage during RL training, not a library that only grows or only shrinks. Its main contribution is a skill lifecycle that can add and remove skills over time. GRPO updates alternate with periodic audits. In each audit, every skill gets a leave-one-out "marginal external contribution" score (MEC, Eq. 3) on routed dev tasks, smoothed with an EMA. Based on that score the skill is kept or retired, and new skills are added where coverage is missing. The paper also gives a theoretical argument for when audit decisions are reliable (Appendix A). With Qwen3-4B, SLIM† reaches 87.5 on ALFWorld and 41.0 on SearchQA. Table 2 has ablations, and the appendix adds seeds, other backbones and WebShop.

Critical strengths

S1. The reframing is clean. "Accumulate" and "eliminate" become special cases of one lifecycle controller, and the controls target the obvious confounds. Fixed Active Set Size (75.6) covers "it's just a shorter prompt", and the frozen Qwen3-4B creator (86.7, Table A7) covers "it's just distillation from o3".

S2. The endpoint behavior holds up across settings. SearchQA ends with an almost empty set, while ALFWorld and WebShop keep compact ones, and set sizes are stable across seeds and backbones (Table A5). That gives the "task-dependent boundary" claim some real support.

Critical weaknesses

W1. The ALFWorld numbers, which carry most of the average gain, don't add up. The Avg. entries and the Table A4 run means are all multiples of 1/128, which points to a 128-episode test set rather than the 134-game official unseen split. The prompt baselines and SLIM without skills all fit one per-type split (35/5/32/13/23/20). The headline SLIM† row does not: 92.9 is not k/35, 78.3 is not k/13, and so on. That is odd, since SLIM and SLIM† use the same policy and test split. EvolveR's Avg. (39.8) also disagrees with its own per-type entries (about 46.1). On top of this, Look has 5 episodes, the bootstrap CI for the main gap is roughly ±9 points, and the Table 2 ablations are single runs. I need to know how every Table 1 number was produced, and I'd like seeds for the ablations.

W2. I can't tell what MEC adds over Skill0. Appendix B.2 describes Skill0's filter as the performance difference with and without each skill, which is essentially Eq. 3. SLIM also drops the zero target and adds expansion, and Table 2 doesn't separate these pieces. Random Audit is a weak control. At 32 dev tasks per audit, a skill's routed subset is a handful of episodes, so the Hoeffding-based bound in Appendix A is vacuous at the thresholds used. Decisions hinge on one or two episodes. The missing controls are a Skill0-style filter with SLIM's schedule and expansion, and SLIM with frequency-based retirement.

Questions for authors
  1. [Clarification] How were the ALFWorld entries in Table 1 computed (episodes per type, single run vs. average)? Why don't the SLIM† and several baseline rows fit the split the other rows imply? This answer matters most for my Soundness score.
  2. [Clarification] Why 128 test episodes rather than the 134 official unseen games, and which split does the 32-task dev set come from?
  3. [Clarification] What is the typical number of routed dev episodes per audited skill, and how many retire/retain decisions rest on a one- or two-episode difference?
  4. [Suggestion] Could you run the two controls in W2? If MEC clearly beats both, my concern largely goes away.
  5. [Clarification] SkillRL uses top-6 retrieval with KL and SLIM uses top-3 without. Does the 12.5-point gap survive matched settings?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/5)

We thank Reviewer JLUo for the rigorous review and constructive suggestions. We address every question below and have incorporated the unified evaluation table and new control experiments into the revised paper.

Review (W1 - Part 1 & Q1): The ALFWorld numbers, which carry most of the average gain, don't add up.

Response: Thank you for catching this aggregation inconsistency. We have thoroughly audited our evaluation logs and confirm that all methods in Table 1 are evaluated on the exact same 128 test episodes, with per-category counts of Pick at 35, Look at 5, Clean at 32, Heat at 13, Cool at 23, and Pick2 at 20, using a single evaluation run with one rollout per episode partitioned into 8 equal batches of 16 episodes.

The discrepancy in per-category decimals arises because two different category-aggregation formulas are used across the standalone inference script and the online-RL evaluation logger. Specifically, the standalone evaluation script used for prompt baselines and the post-training no-skill pass of SLIM computes pooled episode-level category success rates SRcpool=∑bsbc∑bnbc\mathrm{SR}_c^{\mathrm{pool}} = \frac{\sum_b s_{bc}}{\sum_b n_{bc}}, which always equal integer numerators divided by the category totals 35,5,32,13,23,35, 5, 32, 13, 23, and 2020. In contrast, the batched online-RL evaluator used for SLIM$^\dagger$, GRPO, GRPO$^\dagger$, EvolveR, SkillRL$^\dagger$, and Skill0 computes the category success rate within each 16-episode batch and averages those rates across the batches Bc\mathcal{B}_c containing category cc, namely SRcbatch=1∣Bc∣∑b∈Bcsbcnbc\mathrm{SR}_c^{\mathrm{batch}} = \frac{1}{|\mathcal{B}_c|}\sum_{b\in\mathcal{B}_c}\frac{s_{bc}}{n_{bc}}. When category counts nbcn_{bc} vary across batches, SRcbatch\mathrm{SR}_c^{\mathrm{batch}} differs slightly from SRcpool\mathrm{SR}_c^{\mathrm{pool}}; for instance, batch counts [2,1,1,1][2,1,1,1] with successes [1,0,0,1][1,0,0,1] give a batch-averaged score of 37.5% versus a pooled episode score of 2/5=40.0%2/5 = 40.0\%. Meanwhile, because all 8 batches contain exactly 16 episodes, the batch-averaged overall score and episode-pooled overall score are mathematically identical and always equal total successes divided by 128.

For EvolveR, its per-category entries in Table 1 are computed by the same 8-batch evaluator from the raw per-batch outcomes below, which sum to 59/128=46.1%59/128 = 46.1\% overall success, whereas 39.8 in Table 1 is a transcription error from an earlier checkpoint log.

CategoryPer-Batch Episode Counts nbcn_{bc}Per-Batch Successes sbcs_{bc}Batch-Averaged SRcbatch\mathrm{SR}_c^{\mathrm{batch}} (%)Episode-Pooled SRcpool\mathrm{SR}_c^{\mathrm{pool}} (%)
Pick[8, 5, 5, 4, 4, 3, 3, 3][7, 1, 5, 0, 4, 1, 3, 3]67.668.6 (24/35)
Look[0, 0, 0, 1, 1, 0, 2, 1][0, 0, 0, 0, 0, 0, 1, 1]37.540.0 (2/5)
Clean[5, 5, 4, 3, 3, 8, 2, 2][2, 5, 3, 2, 3, 1, 0, 0]49.350.0 (16/32)
Heat[1, 1, 1, 1, 2, 1, 2, 4][0, 0, 0, 0, 0, 1, 0, 1]15.615.4 (2/13)
Cool[1, 4, 3, 5, 3, 2, 3, 2][1, 1, 0, 4, 0, 0, 1, 1]36.034.8 (8/23)
Pick2[1, 1, 3, 2, 3, 2, 4, 4][0, 0, 0, 2, 2, 2, 0, 1]36.535.0 (7/20)

In the revised paper, we have unified all rows in Table 1 under the same batch-averaged aggregation rule from the raw evaluation logs. This unification leaves all other overall Avg. scores and all experimental conclusions completely unchanged. Thank you again for pointing out this critical issue.

Review (W1 - Part 2): ... On top of this, Look has 5 episodes ...

Response: Look contains 5 episodes because we preserve the exact 128-game test set from the released SkillRL [1] benchmark. Because Look accounts for only 5/128=3.91%5/128 = 3.91\% of the overall test set, one additional solved Look episode changes the overall Avg. score by only 0.78 percentage points. In Table 1, SLIM$^\dagger$ outperforms SkillRL$^\dagger$ by 12.5 points, corresponding to 16 additional solved episodes out of 128. Even in the extreme case where all 5 Look episodes contribute to this gap, at least 11 additional solved episodes come from the remaining categories, led by substantial improvements on Clean and Cool. Thus, variation on Look cannot account for SLIM$^\dagger$'s overall performance gain.

AuthorsOpenReview ↗

Rebuttal by Authors (2/5)

Review (W1 - Part 3): ... the bootstrap CI for the main gap is roughly ±9 points ...

Response: Table A6 quantifies sampling variation across the 128 binary test episodes. At the observed success rates on N=128N=128 binary outcomes, the standard independent-proportion standard error is SE(p^1−p^2)≈p1(1−p1)+p2(1−p2)128\mathrm{SE}(\hat{p}_1 - \hat{p}_2) \approx \sqrt{\frac{p_1(1-p_1)+p_2(1-p_2)}{128}}, which gives a 95% confidence interval half-width of 9.4 to 9.5 percentage points, matching our bootstrap intervals. Crucially, both 95% confidence intervals strictly exclude zero, with gains of +13.3 points and a 95% CI of [+3.9,+22.7][+3.9, +22.7] over Skill0, and +12.6 points with a 95% CI of [+3.1,+21.9][+3.1, +21.9] over SkillRL$^\dagger$.

Furthermore, Table A4 separately measures training stability across 3 independent training seeds. With Qwen3-4B, SLIM$^\dagger$ achieves 88.02±1.63%88.02 \pm 1.63\%, compared to 76.30±1.63%76.30 \pm 1.63\% for SkillRL$^\dagger$ and 73.44±0.78%73.44 \pm 0.78\% for Skill0, maintaining mean advantages of +11.72 and +14.58 points. With Qwen3-8B, SLIM$^\dagger$ achieves 89.32±1.19%89.32 \pm 1.19\% versus 85.68±1.19%85.68 \pm 1.19\% for SkillRL$^\dagger$. On SearchQA, the bootstrap 95% CI over SkillRL$^\dagger$ is much tighter at [+2.35,+3.52][+2.35, +3.52]. Together, these results confirm that the gains are consistent across evaluation episodes, independent training seeds, and model scales.

Review (W1 - Part 4): ... and the Table 2 ablations are single runs. I'd like seeds for the ablations.

Response: Following your suggestion, we have extended all ALFWorld ablation variants in Table 2 to 3 independent training seeds and report the mean success rate and sample standard deviation below.

MethodALFWorld Success Rate (Mean ±\pm Std %)
SLIM$^\dagger$ (Full)88.02 ±\pm 1.63
SLIM w/o Retirement73.70 ±\pm 1.19
SLIM w/o Expansion79.43 ±\pm 0.90
Random Audit67.19 ±\pm 1.56
Fixed Active Set Size75.00 ±\pm 0.78

The 3-seed results confirm all ablation conclusions from Table 2, with component gaps of 8.6 to 20.8 points that far exceed the run-to-run standard deviations. We have updated Table 2 in Section 6.3 of the revised paper with these multi-seed ablation results.

AuthorsOpenReview ↗

Rebuttal by Authors (3/5)

Review (W2 - Part 1 & Q3): What is the typical number of routed dev episodes per audited skill, and how many retire/retain decisions rest on a one- or two-episode difference?

Response: We have analyzed all 126 individual MEC audit records from the main ALFWorld training run, excluding the separately conducted general-group audits. Across these 126 audits, each audited skill is associated with an average of 19.7 routed evaluation records, with a median of 21.5 and a range of 7 to 31, including repeated evaluations on the same underlying development tasks.

To directly test sensitivity to one- or two-episode outcome flips, we retrospectively enumerate all worst-case perturbations to at most 1 or 2 binary rollout outcomes across the paired with-skill and skill-disabled evaluations for all 126 audit decisions, holding routing, exposure, and prior audit history fixed.

Outcome Perturbations per AuditTotal Audit Decisions ChangedObserved Retirements Reversed
Up to 1 episode outcome flip3 / 126 (2.4%)0 / 33 (0.0%)
Up to 2 episode outcome flips5 / 126 (4.0%)2 / 33 (6.1%)

Thus, 97.6% and 96.0% of all lifecycle decisions remain unchanged under worst-case one- and two-episode perturbations, and all observed retirements are invariant to single-episode flips due to EMA smoothing with α=0.1\alpha=0.1, minimum exposure nmin⁡=30n_{\min}=30, and patience p=3p=3. We have included this perturbation analysis in Appendix D.4 of the revised paper.

Review (W2 - Part 2): I can't tell what MEC adds over Skill0. Appendix B.2 describes Skill0's filter as the performance difference with and without each skill, which is essentially Eq. 3. SLIM also drops the zero target and adds expansion, and Table 2 doesn't separate these pieces...

Response: Thanks for raising this point. First, Skill0 and SLIM evaluate fundamentally different counterfactual baselines. Skill0 evaluates each skill individually against a zero-skill baseline on a matched sub-task and ranks skills under a forced shrinking schedule [6,3,0][6, 3, 0] toward zero-skill inference. In contrast, SLIM's leave-one-skill-out MEC removes only the audited skill ss from the current active set At\mathcal{A}_t to form At∖{s}\mathcal{A}_t \setminus \{s\} on its routed subset Vt(s)\mathcal{V}_t(s), measuring the marginal value of ss conditioned on the remaining active skills and refreshing baselines sequentially after each removal.

Second, Table 2 already isolates the lifecycle without expansion via SLIM w/o Expansion, which makes zero creator calls and still achieves 79.43±0.90%79.43 \pm 0.90\% across 3 seeds on ALFWorld, outperforming Skill0 at 73.44±0.78%73.44 \pm 0.78\% across 3 seeds (Table A4) by +5.99 points. Our new matched controls in Review (W2 - Part 4 & Q4) below further isolate the exact contribution of leave-one-skill-out MEC when paired with the identical SLIM schedule and expansion module.

Reviewer_5D3sOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLMs often use “skills” in the prompt during reinforcement learning training and at inference to guide agent behavior. All the current prompt update paradigms treat this as a monotonic task (update only in one direction). The paper proposes SLIM, an agentic RL framework that tells which skills can be retained, retired, or added as the model policy changes. The framework calculates the effect of each skill using the leave-one-skill-out validation technique to identify and remove skills from the prompt. They categorize the operation on the skills as: i) High value skills which are retained in the prompt, ii) retiring skills as their contribution becomes negligible as the training evolves and iii) expanding the skill bank of the persistent failures that reveals missing capability coverage. They evaluate their framework on two environments ALFWorld and SearchQA with varied skill sets and train a Qwen3-4B model using GRPO on these tasks. SLIM reaches 87.5% on ALFWorld (vs. 75.0% for SkillRL) and 41.0% on SearchQA (vs. 39.3% for Skill0). They also extend the evaluation to Qwen3-8B and the WebShop environment.

Critical strengths
  1. Clear distinction between the current skill update paradigm as monotonic (SkillRL and Skill0) and the motivation for a new RL framework is very strong. The Fig. 3 contrast (38→73 vs 38→0 vs 38→46→21) shows it well.

  2. Clear distinction from o3 model distillation. With Qwen3-4B creator, the RL framework reaches comparable performance (86.7, against 87.5 with o3) to showcase that the gains are not from the distilled model but rather the RL framework.

  3. Robust evidence - the authors run a bootstrap CI, train on 3 different seeds, a sensitivity sweep and an initialization study (empty/weak/noisy banks to add more supporting evidence to their analysis. They show similar performance using Qwen3-8B and on WebShop environment as well.

Critical weaknesses

"1. Given that ALFWorld audits after every 10 steps and is trained for 120 steps we have 12 audit stages and in which 4 skills are audited from the TopK retrieval. And given that for a skill to be retired it should score low on 3 successive audits. So on a training run the maximum slots audited are 48 and thus the maximum skills that can be retired are 16. The paper reports 33 and 46 skills retired in the original and the noisy runs which contradicts their approach. I feel defining l_t(s) and u_t(s) better would make it more clear.

  1. Although the paper tells us why M=4 is chosen using a cost rationale, it doesn’t dive deep into the exact reason or ablation behind choosing M=4. Given that M plays an important role in the RL framework, ablating it is essential.

  2. On ALFWorld, SLIM without skills at inference (72.7) is below Skill0 (74.2), while SLIM† reaches 87.5, so the headline gain is realized through inference-time skill conditioning. Table A2 shows that SLIM’s final skills add little to an untrained model (63.3 → 65.8), and GRPO† gains only 1.6 over GRPO, which suggests the gain comes from policy–skill co-training rather than from the final skill set itself. The paper should isolate whether the lifecycle (retain/retire/expand) is what matters."

Questions for authors
  1. What is the per task type “n” in Table 1, were all the numbers evaluated on identical game IDs ?
  2. Please reconcile 33 / 46 retirements (Table A3) with 12 audits × ≤4 task-specific skills and p=3. What exactly increments ℓ_t(s) and u_t(s)?
  3. Is there any mechanism to reactivate retired skills? Given that a skill once retired is never in the loop, thinking about this is essential. Maybe a small justification of why using this skill in the prompt would not be required would justify the statement given that there is a chance of catastrophic forgetting.
  4. Why α=0.9?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank Reviewer 5D3s for the detailed review and constructive questions. We address all concerns below and have incorporated these clarifications and ablations into the revised paper.

Review (W1&Q2): Please reconcile 33 / 46 retirements (Table A3) with 12 audits × ≤4 task-specific skills and p=3. What exactly increments ℓ_t(s) and u_t(s)?

Response: Thank you for pointing out this ambiguity in our description of the audit budget. First, the phrase "at most M=4M=4 skills" in Section 5 is imprecise shorthand for our candidate selection rule, which selects task-specific skills in the top M=4M=4 positive usage-frequency tiers, including all task-specific skills tied in usage count within those top 4 frequency values rather than a hard cap of 4 individual skills. In addition, general skills are audited as a separate group (Appendix B.1) and excluded from the task-specific frequency ranking so that high-frequency general skills do not crowd out task-specific candidates. Furthermore, an initial lifecycle audit is executed at step 0 prior to GRPO updates, followed by audits every 10 steps through step 120, giving 13 audit cycles in total. As shown in Algorithm 1, skill expansion is triggered directly from routed validation failure buckets after the candidate audit loop and does not consume an MEC audit slot; newly expanded skills only count toward MEC audits when selected in later audit cycles. Consequently, 12×412 \times 4 does not bound the total number of skill audits; across the 13 audit cycles, SLIM conducts 126 individual candidate MEC audits in the original-bank run and 143 in the noisy-bank run, excluding the separately conducted general-group audits.

Second, regarding the exact update rules for ut(s)u_t(s) and ℓt(s)\ell_t(s), the cumulative exposure ut(s)u_t(s) increments by 1 each time skill ss is routed to a development task during ordinary validation passes, excluding counterfactual leave-one-skill-out reruns. The low-contribution streak ℓt(s)\ell_t(s) updates only when skill ss undergoes a fresh MEC audit, where it increments by 1 if the updated EMA satisfies Δˉt(s)<τretire\bar{\Delta}_t(s) < \tau_{\mathrm{retire}}, resets to 0 if Δˉt(s)≥τretire\bar{\Delta}_t(s) \ge \tau_{\mathrm{retire}}, and remains unchanged during audit cycles where ss is not audited. Routing or baseline refreshes alone do not increment ℓt(s)\ell_t(s). Because auditing starts at step 0, a skill audited at steps 0, 10, and 20 can reach ℓt(s)=3\ell_t(s) = 3 and be retired as early as step 20.

Third, as noted in Appendix D.3, Table A3 counts cumulative retired skills over the entire training run, which includes newly expanded skills that are subsequently retired in later cycles rather than only initial skills. We have clarified the tie-inclusive frequency-tier selection rule, step-0 audit initialization, and separate general-skill accounting in Section 5 and Appendix B.1 of the revised paper.

Review (W2): Although the paper tells us why M=4 is chosen using a cost rationale, it doesn’t dive deep into the exact reason or ablation behind choosing M=4. Given that M plays an important role in the RL framework, ablating it is essential.

Response: Thanks for this suggestion. The budget MM controls how broadly SLIM audits the active skill set at each lifecycle checkpoint. We ablate M∈2,4,8M \in \\{2, 4, 8\\} on ALFWorld while keeping the policy initialization, training protocol, retrieval, audit interval, lifecycle thresholds, and expansion rules identical.

Audit Budget MMALFWorld Success Rate (%)
M=2M = 274.21
M=4M = 4 (Default)87.50
M=8M = 889.10

As shown in the table, the audit coverage budget has a clear impact when set too small, as reducing MM from 44 to 22 substantially lowers accuracy because auditing too few candidates per cycle leaves redundant or noisy skills in the active pool for longer periods. Once MM is large enough to cover the primary routed skills, performance becomes much less sensitive. Specifically, analysis of our training logs shows that skill routing frequencies follow a concentrated power-law distribution where the top M=4M=4 frequency tiers already account for about 80% of routed skill usages, whereas increasing the budget to M=8M=8 more than doubles the audit runtime cost for only a modest accuracy gain. Therefore, we select M=4M=4 to ensure sufficient audit coverage while keeping validation overhead low. We have added this ablation to Appendix D of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Review (W3): On ALFWorld, SLIM without skills at inference is below Skill0, while SLIM† reaches 87.5, so the headline gain is realized through inference-time skill conditioning. Table A2 shows that SLIM’s final skills add little to an untrained model, and GRPO† gains only 1.6 over GRPO, which suggests the gain comes from policy–skill co-training rather than from the final skill set itself.

Response: Thank you for this insightful observation. We agree that on procedural environments like ALFWorld, SLIM optimizes a joint policy-and-skill system rather than a standalone zero-skill policy or a standalone static prompt library. Because Skill0 explicitly trains the policy toward zero-skill inference, its no-skill evaluation is slightly higher than SLIM without skills at 74.2% versus 72.7%, and transferring SLIM's final 21 skills to a frozen untrained policy provides a modest standalone improvement over the 38-skill initial bank (Table A2).

Crucially, however, within this policy-skill co-training paradigm, our controlled comparisons show that static co-training alone is far from sufficient, and dynamic lifecycle management contributes a large additional gain. Specifically, static policy-skill co-training without lifecycle updates corresponds to GRPO$^\dagger$ in Table 1, which reaches only 68.8%, gaining just 1.6 points over no-skill GRPO at 67.2% and trailing SLIM$^\dagger$ at 87.5% by 18.7 points. Similarly, co-training with expansion but without retirement in SLIM w/o Retirement (Table 2) and SkillRL$^\dagger$ (Table 1), as well as co-training under a fixed skill count with LRU replacement in Fixed Active Set Size (Table 2), all fall substantially below SLIM$^\dagger$ under their respective protocols. Therefore, while the final performance is realized through the co-adapted policy and active skill set working together, the retain-retire-expand lifecycle is essential for reaching 87.5% beyond what policy-skill co-training achieves on its own.

Review (Q1): What is the per task type “n” in Table 1, were all the numbers evaluated on identical game IDs ?

Response: The per-type episode counts in Table 1 are Pick with 35, Look with 5, Clean with 32, Heat with 13, Cool with 23, and Pick2 with 20, totaling 128 ALFWorld test tasks. All methods in Table 1, including SLIM and SLIM$^\dagger$, are evaluated on the exact same 128 game IDs.

Review (Q3): Is there any mechanism to reactivate retired skills? Given that a skill once retired is never in the loop, thinking about this is essential. Maybe a small justification of why using this skill in the prompt would not be required would justify the statement given that there is a chance of catastrophic forgetting.

Response: Thanks for raising this question. The current SLIM framework does not include a direct reactivation mechanism for retired skills; instead, when a persistent capability gap emerges on a task region covered by the development split, SLIM relies on failure-driven skill expansion via Eq. (6) to synthesize new support.

We avoid directly reactivating retired skills for two design reasons. First, skills retired due to negative or near-zero MEC are often overly broad, redundant, or poorly aligned with the evolving policy (Figure 5c), so re-inserting them risks reintroducing routing and context noise. Second, when the policy later struggles on a task region due to policy drift or partial forgetting, its current failure modes generally differ from those in early training, making a new skill generated from current failure traces more targeted than an old skill artifact. We note that this expansion-based recovery applies when regressions occur on task regions represented in the development split and reach the expansion trigger thresholds with Perf<τexpand\mathrm{Perf} < \tau_{\mathrm{expand}} and Nt≥nexpandN_t \ge n_{\mathrm{expand}}.

Review (Q4): Why α=0.9?

Response: Thank you for catching this wording ambiguity. The phrase "EMA coefficient 0.9" in Appendix B.1 refers to the retention weight on the previous smoothed estimate Δˉt−1(s)\bar{\Delta}_{t-1}(s), meaning 1−α=0.91-\alpha = 0.9 and α=0.1\alpha = 0.1 in the update rule Δˉt(s)=αΔt(s)+(1−α)Δˉt−1(s)\bar{\Delta}_t(s) = \alpha \Delta_t(s) + (1-\alpha)\bar{\Delta}_{t-1}(s) in Section 4.2. Our implementation uses α=0.1\alpha = 0.1 on the current-round observation and 0.90.9 on the historical EMA to smooth out single-audit sampling noise. We have updated Appendix B.1 in the revised paper to state α=0.1\alpha = 0.1 explicitly.

AuthorsOpenReview ↗

Kind Request for Your Feedback on Our Rebuttal

Thank you very much for your thoughtful review, as well as the valuable questions and constructive suggestions you provided. In our rebuttal and revised paper, we have done our best to address all of your concerns and clarifications. As the author-reviewer discussion period will end in about 1 day, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. If you have any remaining questions or would like further clarification on any point, please feel free to let us know and we will respond immediately. Thank you very much for your time and consideration!

Reviewer_g6nzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how the active external skill set of an LLM agent should evolve during agentic reinforcement learning. Prior skill-based RL methods either treat skills as persistent augmentation and keep growing the skill bank, or treat them as temporary scaffolds and remove them toward zero-skill inference. The authors argue this monotonic assumption is too restrictive, since parametric capacity is finite and skills differ in their marginal value to the evolving policy. They propose SLIM, a framework that couples GRPO policy updates with periodic lifecycle auditing of the active skill set. Each audited skill's marginal external contribution is estimated through leave-one-skill-out validation on a development split, smoothed with a moving average, and used to retain skills that still help, retire skills whose contribution has become negligible after sufficient exposure, and expand the bank when persistent failures reveal missing capability coverage. A supporting analysis bounds the performance loss of sequential retirements. Experiments on an embodied household benchmark and a search-augmented QA benchmark, extended with a second backbone and a web shopping benchmark, show consistent improvements over skill-based and RL baselines, with learned endpoints that keep a compact skill set on procedural tasks and shrink to near empty on search tasks.

Critical strengths
  1. The problem formulation is genuinely valuable. Both accumulation and forced internalization impose a monotonic trajectory on the skill set, and the paper cleanly reframes skill-based RL as a dynamic boundary management problem, with the two prior paradigms recovered as special cases. This conceptual contribution is well motivated by the tension between limited parametric capacity and the routing or context cost of large skill banks.

  2. The proposed mechanism is principled, and the paper convincingly shows that the gains come from this mechanism rather than from incidental effects. The ablations isolate contribution-aware decisions from random skill set perturbation and from simple prompt budget control, and the case study shows that lifecycle decisions track marginal contribution rather than selection frequency. The retention and retirement rules are also guarded by exposure and patience conditions that protect infrequent but useful skills.

  3. The empirical study is thorough. It covers multiple backbones and benchmarks, repeated independent runs with variance reporting, bootstrap checks of the performance gaps, sensitivity to skill initialization and to the choice of skill creator, transfer of the learned skill bank to policies that did not undergo the lifecycle training, explicit data leakage controls, and an analysis of audit overhead. The learned endpoints adapt sensibly to backbone capacity and task type, which supports the central claim rather than just the headline numbers.

Critical weaknesses
  1. The core signal is a local approximation with substantial tuning burden. Marginal contribution is estimated by removing one skill at a time, which ignores interactions among skills and depends on routing behavior on the development split. The full method introduces many thresholds and budgets governing retention, retirement, expansion, exposure, patience, and smoothing, which are tuned per benchmark, and the sensitivity study covers only one benchmark. The possibility that audit decisions overfit the development distribution is not examined.

  2. The headline comparison is partly confounded and the method adds nontrivial cost. The strongest skill-based baseline is run with different retrieval depth and with KL regularization while SLIM omits both, so some of the observed gap may not be attributable to lifecycle management alone. In addition, lifecycle auditing requires extra validation passes and repeated baseline refreshes, and the overhead analysis is mostly descriptive; the practicality of the approach for much larger skill banks is left open.

Questions for authors
  1. When skills interact or substitute for one another, how often does single-skill leave-one-out auditing misrank their value? Did you observe failure cases where sequential retirements removed jointly useful coverage that was later recovered by expansion?

  2. How sensitive are the learned endpoints and the final performance to the size and composition of the development split used for auditing, given that all lifecycle decisions are driven by routed statistics from that split?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank Reviewer g6nz for the positive feedback and constructive comments. We address all concerns below and have incorporated these clarifications and additional experiments into the revised paper.

Review (W1): The core signal is a local approximation with substantial tuning burden. The possibility that audit decisions overfit the development distribution is not examined.

Response: Thanks for raising this point. First, 7 core retrieval and lifecycle parameters are held fixed across all three environments and across both Qwen3-4B and Qwen3-8B backbones, with τretire=0.001\tau_{\mathrm{retire}}=0.001, p=3p=3, 1−α=0.91-\alpha=0.9, τexpand=0.40\tau_{\mathrm{expand}}=0.40, d=10d=10, K=3K=3, and τemb=0.45\tau_{\mathrm{emb}}=0.45 (Appendix B.1 and B.3). Between ALFWorld and SearchQA, 5 scale-dependent parameters are adjusted on a training-split subset to account for the different validation batch sizes of 3232 versus 512512 and episode horizons of 5050 versus 44 steps, while WebShop directly reuses the ALFWorld configuration without re-tuning (Appendix B.3). Within a chosen environment scale, Figure A1 shows that performance remains stable across broad ranges of the decision thresholds τkeep\tau_{\mathrm{keep}}, τretire\tau_{\mathrm{retire}}, and τexpand\tau_{\mathrm{expand}}, while our ablation on MM for Reviewer 5D3s (W2) shows that performance is insensitive once MM is large enough to cover the frequently routed skills.

Second, our evaluation protocol directly tests generalization beyond the development distribution. On ALFWorld, the 32 development tasks come from the seen split, whereas final evaluation uses the disjoint unseen split where SLIM$^\dagger$ reaches 87.5%. On SearchQA, the development split covers only 5 datasets, while the test split includes two unseen out-of-domain benchmarks, PopQA and Bamboogle (Appendix C), where SLIM$^\dagger$ achieves 40.4% and 36.0%, outperforming SkillRL$^\dagger$ at 36.9% and 31.2% and Skill0 at 38.6% and 32.8% (Table 1).

Third, Table A2 shows that transferring the 21 final skills learned by SLIM to frozen Zero-Shot and Few-Shot policies consistently improves unseen ALFWorld accuracy over the 38-skill initial bank across both settings, despite reducing the skill count by 45%. This confirms that the retained skills capture reusable procedural knowledge rather than overfitting to development routing noise.

Review (W2 - Part 1): The headline comparison is partly confounded and the method adds nontrivial cost...

Response: To isolate lifecycle management from retrieval depth and KL regularization, we evaluate a matched SkillRL control using the exact same retrieval cap K=3K=3 and policy-side KL coefficient β=0\beta=0 as SLIM, while keeping all other training, creator, and skill-conditioned evaluation settings identical.

MethodALFWorld (%)SearchQA (%)
SkillRL$^\dagger$ (Original)75.0038.10
SkillRL$^\dagger$ (Matched)70.3136.83
SLIM$^\dagger$ (Ours)87.5041.00

As shown in the table, enforcing K=3K=3 and β=0\beta=0 degrades SkillRL$^\dagger$'s performance on both benchmarks because its original configuration is tailored to its monotonic skill accumulation design. Since SkillRL$^\dagger$ continuously accumulates up to 73 active skills without retirement, reducing KK from 6 to 3 increases retrieval misses caused by redundant candidates, and removing KL regularization allows greater policy drift under noisy skill contexts. Therefore, the performance advantage of SLIM$^\dagger$ stems from dynamic lifecycle management rather than from retrieval depth or KL settings.

Review (W2 - Part 2): ... lifecycle auditing requires extra validation passes and repeated baseline refreshes, and the overhead analysis is mostly descriptive; the practicality of the approach for much larger skill banks is left open.

Response: Thanks for this question. In our shared training stack, total wall-clock time is dominated by online GRPO trajectory generation with n=8n=8 rollouts per prompt and policy gradient updates rather than periodic auditing. On ALFWorld, total training wall-clock time is 17.5 hours for Skill0, 19.2 hours for SkillRL$^\dagger$, and 20.1 hours for SLIM$^\dagger$, where leave-one-skill-out auditing and baseline refreshes account for only 2.4 hours, or 12% of total runtime.

Regarding scalability to much larger skill banks, the auditing cost of SLIM does not scale with the global skill bank size ∣S∣|\mathcal{S}| (Appendix D.8, Table A14). Dense embedding retrieval filters candidates in milliseconds prior to rollout, and lifecycle auditing only evaluates recently routed candidates within the fixed budget. Therefore, even if the global skill bank ∣S∣|\mathcal{S}| scales to thousands of skills, the number of counterfactual validation rollouts per audit cycle remains strictly bounded by the candidate budget and retrieval cap.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Review (Q1): When skills interact or substitute for one another, how often does single-skill leave-one-out auditing misrank their value? Did you observe failure cases where sequential retirements removed jointly useful coverage that was later recovered by expansion?

Response: First, when two skills s1s_1 and s2s_2 act as redundant substitutes, auditing s1s_1 while s2s_2 is still active can assign a near-zero MEC to s1s_1. To prevent retiring both skills in the same cycle, SLIM commits retirements sequentially and immediately refreshes skill routing and the validation baseline under the updated active set At∖{s1}\mathcal{A}_t \setminus \{s_1\} before auditing the next candidate (Algorithm 1). Consequently, when s2s_2 is subsequently audited, its leave-one-out MEC Δt(s2)\Delta_t(s_2) is evaluated relative to At∖{s1}\mathcal{A}_t \setminus \{s_1\} rather than reusing a pre-removal estimate. Because s2s_2 now provides non-redundant support on that routed task region, its newly measured Δt(s2)\Delta_t(s_2) increases; when the resulting smoothed EMA satisfies Δˉt(s2)≥τretire\bar{\Delta}_t(s_2) \ge \tau_{\mathrm{retire}}, Lemma A.5 protects s2s_2 from retirement and its low-contribution streak ℓt(s2)\ell_t(s_2) resets to 00 upon that audit.

Second, regarding failure cases recovered by expansion, our training logs contain two concrete compensatory expansion events in the main ALFWorld training run.

On Heat, SLIM retires heat_order at step 50, after which 7/8 validation tasks still succeed. Following the retirement of heat_state_check at step 60, success drops to 3/8 at step 70, and in all five failed episodes the agent places the object before heating it. SLIM then generates heat_before_delivery from these failure traces to verify heating completion before placement, and paired replay at step 70 shows that adding this skill improves success from 3/8 to 6/8 without losing any prior successes.

On Pick2, SLIM retires pair_counter at step 50, with 6/8 validation tasks remaining successful. Following the retirement of second_object_check at step 60, success declines to 2/8 at step 70, with five of the six failed episodes stopping after delivering only one object. SLIM subsequently generates require_two_distinct_items to verify that two distinct objects have been delivered before terminating, and paired replay improves success from 2/8 to 5/8 without introducing regressions.

Together, these two observed cases illustrate localized capability regressions following sequential retirements and show how failure-driven expansion can restore the missing external support. The paired replays confirm that the newly generated skills recover three failed tasks in each case without harming previously successful tasks.

Review (Q2): How sensitive are the learned endpoints and the final performance to the size and composition of the development split used for auditing, given that all lifecycle decisions are driven by routed statistics from that split?

Response: Utilizing the seen split of ALFWorld, we evaluate sensitivity to both the development set size ∣Ddev∣∈16,32,64|\mathcal{D}_{\mathrm{dev}}| \in \\{16, 32, 64\\} and the sampled task composition across 3 independent stratified draws of size 32 from the seen split.

Dev Split SettingTest Success Rate (%)Final Active Skills
16 tasks84.419
32 tasks, Draw 1 (Default)87.521
32 tasks, Draw 286.720
32 tasks, Draw 388.320
64 tasks89.123

As shown in the table, test success rates and final active skill counts remain highly consistent across the three independent 32-task draws, showing strong robustness to the sampled task composition of the development split. Across different development set sizes, the learned endpoint consistently converges to a compact skill set. Using 16 tasks slightly lowers accuracy because smaller batches accumulate exposure ut(s)≥30u_t(s) \ge 30 more slowly, though it still exceeds SkillRL$^\dagger$ at 75.0% and Skill0 at 74.2% by over 9 points. Doubling the development split to 64 tasks brings only a marginal gain at twice the validation cost, confirming that 32 tasks provide a reliable and efficient operating point.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their time and constructive feedback. To help you navigate the updated paper, we summarize the main revisions and additions below.

  1. Method and Implementation Clarifications (Section 5 and Appendix B.1). We have clarified the candidate selection rule across the top MM positive usage-frequency tiers including ties, the initial lifecycle audit at step 0 prior to GRPO updates, separate group auditing for general skills, the exact update rules for cumulative exposure ut(s)u_t(s) and low-contribution streak ℓt(s)\ell_t(s), and the EMA historical retention weight 1−α=0.91-\alpha=0.9 with current-round weight α=0.1\alpha=0.1.
  2. Unified ALFWorld Category Aggregation (Section 6.2). We have unified all ALFWorld success rates in Table 1 under a consistent batch-averaged aggregation protocol from the raw evaluation logs and corrected the transcription error, leaving all other overall Avg. scores and conclusions unchanged.
  3. Benchmark Split Documentation (Appendix C). We have explicitly documented the split origin, total episode counts, and non-overlapping partition of the 128-game unseen test split and the 32-game seen development split inherited from the released SkillRL benchmark.
  4. Multi-Seed Ablations and Audit Robustness Analyses (Section 6.3 and Appendix D.4). We have extended all ALFWorld ablation variants in Table 2 to 3 independent training seeds, added the worst-case 1- and 2-episode outcome perturbation analysis across all 126 individual skill audit decisions, and included sensitivity experiments over development split sizes ∣Ddev∣∈{16,32,64}|\mathcal{D}_{\mathrm{dev}}| \in \{16, 32, 64\} and independent development draws.
  5. Additional Controlled Comparisons and Audit Budget Ablation (Appendix D.6 and D.7). We have added the audit candidate budget ablation over M∈2,4,8M \in \\{2, 4, 8\\}, the matched SkillRL control with K=3K=3 and β=0\beta=0, and two lifecycle signal controls, namely Skill0-helpfulness + SLIM and Frequency-retirement + SLIM.
  6. Wall-Clock Auditing Overhead (Appendix D.8). We have added concrete wall-clock training and auditing runtime breakdowns across methods.
AuthorsOpenReview ↗

Paper Clarification (1/2)

We would like to thank all reviewers for their thorough reading of our paper and insightful comments. We at first address the most common and central questions raised across reviews to avoid any misunderstandings.

1. Effectiveness of Leave-One-Skill-Out MEC and Dynamic Lifecycle Management

To verify how much of SLIM's improvement comes from leave-one-skill-out marginal external contribution (MEC) and dynamic lifecycle management versus baseline hyperparameter differences, single-skill helpfulness filtering, or static policy-skill co-training, we conduct three sets of controlled experiments under the exact same backbone, initial skill bank, data splits, GRPO settings, and evaluation protocol.

First, to isolate lifecycle management from retrieval depth and KL regularization, we evaluate SkillRL$^\dagger$ (Matched) by setting SkillRL's retrieval cap to K=3K=3 and disabling policy-side KL regularization with β=0\beta=0 to match SLIM.

Second, to isolate leave-one-skill-out MEC from alternative retention and retirement signals under SLIM's exact schedule and expansion module, we evaluate Skill0-helpfulness + SLIM, which replaces leave-one-skill-out MEC with Skill0's single-skill versus zero-skill helpfulness score, and Frequency-retirement + SLIM, which replaces MEC-based retirement with smoothed routing frequency thresholding.

MethodRetrieval Cap KKPolicy-Side KL β\betaALFWorld (%)SearchQA (%)
GRPO$^\dagger$ (Static Co-Training)3068.8037.90
SkillRL$^\dagger$ (Original)60.01 / 0.00175.0038.10
SkillRL$^\dagger$ (Matched)3070.3136.83
Skill06 →\to 3 →\to 0 / 5 →\to 3 →\to 0074.2039.30
Frequency-retirement + SLIM3071.8838.12
Skill0-helpfulness + SLIM3082.0339.73
SLIM$^\dagger$ (Ours)3087.5041.00

Note: Entries separated by / denote ALFWorld / SearchQA configurations (Appendix B.2).

Third, we extend all ALFWorld lifecycle ablations in Table 2 to 3 independent training seeds, where full SLIM$^\dagger$ achieves 88.02±1.63%88.02 \pm 1.63\%, outperforming SLIM w/o Retirement at 73.70±1.19%73.70 \pm 1.19\%, SLIM w/o Expansion at 79.43±0.90%79.43 \pm 0.90\%, Random Audit at 67.19±1.56%67.19 \pm 1.56\%, and Fixed Active Set Size at 75.00±0.78%75.00 \pm 0.78\%. Together, these results confirm that static policy-skill co-training, fixed prompt budgets, routing frequency, and single-skill versus zero-skill filtering all fall short of SLIM$^\dagger$. Evaluating each skill's leave-one-out contribution relative to the remaining active set At∖{s}\mathcal{A}_t \setminus \{s\} and coupling it with retain, retire, and expand updates is essential for learning an effective external capability boundary.

2. ALFWorld Evaluation Split and Table 1 Category Aggregation

All methods in Table 1 are evaluated on the exact same 128 unseen test games directly inherited from the released SkillRL [1] benchmark, with per-category counts of Pick at 35, Look at 5, Clean at 32, Heat at 13, Cool at 23, and Pick2 at 20, alongside 32 development games sampled from the disjoint official seen split.

In our single-run evaluation partitioned into 8 equal batches of 16 episodes, the standalone evaluation script used for prompt baselines and SLIM without skills computes pooled episode-level category success rates SRcpool=∑bsbc∑bnbc\mathrm{SR}_c^{\mathrm{pool}} = \frac{\sum_b s_{bc}}{\sum_b n_{bc}}, whereas the batched online-RL evaluator used for SLIM$^\dagger$, GRPO, GRPO$^\dagger$, EvolveR, SkillRL$^\dagger$, and Skill0 averages the within-batch category success rates across batches containing category cc, namely SRcbatch=1∣Bc∣∑b∈Bcsbcnbc\mathrm{SR}_c^{\mathrm{batch}} = \frac{1}{|\mathcal{B}_c|}\sum_{b\in\mathcal{B}_c}\frac{s_{bc}}{n_{bc}}. Because all 8 batches have an identical size of 16, the overall Avg. score is mathematically identical under both formulas and always equals total successes divided by 128. In the revised paper, we have unified all rows in Table 1 under a single aggregation rule, leaving all other overall Avg. scores and conclusions completely unchanged.

AuthorsOpenReview ↗

Paper Clarification (2/2)

3. Audit Selection Protocol, Routed Sample Size, and Decision Stability

In our implementation, the candidate budget M=4M=4 selects task-specific skills in the top 4 positive usage-frequency tiers including ties within those tiers, while general skills are audited separately (Appendix B.1) and skill expansion is triggered from failure buckets after the audit loop without consuming MEC audit slots (Algorithm 1). With an initial audit at step 0 followed by audits every 10 steps through step 120, SLIM executes 13 audit cycles and performs 126 individual skill audits in the original-bank run and 143 in the noisy-bank run, excluding the separately conducted general-group audits. Table A3 counts cumulative retired skills across the full run, including expanded skills that are subsequently retired (Appendix D.3).

Across the 126 individual MEC audits in the main ALFWorld run, each audited skill is associated with an average of 19.7 routed evaluation records, with a median of 21.5 and a range of 7 to 31, including repeated evaluations on the same underlying development tasks. Retrospective perturbation analysis across all 126 audit decisions shows that 97.6% and 96.0% of lifecycle decisions remain unchanged under worst-case 1- and 2-episode outcome flips, and 100% of observed retirements are invariant to single-episode flips due to EMA smoothing with α=0.1\alpha=0.1, minimum exposure nmin⁡=30n_{\min}=30, and patience p=3p=3. Furthermore, varying the development split size and resampling 3 independent 32-task development draws confirm that both the learned endpoint and test accuracy remain stable.

Dev Split SettingTest Success Rate (%)Final Active Skills
16 tasks84.419
32 tasks, Draw 1 (Default)87.521
32 tasks, Draw 286.720
32 tasks, Draw 388.320
64 tasks89.123

4. Computational Overhead and Audit Budget Scaling

Under our shared training stack on ALFWorld, total wall-clock time is dominated by online GRPO rollouts and policy updates, taking 17.5 hours for Skill0, 19.2 hours for SkillRL$^\dagger$, and 20.1 hours for SLIM$^\dagger$, where leave-one-skill-out auditing and baseline refreshes account for only 2.4 hours, or 12% of total runtime. Ablating the audit budget over M∈2,4,8M \in \\{2, 4, 8\\} on ALFWorld gives test success rates of 74.21%, 87.50%, and 89.10%. Training logs show that skill routing frequencies follow a concentrated power-law distribution where the top M=4M=4 frequency tiers cover about 80% of routed skill usages, whereas M=8M=8 more than doubles the audit runtime cost for a modest gain. Because dense retrieval filters candidates in milliseconds and auditing only evaluates top routed candidates within budget, the per-cycle audit cost is independent of the global skill bank size ∣S∣|\mathcal{S}| (Appendix D.8, Table A14).


References

[1] Peng Xia, et al. SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. arXiv 2602.08234.

AuthorsOpenReview ↗

Thank You to All Reviewers

We would like to express our sincere thanks to all reviewers for your thorough and constructive comments on our paper, and we are deeply encouraged by your engagement and support throughout the discussion phase. We are pleased that reviewers recognize the value of reframing monotonic skill accumulation and forced elimination as a unified dynamic boundary management problem, the principled contribution-aware lifecycle controller, and the rigor of our empirical evaluation and confound controls. The thoughtful questions and suggestions raised during both the initial review and discussion phases have significantly strengthened our work. We will carefully integrate all of your feedback into the final version of the paper. Thank you once again for your time and support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.