acceptodds
Under review as a conference paper at ICLR 2027

Trackable Skill Library Lifecycle Management for Agentic Reinforcement Learning

Abstract

Skill-augmented agentic reinforcement learning (RL) has emerged as an effective approach to improving LLM agents on long-horizon tasks. Despite this progress, individual skills may follow distinct trajectories as the policy evolves, with some becoming redundant after the model internalizes their capabilities, others remaining valuable as external scaffolding, and still others turning harmful. However, existing skill library systems fail to identify each skill’s true contribution due to free-riding among co-retrieved skills, difficulty confounding across tasks, and the absence of skill-specific counterfactual evidence, which prevents reliable skill lifecycle management. We propose TrackRL, an agentic RL framework in which skill tracking drives full lifecycle management of the skill library. Specifically, within each GRPO group, we turn grouped sampling into an online randomized controlled experiment. Independent random injection and within-group contrasts on the same task isolate each skill’s true contribution from co-retrieved skills and task difficulty, and reliability-weighted aggregation accumulates this evidence across groups. As evidence accumulates, we continuously track skill utility, which measures the skill’s causal contribution as external scaffolding, and model mastery, which reflects how far the corresponding capability has been internalized. Along this line, we formulate skill lifecycle management as an evidence-gated state-transition process that governs each skill from generation and admission through protected incubation to retention, retirement, or removal, while a soft capacity bound controls library growth. Extensive experiments show that TrackRL outperforms recent skill-augmented RL baselines in both library-assisted and skill-free evaluation, with further analyses revealing how skill value shifts during policy learning and how evidence-driven state transitions shape skill library evolution.

Primary area:
interpretability and explainable AI
Keywords:
agentic reinforcement learning, agent skills, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vRLHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TrackRL for managing an evolving skill library during agentic reinforcement learning. It randomizes the injection of individual skills within GRPO groups and uses outcome comparisons to track skill utility, model mastery, and accumulated evidence. These statistics guide skill admission, incubation, retention, retirement, removal, and capacity eviction. Experiments on ALFWorld and AppWorld evaluate both skill-assisted and skill-free performance.

Critical strengths
  1. A practical problem and a coherent framework. The paper recognizes that skills can remain useful, become redundant, or turn harmful as the policy evolves. Tracking these changes and using evidence to guide lifecycle decisions is a meaningful extension beyond accumulating or periodically pruning a library.

  2. Evaluation distinguishes external assistance from learned capability. The experiments include skill-assisted and skill-free evaluation, as well as base-model and SFT initialization. TrackRL improves performance even without the library: under cold start, TrackRL (Luna) achieves 72.7% skill-free success on seen ALFWorld tasks, compared with 58.6% for GRPO.

Critical weaknesses
  1. The causal interpretation needs clarification. Reserved episodes omit all skills of a given granularity and enter individual skills’ control arms. This may create differences in co-injected skills between treatment and control, complicating attribution to a single skill. Control-arm success may also reflect assistance from other skills, so it does not directly establish capability internalization.

  2. The correspondence between the paper and implementation is unclear. The paper defines when_to_apply as the retrieval key, whereas the code prioritizes retrieval_obs, derived from the source task and observation. The implementation also supports validation-driven library updates and optional same-task paired evaluation. Without complete run configurations, adherence to the stated evaluation protocol cannot be confirmed.

Questions for authors
  1. Do the reserved skill-free episodes contribute to individual skill utility estimates? If so, how is comparability in co-injected skills maintained between treatment and control? Could the authors report estimates excluding these reserved episodes?

  2. Could the authors provide the actual run configurations, clarify the retrieval key used, and confirm that library updates were disabled during evaluation and both modes used the same tasks?

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We appreciate your insightful feedback and constructive comments. We address each of your comments below.

W1 &Q1: Reserved episodes complicate attribution to a single skill, and control-arm success does not directly establish capability internalization

Thank you for this comment and the related question. We confirm that the reserved skill-free episodes are counted in the control arm of every skill selected in them, as stated in Appendix D.1. Counting these episodes leaves most lifecycle decisions unchanged and yields comparable final performance, and the retired skills are internalized by the policy rather than supported by other skills. We explain these two points in turn.

1. Effect of counting the reserved episodes in the control arm

Each group reserves one task-level and one step-level skill-free episode, in which the selected skills of that granularity are not injected. All other episodes are randomized independently for each skill, so among them the co-injected skills are comparable between the treatment and control arms of a skill. The reserved episode adds to the control arm one episode in which the other skills are withheld as well.

Motivation for reserved skill-free episodes. A training group contributes to a skill’s utility estimate only if both arms contain an episode. With few episodes per group, randomization can leave the control arm empty. Counting the reserved skill-free episode ensures a nonempty control arm without extra rollouts, allowing evidence to accumulate faster for newly admitted skills.

The positive effect of the upward bias on removal. This bias provides a desirable safety property for skill removal. As your comment shows, when co-retrieved skills are helpful, mixing in the skill-free episode introduces a small upward bias to the target skill's utility usu_s. Meanwhile, in our lifecycle management (Eq. 11), permanently removing a harmful skill (Removal) requires its upper confidence bound to fall below zero: us+λ SE(us)<0u_s + \lambda\,\mathrm{SE}(u_s) < 0. This upward bias therefore makes removal slightly more lenient for every skill. A skill whose upper confidence bound still falls below zero despite this bias is clearly harmful. In other words, this bias lowers the false-positive removal rate, i.e., the chance of removing a helpful skill by mistake.

Comparison with and without the reserved episodes. Following your suggestion, we compare the original version with a version that computes the per-skill statistics only from randomized episodes. In this version, the reserved skill-free episodes are used only for the family baseline. We conduct two experiments.

  • Experiment 1: Recomputation from existing training records. Using the training records from the cold-start ALFWorld run, we recompute utility for every skill that passed incubation. The original estimate includes the reserved skill-free episodes in the control arm. The alternative estimate uses only randomized episodes. The estimates differ by 0.014 on average. The utility sign agrees for 96.4% of the skills, and 95.8% of removal and retirement decisions agree. The upward bias depends on how much less often the reserved episodes succeed compared with the randomized control episodes, where other skills can still be injected. In our training records, the reserved episodes succeed only 5.6% less often than the randomized control episodes. This small gap is consistent with success rarely depending entirely on another skill. Since the reserved episode accounts for about one quarter of the control arm, the bias is 1/4×0.056≈0.0141/4 \times 0.056 \approx 0.014, which matches the measured difference.

  • Experiment 2: End-to-end retraining. To further verify this, we retrain TrackRL end to end on ALFWorld with the version restricted to randomized episodes. All settings are kept the same as in the original experiment. The only change is that we compute each skill’s utility and mastery using only randomized episodes. We evaluate the resulting checkpoint in skill-assisted and skill-free modes on the seen and unseen splits and report the success rate (SR, %).

VersionSeen skill-assistedSeen skill-freeUnseen skill-assistedUnseen skill-free
Original78.172.771.964.1
Randomized episodes only78.973.472.764.8

As shown in the table, the two versions reach comparable SR in all four settings, so the final performance of TrackRL is robust to this choice.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

W1 &Q1 (continued): Reserved episodes complicate attribution to a single skill, and control-arm success does not directly establish capability internalization

2. Retired skills are internalized rather than assisted by other skills

Although a control-arm episode for a skill may still contain other skills, compensation by these skills does not explain the high mastery of the retired skills. We explain this from two aspects: the mechanism design of TrackRL limits compensation, and the empirical evidence shows that the retired skills are internalized.

TrackRL's mechanism design limits compensation.

  • Admission filters out near-duplicate skills. TrackRL admits a new skill only if it is dissimilar to all active skills, keeping directly substitutable near-duplicates out of the active library.
  • The control arm includes episodes that also withhold other skills. TrackRL independently randomizes whether each selected skill is injected. Therefore, other task-level skills can also be withheld in a given skill’s control episodes. Each group also reserves one episode in which all task-level skills are withheld. If the policy relies on other skills, it is more likely to fail when those skills are absent. These failures lower the skill’s estimated mastery, making it less likely to be retired.
  • TrackRL can reverse retirement. TrackRL can regenerate and readmit a retired skill if later failures show that the policy still needs it. Skills removed as harmful are ineligible for readmission. If temporary compensation often led to premature retirement, we would expect the policy to fail again on the associated tasks and many retired skills to be regenerated.

Empirical evidence for internalization. We examine the internalization of retired skills directly from two aspects: (1) the skill-free success of the retired skills in the existing training records, and (2) a controlled evaluation of the trained policy with and without other skills.

Skill-free success from the training records. We first analyze the training records from the cold-start ALFWorld run. Each training group reserves one episode with no task-level skills injected. These episodes show how the policy performs when other task-level skills are unavailable. For each of the 14 retired skills, we identify the groups that selected it. We measure SR in the reserved episodes from the first 10 such groups after admission and the last 10 before retirement. Averaged across retired skills, SR rises from 16.4% at admission to 85.7% at retirement, close to the estimated mastery (87.2%). For skills evicted because of non-positive utility, the corresponding SR before eviction is 31.2%. By retirement, the policy can therefore solve most of the associated tasks without any task-level skills. This confirms that the policy has internalized these skills and can solve their tasks without compensation from other skills.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

W1 &Q1 (continued): Reserved episodes complicate attribution to a single skill, and control-arm success does not directly establish capability internalization

Controlled evaluation with and without other skills. Inspired by your comment, we further designed a controlled evaluation to test whether the high mastery comes from internalization or from compensation. Let πinit\pi_{\mathrm{init}} denote the initial policy before RL and πθ\pi_\theta the final trained policy. For each retired skill ss, we evaluate both policies on the ALFWorld evaluation tasks for which ss is retrieved. We use the same tasks across conditions and report SR averaged across retired skills. We consider three skill settings: (1) "None" injects no skill, (2) "Retired skill" injects only ss, and (3) "Other skills" injects the skills retrieved from the active library, which no longer contains ss. If the final policy depends on other skills to achieve high success, its SR should drop substantially from "Other skills" to "None."

PolicySkillsSR (%)
πinit\pi_{\mathrm{init}}None21.4
πinit\pi_{\mathrm{init}}Retired skill42.9
πθ\pi_\thetaNone86.2
πθ\pi_\thetaOther skills86.9
πθ\pi_\thetaRetired skill87.1

The table leads to two findings: (1) Under πinit\pi_{\mathrm{init}}, the retired skill raises SR from 21.4% to 42.9%, so the retired skills carry useful guidance for a policy that has not yet learned them. (2) Under πθ\pi_\theta, removing all other skills changes SR by only 0.7% (86.9% to 86.2%), and injecting the retired skill again brings no further gain (87.1%). The trained policy thus solves these tasks on its own, which supports internalization. Overall, the policy’s ability to solve these tasks on its own explains the high mastery of the retired skills. (1) By design, TrackRL withholds other skills in the control arm for a given skill and can regenerate a retired skill if the policy still needs it. (2) Empirically, the policy already succeeds on the associated tasks without any task-level skill at the time of retirement, and removing all other skills from the trained policy changes SR by only 0.7%. Since this evidence is collected for each retired skill on its own tasks, it shows that the retirement decisions single out skills that the policy has internalized.

In summary, (1) excluding the reserved skill-free episodes makes co-injected skills comparable across treatment and control arms. Utility estimates, lifecycle decisions, and final performance remain similar, supporting attribution to the target skill itself. (2) The trained policy solves tasks associated with retired skills even when no skills are injected, showing that it can succeed without compensation from other skills.

We have described both versions and added these analyses to the revised paper (Section 3 and Appendices F.4 and F.5).

Reviewer_RtnQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TrackRL, a skill-augmented reinforcement learning framework that tracks individual skill contributions as the policy evolves. It uses randomized skill injection within GRPO groups to estimate skill utility and policy mastery, which guide evidence-based skill lifecycle management, including retention, retirement, and removal. Experiments on ALFWorld and AppWorld demonstrate performance improvements over existing skill-augmented RL baselines in both skill-assisted and skill-free settings.

Critical strengths

The proposed within-group randomized skill injection provides an efficient approach to estimating individual skill contributions while mitigating free-riding and task difficulty confounding without additional rollouts. The framework incorporates uncertainty-aware skill tracking and evidence-gated lifecycle decisions, offering a systematic approach to dynamic skill library management. The experiments include multiple benchmarks, skill-assisted and skill-free evaluations, and ablation studies, providing useful insights into the effectiveness of the proposed framework.

Critical weaknesses

The paper emphasizes that a skill's utility changes as the policy evolves, but it remains insufficiently justified why this necessitates retiring skills from the library rather than simply adapting their retrieval or usage. A skill is essentially reusable knowledge applicable in specific contexts, whereas the policy is the decision-making component that changes through learning. Therefore, a skill's low marginal utility under the current policy does not necessarily imply that the skill has lost its long-term value. Although the paper introduces a soft capacity bound and mentions retrieval and storage costs, it remains unclear why policy-dependent retrieval or skill preservation would be insufficient compared to retirement. The authors should more clearly distinguish a skill's current utility from its value as reusable knowledge and provide a stronger justification for why the skill library itself must evolve with the policy.

The justification for skill retirement based on internalization remains unclear. Although GRPO continuously updates the policy, the framework does not explicitly encourage or optimize skill internalization. Yet, high success rates when a skill is withheld are interpreted as evidence of internalization, even though such success may reflect general policy improvement or compensation by other skills still injected in the control arm. The additional utility and uncertainty conditions for retirement do not resolve this ambiguity. Consequently, the mastery metric may conflate internalization with skill redundancy, making retirement difficult to distinguish from utility-based pruning.

The baseline comparisons raise concerns regarding both fairness and completeness. The authors remove the hindsight intrinsic reward originally used by D2Skill, potentially altering its intended training objective. Moreover, the paper states that SkillRL also employs an intrinsic reward, whereas the original SkillRL formulation appears to use only task-outcome rewards. This discrepancy raises questions about the fidelity of the baseline implementations and should be clarified. More importantly, SLIM, which shares highly similar problem formulations, motivations, and overall frameworks for policy–skill library co-evolution and dynamic skill lifecycle management, is cited in the related work but omitted from the experiments. While TrackRL uses a different approach to estimating skill contributions, its meaningful advantages over SLIM in performance, attribution reliability, or computational efficiency remain unclear.

The experimental results are reported without standard deviations, confidence intervals for overall performance, or clear information about repeated runs with different random seeds. Given the stochastic nature of reinforcement learning, skill generation, and randomized skill injection, the reported performance improvements may be sensitive to random initialization and sampling variability. Without repeated experiments, it is difficult to determine whether the observed improvements are statistically reliable or attributable to random fluctuations, particularly when the performance differences between methods are relatively small.

Questions for authors

Please see the weaknesses.

11 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/6)

We appreciate your insightful feedback and constructive comments! We address each of your comments below.

W1: Why retire a skill rather than retrieve it less often or keep it as reusable knowledge?

Thank you for this thoughtful comment. We understand this view, but an unmaintained skill library accumulates skills that are no longer needed, and even harmful ones. This is a widely recognized problem that recent work addresses (SkillOps [1], SLIM [2], MUSE-Autoskill [3]). Retirement in TrackRL does not deny the value of a skill as reusable knowledge. It only keeps the active library aligned with what the current policy still needs. We explain this in three parts: why adapting retrieval alone is not enough, what happens in practice when retirement is disabled, and how a retired skill keeps its value as reusable knowledge.

1. Why adaptive retrieval alone leaves two problems

TrackRL already adapts retrieval to the policy, since reranking combines semantic relevance with the tracked utility. However, leaving internalized skills in the active library causes two problems that reranking cannot fix.

  • Internalized skills still occupy the few injected positions. TrackRL first retrieves candidates mostly by semantic similarity. An internalized skill can remain highly relevant to the task, and its utility is typically close to zero. Repeatedly selecting these skills leaves fewer slots for newer skills that address gaps in the policy’s capabilities.
  • Internalized skills block more specific new skills. TrackRL rejects a new candidate if it is too similar to any active skill. If a basic skill remains active after the policy has mastered the routine, deduplication can reject more specific skills that address the remaining hard cases in the same task family. Retiring the basic skill allows the library to admit skills for those cases.

Retrieval decides which skills are used for a given query, whereas retirement decides which skills take part in retrieval and admission at all. Retirement therefore plays a role that retrieval cannot replace.

2. Ablation of retirement

Setup. To test this directly, we compare TrackRL with two variants on ALFWorld under the same cold-start setting. Both variants keep skill tracking and utility-based reranking, so retrieval still adapts to the policy, and they differ from TrackRL only in which skill exits are disabled. (1) w/o management disables all exits (retirement, harmful removal, and capacity eviction), so every admitted skill stays in the library, which corresponds to preserving all skills. (2) w/o retirement disables only retirement, while harmful removal and capacity eviction work as in TrackRL. Internalized skills therefore stay in the active library, whereas harmful skills are still removed and the library size remains controlled. Comparing these variants isolates the effect of enabling retirement.

Besides SR on the seen split, we report two statistics from the last 20 training steps. "Non-beneficial" is the share of injected skills whose utility is non-positive. "Rejected" is the share of newly generated skills that are rejected by deduplication.

VariantSkill-assisted SR (%)Skill-free SR (%)Non-beneficial (%)Rejected (%)
w/o management71.953.146.238.5
w/o retirement74.264.834.831.2
TrackRL78.172.78.414.6

Results. (1) Without retirement, 34.8% of the injected skills are non-beneficial and 31.2% of the newly generated skills are rejected, compared with only 8.4% and 14.6% for TrackRL. These results are consistent with internalized skills continuing to occupy injection slots and block new skills, even with adaptive retrieval and harmful removal. (2) Without retirement, the skill-assisted and skill-free SR are 3.9% and 7.9% lower than those of TrackRL, and preserving all skills (w/o management) lowers them further. Internalized skills left in the library thus also degrade the policy, both with and without skills. Together, these findings show that neither policy-dependent retrieval nor skill preservation can replace retirement, which resolves your question.

AuthorsOpenReview ↗

Rebuttal by Authors (2/6)

W1 (continued): Why retire a skill rather than retrieve it less often or keep it as reusable knowledge?

3. Retirement preserves the value of a skill as reusable knowledge

Retired skills and harmful skills both leave the library with non-positive utility, but for different reasons. A retired skill is no longer needed because the policy has internalized its capability, whereas a harmful skill misleads the policy.

Setup. To see whether these skills still have value for a policy that has yet to learn those capabilities, we test them on the initial policy before RL. For each skill, we evaluate the initial policy on the ALFWorld evaluation tasks for which that skill is retrieved, once with the skill injected and once with it withheld. We report SR averaged across the skills in each group.

SkillsSR without the skill (%)SR with the skill (%)
Retired21.442.9
Removed as harmful21.410.2

Results. Although both groups have non-positive utility when they leave the library, they affect the initial policy in opposite ways. Retired skills raise its SR from 21.4% to 42.9%, whereas harmful skills lower it from 21.4% to 10.2%. A retired skill therefore remains valid knowledge for a policy that has not learned it, and its low utility only reflects that the current policy no longer needs it. This is also why TrackRL keeps retirement separate from harmful removal. A harmful skill is permanently excluded from the library, whereas a retired skill can be regenerated and readmitted if the policy later loses the capability.

In conclusion, we address your concerns with both analysis and experiments. Adapting retrieval cannot prevent internalized skills from occupying injected positions and blocking new skills, and the ablation shows that this lowers performance. At the same time, retirement does not discard knowledge, since retired skills still help a policy that has not learned them and can be readmitted when needed. Therefore, the skill library needs to evolve with the policy, while the value of each skill as reusable knowledge is preserved.

We have clarified the distinction between the current utility of a skill and its value as reusable knowledge in the revised paper, and added both experiments (Section 3 and Appendix F.6).

[1] Song et al. SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems.

[2] Shen et al. Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning.

[3] Lin et al. MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation.

AuthorsOpenReview ↗

Rebuttal by Authors (3/6)

W2. What drives and verifies internalization, and how does retirement differ from utility-based pruning?

Thank you for this thoughtful comment. In TrackRL, retirement rests on internalization that the framework itself drives and that can be verified for the retired skills. We address your concern from three aspects: (1) how the policy-library co-evolution drives internalization without changing the GRPO objective, (2) two experiments showing that the retired skills are internalized rather than compensated by other skills or redundant, and (3) how retirement differs from utility-based pruning.

1. Policy–library co-evolution drives internalization

TrackRL does not need an additional objective for internalization, because the co-evolution of the policy and the skill library already drives it. Skills are distilled from the failed trajectories of the policy itself, so each skill describes what the policy currently lacks. When a skill is injected, the policy succeeds more often on the corresponding tasks, and GRPO reinforces these successful trajectories. The reward depends only on task completion, with no extra reward for using the skill. This encourages the policy to learn how to complete the task itself. Since injection is randomized, each group also contains episodes of the same task in which the skill is withheld. The policy is therefore continually trained and tested on the task without the skill, and gradually reproduces the behavior that the skill taught. As the policy learns to solve the task on its own, the skill’s utility falls and its mastery rises. TrackRL then retires the skill, and reflection on the remaining failures produces new skills for capabilities the policy still lacks.

Internalization is thus the outcome of this loop: the policy learns from its own experience through the library under the standard GRPO objective. This is reflected in Table 1, where TrackRL reaches 72.7% skill-free SR on the seen split of ALFWorld, compared with 58.6% for GRPO. This addresses your concern that the framework does not explicitly encourage internalization.

2. Retired skills are internalized, not compensated or redundant

We verify this with two experiments on the retired skills, using the training records of the cold-start ALFWorld run.

Skill-free success of the retired skills. Each training group reserves one episode with no task-level skills injected. Success in these episodes is therefore independent of compensation from other task-level skills. For each retired skill, we identify the training groups in which the skill was selected. We then measure the SR of these reserved episodes in the first 10 such groups after admission and the last 10 before the skill left the library. We apply the same procedure to skills evicted with non-positive utility.

SkillsSR at admission (%)SR at exit (%)
Retired16.485.7
Evicted14.831.2

The results are as follows. For retired skills, skill-free SR rises from 16.4% at admission to 85.7% at retirement, close to their estimated mastery of 87.2%. For evicted skills, skill-free SR remains low at 31.2% before eviction. The policy can therefore solve the tasks of the retired skills without any task-level skills, so these skills are internalized rather than compensated by other skills.

Utility before retirement. If a retired skill were simply redundant, its utility would stay around zero from the start. We therefore trace the utility of each skill from the end of incubation to its exit, and report the highest utility it reached and its mastery at exit. We compare the retired skills with the skills that were removed as harmful or evicted.

SkillsHighest utilityMastery at exit
Retired+0.180.87
Removed or evicted−0.030.34

The table leads to two findings. (1) The retired skills reached an average highest utility of +0.18, whereas the removed or evicted skills reached only −0.03. The retired skills therefore did help the policy before they were retired, so they are not redundant. (2) At exit, the mastery of the retired skills is 0.87, compared with 0.34 for the other skills. Accordingly, the utility of the retired skills declined because the policy learned to succeed without them, whereas the removed and evicted skills left the library because they did not help.

Taken together, the two experiments show that a skill is retired only after it has helped the policy and the policy has then learned to solve its tasks without any task-level skills. Since this SR is measured without any task-level skills, and the evicted skills went through the same training but only reach 31.2%, this rise comes neither from compensation by other skills nor from a general improvement of the policy alone. The high mastery of a retired skill therefore reflects internalization, which justifies basing retirement on it.

Reviewer_EGAPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TrackRL to study how the value of external skills changes as a policy learns and how a skill library should be maintained in response. It maintains task and step level skill pools, retrieves and reranks relevant skills, and randomizes their injection within GRPO groups for the same task. TrackRL estimates each skill’s utility from the difference in success rates between episodes with and without that skill, and estimates mastery for task-level skills from success when the selected skill is withheld. Utility and mastery are updated using recency-weighted statistics, while an undiscounted evidence count tracks the accumulated comparisons. These quantities guide skill incubation, retention, retirement, removal, and capacity eviction.

The paper reports skill-assisted and skill-free performance on ALFWorld and AppWorld, under both the base model and an SFT ckpt. The ALFWorld ablations include w/o tracking, which replaces independent skill injection with joint injection or withholding; w/o management, which retains skill generation and tracking but disables retirement, removal, and eviction; and w/o both, which reduces to skill-free GRPO. Additional analyses examine library growth, utility and mastery distributions, lifecycle events, and individual skill trajectories.

Critical strengths
  1. TrackRL treats a single skill as the unit of randomized intervention, changing the injection status of each skill across multiple rollouts of the same task under the same policy. Compared with assigning a shared contribution to co-retrieved skills, this design produces the differing skill combinations needed to investigate individual contributions and uses GRPO’s repeated-sampling structure for that evaluation. This is a valuable methodological choice because it directly targets the attribution problem arising from skill co-occurrence and evaluates skills within the training process of the current policy.

  2. The paper does not decide whether a skill stays or goes on the basis of utility magnitude alone; it separately records the incremental gain a skill provides and the task success achieved when that skill is not provided. Recording both quantities is useful because marginal benefit and performance without the target skill describe different aspects of the policy’s dependence on external guidance. Building on this, TrackRL sets distinct conditions for retirement, removal, and capacity eviction, and uses evidence requirements to protect new skills that have not yet been adequately evaluated. Lifecycle management thereby rests on explicit, testable decision criteria rather than on sorting entries by a single score and deleting them.

Critical weaknesses
  • Weakness 1: Mastery does not distinguish internalization from compensation by other skills

Equations (8)–(9) estimate mastery from success on episodes where the target skill is withheld, but the remaining skills can still help the policy finish the task. A skill that shows zero utility together with high mastery is therefore ambiguous: either it has been internalized, or other external guidance is compensating for it. The family baseline in Eq. (12) does not exclude the second reading. Figure 3 shows that skill-free performance improves, which indicates a gain in the policy's overall capability, but it does not show that individual retirement decisions single out internalized skills. The ambiguity bears on one of the paper's main claims: that utility and mastery together separate ineffective skills from internalized ones and account for their different lifecycle outcomes.

  • Weakness 2: The per-skill utility estimator mixes the target skill’s effect with the effects of jointly withholding other skills.

Appendix D.1 states that the reserved task-level and step-level skill-free episodes enter the withheld-skill statistics of every selected skill at the corresponding granularity. Such an episode withholds every skill at that granularity, whereas the randomized episodes assign each skill independently. The contrast in Eq. (5) can therefore charge the target skill for failures that other missing skills caused. Consider a case in which A has no effect and success depends only on B: one reserved task-level skill-free episode plus seven episodes with independent Bernoulli(1/2) assignments give A an expected within-group contrast of 1/8, conditional on both comparison groups being nonempty. The weighting in Eq. (5) does not remove this bias.

Questions for authors
  • Question 1: Is there any existing evidence directly validates the per-skill utility estimates expect for downstream success rates? For example, at a fixed policy ckpt, do skills estimated to be helpful improve performance, and does withholding skills estimated to be harmful? Such evidence would help determine whether the gains in Figure 3 reflect more accurate attribution rather than changes in the skill combinations used during training.

  • Question 2: For skills retired as internalized, do existing skill-free evaluations on the associated tasks show that the policy can succeed without assistance from other skills? Could the authors connect these outcomes to the existing lifecycle records and explain what additional information mastery and the distinctions between exit types provide.

  • Question 3: How does the implementation handle the reserved skill-free episodes? Please confirm whether they are included in the per-skill statistics used for utility and mastery in Eqs. (5) and (8)–(9). If they are directly included, how do utility signs and removal, retirement, and eviction decisions change when the estimates are restricted to randomized episodes?

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/5)

We appreciate your insightful feedback and constructive comments! We address each of your comments point by point below.

W1: Mastery does not distinguish internalization from compensation by other skills

Thank you for this insightful comment. We explain this from two aspects: the mechanism design of TrackRL limits compensation, and the empirical evidence shows that the retired skills are internalized.

1. TrackRL's mechanism design limits compensation

  • Admission filters out near-duplicate skills. TrackRL admits a new skill only if it is dissimilar to all active skills, keeping directly substitutable near-duplicates out of the active library.
  • The control arm also includes episodes that also withhold other skills. TrackRL independently randomizes whether each selected skill is injected. Therefore, other task-level skills can also be withheld in a given skill’s control episodes. Each group also reserves one episode in which all task-level skills are withheld. If the policy relies on other skills, it is more likely to fail when those skills are absent. These failures lower the skill’s estimated mastery, making it less likely to be retired.
  • TrackRL can reverse retirement. TrackRL can regenerate and readmit a retired skill if later failures show that the policy still needs it. If temporary compensation often led to premature retirement, we would expect the policy to fail again on the associated tasks. These failures would trigger the regeneration of many retired skills.

2. Empirical evidence for internalization

Following your suggestion, we examine the internalization of retired skills directly from two aspects: (1) the skill-free success of the retired skills in the existing training records, and (2) a controlled evaluation of the trained policy with and without other skills.

Skill-free success from the training records. We first analyze the training records from the cold-start ALFWorld run. Each training group reserves one episode with no task-level skills injected. These episodes show how the policy performs when other task-level skills are unavailable. For each of the 14 retired skills in the cold-start ALFWorld run, we identify the training groups where that skill was selected. We then measure success rate (SR) in the reserved episodes from the first 10 such groups after admission and the last 10 before retirement. Averaged across retired skills, it rises from 16.4% at admission to 85.7% at retirement, close to the estimated mastery (87.2%). For skills evicted because of non-positive utility, the corresponding SR before eviction is 31.2%. The policy can therefore already solve most of the associated tasks without any task-level skills by the time of retirement. In other words, these skills have indeed been internalized by the policy, rather than compensated by other skills.

Controlled evaluation with and without other skills. Inspired by your comment, we further designed a controlled evaluation to test whether the high mastery comes from internalization or from compensation by other skills. Let πinit\pi_{\mathrm{init}} denote the initial policy before RL and πθ\pi_\theta the final trained policy. For each retired skill ss, we evaluate both policies on the ALFWorld evaluation tasks for which ss is retrieved. We use the same tasks across conditions and report SR averaged across retired skills. We consider three skill settings: (1) "None" injects no skill, (2) "Retired skill" injects only ss, and (3) "Other skills" injects the skills retrieved from the active library, which no longer contains ss. If the final policy depends on other skills to achieve high success, its SR should drop substantially from "Other skills" to "None."

PolicySkillsSR (%)
πinit\pi_{\mathrm{init}}None21.4
πinit\pi_{\mathrm{init}}Retired skill42.9
πθ\pi_\thetaNone86.2
πθ\pi_\thetaOther skills86.9
πθ\pi_\thetaRetired skill87.1

The table show two findings: (1) Under πinit\pi_{\mathrm{init}}, the retired skill raises SR from 21.4% to 42.9%, so the retired skills carry useful guidance for a policy that has not yet learned them. (2) Under πθ\pi_\theta, removing all other skills changes SR by only 0.7% (86.9% to 86.2%), and injecting the retired skill again brings no further gain (87.1%). The trained policy thus solves these tasks on its own, which supports internalization.

In summary, the policy’s ability to solve these tasks on its own explains the high mastery of the retired skills. (1) By design, TrackRL withholds other skills in control episodes and can regenerate retired skills if needed. (2) Empirically, the policy already succeeds on the associated tasks without any task-level skill by retirement, and removing all other skills from the trained policy changes SR by only 0.7%, showing that TrackRL retires skills the policy has internalized.

We have added both analyses to the revised paper (Section 3 and Appendix F.5).

AuthorsOpenReview ↗

Rebuttal by Authors (2/5)

W2: The per-skill utility estimator mixes the target skill's effect with the effects of jointly withholding other skills

Thank you for the careful analysis and the concrete example. Counting the reserved skill-free episodes in the control arm leaves most lifecycle decisions unchanged and yields comparable final performance. We explain why we count these episodes, why the resulting bias could be positive, and what happens when we exclude them from the statistics.

1. Motivation for reserved skill-free episodes

A training group can contribute to a skill’s utility estimate only if both arms contain at least one episode. With only a few episodes per group, randomization can leave a skill’s control arm empty. Counting the reserved skill-free episode gives the control arm at least one episode without requiring additional rollouts. This allows evidence to accumulate faster for newly admitted skills.

2. The positive effect of the upward bias on removal

This bias provides a desirable safety property for skill removal. As your comment shows, when co-retrieved skills are helpful, mixing in the skill-free episode introduces a small upward bias to the target skill's utility usu_s. Meanwhile, in our lifecycle management (Eq. 11), permanently removing a harmful skill (Removal) requires its upper confidence bound to fall below zero: us+λ SE(us)<0u_s + \lambda\,\mathrm{SE}(u_s) < 0. This upward bias therefore makes removal slightly more lenient for every skill. A skill whose upper confidence bound still falls below zero despite this bias is clearly harmful. In other words, this bias lowers the false-positive removal (FPR) rate, i.e., the chance of removing a helpful skill by mistake.

3. Comparison with and without the reserved episodes

Following your suggestion, we compare the original version with a version that computes the per-skill statistics only from randomized episodes. In this version, the reserved skill-free episodes are used only for the family baseline. We conduct two experiments.

  • Experiment 1: Recomputation from existing training records. Using the training records from the cold-start ALFWorld run, we recompute utility for every skill that passed incubation. The original estimate includes the reserved skill-free episodes in the control arm. The alternative estimate uses only randomized episodes. The estimates differ by 0.014 on average, which is much smaller than the 1/81/8 in your example. The utility sign agrees for 96.4% of the skills, and 95.8% of removal and retirement decisions agree. The upward bias depends on how much less often the reserved episodes succeed compared with the randomized control episodes, where other skills can still be injected. In your example, success depends entirely on skill B. The randomized control episodes succeed half the time, and the reserved episode never succeeds. Since the reserved episode accounts for about one quarter of the control arm, the bias is 1/4×1/2=1/81/4 \times 1/2 = 1/8. In our training records, the reserved episodes succeed only 5.6% less often than the randomized control episodes. This smaller gap is consistent with success rarely depending entirely on another skill. Applying the same calculation gives 1/4×0.056≈0.0141/4 \times 0.056 \approx 0.014, which matches the measured difference.

  • Experiment 2: End-to-end retraining. To further verify this, we retrain TrackRL end to end on ALFWorld with the version restricted to randomized episodes. All settings are kept the same as in the original experiment. The only change is that the utility and mastery of each skill are computed from randomized episodes only. We evaluate the resulting checkpoint in the skill-assisted and skill-free modes on the seen and unseen splits and report SR (%).

VersionSeen skill-assistedSeen skill-freeUnseen skill-assistedUnseen skill-free
Original78.172.771.964.1
Randomized episodes only78.973.472.764.8

As shown in the table, the two versions reach comparable SR in all four settings, so the final performance of TrackRL is robust to this choice.

Overall, we address your concern in three ways. (1) Including the reserved skill-free episodes introduces a small bias (0.014 on average) because success on our tasks rarely depends entirely on another skill. (2) The upward bias makes the removal criterion more conservative, reducing the risk of mistakenly removing a helpful skill. When we recompute the per-skill statistics using only randomized episodes, 95.8% of removal and retirement decisions stay the same. (3) Retraining TrackRL with per-skill statistics computed only from randomized episodes yields comparable final performance. The per-skill utility estimates and our results are therefore robust to whether the reserved episodes count toward the control arm.

We have described both versions and reported the results of both experiments in Appendix F.4 of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (3/5)

Q1: Is there evidence that directly validates the per-skill utility estimates?

Thank you for this constructive suggestion. Following your suggestion, we conducted a single-skill intervention test at a fixed policy checkpoint. In this test, we inject or withhold a single skill while keeping everything else unchanged, and compare the measured effect with the utility that TrackRL estimated for that skill during training.

Setup. We take the checkpoint at training step 100 of the cold-start ALFWorld run and keep the policy fixed. Based on the utility tracked up to this step, we select two groups of skills. (1) Helpful skills are active skills whose utility exceeds zero by two standard errors. (2) Harmful skills are skills removed as harmful, whose upper confidence bound on utility is below zero. For each selected skill, we identify the tasks in the ALFWorld evaluation set for which that skill is retrieved. We run the fixed policy on each task twice, once with the target skill injected and once with it withheld. Both runs use the same other injected skills.

Results. The table below reports the results for the helpful and harmful skills. Tracked utility is the utility accumulated during training up to this checkpoint, and SR change is the difference in success rate between injecting and withholding the skill in the intervention test. Both are averaged over the skills in each group. Sign agreement is the percentage of skills whose SR change has the same sign as their tracked utility. If the utility estimates are accurate, the SR change should be close to the tracked utility and have the same sign.

SkillsTracked utilitySR change (%)Sign agreement (%)
Helpful+0.14+12.091.7
Harmful−0.16−15.0100.0

The table leads to two findings: (1) The measured effects are close to the tracked utility at the group level. Helpful skills have an average utility of +0.14, and injecting them raises SR by 12.0%. Harmful skills have an average utility of −0.16, and withholding them recovers 15.0%. The utility tracked during training therefore reflects the actual effect of the skills on the policy. (2) This consistency also holds for individual skills. The SR change has the same sign as the tracked utility for 91.7% of the helpful skills and 100.0% of the harmful skills, which shows that the per-skill utility estimates are accurate and reliable for lifecycle decisions.

In summary, the test validates the estimates in two ways. (1) At a fixed checkpoint, skills estimated to be helpful do improve performance, and withholding skills estimated to be harmful does recover performance, so the utility estimates reflect the actual effect of each skill. (2) Each comparison holds the policy and the other injected skills fixed. This isolates the contribution of the target skill and directly tests whether TrackRL attributes the effect to that skill.

We have added this experiment to Appendix F.3 of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Response by Authors

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their time, insightful comments, and constructive suggestions. The rebuttal and discussion phase has been highly productive, and we believe the paper has been significantly strengthened in both empirical rigor and presentation.

Strengths Acknowledged by the Reviewers

  1. Efficient and principled per-skill attribution. Randomizing the injection of each selected skill within GRPO groups was recognized as a valuable and efficient approach to the attribution problem caused by skill co-occurrence, mitigating free-riding and task-difficulty confounding without additional rollouts (Reviewers EGAP, RtnQ).
  2. Novel and practical lifecycle perspective. Tracking how skills remain useful, become redundant, or turn harmful as the policy evolves, and using this evidence to guide lifecycle decisions, was viewed as a novel and meaningful extension beyond accumulating or periodically pruning a skill library (Reviewers EGAP, RtnQ, vRLH).
  3. Systematic and testable lifecycle criteria. The separate tracking of utility and mastery, together with distinct conditions for retirement, removal, and eviction and evidence requirements for new skills, was highlighted as a systematic design based on explicit decision criteria for skill lifecycle management (Reviewers EGAP, RtnQ).
  4. Strong and comprehensive empirical validation. The experimental evaluation covers multiple benchmarks, skill-assisted and skill-free modes, cold-start and SFT initialization, and ablation studies, distinguishing external assistance from learned capability and supporting the paper's empirical strength (Reviewers EGAP, RtnQ, vRLH).

Main Concerns Raised by the Reviewers

  1. Evidence for skill internalization. The clarification needed was to distinguish the internalization of retired skills from compensation by co-injected skills and from general policy improvement (Reviewers EGAP, RtnQ, vRLH).
  2. Validation of per-skill utility estimates. Further analysis was needed on the effect of the reserved skill-free episodes on the per-skill utility estimates, along with direct validation of these estimates (Reviewers EGAP, vRLH).
  3. Design rationale for retirement and exit types. Clearer justification was requested for retiring internalized skills rather than retrieving them less often, and for the role of mastery and distinct exit types beyond utility (Reviewers EGAP, RtnQ).
  4. Baseline details and additional comparisons. Clarification of the baseline rewards and an additional comparison with SLIM were requested (Reviewer RtnQ).

Revisions in the Rebuttal and Discussion Phase

  1. Internalization experiments. We analyzed the skill-free success of retired skills and conducted a controlled evaluation with and without the other skills, showing that the retired skills are internalized rather than supported by co-injected skills. We further compared the tasks of retired skills with matched tasks of the same family and with GRPO trained without a skill library, showing that the gains on these tasks go beyond general policy improvement.
  2. Utility validation and robustness analysis. We conducted single-skill interventions at a fixed checkpoint, showing that the tracked utility agrees with the measured effect of each skill. We also showed that the bias from the reserved episodes provides a desirable safety property for skill removal, and that retraining without it barely changes lifecycle decisions or final performance.
  3. Lifecycle design ablations. We compared TrackRL with variants without retirement, without distinct exit types, and with utility-based or capacity-based pruning, showing that retirement and mastery keep the library aligned with what the current policy still needs and improve performance. We also evaluated retired skills on the initial policy, showing that retirement preserves their value as reusable knowledge.
  4. Baseline clarification, SLIM comparison, and stability analysis. We clarified that all baselines use their original rewards. We compared TrackRL with SLIM, showing better performance and more reliable attribution without extra rollouts. We also confirmed the stability of the gains with three-seed results and a sign test over all independent runs.

Final Remarks

We believe these results address the main concerns, while the core contributions acknowledged by the reviewers remain strong. We are grateful that Reviewer EGAP raised the score, that Reviewer vRLH confirmed that our rebuttal addressed the main concerns and will advocate for the acceptance of our paper, and that Reviewer RtnQ found our response effective in addressing the concern on skill retirement. We are committed to continuously improving this work. We sincerely appreciate the reviewers' recognition, suggestions, support, and advocacy.

Sincerely,

Authors of Paper #30626

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.