- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TrackRL for managing an evolving skill library during agentic reinforcement learning. It randomizes the injection of individual skills within GRPO groups and uses outcome comparisons to track skill utility, model mastery, and accumulated evidence. These statistics guide skill admission, incubation, retention, retirement, removal, and capacity eviction. Experiments on ALFWorld and AppWorld evaluate both skill-assisted and skill-free performance.
Critical strengths
-
A practical problem and a coherent framework. The paper recognizes that skills can remain useful, become redundant, or turn harmful as the policy evolves. Tracking these changes and using evidence to guide lifecycle decisions is a meaningful extension beyond accumulating or periodically pruning a library.
-
Evaluation distinguishes external assistance from learned capability. The experiments include skill-assisted and skill-free evaluation, as well as base-model and SFT initialization. TrackRL improves performance even without the library: under cold start, TrackRL (Luna) achieves 72.7% skill-free success on seen ALFWorld tasks, compared with 58.6% for GRPO.
Critical weaknesses
-
The causal interpretation needs clarification. Reserved episodes omit all skills of a given granularity and enter individual skills’ control arms. This may create differences in co-injected skills between treatment and control, complicating attribution to a single skill. Control-arm success may also reflect assistance from other skills, so it does not directly establish capability internalization.
-
The correspondence between the paper and implementation is unclear. The paper defines
when_to_applyas the retrieval key, whereas the code prioritizesretrieval_obs, derived from the source task and observation. The implementation also supports validation-driven library updates and optional same-task paired evaluation. Without complete run configurations, adherence to the stated evaluation protocol cannot be confirmed.
Questions for authors
-
Do the reserved skill-free episodes contribute to individual skill utility estimates? If so, how is comparability in co-injected skills maintained between treatment and control? Could the authors report estimates excluding these reserved episodes?
-
Could the authors provide the actual run configurations, clarify the retrieval key used, and confirm that library updates were disabled during evaluation and both modes used the same tasks?
7 comments
Rebuttal by Authors (2/4)
W1 &Q1 (continued): Reserved episodes complicate attribution to a single skill, and control-arm success does not directly establish capability internalization
2. Retired skills are internalized rather than assisted by other skills
Although a control-arm episode for a skill may still contain other skills, compensation by these skills does not explain the high mastery of the retired skills. We explain this from two aspects: the mechanism design of TrackRL limits compensation, and the empirical evidence shows that the retired skills are internalized.
TrackRL's mechanism design limits compensation.
- Admission filters out near-duplicate skills. TrackRL admits a new skill only if it is dissimilar to all active skills, keeping directly substitutable near-duplicates out of the active library.
- The control arm includes episodes that also withhold other skills. TrackRL independently randomizes whether each selected skill is injected. Therefore, other task-level skills can also be withheld in a given skill’s control episodes. Each group also reserves one episode in which all task-level skills are withheld. If the policy relies on other skills, it is more likely to fail when those skills are absent. These failures lower the skill’s estimated mastery, making it less likely to be retired.
- TrackRL can reverse retirement. TrackRL can regenerate and readmit a retired skill if later failures show that the policy still needs it. Skills removed as harmful are ineligible for readmission. If temporary compensation often led to premature retirement, we would expect the policy to fail again on the associated tasks and many retired skills to be regenerated.
Empirical evidence for internalization. We examine the internalization of retired skills directly from two aspects: (1) the skill-free success of the retired skills in the existing training records, and (2) a controlled evaluation of the trained policy with and without other skills.
Skill-free success from the training records. We first analyze the training records from the cold-start ALFWorld run. Each training group reserves one episode with no task-level skills injected. These episodes show how the policy performs when other task-level skills are unavailable. For each of the 14 retired skills, we identify the groups that selected it. We measure SR in the reserved episodes from the first 10 such groups after admission and the last 10 before retirement. Averaged across retired skills, SR rises from 16.4% at admission to 85.7% at retirement, close to the estimated mastery (87.2%). For skills evicted because of non-positive utility, the corresponding SR before eviction is 31.2%. By retirement, the policy can therefore solve most of the associated tasks without any task-level skills. This confirms that the policy has internalized these skills and can solve their tasks without compensation from other skills.
Rebuttal by Authors (3/4)
W1 &Q1 (continued): Reserved episodes complicate attribution to a single skill, and control-arm success does not directly establish capability internalization
Controlled evaluation with and without other skills. Inspired by your comment, we further designed a controlled evaluation to test whether the high mastery comes from internalization or from compensation. Let denote the initial policy before RL and the final trained policy. For each retired skill , we evaluate both policies on the ALFWorld evaluation tasks for which is retrieved. We use the same tasks across conditions and report SR averaged across retired skills. We consider three skill settings: (1) "None" injects no skill, (2) "Retired skill" injects only , and (3) "Other skills" injects the skills retrieved from the active library, which no longer contains . If the final policy depends on other skills to achieve high success, its SR should drop substantially from "Other skills" to "None."
| Policy | Skills | SR (%) |
|---|---|---|
| None | 21.4 | |
| Retired skill | 42.9 | |
| None | 86.2 | |
| Other skills | 86.9 | |
| Retired skill | 87.1 |
The table leads to two findings: (1) Under , the retired skill raises SR from 21.4% to 42.9%, so the retired skills carry useful guidance for a policy that has not yet learned them. (2) Under , removing all other skills changes SR by only 0.7% (86.9% to 86.2%), and injecting the retired skill again brings no further gain (87.1%). The trained policy thus solves these tasks on its own, which supports internalization. Overall, the policy’s ability to solve these tasks on its own explains the high mastery of the retired skills. (1) By design, TrackRL withholds other skills in the control arm for a given skill and can regenerate a retired skill if the policy still needs it. (2) Empirically, the policy already succeeds on the associated tasks without any task-level skill at the time of retirement, and removing all other skills from the trained policy changes SR by only 0.7%. Since this evidence is collected for each retired skill on its own tasks, it shows that the retirement decisions single out skills that the policy has internalized.
In summary, (1) excluding the reserved skill-free episodes makes co-injected skills comparable across treatment and control arms. Utility estimates, lifecycle decisions, and final performance remain similar, supporting attribution to the target skill itself. (2) The trained policy solves tasks associated with retired skills even when no skills are injected, showing that it can succeed without compensation from other skills.
We have described both versions and added these analyses to the revised paper (Section 3 and Appendices F.4 and F.5).
Rebuttal by Authors (1/4)
We appreciate your insightful feedback and constructive comments. We address each of your comments below.
Thank you for this comment and the related question. We confirm that the reserved skill-free episodes are counted in the control arm of every skill selected in them, as stated in Appendix D.1. Counting these episodes leaves most lifecycle decisions unchanged and yields comparable final performance, and the retired skills are internalized by the policy rather than supported by other skills. We explain these two points in turn.
1. Effect of counting the reserved episodes in the control arm
Each group reserves one task-level and one step-level skill-free episode, in which the selected skills of that granularity are not injected. All other episodes are randomized independently for each skill, so among them the co-injected skills are comparable between the treatment and control arms of a skill. The reserved episode adds to the control arm one episode in which the other skills are withheld as well.
Motivation for reserved skill-free episodes. A training group contributes to a skill’s utility estimate only if both arms contain an episode. With few episodes per group, randomization can leave the control arm empty. Counting the reserved skill-free episode ensures a nonempty control arm without extra rollouts, allowing evidence to accumulate faster for newly admitted skills.
The positive effect of the upward bias on removal. This bias provides a desirable safety property for skill removal. As your comment shows, when co-retrieved skills are helpful, mixing in the skill-free episode introduces a small upward bias to the target skill's utility us. Meanwhile, in our lifecycle management (Eq. 11), permanently removing a harmful skill (Removal) requires its upper confidence bound to fall below zero: us+λSE(us)<0. This upward bias therefore makes removal slightly more lenient for every skill. A skill whose upper confidence bound still falls below zero despite this bias is clearly harmful. In other words, this bias lowers the false-positive removal rate, i.e., the chance of removing a helpful skill by mistake.
Comparison with and without the reserved episodes. Following your suggestion, we compare the original version with a version that computes the per-skill statistics only from randomized episodes. In this version, the reserved skill-free episodes are used only for the family baseline. We conduct two experiments.
Experiment 1: Recomputation from existing training records. Using the training records from the cold-start ALFWorld run, we recompute utility for every skill that passed incubation. The original estimate includes the reserved skill-free episodes in the control arm. The alternative estimate uses only randomized episodes. The estimates differ by 0.014 on average. The utility sign agrees for 96.4% of the skills, and 95.8% of removal and retirement decisions agree. The upward bias depends on how much less often the reserved episodes succeed compared with the randomized control episodes, where other skills can still be injected. In our training records, the reserved episodes succeed only 5.6% less often than the randomized control episodes. This small gap is consistent with success rarely depending entirely on another skill. Since the reserved episode accounts for about one quarter of the control arm, the bias is 1/4×0.056≈0.014, which matches the measured difference.
Experiment 2: End-to-end retraining. To further verify this, we retrain TrackRL end to end on ALFWorld with the version restricted to randomized episodes. All settings are kept the same as in the original experiment. The only change is that we compute each skill’s utility and mastery using only randomized episodes. We evaluate the resulting checkpoint in skill-assisted and skill-free modes on the seen and unseen splits and report the success rate (SR, %).
As shown in the table, the two versions reach comparable SR in all four settings, so the final performance of TrackRL is robust to this choice.