acceptodds
Under review as a conference paper at ICLR 2027

Continual-Skill: Utility Optimization-Driven Co-Evolution of Policies and Skill Libraries

Abstract

Self-improving agents continually accumulate reusable skills, yet they lack an efficient mechanism for determining whether a stored skill still benefits the current policy. Semantic relevance does not reveal a skill’s actual effect on task performance, while explicit with/without-skill evaluation requires costly additional environment interactions. We introduce **Continual-Skill**, a framework that turns ordinary training trajectories into online evidence of skill utility. Its key idea is to randomize skill exposure during training, creating exposed and unexposed observations for candidate skills, and to use inverse-probability weighting to estimate their marginal effects on agent performance. The resulting utility estimates are updated alongside the policy and directly govern skill routing and lifecycle decisions, including promotion, suspension, and archival; persistent task-domain failures additionally trigger new-skill generation. This design allows policy learning and skill reassessment to share the same interaction data, avoiding separate per-skill evaluation rollouts. On ALFWorld and WebShop, Continual-Skill remains robust under increasingly interfered skill banks and improves success rate by up to 40.4 percentage points over strong skill-learning baselines on expanded WebShop banks. Results suggest that continual agent improvement requires treating skill utility as a policy-dependent quantity that must be repeatedly identified and updated throughout learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill Self Evolution, Self Improvement, Agent Systems, LLM Agents.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_R4a7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Continual-Skill, a method that incorporates skills into agentic GRPO training to improve agents’ ability to invoke skills. It makes skills available during LLM rollouts according to prioritized exposure probabilities and uses these probabilities to weight group returns for GRPO updates. Experiments demonstrate its advantages in settings with perturbed skill banks.

Critical strengths
  • The paper addresses an important problem, and the proposed approach is relatively simple and reasonable.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • Reliance on hand-designed heuristics. The method relies heavily on manual design choices, particularly in its central exposure probability mechanism. It also introduces numerous hyperparameters, making their selection potentially challenging. Clearer guidance on how to choose these hyperparameters would improve the method’s practical usability.
  • Unclear source and practical relevance of the gains. The main advantages appear in the larger, perturbed skill banks, while the method shows no clear improvement over the baselines with a clean skill bank. This raises questions about its intended application scenarios and the reasons for its stronger performance in the Medium and Large settings. Are the gains attributable to the larger number of skills, the presence of perturbations, or an interaction between the two? My interpretation is that the improvements may largely stem from the manually designed exposure probabilities suppressing interference from perturbed skills. However, such settings may be less common in practice, where smaller, cleaner skill banks resembling the Small setting may be more representative. Further analysis is needed to clarify when and why the method is beneficial.
  • Insufficient hyperparameter sensitivity analysis. How sensitive is the method to its many hyperparameters, and how do their values affect performance? A more systematic analysis would help establish the robustness of the reported results.
  • Potential confounding effect of the number of injected skills. The proposed method and the baselines inject different numbers of skills. Could this difference substantially affect performance? Comparisons that control for the number of injected skills would help isolate the contribution of the proposed method.
  • Unclear contribution of exposure-based weighting. What effect do the weights based on exposure probabilities in Equation (6) have on performance? An ablation of this weighting mechanism would help establish its contribution.
  • Limited scope of the ablation studies. The effects of individual components may vary substantially across skill banks. Could the authors conduct ablation studies across different bank settings, rather than restricting them to the Large bank in a single environment?
Questions for authors

See Weaknesses.

3 comments

We sincerely appreciate the thorough and constructive feedback provided on our manuscript. In response to the reviewers’ comments, we have carefully addressed each point below.

W1 & W3: Parameter design and sensitivity

We believe the current results support the use of this parameter configuration across environments. The paper uses the same default routing and management parameters on ALFWorld and WebShop and reports results at three skill-bank sizes. These experiments directly test whether the parameters transfer across settings.

Exposure probabilities also follow an explicit rule. The system first specifies the budget for providing skills and then computes probabilities based on the number of candidates. For example, with 10 general-skill candidates and a budget of 2, each candidate has an exposure probability of 0.2; with 10 task-specific skill candidates and a budget of 6, the probability is 0.6. In practice, users choose the exposure budget, and the system computes the corresponding probabilities.

Skill-management parameters specify the evidence required to take action. The default configuration requires at least 40 observations in total, with at least 12 observations in both the exposed and unexposed groups; archival due to negative utility additionally requires two consecutive confirmations. These rules link every state transition to specific observations and decision criteria.

The paper therefore provides both the rationale for parameter selection and results obtained using the same parameter configuration across settings. The sensitivity experiments suggested by the reviewer could further quantify how parameter changes affect performance, while the current results already support the effectiveness of a shared default configuration in the evaluated settings.

W2 & W4: Sources of improvement and the number of provided skills

The improvements in expanded skill banks directly demonstrate the value of our method under skill accumulation. The paper retains the original skills, adds redundant, overly specific, outdated, or conflicting content, and initializes all methods with the same skill bank. On WebShop, our method exceeds the strongest comparison baseline reported in the original paper by 31.40 and 40.40 percentage points in success rate under the Medium and Large conditions, respectively. These are direct comparisons under the same initial skill-bank conditions.

Existing methods perform well in the Small setting, showing that they can effectively use relatively clean initial skill banks. As the banks expand, their performance diverges substantially, indicating that skill accumulation changes the relative performance of these methods. Together, these results clearly identify the conditions in which our method offers its main advantages.

Synthetic expansion enables reproducible evaluation of problems arising from skill accumulation. Expansion simultaneously increases the number of skills and changes their content composition, so the results evaluate the combined effects of these two changes. The perturbation types listed in the paper describe the content being tested; their prevalence in real skill banks would need to be measured using data from actual accumulation.

The number of skills provided by each method is reported in Appendix Table 14. The main experiments retain each method's own exposure strategy and compare complete systems. Matching the number of provided skills would enable a further comparison of skill selection under the same budget. The conclusion supported by the current main results is that, given the same initial skill bank, our complete method achieves a clear advantage under expanded-bank conditions. This advantage reflects the combined effects of skill selection, the number of provided skills, and the management mechanism.

W5 & W6: Probability weighting and component ablations

Equation (6) corrects for bias arising from differences in exposure probability. A skill that is provided more frequently receives more feedback, so directly accumulating this feedback makes the estimate depend on exposure frequency. Equation (6) weights observations by their actual exposure probabilities so that the estimate corresponds to the difference in mean feedback between providing and withholding the skill.

This follows directly from the formula. Let the mean feedback in the two groups be μ1\mu_1 and μ0\mu_0, and let the exposure probability be pp. Under the assumptions stated in the appendix, the weighted expectation is μ1−μ0\mu_1-\mu_0; after removing the weights, the expectation becomes pμ1−(1−p)μ0p\mu_1-(1-p)\mu_0. The latter varies with exposure probability. The paper therefore explains the role of weighting in utility estimation. Quantifying its contribution to final success rate would require a control experiment that removes these weights.

The component experiments provide two concrete pieces of evidence. In Table 3, the full configuration outperforms the configuration with management disabled across all three seeds, improving mean success rate by 2.49 percentage points. In Table 4, the configuration combining semantic information and utility achieves the highest mean success rate. These results support the roles of the corresponding mechanisms in the ALFWorld Large setting.

The evaluation of the complete method covers two environments and three skill-bank sizes, while the decomposition of component contributions focuses on ALFWorld Large. Accordingly, we claim that the complete method is competitive across the tested settings and use ablations to explain the sources of performance in one of those settings. Component experiments across additional settings could further examine how individual contributions vary; the current conclusions are supported by the two bodies of evidence described above.

Thank you for the rebuttal. I recognize that time for additional experiments is limited during the rebuttal period, and I find the authors’ explanations for W1 and W3 satisfactory, as these concerns are less central to my assessment. For W2 and W4, however, experimental evidence is essential to understanding why the method works, under what conditions it succeeds, and when it fails. The results provided so far are therefore insufficient for me to raise my score.

Reviewer_azkROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies problem of managing growing skill banks that accumulate redundant or conflicting guidance by proposing a framework, Continual-Skill. It estimates marginal skill utility through randomized skill exposure and inverse-probability weighting of trajectory feedback during GRPO training, using these estimates for skill routing and lifecycle management without separate per-skill evaluation rollouts. Experiments on noise-expanded skill banks, which approximate the accumulation of redundant, outdated, and conflicting guidance, show comparable performance to baselines on ALFWorld (text-based embodied planning). On WebShop (multi-constraint product search and purchasing), Continual-Skill shows greater robustness than existing skill-augmented agent frameworks as noise increases in the skill bank.

Critical strengths

I think agents increasingly combine evolving internal policies with external memory, making the accumulation of outdated or redundant external knowledge an important yet underexplored bottleneck. Utility-guided skill selection and lifecycle management offer a promising direction for managing this challenge in external skill libraries.

Critical weaknesses

Given the practical nature of the proposed framework, convincing empirical validation is particularly important. However, improvements over the strongest baselines are modest on ALFWorld, while substantial gains on WebShop are concentrated in synthetically expanded skill banks. As the main results are based on a single seed, further evidence would help establish the reliability and generality of these improvements.

Even from a robustness perspective, the practical motivation for the evaluation setting could be better established. The substantial gains are largely observed under synthetically expanded skill banks, while SLIM with the original Small bank achieves comparable or better WebShop performance. This raises the concern of how well the constructed interference reflects realistic skill accumulation, and whether the reported robustness translates into a meaningful practical advantage.

Questions for authors

Could authors clarify why the evaluation uses synthetically expanded skill banks rather than continual settings such as CL-ALFRED [1], where changes in skill utility over time could be examined more directly?

In the noise-expanded skill banks, how many skills were pruned or downweighted during training, and what fraction of them were injected noisy skills rather than original skills?

Could authors discuss whether stronger base models would reduce skill-bank interference on their own, and how this would affect the need for utility-guided skill management?

[1] Online Continual Learning for Interactive Instruction Following Agents

3 comments

We are grateful for the detailed concerns and suggestions provided by the reviewer. Now we respond to these comments.

W1: Stability of performance improvements

Figure 4 already presents the three-seed training dynamics on ALFWorld with the Medium initial skill bank. The corresponding success rates are provided below.

Method / SR (%)Seed 372Seed 373Seed 374Mean ± SD
Skill182.0976.1282.0980.10 ± 3.45
CONTINUAL-SKILL82.8482.8481.3482.34 ± 0.86

We acknowledge that single-seed results in the main table cannot replace stability validation. Within our available computational resources, we added complete three-seed runs for CONTINUAL-SKILL with the WebShop Large initial skill bank, as reported below.

WebShop Large (both SR and Score are reported as percentages)

MethodMetric (%)Seed 1Seed 2Seed 3Mean ± SD
Skill1SR4.606.802.804.73 ± 2.00
Skill1Score13.2422.5712.1215.98 ± 5.74
SkillRLSR6.805.602.805.07 ± 2.05
SkillRLScore30.5718.4924.8124.62 ± 6.04
CONTINUAL-SKILLSR49.8042.4043.2045.13 ± 4.06
CONTINUAL-SKILLScore58.4954.9256.7756.73 ± 1.79

Across these three seeds, CONTINUAL-SKILL outperforms both baselines in the table on success rate and Score. Together with the three-seed ALFWorld results above, these results provide evidence of performance across multiple runs.

W2 / Q: Why use synthetic expansion rather than CL-ALFRED?

The two settings examine different sources of change. CL-ALFRED introduces new behaviors or environments. Our experiments hold the task setting and original skill coverage constant while varying the interference load in the skill bank, thereby examining whether a system can continue to use its accumulated experience effectively as that experience grows. Repeated formulations, overly specific experience, outdated steps, cross-task misuse, and seemingly plausible conflicts correspond to concrete failure modes in experience accumulation. Synthetic construction makes the sources of these factors traceable and facilitates comparisons of management effectiveness; we do not claim that their proportions match those in real deployments. The best performance with the Small bank measures the ability to use a clean skill bank, whereas the expanded-bank experiments measure robustness to increased interference. These should be evaluated separately. Moreover, even with a fixed task distribution, model learning changes a skill's marginal contribution, which is already captured by the paper's policy-dependent definition of utility.

Q: Which skills were actually removed or downweighted?

We reanalyzed skill-management records by skill source for runs with the Large skill banks in both environments.

EnvironmentSkill sourceNumber of skillsArchived at the endSuspended at the endDownweighted at least once during training
ALFWorldOriginal skills44008
ALFWorldSynthetically added skills39610127
ALFWorldOnline-generated skills213012
WebShopOriginal skills541014
WebShopSynthetically added skills48612132
WebShopOnline-generated skills271304

These results show that skill management operates primarily by adjusting opportunities for use, while the system is comparatively cautious about archival and suspension. More directly, 13 of the 27 online-generated skills in WebShop were archived because of insufficient benefit. These skills were not subjected to synthetic noise injection, demonstrating that quality filtering also applies to new experience generated by the system itself.

I appreciate the authors' rebuttal and additional experiments. While the WebShop results show substantial gains under expanded skill banks, the improvements over the strongest baseline on ALFWorld remain marginal, making practical relevance a key concern. The response to W2 clarifies the purpose of the synthetic evaluation, but I remain unconvinced that the observed robustness translates into practical benefits under realistic skill accumulation or sufficiently demonstrates the practical value of skill management. I therefore maintain my original score.

Thank you for highlighting practical relevance. We examined two completed experiments in which the agent starts with an empty skill bank (which represents the real skill accumulation process), generates skills during training, and evaluates them through the lifecycle management mechanism. These experiments directly examine skill accumulation arising from the agent’s own learning process, with no synthetic distractors injected.

The following results are recorded at training step 200:

EnvironmentInitial skillsSkills addedActiveSuspendedArchivedSR (%)Score (%)
ALFWorld0904005083.58—
WebShop01495319547.0058.74

Starting from zero skills, the agent accumulated 90 skills on ALFWorld and 149–150 on WebShop. Management retained 40 and 53–55 skills, respectively, for continued use or exploration. This identifies a concrete workload under natural accumulation: evaluating newly generated instructions and deciding which should continue to receive opportunities for use.

In the actual accumulation from the empty skill bank, the archive records also distinguish the reasons for filtering. In each WebShop run, 95 skills were archived as unprofitable under the configured utility criterion. In ALFWorld, 49 were archived as unprofitable and one for persistent negative utility. Thus, the observed screening workload includes both skills with insufficient demonstrated benefit and a skill with sustained negative evidence. The results show that the performances in the test set is consistent with the ones we presented in Table 2, and the skill management worked as designed. We will add these results in the appendices in later versions of the PDF.

These experiments extend the empirical scope to autonomous skill generation from an empty bank. They demonstrate that the system can generate, evaluate, and selectively retain accumulated skills while achieving the reported task performance. This evidence addresses the occurrence and operation of skill management under natural accumulation.

We would sincerely appreciate it if you could reconsider your assessment and rating according to the additional results

Reviewer_qumuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CONTINUAL-SKLIL, a continuous management framework for llm agent skill libraries that treats utility as a policy-dependent value to be iteratively estimated during training.

Critical strengths
  • The problem framing is well motivated. In particular, the paper clearly identifies the limitation of treating stored skills as having static utility as the policy and task distribution evolve.
  • Using randomized skill exposure together with inverse-probability weighting (IPW) to estimate skill utility from ordinary training trajectories is a novel and appealing idea. It avoids the need for separate per-skill evaluation rollouts.
Critical weaknesses
  • The experimental results lack sufficient statistical evidence. Agent trajectory rewards are likely to be noisy, and IPW estimators can have high variance, especially when exposure probabilities are close to 0 or 1. However, the main results in Table 2 are reported using only a single seed, making it difficult to assess the robustness of the reported gains.
  • One of the paper's central conceptual claims is that skill utility is policy-dependent and therefore should be continually reassessed. However, the proposed estimator averages (Z_{i,s}) over a recent window that may contain observations collected under different policies, task contexts, and routing distributions. It is therefore unclear how well the resulting estimate tracks the utility of a skill under the current policy.
Questions for authors
  • Have the authors considered alternative estimators, such as doubly robust methods, which might reduce the variance of IPW-based utility estimation?
  • On WebShop, performance with the Medium skill bank is higher than with the Small bank for CONTINUAL-SKILL (53.6% vs. 48.6%). Does this indicate that some of the added perturbed skills are actually beneficial rather than purely interfering? A perturbation-type-level analysis of estimated utility and routing frequency could help clarify this result.
3 comments
AuthorsOpenReview ↗

Author Responses to weaknesses and questions

We thank the reviewer for the detailed review. Here are our responses to the concerns mentioned in the review content.

W1/Q1: Trajectory reward variability and the stability of inverse-probability weighting

Our estimator targets the average contribution of a skill under randomized exposure, rather than exact attribution within an individual execution. Equations (6)–(7) in the main text define the trajectory-level contribution and its window average, and Proposition 1 establishes conditional unbiasedness under ordinary randomized exposure. Variability in execution rewards does not alter this identification argument. The paper addresses variance at three levels: centering feedback using a task-category baseline fixed before each batch; preserving comparisons between selected and unselected observations through the exposure design; and deciding whether to take management actions based on the sample size and empirical standard error within the window. The evidence thresholds require at least 40 observations in total and at least 12 on each side. Archival due to negative utility additionally requires the upper bound to remain below the threshold in two consecutive audits. The system therefore accumulates evidence to determine whether a negative effect warrants action, rather than deleting a skill as soon as its estimated utility becomes negative. Conditional unbiasedness concerns whether the estimator targets the intended quantity, whereas the evidence thresholds determine whether a decision is supported with finite samples; the two serve distinct purposes.

W2: Single-seed main results and stability across runs

Indeed, single-seed results in the main table cannot replace stability validation. Within our available computational resources, we added complete three-seed runs for CONTINUAL-SKILL with the WebShop Large initial skill bank, as reported below. These provide empirical results under an additional skill-bank setting. CONTINUAL-SKILL outperforms the other baseline methods in all three seeds.

WebShop Large (both SR and Score are reported as percentages)

MethodMetric (%)Seed 1Seed 2Seed 3Mean ± SD
Skill1SR4.606.802.804.73 ± 2.00
Skill1Score13.2422.5712.1215.98 ± 5.74
SkillRLSR6.805.602.805.07 ± 2.05
SkillRLScore30.5718.4924.8124.62 ± 6.04
CONTINUAL-SKILLSR49.8042.4043.2045.13 ± 4.06
CONTINUAL-SKILLScore58.4954.9256.7756.73 ± 1.79

CONTINUAL-SKILL achieves a substantially higher mean success rate in these runs. The main table covers many tasks, and reproducing each requires substantial computational resources. Given these resource constraints and the trade-off involved in maintaining consistent reporting across the table, we reported the main results using a single seed. Figure 4 also presents multi-seed experiments on ALFWorld. Taken together, the current experiments demonstrate the training stability of our framework.

Q2: Why not use a doubly robust estimator?

Our method relies on accumulating trajectories with randomized exposure so that the average estimate concentrates around the target effect, rather than requiring each individual contribution to be accurate. Proposition 1 establishes conditional unbiasedness under ordinary randomized exposure. With a fixed policy, task distribution, and exposure mechanism, and assuming independent sampling and bounded contribution variance, if the variance of each contribution is at most CC, the average estimator satisfies

\qquad P \left(|\widehat U_n-U|\geq\varepsilon\right) \leq \frac{C}{n\varepsilon^2} \longrightarrow 0.$$ Thus, individual trajectories may exhibit substantial variability, but the probability that the average estimate deviates from the target effect decreases as effective observations accumulate. **The method relies on statistical identification through randomization and aggregation across trajectories, rather than accurate attribution for every rollout.** A doubly robust (DR) estimator can improve estimation efficiency by incorporating a conditional feedback prediction model. However, it is not necessary for the statistical properties above: our exposure probabilities are directly specified by the randomization mechanism, which already allows us to construct unbiased contributions. With finite samples, the system additionally uses sample-size thresholds on both sides and empirical standard errors to determine when to take skill-management actions. Because the policy continually changes during actual training, the paper uses a rolling window to estimate local utility over recent candidate opportunities.

Thank the authors for the rebuttal. The additional multi-seed results address most of my concern about empirical stability, and the clarification of the exposure design makes the IPW variance issue less concerning. However, my main concern remains that the rolling-window estimate mixes data collected under different policies. I will maintain my current score.

Thank you for recognizing the additional stability experiments and the clarification of randomized skill provision. We address your remaining concern about rolling-window estimate by explaining what utility estimates mean when model parameters (i.e. the policy) change.

The window estimates a skill’s average contribution during recent training. Each observation corresponds to the model and task at that opportunity.

Let (\mathcal H_i) contain the model parameters, task, and history before skill provision. Let AiA_i indicate provision with probability pip_i, YiY_i denote feedback, and bib_i be a baseline fixed before randomization. Define

Zi=Ai(Yi−bi)pi−(1−Ai)(Yi−bi)1−pi.Z_i= \frac{A_i(Y_i-b_i)}{p_i} - \frac{(1-A_i)(Y_i-b_i)}{1-p_i}.

Randomized provision gives

E[Zi∣Hi]=E[Yi(1)−Yi(0)∣Hi]=τi,\mathbb E[Z_i\mid\mathcal H_i] = \mathbb E[Y_i(1)-Y_i(0)\mid\mathcal H_i] = \tau_i,

where τi\tau_i is the skill’s expected contribution under that opportunity’s model and task conditions. This identity permits changing model parameters: the current model is included in the conditioning information.

Consequently, Zi−τiZ_i-\tau_i has conditional mean zero and forms a martingale-difference sequence. The martingale-difference property applies to the estimation errors (the policy item itself is not included). In our settings, before each exposure, the current model and provision probability are determined by the available history. The skill is then randomly provided, so IPW gives an expected contribution equal to that opportunity’s utility. Subtracting this utility leaves an error with conditional mean zero. Later model updates depend on earlier outcomes, thus it still keeps this property. This supports analysis without assuming independent, identically distributed trajectories. Related conditional-unbiasedness and martingale arguments are also used in similar adaptive experimental inference works [1].

For a fixed window of nn candidate opportunities, define

\qquad U_W=\frac1n\sum_{i\in W}\tau_i.$$ With bounded centered feedback and provision probabilities uniformly bounded away from zero and one, $$\mathbb E[(\widehat U_W-U_W)^2]=O(1/n).$$ **Thus, observations from different model stages can estimate recent average utility, with a decreasing mean-squared error bound.** Applying this estimate to the current model involves two errors: $$|\widehat U_W-U_{\mathrm{current}}| \le |\widehat U_W-U_W| + |U_W-U_{\mathrm{current}}|.$$ The first item is finite-sample estimation error, and the second is the change from recent average utility to current utility. A negative decision is supported when the margin below the threshold exceeds both errors combined. This makes evidence strength and utility change the relevant criteria. The finite window limits evidence age, while sample requirements and repeated confirmations govern archival. **Training records demonstrate how new evidence changes decisions.** In one ALFWorld Large run, skill A’s estimated utility and upper confidence bound were −0.3278 and −0.0689 at step 45, triggering suspension and further probing. At step 50, they became −0.1060 and +0.3474; the system reset consecutive negative confirmations and continued evaluation. Skill B’s upper bounds remained −0.1444 and −0.1496 at steps 35 and 40. After nine additional selected probes, it still met the negative criterion and was archived. These examples show the estimate’s practical role: improving evidence preserves evaluation opportunities, while persistent negative evidence triggers archival. The theoretical proof establishes why recent observations remain informative under evolving policies, and current-policy accuracy depends on estimation error and local utility change. In later versions of the PDF, we'll add the experimental results and the proof of the settings corresponding to your concerns. We hope this analysis and the training examples clarify your concern. Together with the additional stability results, we sincerely hope you give reconsideration to your assessment. --- **References:** [1] Hadad, V., Hirshberg, D. A., Zhan, R., Wager, S., & Athey, S. (2021). [Confidence intervals for policy evaluation in adaptive experiments](https://doi.org/10.1073/pnas.2014602118). *PNAS*, 118(15), e2014602118.
Reviewer_rLvHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how an agent can determine which skills in a skill library remain useful during continual training. The method randomly provides or withholds candidate skills from the agent, then estimates skill utility from task outcomes using inverse-probability weighting. The same batch of trajectories is used simultaneously for GRPO training and for skill evaluation; utility then drives subsequent retrieval, pausing, and archiving, and task domains that keep failing trigger new skill generation.

The experiments use Qwen2.5-7B-Instruct and compare the original skill library against an expanded skill library augmented with distractors on ALFWorld and WebShop. The main advantage appears on the expanded WebShop skill library, improving success rate by up to 40.4 percentage points over the compared baselines. Ablations respectively disable lifecycle management and skill generation, as well as the utility or semantic information in candidate selection, and the results support the role of these components in the tested settings.

Critical strengths

The method lets skill evaluation and policy training share trajectories, reducing the cost of testing skills one by one. Randomly providing or withholding candidate skills means the training data naturally contains both kinds of observations needed for comparison; after recording the exposure probability, the effect of providing a given skill on task performance can be estimated. A single trajectory can also accumulate evidence for multiple candidate skills at once. This design targets a specific difficulty: the policy keeps changing, so skills need repeated evaluation, while running with/without-skill tests for each skill individually is expensive.

Critical weaknesses
  • Weakness 1: Permanent archival limits adaptation to changes in utility. The Archived state has no re-evaluation or restoration path, and new-skill deduplication includes archived entries. Consequently, even if a skill becomes useful again as policy capabilities change, the system may be unable to exploit it a second time.

  • Weakness 2: Utility aggregation for general skills is too coarse-grained. The method merges a general skill's effects across different task domains into a single utility and manages it globally on that basis. This may cause a skill that is useful only in a particular task domain to be judged as low-value overall. The centralization of the task-group baseline cannot preserve this heterogeneity of effects, which limits the method's ability to identify the conditions under which a skill applies.

  • Weakness 3: No-skill GRPO control is missing. The most pronounced performance advantage appears on the manually expanded distractor skill library, while all main controls and ablations still use the skill library. The current results therefore do not clearly establish how much benefit the complex skill management system provides relative to using no skill library at all during training and evaluation.

Questions for authors
  • Question 1: Are there conditions under which archived skills are re-evaluated or reactivated? If not, how does the system recover a skill when policy changes make an old skill useful again, and how does it avoid being blocked by the deduplication rule that includes archived entries?

  • Question 2: Do general skills retain per-task-domain utility statistics? Can the existing exposure data be used to show whether cross-domain aggregation masks effects that point in opposite directions, and how such skills are managed?

  • Question 3: Could you add results in which neither training nor evaluation provides external skills, under the same initial model, training data, GRPO, and base training trajectory budget? This control would directly help determine the net benefit of skill library management relative to not using a skill library.

4 comments
AuthorsOpenReview ↗

W1 / Q1: Concerns about skill archival

We thank the reviewer for the detailed review. Here we present our responses about these concerns.

W1 / Q1: Concerns about skill archival.

We acknowledge that checking archived skills when deduplicating newly generated skills is a potential limitation that can arise in practice. However, our system imposes stringent conditions on the progressive archival of a skill due to negative utility: archival requires evidence of a persistent negative contribution. First, the total number of observations and the sample sizes in both the selected and control groups must meet the required thresholds. Second, the upper confidence bound on utility must fall below the negative threshold of −0.05, meaning that even the most optimistic value within the estimated interval remains negative. Finally, this condition must be confirmed in two consecutive audits. The first confirmation only triggers suspension, while the system continues probing the skill. If subsequent evidence improves, the count of consecutive negative confirmations is reset.

The following comparison comes from a single ALFWorld training run initialized with the Large skill bank.

Skill / Training stepObservations (selected / control)Mean utilityUpper confidence bound on utilityConsecutive negative confirmationsOutcome
A / 45152 (14 / 138)−0.3278−0.06891Suspended; probing continued
A / 50168 (19 / 149)−0.1060+0.34740Evaluation continued
B / 35208 (13 / 195)−0.3473−0.14441Suspended; probing continued
B / 40248 (22 / 226)−0.3517−0.14962Archived

Skill A advises the agent to “immediately pick up an object that needs to be heated upon seeing it.” After suspension, its utility estimate improved, so the system reset the negative-confirmation count and retained opportunities for further evaluation. Skill B advises the agent to “turn on any desk lamp found switched off during general tasks.” It received nine additional selected probes after suspension. By the next audit, it had accumulated 248 observations, and its upper confidence bound remained at −0.1496, below the negative threshold. Only then was it archived.

This example illustrates the stringent conditions required to archive a skill with negative utility. For overly general advice such as “turn on a desk lamp whenever you see one, regardless of the task,” improvements in model capability alone do not make the advice appropriate.

Archival therefore follows a multi-stage process. The current mechanism does not restore archived skills, but the empirical records show that substantial observation and reassessment opportunities are provided before archival. We have found no evidence that deduplication against archived skills commonly harms subsequent learning; in fact, we did not observe such a case in any of our experiments.

AuthorsOpenReview ↗

W2 / Q2: Utility of general skills

We also thank the reviewer for raising the issue of task-specific utility. The fact that pooled statistics can obscure differences across tasks does not necessarily imply that these differences readily lead to erroneous skill removal. As the reviewer notes, the current system does not separately maintain per-task utility statistics for general skills, and the task-category feedback baseline serves to calibrate feedback. However, the recorded trajectories include task categories, feedback signals, skill-selection indicators, and selection probabilities, allowing us to reconstruct task-specific results using the original utility estimator. We used these records to examine 40 runs comprising 394,144 trajectories.

One concrete example comes from ALFWorld with the Large skill bank. The general skill “Addressing Positional Objectives” instructs the model to identify a reference object from the positional relation in the goal, and then search for or place the target object. Its utility is positive for cooling tasks and negative for examination tasks, confirming the cancellation between positive and negative effects highlighted by the reviewer.

Task categoryCandidate observationsSelectedControlMean utilityStandard error
Cool961581+0.24380.1062
Look801268−0.47100.1825
Pick and place496100396−0.02160.0681

Nevertheless, the actual management outcome shows that this skill remained active and continued to appear in regular training prompts. It was selected for regular prompts 170 times in total, including 145 selections while its routing status was marked as downweighted; usage was still recorded at training step 200. Thus, negative pooled utility did not result in its global deactivation. We also examined archived skills and identified 219 archival decisions involving general skills. For 216 of these, we could exactly reconstruct the evidence window at the time of archival; the remaining three were excluded because of reconstruction inconsistencies. Splitting these windows by task category yielded the following results.

CheckResult
Exactly reconstructed archival decisions for general skills216
Task-category groups meeting the sample-size requirements674
Groups with a utility lower bound above zero under the confidence criterion used in the run0

The retained-skill example demonstrates continued use despite negative pooled utility. The archival analysis found no clear evidence of positive within-task utility under the confidence coefficient of 2.8653 actually used in these runs. The current system still manages general skills globally based on pooled utility and its uncertainty, without automatically creating task-specific versions. These records indicate that utility differences across tasks do exist, but there is insufficient evidence that the system discarded locally beneficial skills. In addition, our work defines a general skill as one whose description covers the needs of the vast majority of tasks. We therefore treat general skills as a unified category; assigning task categories according to each skill's effectiveness across tasks would substantially increase system complexity.

AuthorsOpenReview ↗

W3 / Q3: Missing no-skill control

We thank the reviewer for the suggestion of offering no-skill control. Our primary focus is on maintaining the effectiveness of skill-augmented post-training as a skill bank accumulates redundant, outdated, or conflicting content. We therefore use the synthetically expanded Medium and Large skill banks as the core robustness tests, supplemented by the Small skill bank and no-skill GRPO controls to assess the benefits of skill management under different conditions.

The newly added no-skill GRPO results show that direct post-training is also a competitive option. This is consistent with the scope of our conclusions: under the evaluated expanded-bank conditions, CONTINUAL-SKILL is more robust to interference than existing skill-augmented methods; relative to using no skills at all, its benefits still depend on the quality and configuration of the skill bank. The current experiments support the value of its management mechanism in skill banks subject to interference.

Following the reviewer's suggestion, we provide the additional results for plain GRPO without skills below.

ALFWorld:

MethodPickLookCleanHeatCoolPick2Avg.
GRPO (w/o skills)75.0072.2277.4286.9695.2470.5979.85

WebShop:

MethodSRScore
GRPO (w/o skills)51.8060.35

The initial model, training data, and trajectory configuration match those of CONTINUAL-SKILL, with training conducted for 200 steps.

Dear Reviewer rLvH,

Thank you for your time and careful review, we sincerely appreciate your recognition of our work.

We have posted three separate responses addressing your concerns, including skill archival, utility of no-skills and w/o skill control. We also add some extra experiments accordingly.

Feel free to let us know if you have any further questions according to the response we presented.

Best, Authors

Comments on the paper

1 comment

Dear reviewers,

As we only have a few more days to go before the discussion window closes, please go through the author responses and acknowledge them. You may highlight aspects of your initial review comments that have been satisfactorily addressed by the responses, as well as those which require further clarification.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.