acceptodds
Under review as a conference paper at ICLR 2027

Selm: Self-Evolving Library Management for Multi-Skill Agent Optimization

Abstract

Large language model (LLM) agents increasingly use skills as a lightweight way to acquire task-specific knowledge. Recent work enables skills to evolve from task trajectories. However, these methods often lack systematic management of evolved skills, making it difficult for agents to reuse and accumulate skills across domains, thus limiting performance on subsequent tasks.To reliably accumulate, reuse, and maintain useful skills across tasks, we present SELM (Self-Evolving Library Management), a framework that formulates skill evolution as library-level management. SELM jointly manages skill retrieval, updating, and lifecycle by maintaining skill confidence and utility signals. It introduces utility-aware retrieval to select useful skills, a Map-Reduce framework to generate targeted updates from task-specific experience, and lifecycle controls to retain or remove skills based on their effectiveness. These mechanisms enable the skill library to continuously accumulate useful experience while preserving and pruning skills across tasks, improving agent performance on complex tasks across diverse domains. We evaluate SELM on four agent benchmarks covering diverse tasks and execution environments, and SELM consistently outperforms in most settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill Evolution, Skill Optimization, Multi-Objective Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BSjjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SELM manages agent skills as a library rather than as a single document. Each skill has a task category, a confidence score adjusted by LLM up- and down-votes, and a utility score tracked from task rewards. For each task, SELM retrieves skills from the task's category and a general category, ranked by relevance and utility. It then analyzes trajectories separately for each category, turns the analyses into add, edit, vote, and delete operations, keeps an update only if the selection score improves, and retires skills with low utility or confidence. SELM is evaluated on four benchmarks and three target models, and achieves the best score on most benchmarks.

Critical strengths

S1. The paper addresses a practical problem for multi-skill agents: as skills accumulate, the agent must decide which skills to retrieve, which to update, and which to remove. SELM answers these questions with one concrete design, using per-skill utility and confidence scores, explicit library constraints, and a complete algorithm.

S2. The evaluation is broad. It covers four benchmarks, three target models, and two execution settings, and compares against both single-document and multi-skill baselines. SELM achieves the best score on most benchmarks.

Critical weaknesses

W1. The novelty is limited. Most components come from prior work. The paper states that its fast and slow updates follow prior work, and its related work already cites methods that keep separate skills for different task types, validate candidate skills, and manage a skill library over its lifecycle. On top of these, SELM mainly splits skills by task category and adds per-skill scores for retrieval and removal. Overall, the method is a combination of existing components rather than a new idea.

Questions for authors

Q1. Retrieval and trajectory analysis both rely on a category label for each task. Where do these labels come from for each benchmark, and are they needed at test time?

Q2. The utility score u(s) is the average reward of the tasks that retrieved the skill, with no comparison to the same tasks solved without it. It may therefore reflect task difficulty as much as the skill's usefulness. This matters because utility has a larger weight than relevance in retrieval (wu = 0.6), and skills with utility below 0.05 are retired. How does SELM separate a skill's contribution from the difficulty of the tasks it is retrieved for?

3 comments

Thank you for recognizing our work. We believe SELM's contribution lies in systematic library management for skill, and we will detail its innovation compared to existing work.

W1. Contribution and Novelty

First, SELM introduces a holistic skill library management system. Prior work has made progress in skill acquisition, evolution, and lifecycle management. Methods such as EvoSkill and Memento-Skills primarily focus on acquiring and improving skills, while SkillOps and MUSE-Autoskill explore library health, reliability, and lifecycle management. However, a key challenge remains when skills continuously accumulate and evolve: improvements to individual skills may lead to worse library-wide performance. For example, updates that improve sheet-level tasks may introduce regressions in cell-level tasks, while outdated or ineffective skills may remain retrievable and interfere with future execution. Existing mechanisms for skill evolution do not, by themselves, fully address these interactions between skill updates, retrieval, and retention. Without systematic management of the entire library, these problems will accumulate, making library become increasingly difficult to maintain, and its overall performance may decline despite the growing number of skills.

To address this challenge, SELM introduces a systematic approach to managing the entire skill library. First, SELM analyzes execution trajectories from different task categories separately to identify problems and generate targeted skill updates. This reduces interference between different types of tasks. Second, SELM uses utility and confidence scores to guide skill retrieval and removal, helping retain useful skills and eliminate ineffective ones. Finally, SELM evaluates the updated library on a selection set and accepts changes only when they improve its performance. Together, these mechanisms allow SELM to improve the library as a whole, rather than simply accumulating or optimizing individual skills.

Our experiments support the importance of this management process. Table 3 shows that adding five skills in the first epoch does not immediatly improve accuracy. Table 2 further shows that removing validation, utility scores, or the graveyard reduces performance from 56.5% to 42.5%, 46.3%, and 50.4%, respectively.

Q1. Category sources and test-time requirements

First, SELM does not require ground-truth category annotations at test time, but task categories inferred from task description is needed at test time , for they are important content of our method. When a new task arrives, SELM can leverage an LLM to infer its category from the task description. The category is then used for retrieval. We will describe the category inputs in the appendix.

Second, all methods are evaluated under the same data configurations to ensure fair comparisons. When reproducing existing baselines, we observed that some eimplementations expose task type information to the agent, such as the instruction type in SpreadsheetBench. To maintain consistency with these settings, we provide the same task text and metadata to all baselines, which may include category information in certain benchmarks. We acknowledge that access to complete metadata may influence agent performance. However, since all methods receive identical task information and metadata, we believe the comparisons remain fair and the demonstrated advantages of SELM remain valid.

Q2. Utility estimation and task difficulty

Admittedly, task difficulty can affect skills' utility estimation. However, SELM uses several designs to mitigate its effects on the utility signal and downstream decisions.

First, we prioritize soft rewards for utility updates. Soft rewards retain partial credit when a difficult task falls short of full success, reducing the chance of repeated zero feedback under binary scoring. During retrieval, specialized skills compete within the task category alongside general skills, and ranking combines utility with task relevance.

Second, SELM reduces the influence of individual utility estimation errors by aggregating utility scores during library updates. Instead of relying on a single skill's utility, SELM averages the utility scores of all retrieved skills for each task. This reduces the impact of extreme individual scores on trajectory selection. The selected trajectories are then grouped by task category and jointly analyzed to generate targeted skill updates. Finally, candidate libraries are evaluated on a selection set before the updates are accepted, providing an additional check on overall performance.

Our ablation results support the effectiveness of these mechanisms. On SpreadsheetBench (Table 2), the full SELM achieves 56.5%, compared with 46.3% without utility scores and 49.2% without utility-aware ranking. Removing the validation gate further reduces performance to 42.5%. These results show that utility-based retrieval and library validation both contribute to overall performance. They also suggest that utility provides a useful signal despite its potential estimation bias.

To assess whether utility reflects a skill’s contribution, we conducted a skill-removal experiment across task difficulty levels. The results show that removing the highest utility injected skill reduced performance across difficulty tiers, indicating that high utility skills provide measurable performance benefits across difficulty levels, including challenging tasks, rather than merely being associated with easy tasks. For each task, we compare the full injected skill set with the set that removeing the injected skill with the highest stored utility, holding all other skills and execution settings fixed. The results below shows that removal would lower overall accuracy, dropping 16.6, 8.5, and 12.5 percentage points on easy, medium, and hard tasks, respectively. We will add the table to the appendix later.

DifficultyTasks NumSoft Acc. with full skills (%)Soft Acc. without top-utility skill (%)Δ (pp)
All7769.859.710.1
Easy683.366.716.6
Medium4776.668.18.5
Hard2454.241.712.5

Thank you for the detailed response and additional experiment. My remaining concern is that the contribution is still limited, as the method mainly combines existing update, validation, and lifecycle management components around task categories, so I will maintain my current assessment.

Reviewer_WWM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

he draft introduces SELM, a framework that optimizes an external skill library while keeping the executing LLM frozen. Each skill is associated with a task category, a confidence counter that is updated through voting operations, and a utility estimate derived from an exponential moving average of rewards obtained from tasks that retrieve it. Retrieval combines category matching, relevance, and utility to select the most appropriate skill. Execution trajectories are chosen based on the discrepancy between observed rewards and the stored utility; these trajectories are then analyzed by category and synthesized into library operations through a Map–Reduce process. Candidate libraries undergo selection‑set evaluation, and skills that exhibit low utility or low confidence are retired to a name‑based graveyard. The system supports rapid batch updates for immediate refinement and slower, epoch‑level updates for longer‑term consolidation.

Critical strengths

SELM makes various skill operations explicit such as retrieval, revision, retention, and removal. It connects them through persistent skill statistics. The bounded library, targeted operations, and candidate-library validation provide a practical approach to managing external agent knowledge without retraining model weights. It is important to agent adaptation because it controls the quality of accumulated knowledge, rather than assuming that a larger collection necessarily improves execution.

The experimental results support the potential usefulness of the framework. The paper evaluates four benchmarks with three target models, including configurations with and without Claude Code. SELM generally improves over the reported baselines. Although these results need stronger experimental documentation, their breadth makes the approach more credible than an improvement demonstrated only for one model or one environment.

Critical weaknesses

Equations 7 and 8 assign each retrieved skill the average reward of tasks in which it appeared. With multiple skills retrieved together, this measures an association with success rather than an individual skill’s contribution. A useful skill serving difficult tasks can receive low utility, while an irrelevant skill accompanying easy tasks can receive high utility. Since utility will influences future retrieval and retirement, the authors need to clarify whether the utility estimator can establish the claimed ability to preserve useful skills and retire ineffective ones over time.

The experimental protocol is insufficiently specified. The draft does not report benchmark-specific training, selection, and test sizes or construction procedures. It also omits key execution and optimization settings, including the default optimizer assignment for the main comparisons, training budgets, decoding settings, and so on. These omissions make it difficult to reproduce the comparisons or determine which implemented procedure produced the reported gains.

Questions for authors

What are the exact training, selection, and test splits for each benchmark, and how were they constructed? Which reward (hard, soft, mixed) controls selection on each benchmark? Are the reported results from independent runs, and what variability was observed?

In Table 1, which optimizer model was used for each method? And were optimization calls, execution budgets, and selection-set access comparable?

3 comments

Thank you for valuable advice on our paper. We will clarify distinction between historical utility and individual skill contribution, and for identify the details needed to reproduce Table 1.

Weakness 1. Utility and reward attribution

Admittedly, task difficulty can affect skills' utility estimation. However, SELM uses several designs to mitigate its effects on the utility signal and downstream decisions.

First, we prioritize soft rewards for utility updates. Soft rewards retain partial credit when a difficult task falls short of full success, reducing the chance of repeated zero feedback under binary scoring. During retrieval, specialized skills compete within the task category alongside general skills, and ranking combines utility with task relevance.

Second, SELM reduces the influence of individual utility estimation errors by aggregating utility scores during library updates. Instead of relying on a single skill's utility, SELM averages the utility scores of all retrieved skills for each task. This reduces the impact of extreme individual scores on trajectory selection. The selected trajectories are then grouped by task category and jointly analyzed to generate targeted skill updates. Finally, candidate libraries are evaluated on a selection set before the updates are accepted, providing an additional check on overall performance.

Our ablation results support the effectiveness of these mechanisms. On SpreadsheetBench (Table 2), the full SELM achieves 56.5%, compared with 46.3% without utility scores and 49.2% without utility-aware ranking. Removing the validation gate further reduces performance to 42.5%. These results show that utility-based retrieval and library validation both contribute to overall performance. They also suggest that utility provides a useful signal despite its potential estimation bias.

To assess whether utility reflects a skill’s contribution, we conducted a ablation study. The results show that removing the highest-utility injected skill reduced performance across difficulty tiers, indicating that the selected skills contributed to task performance. For each task, we compare the full injected skill set with the set that removeing the injected skill with the highest stored utility, holding all other skills and execution settings fixed. The results below shows that removal would lower overall accuracy, dropping 16.6, 8.5, and 12.5 percentage points on easy, medium, and hard tasks, respectively. It suggests that the highest utility injected skill contributes in each difficulty tier. We will add the table to the appendix later.

DifficultyTasks NumSoft Acc. with full skills (%)Soft Acc. without top-utility skill (%)Δ (pp)
All7769.859.710.1
Easy683.366.716.6
Medium4776.668.18.5
Hard2454.241.712.5

Weakness 2 / Question 1. Data splits, selection metrics, and variability

Data splits. We preserve each benchmark's original train–test split. Using random seed 42, we divide the original training split into three disjoint subsets in a 2:1:7 ratio: 20% for skill evolution, 10% for validation and library selection, and 70% for internal evaluation. The original test split is reserved for final evaluation after library selection. Selection rewards. For SkillsBench, we conduct separate experiments using hard and soft rewards for library selection. For SpreadsheetBench, OfficeQA, and LiveMath, we use mixed rewards for library selection.

Evaluation metrics and variability. We use hard accuracy as the primary evaluation metric across all four benchmarks. Given the difficulty of SkillsBench, we additionally report soft accuracy to measure partial task completion. Each reported score is averaged over three runs.

Skill management. SELM updates each skill's utility using an exponential moving average of task rewards, prioritizing soft rewards when available. It also updates confidence through upvotes and downvotes. As shown in Algorithm 1, skills are retired when their utility falls below 0.05 or their confidence is non-positive. We will add these details to the revised paper.

Question 2. Optimizer assignments and comparison budget

Optimizer assignments. All methods requiring optimization use the evaluated model itself as the optimizer within each model group in Table 1. Specifically, Claude-Opus-4.6, DeepSeek-V4-Pro, and Qwen3-30B-A3B each serve as both target and optimizer in their respective groups. Qwen3-30B-A3B is deployed locally via vLLM, while the other two models are accessed through APIs. Static baselines use no skill, a fixed expert-written skill, or a fixed one-shot skill generated by Claude-Opus-4.6.

Shared experimental settings. Within each benchmark, model, and harness setting, we keep the shared experimental configurations consistent across methods. These include datasets, scoring metrics, decoding settings, execution limits, timeouts, and planned evolution rounds where applicable. The default configuration uses a batch size of 40, a maximum of 16,384 completion tokens, 24 tool turns, and a 300-second timeout.

Computational budgets. All methods use the same selection set within each benchmark and model setting. We also keep the planned evolution rounds consistent where applicable. The actual number of optimizer calls may differ because methods perform different operations within each round.

Thank the authors for your responses and the additional experimental results, which partially addressed my main concerns. However, I would keep my original rating.

Reviewer_rUqKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SELM (Self-Evolving Library Management), a framework that formulates LLM agent skill evolution as a dynamic library management problem to prevent single-round edits from overwriting useful procedures or retaining failing rules across tasks. Its core contributions are three integrated mechanisms: lifecycle management that tracks skill confidence and utility to retain effective skills and prune failing ones, a category-wise Map-Reduce framework that generates targeted skill updates without cross-task interference, and utility-aware retrieval that ranks skills by both semantic relevance and historical utility. Evaluated across four diverse agent benchmarks along with component ablation studies, SELM achieves state-of-the-art performance over single-document optimization and multi-skill evolution baselines without updating model parameters.

Critical strengths
  • Principled Long-Term Skill Management: Unlike prior methods that apply myopic, single-round edits, SELM tracks skill confidence and utility alongside category-wise Map-Reduce updates. This directly resolves cross-task interference and prevents useful rules from being overwritten during continual agent learning.

  • Strong Cross-Domain Generalization: SELM consistently outperforms single-document and multi-skill evolution baselines across four diverse agent benchmarks without parameter fine-tuning, proving that utility-aware retrieval and lifecycle pruning yield reliable, domain-agnostic gains.

Critical weaknesses
  • The paper leaves task category assignment unspecified, which is consequential because using ground-truth labels gives SELM an unfair advantage and limits applicability to uncategorized tasks.

  • Coarse reward attribution across co-retrieved skills and unvalidated pruning are consequential because valid skills can be prematurely and permanently retired after noisy early failures.

Questions for authors
  • How are task categories γi\gamma_i assigned during training and testing (ground-truth labels vs. LLM routing, and do baselines have equal access), and (Future Work) how sensitive is the hard filter in Eq. (6) to misclassification or multi-category tasks?

  • How are v(s)v(s) and u(s)u(s) initialized, and how does lifecycle pruning (Alg. 1, Line 48) prevent valid skills from being prematurely retired due to noisy early trajectories or shared reward updates across co-retrieved skills (Eq. 7)?

1 comment

Thank you for recognizing our work! In the following sections, we provide further clarification on task categorization in SELM and our experimental configurations.

Question1. Task Categorization and routing robustness

Category sources and access to labels.

First, SELM doesn't rely on predefined task categories. When a new task arrives, SELM can leverage an LLM to infer its category from the task description. The category is then used for retrieval. We will describe the category inputs in the appendix.

Second, all methods are evaluated under the same data configurations to ensure fair comparisons. When reproducing existing baselines, we observed that some eimplementations expose task type information to the agent, such as the instruction type in SpreadsheetBench. To maintain consistency with these settings, we provide the same task text and metadata to all baselines, which may include category information in certain benchmarks. We acknowledge that access to complete metadata may influence agent performance. However, since all methods receive identical task information and metadata, we believe the comparisons remain fair and the demonstrated advantages of SELM remain valid.

Sensitivity of the hard category filter. To assess the sensitivity to hard category filtering, we ran a three-way comparison on SpreadsheetBench with Qwen3-30B-A3B, holding the skill library, test tasks, ranking rule, and execution budget fixed. The three settings were hard filtering with categories mapped from the supplied instruction type, hard filtering with categories inferred by Qwen3-30B-A3B from the task descriptions, and retrieval with category filtering disabled. The scores were 36.43%, 36.07%, and 32.86%, respectively. This result shows that categories inferred from task descriptions retained most of the benefit of hard filtering. We will add the result into appendix and extend this comparison to additional benchmarks in follow-up work.

Question 2. Initialization and premature retirement

Skill initialization. Newly added skills start with utility u0=0.5u_0=0.5 and confidence v0=2v_0=2. The initial utility is the midpoint of the reward range of [0, 1], while the initial confidence allows a skill to survive one downvote. Skills loaded from a saved library may retain their previous values.

Protection against premature retirement. SELM does not impose a minimum-use requirement before retirement. However, EMA smoothing and initial confidence reduce the impact of a single negative observation. With the default EMA setting, one zero-reward update does not reduce utility below the retirement threshold of 0.05, and one downvote only reduces confidence from 2 to 1, avoiding the skill from being retired too early.

Update and validation procedure. CRUD operations(add, edit, upvote, downvote) and utility updates are applied before pruning. The resulting library is then evaluated on the selection set, and a new best snapshot is saved if its score improves. However, this selection process does not roll back unsuccessful updates or guarantee that individual skills are correctly retired. Retirement rules and further analysis. The implemented retirement condition is u(s)<0.05u(s)<0.05 or v(s)≤0v(s)\leq 0. The graveyard holds the 50 most recent retirements and blocks a name while its entry remains in that record.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.