acceptodds
Under review as a conference paper at ICLR 2027

SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems

Abstract

Recent self-evolving agents have shown that skills can be discovered, refined, and accumulated through execution. However, existing skill evolution frameworks typically assume a fixed tool layer and evaluate each skill independently, limiting their ability to repair tool-level failures or reason about interactions among skills. We propose SkillSmith, a synergy-aware skill–tool co-evolution framework. SkillSmith introduces a unified proposal space where reflection produces atomic bundles that jointly modify skills and tools—allowing tools to be wrapped, edited, composed, split, or retired when skill evolution identifies a reusable capability gap. To guide this joint search, SkillSmith maintains an ecological utility model inspired by Lotka–Volterra dynamics, where an interaction matrix estimated from execution traces captures pairwise complementarity and conflict among skills and provides pressure signals for retrieval, mutation prioritization, and retirement. Furthermore, SkillSmith records anti-patterns—failure signatures, attributions, and remedies—to accelerate diagnosis and veto proposals that repeat known mistakes. Experiments on three benchmarks (e.g., WildClawBench) and five Qwen3.5 scales show that SkillSmith consistently outperforms strong baselines, with gains that amplify as task complexity and multi-skill co-activation increase.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm agent, self-evolve

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8CMFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SkillSmith, a skill–tool co-evolution framework for self-improving LLM agents. The framework consists of three main components: (1) a skill ecosystem that allocates limited context resources based on skill utility and interactions, (2) tool evolution that extracts reusable deterministic logic from skills into tools, and (3) conflict-driven failure memory that prevents repeated unsuccessful modifications. Experimental results demonstrate that SkillSmith consistently improves agent performance throughout the evolution process, with larger gains observed as the backbone model scales up.

Critical strengths
  1. Well-motivated problem formulation and promising empirical results. SkillSmith addresses a meaningful problem in the self-evolution of LLM agents. The reported performance improvements are substantial, and the paper provides extensive analyses to support the effectiveness of the proposed framework.

  2. Rigorous and well-structured formalization. The paper presents a comprehensive mathematical formulation of skill–tool co-evolution, with theoretical analyses supporting several key design choices.

Critical weaknesses
  1. Limited scale of the skill and tool libraries. The skill and tool libraries considered in the experiments are relatively small compared with those that may arise in real-world agent systems. This raises concerns about whether the proposed setting adequately captures the context-budget constraints that motivate the framework. In particular, it is unclear to what extent the context budget becomes a binding constraint in the current experiments, and whether the proposed approach would remain effective with substantially larger skill and tool libraries.

  2. Potentially unreliable skill utility estimation due to limited training data. The training sets for OfficeQA and SealQA are very small, both in absolute size and relative to the full datasets (24/246 for OfficeQA and 11/111 for SealQA). This may substantially affect the reliability of skill utility estimation based on randomized skill masking, particularly when certain skills are more relevant to the limited training samples than to the overall task distribution. Consequently, the estimated individual and pairwise skill contributions may be noisy or biased, potentially affecting subsequent context allocation decisions.

Questions for authors
  1. The experimental comparisons are somewhat limited, and different baselines are used across benchmarks. Specifically, Table 1 compares SkillSmith against EvoSkill, whereas Figure 3 compares it against SkillClaw. It would be helpful if the authors could provide more comprehensive comparisons across benchmarks or clarify the rationale behind this experimental design.

  2. An interesting observation is that SkillSmith achieves larger performance gains as the backbone model scales up. To further investigate the generalizability of this trend, it's benefit for the authors to evaluate SkillSmith with SOTA proprietary LLMs.

  3. Disentangling the effects of Executor and Reflector scaling. In the current experimental setup, the Executor and Reflector use the same backbone model, making it difficult to determine whether the observed scaling benefits primarily arise from improved task execution or more effective evolutionary reflection. Additional experiments with heterogeneous model configurations, such as a small-model Executor paired with a large-model Reflector and vice versa, would help disentangle these effects.

  4. Figures 1 and 2 suffer from several visual rendering and alignment issues that significantly affect their readability. I recommend carefully revising these figures to improve their visual clarity and presentation quality.

4 comments

Thank you for your careful review and valuable comments. We have added the corresponding experiments, and changes in the revised manuscript are marked in blue. We respond to each point below.

Q1. Rationale for the baseline comparisons

We thank the reviewer for this comment. Baselines were chosen according to the setting each method is designed for.

OfficeQA and SealQA are static question-answering tasks with few interactions between skills. This is exactly the setting EvoSkill is designed for, so it is the strongest and fairest baseline on these two benchmarks. SkillClaw is designed for settings with multiple users, continuously arriving data, and substantial interactions within the skill library; on these two benchmarks its collective-evolution mechanism has nothing to work with, so the comparison would be of limited value.

WildClawBench is the opposite: it involves multi-day continual evolution with frequent skill co-activation. EvoSkill has no ecosystem-level design and performs only modestly there, so SkillClaw is the matching strong baseline. In response to the reviewer, we added a full WCB comparison on Qwen3.5-35B (N = 5, Day 6, averaged over the four task categories):

MethodWCB accuracy (%)Gain over no-evolution baseline (pp)
No-evolution baseline19.73 ± 1.47—
EvoSkill20.73 ± 1.47+1.00 ± 2.08
SkillClaw27.70 ± 1.62+7.97 ± 2.19
SkillSmith34.75 ± 1.49+15.02 ± 2.09

On WCB, EvoSkill improves over the no-evolution baseline by only 1.0 percentage point, within the error range. In addition, SkillSmith and SkillClaw are concurrent work, and when this work was carried out, few comparable methods existed in this direction. On each benchmark we chose the strongest method that best matched it at the time.

Q2. Performance with stronger backbones (new Appendix E.3, p.24, Figure 12, Table 6)

To examine whether the trend "the stronger the backbone, the larger SkillSmith's gain" holds for top closed-source models, we built Chronos Bazaar, a 30-round dynamic adversarial sandbox with extreme context pressure, skill conflicts, and non-stationary rule changes. We evaluate the state-of-the-art closed-source models Claude Fable 5.2 and GPT-6-Astra side by side with the open-source model family.

Task setup and hybrid evaluation protocol

The agent acts as an intergalactic anti-money-laundering investigator and high-frequency arbitrage officer. Under a strict context window (C=2048C=2048 tokens) and a budget of 60 validated modifications (B≤60B \le 60), it goes through three phases of evolution:

See Appendix E.3 for the task setup and phase-specific rules.

Results across backbones (5 independent runs, mean ±\pm SD)

BackboneFinal (D30, %)Recovery (D25, %)Payload (%)Bundle pass (%)Tool errors (%)Conflicts left (pairs)Validation budget used (b≤60b \le 60)
Qwen3.5-9B6.5±1.86.5 \pm 1.86.3±1.56.3 \pm 1.531.2±4.131.2 \pm 4.118.4±4.218.4 \pm 4.228.6±3.928.6 \pm 3.90.64±0.080.64 \pm 0.0860.0±0.060.0 \pm 0.0 (exhausted)
Qwen3.5-35B19.5±2.419.5 \pm 2.423.5±2.623.5 \pm 2.648.5±3.548.5 \pm 3.542.5±3.642.5 \pm 3.614.2±2.414.2 \pm 2.40.31±0.050.31 \pm 0.0557.4±2.657.4 \pm 2.6
Qwen3.5-122B64.5±1.564.5 \pm 1.546.0±2.146.0 \pm 2.174.0±2.274.0 \pm 2.271.8±2.571.8 \pm 2.55.4±1.15.4 \pm 1.10.12±0.030.12 \pm 0.0346.2±3.146.2 \pm 3.1
GPT-6-Astra92.1±0.792.1 \pm 0.786.8±1.186.8 \pm 1.191.6±0.891.6 \pm 0.892.4±1.092.4 \pm 1.00.9±0.30.9 \pm 0.30.04±0.010.04 \pm 0.0133.2±2.033.2 \pm 2.0
Claude Fable 5.294.3±0.694.3 \pm 0.691.2±0.991.2 \pm 0.989.8±1.189.8 \pm 1.190.5±1.290.5 \pm 1.21.3±0.41.3 \pm 0.40.02±0.010.02 \pm 0.0131.4±1.831.4 \pm 1.8

Analysis: why stronger backbones unlock larger gains

Cross-layer restructuring has a capability threshold. Weaker models (9B/35B) are limited in synchronizing code across multiple components and in abstraction. Their first-pass rate for atomic bundles is very low (18.4%18.4\%–42.5%42.5\%), so they struggle to move deterministic computation into tools, and the context window remains crowded with redundant instructions (payload <50%<50\%). The top models exceed a 90%90\% pass rate, quickly slim down their skills, and free the scarce context budget CC for higher-level strategy.

  • Moving logic back from tools to skills when the environment changes. After the rule drift on D21, the 35B model keeps piling rigid regular-expression patches (PATCH) onto the invalidated tool. These happen to pass the small validation set DvalD_{\text{val}} (reaching 27.0%27.0\% on D24), but trigger snowballing false-freeze penalties on the test stream, and accuracy falls steadily to 19.5%19.5\%. In contrast, Claude Fable 5.2 and GPT-6-Astra recognize that the task is no longer context-independent: on D22 they decisively execute INLINE to retire the old tool and return pun interpretation to the skill layer, and on D23 they extract the pure weather lookup into a new tool, rapidly recovering to 86.8%86.8\%–91.2%91.2\%.

Where the advantages of Claude Fable 5.2 and GPT-6-Astra come from. GPT-6-Astra excels in engineering quality and sample efficiency: the highest first-pass rate for code, the best tool compression, and the lowest budget use. Claude Fable 5.2 excels in adapting to non-stationary environments and in deeper strategic reasoning: it resolves multi-skill conflicts faster and restructures more decisively after the rule change.

  • Strong backbones compound the value of failure memory (Nogood). From the minimal failing subset isolated by ddmin, the top models infer the underlying mechanism in one step and generate precise remedy constraints. They converge early on D27, stop making ineffective modifications, and achieve a clear lead using only about half of the budget.

Further direct evidence comes from WikiSkill [1]: on Qwen 4B/9B/27B, the gains are +12.3/+17.5/+23.9, and its abstract states that larger models benefit more from evolved skills. Indirect support comes from two works showing that self-improvement has a capability threshold. In SkillsBench [2], self-written skills yield zero or negative gains on most backbones, and only the latest Opus 4.6 is positive (+1.4, possibly within noise). Sample More, Reflect Less [3] shows that for small 1.5B–7B models, self-reflection loses to repeated sampling in every case at equal token cost.

[1] Tang et al., WikiSkill, arXiv:2608.27454, 2026 [2] SkillsBench, arXiv:2602.12670, 2026 [3] Mirzaei, Sample More, Reflect Less, arXiv:2607.28576, 2026

Q3. Scale of the executor and the reflector (updated in §4.3, p.10, Figure 10)

We thank the reviewer for this suggestion. In the original experiments, the executor and the reflector used the same frozen backbone, so the effects of their respective scales could not be separated. We have therefore added a cross-scale experiment over executor M_E × reflector M_R (N = 5; parentheses give the gain over the no-evolution baseline in percentage points).

OfficeQA

M_E \ M_RNo evolution9B27B35B122B
9B14.017.7 (+3.7)20.8 (+6.8)22.4 (+8.4)25.6 (+11.6)
27B26.128.5 (+2.4)34.2 (+8.1)36.8 (+10.7)41.5 (+15.4)
35B34.837.6 (+2.8)43.1 (+8.3)46.2 (+11.4)52.8 (+18.0)
122B52.655.4 (+2.8)63.5 (+10.9)67.2 (+14.6)72.3 (+19.7)

SealQA

M_E \ M_RNo evolution9B27B35B122B
9B5.77.8 (+2.1)9.6 (+3.9)10.5 (+4.8)12.8 (+7.1)
27B10.211.8 (+1.6)15.1 (+4.9)16.8 (+6.6)20.4 (+10.2)
35B13.315.0 (+1.7)18.6 (+5.3)20.9 (+7.6)25.4 (+12.1)
122B20.422.8 (+2.4)27.5 (+7.1)30.1 (+9.7)33.7 (+13.3)

Conclusion: the reflector sets the ceiling on improvement, and the executor's scale amplifies it.

① The reflector has a capability threshold. With M_R = 9B, the gain is only +1.6 to +3.7 regardless of executor size. Only when the reflector reaches 27B or larger do the gains widen substantially (e.g., from +2.8 to +10.9 at M_E = 122B). This is consistent with prior findings on self-reflection methods: self-diagnosis and strategy revision require causal attribution across trajectories and depend on a strong base model (Reflexion, Shinn et al., 2023; Self-Refine, Madaan et al., 2023). Reflectors below 30B often manage only superficial fixes.

② The tool layer transfers a strong reflector's capability to a weak executor. With M_E = 35B and M_R = 122B, OfficeQA reaches 52.8, on par with the 122B no-evolution baseline (52.6), and SealQA reaches 25.4, exceeding the 122B baseline (20.4). The reason is that EXTRACT and PATCH turn deterministic logic (e.g., merged-cell parsing, unit conversion) into tool code, so execution no longer depends on the LLM's stochastic interpretation. A small executor only needs to pass arguments according to the tool signature to directly reuse the larger model's abstractions. This is consistent with the finding that "large models make tools, small models use them" (LATM, Cai et al., 2024; Voyager, Wang et al., 2023; ADAS, Hu et al., 2024).

③ Context offloading benefits small executors more. Small models are more sensitive to long contexts and conflicting instructions (β_ij < 0) (Lost in the Middle, Liu et al., 2024). The 122B reflector allocates 74% of the context to skills with positive utility and precisely removes conflicting skills, leaving only 0.12 conflicting pairs per task, compared with 0.64 for the 9B reflector. This directly reduces the cognitive load on small executors.

Q4. Figure readability (updated in Figure 1, p.2, and Figure 3, p.4)

We thank the reviewer for the detailed comments. We have redrawn Figure 1 and the original Figure 2: we fixed rendering artifacts and element alignment, unified fonts, colors, and labels, and increased the resolution. Because the revision adds a figure quantifying the motivation (now Figure 2), the original Figure 2 is now Figure 3. The new Figure 1 is on page 2 and Figure 3 on page 4, with all changes marked in blue.

Thank you again for the time you have devoted to reviewing our paper. Your comments have made our baseline comparisons, scale analysis, and figure presentation more complete. If you have any further questions, we would be happy to discuss them. 🌸

Thank you for the authors' detailed responses and additional experiments, including the clarification of the baseline selection and the analysis of performance differences across executor–reflector configurations of different model scales. However, I still have two concerns regarding the newly introduced Chronos Bazaar experiment:

  1. Why did the authors introduce a new evaluation environment instead of conducting the experiments on the existing benchmarks, namely OfficeQA, SealQA, and WildClawBench?
  2. My original Q2 aimed to examine whether the observed trend that SkillSmith achieves larger performance gains as the backbone model scales up also generalizes to SOTA proprietary LLMs. However, although the authors constructed a new and relatively elaborate evaluation environment, they did not report the performance gains of SkillSmith over a no-evolution baseline or other evolution baselines under the same backbone model. This mismatch between the experimental design and the original evaluation objective is somewhat puzzling. I believe the authors should provide a clearer justification for the purpose and design of this additional experiment.

Given these concerns, I have decided to maintain my original rating.

Reviewer_NuAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillSmith, a framework that jointly evolves agent skills and tools under limited context and iteration budgets. It combines interaction-aware skill allocation, typed operations for repairing tools and transferring logic between skills and tools, and version-scoped failure memory to avoid repeating unsuccessful edits. Experiments on OfficeQA, SealQA, and WildClawBench show improvements over skill-only evolution baselines, with ablations supporting the contributions of the three components.

Critical strengths
  1. The paper offers a reasonable insight. It treats the skill-tool boundary as an optimization variable rather than fixing the tool layer and editing skills only. This is a useful angle for studying where logic should live in an agent's external capability library.
  2. The algorithms proposed in the paper make sense. The identification result for randomized masking is sound under its assumptions. Theorem A.5 shows that centering makes the pair features orthogonal to arbitrary task difficulty, so pairwise interaction effects can be identified without modeling the task. PATCH, RETYPE, EXTRACT, and INLINE cover the basic ways logic can move across the boundary, and committing interface changes as atomic bundles with caller updates is the right engineering response to exact-match dependencies.
Critical weaknesses
  1. The three mechanisms are loosely coupled rather than jointly optimized. Context allocation is a continuous resource-allocation problem, tool evolution is a discrete program-refactoring problem, and nogood learning is a search-pruning problem. Their objectives, search spaces, and timescales differ. As a result, "co-evolution" reads as a system narrative rather than a formalized algorithm.
  2. The experimental design does not establish that the gains come from the co-evolution design. The main comparison is the full system against the skill-only EvoSkill baseline. This shows that adding a tool layer helps, but not that the proposed design is better than a strong system that combines existing components. The prevalence of the three motivating problems is never measured, so the reader cannot tell whether duplicated logic, skill interactions, and repeated failed edits are common or rare in the experiments. The ablation is also incomplete: it omits OfficeQA, runs at a single model size, and reports final library counts that differ by only a few components. the paper cannot separate "better design" from "more components. And the reported results are not yet reliable or complete enough to support the claims. WildClawBench appears only as a trajectory figure, without a table, final numbers, or variance; the ablation has no error bars and is run at one scale; the standard deviations in Table 1 are identical across methods at each model size, which is unexpected for sample SDs over five independent runs, and no significance testing is reported. Code, prompts, and a complete hyperparameter table are missing.
Questions for authors
  1. If you combine EvoSkill with a strong tool-repair method and a strong memory baseline under a simple orchestrator, and compare under the same budget, does SkillSmith still win? If yes, the design is responsible for the gain; if not, the contribution is better described as system integration?
  2. Can you quantify the prevalence of the three motivating problems in your logs? Count how many skills contain duplicated logic, how many tokens the duplication costs, how often skills co-activate with positive or negative interactions, how often rejected proposals recur, and how much budget vetoes save. Without these numbers, the motivation is plausible but not established?
5 comments

Thank you for your careful review and valuable comments. Changes in the revised manuscript are marked in blue. We respond to each point below.

Q1. Comparison with a strong composite baseline (added in E.1, pp.23–24, Table 5)

We thank the reviewer for raising this key point. EvoSkill, ToolMaker, and ReasoningBank correspond to SkillSmith's skill evolution, tool repair, and experience reuse, respectively. Combining the three under a simple orchestrator gives a direct test of whether SkillSmith still has an advantage when the baseline has the same three capabilities, and if so, where that advantage comes from.

Composite baseline design

ModuleImplementation in the composite baseline on OfficeQA
Skill evolutionEvoSkill revises skills from failed trajectories and keeps candidates via Pareto selection.
Tool repairToolMaker (ACL'25) iteratively diagnoses and revises tool code from execution feedback, e.g., fixing column misalignment caused by merged headers.
Experience memoryReasoningBank (ICLR'26) distills strategies from successful and failed trajectories for retrieval on later similar tasks.
OrchestratorEach round runs one module and validates its update independently; later rounds can build on accepted modifications.

Results (OfficeQA, 122B; E.1, p.23)

MethodCalls/iterItersTotal callsTotal tokens (M)Acc
EvoSkill42.618766.818.463.8%
EvoSkill + ToolMaker + ReasoningBank48.316772.820.368.7%
SkillSmith51.313666.916.072.3%

Analysis

1. Combining modules is not the same as coordinating them. Without cross-layer coordination and ecosystem control of the context, the composite baseline is essentially three independent reflective optimizers (in the style of GEPA) competing for the same iteration budget BB and context budget CC. The difference shows up in three places:

  • Context. ReasoningBank's memory entries compete with skills for CC, and some of them merely restate what the skills already contain; EvoSkill's fixes can only accumulate as text inside skills. SkillSmith uses EXTRACT to move shared deterministic logic out of the window (Eq. 9), and the skill ecosystem then reallocates the freed space by marginal utility.
  • Search. SkillSmith samples proposals in the joint space of skill edits and tool operations, whereas ToolMaker + EvoSkill run in sequence: the orchestrator lets only one module act per round, with EvoSkill revising skills while tools are fixed and ToolMaker repairing tools while skills are fixed. This amounts to coordinate ascent alternating between two subspaces.
  • Iterations. ReasoningBank's lessons are only soft prompts, so the LLM may still generate substantively identical modifications, each of which is rejected only after a full run on the 17 validation samples. SkillSmith's version-scoped nogoods veto such proposals before CHECK. This explains why SkillSmith converges in 13 iterations, while the composite baseline remains lower after 16.

The composite baseline's per-iteration cost (48.3) lies between the two: ToolMaker's sandbox retries and ReasoningBank's per-round memory distillation add calls, but it has no delta-debugging replay. SkillSmith costs more per iteration, but a larger share of its iterations turns into effective improvements, so its total number of calls is the lowest.

2. Limits when the budgets are relaxed.

  • B→∞B\to\infty: a simple orchestrator can rely on extensive trial and error until it happens to fix the tool and the skill correctly in the same round, approaching the correct coordination; the advantage in iterations then disappears.
  • C→∞C\to\infty: once the window has slack (y0∗>0y_0^*>0), the marginal token value λ=fˉ(y∗)=0\lambda=\bar f(y^*)=0, the capacity gain from lower costs in Proposition 3.2, U∗(c′)>U∗(c)U^*(c')>U^*(c), degenerates to equality, and the advantage in context disappears.
  • Even with both budgets unlimited, logic that stays in skills still has to be simulated by the LLM in natural language every time and is subject to random reasoning errors, whereas logic extracted into tools is executed deterministically by the machine (§2.1).

3. When the three are equivalent. Suppose (1) the tools are fixed and free of defects (ΔT=∅\Delta\mathcal T=\emptyset); (2) each task activates only one skill, or context is extremely abundant (so βij\beta_{ij} plays no role and there is no competition for capacity); and (3) the modification space is small enough that typed tool operations, ecosystem allocation, and nogoods are never triggered. Then SkillSmith, EvoSkill, and the composite baseline all reduce to reflective prompt evolution over a Pareto front, and their differences should fall within noise.

Conclusion. SkillSmith's advantage does not come from having an extra module, but from letting skills, tools, and failure experience co-evolve under the same budgets.

Q2. Source of the motivation (added on p.2)

We thank the reviewer for this suggestion and fully agree that the motivation needs empirical support rather than argument alone. We have therefore added Fig. 2 (p.2), which quantifies logs of skill-only evolution on a mixed τ²-bench stream, with one set of measurements for each of the three motivations.

MotivationMetricValue
① Logic lives in the wrong layerSkills containing duplicated logic34 / 36
Share of duplicated logic implementable as tools89%
Share of library tokens taken by redundant copies19%
② Value is collectiveEpisodes co-activating ≥2 skills90%
Episodes loading contradictory skills32%
Episodes loading duplicated skills36%
③ Failures are discardedProposals that revisit an already rejected edit38%
Of which rejected again82%
Evolution budget spent on edits that are revisited and rejected again22%

Analysis:

① Duplication is systematic, because the boundary is fixed. Nearly all skills (34/36) restate logic from other skills. 90% of the duplication occurs within the same domain and only 5% across domains, which indicates that this is not incidental textual similarity: a domain's shared procedures have been copied into each skill separately. More importantly, 89% of the duplicated logic is deterministic (e.g., identity verification, device repair) and belongs in the tool layer.

② Skills cannot be evaluated in isolation. 90% of episodes load several skills at once, so single-skill evaluation does not cover actual execution. In 32% of episodes, the loaded skills give contradictory instructions; for example, the retail skill asks for the email first, while the banking skill asks for the full name first. Such conflicts surface only when the skills appear together, which requires explicitly estimating βᵢⱼ and accounting for it in allocation. This is the direct motivation for randomized masking and the second-order model.

③ Repeated failures keep consuming the budget. 38% of proposals revisit a modification that was already rejected, and 82% of these fail again, showing that without failure memory, reflection repeatedly converges to the same wrong modification. By iteration 15, 22% of the cumulative budget is spent on such repeats, and this share grows with iterations. These 22% are an upper bound on the budget the nogood veto is designed to recover. The ablation in Table 2 is consistent with this: without nogoods, WCB accuracy drops from 41.4 to 38.2 and the regression rate rises from 2.1% to 7.6%, showing that failure memory both saves budget and makes evolution more stable.

Other

We address the remaining points in the weaknesses one by one below.

Significance tests (new Appendix E.2, p.24). For the 9 benchmark × scale combinations in Table 1, we ran two-sided Welch t-tests (5 runs per setting) with Holm correction. SkillSmith significantly outperforms both EvoSkill and the Base Agent in 8 of the 9 settings (adjusted p < 0.01). The only exception is SealQA at 9B: +1.1 over EvoSkill (p = 0.22) and +2.1 over the Base Agent (p = 0.085), which are not significant with 5 runs.

Stability (updated in Figure 5, Table 1, and Table 2, p.8). We ran 5 independent runs on WCB to measure the standard deviation and added "mean ± 1 SD" shaded bands to the trajectory plots. We reran the experiments in Table 1 and can provide the run logs. Table 2 now reports standard deviations for all variants.

Reproducibility (Appendix H, pp.30–35). The complete code is included in the supplementary material submitted with the paper. Prompt templates for all LLM components have been added to Appendix H.

Thank you again for the time you have devoted to reviewing our paper. Your comments have made our comparison with the composite baseline and the empirical support for our motivation more complete. If our responses have addressed your concerns, we would be grateful if you would consider adjusting your score. If you have any further questions, we would be happy to discuss them. 🌸

Thank you for the revision. The revised statistics and prompts address my earlier reporting concerns, and the composite comparison is useful.

I am still unsure whether the EvoSkill baseline retains its helper scripts and proposal history, including rejected edits. These are already part of EvoSkill, so the description of text-only repairs and discarded failures needs clarification. Table 5 also gives no uncertainty for the 68.7% versus 72.3% comparison, and the common stopping rule remains unclear. Without these details, I find the claim that coordination accounts for 42% of the gain premature.

The interaction experiments leave a similar question. Figure 2 uses randomized skill injection, with skills from another benchmark in 80% of co-active episodes. Some of these conflicts may disappear with ordinary relevance filtering. Do Figures 6 and 14 use the same setup? Absolute task scores for Full, −Ecosys, and −Pair under a documented retrieval protocol would help distinguish the benefit of pairwise modeling from filtering irrelevant skills. Normalized losses do not show the size of that benefit.

These questions still affect the evidence for the proposed coordination mechanism. The revision improves the paper, but I am keeping my score at 2.

Reviewer_PJxHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SkillSmith, a framework that jointly evolves an agent's skill library and tool library. It introduces an interaction-aware mechanism for allocating limited context to useful skills, a set of structured operations for updating tools and their dependent skills, and a failure-memory mechanism that prevents the system from repeatedly proposing unsuccessful changes. Experiments on OfficeQA, SealQA, and WildClawBench show improvements over skill-only evolution, with additional ablations and cost comparisons.

Critical strengths

S1. A meaningful extension beyond skill-only evolution.

I like that the paper treats tools as something the agent can improve, rather than assuming they remain fixed. The idea of updating tool interfaces together with their dependent skills is practical and addresses a real limitation of skill-only approaches. The structured tool operations also make the framework easier to understand and reason about.

S2. Encouraging results across models and benchmarks.

The main results are fairly convincing. SkillSmith consistently improves over EvoSkill on OfficeQA and SealQA, with gains reaching 10.2 and 9.8 percentage points at the largest model scale. The ablation results and the lower reported evolution cost provide further support for the framework.

Critical weaknesses

W1. The contribution of the interaction-aware skill mechanism is not sufficiently isolated.

The −Ecosys ablation removes context allocation, interaction-aware retrieval, and skill retirement together. As a result, it is difficult to tell whether the gains come from modeling pairwise skill interactions or simply from better retrieval and library management. Since interaction-aware allocation is a central contribution, I would like to see more direct evidence that the interaction terms and randomized masking are necessary.

W2. The evidence for robustness is still somewhat limited.

Table 1 reports variation across five runs, but the ablation results and WildClawBench evolution curves do not provide comparable uncertainty estimates. This matters because the latter experiments support several of the paper's main claims about reliability and long-term improvement. The relatively small training and validation splits also make it important to establish that the improvements are stable across different splits and evolution runs.

Questions for authors

Q1. Could the authors provide a more targeted ablation of the skill ecosystem, particularly removing the pairwise interaction terms while keeping the remaining allocation mechanism unchanged? This would help clarify how much of the improvement comes specifically from interaction-aware modeling.

Q2. How stable are the WildClawBench results and the component ablations across independent evolution runs? Reporting the mean and standard deviation, along with the number of runs, would help assess whether the observed differences are reliable.

2 comments

Thank you for your careful review and constructive comments. Changes in the revised manuscript are marked in blue. We respond to each point below.

Q1. More targeted ablations (updated in §4.2, Figure 6, p.9; Appendix G, pp.26–29)

Thank you for this suggestion. We agree that −Ecosys removes too much at once. Below we first state exactly what this ablation removes, and then present the results together with the theoretical analysis.

What is ablated. The skill ecosystem models utility as (Eq. 2)

Ũ(p) = Σ_i u_i p_i + ½ Σ_{i≠j} β_ij p_i p_j

The pairwise interaction term β_ij is used in exactly three places:

  1. Allocation fitness (Eq. 4): f_i = (C/c_i)(1−p_i)·g_i, where g_i = u_i + Σ_j β_ij p_j.
  2. Retrieval score: α·sim_i + γ·log p_i + ζ·Σ_{j∈A} β̄_ij.
  3. Interaction site: a skill pair is flagged and repaired when Pr(β_ij < 0 | data) > 1−ε.

Everything else is identical across all variants: the loading model (Eq. 3), the Hedge update (Eq. 5), the retirement rule, and the sim and log p_i terms of the retrieval score.

VariantTerm removedWhat it measures
−Pairβ_ij removed from all three places: fitness becomes g_i = u_i, the ζ term is dropped from retrieval, and the interaction site is disabledContribution of all interaction-aware modeling
−Pair (alloc.)Only Σ_j β_ij p_j in the fitnessContribution of interaction-aware allocation
−Pair (retr.)Only ζ·Σ_{j∈A} β̄_ij in the retrieval scoreContribution of interaction-aware retrieval
−Pair (site)Only the interaction siteContribution of conflict repair
β≡0No interaction term in the model (τ_β² = 0 in Eq. 7); fitness reverts to γ_i, and the ζ term and interaction site are disabled with itPerformance when interactions are not modeled at all

β≡0 and −Pair differ in only one respect. When τ_β² = 0, the estimated main effect is γ_i = u_i + (1−ρ) Σ_j β_ij π_ij/π_i (Eq. 16), which still carries the interactions the skill experiences on average. −Pair uses the interaction-free u_i (Eq. 8), and is therefore the stricter test.

Results (new Appendix G.8, Figure 14, p.29). The table gives each variant's loss as a share of the ecosystem gain, defined as the improvement of Full over a system with the entire ecosystem mechanism removed and retrieval by similarity only.

# Skills−Pair (alloc.)−Pair (retr.)−Pair (site)Sum of the three−Pairβ≡0
124%11%8%23%56%31%
243%14%6%23%53%26%
485%15%8%27%52%28%

Below we explain the theoretical reason behind each number.

−Pair (alloc.): 3–5%. Replacing g_i with ĝ_i = u_i introduces a fitness error e_i = (C/c_i)(1−p_i) Σ_j β_ij p_j, and along the update direction

dŨ/dt = Cov_y(f, f̂) ≥ sd_y(f)·[sd_y(f) − sd_y(e)]

That is, the update is guaranteed to keep increasing utility only when the interaction term is less dispersed than the fitness itself. After convergence, for two conflicting skills, Full crowds out the weaker one once the conflict exceeds that skill's main effect (β_ij < −u_i). Under ĝ_i = u_i this never happens, and both skills keep their share (Figure R1). On its own this loss is small, because the ζ term in retrieval and the interaction site remain active and avoid or repair the conflicting pairs that are kept.

−Pair (retr.): 11–15%. Once a skill is selected, the ζ term down-weights candidates that conflict with it. It is the largest single contributor of the three, because it acts directly on which skills each task actually loads.

−Pair (site): 6–8%. The interaction site acts only on skill pairs judged to be in conflict.

−Pair: 52–56%, about twice the sum of the three (23–27%). A loss occurs only when a conflicting pair is loaded together, and any one of the three uses can prevent this: allocation crowds out one of the two, retrieval does not select both, and the site repairs the pair directly. When one use is removed, the other two compensate; protection is lost only when all three are removed. The number of conflicting pairs loaded per task in the model confirms this: 0.07–0.10 for Full, 0.09–0.21 for the single-use ablations, and 0.76–1.02 for −Pair, comparable to removing the ecosystem entirely (0.77–0.95).

β≡0: 26–31%, between Full and −Pair. With ĝ_i = u_i + θ Σ_j β_ij p_j, the utility loss relative to Full is second order, Δ ∝ (1−θ)². −Pair corresponds to θ = 0 and β≡0 to θ = 1−ρ, since by Eq. 16, under independent activation, γ_i = u_i + (1−ρ) Σ_j β_ij p_j. Hence, at the allocation step with ρ = 0.2, β≡0 loses only about 4% as much as −Pair; its loss comes mainly from the ζ term and the interaction site being disabled with it.

In short

  1. Interaction-aware modeling accounts for about half of the ecosystem gain.
  2. The system does not collapse when one use is missing; the other mechanisms compensate.

Q2. Stability (updated in Table 2 and Figure 5, p.8; evaluation setup in Appendix D.1 and D.4, pp.20 and 23)

Thank you for the suggestion. We have rerun all WildClawBench and component-ablation experiments with N = 5 independent evolution runs, report mean ± sample standard deviation, and updated Table 2 and Figure 5 (p.8) accordingly.

Evaluation setup. The SealQA test set has 92 questions, and a frozen judge evaluates each response three times independently, with the majority vote taken. WCB follows a 6-day evolution protocol covering 38 tasks in 4 categories, with 380 trials per day; we report Day-6 results.

Table A: Component ablation (Qwen3.5-122B, N = 5; parentheses give the difference from Full and the Welch t value)

VariantSealQA (%)WCB Avg. (%)
SkillSmith (Full)33.7 ± 1.341.4 ± 1.6
Skill-only29.3 ± 1.7 (−4.4, t = 4.6)34.6 ± 1.6 (−6.8, t = 6.7)
FreeTool31.5 ± 2.1 (−2.2, t = 2.0)37.8 ± 2.2 (−3.6, t = 3.0)
−Ecosys26.6 ± 1.9 (−7.1, t = 6.9)39.1 ± 1.8 (−2.3, t = 2.1)
−Nogood30.4 ± 1.9 (−3.3, t = 3.2)38.2 ± 2.0 (−3.2, t = 2.8)

Table B: WCB Day-6 results (SkillClaw / SkillSmith, mean ± SD, N = 5)

ScaleSocial InteractionSearch & RetrievalCreative SynthesisSafety & Alignment
9B28.3 ± 4.7 / 33.3 ± 3.710.0 ± 2.3 / 13.6 ± 2.45.4 ± 1.2 / 9.1 ± 2.213.0 ± 2.7 / 15.0 ± 2.2
35B49.9 ± 4.7 / 58.2 ± 3.922.6 ± 2.6 / 29.9 ± 2.614.4 ± 2.3 / 21.0 ± 2.323.9 ± 2.8 / 29.9 ± 2.9
397B66.7 ± 3.9 / 73.2 ± 3.938.2 ± 2.8 / 48.0 ± 3.024.7 ± 2.1 / 31.8 ± 2.735.1 ± 3.2 / 40.1 ± 3.1

Conclusion: the results are stable across independent evolution runs.

  1. Run-to-run variation in the ablations is small. SDs across variants range from 1.3 to 2.2 percentage points. Full has the lowest SD (1.3 / 1.6), about 4% of its mean. Its margin over each ablated variant is 2.2–7.1 percentage points; 6 of the 8 comparisons reach p < 0.05, and the other 2 point in the same direction with p < 0.1.
  2. Variation on WCB does not accumulate over evolution. SkillSmith's Day-6 SDs range from 2.2 to 3.9 percentage points. The mean SD over the 12 cells is 2.8 on Day 1 (before any evolution) and 2.9 on Day 6, essentially unchanged, so six days of evolution do not amplify run-to-run differences.

Sources of variation and reasons for stability.

  • Source of variation. Run-to-run differences come mainly from sampling randomness in the reflection and proposal stages: under different seeds the reflector generates different proposals, so the evolution trajectories diverge slightly.
  • Typed operations and atomic bundles constrain the search space. Proposals are restricted to a small set of typed operations, which limits how far sampling randomness can push trajectories apart. FreeTool, which removes this constraint, has the highest SD (2.1 / 2.2), while Full has the lowest.
  • The nogood veto reduces path dependence. Modifications that have already failed are not proposed again, so trajectories under different seeds are more consistent; correspondingly, the SD of −Nogood is higher (1.9 / 2.0).
  • Utility estimates are unbiased with respect to task difficulty. Theorem A.5 shows that centered pairwise features are orthogonal to arbitrary task difficulty. The estimated interaction matrix therefore does not drift with the difficulty mix of the small training set, and context-allocation decisions stay consistent across runs.

Thank you again for the time you have devoted to reviewing our paper. If our responses have addressed your concerns, we would be grateful if you would consider adjusting your score. If you have any further questions, we would be happy to discuss them. 🌸

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We are glad that reviewers found treating the skill–tool boundary as an optimization variable a meaningful extension beyond skill-only evolution (PJxH, NuAq), the formalization and design sound (NuAq, 8CMF), and the results encouraging (PJxH, 8CMF).

The main additions in the revision (marked in blue):

  1. Interaction modeling, isolated (PJxH). Removing the pairwise terms β_ij alone loses about half of the ecosystem gain (App. G, Fig. 14).
  2. Composite baseline (NuAq). EvoSkill + ToolMaker + ReasoningBank reaches 68.7% on OfficeQA; SkillSmith reaches 72.3% with 14% fewer calls. About 42% of SkillSmith's gain over EvoSkill comes from coordination rather than added components (App. E.1).
  3. Measured motivation (NuAq). In skill-only evolution, 34/36 skills duplicate logic, 32% of episodes load contradictory skills, and 22% of the budget goes to re-proposing rejected edits (Fig. 2).
  4. Reliability (PJxH, NuAq). WildClawBench and all ablations now report N = 5 runs; SkillSmith is significant in 8 of 9 settings under Holm-corrected Welch t-tests (Tables 1–2, Fig. 5, App. E.2).
  5. Scaling (8CMF). New results with Claude Fable 5.2 and GPT-6-Astra (App. E.3), an executor × reflector cross-scale study (Fig. 10), and a full baseline comparison on WildClawBench.
  6. Presentation and reproducibility (8CMF, NuAq). Redrawn Figs. 1 and 3; code in the supplement; all prompts in App. H.

Details are in the individual responses. We would be glad to discuss any remaining questions. 🌸🌸🌸

AuthorsOpenReview ↗

A kind reminder before the discussion deadline

Dear Reviewers PJxH, NuAq, and 8CMF,

Thank you again for your thoughtful reviews and for the time you have devoted to our paper. As the author–reviewer discussion period ends today (November 18, AoE), we would like to kindly remind you of our responses and revised manuscript (changes in blue).

In brief, we have addressed your main concerns:

  • PJxH: targeted ablations isolating the pairwise interaction terms, and N = 5 runs for WildClawBench and all ablations.
  • NuAq: a composite strong baseline (EvoSkill + ToolMaker + ReasoningBank), quantitative evidence for the three motivating problems, significance tests, and code and prompts for reproducibility.
  • 8CMF: a full baseline comparison on WildClawBench, results with proprietary backbones, an executor × reflector scaling study, and redrawn figures.

We fully understand how busy this period is. If you have a moment, we would greatly appreciate it if you could check whether our responses resolve your concerns, and, if so, consider updating your assessment. We are happy to answer any remaining questions before the deadline.

Thank you very much.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.