- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SkillSmith, a skill–tool co-evolution framework for self-improving LLM agents. The framework consists of three main components: (1) a skill ecosystem that allocates limited context resources based on skill utility and interactions, (2) tool evolution that extracts reusable deterministic logic from skills into tools, and (3) conflict-driven failure memory that prevents repeated unsuccessful modifications. Experimental results demonstrate that SkillSmith consistently improves agent performance throughout the evolution process, with larger gains observed as the backbone model scales up.
Critical strengths
-
Well-motivated problem formulation and promising empirical results. SkillSmith addresses a meaningful problem in the self-evolution of LLM agents. The reported performance improvements are substantial, and the paper provides extensive analyses to support the effectiveness of the proposed framework.
-
Rigorous and well-structured formalization. The paper presents a comprehensive mathematical formulation of skill–tool co-evolution, with theoretical analyses supporting several key design choices.
Critical weaknesses
-
Limited scale of the skill and tool libraries. The skill and tool libraries considered in the experiments are relatively small compared with those that may arise in real-world agent systems. This raises concerns about whether the proposed setting adequately captures the context-budget constraints that motivate the framework. In particular, it is unclear to what extent the context budget becomes a binding constraint in the current experiments, and whether the proposed approach would remain effective with substantially larger skill and tool libraries.
-
Potentially unreliable skill utility estimation due to limited training data. The training sets for OfficeQA and SealQA are very small, both in absolute size and relative to the full datasets (24/246 for OfficeQA and 11/111 for SealQA). This may substantially affect the reliability of skill utility estimation based on randomized skill masking, particularly when certain skills are more relevant to the limited training samples than to the overall task distribution. Consequently, the estimated individual and pairwise skill contributions may be noisy or biased, potentially affecting subsequent context allocation decisions.
Questions for authors
-
The experimental comparisons are somewhat limited, and different baselines are used across benchmarks. Specifically, Table 1 compares SkillSmith against EvoSkill, whereas Figure 3 compares it against SkillClaw. It would be helpful if the authors could provide more comprehensive comparisons across benchmarks or clarify the rationale behind this experimental design.
-
An interesting observation is that SkillSmith achieves larger performance gains as the backbone model scales up. To further investigate the generalizability of this trend, it's benefit for the authors to evaluate SkillSmith with SOTA proprietary LLMs.
-
Disentangling the effects of Executor and Reflector scaling. In the current experimental setup, the Executor and Reflector use the same backbone model, making it difficult to determine whether the observed scaling benefits primarily arise from improved task execution or more effective evolutionary reflection. Additional experiments with heterogeneous model configurations, such as a small-model Executor paired with a large-model Reflector and vice versa, would help disentangle these effects.
-
Figures 1 and 2 suffer from several visual rendering and alignment issues that significantly affect their readability. I recommend carefully revising these figures to improve their visual clarity and presentation quality.
4 comments
re 2/3
Q2. Performance with stronger backbones (new Appendix E.3, p.24, Figure 12, Table 6)
To examine whether the trend "the stronger the backbone, the larger SkillSmith's gain" holds for top closed-source models, we built Chronos Bazaar, a 30-round dynamic adversarial sandbox with extreme context pressure, skill conflicts, and non-stationary rule changes. We evaluate the state-of-the-art closed-source models Claude Fable 5.2 and GPT-6-Astra side by side with the open-source model family.
Task setup and hybrid evaluation protocol
The agent acts as an intergalactic anti-money-laundering investigator and high-frequency arbitrage officer. Under a strict context window ( tokens) and a budget of 60 validated modifications (), it goes through three phases of evolution:
See Appendix E.3 for the task setup and phase-specific rules.
Results across backbones (5 independent runs, mean SD)
| Backbone | Final (D30, %) | Recovery (D25, %) | Payload (%) | Bundle pass (%) | Tool errors (%) | Conflicts left (pairs) | Validation budget used () |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | (exhausted) | ||||||
| Qwen3.5-35B | |||||||
| Qwen3.5-122B | |||||||
| GPT-6-Astra | |||||||
| Claude Fable 5.2 |
Analysis: why stronger backbones unlock larger gains
Cross-layer restructuring has a capability threshold. Weaker models (9B/35B) are limited in synchronizing code across multiple components and in abstraction. Their first-pass rate for atomic bundles is very low (–), so they struggle to move deterministic computation into tools, and the context window remains crowded with redundant instructions (payload ). The top models exceed a pass rate, quickly slim down their skills, and free the scarce context budget for higher-level strategy.
- Moving logic back from tools to skills when the environment changes. After the rule drift on D21, the 35B model keeps piling rigid regular-expression patches (
PATCH) onto the invalidated tool. These happen to pass the small validation set (reaching on D24), but trigger snowballing false-freeze penalties on the test stream, and accuracy falls steadily to . In contrast, Claude Fable 5.2 and GPT-6-Astra recognize that the task is no longer context-independent: on D22 they decisively executeINLINEto retire the old tool and return pun interpretation to the skill layer, and on D23 they extract the pure weather lookup into a new tool, rapidly recovering to –.
Where the advantages of Claude Fable 5.2 and GPT-6-Astra come from. GPT-6-Astra excels in engineering quality and sample efficiency: the highest first-pass rate for code, the best tool compression, and the lowest budget use. Claude Fable 5.2 excels in adapting to non-stationary environments and in deeper strategic reasoning: it resolves multi-skill conflicts faster and restructures more decisively after the rule change.
- Strong backbones compound the value of failure memory (
Nogood). From the minimal failing subset isolated byddmin, the top models infer the underlying mechanism in one step and generate precise remedy constraints. They converge early on D27, stop making ineffective modifications, and achieve a clear lead using only about half of the budget.
Further direct evidence comes from WikiSkill [1]: on Qwen 4B/9B/27B, the gains are +12.3/+17.5/+23.9, and its abstract states that larger models benefit more from evolved skills. Indirect support comes from two works showing that self-improvement has a capability threshold. In SkillsBench [2], self-written skills yield zero or negative gains on most backbones, and only the latest Opus 4.6 is positive (+1.4, possibly within noise). Sample More, Reflect Less [3] shows that for small 1.5B–7B models, self-reflection loses to repeated sampling in every case at equal token cost.
[1] Tang et al., WikiSkill, arXiv:2608.27454, 2026 [2] SkillsBench, arXiv:2602.12670, 2026 [3] Mirzaei, Sample More, Reflect Less, arXiv:2607.28576, 2026
re 3/3
Q3. Scale of the executor and the reflector (updated in §4.3, p.10, Figure 10)
We thank the reviewer for this suggestion. In the original experiments, the executor and the reflector used the same frozen backbone, so the effects of their respective scales could not be separated. We have therefore added a cross-scale experiment over executor M_E × reflector M_R (N = 5; parentheses give the gain over the no-evolution baseline in percentage points).
OfficeQA
| M_E \ M_R | No evolution | 9B | 27B | 35B | 122B |
|---|---|---|---|---|---|
| 9B | 14.0 | 17.7 (+3.7) | 20.8 (+6.8) | 22.4 (+8.4) | 25.6 (+11.6) |
| 27B | 26.1 | 28.5 (+2.4) | 34.2 (+8.1) | 36.8 (+10.7) | 41.5 (+15.4) |
| 35B | 34.8 | 37.6 (+2.8) | 43.1 (+8.3) | 46.2 (+11.4) | 52.8 (+18.0) |
| 122B | 52.6 | 55.4 (+2.8) | 63.5 (+10.9) | 67.2 (+14.6) | 72.3 (+19.7) |
SealQA
| M_E \ M_R | No evolution | 9B | 27B | 35B | 122B |
|---|---|---|---|---|---|
| 9B | 5.7 | 7.8 (+2.1) | 9.6 (+3.9) | 10.5 (+4.8) | 12.8 (+7.1) |
| 27B | 10.2 | 11.8 (+1.6) | 15.1 (+4.9) | 16.8 (+6.6) | 20.4 (+10.2) |
| 35B | 13.3 | 15.0 (+1.7) | 18.6 (+5.3) | 20.9 (+7.6) | 25.4 (+12.1) |
| 122B | 20.4 | 22.8 (+2.4) | 27.5 (+7.1) | 30.1 (+9.7) | 33.7 (+13.3) |
Conclusion: the reflector sets the ceiling on improvement, and the executor's scale amplifies it.
① The reflector has a capability threshold. With M_R = 9B, the gain is only +1.6 to +3.7 regardless of executor size. Only when the reflector reaches 27B or larger do the gains widen substantially (e.g., from +2.8 to +10.9 at M_E = 122B). This is consistent with prior findings on self-reflection methods: self-diagnosis and strategy revision require causal attribution across trajectories and depend on a strong base model (Reflexion, Shinn et al., 2023; Self-Refine, Madaan et al., 2023). Reflectors below 30B often manage only superficial fixes.
② The tool layer transfers a strong reflector's capability to a weak executor. With M_E = 35B and M_R = 122B, OfficeQA reaches 52.8, on par with the 122B no-evolution baseline (52.6), and SealQA reaches 25.4, exceeding the 122B baseline (20.4). The reason is that EXTRACT and PATCH turn deterministic logic (e.g., merged-cell parsing, unit conversion) into tool code, so execution no longer depends on the LLM's stochastic interpretation. A small executor only needs to pass arguments according to the tool signature to directly reuse the larger model's abstractions. This is consistent with the finding that "large models make tools, small models use them" (LATM, Cai et al., 2024; Voyager, Wang et al., 2023; ADAS, Hu et al., 2024).
③ Context offloading benefits small executors more. Small models are more sensitive to long contexts and conflicting instructions (β_ij < 0) (Lost in the Middle, Liu et al., 2024). The 122B reflector allocates 74% of the context to skills with positive utility and precisely removes conflicting skills, leaving only 0.12 conflicting pairs per task, compared with 0.64 for the 9B reflector. This directly reduces the cognitive load on small executors.
Q4. Figure readability (updated in Figure 1, p.2, and Figure 3, p.4)
We thank the reviewer for the detailed comments. We have redrawn Figure 1 and the original Figure 2: we fixed rendering artifacts and element alignment, unified fonts, colors, and labels, and increased the resolution. Because the revision adds a figure quantifying the motivation (now Figure 2), the original Figure 2 is now Figure 3. The new Figure 1 is on page 2 and Figure 3 on page 4, with all changes marked in blue.
Thank you again for the time you have devoted to reviewing our paper. Your comments have made our baseline comparisons, scale analysis, and figure presentation more complete. If you have any further questions, we would be happy to discuss them. 🌸
Thank you for the authors' detailed responses and additional experiments, including the clarification of the baseline selection and the analysis of performance differences across executor–reflector configurations of different model scales. However, I still have two concerns regarding the newly introduced Chronos Bazaar experiment:
- Why did the authors introduce a new evaluation environment instead of conducting the experiments on the existing benchmarks, namely OfficeQA, SealQA, and WildClawBench?
- My original Q2 aimed to examine whether the observed trend that SkillSmith achieves larger performance gains as the backbone model scales up also generalizes to SOTA proprietary LLMs. However, although the authors constructed a new and relatively elaborate evaluation environment, they did not report the performance gains of SkillSmith over a no-evolution baseline or other evolution baselines under the same backbone model. This mismatch between the experimental design and the original evaluation objective is somewhat puzzling. I believe the authors should provide a clearer justification for the purpose and design of this additional experiment.
Given these concerns, I have decided to maintain my original rating.
re 1/3
Thank you for your careful review and valuable comments. We have added the corresponding experiments, and changes in the revised manuscript are marked in blue. We respond to each point below.
Q1. Rationale for the baseline comparisons
We thank the reviewer for this comment. Baselines were chosen according to the setting each method is designed for.
OfficeQA and SealQA are static question-answering tasks with few interactions between skills. This is exactly the setting EvoSkill is designed for, so it is the strongest and fairest baseline on these two benchmarks. SkillClaw is designed for settings with multiple users, continuously arriving data, and substantial interactions within the skill library; on these two benchmarks its collective-evolution mechanism has nothing to work with, so the comparison would be of limited value.
WildClawBench is the opposite: it involves multi-day continual evolution with frequent skill co-activation. EvoSkill has no ecosystem-level design and performs only modestly there, so SkillClaw is the matching strong baseline. In response to the reviewer, we added a full WCB comparison on Qwen3.5-35B (N = 5, Day 6, averaged over the four task categories):
On WCB, EvoSkill improves over the no-evolution baseline by only 1.0 percentage point, within the error range. In addition, SkillSmith and SkillClaw are concurrent work, and when this work was carried out, few comparable methods existed in this direction. On each benchmark we chose the strongest method that best matched it at the time.