acceptodds
Withdrawn from ICLR 2027

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent, self-improvement, skill

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fyDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes WikiSkill, which keeps a separate wiki to store past agent experience and uses it to improve skills over time. the wiki records useful patterns, failures and previous skill updates. experiments on 5 benchmarks and 5 models show better performance than existing skill evolution methods, with some interesting cross-model transfer results.

Critical strengths
  1. the results are pretty solid across different models and tasks. the improvement over existing methods is quite consistent.
  2. the cross-model skill transfer results are interesting. i also appreciate that the authors included ablations to study the wiki component.
Critical weaknesses
  1. the main wiki ablation also removes the Wiki Maintainer, so its hard to tell if the gain comes from the structured wiki itself or simply from having another agent summarize past experience. a simpler memory baseline with the same information would make this much more convincing.
  2. the method adds quite a bit of extra LLM work, but there is no clear comparison of actual compute cost against the baselines. also, the experiments only cover a few evolution rounds, so the long-term benefit of keeping all this knowledge is still unclear.
Questions for authors
  1. have you tried just keeping a simple summary of past experiences instead of the full wiki structure, with the same compute budget?
  2. how much more does WikiSkill actually cost compared to the baselines in terms of tokens or time?
Reviewer_GChPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WikiSkill, a skill-evolution framework that separates raw execution traces, a persistent knowledge base called the Wiki Layer, and executable procedural skills. While candidate skill edits are gated by validation performance and rolled back if they degrade accuracy, the Wiki Layer continuously accumulates root-cause patterns, evolution logs, and proposal acceptance histories across iterations to guide a ReAct-based Skill Proposer. Evaluated across five benchmarks and five open-weight and closed models, WikiSkill consistently outperforms existing skill-evolution baselines, demonstrates that skill evolution complements model scaling, and shows that evolved skills transfer effectively across model families.

Critical strengths
  • Decoupling persistent diagnostic knowledge from reversible procedural skills is consequential because retaining an append-only audit trail of failure patterns and rejected diffs prevents the optimizer from repeating failed edits and yields a 15-point average gain over memoryless proposers.

  • The cross-model skill transfer and scaling analyses are consequential because they empirically separate skill discovery from skill execution, showing that evolved skills both amplify larger models and allow smaller models to surpass larger no-skill baselines.

Critical weaknesses
  • Injecting all active skills directly into the system prompt and growing the wiki without pruning are consequential because bypassing skill retrieval and memory management limits scalability to larger skill libraries or long-horizon evolution runs.

  • Gating candidate skills on tiny validation splits of 10 to 40 examples with strict monotonic thresholding is consequential because validation noise can cause overfitting or early stopping, while rejecting neutral intermediate edits blocks multi-step skill refinements.

Questions for authors
  • In Table 2, how are cross-model transfer scores aggregated across the three independent evolution runs of each source model, and how sensitive is validation gating to noise on small splits like SealQA with 10 validation tasks?

  • Since the Wiki Maintainer receives the full wiki context at every step without pruning, how does prompt length scale over more iterations, and how does WikiSkill perform when paired with top-k skill retrieval instead of full-prompt injection?

Reviewer_ydyWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WikiSkill, a framework for iterative skill evolution in LLM agents. The core idea is to separate an agent's workspace into three layers: a Raw Layer of immutable execution traces, a Wiki Layer that stores persistent structured knowledge distilled from those traces, and a Skill Layer containing executable procedural skills. At each iteration, an Inference Agent runs tasks with the current skills, a Wiki Maintainer consolidates sampled successes and failures into pattern pages and logs, a ReAct-style Skill Proposer consults the wiki and selected traces to propose one skill creation or edit, and a validation gate accepts only skill updates that improve held-out validation performance. Unlike skills, the wiki persists even when a proposed skill edit is rejected.

The evaluation spans five benchmarks and five models. WikiSkill obtains the best average performance among the evaluated skill-evolution methods for every model in Table 1, and the paper further studies model scaling, cross-model skill transfer, negative transfer, and wiki/skill dynamics. The results are promising and the system design is practical. I view WikiSkill as a framework for evolving reusable skill instructions through a persistent knowledge workspace, while keeping model parameters and the task-execution harness fixed. My main concern is that the central mechanism is not cleanly isolated: the decisive ablation changes persistent knowledge, the presence of the Wiki Maintainer, the representation of historical information, and optimization compute at the same time. Thus, the experiments currently show that the full WikiSkill workflow is effective, but they do not yet isolate the incremental contribution of persistence and structured wiki organization over simpler, compute-matched historical representations.

Critical strengths
  1. The three-layer architecture is intuitive: immutable traces preserve evidence, the wiki organizes optimizer-side knowledge, and deployable skills are separately validation-gated. Proposal diffs, validation outcomes, rollback, and skill-impact records make the evolution process easier to inspect. Keeping the wiki out of the default Inference Agent context is also a thoughtful design choice because it attempts to evaluate deployable skill quality rather than direct access to a memory store.

  2. The paper evaluates five heterogeneous benchmarks and five models from three families, repeats the full evolution process three times, starts all methods from an empty skill set, and reports paired-bootstrap comparisons. In Table 1, WikiSkill improves over the strongest competing baseline in macro-average by 3.3, 5.1, 10.0, 5.8, and 12.0 points for the five evaluated models. The cross-model transfer study is also informative because it separates skill discovery from skill execution and includes a useful discussion of harmful transfer on SpreadsheetBench.

Critical weaknesses
  1. The paper itself notes that EvoSkill keeps proposal/outcome histories, Trace2Skill consolidates lessons across trajectories, and SkillOpt uses rejected-edit feedback and epoch-level meta guidance. In particular, SkillOpt already has optimizer-only meta memory that records effective and rejected edits across epochs. Closely related systems also include Memento-Skills, which evolves persistent structured Markdown skills without parameter updates, and Meta Context Engineering, which co-evolves skills and external context artifacts from execution history. WikiSkill's increment appears to be a more explicit, filesystem-backed, structured knowledge layer (patterns/, logs.md, and skill-impact.md) between traces and deployable skill artifacts, rather than the general existence of non-deployed optimizer memory. This is a reasonable and potentially useful systems contribution, but the paper should more precisely explain what is new relative to prior skill-evolution memories and persistent agent-memory systems such as Reflexion/ExpeL, ReasoningBank, Agent Workflow Memory, and Voyager-style memory/skill libraries.
  2. Table 3 is the key evidence for the wiki, but the paper states that when the Skill Proposer has no wiki access, the Wiki Maintainer is also removed. Therefore, the main comparison from 48.7 to 63.7 average performance changes multiple factors at once: the presence of an additional analysis agent, additional optimizer calls, the amount and form of historical information, persistent cross-iteration state, and the interface available to the ReAct proposer. This supports the complete WikiSkill workflow, but not the stronger claim that persistence and structured wiki organization are the key drivers. A more diagnostic comparison would keep the same overall Maintainer/Proposer roles and a comparable optimization budget, while comparing at least: (i) a persistent wiki versus a wiki reset every iteration, and (ii) a structured wiki versus a matched flat history or flat summary containing the same observations and proposal outcomes. The current ablation is also limited to Gemini-3.5-Flash and four benchmarks.
  3. Appendix E reports optimizer API-call complexity, but API-call count is not the same as token, latency, or monetary cost. WikiSkill uses one Maintainer call plus roughly 10--20 ReAct proposer turns per iteration; the proposer also receives a summary of all training outcomes and may inspect trace files, so total input tokens can grow with the training set and with accumulated wiki contents even if the number of optimizer calls is constant under full-batch processing. The paper should report actual optimizer model/role mappings, batch sizes, candidate and validation counts, input/output tokens, latency, and monetary cost, and ideally provide equal-budget performance curves. Statistically, the paper repeats the full evolution process three times and uses paired bootstrap testing, which is good, but the appendix only describes resampling test tasks. It is unclear how the test accounts for run-level variation across independently evolved skill sets. Per-run scores and confidence intervals reflecting both task-level and evolution-level variability would make the conclusions more reliable.
  4. The strict-improvement validation gate is simple and auditable, but some validation splits are small: for example, SealQA has only 10 validation examples, while LiveMath and ALFWorld have 18 each. A single binary outcome can therefore move the validation score by 10 or 5.56 points. The Gemini-3.5-Flash ALFWorld case illustrates a limitation of validation saturation: validation reaches 100% before skill evolution, so no skills are evolved, while test performance remains 85.9%. This does not invalidate the reported test results, nor does it prove that further evolution would improve test performance, but it motivates a sensitivity analysis for validation split size, repeated splits, confidence-aware gating, or early stopping. Similarly, the experiments run for eight iterations with training splits of 16--80 tasks and final workspaces of about 6.3--8.9 pattern pages on average. This is useful initial evidence, but it is still a limited test of long-term compounding knowledge. The paper would be stronger if it reported how wiki entries are corrected, deduplicated, or pruned over longer runs.
  5. The paper provides many useful implementation details, but some key parameters remain unclear or inconsistent, including decoding temperature, token limits, per-task action budgets, exact model revisions, split IDs/seeds, and baseline-specific switches. The main text refers to logs.md while an appendix prompt appears to refer to log.md. OfficeQA is evaluated in a controlled setting with pre-parsed oracle reference pages provided as initial document evidence; this is a reasonable benchmark setup, but the result should not be over-interpreted as demonstrating complete end-to-end document retrieval ability. The paper should also clarify a prompt statement suggesting that the index determines what the Inference Agent reads, since the default method explicitly prevents the Inference Agent from accessing the wiki during training rollouts.
Questions for authors
  1. Under a comparable Maintainer/Proposer setup and optimization budget, how does a persistent wiki compare with a wiki reset every iteration, and with a matched flat history or summary containing the same observations and proposal outcomes?
  2. What are the actual token, latency, and monetary costs of each method? Were optimizer models, interaction limits, and total optimization budgets matched across WikiSkill and the baselines?
  3. How are the three independent evolution runs incorporated into the paired-bootstrap test? Could the authors report per-run scores and confidence intervals that include both task sampling and evolution randomness?
  4. How sensitive are accepted skills and final test scores to validation split size, strict-improvement gating, and the 100% validation early-stop rule?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.