- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
this paper proposes WikiSkill, which keeps a separate wiki to store past agent experience and uses it to improve skills over time. the wiki records useful patterns, failures and previous skill updates. experiments on 5 benchmarks and 5 models show better performance than existing skill evolution methods, with some interesting cross-model transfer results.
Critical strengths
- the results are pretty solid across different models and tasks. the improvement over existing methods is quite consistent.
- the cross-model skill transfer results are interesting. i also appreciate that the authors included ablations to study the wiki component.
Critical weaknesses
- the main wiki ablation also removes the Wiki Maintainer, so its hard to tell if the gain comes from the structured wiki itself or simply from having another agent summarize past experience. a simpler memory baseline with the same information would make this much more convincing.
- the method adds quite a bit of extra LLM work, but there is no clear comparison of actual compute cost against the baselines. also, the experiments only cover a few evolution rounds, so the long-term benefit of keeping all this knowledge is still unclear.
Questions for authors
- have you tried just keeping a simple summary of past experiences instead of the full wiki structure, with the same compute budget?
- how much more does WikiSkill actually cost compared to the baselines in terms of tokens or time?