- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies training-free procedural memory for LLM agents: how an agent that solves multi-step tasks through a shell can carry knowledge from successful and failed attempts into future attempts, keeping what matters without accumulating an unnecessary amount of information. The proposed method, WASP-Mem, follows Karpathy's "LLM wiki" idea: an LLM consolidator maintains a set of interlinked markdown pages built from execution traces and feedback. Around this method, the paper runs a controlled study of four design choices, each varied independently:
- the consolidation method (Free Form, Schema Form and Schema Form-ED, compared to No Memory, No Consolidation, Insights and SkillX);
- the feedback type (Self-Judge, Checkpoint-binary, Checkpoint-binary + Human, additive);
- the ingestion scope (successful traces only vs. successes and failures);
- the number of refinement rounds (0, 1, 2–4). Experiments use TheAgentCompany (127 tasks) and τ²-Bench Retail, with three LLMs. The paper reports that the free-form wiki performs best, and that human diagnosis of failures improves results.
Critical strengths
- The problem is relevant and well motivated: letting agents improve on recurring workflows without retraining is a common practical need.
- The experimental design is clear. Varying four design choices independently makes the effect of each choice easy to follow.
- Results are reported with three different language models, which behave similarly; this suggests the main findings are not specific to one model.
- Results are reported as both checkpoint pass rate and fully-passed rate, following common practice in the community.
- The role of human feedback is an interesting finding: diagnosing failures improves scores, which gives practical guidance on where human effort helps.
Critical weaknesses
- Novelty. The core method is close to Karpathy's LLM wiki, which the paper cites as its basis. The main contribution therefore seems to be the evaluation, in particular the comparison of how wiki pages should be structured and, with that, how deduplication is handled. It remains slightly unclear how the method differs from some of the cited prior work. The related work presents cross-session consolidation and deduplication as what distinguishes WASP-Mem. AutoManual (Chen et al., 2024) already uses an LLM-based consolidator agent that merges related rules and deletes redundant ones in its procedural rule set, which seems close to this idea. Memp (Fang et al., 2025) also revises and discards memory entries. It would help if the paper stated more clearly what WASP-Mem adds beyond these works.
- Same tasks for training and test. On TheAgentCompany, the training and test sets are the same 127 tasks, and Experiments 2–4 (feedback type, ingestion scope, refinement) are run only on this benchmark. Only Experiment 1 is also run on τ²-Bench, where train and test tasks are disjoint. The paper motivates this choice as modelling recurring enterprise automation, and this is a legitimate setting. However, the conclusions in the title and abstract are stated as general findings about procedural memory, not as findings about re-solving tasks the agent has already seen.
- Task-specific memory vs. transferable knowledge. Because train and test are the same on TheAgentCompany, the memory may hold explicit information on how to execute a specific task rather than transferable procedural knowledge. The paper does not show which of the two the stores contain, so the claim of transferability is not backed for Experiments 2–4.
- Scale of the evaluation. The benchmarks are small (127 tasks on TheAgentCompany, and a disjoint test set of 40 tasks on τ²-Bench), which limits how general the results are. On the other hand, the main trends are consistent across three different language models, which makes it more likely that the large effects are real. Smaller differences between methods should be read with caution at this scale.
- Scope. The evaluation covers recurring enterprise workflows only. It is unclear whether the findings carry over to other domains.
Questions for authors
- How does WASP-Mem differ from Karpathy's LLM wiki and from procedural-memory methods that already consolidate or revise their memory, in particular AutoManual (with its consolidator agent) and Memp? If I have misread these works, a short clarification would help.
- Can you show that the memory stores contain transferable procedural knowledge rather than task-specific execution details, e.g. by releasing the final stores or by evaluating Experiments 2–4 on held-out tasks?