acceptodds
Withdrawn from ICLR 2027

WASP-Mem: LET AGENT ORGANIZE PROCEDURAL MEMORY. HUMAN SPEND EFFORT ON DIAGNOSIS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

28 views

Abstract

LLM agents struggle to automate complex enterprise procedures because static, pre-built harness cannot capture proprietary policies, undocumented behaviors, or state-dependent recovery paths that surface only during execution. Procedural memory accrued from agent experience offers a training-free path to closing this gap—yet unlike episodic records or semantic facts, it must consolidate repeated successes and diagnosed failures into reusable actions and recovery strategies. We introduce WASP-Mem (Wiki-based Agent-Synthesized Procedural Memory), a nimble but effective method that distills execution traces and feedback into persistent, interlinked wiki pages, enabling continual agent learning. On TheAgentCompany benchmark, WASP-Mem raises the checkpoint pass rate by 4.6–6.1% over the strongest baseline we compared across three models while requiring 57–61% less storage. We further present a systematic controlled study spanning consolidation method, feedback type, trace ingestion scope, and refinement rounds. Our results reveal that prescriptive knowledge organization such as human defined schemas and explicit deduplication rules hurts performance rather than closing the procedural-knowledge gap. In contrast, human diagnosis converts failed traces into corrective knowledge and drives sustained improvement: a single round of human diagnosis outperforms four rounds of self-judgment by 7.0–10.9%. These findings point to a simple division of labor: let agents organize procedural memory, and invest human effort where it matters most—diagnosis.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Continual Learning, LLM Agents, Memory Consolidation, Procedural Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BQ67OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training-free procedural memory for LLM agents: how an agent that solves multi-step tasks through a shell can carry knowledge from successful and failed attempts into future attempts, keeping what matters without accumulating an unnecessary amount of information. The proposed method, WASP-Mem, follows Karpathy's "LLM wiki" idea: an LLM consolidator maintains a set of interlinked markdown pages built from execution traces and feedback. Around this method, the paper runs a controlled study of four design choices, each varied independently:

  1. the consolidation method (Free Form, Schema Form and Schema Form-ED, compared to No Memory, No Consolidation, Insights and SkillX);
  2. the feedback type (Self-Judge, Checkpoint-binary, Checkpoint-binary + Human, additive);
  3. the ingestion scope (successful traces only vs. successes and failures);
  4. the number of refinement rounds (0, 1, 2–4). Experiments use TheAgentCompany (127 tasks) and τ²-Bench Retail, with three LLMs. The paper reports that the free-form wiki performs best, and that human diagnosis of failures improves results.
Critical strengths
  • The problem is relevant and well motivated: letting agents improve on recurring workflows without retraining is a common practical need.
  • The experimental design is clear. Varying four design choices independently makes the effect of each choice easy to follow.
  • Results are reported with three different language models, which behave similarly; this suggests the main findings are not specific to one model.
  • Results are reported as both checkpoint pass rate and fully-passed rate, following common practice in the community.
  • The role of human feedback is an interesting finding: diagnosing failures improves scores, which gives practical guidance on where human effort helps.
Critical weaknesses
  1. Novelty. The core method is close to Karpathy's LLM wiki, which the paper cites as its basis. The main contribution therefore seems to be the evaluation, in particular the comparison of how wiki pages should be structured and, with that, how deduplication is handled. It remains slightly unclear how the method differs from some of the cited prior work. The related work presents cross-session consolidation and deduplication as what distinguishes WASP-Mem. AutoManual (Chen et al., 2024) already uses an LLM-based consolidator agent that merges related rules and deletes redundant ones in its procedural rule set, which seems close to this idea. Memp (Fang et al., 2025) also revises and discards memory entries. It would help if the paper stated more clearly what WASP-Mem adds beyond these works.
  2. Same tasks for training and test. On TheAgentCompany, the training and test sets are the same 127 tasks, and Experiments 2–4 (feedback type, ingestion scope, refinement) are run only on this benchmark. Only Experiment 1 is also run on τ²-Bench, where train and test tasks are disjoint. The paper motivates this choice as modelling recurring enterprise automation, and this is a legitimate setting. However, the conclusions in the title and abstract are stated as general findings about procedural memory, not as findings about re-solving tasks the agent has already seen.
  3. Task-specific memory vs. transferable knowledge. Because train and test are the same on TheAgentCompany, the memory may hold explicit information on how to execute a specific task rather than transferable procedural knowledge. The paper does not show which of the two the stores contain, so the claim of transferability is not backed for Experiments 2–4.
  4. Scale of the evaluation. The benchmarks are small (127 tasks on TheAgentCompany, and a disjoint test set of 40 tasks on τ²-Bench), which limits how general the results are. On the other hand, the main trends are consistent across three different language models, which makes it more likely that the large effects are real. Smaller differences between methods should be read with caution at this scale.
  5. Scope. The evaluation covers recurring enterprise workflows only. It is unclear whether the findings carry over to other domains.
Questions for authors
  1. How does WASP-Mem differ from Karpathy's LLM wiki and from procedural-memory methods that already consolidate or revise their memory, in particular AutoManual (with its consolidator agent) and Memp? If I have misread these works, a short clarification would help.
  2. Can you show that the memory stores contain transferable procedural knowledge rather than task-specific execution details, e.g. by releasing the final stores or by evaluating Experiments 2–4 on held-out tasks?
Reviewer_bX6yOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WASP-Mem, a training-free procedural-memory method for LLM agents on enterprise-automation tasks. It frames continual learning as a multi-round execute–consolidate cycle where the model and harness stay fixed and only an external memory store updates. WASP-Mem consolidates execution traces and feedback into persistent, interlinked markdown "wiki" pages reached from an index.md router, with three interface operations (Ingestion, Query, Lint) and agent-managed implicit deduplication (the consolidator reads the existing wiki before writing). Three variants form a ladder of increasing prescriptive structure: Free Form (unconstrained pages), Schema Form (human-defined Procedure/Action/Lesson page types), and Schema Form-ED (adds explicit deterministic deduplication). The core empirical contribution is a controlled study on TheAgentCompany (127 tasks, 542 checkpoints; train=test) and τ²-Bench Retail (disjoint train/test) across Claude Opus 4.6, GPT-5.4, and GLM-5, varying four dimensions: consolidation method, feedback type, ingestion scope, and refinement rounds. Headline findings: Free Form leads or ties checkpoint pass rate in all six model–benchmark settings (+10.9–14.5 pp over No Memory) with 57–61% smaller stores than the strongest baseline; prescriptive schemas and explicit deduplication hurt performance; failed traces help only when feedback is diagnostic; and one round of checkpoint-binary + human diagnosis beats four rounds of self-judgment by 7.0–10.9 pp. The paper distills a design principle: let the agent organize memory, spend human effort on diagnosis.

Critical strengths
  1. The controlled factorial study is the real contribution, and it is well-designed. Rather than proposing one more memory system and claiming SOTA, the paper isolates four consolidation dimensions and holds the others fixed, which lets it make interpretable claims about what makes consolidation work. Two results are genuinely useful to the field and run against intuition: prescriptive human schemas and explicit deduplication consistently underperform the Free Form variant (schema: −2.1–6.1 pp with 2.3–3.5× larger stores; explicit dedup: 1.5–6.8 pp below Free Form with 3.3–7.2× larger stores), and the feedback×ingestion interaction shows failed traces add noise under Self-Judge but +9.6–10.5 pp under diagnostic feedback. The rollout-volume control (Table 12, Fig. 5) is a nice touch: pooled Insights on more rollouts (635 vs 508) still trails Free Form R4 by 6.1–7.1 pp, which meaningfully rules out "more data" as the explanation. This is the kind of mechanism-level evidence that transfers beyond the specific system.

  2. Consistency across three frontier models and two benchmark structures. The headline ordering holds for Claude Opus 4.6, GPT-5.4, and GLM-5, and across both the train=test recurring setting (TheAgentCompany) and the disjoint-split in-domain-generalization setting (τ²-Bench Retail). Effects that survive three independent backbones and two evaluation protocols are more credible than a single tuned result, and this breadth is what gives the "let agents organize memory" principle its weight.

Critical weaknesses
  1. Every configuration was run once, with no variance quantified — and several load-bearing claims sit inside the noise band the authors themselves flag. The paper is commendably explicit about this (Appendix F), but the issue is consequential because some conclusions depend on pp gaps comparable to plausible single-run variance. The headline consolidation-method result on τ²-Bench includes a tie (Free Form ties SkillX at 81.2% for Claude), and several cross-condition contrasts are small (e.g., Task-binary vs Checkpoint-binary at 0.2–1.7 pp, which the authors rightly bracket as unresolved). Without even a handful of seeds on the central contrasts — Free Form vs the best baseline per cell, and the schema-hurts-performance claim — I cannot distinguish a robust ordering from favorable single draws. Agent rollouts are stochastic; a method paper whose entire value is a ranking of methods needs to show that ranking is stable. This is the single biggest barrier to a higher score.

  2. The primary benchmark uses train=test, which confounds "learned reusable procedure" with "memorized the specific task," and the stronger generalization test is thin. On TheAgentCompany the same 127 tasks are used for training and evaluation, mitigated only by "strict post-consolidation guardrails" against case-specific leakage that are asserted but not characterized in the main text. Because the multi-round refinement result (the headline 85.6–86.7% at R4, and the Fig. 3b task-completion story) is measured by re-executing the same tasks the memory was built from, it is impossible to tell from this setting how much is reusable procedural knowledge versus task-specific recall — exactly the distinction the paper's own Table 2 says defines procedural memory. τ²-Bench Retail provides the disjoint-split evidence, but it is a single domain with one custom-injected procedure, and the multi-round and feedback/ingestion studies are run only on the train=test benchmark. The central claim ("close the residual procedural-knowledge gap") would be much better supported if the refinement and feedback results were replicated on held-out task families.

Questions for authors
  1. Can you provide multi-seed runs (even 3) for the two central contrasts — Free Form vs the per-cell best baseline (Table 1), and Free Form vs Schema / Schema-ED — with std or min–max? Which of the sub-6-pp orderings survive? (Clarification — directly determines whether the main ranking holds.)

  2. What exactly are the "strict post-consolidation guardrails" that prevent case-specific leakage on TheAgentCompany, and how were they validated? Is there a measurement of how often a retrieved page contains task-specific answers versus generalizable procedure? (Clarification — bears on whether the headline gains reflect procedural learning.)

  3. Can the multi-round (R0–R4) and feedback/ingestion experiments be repeated on τ²-Bench's disjoint split, or any held-out task family? If the monotonic refinement gains persist off the training tasks, the central claim strengthens considerably; if not, that bounds the contribution. (Clarification.)

  4. The schema variants inherit a fixed link grammar and the "Schema omission" failure (Table 13) suggests schemas drop executable detail. Is the schema penalty an artifact of this particular schema and link grammar, or robust to a better-designed schema? A single improved schema would show whether "prescriptive structure hurts" is fundamental or implementation-specific. (Clarification — the paper generalizes this to a "bitter lesson," so its scope matters.)

  5. Human diagnosis costs ~6 annotator-hours per rollout batch and drives the strongest results. How consistent is the diagnosis across annotators (was it one annotator?), and how sensitive are the R4 gains to diagnosis quality? (Clarification — the headline human-diagnosis claim rests on this.)

  6. On TheAgentCompany, Free Form ties or is beaten on fully-passed rate by Schema Form in some cells (e.g., Claude 48.0% vs 48.8%) even while leading checkpoint pass rate. Fig. 6 attributes this to partial-credit distribution. Which metric should a practitioner optimize, and does the "Free Form wins" claim hold on the all-or-nothing metric? (Clarification.)

Reviewer_QzUbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduced WASP-Mem, which is a training-free procedural memory method in which an LLM agent consolidates execution traj and feedback into a interlinked markdown wiki. The paper constructed a controlled study on The AgentCompany and a modified tau2-bench retain with a withheld procedure. Models include Opus 4.6, GPT 5.4, GLM-5 which are great. A bunch of variations are tested. Main findings include the agent-organized wiki outperform perscribed schemas with smaller store, that failed traces help only with informative feedback.

Critical strengths
  • It's a systematic controlled study with a practical framing. Multiple frontier models are tested, I can see it's a recent work (compared to the other works using Gemini 2.5 etc.)
  • Varying methods, feedback type, ingestion scope, etc. Main findings and effects are consistent across all 3 models.
  • I do appreciate the authors are being honest and transparent about the limitations. API costs, roll-out volume control, are all certain questions to be asked nowadays.
Critical weaknesses
  • The most critical, dying weakness about this paper (and why I gave clear reject) is that this main controlled study has fundamental mistakes in settings: why evaluation based on the same tasks used for consolidation, with feedback targeting the evaluated checkpoints? For example, on TheAgentCompany, "training and test sets contain the same 127 tasks", and R1–R4 re-execute the same tasks. In Figure 4, an annotator explains each failed checkpoint in the Checkpoint-binary + Human condition, and the next round is scored on those same checkpoints. The only held-out evaluation is tau2 with 40 test tasks, covering just the consolidatio-method comparison, so the others are not tested at all under this held-out. The central aims of this paper are not supported.
  • Many experimental results are unclear, e.g., the "strict post-consolidation guardrails" are not described.
  • several conclusions are relying on small single-run differences and a single schema instantiation. Each configuration was run once, and the claim that prescriptive organization "hurts" relies on 2.1–6.1 pp gaps in checkpoint pass rate. So I wonder: what exactly were the "strict post-consolidation guardrails" against case-specific leakage?
Questions for authors

I don't have too many questions to ask actually, because I think there're some substantial changes needed to make this paper sound solid. I would encourage authors re-design the experiment, run, and resubmit in the next cycle. Some questions that could be beneficial to address:

  • can you report the feedback-type, ingestion-scope, and refinement-round experiments on held-out tasks? How large is the Checkpoint-binary + Human advantage when test tasks were never diagnosed?
  • can you provide repeated runs for the key contrasts?
  • to support the general claim about prescriptive schemas, can you consider evaluating more than one schema design and comparing with additional procedural-memory methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.