- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether language model based rewriting improves long-term memory or irreversibly removes information. It compares full-context injection, FIFO storage, several rewriting strategies, and a proposed method called PW-MLM across AMemGym, LoCoMo, and two LongMemEval settings. PW-MLM stores user messages verbatim and adds an extracted fact tier. At query time, it ranks entries, hides older semantically similar entries, fills a fixed token budget, and answers several related questions together when possible. The paper reports that aggressive rewriting produces much smaller memory stores and lower question answering performance. PW-MLM generally approaches full-context quality while injecting fewer tokens.
Critical strengths
-
The paper asks a specific and testable question about the write side of agent memory. The strongest evidence appears in Table 3. Removing the verbatim tier while retaining the fact tier reduces the judge score from 62.0 to 49.8 on AMemGym and from 54.9 to 35.6 on LoCoMo. This controlled ablation supports the narrower conclusion that an accessible raw archive helps recover details lost during extraction.
-
The reporting is unusually auditable. The paper includes prompts, fixed hyperparameters, storage sizes, query tokens, category breakdowns, and code. In the roughly 120k token setting, PW-MLM obtains a judge score of 48.8 using about 3,909 injected tokens, compared with 25.2 for FIFO and at most 22.8 for the Qwen rewriting baselines. This is a meaningful result even if the paper's broader interpretation requires revision.
Critical weaknesses
-
The experiment does not isolate rewriting from the read policy, information scope, or backbone. PW-MLM uses a 3,900 token budget, top 30 retrieval, recency weighting, temporal conflict resolution, batched answering, and a dedicated answer prompt. The Qwen rewriting baselines inject only 54 to 684 tokens per question. A fair write-side comparison would keep this read pipeline fixed and replace only raw entries with rewritten entries. The stored information also differs. PW-MLM's extraction prompt explicitly includes information stated by the assistant. The AWE prompt explicitly excludes assistant responses, while AWI only extracts user facts. Some of the assistant-side performance gap therefore follows directly from the prompt rather than from rewriting. The backbone comparison is incomplete. Qwen3.7-plus evaluates all methods except A-Mem. GLM-5.3-Flash evaluates only A-Mem and PW-MLM. The paper says A-Mem uses its official GLM backbone, but this does not explain why A-Mem cannot be run with Qwen or why the remaining baselines are absent under GLM. Backbone and memory strategy effects are therefore inseparable.
-
This omission is consequential because A-Mem outperforms PW-MLM under GLM on every external benchmark. Its judge scores are 61.5 versus 59.0 on LoCoMo, 84.6 versus 72.1 on LME-S, and 78.0 versus 53.2 on LME-Oracle. On LoCoMo, A-Mem also uses fewer query tokens. Reclassifying A-Mem as near-verbatim after observing these results does not resolve the contradiction. It instead suggests that compression ratio or retained information, rather than rewriting as a binary property, is the relevant variable. The history regimes and cost comparison do not support several headline claims. On LoCoMo, the No Memory baseline injects 12,160 tokens from a store of 12,221 tokens. It is therefore not truncated. The experiment shows that other methods use smaller retrieval budgets, not that full-history injection is infeasible. Each history tier is also a different benchmark with different question types, history construction, and metrics. Cross-tier differences cannot be attributed only to history length. PW-MLM amortizes one context over batches of questions. On AMemGym, ten questions share a 3,900 token context, producing the reported cost of about 397 tokens per question. The paper does not say whether the No Memory and rewriting baselines receive equivalent batching. If full-context questions were batched in the same way, much of the reported 13 times cost reduction would disappear. Write-time costs are also excluded, including PW-MLM fact extraction every six turns and the construction cost of competing memory systems. Store size alone is not a measure of forgetting. Mem0 uses a much smaller store on LME-S but reaches 64.6 judge points with 584 query tokens, compared with 64.5 and 736 for PW-MLM. AWI also ties PW-MLM at 62.0 on AMemGym while using 117 rather than 397 query tokens. These results contradict the claim that PW-MLM lies on the unique Pareto frontier. All experiments use one seed. Qwen3.7-plus is simultaneously the main answering backbone, the LLM judge, and the generator of AMemGym. There is no independent human evaluation of the open-ended results.
Questions for authors
-
Why is A-Mem not evaluated with Qwen3.7-plus, and why does the GLM-5.3-Flash panel contain only A-Mem and PW-MLM? Can the authors provide a complete model by method comparison so that backbone and memory strategy effects can be separated?
-
Can the authors compare a verbatim store and each rewriting store under exactly the same read policy? This would require the same 3,900 token budget, top 30 retrieval, conflict resolution, batching, answer prompt, and information scope. A plain verbatim top-k baseline is also needed.
-
Are No Memory, FIFO, AWI, AWE, Mem0, and A-Mem batched across questions in the same way as PW-MLM? Please report context tokens per model call before amortization and include write-time extraction, consolidation, embedding, and retrieval costs.
-
On what basis is A-Mem classified as a rewriting baseline when it stores 123,472 tokens in the long setting and substantially outperforms PW-MLM? Which consistency, consolidation, eviction, and MERGE prompts were active in the reported PW-MLM runs?