- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Melo, a closed-loop long-term memory framework for LLM agents motivated by cognitive accounts of attentional bias, prediction error, and reconsolidation. The core idea is that evidence observed after a memory-supported response (user corrections, tool outputs, task outcomes) should be returned to the persistent state used by later queries. The implementation is a deterministic, rule-based pipeline: a write score computed from adapter-supplied importance, confidence, profile overlap, and noise tags; tag-based routing into working, episodic, semantic, and procedural memory; a binary source/tag trigger for feedback admission; consolidation of three or more procedure events into one workflow summary; and exponential time-decay retention. The authors also construct CMLM-Assist, a synthetic, template-based benchmark with 1,920 questions (1,536 held out). Experiments on LoCoMo and CMLM-Assist with DeepSeek-V4-Flash, Qwen3-14B, and GPT-4o-mini show that Melo achieves the highest F1, BLEU-1, ROUGE-L, and METEOR against six baselines (Vanilla RAG, Mem0, A-Mem, MemoryOS, MemoryBank, MemGPT/Letta), and a matched GPT-4o-mini ablation shows that removing each of four components lowers F1.
Critical strengths
Clear problem definition and transparent scoping. The paper gives a precise four-condition definition of a closed loop (memory use -> feedback admission -> state change -> later consultation) and separates it from outcome metrics such as accuracy and retention. The Monday -> Friday deadline example makes the target failure mode concrete. The authors are also candid about the limits of their evidence, including the non-equal retrieval budget, the author-constructed benchmark, low conflict-resolution scores, and unvalidated coefficients.
Auditable design with provenance. All operators are deterministic for a fixed configuration, and consolidated procedures keep the identifiers of the records they replace. For personal-assistant memory, inspectable provenance is a practical property that many LLM-driven memory pipelines lack, and the audit artifacts in Table 8 are potentially reusable.
Critical weaknesses
Confounded comparison and missing control experiments. Melo uses top-k = 20 (expandable to 60 on CMLM-Assist), while Vanilla RAG uses top-k = 5 and the external systems keep their native settings. The main tables therefore cannot separate Melo's memory mechanisms from a 4-12x larger evidence context. Several baseline numbers also look under-configured: MemGPT/Letta gets 9.5-12.1 F1 on LoCoMo, MemoryBank gets 1.34 F1 on CMLM-Assist (DeepSeek), and Mem0 on LoCoMo with GPT-4o-mini (28.54) seems well below what the original Mem0 paper reports. In the LoCoMo GPT-4o-mini block, Vanilla RAG, Mem0, A-Mem, and MemoryOS are almost identical on every metric, which suggests they may be getting near-identical contexts. Moreover, the matched-ablation control reaches 77.00 F1 while the main-table GPT-4o-mini result is 46.59, so configuration alone moves F1 by about 30 points. The following controls are needed: Vanilla RAG (and at least one memory baseline) at top-k = 20 and at a matched token budget, with the same backend and prompt as Melo. Baselines under the matched-ablation protocol (TF-IDF, top-k = 20, same prompt) where Melo reaches 77.00. A Full Context reference on LoCoMo. This is consequential because, without these controls, the main empirical claim cannot be attributed to Melo's memory design rather than to budget and configuration.
The closed-loop mechanism is largely inactive where it is evaluated. On LoCoMo, source records are mapped to protected episodic records, the deliberation trigger and retrieval gate are disabled, and there are no declared correction or task-outcome events for Eq. (6) to admit. Feedback admission, consolidation, and retention therefore seem mostly inert on the only independent benchmark, and the LoCoMo gains are more likely explained by retrieval scoring and budget. On CMLM-Assist, the direct test of the return path is the "w/o Feedback-to-Memory Update" ablation, but its diagnostic (procedures formed 96 -> 0) holds by construction. Conflict-resolution F1 stays below 8% on every backbone, and per-type results for temporal update, the case that motivates the paper, are not reported. The evaluation is also offline replay, so the loop never responds to Melo's own outputs. This is consequential because the closed loop is the main contribution, while the experiments mostly show the effect of profile-aware retrieval.
Oracle-like metadata in CMLM-Assist. In CMLM-Assist, importance, confidence, noise tags, and semantic tags are generated from the scenario specification, and Melo's rules consume exactly these fields (noise penalty in Eq. (9), tag routing in Eq. (11) including a benchmark-specific "university" tag, noise removal in Eq. (14)-(16)). Appendix D.6 shows that noise records score 0-0.083 and valid records score 0.586-0.878, so the distractors are removed by labels rather than by any inferred filtering. Baselines presumably do not use this metadata. In addition, all coefficients are hand-set on a development split that shares templates with the test split, with no sensitivity study. This is consequential because the largest margins (e.g., 46.59 vs. 20.58 F1) come from the benchmark whose construction matches Melo's input schema, while a real host would have to produce this metadata itself.
Limited novelty and overstated cognitive framing. The authors acknowledge that no individual operator is novel, and each is a simple rule (set-membership trigger, count-based consolidation, fixed exponential decay, keyword routing). Event-triggered revision, provenance, and procedural skill formation already appear in Reflexion, Voyager, Mem0, and A-Mem, so the claimed distinction (trigger, timing, provenance) is narrow. Prediction error and reconsolidation are central in the title and method, but the implementation uses none of their computational content (no error magnitude, no lability window). It is unclear what is conceptually new beyond an auditable rule-based memory pipeline.
Questions for authors
What are the results when Vanilla RAG and at least one memory baseline are run at top-k = 20 and at a matched token budget, and under the matched-ablation protocol where Melo reaches 77.00 F1? Does the same improvement trend hold?
On LoCoMo, which Melo components are actually active? How many feedback events are admitted, how many procedures are formed, and how many records are removed by retention? Could the authors provide a LoCoMo ablation?
How does Melo perform on CMLM-Assist when the importance, confidence, noise, and semantic tags are removed or inferred from raw text by an LLM? Conversely, how do the baselines perform when given the same metadata?
Could the authors report per-type results for all eight CMLM-Assist question types, especially temporal update and negative abstention?
Why are several baselines far below Vanilla RAG and below their published numbers, and why are the four GPT-4o-mini LoCoMo baselines nearly identical? Did any baseline fall back to a default path or receive empty memory?
Only lexical metrics are reported, and Melo's answers are about 2.5x longer than the references (12.43 vs. 4.95 tokens). Could the authors add an LLM-as-judge evaluation on LoCoMo and a length-controlled analysis?
How sensitive are the results to the hand-set coefficients in Eq. (7)-(9) and (13)-(15)? Also, what are the per-query latency, number of LLM calls, and context tokens for Melo compared with the baselines?