- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper generates months of consistent mail, calendar, drive, photo records for a virtual user from one hidden event timeline. it then compiles 1,034 memory questions with known evidence from the timeline, without human authoring. It evaluates five memory systems under one harness on answer quality, record-level retrieval and cost.
Critical strengths
Deriving records & questions from one hidden timeline is a good way to get verifiable ground truth. Premise-perturbation questions and openable office files go beyond conversational benchmarks. I checked the reported numbers. Overall scores in Table 1 equal the weighted means of subtypes, the Oracle score recomputes to 93.5 on the 781 questions, and Table 9 matches the multi-hop means in Table 1. The paper also reports results that work against its own claims, like the 7.3 pt run-to-run difference for Mem0.
Critical weaknesses
Most of the benchmark is not audited? Only 192 of 1,034 questions (L01 and L11, both Chinese) were checked one by one, and the 11 released lives were not sent for record review (App. D). This matters for the rankings. Using Table 1 and Table 10, on the 192 audited questions EverOS is 74.7 and Mem0 58.9, but on the other 842 EverOS is about 66.7 and Mem0 about 69.0. EverOS goes from 3rd to last. All ablations also run only on these two lives, which were picked because annotators read Chinese, not at random. Mem0 also looks atypically bad here (54.2 on memory conflict in Table 14 vs 78.8 overall). So does the claim that a stronger reader widens the Mem0-RAG gap hold on the full set? Please report per-life scores, since there are only 11 lives.
Run-to-run variance is close to the gaps used to rank systems. Table 1 ranks systems 1.1-1.4 pts apart (Letta 77.9 vs RAG 76.5, EverOS 68.2 vs Mem0 67.1) with one run and one judge run. In App. H, RAG changes by 2.6 pts between two runs even though its ingestion has no LLM call, and Mem0 changes verdict on 53 of 192 questions. Please add repeated runs and intervals for Table 1.
The gates likely make the question set easier than real use. Gblind keeps only questions a GPT model can answer and locate among 5 lexical competitors, and failed ones get more locating cues added. 36% of candidates were dropped, mostly multi-hop (Table 6). So the kept multi-hop questions are the easy-to-locate ones?
Mem0’s recall is inflated since facts from a chunk of up to 8 records are credited to all of them (App. G). But sec 4.2 uses Mem0 being only 2.4 pts below RAG on Recall@30 as evidence that extraction loses details. This comparison doesnt hold with biased recall.
The ≥6-record drop (n=49, fig 3, Table 9) looks a lot like the time-slice scores (EverOS 41.8 vs 41.0, others around 8-12 in both), so this group is likely mostly time slices. The harness also turns off agentic retrieval (sec 4.1), so each system gets one similarity query. So is the drop about how memory is organized, or about the harness?
The judge protocol used in the paper is not validated. The 93.8% judge-human agreement comes from 80 pairs of the first compiled set, not the released 1,034, and was measured before the three-level verdict was introduced (App. D).
Everything uses GPT models - generation, question writing, blind validation, extraction, reader and judge. Blind validator only keeps questions a GPT model can answer from the evidence, so absolute scores and the Oracle may favor GPT.
Photo questions are answered from captions not images, so they dont really test multimodal memory (Oracle 59.5). Sec 4.4 says no current system stores multimodal content, but MIRIX, Mem0 with vision and RAG-Anything do.
LLM suggestion checking against relevant papers- the paper misses three relevant works published before the July 17 cutoff: M3Exam (arXiv 2606.07402) evaluates multimodal memory systems, DynamicMem (arXiv 2606.22877) also finds most failures come from retrieval, and PersonaTrace (EACL 2026 Industry) also synthesizes digital footprints.
Questions for authors
Can the authors audit a stratified sample from the nine unaudited lives and report the error rate?
Can the authors report per-life scores and confidence intervals (repeated runs or bootstrap over lives) for Table 1?
Do the reader and k ablations hold on the other nine lives?
Can you compare Mem0 and RAG recall with exact record attribution?
What is the subtype mix of the ≥6-record group?
How well does the final three-level judge agree with humans on the released questions?
Do the results hold with a reader and judge from another model family?
Happy to raise the score if you properly address or rebut these questions.
2 comments
Thanks for the response. The recall fix is good, esp since you caught the RAG attribution bug yourselves. The cross-family reader and judge runs and the per-life CIs also make the ranking claims more honest.
I’m keeping my score tho. The main findings now only hold for the harness setup. The time-slice drop holds for single-query similarity retrieval at k=30, and the two-step baseline wasnt run. So is this about how the systems organize memory, or about the harness? we cant tell from the current results. The detail loss finding is also weaker now that the recall evidence is withdrawn.
2.6% error on the audited lives is fine. But both audited lives are Chinese and the system ranking flips by language, so why would the rate transfer to the other nine lives? A small audit of one non-Chinese life would help a lot.
The pipeline is useful. For a benchmark paper I’d want either checked data across languages or findings that hold outside this harness setup, and the paper doesnt have either yet.
Again happy to raise score if you justify yourself/add more evidence.
We thank the reviewer for checking the arithmetic and for the concrete list. References are to the revised PDF; "Oracle" is the column the revision calls the gold-evidence reader.
Q1. Not within the discussion period: annotators who read English, Japanese or Portuguese natively cost considerably more than our Chinese-speaking team and could not be organised in time. Human review is sampled calibration at three points (outsourced review of development-life records, 99/99/85/97%; our audit of two Chinese lives, 195 compiled questions, 3 dropped, 2 ground truths corrected, none rewritten, 2.6%; the judge on 80 pairs), while code validators run on every question. The revision marks the nine lives as unaudited compiler output and reports the groups separately: on the other 842 questions EverOS is 66.7 and Mem0 68.9. The swap follows language, not audit status: on L10, the unaudited Chinese life, EverOS again leads Mem0 (76.4 vs 61.5), and overall the two lie within each other's intervals (Appendix D). All data will be released for independent audit.
Q2. Appendix D gives per-life and per-language scores (the verbatim stores are top two in 10 of 11 lives). Bootstrap 95% intervals over questions: RAG 76.5 [74.1, 78.8]; Mem0 67.1 [64.4, 69.8]; Letta 77.9 [75.5, 80.2]; EverOS 68.2 [65.6, 70.8]; Graphiti 71.8 [69.2, 74.3]. Paired differences: Letta − RAG 1.4 [−0.4, 3.1]; RAG − Graphiti 4.7 [2.2, 7.2]; Graphiti − EverOS 3.6 [0.6, 6.6]; EverOS − Mem0 1.2 [−1.7, 4.2]. The revision therefore ranks in three groups (verbatim stores; Graphiti; EverOS and Mem0). No repeated runs were added beyond the two in Appendix I; the intervals replace run-to-run deviations (W2).
Q3. Reader: yes, on all eleven lives Claude Opus 4.5 scores 1.9–5.0 below GPT-5.6 on every system (Oracle −3.1) and Gemini 3.1 Pro 1.9–4.6 (Oracle −1.9) (Q7). k: not re-run on the nine lives; the captions say so (Appendix I).
Q4. Correct, and wider than Mem0: the harness also credited verbatim-RAG memories to the whole ingestion chunk, so the submission's single-record claim was wrong. Strict Recall@30: RAG 64.8, Mem0 50.2 (an estimate), Letta 62.4; multi-hop full recall 20.1/9.1/20.1. The §4.2 comparison is withdrawn; detail loss now rests on the reader ablation (verbatim stores +10–13, Mem0 +0.6), the drive gap (48.7 vs 83.1) and the extractor ablation. Table 1 reports strict values, Appendix H both.
Q5. All 49 such questions are time slices, and the revision labels the group so (§4.2, Appendix H): whole-day aggregation is where single-query retrieval fails and only EverOS's episodes survive it. The finding is conditional on single-query similarity retrieval; the two-step baseline (locate the anchor, fetch the day) was not run, and the claim is restricted accordingly. Only 4 of 61 time-slice questions contain a date, so a date filter does not apply.
Q6. No human figure exists for it on the released questions; the revision says the 93.8% predates it. Two judges from different families agree on 87.6/85.9/85.0% of 5,951 labels (κ 0.73/0.72/0.71) for the GPT-5.6, Claude and Gemini answers, mostly on knowledge updates (Appendix J); a hand review of the 14 answers the judge flagged as uncertain found 2 judge errors and 2 dataset defects, since corrected (Appendix E).
Q7. Yes. Claude Opus 4.5 and Gemini 3.1 Pro re-answered the same contexts and Claude Haiku 4.5 re-graded every answer: six reader × judge cells (Appendix J). In every cell the verbatim stores are first and second, Graphiti third, Mem0 and EverOS within each other's intervals; RAG − Mem0 is +8.0 to +14.2; on time slices single-query systems score 0.8–17.2, EverOS 34–42, Oracle 67–77. The new readers score 1.9–5.0 below GPT-5.6 on every system and 1.9–3.1 on the Oracle, so the difference is reader capability. The blind validator was not swapped; Oracle scores of 90.5 and 91.7 under the other families bound the selection effect at a few points.
Remaining weaknesses. W3: the round-trip gate selects for what the validator's family can answer; the revision says so (§3.3, limitations). Dropped candidates are mostly leakage (85), vague anchors (80) and unresolvable references (47), not validator failure (44), and the kept set is not easy to retrieve (best system 59 on multi-hop). W8: §4.4 now names MIRIX, Mem0 with vision input and RAG-Anything as storing multimodal content; photo questions test caption-based memory, and no vision-capable gold-evidence reader was run (Appendix F). W9: M³Exam, DynamicMem and PersonaTrace are added to §2.
Full tables and per-life numbers are in the revised PDF (Appendices D, H, I, J) and in our global response.