acceptodds
Under review as a conference paper at ICLR 2027

LIFEFORGE: SYNTHETIC DIGITAL LIVES FOR MULTI-SOURCE AGENT MEMORY EVALUATION

Abstract

AI assistants increasingly build memory from a user's personal data spread across many sources rather than from conversation alone. Evaluating them requires multi-source personal data with verifiable ground truth, which real users cannot provide for privacy reasons, and existing memory benchmarks remain almost exclusively conversational. We present LifeForge, a two-stage framework that synthesizes a virtual user's multi-source data and compiles memory questions from it without human authoring. Stage one derives every record from one hidden event timeline, projecting each event into mutually consistent calendar entries, emails, office files and photos. Because every record traces back to a known event, Stage two can select the evidence for a question and keep only questions whose answer is unique. We release 11 digital lives in six countries and four languages, 29K records in all, and 1,034 questions covering four memory operations. Scoring five open-source memory systems under one harness, on answer quality, record-level retrieval and the cost of writing and reading memory, exposes two shared failures. They retrieve records one at a time, so most collapse once an answer has to gather many records from one stretch of time; giving the same reader the gold evidence closes most of that gap, which puts the fault in retrieval. They extract facts at ingestion and discard the details questions ask about, a loss a stronger reader widens rather than repairs.

Primary area:
datasets and benchmarks
Keywords:
memory benchmark, agent memory, memory evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G4RhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper generates months of consistent mail, calendar, drive, photo records for a virtual user from one hidden event timeline. it then compiles 1,034 memory questions with known evidence from the timeline, without human authoring. It evaluates five memory systems under one harness on answer quality, record-level retrieval and cost.

Critical strengths

Deriving records & questions from one hidden timeline is a good way to get verifiable ground truth. Premise-perturbation questions and openable office files go beyond conversational benchmarks. I checked the reported numbers. Overall scores in Table 1 equal the weighted means of subtypes, the Oracle score recomputes to 93.5 on the 781 questions, and Table 9 matches the multi-hop means in Table 1. The paper also reports results that work against its own claims, like the 7.3 pt run-to-run difference for Mem0.

Critical weaknesses

Most of the benchmark is not audited? Only 192 of 1,034 questions (L01 and L11, both Chinese) were checked one by one, and the 11 released lives were not sent for record review (App. D). This matters for the rankings. Using Table 1 and Table 10, on the 192 audited questions EverOS is 74.7 and Mem0 58.9, but on the other 842 EverOS is about 66.7 and Mem0 about 69.0. EverOS goes from 3rd to last. All ablations also run only on these two lives, which were picked because annotators read Chinese, not at random. Mem0 also looks atypically bad here (54.2 on memory conflict in Table 14 vs 78.8 overall). So does the claim that a stronger reader widens the Mem0-RAG gap hold on the full set? Please report per-life scores, since there are only 11 lives.

Run-to-run variance is close to the gaps used to rank systems. Table 1 ranks systems 1.1-1.4 pts apart (Letta 77.9 vs RAG 76.5, EverOS 68.2 vs Mem0 67.1) with one run and one judge run. In App. H, RAG changes by 2.6 pts between two runs even though its ingestion has no LLM call, and Mem0 changes verdict on 53 of 192 questions. Please add repeated runs and intervals for Table 1.

The gates likely make the question set easier than real use. Gblind keeps only questions a GPT model can answer and locate among 5 lexical competitors, and failed ones get more locating cues added. 36% of candidates were dropped, mostly multi-hop (Table 6). So the kept multi-hop questions are the easy-to-locate ones?

Mem0’s recall is inflated since facts from a chunk of up to 8 records are credited to all of them (App. G). But sec 4.2 uses Mem0 being only 2.4 pts below RAG on Recall@30 as evidence that extraction loses details. This comparison doesnt hold with biased recall.

The ≥6-record drop (n=49, fig 3, Table 9) looks a lot like the time-slice scores (EverOS 41.8 vs 41.0, others around 8-12 in both), so this group is likely mostly time slices. The harness also turns off agentic retrieval (sec 4.1), so each system gets one similarity query. So is the drop about how memory is organized, or about the harness?

The judge protocol used in the paper is not validated. The 93.8% judge-human agreement comes from 80 pairs of the first compiled set, not the released 1,034, and was measured before the three-level verdict was introduced (App. D).

Everything uses GPT models - generation, question writing, blind validation, extraction, reader and judge. Blind validator only keeps questions a GPT model can answer from the evidence, so absolute scores and the Oracle may favor GPT.

Photo questions are answered from captions not images, so they dont really test multimodal memory (Oracle 59.5). Sec 4.4 says no current system stores multimodal content, but MIRIX, Mem0 with vision and RAG-Anything do.

LLM suggestion checking against relevant papers- the paper misses three relevant works published before the July 17 cutoff: M3Exam (arXiv 2606.07402) evaluates multimodal memory systems, DynamicMem (arXiv 2606.22877) also finds most failures come from retrieval, and PersonaTrace (EACL 2026 Industry) also synthesizes digital footprints.

Questions for authors

Can the authors audit a stratified sample from the nine unaudited lives and report the error rate?

Can the authors report per-life scores and confidence intervals (repeated runs or bootstrap over lives) for Table 1?

Do the reader and k ablations hold on the other nine lives?

Can you compare Mem0 and RAG recall with exact record attribution?

What is the subtype mix of the ≥6-record group?

How well does the final three-level judge agree with humans on the released questions?

Do the results hold with a reader and judge from another model family?

Happy to raise the score if you properly address or rebut these questions.

2 comments

We thank the reviewer for checking the arithmetic and for the concrete list. References are to the revised PDF; "Oracle" is the column the revision calls the gold-evidence reader.

Q1. Not within the discussion period: annotators who read English, Japanese or Portuguese natively cost considerably more than our Chinese-speaking team and could not be organised in time. Human review is sampled calibration at three points (outsourced review of development-life records, 99/99/85/97%; our audit of two Chinese lives, 195 compiled questions, 3 dropped, 2 ground truths corrected, none rewritten, 2.6%; the judge on 80 pairs), while code validators run on every question. The revision marks the nine lives as unaudited compiler output and reports the groups separately: on the other 842 questions EverOS is 66.7 and Mem0 68.9. The swap follows language, not audit status: on L10, the unaudited Chinese life, EverOS again leads Mem0 (76.4 vs 61.5), and overall the two lie within each other's intervals (Appendix D). All data will be released for independent audit.

Q2. Appendix D gives per-life and per-language scores (the verbatim stores are top two in 10 of 11 lives). Bootstrap 95% intervals over questions: RAG 76.5 [74.1, 78.8]; Mem0 67.1 [64.4, 69.8]; Letta 77.9 [75.5, 80.2]; EverOS 68.2 [65.6, 70.8]; Graphiti 71.8 [69.2, 74.3]. Paired differences: Letta − RAG 1.4 [−0.4, 3.1]; RAG − Graphiti 4.7 [2.2, 7.2]; Graphiti − EverOS 3.6 [0.6, 6.6]; EverOS − Mem0 1.2 [−1.7, 4.2]. The revision therefore ranks in three groups (verbatim stores; Graphiti; EverOS and Mem0). No repeated runs were added beyond the two in Appendix I; the intervals replace run-to-run deviations (W2).

Q3. Reader: yes, on all eleven lives Claude Opus 4.5 scores 1.9–5.0 below GPT-5.6 on every system (Oracle −3.1) and Gemini 3.1 Pro 1.9–4.6 (Oracle −1.9) (Q7). k: not re-run on the nine lives; the captions say so (Appendix I).

Q4. Correct, and wider than Mem0: the harness also credited verbatim-RAG memories to the whole ingestion chunk, so the submission's single-record claim was wrong. Strict Recall@30: RAG 64.8, Mem0 50.2 (an estimate), Letta 62.4; multi-hop full recall 20.1/9.1/20.1. The §4.2 comparison is withdrawn; detail loss now rests on the reader ablation (verbatim stores +10–13, Mem0 +0.6), the drive gap (48.7 vs 83.1) and the extractor ablation. Table 1 reports strict values, Appendix H both.

Q5. All 49 such questions are time slices, and the revision labels the group so (§4.2, Appendix H): whole-day aggregation is where single-query retrieval fails and only EverOS's episodes survive it. The finding is conditional on single-query similarity retrieval; the two-step baseline (locate the anchor, fetch the day) was not run, and the claim is restricted accordingly. Only 4 of 61 time-slice questions contain a date, so a date filter does not apply.

Q6. No human figure exists for it on the released questions; the revision says the 93.8% predates it. Two judges from different families agree on 87.6/85.9/85.0% of 5,951 labels (κ 0.73/0.72/0.71) for the GPT-5.6, Claude and Gemini answers, mostly on knowledge updates (Appendix J); a hand review of the 14 answers the judge flagged as uncertain found 2 judge errors and 2 dataset defects, since corrected (Appendix E).

Q7. Yes. Claude Opus 4.5 and Gemini 3.1 Pro re-answered the same contexts and Claude Haiku 4.5 re-graded every answer: six reader × judge cells (Appendix J). In every cell the verbatim stores are first and second, Graphiti third, Mem0 and EverOS within each other's intervals; RAG − Mem0 is +8.0 to +14.2; on time slices single-query systems score 0.8–17.2, EverOS 34–42, Oracle 67–77. The new readers score 1.9–5.0 below GPT-5.6 on every system and 1.9–3.1 on the Oracle, so the difference is reader capability. The blind validator was not swapped; Oracle scores of 90.5 and 91.7 under the other families bound the selection effect at a few points.

Remaining weaknesses. W3: the round-trip gate selects for what the validator's family can answer; the revision says so (§3.3, limitations). Dropped candidates are mostly leakage (85), vague anchors (80) and unresolvable references (47), not validator failure (44), and the kept set is not easy to retrieve (best system 59 on multi-hop). W8: §4.4 now names MIRIX, Mem0 with vision input and RAG-Anything as storing multimodal content; photo questions test caption-based memory, and no vision-capable gold-evidence reader was run (Appendix F). W9: M³Exam, DynamicMem and PersonaTrace are added to §2.

Full tables and per-life numbers are in the revised PDF (Appendices D, H, I, J) and in our global response.

Thanks for the response. The recall fix is good, esp since you caught the RAG attribution bug yourselves. The cross-family reader and judge runs and the per-life CIs also make the ranking claims more honest.

I’m keeping my score tho. The main findings now only hold for the harness setup. The time-slice drop holds for single-query similarity retrieval at k=30, and the two-step baseline wasnt run. So is this about how the systems organize memory, or about the harness? we cant tell from the current results. The detail loss finding is also weaker now that the recall evidence is withdrawn.

2.6% error on the audited lives is fine. But both audited lives are Chinese and the system ranking flips by language, so why would the rate transfer to the other nine lives? A small audit of one non-Chinese life would help a lot.

The pipeline is useful. For a benchmark paper I’d want either checked data across languages or findings that hold outside this harness setup, and the paper doesnt have either yet.

Again happy to raise score if you justify yourself/add more evidence.

Reviewer_U8gROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the evaluation of assistant memory built from a user's personal data spread over several sources (mail, calendar, cloud drive, photos), a setting where real data cannot be collected and conversational benchmarks do not apply. LIFEFORGE is a two-stage generator. Stage one expands a short free-text brief into a hidden provenance structure (a persona, recurring series, and an event timeline generated in 14-day windows) and projects each event onto the platforms it would plausibly touch, producing threaded mail, RRULE calendars with exception rows, openable office files and face-consistent photos, with code-computed validators and two halting gates enforcing consistency. Stage two compiles questions from the hidden structure: selectors pick the evidence, an LLM phrases the question from the visible view only, and three gates (a leak scan, a blind re-answering validator with lexical competitors, and type-specific checks) keep, repair or drop candidates, so that no human writes or rewrites a question. The released LIFEFORGE-BENCH holds 11 lives in six countries and four languages (28,952 records, 3.6K files, 3.6K photos) and 1,034 questions over single-hop, multi-hop, abstention and preference operations, 781 of them with record-level evidence sets; 192 questions from two Chinese-language lives were audited item by item by the authors. Five open-source memory systems (verbatim RAG, Mem0, Letta, EverOS, Graphiti) are evaluated under one harness with a shared embedding model, reader (GPT-5.6-sol), judge (GPT-5-mini) and k = 30. The verbatim stores lead (Letta 77.9, RAG 76.5); extraction-based systems trail by 4 to 11 points and lose most on document-content questions; every system but EverOS collapses on time slices and on questions needing six or more records, while an oracle given the gold evidence scores 93.5, which the authors read as placing the fault in retrieval; a weaker reader costs the verbatim stores 10 to 13 points but Mem0 only 0.6, so detail lost at extraction is not recoverable by the reader. Ablations over k, reader, Mem0's extractor, Graphiti's episode granularity and run-to-run variance are reported on the 192 audited questions.

Critical strengths

S1. Timely problem and a clear argument. The three coupled obstacles (no real data, naive synthesis breaks consistency, no traceable provenance) are well articulated, and the design answers each: one hidden timeline as the single source of truth, projection rather than independent generation, and questions compiled from the provenance so that evidence sets are complete by construction.

S2. The generation pipeline is principled and the engineering is serious. The windowed factorization keeps prompts bounded with O(K) calls and a resumable chain; schedules are expanded in code because LLMs miscompute dates; triggered_by links and stable contact identifiers carry information across windows and name variants; files are accepted only if the format's SDK opens them; photos are conditioned on reference portraits. Table 2 is a careful inventory of validators with repair or fallback per step, and the identifiability block's "stop rather than degrade" rule is the right instinct.

S3. Question compilation is thoughtfully gated. Evidence is selected before phrasing, the question writer sees only the visible view, the repair loop may add locating cues but not change the answer, and the layered ground truth with a factual-error veto is motivated by a measured judge effect (8–9 percent pass on long free-text ground truths against 74 percent on short ones). Memory-conflict questions that corrupt one detail of a real record probe a failure mode existing benchmarks rarely test, and the stale_gt flag is a sensible guard for knowledge-update currency.

S4. The system comparison follows the MemDelta recipe where it can: fixed embedding model, reader, judge and k; one record per message; record-level retrieval metrics; ingestion and reader-token cost; an Oracle column; k and reader ablations; extractor and episode-granularity ablations; repeated runs. The finding that a stronger reader widens rather than closes the extraction gap is crisp, well supported by Table 13, and practically useful, as is the cost picture (Graphiti's 186K calls and 693M tokens buy third place; Letta leads with zero ingestion calls).

S5. Unusual candor about what was and was not verified: the 192/842 audit split, the per-system attribution bias in retrieval metrics, Graphiti as a lower bound, Mem0's 7.3-point run-to-run variation, photos entering only as captions, EverOS's unmeterable cost. The reproducibility statement (seeded compilation, content-hash identifiers, persisted logs for every call and dropped candidate) is strong.

S6. The paper is well organized and readable; the appendices give quotas, thresholds, worked examples and the judge protocol at a level of detail that would let others reproduce the compiler's behavior on the released lives.

Critical weaknesses

W1. The quality of 81 percent of the released questions is inferred rather than measured. Of 1,034 questions, 192 (two Chinese lives) were audited item by item, by members of the author team rather than independent annotators; the other 842, in languages the team does not read, are raw compiler output. The evidence offered for them is an 80-pair review drawn from 75 questions of a single compiled set (87.5 percent directly usable, 11.3 usable with flaws, 1.3 unusable) and record pass rates measured on development lives rather than the released ones, which "were not sent for outsourced review". Two numbers that would let a reader calibrate the raw output are missing: how many of the 192 audited questions had their ground truth, evidence set or text changed by the audit, broken down by failure type (non-unique answer, stale knowledge-update state, wrong detail, unanswerable from the visible view), and how many stale_gt corrections were applied. The drive pass rate of 85 percent is measured after the final fixes and concerns wrong contact names and contradictions with the event or an earlier file version, which bear directly on the 77 drive single-hop questions and on chains spanning several documents. Correcting ground truths during the release audit also sits uneasily with the claim that humans never rewrite anything. A stratified independent audit across all 11 lives, with native-speaker annotators for English, Japanese and Portuguese and reported correction rates, would move this from inferred to measured.

W2. The entire stack is one vendor's model family, which the paper's own adopted protocol warns against. The lives are written by GPT-5.5, questions are phrased and blind-validated by GPT-5.4, answers are written by three GPT-5.6 variants (or GPT-4.1-mini), extraction for three systems runs on GPT-4.1-mini, and grading is by GPT-5-mini. The blind validator is a round-trip filter: a question survives only if a GPT-5.4 model answers it correctly from the visible records, so the released set is selected for what this family finds answerable, and the Oracle score of 93.5 obtained by a sibling reader is in part a measurement of that selection. MemDelta, cited as the protocol being followed, calls for stratification by model family, but the "three readers from the same series" vary within a family rather than across families. At least one reader and one judge from a different family are needed, and ideally one life re-gated with a different blind validator, to show that the benchmark and the diagnoses drawn from it are not family-specific.

W3. Retrieval metrics computed under system-specific attribution rules are compared in one row and argued from. The paper states that Mem0 credits each fact to all records of its chunk, EverOS can be attributed only at session level, and Graphiti's entity summaries never count (a lower bound that offline resolution raises by about 20 points). Tables 1 and 8 nevertheless place these numbers side by side, and Sections 4.2 and 4.4 reason from them, for example that Mem0's Recall@30 on memory conflicts is within three points of RAG's, so the 12.9-point score gap must be detail loss. Under chunk-level attribution Mem0's true recall could be far lower, and the inference does not follow.

W4. Equal k is not an equal budget, the context caps are asymmetric, and the harness disables the capabilities whose absence it then diagnoses. Thirty Mem0 facts carry 2.9K reader tokens against 9.3K for thirty verbatim records, and Table 12 shows Mem0 still rising from k = 20 to k = 30 on single-hop (62.0 to 64.8) and multi-hop (28.7 to 33.1), so part of its deficit is budget starvation rather than extraction loss. A token-matched comparison (Mem0 and Graphiti at the k that equalizes reader tokens) is needed before the extraction conclusion is clean, even though the reader ablation supports it independently. The context cap is 150K characters for the verbatim stores and 60K for the others, truncating EverOS 122 times and Graphiti 23 times against 3 for the verbatim stores, no reason is given. And EverOS is the system whose long episodes carry the time-slice advantage. Finally, agentic retrieval and metadata filters were disabled for every system and each returns its 30 nearest memories by similarity, so the conclusions that "the problem is the unit of retrieval" and that memory "should be organised around time" are partly properties of this configuration. A date-filtered or BM25-hybrid baseline (the dates in time-slice questions are lexical anchors) and a full-context baseline for the five lives under one million tokens would show how much of the collapse at six or more records is fundamental.

W5. The Oracle overstates what perfect retrieval at k = 30 would deliver. It receives the gold records alone (1.6K tokens against 9 to 13K for the systems), in chronological order rather than a ranking order, and for time slices an extra anchor record that is not in the evidence set. Each of these helps the reader independently of retrieval, so "the multi-hop gap lies in retrieval, not in the reader" is established only up to these confounds.

W6. The ablation subset is not representative of the benchmark, and per-life results are absent. On the two audited Chinese lives Mem0 scores 58.9 against 67.1 overall and EverOS 74.7 against 68.2, so the ordering among the extraction systems reverses between Appendix H and Table 1, and Mem0's memory-conflict score is 54.2 there against 78.8 overall. This points to strong language effects in LLM extraction that the paper never surfaces, because no per-life or per-language table is given.

W7. Realism and difficulty of the synthetic lives are asserted, not measured. There is no distributional comparison with a real corpus (thread length, reply latency, message length, vocabulary, entity repetition against Enron or a public calendar dump), no human real-versus-synthetic discrimination test, and no check of coherence across non-adjacent windows, although the factorization lets an event see only the previous window. By construction the records of one event agree, so the reconciliation the introduction motivates, records "written at different times by different people", is never tested under real-world disagreement. The benchmark measures a clean regime that favors verbatim retrieval, and the recommendation to layer extraction over verbatim records may not hold where deduplication and normalization matter. Preference questions reward regularities the generator itself imposed (a naming pattern is a content-specification choice before it is a habit).

W8. A large share of the benchmark does not discriminate. Fictitious-entity questions are solved at 100 percent by every system (88 questions); single-hop mail and calendar sit at 93 to 97 for the verbatim stores. Preference scores span 76.7 to 80.3 and fall to 65 to 78 with a smaller reader regardless of memory, which marks them as a reader test; and photo questions (37) are answered from a caption that even the Oracle cannot use beyond 59.5. The overall weighted score is therefore dominated by items that separate nothing. The paper should report a hard subset (multi-hop, drive, memory conflict) as a headline beside the overall score, and validate photo-question answerability with a vision-capable oracle that sees the image rather than the caption. A benchmark release on already-good-enough evaluation cuts its value. Because it might signals the problem is near-solved by the existing system/solution.

Questions for authors

Q1. For the 192 audited questions, how many had the ground truth, evidence set or question text changed, by failure type? How many stale_gt corrections were applied during the release audit?

Q2. Which compiled set did the 80-pair usability review cover, and in which language? Would you run a stratified independent audit over all 11 lives with native-speaker annotators?

Q3. Can you report results with a reader and a judge from a different model family, and re-gate one life with a different blind validator to measure how many keep/drop decisions depend on the validator's family?

2 comments

Q1. Of 195 compiled questions for the two audited lives, 3 were dropped (single-hop) and 2 ground truths corrected (one stale_gt knowledge update; one single-hop); none rewritten, no evidence set changed; 2.6%. That is the one stale_gt correction. "Humans never rewrite" is made precise in the revision: humans never write or rewrite questions, only drop or correct a ground truth. Human review is sampled calibration at three points (outsourced review of development-life records, 99/99/85/97%; this audit; the judge on 80 pairs and 14 flagged answers), while code validators run on every question. Our annotators read Chinese natively, hence both audited lives are Chinese (Appendix E).

Q2. The first compiled set of L11, in Chinese, by twelve team members. An independent native-speaker audit of the other nine lives was not possible in the discussion period (such annotators cost considerably more and could not be organised in time). The revision marks the nine lives as unaudited compiler output, reports audited vs unaudited and per-language scores (Appendix D); all data will be released.

Q3. Reader and judge: done, findings hold; re-gating with a non-GPT validator: not done, bound below. Claude Opus 4.5 and Gemini 3.1 Pro re-answered the same contexts and Claude Haiku 4.5 re-graded every answer: six cells (Appendix J). In every cell the verbatim stores are first and second, Graphiti third, Mem0 and EverOS within each other's intervals; RAG − Mem0 is +8.0 to +14.2; on time slices single-query systems score 0.8–17.2, EverOS 34–42, Oracle 67–77. Claude scores 1.9–5.0 below GPT-5.6 on every system (Oracle −3.1), Gemini 1.9–4.6 (−1.9); the Oracle moves with the systems, so this is reader capability. Haiku agrees with GPT-5-mini on 87.6/85.9/85.0% of labels (κ 0.73/0.72/0.71); knowledge updates are the sensitive category; the aggregate moves ≤1.5 on GPT answers and ≤4.8 on the others. Validator: gold-evidence scores of 90.5 and 91.7 vs 93.5 bound family fit at 2–3 points; validator failure accounts for 44 of 469 dropped candidates; the revision states the selection effect in §3.3 and bounds it in §4.3.

Regarding the remaining weaknesses, all points are addressed in the revised manuscript. For retrieval (W3), we now report a strict Recall@30 that credits verbatim-RAG memories, such as Mem0’s, only to the records they contain rather than to every record in the ingestion chunk: RAG 64.8, Mem0 50.2, Letta 62.4, EverOS 67.1 (session-level) and Graphiti 68.9 (lower bound). Table 1 reports the strict values and Appendix H reports both protocols. Because Mem0 trails RAG by 14.6 points under this protocol, part of its deficit is attributed to retrieval, and the extraction claim is supported by the reader ablation, the drive gap and the extractor ablation. For retrieval budget (W4), the context caps are fixed harness settings, not tuned values: 150K tokens so that 30 full records fit, and 60K for the other systems, with truncation counts given in Appendix F (EverOS 122, Graphiti 23). The revision notes that part of Mem0’s deficit may reflect budget (Table 14) and scopes the claim to single-query similarity retrieval. Since only 4 of the 61 time-slice questions contain a date, a date filter cannot precede anchor location, so two-step retrieval, BM25-hybrid and full-context baselines are identified as future work. The oracle setting (W5) is now named the gold-evidence reader, and Appendix F states how it differs from the standard setting: gold records only (1.6K vs. 9–13K tokens), chronological order, and the anchor record for time-slice questions. The corresponding claim reads “most of the multi-hop gap closes when the reader is given the evidence.” Per-life and per-language scores (W6) are reported in Appendix D. They show that Mem0’s extractor loses most on Chinese (59.8 vs. 71.0 on English, consistently under the Claude and Gemini readers), and that EverOS’s 74.7 on the audited lives is its best case (it ranks last on five of the six English lives). On data realism (W7), Appendix C adds the mail distributions, and a limitations paragraph notes that the corpus has not been compared with a real corpus, that no real-vs-synthetic or cross-window consistency test has been run, and that in this clean regime the records of one event agree, so reconciliation is outside the current scope. For preference questions we report the spread of supporting evidence. Finally (W8), Table 1 and Appendix D add a hard-subset headline over the 548 multi-hop, drive and memory-conflict questions: RAG 67.5, Mem0 53.9, Letta 70.1, EverOS 58.4 and Graphiti 64.1. Photo questions are clarified as testing caption-based memory, and no vision-input reader was evaluated (Appendix F). §4.4 states that none of the five evaluated systems stores multimodal content, whereas MIRIX, Mem0 with vision input and RAG-Anything do.

Reviewer_U8gROpenReview ↗

Official Comment by Reviewer U8gR

Thank you for your clarifications. It address most of my questions and concerns, however due to the data quality setback that a significant share of the benchmark does not discriminate, I am maintaining my original rating.

Reviewer_v6tiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LifeForge, a framework for building memory benchmarks from synthetic multi-source personal data. In Stage one, a short free-text brief is expanded into a hidden persona, a set of recurring series and an event timeline, which is generated in 14-day windows. Each event is then projected into mail, calendar entries, office files and photos, so every record traces back to the event it came from. In Stage two, questions are compiled from this hidden structure. The evidence is selected first, an LLM phrases the question from the visible records only, and code and LLM gates filter the candidates for answer leakage and uniqueness. The released benchmark, LifeForge-Bench, has 11 lives in six countries and four languages, with 28,952 records and 1,034 questions. The questions cover single-hop, multi-hop, abstention and preference questions. The data are validated in three ways: by these code and LLM checks, by a human audit of the 192 questions from two of the lives, and by a quality review run on development lives. The authors evaluate five open-source memory systems under one harness and report two findings. Systems fail when an answer needs many records from one stretch of time. Systems that extract facts at ingestion lose the details that questions ask about.

Critical strengths

S1. The records follow real platform formats rather than plain text logs. Mail is threaded, calendars use recurrence rules with exception rows, office files open in their format’s software, and photos are generated images. Questions can therefore target document bodies and photo content, which conversational benchmarks cannot. This is consequential because it is what distinguishes the benchmark from prior multi-source work. That includes LifeBench, whose traces the paper itself describes as “text logs, with photos appearing only as entries” (p. 3).

S2. 781 questions carry their exact evidence records. The evaluation can therefore report record-level retrieval recall next to answer scores, alongside ingestion cost and reader input tokens. This is consequential because it lets the benchmark diagnose why a memory system fails, in retrieval or in reading, rather than only whether it fails. That value depends on the labels being correct, which is the subject of the first weakness.

Critical weaknesses

W1. Most of the benchmark has never been checked by a human. Only 192 of the 1,034 questions were audited item by item, all from two Chinese-language lives, by “native Chinese speakers on our team” (p. 7). The other nine sets, including the third Chinese life, “are raw compiler output” (p. 7). The figures offered for what that raw output is worth were not measured on it. The question-usability rate (87.5% directly usable) and the 93.8% judge–human agreement come from 80 question–answer pairs of “the first compiled set” (p. 20). The record pass rates come from development lives (W5). The paper also does not say how many of the 192 audited questions were corrected or dropped, which would be the most direct estimate of the error rate in the other 842. All ablations run on these same 192 questions (p. 8), so their conclusions rest on two lives in one language.

W2. The paper overstates its novelty relative to work it cites, and omits close prior work. The introduction presents its “key idea” as generating the provenance first and deriving both the records and the questions from it (p. 1). Section 2 itself credits this idea to earlier work. LoCoMo “expands persona-conditioned event graphs into dialogues”, LifeBench “writes questions during the simulation so that evidence is grounded”, and of Setoka’s event tree the authors write “We build on this idea” (p. 3). TimelineQA (Tan et al., 2023) is not cited at all, although it already generated synthetic lifelogs “of imaginary people”. It also found that aggregate questions are answered best only when the ground-truth set of episodes is supplied, a close parallel to this paper’s main Oracle finding. Finally, Section 4.4 states that existing benchmarks “build memory from conversations alone, so their evidence never spans platforms” (p. 9). This contradicts the multi-source benchmarks the paper itself describes in Section 2. What appears genuinely new is the platform-faithful rendering (threaded mail, recurring calendar entries with exceptions, openable files, generated photos), the question gates and the record-level evidence sets. The contribution would be clearer framed around these.

W3. The lives are generated in 14-day windows, a design that limits their long-range coherence, and the paper never tests that coherence. Each window conditions only on the persona PP, the series state Sk−1S_{k-1} and the previous window’s events Ek−1E_{k-1} (Eq. 1, p. 4). The authors acknowledge that “a window cannot see events older than the previous window” (p. 5). Information older than one window therefore survives only through the fixed persona, the recurring-series state and triggered_by links. This works for recurring commitments and for processes that stay active in every window. It does not obviously work for long-running, intermittent processes, which are common in real personal data. A paper submission is a good example: the submission, weeks of silence, then reviews, a rebuttal and a decision months later. When the decision email is generated, the model has never seen the submission. LifeBench (Cheng et al., 2026), which the paper calls the closest prior work, avoids this by planning a year-long plot outline before generating daily detail. It is unclear why LifeForge did not adopt a top-down plan or a compressed history instead. Several further points compound this:

  • the claim that the horizon is “a freely settable parameter” (p. 5) is never tested for coherence;
  • the longest released life is 133 days (Table 4), against one year for LifeBench;
  • every life starts empty on the first day of its horizon, whereas a real account carries years of older mail and files that act as distractors for retrieval;
  • the lives’ temporal statistics, such as how long threads go quiet or how often events occur, are never compared with real accounts, for example the Enron corpus that the paper cites.

Since the benchmark is motivated by memory over “months” of multi-source data (p. 1), this design directly limits what it can claim to measure.

W4. The persona never changes during the horizon, although a person’s status plausibly does within three months. The persona PP is written once from the brief, and every window is conditioned on it unchanged (Eq. 1, p. 4). The only part that can change is the recurring series, which can start, stop or change (the series delta Δk\Delta_k, p. 14). How does the proposed pipeline handle such a thing? or this situation just gets exclude from the benchamrk? For example, the life L03 is a “senior undergraduate, job seeking” (Table 5). If that user finds a job during the horizon, the change is recorded only as events, while every later window is still conditioned on a job-seeking persona. The design is therefore inverted: the attributes that should evolve are frozen, while the event history, which should persist, is cut off after one window (W3). The same assumption reaches the preference questions. Habits are kept if at least three records support them, with no stated check on when those records occur (p. 5), so a habit that changed during the horizon would yield an ambiguous ground truth. Prior work designs around this. PersonaMem (Jiang et al., 2025) updates a dynamic part of the persona through timestamped events and tests those updates, and LifeBench (Cheng et al., 2026) carries change through a year-long plot outline.

W5. Record contents are never checked for consistency, although contradictions between independently generated records, which “silently corrupt the ground truth” (p. 1), are the problem the method is built to solve. However, this problem is only hanlded structurally but not content-verified in the later context of the writeup. I can somehow see how this problem is handled partly via the construction of the stages but still the main problem, i.e., the constradiction between records’ content (not the structure like the email fields, etc), is not verfied.

The stages construction only validates structural errors such as schemas and identifier syntax; the cadence grammar of recurring series (for example, weekly needs a weekday set); slot coverage, name uniqueness and revision equality; whether files open in their format’s library. And the only check that reads a body is whether a mail body mentions its attachment, and it is performed by an LLM even though Table 2 is captioned “deterministic validators”. Nothing checks that the times, people, amounts or conclusions written in an email body or a document agree with the source event, with the calendar entry of that event, or with an earlier version of the same file.

The authors’ own development review found exactly these failures: drive failures are “mostly wrong contact names inside a document and contradictions with the event or an earlier version of the file”, with an 85% pass rate (p. 20; Table 6). That review covered development lives (12 lives for calendar and mail, 3 for drive and photos), and “the 11 released lives … were not sent for outsourced review” (p. 20), while single-hop drive and mail questions, chains and knowledge updates are all graded on such contents. The claims that records “agree by construction” (p. 2) and that HH “serves as ground truth for any question about D\mathcal{D}” (p. 4) is thus not proved for the content in the records. To assess whether the released lives are valid, the content level check is needed just like how the development set gets tested.

W6. The repair step can introduce new inconsistencies, and how often it runs is never reported. A failing output is not regenerated but sent to a repair call “that sees only the failing items”. After three attempts it falls back to a default such as a canceled occurrence (p. 6; Table 2). The paper’s argument for checking every call is that errors propagate to later windows (p. 6). That explains why each output must be checked before the next window uses it, but not why repairing is better than regenerating. A repair that sees only the failing item can pass the rule it failed while contradicting records it cannot see. For example, a rescheduled meeting may no longer match mail already written about it. Because the checks are structural (W5), such a contradiction would go undetected. Repaired and fallback outputs are also not samples from the model’s distribution restricted to valid outputs, which is how Eq. (2) frames the procedure. In addition, one big concern on the existing gepa and other prompt-optimizer is that they will always try to make things work by attaching more finer grain content to the prompt, and I see a similar concern for the repair in Stage two, which fixes a failing question by adding locating cues such as an approximate time, topic or people (p. 6): would the repair operation just try to make things work and thus invent something that seemingly true? Thus, I think the paper should report how often repairs and fallbacks occur in each life, and whether a repaired output is checked against the records already written for the same event.

W7. The answer-uniqueness guarantee rests on a comparison with five “competitor” records whose selection the paper never specifies. The paper’s main advantage over prior benchmarks is that every question matches exactly its evidence set, so its answer is unique (Abstract; p. 6). The authors state that this “cannot be checked exactly”. The decision is therefore left to an LLM check, GblindG_{\text{blind}}, which judges whether the question “singles out its evidence among the competitors” (p. 6). The competitors are defined in a single clause, “the five pool records lexically closest to qq” (p. 6), and nothing else in the paper says how they are found. Unspecified points:

  • which lexical scoring method is used;
  • how Chinese and Japanese text, used in 4 of the 11 lives, is tokenized (since those languages’ tokenization is greatly differ from language like English);
  • whether the evidence records themselves are excluded from the five;
  • how a photo, whose visible view is only an image and a timestamp, is matched lexically;
  • whether the competitors are recomputed after a repair rewrites the question.

If the evidence records are not excluded, a multi-hop question with five or more evidence records may face no real competitor at all. A lexical match also misses records that refer to the same thing in other words, although names may be translated or abbreviated across platforms (p. 14). Maybe an embedding-based (dense) retrieval check is also needed.

Questions for authors
  1. (W3) Section 3.1 says the projection step fetches the records of a referenced event “whatever its window” (p. 5), yet the event writer in window kk sees only the events of window k−1k-1. Can an event’s triggered_by link point further back than the previous window, and if so, how does the model know about the older event? How is a process with a long quiet gap generated, such as a submission whose decision arrives months later? What is the distribution of time gaps within the causal lines used for chain and knowledge-update questions?

  2. (W4) Does any attribute of the persona change during the horizon, other than through recurring series? For preference questions, how is a habit that changed during the horizon detected and excluded?

  3. (W5) What fraction of released records contain content that contradicts their source event, another record of the same event, or an earlier version of the same file?

  4. (W6) Per life, how many repair calls, fallback cancellations and fallbacks to direct file generation occurred? Is a repaired output checked against the records already written for the same event, or only against the rule it failed?

  5. (W7) How are the five competitors K(s)\mathcal{K}(s) selected: which lexical scorer and tokenization (including for Chinese and Japanese), are evidence records excluded, how are photos matched, and are competitors recomputed after repair? How often does a validator from a different model family agree with GPT-5.4’s keep or drop decisions? For sub-types whose selection rule can be evaluated on HH, such as all records of one day or one contact’s records within a month, could uniqueness be counted exactly in code instead?

  6. When Gate 1 or Gate 2 halts the pipeline, is the life regenerated, resumed from a saved prefix, or corrected by hand? How many lives were generated, halted and released, and are the 12 development lives disjoint from the 11 released lives?

2 comments

Q1. Yes, whenever the earlier event is in the writer's context. The window is the scheduling unit, not an information boundary. The event writer for window k sees the persona, the series state, window k−1 and the most recent occurrence of every series due in k, wherever it lies, and may link to any event in that context; projection and document writers then fetch the referenced event with all its records from whatever window. What never enters is an event nothing in the context refers to (§3.1, Appendix A). A long-silent process is generated only if a series or a chain of links carries it; the submission-then-decision example is not (limitations). 93% of events carry a link and 1.2% of links reach past the previous window. Gaps: median 1 day, 90th percentile 7 days, none over 35; knowledge-update states are 54 days apart at the median.

Q2. No: the persona is fixed over the horizon, change is carried by series state and events, and later windows still condition on the original persona (§3.1, limitations). A changed habit is not detected; the supporting records of a habit span 80 days at the median (154 of 165 ≥ 30 days) and 155 of 165 habits are supported over at least two thirds of the horizon. We will add a selector check that support cover the horizon up to τ.

Q3. Not measured on the released records. Table 2 is structural checks plus one LLM check; the development review (Appendix E, drive 85%) is our only content-level measurement and was not run on the released lives. The revision says so (§3.3, limitations) and corrects the Table 2 caption.

Q4. Stage one: 14 repair calls (file-name collisions, in 4 lives), 4 series-delta retries, no occurrence repair, so the cancellation fallback was never used. Stage two: 469 of 1,516 candidates dropped (31%); 83 of 1,037 questions repaired. A repair may only add a locating cue present in the evidence and re-enters all gates; the revision says validate-and-repair approximates Eq. (2). A repaired output is checked against the rule it failed and the chunk's validators, not against sibling records' content (Q3).

Q5. The five records with the highest lexical overlap: Latin words plus character bigrams for CJK text, length-normalised, evidence records excluded, same platform for single-hop, photos matched on their scene description, recomputed after every repair (Appendix B). No embedding variant was run. No validator from another family was run; given the gold evidence, Claude Opus 4.5 and Gemini 3.1 Pro score 90.5 and 91.7 against GPT-5.6's 93.5, bounding family fit at 2–3 points. Time-slice and contact-bridge selectors already count their predicate in code.

Q6. Halts were rare; the data named in the halt report was corrected by hand so generation could continue, lives were neither regenerated nor rewritten, and questions are compiled only after both gates pass. The development lives reviewed (12 for calendar and mail, 3 for drive and photos) come from different briefs and are disjoint from the 11 released lives.

Regarding the remaining weaknesses, the corresponding descriptions have been revised in the manuscript. On human validation (W1), the revision describes human review as sampled calibration at three points: outsourced review of the development-life records (99/99/85/97%), our own audit of two Chinese lives (195 compiled questions, of which 3 were dropped and 2 corrected, 2.6%), and a check of the judge (80 pairs, 14 flagged answers). The audited lives are Chinese because our annotators are native Chinese readers. Native-speaker annotation for the other languages is considerably more costly and could not be arranged within the revision period. The scores on the nine unaudited lives, the per-life table and the cross-family reader ablation over all eleven lives show the same system ordering, the same extraction gap and the same time-slice collapse (Appendices D and J). The nine unaudited lives are labelled as compiler output, and all data will be released for independent audit. On positioning (W2), the revision credits Setoka and LifeBench with provenance-first generation and states our contribution as a data-generation pipeline together with its matching question compiler. TimelineQA is now cited, and the corresponding sentence in §4.4 now refers to “conversational benchmarks”. On long-range coherence (W3), the windowed factorisation keeps every prompt bounded and every generation step checkable. A year-long outline, by contrast, would have to stay consistent with the series state at every window, which is beyond the scope of this design. The paper therefore makes no claim of coherence across non-adjacent windows. The lives contain no mail older than the horizon, the mail distributions are reported in Appendix C, and the limitations section states that they have not been compared with a real corpus.

Thank you for the detailed response. The clarifications on prior work, repairs, and competitor selection address several of my concerns. However, my main concern remains: content consistency has not been evaluated on the released records, leaving uncertainty about the reliability of the ground truth. The additional results across lives are helpful, but do not directly address this issue. The handling of changing preferences also remains unresolved, since support spread across the horizon does not rule out a preference change. I am therefore keeping my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.