- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper releases RealCompanion, ten real users chatting with an AI companion for up to 120 days (27,218 messages), with derived profiles, personas, chat and question benchmarks, and a traceable reasoning record for every chat label. It finds that only 3.4% of messages actually need memory (median 2,157 messages back), that nothing they tried can tell when memory is needed, and that Claude Opus 5.5, GPT 5.6-sol and Gemini 3.8 Flash rebuild personas equally well (F1 0.69-0.70) at a 31x difference in tokens.
Critical strengths
-
Real long-term data is rare, and this fills a real gap. The probes are actual messages users sent over months, not synthetic dialogs written around a planted fact.
-
The labels are easy to check. Every label points to message IDs that actually exist, each chat item comes with its step-by-step trace, and they even ship the rows they withdrew and the audit results.
-
The main takeaway is useful for anyone building memory systems. Recency looks great overall (95.9%) but almost never finds the far-back message (2.2%), and "do I need memory here?" is basically a coin flip on real messages (0.530) while being easy on written questions (0.802).
-
The controls are nicely done. The sanity checks behave as they should, and just renaming the context to "retrieved memories" bumps memory use by 10-14 points, which is a neat finding.
Critical weaknesses
-
The persona reconstruction comparison isn't apples to apples, opus 5.5 mostly worked from a compressed summary on U09, Gemini read long chats in chunks, and each system ran in a different setup, so the "31x cost" is really about the setups, and the "70-80 extra fields" mostly comes from a ground-truth persona that only saw the first 14,000 characters (G.6).
-
The labels are weakest exactly where the main claims lean on them. Most memory-needing items come from a sweep dated 2026-09-14, the strict reading comes from one person's final pass dated 2026-09-17 (which marked 79% as "not on screen" vs. about 14% before), stage A's reasoning is wrong on 25.8% of rows, and the question track's error rate is 28.8% in D.8 but 3.8% in the Datasheet.
-
The math promises more than it delivers: Proposition 1 only gives a lower bound if false positives are near zero, which the strict reading shows they aren't, and Remark 1 is just wrong (when π = 1 you get γ1 directly). The 96% split also rests on only 40 items with no error bars, compares against a C2 built from the same context the gold was written from, and uses a single older generator.
-
It feels unfinished. The memory-system runs stopped partway (Supermemory on 2 users from an older release, mem0 on part of 1 user), and plenty of numbers don't match across sections (207 vs. 216 empty slots, 3.2% vs. 3.7% census rate, a mislabeled 60.8% misfire).
-
The paper is too long and hard to follow. At 61 pages with a 44-page appendix that repeats itself a lot, it is tiring to read, and the main text leans on so many made-up terms that you have to dig through the appendix to understand it.
Questions for authors
-
Is the question-track error rate 28.8% or 3.8%?
-
What happens to the persona results if all three systems read the full history in the same setup, scored against a ground truth that isn't cut off?
-
Could a second person re-check the on-screen judgments, especially the final pass?
-
Will the Supermemory and mem0 runs be finished on the current release for all ten users?