acceptodds
Withdrawn from ICLR 2027

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release RealCompanion, ten real relationships with an AI companion: 27,218 messages over 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9% of probes and 2.2% of those that need memory, and at the natural rate 96% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.

Primary area:
datasets and benchmarks
Keywords:
Human-AI Conversation, Personalization, Benchmark, Long-term memory, Conversational Agents, retrieval evaluation, reasoning traces

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BmrGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper releases RealCompanion, ten real users chatting with an AI companion for up to 120 days (27,218 messages), with derived profiles, personas, chat and question benchmarks, and a traceable reasoning record for every chat label. It finds that only 3.4% of messages actually need memory (median 2,157 messages back), that nothing they tried can tell when memory is needed, and that Claude Opus 5.5, GPT 5.6-sol and Gemini 3.8 Flash rebuild personas equally well (F1 0.69-0.70) at a 31x difference in tokens.

Critical strengths
  1. Real long-term data is rare, and this fills a real gap. The probes are actual messages users sent over months, not synthetic dialogs written around a planted fact.

  2. The labels are easy to check. Every label points to message IDs that actually exist, each chat item comes with its step-by-step trace, and they even ship the rows they withdrew and the audit results.

  3. The main takeaway is useful for anyone building memory systems. Recency looks great overall (95.9%) but almost never finds the far-back message (2.2%), and "do I need memory here?" is basically a coin flip on real messages (0.530) while being easy on written questions (0.802).

  4. The controls are nicely done. The sanity checks behave as they should, and just renaming the context to "retrieved memories" bumps memory use by 10-14 points, which is a neat finding.

Critical weaknesses
  1. The persona reconstruction comparison isn't apples to apples, opus 5.5 mostly worked from a compressed summary on U09, Gemini read long chats in chunks, and each system ran in a different setup, so the "31x cost" is really about the setups, and the "70-80 extra fields" mostly comes from a ground-truth persona that only saw the first 14,000 characters (G.6).

  2. The labels are weakest exactly where the main claims lean on them. Most memory-needing items come from a sweep dated 2026-09-14, the strict reading comes from one person's final pass dated 2026-09-17 (which marked 79% as "not on screen" vs. about 14% before), stage A's reasoning is wrong on 25.8% of rows, and the question track's error rate is 28.8% in D.8 but 3.8% in the Datasheet.

  3. The math promises more than it delivers: Proposition 1 only gives a lower bound if false positives are near zero, which the strict reading shows they aren't, and Remark 1 is just wrong (when π = 1 you get γ1 directly). The 96% split also rests on only 40 items with no error bars, compares against a C2 built from the same context the gold was written from, and uses a single older generator.

  4. It feels unfinished. The memory-system runs stopped partway (Supermemory on 2 users from an older release, mem0 on part of 1 user), and plenty of numbers don't match across sections (207 vs. 216 empty slots, 3.2% vs. 3.7% census rate, a mislabeled 60.8% misfire).

  5. The paper is too long and hard to follow. At 61 pages with a 44-page appendix that repeats itself a lot, it is tiring to read, and the main text leans on so many made-up terms that you have to dig through the appendix to understand it.

Questions for authors
  1. Is the question-track error rate 28.8% or 3.8%?

  2. What happens to the persona results if all three systems read the full history in the same setup, scored against a ground truth that isn't cut off?

  3. Could a second person re-check the on-screen judgments, especially the final pass?

  4. Will the Supermemory and mem0 runs be finished on the current release for all ten users?

Reviewer_LhLMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents RealCompanion - a dataset of 10 longitudinal conversations of humans and their AI companions, consisting of over 27,000 text messages recorded over up to 120 days, supplemented with additional data such as profile, persona, chat ground truth, question set. The authors’ main contributions are the creation of the dataset and the three findings that emerge, as stated in the abstract.

Critical strengths

The authors present a valuable dataset/benchmark for evaluating language model reasoning.

Critical weaknesses

The entire article is written in a very unclear way, making it difficult for the reader to follow. Some crucial details of the benchmark construction procedure are missing (annotation protocol / models used etc.).

Questions for authors
  • The findings reported in the abstract are formulated in an unclear manner, making it difficult to discern exactly what they refer to without reading the entire paper.
  • The Introduction section needs to be rewritten, as it is written in an unclear and messy manner. Some (not all) problems with the Introduction section:
    • The opening sentence of the Introduction section (line 32, “People now spend a long time talking to AI companions “) is overly broad, informal and lacks sources.
    • The claim that understanding a person comes down to answering just two questions (lines 32-35) is oversimplification.
    • Line 36: The sentence “a claim about a person can only be checked against what that person actually said” seems questionable since claims about people can be also checked against data about them that originate from other sources than their statements.
    • Lines 38-39: “Ten real ones” - real what? “two questions on one of them” - “on” or “from”? one of what? Etc.
  • The details of the construction of the benchmark’s dataset are not clearly stated in the main text (some are partially answered in the AI Use Statement section), in particular:
    • Which model performed the anonymization of the conversations? Did humans review the results of the anonymization?
    • Which language models were used to construct the question set, the chat ground truth, and reasoning traces? What was the role of human reviewers in this process?
    • Who were the human annotators that performed the annotations/audit (if any)? What was the annotation/audit protocol?
Reviewer_tqQ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RealCompanion, a benchmark for evaluating human understanding and conversational memory in AI companions. The dataset contains 27,218 anonymized messages from ten real human-companion relationships spanning 36-120 days. From these conversations, the authors construct user profiles and personas, natural chat probes, and authored QA pairs, with labels grounded in specific conversation turns.

The paper evaluates retrieval, memory-need detection, the effect of different context conditions, and profile/persona reconstruction. A main finding is that explicit long-range memory demand is relatively rare in natural conversations, but when it occurs, the relevant information can be hundreds or thousands of turns back. The experiments also show a substantial gap between natural user turns and authored questions: memory-need detection is close to chance on natural turns but considerably easier on authored QA. Finally, the evaluated agent systems achieve similar persona reconstruction performance despite large differences in inference cost.

Critical strengths

1. Evaluation on real longitudinal conversations. RealCompanion is built from multi month human companion conversations and includes probes taken directly from natural user turns, rather than relying only on synthetic personas or authored questions. This is particularly valuable because the paper shows that authored questions make memory evaluation substantially easier than natural conversation. The dataset therefore provides a more realistic setting for evaluating conversational memory.

2. A useful analysis of how rarely memory is actually needed. The paper measures the base rate of memory demand in natural conversations and shows that only 1.3-3.4% of user turns require distant memory. This makes the decomposition in Proposition 2 important: when memory-demanding turns are rare, an overall context-ablation score can be dominated by turns that do not require memory at all. I found this to be a useful point for how conversational memory systems should be evaluated and reported.

Critical weaknesses

1. Ground truth problems affect two main results.

Persona reconstruction. Appendix G.6 shows that the persona ground truth was built from truncated inputs. For U07 to U10, only 2 to 6 of the original 18 to 20 grounded dimensions remained in the final ground truth. As a result, a system can recover a correct dimension from the full conversation and still be counted as wrong. In fact, 83 to 94% of the reported false positives on these users were dimensions that the pipeline had originally grounded. More importantly, Appendix G.6 shows that using the original grounded dimensions changes the trend with conversation length. Despite this, Sections 6 and 7 still interpret the low precision as model over attribution and conclude that “scale does not close the gap.” I do not think this conclusion is supported by the current evaluation. The main results should be recomputed using ground truth without this truncation.

QA track quality. Table 26 reports that 28.8% of the questions still have content defects after three repair stages. In contrast, Appendices H.2 and I.5 report only 3.8% after four repair passes. I could not find any description of this fourth pass. This is a large difference and directly affects how much the QA results can be trusted. The paper should clarify which number applies to the released dataset and explain how the defect rate dropped from 28.8% to 3.8%.

2. Several key comparisons change multiple things at the same time.

Context ablation. C2 and C1 differ in more than the memory given to the model. They also use different headings, message order, and context length. This matters because the paper itself shows with C3 and C3m that changing the heading alone changes memory use by 10 to 14 points. Therefore, the current C2 and C1 comparison does not tell us how much of the improvement actually comes from providing the relevant memory. A controlled comparison that keeps the heading and context length the same is needed.

Retrieval. BM25 is the only content based retriever evaluated, even though every message was rewritten by an LLM during anonymization. The paper does not measure how much this rewriting affects retrieval. This is particularly important because the paper itself shows that changing the wording of the questions to better match the cited turns increases BM25 Recall@5 from 0.356 to 0.480. This suggests that BM25 is quite sensitive to wording in this dataset. A dense or hybrid retriever is needed before concluding that natural conversational callbacks are inherently difficult to retrieve.

Cost comparison. The paper highlights a 31× difference in token cost, but the systems were run in very different ways. Claude used a multi step agent setup with file tools and 1,716 calls, while Codex used 138 single requests with the conversation history directly included. Appendix G.7 also states that these call counts are not directly comparable. The 31× difference may therefore come largely from the evaluation setup rather than the models themselves. This limitation should be made clear when the cost result is presented.

Questions for authors
  1. Which QA defect rate applies to the released dataset, 28.8% in Table 26 or 3.8% in Appendices H.2 and I.5? If 3.8% is correct, what was done in the fourth repair pass?
  2. Appendix G.6 shows that using the persona labels before truncation changes the trend with conversation length. Can you report precision, recall, and F1 using these untruncated labels? Does the conclusion that “scale does not close the gap” still hold?
  3. Can you compare C1 and C2 while keeping the heading, message order, and context length the same? The paper shows that changing the heading alone shifts memory use by 10 to 14 points, so it is unclear how much of the current difference comes from the relevant memory itself.
  4. Do the retrieval results hold with a dense or hybrid retriever? BM25 seems quite sensitive to wording in this dataset, as Recall@5 increases from 0.356 to 0.480 after the question wording is repaired. Since all messages were also rewritten during anonymization, this seems important for interpreting the retrieval results.
  5. How reliable are the referent_on_screen labels? These labels reduce the number of memory bearing probes from 404 to 250 and the estimated memory demand from 3.4% to 1.3%, but they were produced by a single reader. Can you report agreement with a second annotator on a subset?
  6. Were participants explicitly told that the public release would include their rewritten conversations and the generated psychotherapy_note summaries? The Ethics Statement also reports re-identification accuracy of 83% with stylometry and 98% with an LLM, but I could not find the adversary evaluation referenced in Appendix B.5. Could you provide its setup and results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.