acceptodds
Withdrawn from ICLR 2027

KlaraBench: Benchmarking Memory Between the Lines for AI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term personalized agents need not only to remember facts and preferences explicitly expressed by users, but also to infer individual-level behavioral representations from past behavior that can transfer to new scenarios. Recent benchmarks evaluate cross-record integration and behavioral patterns, but few directly assess whether models can transfer user representations induced from behavioral evidence to novel scenarios under strict semantic isolation. To this end, we propose , a memory benchmark grounded in the Big Five personality framework that evaluates cross-scenario behavioral trait induction and transfer. tests whether memory systems can infer stable user characteristics from behavioral evidence across multiple unrelated scenarios and apply them to novel contexts. We evaluate four base language models and four external memory systems, with Profile Induction as a profile-based aggregation reference. Under the unified evaluation protocol, the tested retrieval-based memory systems do not consistently outperform full-history input, with performance varying substantially across base models. Counterfactual analysis shows that responses generated by external memory systems exhibit weak persona-conditioned specificity. Scenario-isolation interventions reveal that model performance improves substantially when the history contains the same decision object as the test request, indicating that strict scenario isolation removes a local matching shortcut and makes the task more dependent on cross-scenario behavioral abstraction.

Primary area:
datasets and benchmarks
Keywords:
long-term memory, personalized agents, behavioral understanding, cross-scenario transfer, personality traits

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fSaHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KlaraBench tests whether an agent can infer stable user traits from past behavior and use them in new situations, which the paper calls "memory between the lines". It builds on the Big Five. Each of 80 full-factorial personas has three evaluated dimensions set high or low and two fixed at medium. A persona's history holds 47 LLM-generated behavioral dialogues and 55 irrelevant ones, and test scenarios share no decision object with the history.

There are two tasks. One is a set of 720 multiple-choice items covering 1, 2 or 3 dimensions. The other is an Open Test of 80 instances (20 openings x 4 personas), scored for per-dimension alignment by a single judge, gemini-3.5-flash-lite. The paper also adds a counterfactual persona-specificity metric. Four base models are run without history, with evidence only, with full history, with an induced profile and with the true profile. Four memory systems are tested as well (A-Mem, MemoryBank, MemPalace, LightMem).

Full history helps on both tasks, and every memory system is below full-history prompting on multiple choice. Profile Induction gets the best Open Test alignment and specificity. Swapping in one same-object dialogue raises Open Test alignment by 7.5 to 15.0 points. A human study with 51 people finds trait poles easier to recognize than dimensions. The supplement releases the multiple-choice items and persona histories.

Critical strengths
  1. The benchmark targets a capability that current memory benchmarks do not isolate, and it tests that isolation directly. In the overlap intervention, request, target and scoring stay fixed and only one history dialogue is swapped. In Table 3a, a same-object dialogue raises alignment by 7.5 to 15.0 points over a matched unrelated one, though the interval crosses zero for DeepSeek-V4-Flash. The specificity metric also behaves as it should. It is near zero with no history (-0.001) and clearly positive with the true profile (+0.246). Releasing the multiple-choice data and histories is useful too.
Critical weaknesses
  1. The 3-dimension multiple-choice items may be answerable from the options alone. As §3.2 describes, each distractor differs from the answer on exactly one dimension. On every dimension, three of the four options then carry the correct level, so the answer is the per-dimension majority among the options. Table 1 fits this reading. Under Direct Prompting, 3-dimension accuracy is higher than 2-dimension accuracy for all four models, although 3-dimension items need more traits to be judged jointly. LightMem shows the same pattern, for example 26.7 on 2-dimension versus 44.2 on 3-dimension items with Qwen3.8-27B. Without a No History or options-only result per item type, I cannot tell how much of the multiple-choice score reflects trait transfer.

  2. The Open Test rests on one LLM judge, and no agreement with humans or a second judge is reported. App. A itself notes that another judge "can shift the level substantially". Table 8 shows that under this judge the true trait labels score 12.0 points below a self-induced profile. To me this suggests the rubric rewards explicit trait display more than fit to the persona. The supplement does not include the Open Test scenarios or the rubric.

Questions for authors
  1. What do No History and an options-only baseline score on each item type, especially on 3-dimension items? Could the 3-dimension distractors be redesigned so the answer is not the center of the options?
  2. Can you report agreement between this judge and human raters, or a judge from another model family, on a subset of Open Test cells? Why does the true profile score below the induced one?
Reviewer_reABOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces KlaraBench, a benchmark for testing whether AI agents can infer users’ behavioral tendencies from past interactions and apply them to new situations. It constructs 80 synthetic personas using the Big Five personality framework, with dialogue histories containing behavioral evidence and irrelevant interactions. Evaluation includes 720 multiple-choice questions and 80 open-ended tasks, with controls to reduce overlap between historical and test scenarios. Experiments compare four language models and four memory systems, finding that external memory systems do not consistently outperform full-history input. Additional analyses examine persona specificity, scenario overlap, and human recognition of the intended personality signals.

Critical strengths

None

Critical weaknesses
  1. The benchmark’s validity is not sufficiently established. Its synthetic histories and target responses follow prescribed mappings from personality traits to behaviors, which may create a narrow behavioral distribution and favor methods that recover those mappings. The fixed composition of 47 behavioral-evidence dialogues and 55 irrelevant dialogues also lacks justification as a realistic balance of useful signals and noise. Human recognition of the intended traits does not resolve this concern, while the study connecting participants’ self-reported traits to their choices provides only weak evidence. Consequently, it remains unclear whether benchmark performance reflects an ability to understand real users.

  2. The experiments provide limited insight beyond this particular construction. Findings such as improved performance with overlapping scenarios are expected, while the advantage of profile induction may follow directly from generating the data around predefined personality profiles. The existing controls do not distinguish these effects from broader limitations of memory systems. This limits the conclusions that can be drawn from the reported comparisons and weakens the benchmark’s contribution.

Questions for authors
  1. What evidence rules out models exploiting recurring trait-expression patterns introduced by the generator? Do the main findings hold when those patterns change while the underlying behavioral tendencies remain the same?

  2. Beyond recognizing the intended traits, what evidence shows that the synthetic histories and target choices reflect real users’ behavior? The self-report study provides weak evidence; is there additional validation supporting this connection?

Reviewer_r9FrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes KlaraBench to test whether memory systems can infer a user’s personality-related tendencies from past conversations and apply them to new situations. It uses synthetic personas and evaluates both multiple-choice answers and generated responses. The main distinction is separating the concrete decisions appearing in the history from those in the test.

Critical strengths

I think studying whether an assistant can learn broader patterns about a user is useful. Remembering individual facts is not enough for good personalization. I also appreciate that the paper includes generated responses, several memory systems, and a human validation study.

Critical weaknesses
  1. The novelty is not clear enough to me. Setoka [1] already evaluates inferring personality traits from behavior across different events. LikeBench [2] studies adapting responses to hidden personality and preferences learned through conversation, and PersonaMem-v3 [3] includes responses based on inferred hidden personas. I appreciate that the paper discusses Setoka and PersonaMem-v3.

  2. I am concerned about transferring a fixed personality profile across situations. People can behave differently with friends, family, and colleagues, even when their general personality is relatively stable. Research also shows that personality-related behavior depends meaningfully on the situation [4]. The paper acknowledges that it targets stable traits rather than changes over time. Still, an assistant needs to recognize when an old tendency applies, when the current situation should override it, and when new evidence should change its understanding. I would like to see how the benchmark distinguishes these cases. Otherwise, consistently applying the generated profile could receive a good score without establishing that the assistant understands the particular person in context.

  3. The human validation does not yet convince me that the generated personalities reflect real users sufficiently well. The studies showing that people can recognize the intended signals are useful. However, recognizing what a generated example is meant to express is different from showing that it predicts how a real person would behave. The self-report study addresses this more directly, but its association with scenario choices was not statistically reliable. I would like stronger evidence using real users’ histories and later choices, or their judgments of the personalized responses.

  4. I would also like an updated evaluation with newer models such as GPT-6, Opus 5.5, and Gemini 3.8 or 4. The current evaluation uses Qwen3.8-27B, DeepSeek-V4-Flash, GPT-5.6-Luna, and Doubao-Seed-2.0-Mini. I am not assuming that newer models solve the task, but the results vary substantially with the backbone, so this would help establish whether the findings still hold. I appreciate that the paper tests external memory systems. However, those experiments also leave open how the findings extend to deployed assistants such as ChatGPT or Codex, with their own memory and interaction mechanisms.

I find the problem interesting, but I am not yet convinced that the benchmark’s contribution is sufficiently distinct or that its scores establish realistic user understanding. I would reconsider if the authors clarify the difference from prior work and provide stronger support for the behavioral assumptions.

References

[1] Zeng et al. Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data. arXiv preprint, 2026.

[2] Rahman et al. LikeBench: Evaluating Subjective Likability in LLMs for Personalization. arXiv preprint, 2025.

[3] Jiang et al. PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks. arXiv preprint, 2026.

[4] Fleeson. Situation-Based Contingencies Underlying Trait-Content Manifestation in Behavior. Journal of Personality, 2007.

Questions for authors
  1. What does separating historical and test decisions reveal that the closest existing benchmarks cannot?

  2. How does the benchmark handle cases where the current situation should override an inferred personality tendency?

  3. What evidence shows that its scores predict appropriate responses for real users?

  4. Do the main findings persist with newer models, and how far should they extend to deployed assistants?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.