- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
KlaraBench tests whether an agent can infer stable user traits from past behavior and use them in new situations, which the paper calls "memory between the lines". It builds on the Big Five. Each of 80 full-factorial personas has three evaluated dimensions set high or low and two fixed at medium. A persona's history holds 47 LLM-generated behavioral dialogues and 55 irrelevant ones, and test scenarios share no decision object with the history.
There are two tasks. One is a set of 720 multiple-choice items covering 1, 2 or 3 dimensions. The other is an Open Test of 80 instances (20 openings x 4 personas), scored for per-dimension alignment by a single judge, gemini-3.5-flash-lite. The paper also adds a counterfactual persona-specificity metric. Four base models are run without history, with evidence only, with full history, with an induced profile and with the true profile. Four memory systems are tested as well (A-Mem, MemoryBank, MemPalace, LightMem).
Full history helps on both tasks, and every memory system is below full-history prompting on multiple choice. Profile Induction gets the best Open Test alignment and specificity. Swapping in one same-object dialogue raises Open Test alignment by 7.5 to 15.0 points. A human study with 51 people finds trait poles easier to recognize than dimensions. The supplement releases the multiple-choice items and persona histories.
Critical strengths
- The benchmark targets a capability that current memory benchmarks do not isolate, and it tests that isolation directly. In the overlap intervention, request, target and scoring stay fixed and only one history dialogue is swapped. In Table 3a, a same-object dialogue raises alignment by 7.5 to 15.0 points over a matched unrelated one, though the interval crosses zero for DeepSeek-V4-Flash. The specificity metric also behaves as it should. It is near zero with no history (-0.001) and clearly positive with the true profile (+0.246). Releasing the multiple-choice data and histories is useful too.
Critical weaknesses
-
The 3-dimension multiple-choice items may be answerable from the options alone. As §3.2 describes, each distractor differs from the answer on exactly one dimension. On every dimension, three of the four options then carry the correct level, so the answer is the per-dimension majority among the options. Table 1 fits this reading. Under Direct Prompting, 3-dimension accuracy is higher than 2-dimension accuracy for all four models, although 3-dimension items need more traits to be judged jointly. LightMem shows the same pattern, for example 26.7 on 2-dimension versus 44.2 on 3-dimension items with Qwen3.8-27B. Without a No History or options-only result per item type, I cannot tell how much of the multiple-choice score reflects trait transfer.
-
The Open Test rests on one LLM judge, and no agreement with humans or a second judge is reported. App. A itself notes that another judge "can shift the level substantially". Table 8 shows that under this judge the true trait labels score 12.0 points below a self-induced profile. To me this suggests the rubric rewards explicit trait display more than fit to the persona. The supplement does not include the Open Test scenarios or the rubric.
Questions for authors
- What do No History and an options-only baseline score on each item type, especially on 3-dimension items? Could the 3-dimension distractors be redesigned so the answer is not the center of the options?
- Can you report agreement between this judge and human raters, or a judge from another model family, on a subset of Open Test cells? Why does the true profile score below the induced one?