- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EVINCE, a memory framework for personalized LLM agents. EVINCE retrieves relevant excerpts and adds a learned interpretation note that states the request-relevant preference, subject, and withdrawal status while preserving the original evidence. Retrieval-aligned training exposes the interpreter to actual retrievals when they contain annotated user turns. On PersonaMem-v2, EVINCE outperforms several baselines and full-context prompting while using fewer agent input tokens.
Critical strengths
The paper identifies an important issue in personalization: raw retrieval preserves source evidence but leaves the agent to infer its implication, while compressed memories may lose context. The evidence-plus-interpretation interface is simple and appealing. The matched-evidence RAW, REF, and MIX controls are particularly useful because they isolate the value of the interpretation note while holding retrieved excerpts fixed. The evidence-budget and computational-cost analyses are also clearly presented.
Critical weaknesses
The evaluation scope is narrow: the main experiments use PersonaMem-v2 response selection, one interpreter backbone, and multiple-choice-style personalization tasks. Generalization to open-ended responses, multimodal histories, and broader agent tasks is unclear. The method also depends on supervised preference-state annotations and retrieval-aligned training, but the paper does not adequately explain how such supervision would be obtained in real systems. The literal substring containment rule highlights a mismatch between the training criterion and semantic sufficiency. Finally, outperforming full context should be interpreted cautiously because EVINCE injects a task-specific interpretation that may encode benchmark structure.
Questions for authors
- How does EVINCE perform on open-ended generation tasks rather than multiple-choice response selection?
- How sensitive are the gains to the interpreter backbone and retrieval noise?
- Can retrieval-aligned training use semantic entailment rather than literal substring containment?
- When the interpretation note is wrong but the evidence is correct, does the agent over-trust the note?
Thank you for your valuable comments.
Q1. Open-ended generation
The three new benchmarks in the general response require the agent to generate answers instead of choosing an option.
EVINCE scores highest on LoCoMo and LongMemEval-S and is within 1.05 points of Full Context on MemoryAgentBench. The general response lists all six baselines.
Multimodal histories and broader agent tasks
EVINCE and the benchmarks we use are text-only. MemoryAgentBench moves the evaluation toward broader agent settings because it tests memory in incremental multi-turn interaction. EVINCE leads every memory baseline there by at least 19.76 points.
How supervision would be obtained in real systems
The interpreter is trained once on the PersonaMem-v2 annotations of the preference, its subject, and its withdrawal status. It then generalizes without new labels. We use the same interpreter as is on LoCoMo, LongMemEval-S, and MemoryAgentBench. These benchmarks have no such annotations, and EVINCE still scores highest on LoCoMo and LongMemEval-S. Annotation is therefore a one-time training cost and not a requirement in each new domain. Deployed assistants also produce a signal for each field. Explicit forget or correction requests mark withdrawal. Speaker turns and named third parties mark the subject. LLM extraction confirmed by the user can supply the preference content. We will add this discussion to the paper.
Q3. Semantic entailment instead of literal substring containment
Yes. The coverage check is a separate component, and an NLI model or LLM judge could replace it. We chose literal containment because its errors are safe. When it accepts a retrieval, the annotated user turns appear word for word in the input. When it rejects a retrieval, the example keeps its reference input, which is still a valid training example. An entailment check would accept more than the current 65.23% of retrievals. However, it could pair a target with evidence that only partly paraphrases it.
Outperforming Full Context
The interpreter never sees answer options and is instructed never to output option letters, so the note cannot encode the answer options of the benchmark. The new benchmarks give a direct test outside PersonaMem-v2. Neither the selector nor the interpreter was trained on their data. EVINCE still beats Full Context by 1.64 points on LoCoMo and by 6.60 points on LongMemEval-S, and it is within 1.05 points on MemoryAgentBench.
Q2. Sensitivity to the interpreter backbone and to retrieval noise
The interpreter in all experiments is Qwen3.5-9B. On the agent side, notes generated once for the Qwen agent also help the DeepSeek agent without regeneration. They add 6.50 and 5.22 points over raw excerpts.
Two results address retrieval noise. With Qwen at 32k, the note adds 8.7 points over raw excerpts when the retrieval covers the annotated reference. It still adds 3.5 points when the retrieval misses it. Adding lower-ranked excerpts also helps. Going from top-2 to top-8 raises accuracy steadily, from 62.24 to 68.86 at 32k and from 48.82 to 60.40 at 128k (Figure 3). The interpreter and the agent therefore handle extra, lower-ranked blocks well.
Q4. When the note is wrong but the evidence is correct, does the agent over-trust the note?
We tested this directly. We took notes whose fields a blinded audit judged supported. We flipped either the subject or the withdrawal status and answered with and without the excerpts. A positive difference-in-differences means that keeping the excerpts protects accuracy against the wrong note (Appendix D.3).
Keeping the excerpts protects most strongly against a wrong withdrawal status at 32k, by 36.7 points. In that setting the agent uses the evidence to override a wrong note.
Notes that contradict the evidence are also rare. A blinded audit of stratified queries checked each field of the deployed interpreter. Each field is judged to contradict the excerpts on only 1% to 6% of the audited queries (Table 14).
We hope that our responses have addressed your concerns, and we would be grateful if you would consider raising your score.