acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Preserving Interpretation for Personalized LLM Agents

Abstract

Personalized LLM agents must turn past interactions into responses that fit a user's current needs. Retrieved passages preserve the original history but leave the agent to work out what they imply for the current request. Compressed memories state an implication but can drop the context that shows whose preference it is and whether the user has withdrawn it. We propose Evince (EVidence-preserving INterpretation of Conversational Excerpts), a memory framework that interprets retrieved history for the current request. A learned interpreter states the relevant preference, its subject, and its withdrawal status, and a frozen LLM agent receives this note together with the unchanged excerpts. To align training with deployment, retrieval-aligned training replaces an annotated training input with the selector's actual retrieval whenever that retrieval contains the annotated user turns, keeping every preference target unchanged. On PersonaMem-v2 with 32k and 128k histories, Evince outperforms seven baselines in overall accuracy with two LLM agents, by up to 9.46 points over the strongest, while the agent reads far fewer tokens than with the full history. Ablations show that the note improves accuracy with the retrieved evidence held fixed, that retrieval-aligned training adds further gains, and that evidence and interpretation play complementary roles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Personalization, Memory, Preference-aware retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iw78OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EVINCE, a memory framework for personalized LLM agents. EVINCE retrieves relevant excerpts and adds a learned interpretation note that states the request-relevant preference, subject, and withdrawal status while preserving the original evidence. Retrieval-aligned training exposes the interpreter to actual retrievals when they contain annotated user turns. On PersonaMem-v2, EVINCE outperforms several baselines and full-context prompting while using fewer agent input tokens.

Critical strengths

The paper identifies an important issue in personalization: raw retrieval preserves source evidence but leaves the agent to infer its implication, while compressed memories may lose context. The evidence-plus-interpretation interface is simple and appealing. The matched-evidence RAW, REF, and MIX controls are particularly useful because they isolate the value of the interpretation note while holding retrieved excerpts fixed. The evidence-budget and computational-cost analyses are also clearly presented.

Critical weaknesses

The evaluation scope is narrow: the main experiments use PersonaMem-v2 response selection, one interpreter backbone, and multiple-choice-style personalization tasks. Generalization to open-ended responses, multimodal histories, and broader agent tasks is unclear. The method also depends on supervised preference-state annotations and retrieval-aligned training, but the paper does not adequately explain how such supervision would be obtained in real systems. The literal substring containment rule highlights a mismatch between the training criterion and semantic sufficiency. Finally, outperforming full context should be interpreted cautiously because EVINCE injects a task-specific interpretation that may encode benchmark structure.

Questions for authors
  1. How does EVINCE perform on open-ended generation tasks rather than multiple-choice response selection?
  2. How sensitive are the gains to the interpreter backbone and retrieval noise?
  3. Can retrieval-aligned training use semantic entailment rather than literal substring containment?
  4. When the interpretation note is wrong but the evidence is correct, does the agent over-trust the note?
1 comment

Thank you for your valuable comments.

Q1. Open-ended generation

The three new benchmarks in the general response require the agent to generate answers instead of choosing an option.

MethodLoCoMo (F1)LongMemEval-S (acc.)MemoryAgentBench (score)
Full Context65.9669.8063.12
RAG56.8373.40—
Mem062.1845.6041.25
Rewrite43.3836.6042.31
EVINCE67.6076.4062.07

EVINCE scores highest on LoCoMo and LongMemEval-S and is within 1.05 points of Full Context on MemoryAgentBench. The general response lists all six baselines.

Multimodal histories and broader agent tasks

EVINCE and the benchmarks we use are text-only. MemoryAgentBench moves the evaluation toward broader agent settings because it tests memory in incremental multi-turn interaction. EVINCE leads every memory baseline there by at least 19.76 points.

How supervision would be obtained in real systems

The interpreter is trained once on the PersonaMem-v2 annotations of the preference, its subject, and its withdrawal status. It then generalizes without new labels. We use the same interpreter as is on LoCoMo, LongMemEval-S, and MemoryAgentBench. These benchmarks have no such annotations, and EVINCE still scores highest on LoCoMo and LongMemEval-S. Annotation is therefore a one-time training cost and not a requirement in each new domain. Deployed assistants also produce a signal for each field. Explicit forget or correction requests mark withdrawal. Speaker turns and named third parties mark the subject. LLM extraction confirmed by the user can supply the preference content. We will add this discussion to the paper.

Q3. Semantic entailment instead of literal substring containment

Yes. The coverage check is a separate component, and an NLI model or LLM judge could replace it. We chose literal containment because its errors are safe. When it accepts a retrieval, the annotated user turns appear word for word in the input. When it rejects a retrieval, the example keeps its reference input, which is still a valid training example. An entailment check would accept more than the current 65.23% of retrievals. However, it could pair a target with evidence that only partly paraphrases it.

Outperforming Full Context

The interpreter never sees answer options and is instructed never to output option letters, so the note cannot encode the answer options of the benchmark. The new benchmarks give a direct test outside PersonaMem-v2. Neither the selector nor the interpreter was trained on their data. EVINCE still beats Full Context by 1.64 points on LoCoMo and by 6.60 points on LongMemEval-S, and it is within 1.05 points on MemoryAgentBench.

Q2. Sensitivity to the interpreter backbone and to retrieval noise

The interpreter in all experiments is Qwen3.5-9B. On the agent side, notes generated once for the Qwen agent also help the DeepSeek agent without regeneration. They add 6.50 and 5.22 points over raw excerpts.

Two results address retrieval noise. With Qwen at 32k, the note adds 8.7 points over raw excerpts when the retrieval covers the annotated reference. It still adds 3.5 points when the retrieval misses it. Adding lower-ranked excerpts also helps. Going from top-2 to top-8 raises accuracy steadily, from 62.24 to 68.86 at 32k and from 48.82 to 60.40 at 128k (Figure 3). The interpreter and the agent therefore handle extra, lower-ranked blocks well.

Q4. When the note is wrong but the evidence is correct, does the agent over-trust the note?

We tested this directly. We took notes whose fields a blinded audit judged supported. We flipped either the subject or the withdrawal status and answered with and without the excerpts. A positive difference-in-differences means that keeping the excerpts protects accuracy against the wrong note (Appendix D.3).

HistoryFlipped fieldnDifference-in-differences (pp)
32kWithdrawal180+36.7
32kSubject122+4.9
128kWithdrawal128+10.2
128kSubject890.0

Keeping the excerpts protects most strongly against a wrong withdrawal status at 32k, by 36.7 points. In that setting the agent uses the evidence to override a wrong note.

Notes that contradict the evidence are also rare. A blinded audit of stratified queries checked each field of the deployed interpreter. Each field is judged to contradict the excerpts on only 1% to 6% of the audited queries (Table 14).

We hope that our responses have addressed your concerns, and we would be grateful if you would consider raising your score.

Reviewer_zER1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses LoRA to train an evidence selector and a preference interpreter that output the queries' preference. It is tested with Qwen and 32k histories. Appending the interpretation to the same retrieved excerpts raises response-selection accuracy from 60% to 66%. Retrieval-aligned training adding 1.48 percentage points over reference-trained interpretation.

Critical strengths

The matched-evidence comparison keeps the retrieved excerpts fixed. The answering model receives exactly the same excerpts, either alone or with the interpreter's note, so the accuracy difference shows what the note adds rather than a change in what was retrieved.

Critical weaknesses
  1. Retrieval-aligned training adds no more than 2.88% in the tested experiment settings. More importantly, the paper does not compare the LoRA interpreter with the same base model prompted to produce the same preference state. Without that comparison, I do not see why this needs fine-tuning rather than a skill or system prompt that asks the model to check preference content, subject, and withdrawal status.
  2. All experiments use response selection on PersonaMem-v2, and the interpreter is trained using that benchmark's official preference annotations. I am concerned that the method may mainly learn this particular task format; results on another dataset or on free-form responses would make its value for personalization more convincing.
Questions for authors
  1. Have you tried giving the same retrieved excerpts to the base model without LoRA and asking it to produce the same preference state? What accuracy gap remains between that prompted interpreter and the trained one?
  2. Could the frozen LLM agent perform this interpretation itself before selecting a response, using a system prompt or skill? I would like to see how that simpler setup compares in accuracy.
4 comments

Thank you for your valuable comments.

We first clarify the scope. The paper evaluates two answering agents, Qwen3.5-397B-A17B and DeepSeek-V4-Flash. It uses 32k and 128k histories and all 5,000 test queries in each setting. The change from 60% to 66% in your summary is the Qwen 32k setting. With the excerpts held fixed, the note adds 5.80 and 2.92 points for Qwen at 32k and 128k, and 6.50 and 5.22 points for DeepSeek.

Q1. The base model without LoRA, prompted to produce the same preference state

The revised paper includes this setting (Section 4.3.1 and Appendix D.1). The base Qwen3.5-9B receives the same excerpts, instruction, and output format as the trained interpreter. Its state is rendered and added to the excerpts in the same way.

Interface, same excerpts and agentQwen 32kQwen 128kDeepSeek 32kDeepSeek 128k
Raw excerpts60.1650.9255.1446.28
+ instruction to check attribution and withdrawal60.8451.5256.2047.74
+ note from base Qwen3.5-9B, no LoRA59.3850.5854.4045.64
+ trained free-text note64.1652.2456.9448.98
+ trained structured note (EVINCE)67.0454.8661.9452.80
Trained structured − base-model note+7.66+4.28+7.54+7.16

The base-model note falls slightly below raw excerpts in all four settings. The trained interpreter beats it by 4.28 to 7.66 points. A free-text interpreter trained on the same inputs also falls short of the structured one, by 2.62 to 5.00 points.

Q2. Could the frozen agent do this interpretation itself with a system prompt or skill?

The row "+ instruction to check attribution and withdrawal" tests exactly this. The agent receives the same excerpts and an instruction to check two things itself. It must check whether the relevant information belongs to the user and whether the user has withdrawn it. The agent writes out its reasoning before it answers, as the traces in Appendix D.5 show. This setting therefore lets a 397B model do the check in its own reasoning. The instruction changes accuracy by only 0.60 to 1.46 points. The trained note beats the instructed agent by 3.34 to 6.20 points across the four settings.

The results by preference type show what training adds (Table 7). The instruction does not help when the preference is implicit. On the Neutral, Anti-stereotypical, and Health types, the instructed agent reaches 56.41, 56.73, and 50.18. The trained note reaches 67.48, 70.29, and 69.54. The hard step is to infer which preference the excerpts imply for the current request, for example from a writing task or a described behavior. Training improves exactly this step.

W1. Contribution of retrieval-aligned training

Retrieval-aligned training beats the reference-trained interpreter in all four settings. The gain is 1.48 and 1.18 points for Qwen and 2.88 and 1.78 points for DeepSeek, and it adds no cost at inference. The main case for fine-tuning comes from Q1. Training the interpreter adds 4.28 to 7.66 points over the same model prompted with the same format.

W2. Dependence on the task format and annotations of PersonaMem-v2

We added results on three new benchmarks, shown in full in the general response.

BenchmarkEVINCEFull ContextStrongest memory baseline
LoCoMo (F1)67.6065.9662.18, Mem0
LongMemEval-S (accuracy)76.4069.8073.40, RAG
MemoryAgentBench (score)62.0763.1242.31, Rewrite

EVINCE scores highest on LoCoMo and LongMemEval-S and is within 1.05 points of Full Context on MemoryAgentBench. Within PersonaMem-v2, the interpreter is kept apart from the answer format. At inference, the selector and interpreter see only the request and the history. They never see answer options, gold answers, or annotations. The interpreter is also instructed never to output answer options or letters. The official annotations are used only to build training targets.

We hope that our responses have addressed your concerns, and we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer;

Dear Reviewer; Thank you again for taking the time to review our responses. As the rebuttal period will be ending soon, we would greatly appreciate it if you could kindly revisit our responses and let us know whether any concerns or questions remain unaddressed.

We would be more than happy to provide further clarification or additional explanations if anything is still unclear. If our responses and the additional experiments have adequately addressed your concerns, we would also sincerely appreciate your reconsidering the overall score in light of the new evidence. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Followup on Rebuttal

Dear Reviewer,

Just a friendly knock on the door. The discussion period ends soon, and we’d love to hear what you think of our response.

Following your suggestions, we added new experiments, polished the writing throughout, and answered each of your questions in detail.

If these changes address your concerns, we’d be grateful if you would consider raising your score, and if anything is still unclear, we’re happy to explain further. Thanks again for your time and helpful feedback!

Best regards,

The Authors

AuthorsOpenReview ↗

followup on rebuttal

Dear Reviewer,

Thank you again for asking whether prompting alone could explain EVINCE’s gains. With the same retrieved excerpts, the trained interpreter outperformed the base model without LoRA by 4.28–7.66 percentage points and the directly prompted answering agent by 3.34–6.20 points across both agents and both history lengths. We also evaluated EVINCE on three additional benchmarks. It outperformed both full context and the strongest memory baseline on LoCoMo and LongMemEval-S, and finished 1.05 points below full context on MemoryAgentBench.

As rebuttal period is going to end soon, we would welcome any remaining questions. If these results address your concerns, we would be grateful if you would consider raising your score.

Best regards, The authors

Reviewer_jDGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a memory framework for personalized LLM agents, which retrieves memory from the history, uses it to understand the current task, and preserves the original evidence. However, the proposed method lacks novelty and lacks solid experiments to support its performance.

Critical strengths
  1. For current personalized LLM agents, the paper proposes a memory framework that differs from direct retrieval and simple compressed memory.
  2. It proposes Retrieval-Aligned Training to address the input distribution gap between training and deployment of the interpreter.
Critical weaknesses
  1. The proposed method, EVINCE, lacks novelty and offers limited improvement in method design. One of its main components, Retrieval-Aligned Training, lacks a literature review comparing it with existing strategies, e.g., RAFT (COLM’24), RA-DIT (ICLR’24), and Chain-of-Note (EMNLP’24).
  2. The experimental results are insufficient. The method is only evaluated on a single benchmark, PersonaMem-v2. More experiments on LoCoMo, LongMemEval, etc. are needed to reveal the advantage of this method. The proposed Retrieval-Aligned Training also lacks an ablation study.
Questions for authors
  1. In Figure 1, a clearer explanation of why your method wins would help me better understand your work.
  2. Are there further comprehensive experiments on more benchmarks, such as LoCoMo, LongMemEval, or other representative personalized LLM benchmarks, to help understand the improvement? In addition, an ablation experiment is needed to isolate the contribution of the main component, Retrieval-Aligned Training.
  3. For Retrieval-Aligned Training, how do you deal with wrong retrieval samples?
  4. Providing a literature review on RAG-related methods (e.g., those mentioned above) would help understand your contribution.
6 comments
AuthorsOpenReview ↗

Response to Reviewer jDGN (1/3)

Thank you for the pointers to related work and benchmarks.

W1 / Q4. Novelty and comparison with RAFT, RA-DIT, and Chain-of-Note

Appendix E of the revision adds this comparison. It also covers Self-RAG, InstructRAG, RECOMP, and Yoran et al.

Trained modelIntermediate outputAnswering modelTraining example whose retrieval misses the evidence
RA-DITGenerator and retrieverNoneFine-tunedRetrieved chunks, answer target kept
RAFTGeneratorChain-of-thought answer quoting the oracleFine-tunedOracle removed from a fraction of examples, answer target kept
Chain-of-NoteGeneratorReading note on the relevance of each documentFine-tuned or promptedNotes mark irrelevant documents, and the model may answer "unknown"
EVINCESeparate 9B interpreterPreference, its subject, and its withdrawal statusFrozenReference input kept, so the input always supports the target

The table shows three differences.

  • Where the learning resides. RAFT, RA-DIT, and Chain-of-Note teach the answering model to read retrieved text. The benefit therefore stays in that model's weights. EVINCE keeps the agent frozen, so its memory can serve an agent that was never trained with it. In the paper, the DeepSeek agent receives the memories prepared for the Qwen agent without any change. These memories raise its accuracy by 6.50 and 5.22 points over the same raw excerpts at 32k and 128k.
  • What the intermediate output states. Chain-of-Note and Self-RAG judge whether a document is relevant. In personal memory, a relevant statement can still belong to another person or be withdrawn by the user. Our note states these two properties. With the same excerpts, the explicit subject and withdrawal cues add 1.60 and 1.30 points over a neutral preference sentence (Table 3).
  • Whether the evidence is kept. RECOMP replaces the documents with a summary. When EVINCE drops the excerpts and keeps only the state, accuracy falls by 9.68 and 6.26 points and ends below raw excerpts alone (Table 3).

Our contribution is therefore the interface itself. It adds a separately trained interpretation to retained evidence for a frozen agent, and this design makes the matched-evidence analysis in the paper possible. Retrieval-aligned training also treats missed retrievals differently from these methods, as we explain under Q3.

AuthorsOpenReview ↗

Response to Reviewer jDGN (2/3)

Q1. Why EVINCE wins in Figure 1

The user asks for peaceful outdoor activities for weekend mornings. Three earlier interactions matter. In H1, the assistant suggested tai chi followed by a slow walk. In H2, the user asked the assistant not to suggest tai chi again. In H3, the user asked for help with a newsletter draft about lingering by a fence to watch birds. The correct answer is birdwatching, and it depends on H3. Yet H3 is a writing task rather than a stated preference, and it shares few words with the request. Option D, a slow walk or quiet sitting, echoes the slow walk from H1.

MemoryWhat the agent receivesAnswer
RAGH1, the assistant's tai chi suggestionC, tai chi
Mem0A fragment of the draft without the passage about watching birdsD, slow walk or quiet sitting
Raw (our selector's excerpts, no note)Excerpts that include H3D, slow walk or quiet sitting
EVINCEThe same excerpts and a note stating the inferred user preference "Likes birdwatching in local parks"B, birdwatching

Each memory fails at a different step. RAG retrieves a passage that looks similar to the request. That passage is the assistant's suggestion, and the user later asked to drop tai chi. Mem0 builds its memory before the request exists and keeps only a fragment of the draft. Raw retrieves H3, but the passage alone reads as writing material and not as a user preference. EVINCE retrieves H3 and also states what H3 implies for this request. The reasoning of the agent cites the note directly. It rules out tai chi because the user asked to forget it, and it rejects the slow walk as a generic option that does not use the inferred preference. Raw and EVINCE receive identical excerpts. The change from D to B in this case therefore comes from the note.

Why EVINCE outperforms other memory methods in general

Each step in the case matches a design choice that the paper measures across all queries.

  • Retrieval by utility rather than similarity. RAG ranks history by similarity to the request. This favors passages that repeat the wording of the request, such as an earlier assistant suggestion. Our selector instead learns how much each block helps an agent answer correctly (Section 3.4). It can therefore rank an indirect clue like H3 highly. Without a note, the excerpts from our selector reach 60.58 and 51.76 with Qwen at 32k and 128k. RAG reaches 45.98 and 44.48.
  • Interpretation at request time rather than storage time. Rewrite, Mem0, A-Mem, and MemListener build their memory before any query arrives. They must decide what to keep without knowing the later request. A clue inside a writing task does not look like a preference when it is stored. Its relevance appears only when the user later asks for an activity. EVINCE stores the original interactions and interprets them when a request arrives. None of the five memory baselines beats Full Context in any of the four settings, while EVINCE beats it in all four (Table 1).
  • Explicit subject and withdrawal status. A relevant passage can still be the wrong evidence. In the case, H1 is the assistant's suggestion and H2 withdraws it. Raw excerpts leave the agent to work this out, and a summary may drop the context that shows it. Our note states both properties and tells the agent whether the preference may guide the response. These cues add 1.60 and 1.30 points over a neutral preference sentence. Queries whose note assigns the preference to another person improve by a net 8.1 and 8.4 points, which is more than the overall gain (Appendix D.4). Asking the agent to do this check itself adds at most 1.46 points (Table 2).
  • Retained evidence. A compact memory keeps only what it judged important when it was built, and the agent cannot recover the rest. EVINCE adds the note to the unchanged excerpts, so details that the note omits stay available. Removing the excerpts costs 9.68 and 6.26 points.

Full Context loses no information, but the agent must find a sparse clue in a long input. On knowledge queries, the full history lowers the accuracy of Qwen at 32k from 64.44 without memory to 51.94. EVINCE reaches 68.45 on the same queries (Table 1), with an average agent input of about 5.1k tokens. Tables 1 and 4 also separate the two learned parts of EVINCE. Raw differs from RAG in its learned selector and scores 14.60 and 7.28 points higher. The note then adds 5.80 and 2.92 points on identical excerpts.

AuthorsOpenReview ↗

Response to Reviewer jDGN (3/3)

W2 / Q2. Evaluation on more benchmarks

We added results on three new benchmarks, shown in full in the general response. EVINCE is best on LoCoMo, with 67.60 against 65.96 for Full Context. It is also best on LongMemEval-S, with 76.40 against 73.40 for RAG. On MemoryAgentBench, EVINCE leads every memory baseline by at least 19.76 points and is within 1.05 points of Full Context.

W2 / Q2. Ablation of retrieval-aligned training

The submission contains this ablation in Table 2 and Section 4.3.1. These are Table 4 and Section 4.3.3 in the revision. Ref and Mix share the base model, training queries, targets, and number of updates. They differ only in their training inputs. Ref trains on the annotated reference plus up to three distractors from the same history. This matches the RAFT input of an oracle document with distractors, with the oracle always present. The contrast therefore compares retrieval-aligned training with a RAFT-style input for the same interpreter.

AgentHistoryRawRefMixMix − Ref
Qwen32k60.5864.9066.38+1.48
Qwen128k51.7653.5054.68+1.18
DeepSeek32k55.2658.8861.76+2.88
DeepSeek128k46.3649.8051.58+1.78

Mix beats Ref in all four settings. It also raises the Forget accuracy of Qwen by 4.68 and 4.39 points and improves seven of nine scenarios at each length.

Q3. How do you deal with wrong retrieval samples?

During training, we never pair a retrieval that misses the annotated evidence with the target. If the retrieval does not contain every annotated user turn, the example keeps its reference input. This applies to 4,149 of the 11,934 training examples. Queries and targets stay unchanged, and we never insert the missing reference into a retrieval. This is our main difference from RAFT. RAFT removes the oracle from some examples but keeps the answer target, so that the generator learns to memorize answers. For an interpreter, a target that the input does not support would teach it to invent preferences. Covered retrievals can still contain irrelevant blocks among their four excerpts, and these serve as natural distractors.

At inference, the interface guards against wrong retrievals in two ways. First, the excerpts stay in the agent's input under a header that marks the note as fallible and the original history as authoritative. Second, if the interpreter output is invalid, the agent simply receives the raw excerpts. With Qwen at 32k, the note adds 8.7 points over raw excerpts when the retrieval covers the annotated reference. It still adds 3.5 points when the retrieval misses it.

We also tested notes that are wrong on purpose. We took notes that a blinded audit judged supported, flipped one field, and answered with and without the excerpts. A positive difference-in-differences means that keeping the excerpts protects accuracy against the wrong note (Appendix D.3).

HistoryFlipped fieldnDifference-in-differences (pp)
32kWithdrawal180+36.7
32kSubject122+4.9
128kWithdrawal128+10.2
128kSubject890.0

Keeping the excerpts protects most strongly against a wrong withdrawal status at 32k, by 36.7 points.

We hope that our responses have addressed your concerns, and we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their feedback. Reviewers zER1 and iw78 note that the matched-evidence controls isolate what the note adds. Reviewer iw78 also finds the evidence-plus-interpretation interface simple and appealing. The reviews share three concerns, and we address them below.

1. Results on three new benchmarks (jDGN, zER1, iw78)

MethodLoCoMo (F1)LongMemEval-S (acc.)MemoryAgentBench (score)
Full Context65.9669.8063.12
RAG56.8373.40—
Rewrite43.3836.6042.31
Mem062.1845.6041.25
A-Mem26.479.6041.42
MemListener39.3334.8039.14
EVINCE67.6076.4062.07

EVINCE scores highest on LoCoMo and LongMemEval-S. It is 1.64 points above Full Context on LoCoMo and 3.00 points above RAG on LongMemEval-S. On MemoryAgentBench, EVINCE leads every memory baseline by at least 19.76 points and is within 1.05 points of Full Context.

2. Prompting versus training the interpreter (zER1, jDGN)

Section 4.3.1 and Table 2 of the revised paper compare memory interfaces. All rows use the same top-4 excerpts and the same agent.

InterfaceQwen 32kQwen 128kDeepSeek 32kDeepSeek 128k
Raw excerpts60.1650.9255.1446.28
+ instruction to check attribution and withdrawal60.8451.5256.2047.74
+ note from base Qwen3.5-9B, same prompt, no LoRA59.3850.5854.4045.64
+ trained free-text note64.1652.2456.9448.98
+ trained structured note (EVINCE)67.0454.8661.9452.80

The trained note beats the base-model note by 4.28 to 7.66 points. It beats the instructed agent by 3.34 to 6.20 points. Prompting the same 9B model does not recover the gain of training, and neither does instructing the answering agent.

3. Related work and retrieval-aligned training (jDGN, zER1, iw78)

The new Appendix E covers RA-DIT, RAFT, Chain-of-Note, Self-RAG, InstructRAG, RECOMP, and Yoran et al. Most of these methods teach the answering model itself to read retrieved text. RECOMP instead trains a separate model whose summary replaces the retrieved documents. EVINCE trains a separate interpreter and keeps the agent frozen. The note from the interpreter is added after the unchanged excerpts. The note states whose preference it is and whether the user has withdrawn it. It does not judge whether a document is relevant. Training the interpreter adds 4.28 to 7.66 points over the base model with the same prompt. Retrieval-aligned training adds a further 1.18 to 2.88 points over the reference-trained interpreter in all four settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.