acceptodds
Under review as a conference paper at ICLR 2027

CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents

Abstract

Personalized language agents keep a persistent memory so they can adapt to a user over weeks rather than turns, and that memory is also the attack surface. When an agent reads a statement that contradicts what it already believes, it has no principled way to separate a genuine change of mind from a temporary context switch, a sarcastic aside, or an instruction smuggled in through a retrieved document. Personalization methods treat every update as authentic, memory defenses reject untrusted content without modeling how real preferences evolve, and the two failure modes therefore trade against one another. We formulate the problem as a continuous-time partially observable decision process over a latent user state, and we prove that the error of any rule reading only recency and provenance is bounded below by how closely a feasible adversary can imitate the statistics of a legitimate revision, that conditioning on the interaction history is never worse and is strictly better under a stated condition, and that a bounded-cost clarifying question enlarges the achievable error region under a stated condition. CAPTURE tracks the latent state with a neural differential equation, writes hypotheses into a ledger stratified by timescale, queries the user when its belief is too uncertain to act on, and audits the causal influence of its own citations by counterfactual removal. On 480 held-out episodes from 96 users, CAPTURE reaches a 71.5% win rate against 69.3% for a baseline given identical supervision and 66.1% for the strongest heuristic, holds fixed-policy poisoning to 11.5%, and adheres to 83.5% of genuine updates. An adaptive attacker with the released weights raises poisoning to 24.7%, at which point a provenance filter is marginally more secure and considerably less adaptive, so we report both numbers rather than the favorable one. Because a system trained against our own generator could be learning that generator, we also evaluate the frozen system zero-shot on an independently built benchmark and replay the interaction histories of 40 participants, each spanning two to three weeks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Personalized LLM Agents, Preference Drift, Memory Poisoning, Persistent Memory, LLM Security, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FbYjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a real tension in personalized agents with persistent memory. Suppose a new statement contradicts a stored preference. It may be any of four things:

  • a genuine change of mind;
  • a context-specific preference;
  • benign noise, such as sarcasm or a typo;
  • a poisoning attempt, for example via a retrieved document, or text the user pastes from an attacker-controlled source ("same-origin" laundering).

Personalization methods accept every update. Injection defenses reject untrusted channels. So each fails on the case the other handles.

Theory (§2, App. A). For any rule that reads only recency and provenance, the false-update rate plus the rate of rejecting genuine updates is at least 1 minus the total-variation distance an adversary can achieve (Thm 2). Conditioning on the full history never hurts, and helps strictly under a stated condition (Thm 3). A bounded-cost clarifying question enlarges the achievable error region (Thm 4).

Method (CAPTURE). CAPTURE is a pipeline of five components:

  1. a Qwen2.5-3B + LoRA extractor that produces a structured preference hypothesis;
  2. an "authenticity gate": a latent belief evolving by a neural ODE between events, plus a GRU update, which chooses among six memory actions (retain, add, scope, revise, quarantine, clarify) and asks the user when the action entropy is high;
  3. a three-layer ledger (stable, contextual and transient preferences), each layer with a fixed decay rate, plus a quarantine state;
  4. a best-of-N selector under a fixed safety constraint;
  5. a counterfactual citation audit, which removes each cited memory, re-decodes, and checks whether the answer changes.

Evaluation. A new synthetic benchmark, D-PrefGuard, has 2,400 episodes of 12–20 turns across five tracks, with 480 test episodes from 96 users. Against eight baselines, CAPTURE reaches a 71.5% win rate, 11.5% poisoning, 83.5% adherence to genuine updates, and 8.9% false updates. A supervision-matched Transformer reaches 69.3% and 15.9%. Several further checks probe robustness:

  • an adaptive attacker raises poisoning to 24.7%, statistically tied with Provenance-only at 22.1%;
  • 240 attacks from twelve independent red-teamers;
  • zero-shot evaluation on HorizonBench;
  • offline replay of 40 participants' 2–3-week histories.
Critical strengths

**S1. An important, well-posed problem, and an unusually rigorous and honest evaluation.**Separating genuine preference change from memory poisoning is a real deployment problem, and Fig. 1 shows that the two existing families sit at opposite ends of the trade-off. The evaluation goes well beyond the norm:

  • Supervision is separated from architecture. A supervision-matched Transformer receives the same labels, extractor, clarification channel, selector and safety scorer (Q4).
  • Clarification is controlled for. A query-budget-matched StateMem shows that asking alone does not explain the margin.
  • The optimization target is kept apart from the metric. The reward model used by the selector is separate from the one used to score regret.
  • Strong attackers are tested. An adaptive attacker with the released weights is included, along with 240 attacks from twelve red-teamers, analyzed with a crossed two-way bootstrap.
  • Results are checked beyond the paper's own generator, through zero-shot HorizonBench and a participant replay.
  • Unfavorable results are reported plainly. Provenance-only ties CAPTURE under adaptive attack, CAPTURE does not lead on HorizonBench belief-revision accuracy, and falsification criteria are stated in advance.

S2. A clear empirical answer to a meaningful question. On held-out data CAPTURE improves both axes at once relative to Provenance-only: poisoning falls from 20.0% to 11.5%, adherence rises from 74.1% to 83.5%, and false updates fall from 19.6% to 8.9%. The gain is concentrated where theory predicts. On same-origin contamination, attack success is 17.5% against 43.6% for Provenance-only. On context switches, CAPTURE chooses the correct "scope" action at 71% of changepoints against 4% for StateMem (App. H). The ordering of methods holds under independently authored attacks (18.3% vs 26.3% for Provenance-only).

Critical weaknesses

W1. The architectural contribution is small relative to the system's complexity, and much of it does not hold up beyond the paper's own benchmark or under adaptive attack.

  • Supervision explains most of the gain. The supervision-matched Transformer closes most of the gap: CAPTURE leads by only +2.2 win-rate points [0.6, 3.9] and −4.4 poisoning points (Q4). On human labels the action macro-F1 is 0.71 vs 0.68, a difference that is not significant (App. E).
  • On external data the gain shrinks. Against the matched Transformer it is significant on HorizonBench contradiction resolution (+0.04) and replay win rate (+1.7, p = 0.031). It is not significant on belief-revision accuracy (p = 0.27) or replay poisoning (p = 0.064). The authors call this "a modest result."
  • The headline components contribute little.
  • Removing the neural ODE costs only +2.1 pp poisoning and +0.02 regret (Table 6).
  • A discrete gate with an explicit Δt input recovers about 55% of the continuous-time advantage (App. J).
  • The episodes have only 12–20 events, with a median gap of 8 hours, so the case for continuous-time dynamics is thin.
  • Most of the security result comes from the gate classifier (+22.0 pp) and the provenance feature (+14.0 pp).
  • **Under adaptive attack, CAPTURE is not more secure than a training-free heuristic.**Poisoning more than doubles, to 24.7%, and CAPTURE is statistically tied with Provenance-only (difference 2.6 [−1.9, 7.2], p = 0.29). The adaptive attack is also limited: surrogate-state paraphrase search with origin and timing held fixed.
  • The system is costly. It has five components (extractor, ODE gate, three-layer ledger with quarantine, best-of-N safety selector, counterfactual re-decoding audit). It adds about 2.5 s and 3,850 tokens per turn (App. B, Table 6), against 480 ms for Provenance-only.

So the paper's own evidence supports "supervised authenticity classification over history, plus quarantine and scoping," rather than the specific ODE-and-ledger architecture.

W2. The theory is standard and loosely connected to the method, and the evaluation leans on the authors' own synthetic generator.

  • The theorems are textbook results.
  • Thm 2 is the classical Neyman–Pearson / Le Cam bound for testing between two distributions.
  • Thm 3 is the data-processing inequality for total variation.
  • Thm 4 follows largely from stipulating an accurate clarification channel. The authors themselves call it "close to definitional" (Remark 5) and say that "the tools are standard statistical decision theory."

The abstract's "we prove…" therefore overstates the novelty. - The theory doesn't justify the architecture. Thm 3 says any history-conditioned rule is at least as good as recency-plus-provenance. That supports some learned gate, but not the neural ODE or the ledger. The continuous-time POMDP and the dynamic-regret objective (Eq. 1) are never solved or optimized; the gate is trained by supervised six-class cross-entropy. - The labels come from the same program that generates the test data. The gate's action labels come from the generation program and "were not validated by humans at the event level" (App. B). They agree with independent human labels only 83.6% of the time (App. E), and the human check of the benchmark covers only 300 episodes, for track and changepoint. Disjoint generator families reduce this circularity but don't remove it. - The real-user evidence is weaker than "real users" suggests. - Participants used a neutral assistant. - Attacks were injected afterwards by the authors. - When CAPTURE asked a clarifying question, the stored annotation answered with zero error (oracle-assisted).

The authors acknowledge that poisoning robustness is not shown to transfer beyond author-constructed contamination (§6). - Episodes are short. At 12–20 turns, they cannot test the "weeks rather than turns" setting the abstract motivates.

Questions for authors

Clarifications that could change my assessment:

  1. Which latency figure is correct: "roughly 630 ms per turn" (§1) or 2,480 ms added per turn (App. B, Table 6)? What is the breakdown by component, including the audit's extra decode, and the end-to-end latency relative to the backbone? Note that in Table 6 the full system (2,480 ms) minus the variant without the gate (1,850 ms) is exactly 630 ms. The introduction's figure therefore appears to be the gate's cost alone, reported as the cost of the whole method. By the same subtraction, the causal audit alone costs about 1,030 ms while changing adherence by only 0.4 points and poisoning by 0.6 points.
  2. Given that the supervision-matched Transformer recovers most of the gain and that Discrete + Δt recovers about 55% of the continuous-time advantage, what evidence specifically supports the neural ODE over a simpler gate? Please report a Transformer variant that also has the three-layer ledger, to separate the ledger's contribution from the ODE's. Please also ablate the quarantine state on its own. App. H reports that 58% of blocked poisoning attempts are resolved by quarantine followed by non-corroboration, so it may be a key mechanism, yet Table 6 never removes it.
  3. Can you strengthen the adaptive attacker? For example, let it choose injection timing and channel, which Assumption 4 permits; run multi-turn slow attacks; or attack the extractor as well as the gate. How does CAPTURE compare with Provenance-only then?
  4. How much does gate performance depend on the generation program's labels? Can you train the gate, at least partly, on the 720 human-labeled events, or report test results on human-labeled event subsets broken down by track?
  5. Can the user study include live interaction with CAPTURE, at least on a subset, so that clarifying questions are answered by real users and their acceptance of one question per twelve turns is measured?
  6. Please explain why Provenance-only's FUR + RRL matches 1 − TV̂_φ to four decimals on the matched split (Table 4). Does the result change under a different binning, or when the TV estimate and the rule's errors are computed on separate samples?
  7. The safety selector is ablated only for CAPTURE (Table 6: safety attack success rises from 8.0% to 23.0%, and adherence rises 4.0 points). Because every method shares the selector, please report Table 1 with the selector removed for all methods, to show whether CAPTURE's relative advantage on win rate, adherence, poisoning and safety depends on it. Please also report sensitivity to τ_safe, w_p/w_s and N, and an ablation of the −β·r_s penalty when the r_s ≤ τ_safe constraint is kept.
  8. Please specify the four undescribed baselines (Recency RAG, Full history, Static summary, MemGPT default): their retrieval, recency weighting and summarization rules, and how each was configured or tuned. Please also add the supervision-matched Transformer to Figure 1. In addition, can you compare against existing systems: A-MemGuard (a memory-poisoning defense), Mem0 and Zep/Graphiti (deployed memories with update and invalidation logic), and a model-level injection defense such as SecAlign, combined with provenance?
  9. The HorizonBench evaluation contains no attacks, so outside your own generator only preference tracking is validated, not poisoning robustness. Could you inject your attack families into HorizonBench (or LongMemEval's knowledge-update histories), as you did in the participant replay, so that poisoning robustness is tested on an evolving-preference benchmark you did not build?
  10. On the LLM judge:
    • Please report judge–human and human–human agreement on the same basis (e.g., judge vs each individual rater, or leave-one-rater-out), with Cohen's κ, broken down by track and by method.
    • Please run a direct human comparison of CAPTURE against the supervision-matched Transformer on a subset.
    • Please state how the response-level poisoning and adherence labels are produced, and how reliable they are.
  11. Please also define "Lag" and its unit, add intervals to every Table 1 cell, and explain why CAPTURE's safety attack success is below the unpersonalized model's. 11. How does CAPTURE perform on episodes much longer than 12–20 turns, closer to the "weeks" setting the abstract motivates?
Reviewer_HY8nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors study memory mechanisms in LLM harnesses. They recognize an ambiguity between a user change of preference and a malicious injection of memories. To address that, they develop a method that learns to distinguish benign from suspicious changes in user preferences.

Critical strengths

None

Critical weaknesses

I have read Sections 3 and 4 multiple times, but am still finding the content incomprehensible. To give a concrete example - is the authenticity gate a model, and if so, what kind? A GRU is mentioned in passing, without any details. Does the gate receive any input besides hth_t? How is the rest of the gate implemented? Does it use an LLM, given that it receives the hypothesis? What does each action actually do? The actions are named, but their operations are not explained.

These questions arise from just one paragraph, but similar misunderstandings affect the presentation of the paper’s core contribution. I recommend that the authors revise their writing substantially and provide enough detail for readers to understand and properly evaluate their work.

Questions for authors

None

Reviewer_MyQjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how a personalized LLM agent should handle a new statement that contradicts its stored beliefs about the user, which may be a genuine change of preference, a temporary context switch, noise, or a poisoning attack. The authors formulate memory updates as a partially observable decision problem. They prove that any rule that reads only recency and provenance has an error bounded below by a total-variation term, and that conditioning on the full history or asking a clarifying question can do better. CAPTURE combines an LLM-based hypothesis extractor, a neural-ODE authenticity gate, a three-layer timescale ledger, a safety-bounded response selector, and a counterfactual citation audit. The authors evaluate it on D-PrefGuard, a new LLM-generated benchmark, together with a supervision-matched Transformer baseline, adaptive and red-team attacks, zero-shot evaluation on HorizonBench, and an offline replay of histories from 40 participants.

Critical strengths
  1. The problem is well motivated. Personalization methods tend to accept every update, while injection defenses tend to reject updates that do not come directly from the user. The paper frames this as a single trade-off. The same-origin analysis connects the theory to the experiments: Provenance-only reaches 43.6% attack success on same-origin contamination, compared with 17.5% for CAPTURE.
  2. The evaluation is broad and reports unfavorable results openly. It includes a baseline trained on the same supervision, an adaptive attacker, attacks written by external red-teamers, zero-shot evaluation on HorizonBench, and a replay study with real users. The authors also report that CAPTURE no longer outperforms a provenance filter under adaptive attack.
Critical weaknesses
  1. The technical contribution is limited relative to the system’s complexity. CAPTURE improves over the supervision-matched Transformer, but the gains are modest, and simpler temporal modeling recovers much of the neural ODE’s advantage. The experiments do not clearly establish that the additional components and computational cost are justified. The theory mainly applies standard statistical results and offers limited guidance on why this particular architecture is needed.
  2. The security advantage weakens under adaptive attack, and external validation remains limited. CAPTURE reaches 24.7% attack success against 22.1% for Provenance-only, with no significant difference. Gate labels and test episodes come from the same generation program. HorizonBench contains no attacks, while the replay study uses author-constructed attacks and stored annotations to answer clarifying questions. Evidence for poisoning robustness beyond the authors’ generator therefore rests on 240 independently authored red-team attacks.
  3. Several reports are inconsistent. The introduction states about 630 ms per turn, whereas Table 6 and Appendix B report 2,480 ms added to the backbone. Table 2 says independent-attack intervals await crossed resampling, but Appendix E gives [14.0, 23.1] for the same attacks. Appendix H says StateMem is absent from the relevant results in Appendix I, but Table 7 includes it.
Questions for authors
  1. Please explain the inconsistencies in the reported results. Which latency is correct, 630 ms or 2,480 ms per turn, and what is the 630 ms figure measured against? Were the interval for independently authored attacks in Appendix E and the StateMem results in Table 7 computed after the main text was written, or do they come from different runs?
  2. What does the architecture contribute beyond the supervision? Could a simpler system, such as the supervised Transformer combined with timescale layering, match CAPTURE at similar latency? An ablation that adds the ledger to the Transformer baseline, and a comparison at matched compute, would help isolate the contribution of the neural ODE.
  3. How robust is the gate outside the authors' own generator?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.