acceptodds
Under review as a conference paper at ICLR 2027

Controllable Memory Evolution: Mitigating Memory Misevolution in LLM Agents via Relaxation

Abstract

LLM agents can evolve at inference time by accumulating memories from past interactions and distilling them into reusable procedures. However, recent work also highlights the risks of such evolution: contaminated memories or experiences applied beyond their original scope may steer agents toward unsafe behavior. To investigate these risks, we analyze how agents rely on memory and find that misled cases align more strongly with harmful memory than stable cases that remain correct, while retaining memory in context also impairs self-assessment and key-point coverage. Based on these findings, we propose Relaxation of Agent Memory Adoption (RAMA), a transferable memory-relaxation mechanism that controls the degree to which evolving memory influences agent behavior using the model's own memory-free behavior as a self-supervised anchor. RAMA leaves both the model parameters and memory contents unchanged while balancing memory utility against the risk of over-adoption. On MemEvoBench, RAMA brings misled response rates close to the no-memory reference across QA and workflow tasks for Qwen, Mistral, and Gemma, while improving key-point coverage. In a zero-shot transfer setting, RAMA also reduces grader-judged attack success rates on ActBench, a multi-turn tool-use agent benchmark, by percentage points for both Mistral and Qwen, with online refinement providing further reductions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Memory Misevolve

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_AWiBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on memory mis-evolution in LLM agents, where experience accumulated in memory steers the agent toward incorrect or unsafe answers. It proposes RAMA, inserts trainable latent tokens at memory-query boundaries and optimizes them so that the KL drift is scaled to a fraction δ\delta of the uncontrolled memory-induced drift. The novelty lies in formulating mitigation as self-supervised memory relaxation anchored, which requires no task labels or harmful-entry annotations.

Critical strengths

The motivation is clear as section 3 grounds the design in evidence.

The proposal sounds cheap, i.e. about 10 GPU-minutes of offline training and under 0.5% inference overhead

Critical weaknesses

Statistical support is thin, tbh. The QA track has 85 evaluation cases and the workflow track 58, RAMA is averaged over three optimization seeds with no variance reported, and baselines are run once

tthe theoretical analysis does not establish what matters in practice. Proposition 1 follows from the residual assumption via Pinsker's inequality and the reverse triangle inequality, i.e. it restates the target handed to the optimizer rather than proving that the 80-step Adam procedure reaches it, and the auxiliary condition DP(x,P)>(δd0+ϵ)/2D_P(x,P) > (\delta d_0 + \epsilon)/2 is never checked empirically. The assumptions are close to the conlcusions as in Proposition 2

baselines miss the closest related methods. Beyond CK-PLUG (Bi et al., 2025), the tested defenses are a reminder prompt. Decoding-time control of context reliance has untested alternatives, e.g. layer-contrastive decoding (Chuang et al., ICLR 2024), contrastive decoding against a weaker model distribution (Li et al., ACL 2023), and activation steering (Li et al., NeurIPS 2023), and the trade-off between parametric and contextual knowledge has been mapped before (Mallen et al., ACL 2023). Results against at least one method from these families are needed, imo.

the analysis and metrics leave room for simpler explanations. The Section 3 association between misevolution and memory alignment has no control for generic context distraction, i.e. prior work shows models chase irrelevant context and defer to context under knowledge conflict (Longpre et al., EMNLP 2021), (Wu et al., COLM 2024), so misled behavior need not reflect memory-specific over-adoption.

practical guidance and reproducibility are insufficient. Static RAMA with δ=0.25\delta=0.25 nearly fails on Gemma workflow, Δ\Delta 25.9 against 44.9 for uncontrolled memory, whereas δ=0.1\delta=0.1 works, and the paper offers no procedure for choosing δ\delta before deployment

Questions for authors

refer to the weakness

Reviewer_oHKxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how harmful or biased memory entries mislead LLM agents over time. It proposes RAMA, which inserts learnable vectors between the memory and the query to reduce memory influence relative to the model's memory-free output. It is evaluated on MemEvoBench and ActBench.

Critical strengths
  1. Memory safety in self-evolving agents is a relevant problem.
  2. The method leaves model weights and memory unchanged and adds little inference cost.
Critical weaknesses
  1. Only safety is measured, not task performance: the method does not separate harmful from useful memory. It lowers attention to the whole memory block (Section 5.5), and the best results (δ=0.1) are close to not using memory. The only utility metric on MemEvoBench is key-point coverage of the safe reference answer, which is tied to the misled rate. Task correctness, including cases the method turns from correct to wrong, is not reported.
  2. ActBench pairs each attack with a benign task, but only ASR is reported. A lower ASR may simply mean the agent does not complete the task. How Z is trained and used is unclear: the method is described in two sentences, without a diagram, input format, examples, or code. Where Z is inserted (layer, chat template, multi-step tasks in ActBench) is not specified.
  3. Z is fit on a small split of each benchmark track (23 and 18 cases). It is unclear what to train on for a new setting and how to choose δ.
  4. The claimed control is not verified: the paper does not report whether the target drift ratio is reached on test inputs. The objective covers only the first eight tokens, and results do not change with this window (Figure 4b). The effect may come from changed response openings, and no example responses are shown.
  5. The baseline comparison is incomplete: the defense proposed in MemEvoBench (ModTool) is not compared. On the workflow track, GPT Drop performs similarly to RAMA.
  6. The motivation analysis is weak: Eq. 1 is unclear. The response prefix is not specified, and a harmful memory entry is used as the query. Misled cases are selected by outcome, so their closer distance to harmful memory and lower coverage largely follow from the selection.
Questions for authors

Results vary across models and tracks. Gemma needs δ=0.1 on the workflow track (Δ above 20 at δ=0.25), while Qwen works at 0.25. Online refinement is also not always better than static (e.g., Qwen QA, R1). How can these differences be explained?

Reviewer_9SPkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory misevolution in memory-augmented LLM agents. An analysis on MemEvoBench suggests that misled cases rely more heavily on harmful memory, which also impairs self-assessment and key-point coverage. The proposed RAMA inserts learnable latent vectors between the memory and the query and, anchored to the model's memory-free output, scales the memory-induced drift over the first eight tokens to a fraction δ\delta of its original value. Across three model families, RAMA brings MemEvoBench misled rates close to the no-memory level and lowers ActBench attack success by about 38 percentage points. The manuscript also contains numerous typographical and grammatical errors, e.g., "Anwser" (Tables 1 and 2), "secnario" (Section 1), "propoase" and "implemation" (Section 4), "anlysis" and "biase" (Section 4.2), "qeuries" (Section 5.2), and an incomplete sentence at the end of the key-point coverage paragraph in Section 3.2, so careful proofreading is needed

Critical strengths
  1. The analysis in Section 3, which compares misled and stable cases in terms of distance to harmful memory, self-assessment, and key-point coverage, provides concrete empirical motivation for attributing misevolution to over-reliance on memory.

  2. The method is lightweight: it leaves model weights and memory contents unchanged, requires no labels, adds negligible inference overhead, and consistently reduces misled rates across three model families and two benchmarks.

Critical weaknesses
  1. The paper never defines memory or misevolution, and does not describe how memory is written, managed, or retrieved. In the experiments, memory is a fixed text pool to which responses are appended, and RAMA intervenes only at read time, so the claim in Section 2.1 that it applies to any type of memory is untested.

  2. The Section 5.3 baseline imitates greedy memory-free tokens, whereas RAMA matches full distributions, so the comparison does not isolate the benefit of a nonzero relative drift, especially since the main results use δ=0.1\delta = 0.1.

  3. Eq. (3) controls only the magnitude of drift over the first eight positions, not its direction or later behavior, while Proposition 2 assumes that the direction is preserved.

  4. Utility is measured by reference-answer coverage, which no-memory models already largely achieve (0.710 to 0.856), so the preservation of memory-only information is untested.

  5. ActBench results compare only with the undefended agent and report only attack success, so lower attack success may simply reflect lower task completion.

  6. Each round uses a different query, so round-to-round changes in misled rates cannot be attributed to memory accumulation alone.

  7. The training and evaluation sets are small, seed variance and ActBench episode counts are not reported, and the selection of δ\delta is unspecified despite its strong dependence on model and task.

Questions for authors

Questions that could change my assessment:

  1. How is memory written, managed, and retrieved in the MemEvoBench workflow track and in ActBench?

  2. How does RAMA compare with direct KL distillation toward the memory-free distributions (a zero-target version of Eq. (3)) using the same latent embeddings, token positions, and optimization budget?

  3. Does Assumption 1 hold empirically, and what drift is achieved beyond the first eight positions?

  4. What are the task completion and refusal rates on ActBench in Figure 3(a)?

  5. How was δ\delta chosen for Tables 1 and 2 without using evaluation results, and what is the variance across the three training seeds?

Reviewer_ewkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAMA learns five latent embeddings to reduce memory-induced output drift toward a memory-free reference while keeping the base model and memory contents fixed. Experiments on MemEvoBench and ActBench's memory-and-skill subset show lower misled and attack success rates, with some evidence that useful memory content is retained.

Critical strengths
  1. On held-out QA cases that became misled, retaining the mixed memory pool lowers recognition of improper answers from 55% to 6% for Qwen and from 59% to 13% for Gemma (Fig. 1).
  2. The same-parameter imitation baseline helps isolate the value of the relaxation objective. RAMA improves both misled rate and key-point coverage over this baseline (Fig. 2).
  3. The controller transfers from single-turn training to multi-turn tool use. Offline transfer lowers ActBench attack success by 38 percentage points for Qwen and Mistral, with further gains from online refinement (Fig. 3).
Critical weaknesses
  1. Table 6 supports utility retention, but memory-free coverage is already 0.710-0.856. It remains unclear whether RAMA preserves information that can only be learned from memory, especially when that information conflicts with the memory-free answer.
  2. The ActBench evaluation in this paper reports only attack success and compares RAMA only with no defense (Sec. 5.4). Without legitimate task success and competing defenses, the practical benefit of transfer is hard to assess.
Questions for authors
  1. On cases where benign memory corrects an incorrect memory-free answer, how much of that improvement does RAMA retain? Please separate these cases from those already answered correctly without memory.
  2. On the same ActBench subset, what are legitimate task success and attack success for RAMA, no-memory execution, and a strong applicable defense baseline from Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.