acceptodds
Withdrawn from ICLR 2027

EvoMemory: Learning Future Utility for Bounded Memory Replacement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) increasingly rely on long-term memory to support interactions that extend beyond a fixed context window. Yet memory capacity is inherently limited: as new experiences continually arrive, a system must decide not only what to store, but also what to retain, replace, or discard. Existing memory systems often rely on local signals such as recency, similarity, or standalone importance, which do not directly capture how a memory update will affect future task performance. We introduce **EvoMemory**, a framework that learns bounded memory replacement based on future utility. We formulate replacement as a sequential decision problem in which each incoming memory induces alternative successor memory states. During training, EvoMemory evaluates these counterfactual states on future examples to construct utility supervision and learns a contextual value model to score their downstream utility. At deployment time, the learned policy updates memory using only the current bank and incoming memory, without access to future examples. We evaluate EvoMemory on two complementary benchmarks, JRE-L and WikiLarge, against heuristic and random memory-management strategies. On JRE-L, EvoMemory achieves a held-out utility of 0.4719, compared with 0.4599 for five-seed random replacement and 0.4590 for the strongest deterministic heuristic. On WikiLarge, EvoMemory achieves a held-out utility of 0.8152, compared with 0.7887 for retaining the initial memory bank and 0.7726 for five-seed random replacement. Across both benchmarks, paired tests on held-out examples confirm significant improvements over all evaluated baselines after multiple-comparison correction. These results support future-utility learning as an effective principle for bounded memory replacement, where useful updates depend on their consequences for the resulting memory state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Memory, Long-Term Memory, Memory Management, Bounded Memory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A42YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies bounded memory replacement for LLM systems and proposes EvoMemory. Given a current memory bank and an incoming memory, the method enumerates KEEP and all KK single-item replacements, producing K+1K+1 candidate successor states. During training, these candidates are evaluated on the next HH examples to construct future-utility supervision for a contextual value model. At deployment, the model predicts candidate utilities without future access and performs a replacement only when its predicted advantage over KEEP exceeds a margin. The method is evaluated on JRE-L and a sequential memory stream constructed from WikiLarge.

Critical strengths

The paper addresses a practically relevant problem: replacement under a fixed memory budget. Contextually valuing complete successor memory states is also intuitively preferable to independently scoring individual memories.

Critical weaknesses

1. Related-work coverage and novelty positioning are insufficient.

The literature coverage is surprisingly limited, especially for recent learned and self-evolving memory-management methods. A particularly concerning omission is the 2025 work “Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory,” which studies continuously evolving memory over sequential task streams and even uses essentially the same name, yet is neither cited nor distinguished from this submission. Other closely related learned approaches such as Memory-R1, which learns explicit memory-management operations from downstream outcomes, are also absent. Moreover, although the manuscript itself cites A-MAC, Mem-α\alpha, and future-utility-based memory-management work, none of these stronger learned approaches is included as an experimental baseline. Consequently, the novelty claim is difficult to assess and appears substantially under-positioned.

2. The method remains fundamentally one-step greedy despite being framed as a sequential decision problem.

The future utility Vt(a)V_t(a) evaluates a single successor bank BtaB_t^a on the next HH examples, without rolling out subsequent memory-management decisions. Thus, the method is non-myopic with respect to the evaluation horizon, but remains one-step greedy with respect to the control horizon. A locally preferred successor bank is not guaranteed to be a useful precursor to the globally optimal future memory trajectory. The paper provides no analysis of this approximation, making the sequential-decision framing stronger than what the actual optimization procedure supports.

3. The experimental evidence is too weak to support the claimed generality.

The evaluation is restricted to two benchmarks, one frozen target LLM, and a single stream ordering per benchmark, which the authors themselves acknowledge as limitations. WikiLarge is not originally a memory benchmark and is converted into a synthetic stream of “reusable simplification principles”; JRE-L interactions are likewise converted into abstract memories. The construction of these memories is insufficiently specified and its robustness is not evaluated. The main baselines are also weak—KEEP, FIFO, Similarity, Random, and an LLM-based Importance heuristic—and the only learned item-level control makes no updates under its selected margin, effectively reducing to KEEP. Without stronger learned baselines, multiple stream seeds, and more natural long-term-memory settings, the experiments do not substantiate the main claim.

4. Presentation and reproducibility require substantial improvement.

The manuscript appears insufficiently polished. Figure 1 is difficult to read at normal scale, and central equations are not numbered. More importantly, although the paper states that memory-construction details are provided in the appendices, the appendices largely state that interactions or WikiLarge pairs are converted into abstract memories/reusable principles without sufficiently specifying the model, prompt, and full generation procedure. This makes an important component of the experimental pipeline difficult to reproduce.

Questions for authors
  1. Could the authors clearly articulate the technical distinction from prior Evo-Memory, Memory-R1, A-MAC, and related learned memory-management approaches, and explain the absence of competitive learned baselines?
  2. What justifies treating fixed-horizon evaluation of a single successor state as an adequate approximation to the sequential memory-management objective? Have the authors considered an oracle or planning baseline that rolls out subsequent replacement decisions?
  3. Could the authors provide the complete abstract-memory construction procedure and results across multiple memory-construction settings, stream permutations/random seeds, and stronger learned baselines? These results would materially affect my assessment.
Reviewer_ruAhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates memory replacement strategies for a fixed-capacity external memory bank paired with a frozen language model. When a new memory arrives, the proposed system, EvoMemory, evaluates whether to keep the current memory bank as-is or replace an existing item. During offline training, candidate replacement banks are evaluated on a lookahead window of future examples to compute ground-truth utility differences, which are then used to train a contextual scoring model via a joint soft-ranking and utility-gain regression objective. At test time, the scoring model evaluates candidate memory swaps without looking at future examples, executing a replacement only if the predicted performance gain exceeds a tuned threshold.

Critical strengths
  1. The paper isolates a concrete memory-management problem and provides a practical offline–online decomposition.

  2. The reported improvements are supported by several useful controls and a separate held-out evaluation.

Critical weaknesses
  1. Unclear Conceptual Novelty and Overstated Foresight

The paper needs a clearer positioning against existing memory-valuation methods like MemRL [1], Memento [2], EMR [3], etc., which already use memory critics to guide bounded replacement.

  1. Incomplete Baseline Isolation and Conservative Updating

The experiments do not fully separate eviction quality from conservative admission. Baselines like FIFO, Similarity, and Random unconditionally admit every item, whereas EvoMemory uses a tuned margin threshold and executes very few replacements (e.g., only 10 updates across 139 steps on WikiLarge). Matched-rate heuristic baselines and a context-aware action-scoring model trained on the same utility labels are needed to isolate the specific advantage of the successor-state formulation.

  1. Narrow Evaluation Scope and High Variance Risk

The empirical evaluation is quite small-scale, featuring only 26 decision points on JRE-L and 139 on WikiLarge, tested on a single stream ordering with a single frozen LLM and small bank sizes (K=2K=2 to 88). Because so few updates are accepted, final bank performance may be highly sensitive to stream order, seed initialization, or initial bank contents, none of which are quantified in the significance tests.

  1. Pipeline-Specific Utility

Since the offline utility targets depend directly on the exact downstream model, retrieval policy, and prompt structure, the learned valuation function is tightly coupled to the training setup. Without demonstrating transfer to a second memory pipeline or explicitly restricting the claimed scope, it remains unclear whether the method offers a reusable replacement module or a setup-specific solution.

[1] Zhang, S., Wang, J., Zhou, R., Liao, J., Feng, Y., Li, Z., ... & Wen, M. (2026). Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory. arXiv preprint arXiv:2601.03192.

[2] Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., ... & Wang, J. (2025). Memento: Fine-tuning llm agents without fine-tuning llms. arXiv preprint arXiv:2508.16153.

[3] Han, M., Kang, M., Jung, H., & Hwang, S. J. (2019, July). Episodic memory reader: Learning what to remember for question answering from streaming data. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 4407-4417).

Questions for authors
  1. How does EvoMemory compare conceptually to prior utility-learning approaches like MemRL and EMR's contextual whole-memory critic? To isolate the specific advantage of explicit successor-state encoding, could you compare against a context-aware action-scoring baseline that receives the complete current bank and incoming memory, trained on the same utility labels with equivalent capacity and margin tuning?

  2. During the HH-example utility evaluation in Section 3.2, is the candidate bank held completely static without modeling future replacements? How much of the performance gain from H>1H > 1 persists if supervision uses a randomly sampled subset of examples rather than a consecutive temporal window? This clarification is essential for distinguishing true temporal foresight from simple utility averaging.

  3. Is EvoMemory designed as a general training recipe that must be re-run whenever the memory setup changes, or is the learned scoring model expected to transfer directly across different systems?

Reviewer_H1HQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes EvoMemory, a learned memory replacement method that selects which memory to keep or replace based on its expected future utility. Instead of scoring memories independently, it evaluates candidate memory states using future task performance and trains a value model to guide online replacement decisions.

Critical strengths
  1. EvoMemory evaluates candidate memory states as a whole rather than scoring memories independently, allowing it to account for interactions between stored memories and make more informed replacement decisions.
  2. EvoMemory uses future task utility to supervise memory replacement decisions and moves the expensive utility evaluation offline. This design provides a principled learning signal while keeping online decision-making efficient and avoiding additional LLM calls.
Critical weaknesses
  1. The paper mainly compares EvoMemory with heuristic baselines, leaving it unclear how it performs against existing learned memory management methods such as memory-R1. Comparisons with more relevant approaches would better demonstrate its advantages.
  2. Since the value model is trained on utility signals from specific datasets, it may be vulnerable to distribution shift when applied to benchmarks with different data distributions. This could limit the generalization of the learned replacement policy, yet the paper does not evaluate EvoMemory on any out-of-distribution benchmarks, leaving this concern unaddressed.
  3. The paper does not evaluate EvoMemory on long-term conversational memory or multi-turn dialogue tasks such as LongMemEval or MemoryAgentBench, where memory utility may depend on information accumulated across interactions. Its effectiveness in these realistic settings remains unclear.
Questions for authors

How are the current memory banks BtB_t generated during offline training and validation, and how many trajectories and training seeds are used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.