acceptodds
Under review as a conference paper at ICLR 2027

Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory

Abstract

Memory plays a central role in enabling large language models (LLMs) to operate over sequential tasks by accumulating and reusing experience over time. However, existing evaluations of LLM memory mostly rely on aggregate metrics such as final hold-out accuracy or cumulative online performance. We argue that these metrics can be misleading: they collapse distinct memory behaviors into a single number and obscure critical failure modes such as forgetting and negative transfer. We introduce **SeqMem-Eval** to diagnose the behavior of sequentially evolving LLM memory. Drawing inspiration from continual learning, the framework targets a distinct test-time setting in which memory is external, prompt-mediated, and updated without changing model parameters. Beyond measuring whether the final memory state improves performance, **SeqMem-Eval** examines how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference. Specifically, it measures online utility, hold-out generalization, backward transfer, forgetting and efficiency, providing a finer-grained view of whether memory updates help current tasks, generalize to unseen tasks, improve past predictions, or degrade previously acquired knowledge. Through extensive experiments across diverse tasks and memory methods, we uncover several previously overlooked phenomena. In particular, we show that higher final or cumulative accuracy does not necessarily imply better memory quality: many methods exhibit strong performance gains while suffering from significant forgetting or negative transfers. Moreover, different memory designs exhibit distinct trade-offs between adaptability and stability, which are invisible under standard evaluation metrics. Our findings show that aggregate metrics systematically miss several recurring failure modes, suggesting that a multi-dimensional perspective is essential for understanding LLM memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Memory-Augmented Systems, Sequential Inference, Continual Learning, Retrieval-Augmented Generation, Memory Evaluation, Test-Time Adaptation, Knowledge Accumulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1KNGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEQMEM-EVAL, a diagnostic evaluation framework for LLM systems with evolving external memory. Instead of reporting only an endpoint score, it reports online utility, peak-to-end degradation, recovery, hold-out generalization, backward transfer, forgetting, and efficiency. The paper evaluates several memory systems on six task settings with Qwen3-8B and MiniMax-M2.7. The central message is useful: an endpoint score can hide unstable or non-generalizing memory behavior. However, the current evidence does not yet establish that the proposed trajectory metrics reliably measure memory-induced learning and forgetting, and the comparison set is incomplete for an ICLR 2027 submission.

Critical strengths
  1. The paper addresses an important and under-emphasized evaluation problem. Inspecting the full evolution of memory is more informative than relying on a single final accuracy, especially for systems that may improve temporarily and then degrade.

  2. The evaluation protocol is relatively broad. It covers two backbones, programming, mathematical and domain reasoning, API use, and ALFWorld, and reports both performance and cost. The appendices also include targeted analyses of task order, retrieval configuration, capacity, and paired bootstrap uncertainty.

  3. The framework is clearly described and practically usable. The proposed metrics provide a common vocabulary for discussing online utility, transfer, retention, and efficiency, and the qualitative trajectory examples make the motivation easy to understand.

Critical weaknesses
  1. The validity of the forgetting and transfer metrics is not sufficiently established. Most metrics use a single binary generation per item, with non-zero sampling temperatures (0.7 for Qwen3 and 1.0 for MiniMax). Consequently, a fixed memory state can produce different answers on repeated evaluations. In particular, the max-over-checkpoints operation in F(t) can create a positive apparent forgetting value even when the memory is never updated: an item may be correct at one stochastic evaluation and incorrect at the endpoint. The current paired bootstrap quantifies uncertainty across test items, but it does not isolate this generation-level noise. The paper should include a frozen-memory/no-update null experiment with repeated generations, or otherwise estimate and correct for decoding variance. A matched memory-free trajectory is also needed when attributing changes in PED, MER, and BWT to memory rather than to task difficulty and stream order.

  2. The baseline coverage is incomplete, and this weakens the novelty claim. The main comparisons are concentrated on methods from 2024–2025. At minimum, the authors should discuss and, where the protocol is applicable, evaluate recent 2026 work such as ACE, which performs online context adaptation through incremental generation, reflection, and curation, and Live-Evo, which maintains experience and guideline banks while reinforcing useful experiences and down-weighting stale ones. The paper cites ReasoningBank but does not include it as an experimental baseline; this omission should be explained. TAME is also relevant to the discussion of trustworthiness during test-time memory evolution, while EvoMemBench provides a 2026 standardized benchmark covering in-episode/cross-episode and knowledge/execution evolution. These works do not all have identical goals, so they need not all be reproduced, but the paper should explicitly position SEQMEM-EVAL against them. Without this update, the contribution appears closer to a useful synthesis of existing continual-learning and memory-evaluation ideas than a clearly new evaluation framework.

  3. The experimental conclusions are broader than the robustness evidence supports. The full metric grid is not tested across task orders: the order study uses only Qwen3 HumanEval and three methods. The OOD analysis is also limited to memory learned from MATH500 on one backbone. In addition, ExpeL-MT has up to three attempts per task, whereas the single-try and memory-free controls are not clearly matched for attempts and cost. This makes method-level comparisons difficult, particularly when the paper interprets accuracy and trajectory differences as memory quality. The authors should either narrow the claims or report more clearly cost-matched, retry-matched controls and uncertainty across independent decoding runs.

Questions for authors
  1. Under a frozen-memory null with repeated stochastic generations, what values do F(t), BWT(t), PED, and MER take? Would the main qualitative conclusions remain after subtracting this null variation?

  2. Can the authors provide a complete table mapping each baseline to its publication/preprint date, memory-update mechanism, number of attempts, model calls, and token budget? This would make the comparisons substantially easier to interpret.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1KNG (1/5)

W1.1/Q1 (frozen-memory null): The validity of the forgetting and transfer metrics is not sufficiently established. Most metrics use a single stochastic generation per item at a non-zero temperature, so a fixed memory state can give different answers on repeated evaluations. The max over checkpoints in F(t) can then create apparent forgetting even when the memory is never updated, and the paired bootstrap does not isolate this generation-level noise. The paper should include a frozen-memory null with repeated generations, or otherwise estimate and correct for decoding variance. Q1 asks what values F(t), BWT(t), PED, and MER take under such a null, and whether the main conclusions remain after subtracting it.

Thank you for the valuable suggestion. All methods are evaluated with the same decoding settings and the same protocol, so the evaluation itself does not favor any method. In addition, BWT and forgetting are computed only on the Qwen3-8B runs. Although these runs use a non-zero temperature, they also use a fixed sampling seed. Under this setting, answers generated for the same prompt are very similar. For example, when we ranked the HumanEval problems by difficulty, we ran the memory-free model five times with the same seed, and two of these runs differed in correctness on only 1.1% of the problems on average.

For a stricter test, we ran a frozen-memory null experiment with a different seed for each repetition. We ran it separately for every method in Figure 3, using the method's own memory. For each item, we kept this memory fixed at the state in which the item was first answered and re-generated the answer 8 to 10 times. This gives the success rate of each item when the memory never changes. Table R1 reports the resulting values as ranges over methods. Under the null, BWT(t) has an expected value of zero, because it is a paired difference between two draws with the same success rate. F(t) is positive, because it takes a maximum over checkpoints. It also grows with t, since the maximum then covers more checkpoints. PED and MER are computed from the whole online trajectory, and their frozen-memory null is the memory-free trajectory requested in W1.2. We therefore answer this part of Q1 in W1.2.

Table R1. Values of F(t) and BWT(t) under the frozen-memory null for a single generation per item, as in the paper, given as ranges over methods. F(t) is shown at the longest horizon (t = 32 on HumanEval and ALFWorld, t = 128 on MATH500).

Metric under the nullHumanEvalMATH500ALFWorld
F(t)0.02 to 0.060.08 to 0.110.19 to 0.28
BWT(t), expected value000
BWT(t), 95% range±0.03 to ±0.06±0.05 to ±0.07±0.18

Subtracting this zero null leaves BWT unchanged, and decoding noise only adds the variation shown in Table R1. For forgetting, subtracting the null lowers F(t), and the reduction is larger at long horizons. The absolute values of F(t) therefore become smaller than those reported in the paper. However, a positive F(t) under the null does not change how we use this metric. In our analysis, F(t) serves mainly to compare how much different methods forget over the same horizon, and every method is measured with the same estimator. What matters for these comparisons is whether the order of methods is preserved. After subtracting the null from the F(t) values in the paper, we compared every pair of methods at the two longest horizons of each task. All 15 pairs whose difference is significant, with a 95% item-bootstrap interval that excludes zero, keep their original order.

In summary, under a frozen memory, BWT has zero expectation and F(t) has a positive floor that grows with t. Subtracting the null leaves BWT unchanged and lowers the absolute values of F(t), but the order of methods at long horizons is preserved. In the revised paper, we will report the null values together with F(t), add these results to the appendix, and state Finding 7 more precisely. Thank you again for this valuable suggestion, which has made our analysis deeper and our conclusions more complete.

AuthorsOpenReview ↗

Response to Reviewer 1KNG (2/5)

W1.2 (memory-free trajectory): A matched memory-free trajectory is needed to attribute changes in PED, MER, and BWT to memory rather than to task difficulty and stream order.

Thank you for the helpful suggestion. We will add the matched memory-free trajectory, on the same streams in the same order, to Figure 2, and its PED and MER to Table 6. To account for decoding noise, we ran this trajectory five times with different seeds on each task. Table R2 lists their PED, MER, and BWT. PED and MER are computed from the whole online trajectory. Every method starts the stream without any experience from it, so freezing the memory for the whole trajectory corresponds to the memory-free run. Table R2 therefore also gives the values of PED and MER under the frozen-memory null asked in Q1.

Table R2. Memory-free trajectory over five runs with different seeds on each task. PED and MER are given as the mean and range over the runs, and BWT as its expected value and 95% range.

Memory-freeHumanEvalMATH500ALFWorld
PED0.154 (0.131 to 0.169)0.061 (0.051 to 0.079)0.239 (0.193 to 0.264)
MER0.003 (0.002 to 0.004)0.012 (0.007 to 0.016)0.010 (0.004 to 0.015)
BWT0 (±0.03)0 (±0.06)0 (±0.18)

These runs show that PED and MER depend on the stream itself, including its difficulty profile and task order. On HumanEval, the memory-free trajectory already has a PED of 0.154 with an MER of 0.003. On HumanEval and MATH500, the PED of every method lies within −0.06 to +0.04 of the memory-free value, and MER is at most 0.03. Appendix B.7.3 also shows that PED and MER change with the task order. We will add an explanation of this dependence to the revised paper and report PED and MER relative to the memory-free trajectory. BWT is not affected by the difficulty of the stream in this way. It compares the same item before and after t memory updates, so the difficulty of the item cancels out. Without memory, the item gets exactly the same prompt at both points, so the memory-free BWT is zero in expectation and stays within the range in Table R2. What matters most for comparing methods is that their order does not change. The memory-free value is the same for all methods on a stream, so subtracting it shifts all methods equally and leaves their order unchanged. Departures from this common reference still reveal effects of memory. On ALFWorld, the PED of AWM is 0.486, far above the range of the memory-free runs (0.193 to 0.264), and the MER of ExpeL-MT is 0.150, while the memory-free runs reach at most 0.015. Thank you again for the valuable advice, which has made our analysis of the online trajectories deeper and more complete.

AuthorsOpenReview ↗

Response to Reviewer 1KNG (3/5)

W2.1: The baseline coverage is incomplete, and the main comparisons focus on methods from 2024 to 2025. The paper should discuss and, where the protocol applies, evaluate recent work such as ACE and Live-Evo. ACE adapts its context online through incremental generation, reflection, and curation. Live-Evo keeps experience and guideline banks and down-weights stale experiences. The paper should also explain why ReasoningBank is cited but not used as a baseline.

Thank you for the valuable suggestion. The goal of SeqMem-Eval is to provide a way to evaluate memory as it evolves, rather than to rank every available memory method. For this purpose, we chose one representative method from each main family of memory designs. These are instance retrieval (ExpRecent and ExpRAG), cheatsheet curation (DC-RS), workflow induction (AWM), graph memory (G-Memory), and reflection-based insights (ExpeL). The value of the diagnostics does not depend on how recent the evaluated methods are. They apply to any memory method that runs on a sequential task stream.

ACE, ReasoningBank, and Live-Evo are important recent methods, and their core mechanisms are represented among the families we evaluate. ACE builds on Dynamic Cheatsheet, which we evaluate through DC-RS, and refines the evolving context through itemized incremental updates. ReasoningBank distills reasoning strategies from both successful and failed experiences and retrieves them by embedding similarity. Its distillation step is close to the experience extraction in ExpeL, and its retrieval step is close to ExpRAG. Live-Evo keeps an experience bank and a guideline bank, which correspond to the retrieved experiences and the distilled insights that ExpeL maintains. Its reinforcement of useful experiences and down-weighting of stale ones resemble the upvote and downvote operations that ExpeL uses to manage its insights. Each method also brings its own new designs, such as the incremental delta updates of ACE, the label-free LLM judgment of ReasoningBank, and the explicit reweighting of experiences in Live-Evo. This reweighting also makes Live-Evo a natural subject for our forgetting diagnostics, which measure whether useful experience is retained as the memory changes.

Evaluating a new method with the full protocol is costly, because the whole sequential process must be rerun on every task and backbone (see W3.1). This cost does not allow complete runs of new methods within the current budget. Since SeqMem-Eval only needs access to intermediate memory states, all three methods can be evaluated with the same protocol and code, and we regard this extension as future work. We will discuss and cite these works in the revised related work section. Thank you again for the valuable suggestion, which has made our discussion of related methods more complete.

W2.2: TAME is relevant to trustworthiness during test-time memory evolution, and EvoMemBench is a 2026 standardized benchmark covering in-episode and cross-episode evolution of knowledge and execution. The paper should explicitly position SeqMem-Eval against these works. Without this, the contribution appears closer to a useful synthesis of existing continual-learning and memory-evaluation ideas than a clearly new evaluation framework.

Thank you for the insightful comment. SeqMem-Eval is a diagnostic evaluation framework. It is neither a benchmark nor a new memory method.

TAME studies whether memory stays trustworthy as it evolves at test time. SeqMem-Eval studies whether memory stays useful as it evolves. The two views are complementary, since a reliable memory should be both trustworthy and useful. EvoMemBench, which is concurrent with our work, provides a standardized benchmark across in-episode and cross-episode settings and across knowledge and execution evolution. It summarizes performance mainly through aggregate scores. Our diagnostics can be applied to its task streams to show how the memory evolves along the way. To our knowledge, none of the works mentioned here measure backward transfer or forgetting as a function of the horizon. This kind of analysis is still largely missing in current evaluations of LLM memory, and it is what SeqMem-Eval is designed to provide.

Continual learning gives a natural starting point for these diagnostics, but the setting of LLM memory required new tools. The model stays fixed, there are no task boundaries, and memory affects predictions only through retrieval and prompting. We therefore added measures with no direct counterpart in continual learning. These include the oracle ceiling for immediate validity, the trajectory measures PED and MER, the hold-out trend with paired trajectory inference, and the checkpoint approximation that makes retrospective evaluation affordable. We hope these diagnostics can help evaluate future memory methods, including the ones the reviewer mentioned.

Reviewer_trVuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SEQMEM-EVAL to diagnose the behavior of sequentially evolving LLM memory. It measures online utility, hold-out generalization, backward transfer, forgetting, and efficiency. These metrics provide a more detailed view of whether memory updates help current tasks, generalize to unseen tasks, improve past predictions, or degrade previously acquired knowledge.

Critical strengths

This paper introduces SEQMEM-EVAL to diagnose the behavior of sequentially evolving LLM memory rather than relying on a single evaluation metric. It evaluates memory from multiple angles, including online utility, hold-out generalization, backward transfer, forgetting, and efficiency.

Critical weaknesses
  1. I do not fully agree with the claim that existing evaluations of LLM memory mostly rely on aggregate metrics. The authors do not sufficiently distinguish the research goals of prior work before treating the lack of these diagnostics as a deficiency. A better question is which existing conclusions would be materially affected by omitting these diagnostics.

  2. The metrics discussed in this paper are almost entirely standard measures from continual learning. As a result, the work reads more like an application of these metrics to evaluate LLM memory. The paper also does not provide much new understanding, which makes the overall contribution relatively low.

  3. This paper analyzes LLM memory from an empirical perspective, drawing inspiration from continual learning. However, there are several works that provide theoretical understanding of memory buffers in continual learning. I think the authors need to discuss how their approach differs from those theoretical studies.

  4. The paper "Benchmarking Continual Agent Memory for Online Learning, Transfer, and Forgetting" seems to overlap with this work. Could the authors discuss the differences between the two?

  5. A deeper analysis of BWT/forgetting could strengthen the paper’s explanatory contribution. Which subsequent experiences reliably improve performance on previously encountered tasks, and which memory components produce these gains? Under what conditions do these gains give way to interference?

Questions for authors

Please see Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer trVu (1/3)

W1: The claim that existing evaluations of LLM memory mostly rely on aggregate metrics is not fully convincing. The paper treats the lack of these diagnostics as a deficiency without sufficiently distinguishing the research goals of prior work. A better question is which existing conclusions would be materially affected by omitting these diagnostics.

Thank you for this thoughtful comment. The reviewer's question nicely articulates the core idea of our paper, and we will state this idea more explicitly in the revised paper. Our experiments answer this question from several angles. The most direct evidence is in Appendix B.4, where we compare all 292 eligible method pairs with a paired bootstrap. Among them, 197 pairs (67%) cannot be distinguished by final HoldOutAcc. However, 23 of these pairs show opposite hold-out trends, and the difference between their trends is significant. Another 4 pairs show trends of the same sign but with a significant difference in magnitude. Judged by the final score alone, these methods would look indistinguishable, although their memories evolve in different ways. The main text shows similar cases. Methods with similar OnlineAcc can differ greatly in PED and MER (Finding 2). Online gains often do not carry over to the hold-out set (Finding 3), and performance on earlier tasks is often lost after further memory updates (Finding 7). In all these cases, conclusions drawn from a single score, such as "two methods perform comparably" or "higher online accuracy means better memory", would be materially affected. Thank you again for this thoughtful comment, which has made the motivation of our paper clearer.

W2: The metrics are almost entirely standard measures from continual learning, so the work reads more like an application of these metrics to LLM memory. The paper also provides little new understanding, which makes the overall contribution relatively low.

Thank you for the valuable feedback. As LLM agents are deployed over longer task streams, their memory becomes a state that keeps changing rather than a fixed store. Whether this memory stays useful as it changes is therefore a basic question for building reliable agents. SeqMem-Eval is a diagnostic evaluation framework for this question. Its diagnostics are not tied to particular tasks or memory methods. They can be applied to any memory method on any sequential task stream, including the streams used by existing benchmarks. What they add is a view of how the memory itself evolves, such as whether its gains last, transfer to unseen tasks, and are retained.

Continual learning offers a natural starting point for this question, which is why SeqMem-Eval draws on its diagnostic ideas. The setting, however, differs in important ways. The model stays fixed, there are no task boundaries, and each instance is seen only once. The memory is text written by the LLM itself, and it affects predictions only through retrieval and prompting. Forgetting can therefore come from eviction, overwriting, or changes in what is retrieved, rather than from gradient interference. For this reason, we rebuild the diagnostics around memory states. We measure backward transfer and forgetting for each instance across temporal horizons, and we add measures with no direct counterpart in continual learning. These include the oracle ceiling for immediate validity, the trajectory measures PED and MER, and the hold-out trend with paired trajectory inference.

With these tools, SeqMem-Eval reveals several behaviors that stay hidden when memory is judged by a single score. Online gains often do not carry over to unseen tasks (Finding 3). Feedback is used inefficiently, and even the best configuration realizes only about 55% of its ceiling for immediate improvement (Finding 6). The diagnostics also help explain where forgetting comes from. In ExpeL-MT, eviction from the bounded insight pool removes all rules of an earlier subdomain, even though the retrieved experiences stay relevant. Simply enlarging the pool avoids this loss but hurts online performance (Appendix G). Observing and explaining such behaviors requires following the memory as it evolves. This kind of analysis is still largely missing in current evaluations of LLM memory, and it is what SeqMem-Eval is designed to provide. These findings could inspire future memory designs that validate updates, make better use of feedback, and protect useful knowledge from being overwritten.

AuthorsOpenReview ↗

Response to Reviewer trVu (2/3)

W3: The paper analyzes LLM memory empirically with inspiration from continual learning, but several works provide theoretical understanding of memory buffers in continual learning. The authors should discuss how their approach differs from these theoretical studies.

Thank you for the insightful suggestion. Theoretical studies of memory in continual learning analyze learners that update a parametric model over a sequence of tasks, often with a buffer that stores past samples for replay. They show, for example, that optimal continual learning requires perfect memory (Knoblauch et al., 2020) and that memory must grow linearly with the number of tasks (Chen et al., 2022). They also characterize how buffer size affects forgetting (Banayeeanzade et al., 2025) and show that replay can even increase forgetting (Mahdaviyeh et al., 2025).

Our setting differs in how memory works. In this line of theory, memory matters because it shapes how the model parameters are updated. In our setting, the parameters never change, and memory acts only by being retrieved into the context. The memory content is also different. Besides raw trajectories, it often contains insights, workflows, or cheatsheets that the LLM itself writes, rewrites, and evicts. Forgetting therefore comes from eviction, overwriting, or changes in what is retrieved, rather than from gradient interference. These mechanisms lie outside the assumptions of existing theory.

The two lines of work therefore serve different purposes, and each has its own strengths. Theory gives precise, provable insight under controlled assumptions, such as how much memory is necessary or when replay helps. SeqMem-Eval targets real LLM memory systems that do not yet fit such assumptions, and it measures how they actually behave. The two views can also inform each other. Theory shows that memory must grow with the number of tasks and that more replay does not always reduce forgetting. We observe related patterns in LLM memory. A bounded insight pool can erase earlier subdomains, and a larger pool avoids this loss but hurts online performance (Appendix G). The causes, however, lie in eviction and in how the memory is retrieved and used, rather than in gradient interference. Our diagnostics could therefore offer empirical targets for future theory on LLM memory.

W4: The paper "Benchmarking Continual Agent Memory for Online Learning, Transfer, and Forgetting" seems to overlap with this work. The authors should discuss the differences between the two.

Thank you for pointing us to this related work, and we will cite it in the revised paper. The paper introduces AgentMemoryBench (Ma et al., 2026), a benchmark that jointly evaluates system memory and personal memory across six datasets and five evaluation modes. It also proposes a new memory method, MEMs, which coordinates several memory stores. Its main questions are whether a single memory design can handle mixed task streams, and whether persistent memory can be distinguished from in-context learning baselines.

The main difference lies in the type of contribution. SeqMem-Eval is not a benchmark, and it does not propose a new memory method. It is a diagnostic evaluation framework that specifies what to measure about an evolving memory state. It can therefore be applied to any memory method on any sequential task stream, including those in AgentMemoryBench. Its goal is to understand how memory behaves over time.

This difference also shows in how retention is measured. AgentMemoryBench periodically re-tests a random subset of earlier samples and summarizes the relative drop into a single forgetting rate. SeqMem-Eval evaluates every instance before its own memory update and again after tt later updates, for a range of horizons tt. This gives backward transfer and forgetting as functions of the horizon. Our BWT is signed, so it captures both consolidation and interference, and its t=1t=1 case, immediate validity, comes with an oracle ceiling. We further follow hold-out generalization along the stream with paired trajectory inference, and trace the causes of forgetting inside the memory (Appendix G). AgentMemoryBench, in turn, covers aspects that we do not study, such as personal memory, mixed task streams, and repair under erroneous feedback. We therefore see the two works as complementary. Thank you again for pointing us to this work, which has made the positioning of our paper more complete.

AuthorsOpenReview ↗

Response to Reviewer trVu (3/3)

W5: A deeper analysis of BWT and forgetting could strengthen the explanatory contribution. Which later experiences reliably improve performance on earlier tasks, and which memory components produce these gains? Under what conditions do these gains turn into interference?

Thank you for the constructive suggestion. These questions are closely related to our case study in Appendix G, which traces how the memory of ExpeL-MT evolves on MMLU-Pro-Engineering. The case study gives concrete answers for this method. Gains mainly come from recent experiences on the same topic (see also Finding 5). While the stream covers thermodynamics, ExpeL-MT distills thermo-related rules into its insight pool, and a related hold-out question stays mostly correct. These gains come from the insight pool rather than from retrieved trajectories, since retrieval stays relevant throughout the stream while the prediction fails once the rules disappear. The gains turn into interference when the bounded pool is full and the stream shifts to other topics. New rules then overwrite the old ones, because eviction does not consider their future usefulness. The same pattern appears in the aggregate hold-out accuracy over all 96 questions. Enlarging the pool avoids this loss but makes the memory harder to use, which reveals a trade-off between retention and effective use. We hope this case study can inspire similar analyses of other memory methods.

This analysis is specific to one method. In future work, we plan to group memory methods by their design and break them down into components such as retrieval, abstraction, and eviction. This would show which components help which kinds of memory methods, and under what conditions. Such an analysis could offer practical guidance for designing LLM memory.

Reviewer_rnWsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that current evaluations of sequentially evolving memory LLMs rely too heavily on aggregate metrics, which obscure critical failure modes such as catastrophic forgetting. The authors propose a comprehensive diagnostic framework to measure online utility, generalization to retained data, backward transfer, the extent of forgetting, and efficiency. Extensive experiments across multiple benchmarks demonstrate that even when final accuracy scores are identical, the underlying memory dynamics can differ significantly, underscoring the necessity for multidimensional evaluation.

Critical strengths

The SEQMEM-EVAL framework introduced marks a major shift in the evaluation paradigm, moving from static final-score assessments toward fine-grained, trajectory-level diagnostics of LLM memory systems.

The study provides robust empirical validation, evaluating various memory mechanisms across a range of reasoning and interactive tasks within a standardized sequential protocol.

The research offers profound and practical insights into the mechanisms behind memory failure. For instance, it reveals how data-eviction mechanisms—necessitated by capacity constraints—can compromise previously useful contextual information.

Critical weaknesses
  1. Diagnosing backward transfer and forgetting requires exhaustive retrospective inference over the history of task distributions, which entails an extremely high computational cost. It limits the framework's practical scalability when handling long-horizon, sequential task streams.

  2. The experimental methodology relies primarily on static, fixed-order task sequences, which precludes a rigorous quantification of algorithmic variance and non-stationary learning dynamics induced by randomized task permutations, leaving the robustness of the memory system—in the face of varying sequence orders—insufficiently explored.

  3. Empirical evaluation is confined to prompt-mediated, non-parametric external memory mechanisms. As the study lacks analysis concerning parametric memory adaptation (such as co-evolution at the policy level or recursive reinforcement learning), its applicability to the broader paradigm of "self-evolving agents" remains limited.

Questions for authors

refer to weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer rnWs (1/2)

W1: Diagnosing BWT and forgetting requires exhaustive retrospective inference over past tasks. The computational cost is very high and limits the framework's scalability on long-horizon task streams.

Thank you for this valuable comment. We agree that BWT and forgetting add extra cost, because they need to re-evaluate past tasks under later memory states. However, SeqMem-Eval is modular, and its metrics can be used on demand. Users do not need to compute all of them in every evaluation. The online utility and efficiency metrics come directly from the online run and need no extra inference. The hold-out metrics only require evaluating a fixed hold-out set at a few checkpoints. BWT and forgetting can be added when retention and transfer matter for the application and the budget allows. Their cost can also be controlled. We only re-evaluate past tasks at a small set of horizons, and forgetting reuses the same evaluations as BWT. For long task streams, users can further use fewer horizons or evaluate only every kk-th starting task. In addition, these re-evaluations do not update the memory, so they are independent of each other and can run in parallel after the online run.

W2: The experiments mainly rely on a fixed task order. This prevents a rigorous quantification of the variance and non-stationary learning dynamics caused by random task permutations, so the robustness of memory systems to different task orders is not sufficiently explored.

Thank you for the insightful suggestion. We would like to note that the paper already includes a task-order study in Appendix B.7.3. On the Qwen3-8B/HumanEval stream, we evaluate ExpRecent (k=3k=3), ExpRAG (k=3k=3), and ExpeL-MT under five task orders. These include the original order, two random permutations, and two difficulty-ordered stress tests. Tables 9 and 10 report the mean and standard deviation across orders for online and hold-out accuracy, PED, MER, TrendHO\text{Trend}_\text{HO}, BWT, and forgetting. Several key findings hold under all five orders. ExpeL-MT achieves the highest OnlineAcc, ExpRAG has the lowest HoldOutAcc (with one tie), and ExpeL-MT is the only method with positive BWT@8. In the main experiments, all methods share the same fixed order by design. Every method thus sees exactly the same experiences in the same sequence, so the differences between methods can be attributed to their memory mechanisms rather than to different task streams.

We fully agree that robustness to task order is an important question. Task order is an intrinsic factor in sequential memory evaluation, because different orders can expose different adaptation, interference, and forgetting behaviors. It is therefore worth studying in its own right. SeqMem-Eval can be directly applied to multiple task orders to measure the variance of each metric, as Appendix B.7.3 shows. The main obstacle is cost. Each new order requires rerunning the whole sequential process, including memory updates, hold-out evaluation, and retrospective evaluation. The online steps also cannot be parallelized, because each task uses the memory produced by the previous one. Running just one additional order over all methods and tasks on a single backbone takes more than 400 hours. Under our current computational budget, we have done our best to cover this aspect with the targeted study in Appendix B.7.3. We regard a systematic study of order robustness, with more methods, tasks, and permutations, as an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer rnWs (2/2)

W3: The evaluation only covers prompt-mediated, non-parametric external memory. It lacks analysis of parametric memory adaptation, such as policy-level co-evolution or recursive reinforcement learning, which limits its applicability to the broader paradigm of self-evolving agents.

Thank you for the constructive advice. We agree that parametric adaptation, such as policy-level co-evolution and recursive reinforcement learning, is a valuable direction for self-evolving agents. We view it and prompt-based memory as two complementary lines of work, each with its own strengths. Parametric methods can internalize experience into the model itself and may support longer-term policy adaptation, but they require access to model weights and additional training. Prompt-based memory keeps the model fixed. It works with models that are only accessed through APIs, as in our MiniMax-M2.7 experiments. It adapts at test time without any training, and it stores experience as readable text that can be inspected and edited. Prompt-based memory is therefore a promising direction in its own right, and many current memory-augmented agents adopt it. Our paper deliberately focuses on this setting, as stated in the abstract and Section 2. A fixed LLM also allows a fair comparison, because all memory methods share the same backbone and differ only in how they build and use memory.

Moreover, the SeqMem-Eval metrics do not depend on the form of memory. They only require evaluating the system at intermediate states. For a parametric or hybrid system, the state can include both the model parameters and the external memory. The online, hold-out, BWT, and forgetting metrics then apply without change, so the framework itself is not limited to prompt-based memory. In practice, this requires saving intermediate checkpoints, such as LoRA adapters, so that past states can be re-evaluated. We have discussed this direction, including SkillRL (Xia et al., 2026), in the Limitations (Appendix H). Jointly evaluating memory evolution and policy-level adaptation is a natural next step of this work.

AuthorsOpenReview ↗

A kind reminder

Dear Reviewer rnWs,

We wanted to briefly follow up before the discussion period ends. We hope that our rebuttal has helped clarify the concerns raised in your review. Please let us know if any questions remain or if further evidence or explanation would be useful.

Thank you again for your time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.