- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SEQMEM-EVAL, a diagnostic evaluation framework for LLM systems with evolving external memory. Instead of reporting only an endpoint score, it reports online utility, peak-to-end degradation, recovery, hold-out generalization, backward transfer, forgetting, and efficiency. The paper evaluates several memory systems on six task settings with Qwen3-8B and MiniMax-M2.7. The central message is useful: an endpoint score can hide unstable or non-generalizing memory behavior. However, the current evidence does not yet establish that the proposed trajectory metrics reliably measure memory-induced learning and forgetting, and the comparison set is incomplete for an ICLR 2027 submission.
Critical strengths
-
The paper addresses an important and under-emphasized evaluation problem. Inspecting the full evolution of memory is more informative than relying on a single final accuracy, especially for systems that may improve temporarily and then degrade.
-
The evaluation protocol is relatively broad. It covers two backbones, programming, mathematical and domain reasoning, API use, and ALFWorld, and reports both performance and cost. The appendices also include targeted analyses of task order, retrieval configuration, capacity, and paired bootstrap uncertainty.
-
The framework is clearly described and practically usable. The proposed metrics provide a common vocabulary for discussing online utility, transfer, retention, and efficiency, and the qualitative trajectory examples make the motivation easy to understand.
Critical weaknesses
-
The validity of the forgetting and transfer metrics is not sufficiently established. Most metrics use a single binary generation per item, with non-zero sampling temperatures (0.7 for Qwen3 and 1.0 for MiniMax). Consequently, a fixed memory state can produce different answers on repeated evaluations. In particular, the max-over-checkpoints operation in
F(t)can create a positive apparent forgetting value even when the memory is never updated: an item may be correct at one stochastic evaluation and incorrect at the endpoint. The current paired bootstrap quantifies uncertainty across test items, but it does not isolate this generation-level noise. The paper should include a frozen-memory/no-update null experiment with repeated generations, or otherwise estimate and correct for decoding variance. A matched memory-free trajectory is also needed when attributing changes inPED,MER, andBWTto memory rather than to task difficulty and stream order. -
The baseline coverage is incomplete, and this weakens the novelty claim. The main comparisons are concentrated on methods from 2024–2025. At minimum, the authors should discuss and, where the protocol is applicable, evaluate recent 2026 work such as ACE, which performs online context adaptation through incremental generation, reflection, and curation, and Live-Evo, which maintains experience and guideline banks while reinforcing useful experiences and down-weighting stale ones. The paper cites ReasoningBank but does not include it as an experimental baseline; this omission should be explained. TAME is also relevant to the discussion of trustworthiness during test-time memory evolution, while EvoMemBench provides a 2026 standardized benchmark covering in-episode/cross-episode and knowledge/execution evolution. These works do not all have identical goals, so they need not all be reproduced, but the paper should explicitly position SEQMEM-EVAL against them. Without this update, the contribution appears closer to a useful synthesis of existing continual-learning and memory-evaluation ideas than a clearly new evaluation framework.
-
The experimental conclusions are broader than the robustness evidence supports. The full metric grid is not tested across task orders: the order study uses only Qwen3 HumanEval and three methods. The OOD analysis is also limited to memory learned from MATH500 on one backbone. In addition, ExpeL-MT has up to three attempts per task, whereas the single-try and memory-free controls are not clearly matched for attempts and cost. This makes method-level comparisons difficult, particularly when the paper interprets accuracy and trajectory differences as memory quality. The authors should either narrow the claims or report more clearly cost-matched, retry-matched controls and uncertainty across independent decoding runs.
Questions for authors
-
Under a frozen-memory null with repeated stochastic generations, what values do
F(t),BWT(t),PED, andMERtake? Would the main qualitative conclusions remain after subtracting this null variation? -
Can the authors provide a complete table mapping each baseline to its publication/preprint date, memory-update mechanism, number of attempts, model calls, and token budget? This would make the comparisons substantially easier to interpret.
8 comments
Response to Reviewer 1KNG (2/5)
W1.2 (memory-free trajectory): A matched memory-free trajectory is needed to attribute changes in PED, MER, and BWT to memory rather than to task difficulty and stream order.
Thank you for the helpful suggestion. We will add the matched memory-free trajectory, on the same streams in the same order, to Figure 2, and its PED and MER to Table 6. To account for decoding noise, we ran this trajectory five times with different seeds on each task. Table R2 lists their PED, MER, and BWT. PED and MER are computed from the whole online trajectory. Every method starts the stream without any experience from it, so freezing the memory for the whole trajectory corresponds to the memory-free run. Table R2 therefore also gives the values of PED and MER under the frozen-memory null asked in Q1.
Table R2. Memory-free trajectory over five runs with different seeds on each task. PED and MER are given as the mean and range over the runs, and BWT as its expected value and 95% range.
| Memory-free | HumanEval | MATH500 | ALFWorld |
|---|---|---|---|
| PED | 0.154 (0.131 to 0.169) | 0.061 (0.051 to 0.079) | 0.239 (0.193 to 0.264) |
| MER | 0.003 (0.002 to 0.004) | 0.012 (0.007 to 0.016) | 0.010 (0.004 to 0.015) |
| BWT | 0 (±0.03) | 0 (±0.06) | 0 (±0.18) |
These runs show that PED and MER depend on the stream itself, including its difficulty profile and task order. On HumanEval, the memory-free trajectory already has a PED of 0.154 with an MER of 0.003. On HumanEval and MATH500, the PED of every method lies within −0.06 to +0.04 of the memory-free value, and MER is at most 0.03. Appendix B.7.3 also shows that PED and MER change with the task order. We will add an explanation of this dependence to the revised paper and report PED and MER relative to the memory-free trajectory. BWT is not affected by the difficulty of the stream in this way. It compares the same item before and after t memory updates, so the difficulty of the item cancels out. Without memory, the item gets exactly the same prompt at both points, so the memory-free BWT is zero in expectation and stays within the range in Table R2. What matters most for comparing methods is that their order does not change. The memory-free value is the same for all methods on a stream, so subtracting it shifts all methods equally and leaves their order unchanged. Departures from this common reference still reveal effects of memory. On ALFWorld, the PED of AWM is 0.486, far above the range of the memory-free runs (0.193 to 0.264), and the MER of ExpeL-MT is 0.150, while the memory-free runs reach at most 0.015. Thank you again for the valuable advice, which has made our analysis of the online trajectories deeper and more complete.
Response to Reviewer 1KNG (3/5)
W2.1: The baseline coverage is incomplete, and the main comparisons focus on methods from 2024 to 2025. The paper should discuss and, where the protocol applies, evaluate recent work such as ACE and Live-Evo. ACE adapts its context online through incremental generation, reflection, and curation. Live-Evo keeps experience and guideline banks and down-weights stale experiences. The paper should also explain why ReasoningBank is cited but not used as a baseline.
Thank you for the valuable suggestion. The goal of SeqMem-Eval is to provide a way to evaluate memory as it evolves, rather than to rank every available memory method. For this purpose, we chose one representative method from each main family of memory designs. These are instance retrieval (ExpRecent and ExpRAG), cheatsheet curation (DC-RS), workflow induction (AWM), graph memory (G-Memory), and reflection-based insights (ExpeL). The value of the diagnostics does not depend on how recent the evaluated methods are. They apply to any memory method that runs on a sequential task stream.
ACE, ReasoningBank, and Live-Evo are important recent methods, and their core mechanisms are represented among the families we evaluate. ACE builds on Dynamic Cheatsheet, which we evaluate through DC-RS, and refines the evolving context through itemized incremental updates. ReasoningBank distills reasoning strategies from both successful and failed experiences and retrieves them by embedding similarity. Its distillation step is close to the experience extraction in ExpeL, and its retrieval step is close to ExpRAG. Live-Evo keeps an experience bank and a guideline bank, which correspond to the retrieved experiences and the distilled insights that ExpeL maintains. Its reinforcement of useful experiences and down-weighting of stale ones resemble the upvote and downvote operations that ExpeL uses to manage its insights. Each method also brings its own new designs, such as the incremental delta updates of ACE, the label-free LLM judgment of ReasoningBank, and the explicit reweighting of experiences in Live-Evo. This reweighting also makes Live-Evo a natural subject for our forgetting diagnostics, which measure whether useful experience is retained as the memory changes.
Evaluating a new method with the full protocol is costly, because the whole sequential process must be rerun on every task and backbone (see W3.1). This cost does not allow complete runs of new methods within the current budget. Since SeqMem-Eval only needs access to intermediate memory states, all three methods can be evaluated with the same protocol and code, and we regard this extension as future work. We will discuss and cite these works in the revised related work section. Thank you again for the valuable suggestion, which has made our discussion of related methods more complete.
W2.2: TAME is relevant to trustworthiness during test-time memory evolution, and EvoMemBench is a 2026 standardized benchmark covering in-episode and cross-episode evolution of knowledge and execution. The paper should explicitly position SeqMem-Eval against these works. Without this, the contribution appears closer to a useful synthesis of existing continual-learning and memory-evaluation ideas than a clearly new evaluation framework.
Thank you for the insightful comment. SeqMem-Eval is a diagnostic evaluation framework. It is neither a benchmark nor a new memory method.
TAME studies whether memory stays trustworthy as it evolves at test time. SeqMem-Eval studies whether memory stays useful as it evolves. The two views are complementary, since a reliable memory should be both trustworthy and useful. EvoMemBench, which is concurrent with our work, provides a standardized benchmark across in-episode and cross-episode settings and across knowledge and execution evolution. It summarizes performance mainly through aggregate scores. Our diagnostics can be applied to its task streams to show how the memory evolves along the way. To our knowledge, none of the works mentioned here measure backward transfer or forgetting as a function of the horizon. This kind of analysis is still largely missing in current evaluations of LLM memory, and it is what SeqMem-Eval is designed to provide.
Continual learning gives a natural starting point for these diagnostics, but the setting of LLM memory required new tools. The model stays fixed, there are no task boundaries, and memory affects predictions only through retrieval and prompting. We therefore added measures with no direct counterpart in continual learning. These include the oracle ceiling for immediate validity, the trajectory measures PED and MER, the hold-out trend with paired trajectory inference, and the checkpoint approximation that makes retrospective evaluation affordable. We hope these diagnostics can help evaluate future memory methods, including the ones the reviewer mentioned.
Response to Reviewer 1KNG (1/5)
W1.1/Q1 (frozen-memory null): The validity of the forgetting and transfer metrics is not sufficiently established. Most metrics use a single stochastic generation per item at a non-zero temperature, so a fixed memory state can give different answers on repeated evaluations. The max over checkpoints in F(t) can then create apparent forgetting even when the memory is never updated, and the paired bootstrap does not isolate this generation-level noise. The paper should include a frozen-memory null with repeated generations, or otherwise estimate and correct for decoding variance. Q1 asks what values F(t), BWT(t), PED, and MER take under such a null, and whether the main conclusions remain after subtracting it.
Thank you for the valuable suggestion. All methods are evaluated with the same decoding settings and the same protocol, so the evaluation itself does not favor any method. In addition, BWT and forgetting are computed only on the Qwen3-8B runs. Although these runs use a non-zero temperature, they also use a fixed sampling seed. Under this setting, answers generated for the same prompt are very similar. For example, when we ranked the HumanEval problems by difficulty, we ran the memory-free model five times with the same seed, and two of these runs differed in correctness on only 1.1% of the problems on average.
For a stricter test, we ran a frozen-memory null experiment with a different seed for each repetition. We ran it separately for every method in Figure 3, using the method's own memory. For each item, we kept this memory fixed at the state in which the item was first answered and re-generated the answer 8 to 10 times. This gives the success rate of each item when the memory never changes. Table R1 reports the resulting values as ranges over methods. Under the null, BWT(t) has an expected value of zero, because it is a paired difference between two draws with the same success rate. F(t) is positive, because it takes a maximum over checkpoints. It also grows with t, since the maximum then covers more checkpoints. PED and MER are computed from the whole online trajectory, and their frozen-memory null is the memory-free trajectory requested in W1.2. We therefore answer this part of Q1 in W1.2.
Table R1. Values of F(t) and BWT(t) under the frozen-memory null for a single generation per item, as in the paper, given as ranges over methods. F(t) is shown at the longest horizon (t = 32 on HumanEval and ALFWorld, t = 128 on MATH500).
Subtracting this zero null leaves BWT unchanged, and decoding noise only adds the variation shown in Table R1. For forgetting, subtracting the null lowers F(t), and the reduction is larger at long horizons. The absolute values of F(t) therefore become smaller than those reported in the paper. However, a positive F(t) under the null does not change how we use this metric. In our analysis, F(t) serves mainly to compare how much different methods forget over the same horizon, and every method is measured with the same estimator. What matters for these comparisons is whether the order of methods is preserved. After subtracting the null from the F(t) values in the paper, we compared every pair of methods at the two longest horizons of each task. All 15 pairs whose difference is significant, with a 95% item-bootstrap interval that excludes zero, keep their original order.
In summary, under a frozen memory, BWT has zero expectation and F(t) has a positive floor that grows with t. Subtracting the null leaves BWT unchanged and lowers the absolute values of F(t), but the order of methods at long horizons is preserved. In the revised paper, we will report the null values together with F(t), add these results to the appendix, and state Finding 7 more precisely. Thank you again for this valuable suggestion, which has made our analysis deeper and our conclusions more complete.