acceptodds
Withdrawn from ICLR 2027

When Memory Hurts: MemoryForge for Diagnosing Memory Utility in AI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent memory is usually evaluated by what it retains or retrieves, not by whether remembered experience improves the next task. We introduce MemoryForge, the first controlled evaluation framework, to our knowledge, that branches native agent memory from a shared frozen acquisition state to isolate the downstream behavioral effects of persistence, consolidation, full-history availability, and vetted content. These effects are instantiated as five task-paired interventions: Stateless, Frozen Context, Raw Memory, Dreamed Memory, and Oracle guidance. The protocol separately measures condition-independent artifact correctness and condition-aware process compliance, then records their joint attainment. On FinStateBench, three Claude tiers crossed with three native agent runtimes yield 1,350 executions over 30 held-out tasks. Oracle achieves the highest mean artifact score (0.971) but only 45.2% joint pass, compared with 72.6% for Stateless; Raw reaches 49.3%. Moreover, 58 of the 89 Stateless passes lost under Raw also fail under Oracle, exceeding the overlap expected under independence (44). Under Raw memory, runtime accounts for 78.0% of between-system variation, whereas model tier accounts for 3.1%. The five observed branches nevertheless cover 90.4% of cells, 17.8 points above the best fixed condition, with gains in all nine systems. These results expose a Memory Utility Gap: useful behaviors exist, but current agents do not reliably select, adapt, and execute the right memory mode for each task.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, LLM Agents, Memory Evaluation, Memory Utility, Long-Term Memory, Memory Consolidation, Negative Transfer, Agent Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rU9uOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

MemoryForge proposes a controlled evaluation framework to measure whether memory actually improves subsequent agent execution. This benchmark evaluates identical agent-task combinations under five conditions (Stateless--No prior memory, Frozen Context--A complete record of the acquisition process, Raw Memory -- Native persistent memory, Dreamed Memory --Consolidated and organized native memory, and Oracle guidance--Verified procedural guidance) while keeping the shared "acquisition state" constant. The evaluation covered 30 types of financial analysis tasks, involving a total of 1,350 runs across three Claude models and three agent runtimes. The authors report that the joint-pass rate was 72.6% for the "Stateless" condition, compared to 49.3% for "Raw Memory" and 45.2% for "Oracle."

Critical strengths

The idea that memory may actually harm LLM agent performance is interesting

Critical weaknesses

The paper appears incoherent and does not provide clear explanation of various terms and its studied subjects. (E.g., native runtime, native consolidation operation, native state, native-state snapshots, native agent memory, native post-build state)

Besides, the primary evaluation metric is not comparable across memory conditions. The reported 72.6% versus 49.3% performance comparison relies on the paper's joint execution criterion, which combines artifact correctness with process compliance. However, process compliance explicitly includes a memory-use dimension whose requirements differ by experimental condition. For example. Stateless rewards independent execution without previous memory. Raw and Dreamed require evidence of appropriate retrieval and application. Oracle has its own authorized-use requirements.

Furthermore, artifact correctness barely changes under Raw Memory, and the artifact evaluator itself has concerning validity problems. For example, in Task 33, the Stateless execution uses fabricated identifiers and randomized values but receives an artifact score of 1. The authors correctly note that this does not establish a verified market-analysis success, but this undermines the interpretation of executable artifact scores as reliable evidence of real task correctness.

Questions for authors
  • Can the results be reproduced in opensource model frameworks, in other proprietary models, and other models of different generation?
Reviewer_2m5bOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemoryForge, a controlled evaluation framework for assessing the downstream utility of agent memory. It compares five memory conditions using shared frozen acquisition states and introduces FinStateBench, comprising 10 acquisition tasks and 30 evaluation tasks. Experiments across three Claude models and three agent runtimes show that memory often reduces joint-pass rates despite maintaining similar artifact correctness. The authors further highlight the importance of runtime design, memory quality, and task-dependent memory selection.

Critical strengths
  1. The paper shifts the focus from memory retention and retrieval to downstream task utility, addressing an important question of whether memory actually improves agent execution.

  2. The shared frozen-state protocol enables paired comparisons across memory conditions, reducing confounding factors. The evaluation across multiple models and agent runtimes provides useful insights into how memory affects agent behavior.

Critical weaknesses
  1. Questionable Evaluation Rubric. The joint-pass metric combines artifact correctness with process compliance, including a memory-use dimension that applies different criteria across memory conditions. Since the substantial performance degradation is largely driven by this dimension, it is unclear whether the reported negative effects reflect actual task failures or biases introduced by the evaluation rubric, especially under LLM-as-a-Judge. A condition-independent evaluation would strengthen the conclusions.

  2. Limited Model Diversity. The experiments exclusively use three models from the Claude family (Haiku, Sonnet, and Opus). Without evaluating other model families, such as GPT or open-source models (e.g., Qwen), it is difficult to determine whether the findings generalize beyond Claude-based agents. This limitation also weakens the claim that agent runtime matters more than model capability.

  3. Insufficient Failure Mechanism Analysis. While the paper shows that memory-enabled conditions can lead to lower joint-pass rates, it provides limited analysis of why these failures occur. It remains unclear whether failures arise from incorrect memory retrieval, inappropriate memory application, context interference, or other execution issues. More detailed trajectory-level analysis would help substantiate the proposed memory selection, adaptation, and verification bottlenecks.

Questions for authors

The Stateless baseline already achieves a high artifact score of 0.952, while Raw and Dreamed Memory show little improvement. This raises concerns about whether the evaluation tasks genuinely require memory to be completed. If agents can solve most tasks without relying on prior experience, how can the benchmark effectively assess the benefits of memory? Could the authors clarify how memory-dependent tasks are constructed and provide a more detailed analysis distinguishing tasks that genuinely require memory from those that can be solved independently?

Reviewer_9wfTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemoryForge, a framework that branches five memory conditions from a shared frozen acquisition state to evaluate their effects on downstream agent execution. Across 1350 executions involving three models and three native runtimes on FinStateBench, the authors separately measure artifact correctness, process compliance, and joint pass rates. They find that semantic memory quality does not consistently translate into execution gains and argue for task-dependent memory selection, adaptation, and verification.

Critical strengths
  1. The shared-state, task-paired experimental design is valuable.Memory conditions share the same acquisition origin, while evaluation tasks are independently initialized without cross-branch writeback. These controls reduce confounding from acquisition differences and state contamination, providing a clearer basis for comparing persistence, consolidation, and full-history availability. This controlled protocol is the paper’s strongest contribution.

  2. Preserving native runtimes gives the evaluation practical systems relevance.The framework evaluates complete systems with their native memory representations, retrieval mechanisms, and consolidation operations, rather than only substituting memory text in prompts. Comparing models and runtimes can expose consequential deployment-level differences and gives the framework potential value as a reusable diagnostic tool.

Critical weaknesses
  1. The experiments establish variability in memory utility more convincingly than they establish its underlying cause.The paper primarily attributes the results to failures in memory selection, adaptation, and execution control. However, the supporting evidence largely consists of aggregate performance differences and overlapping failure sets. Shared failures under Raw and Oracle do not rule out alternative explanations such as intrinsic task difficulty, mismatched guidance, or additional context burden. Without systematic analysis of what was retrieved, how it was applied, and where execution failed, the proposed control-bottleneck explanation remains insufficiently established.

  2. Hindsight branch coverage does not yet establish actionable benefits from task-dependent memory selection.The paper presents the increase from 72.6% to 90.4% success coverage across five conditions as selection headroom. However, each system–task–condition cell contains only one execution, and there is no comparison against repeated executions of a fixed condition under the same execution budget. Consequently, the observed gain may combine stable complementarity between memory modes with ordinary stochastic retry gains. This limits its value as evidence for the practical benefits of memory-mode selection.

Questions for authors
  1. For tasks that fail under both Raw and Oracle, is there systematic trajectory evidence showing that suitable guidance was available but failed during selection, adaptation, or verification? What evidence distinguishes control failures from mismatched content or context interference?
  2. What success coverage is obtained by repeatedly running Stateless or the best fixed condition under the same execution budget? Are the coverage advantage of the five memory modes and task-specific preferences between modes stable across repeated executions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.