- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
MemoryForge proposes a controlled evaluation framework to measure whether memory actually improves subsequent agent execution. This benchmark evaluates identical agent-task combinations under five conditions (Stateless--No prior memory, Frozen Context--A complete record of the acquisition process, Raw Memory -- Native persistent memory, Dreamed Memory --Consolidated and organized native memory, and Oracle guidance--Verified procedural guidance) while keeping the shared "acquisition state" constant. The evaluation covered 30 types of financial analysis tasks, involving a total of 1,350 runs across three Claude models and three agent runtimes. The authors report that the joint-pass rate was 72.6% for the "Stateless" condition, compared to 49.3% for "Raw Memory" and 45.2% for "Oracle."
Critical strengths
The idea that memory may actually harm LLM agent performance is interesting
Critical weaknesses
The paper appears incoherent and does not provide clear explanation of various terms and its studied subjects. (E.g., native runtime, native consolidation operation, native state, native-state snapshots, native agent memory, native post-build state)
Besides, the primary evaluation metric is not comparable across memory conditions. The reported 72.6% versus 49.3% performance comparison relies on the paper's joint execution criterion, which combines artifact correctness with process compliance. However, process compliance explicitly includes a memory-use dimension whose requirements differ by experimental condition. For example. Stateless rewards independent execution without previous memory. Raw and Dreamed require evidence of appropriate retrieval and application. Oracle has its own authorized-use requirements.
Furthermore, artifact correctness barely changes under Raw Memory, and the artifact evaluator itself has concerning validity problems. For example, in Task 33, the Stateless execution uses fabricated identifiers and randomized values but receives an artifact score of 1. The authors correctly note that this does not establish a verified market-analysis success, but this undermines the interpretation of executable artifact scores as reliable evidence of real task correctness.
Questions for authors
- Can the results be reproduced in opensource model frameworks, in other proprietary models, and other models of different generation?