- Rating
- 4
- Confidence
- 3
- Informativeness
- 3(+2)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This project presents a pre-registered empirical audit of experience memory in LLM agents. Using a 33-arm dissection of an ExpeL-style memory bank, the authors study memory presence, memory content, retrieval relevance, evidence instructions, enforcement, and episode-completion mechanics. On Qwen3.5-27B, the paper finds little evidence that the memory bank itself improves EM or induces the hypothesized search shortcut. A post-hoc analysis instead suggests that budget extension and a terminal forced-finish mechanism account for a substantial fraction of the gains observed in instruction-bearing arms. On gpt-oss-120b, however, memory provides a clear benefit and stored answers account for much of this gain.
Critical strengths
-
The paper studies an important attribution problem in agent-memory evaluation: whether measured gains actually come from memory content or from other execution and evaluation choices.
-
The experimental protocol is unusually careful, including pre-registration, paired tests, equivalence testing, multiple-comparison correction, seed-based null controls, and a second-agent replication.
-
The paper transparently reports null results and findings that contradict its original hypotheses.
Critical weaknesses
-
The strongest empirical effect comes from lightweight execution-level interventions (extra calls and a natural-language forced-finish prompt), rather than from a new memory mechanism. Therefore, the contribution depends heavily on whether this completion-mechanics confound generalizes beyond the specific experimental setting.
-
The evaluated memory system is quite narrow: a small static ExpeL-style text bank built only from successful episodes. Moreover, the PROCEDURE field is not distilled from the actual execution trajectory, since the distiller only observes the question, search count, and final answer. This makes the studied memory closer to distilled textual advice plus a historical outcome than to a full episodic trajectory memory.
-
More importantly, the paper does not empirically consider representative mainstream memory systems. The related-work section discusses Reflexion, ReasoningBank, MemRL, SEEM, and other recent utility-aware, adaptive, or structured memory approaches, but none of them is evaluated as a baseline or replication setting. As a result, it is unclear whether the reported content-level null results and completion-mechanics confound persist for modern dynamic retrieval, memory-writing/retention, structured episodic memory, or self-evolving procedural memory. This limits the generality of the paper's broader framing around "experience memory."
-
The source of the +Complete improvement remains ambiguous. The terminal forced-finish answer is not evidence-checked, and the model already achieves non-trivial closed-book EM. Thus, part of the gain may come from forcing the model to guess using parametric knowledge or benchmark memorization rather than from recovering unfinished evidence-grounded reasoning. Since retrieved passages and model outputs were not retained, this alternative explanation cannot currently be ruled out.
-
The key completion-mechanics decomposition is post-hoc and was motivated by an asymmetry introduced during gating, where A6/A8 were applied only to instruction-bearing arms. The matched-mechanics memory effect remains statistically inconclusive, so the residual value of memory is still unresolved.
-
The presentation would benefit substantially from a clear architecture/experimental-pipeline figure. The current figures mainly show result plots, but there is no single figure explaining memory construction, retrieval, the ReAct loop, A6/A8 completion mechanics, and how the major experimental arms differ. For a paper whose main goal is to disentangle these components, this makes the central story unnecessarily difficult to follow.
Questions for authors
-
Can the authors separate forced-finish gains that are supported by evidence retrieved during the episode from gains that come from parametric knowledge or unsupported guessing?
-
Do the same completion-mechanics effects persist for representative modern memory systems, such as ReasoningBank, MemRL, structured episodic memory (e.g., SEEM), or other adaptive memory-writing/retrieval methods? Without such experiments, how broadly should the paper's conclusions about experience memory be interpreted?
-
How would the conclusions change if memories were distilled from the actual executed trajectories rather than from the question, search count, and final answer?
-
Could the authors add a clear pipeline figure showing memory construction/retrieval, the agent execution loop, and the exact locations of the +Complete, +Cite, and +Enforce interventions?
2 comments
Response to Reviewer ruwm (Part 2/2)
W6/Q1. Evidence-grounded or parametric? Correct forced answers are overwhelmingly strings the agent retrieved in that episode, not closed-book guesses. We re-ran Mem+Complete with passage logging (s23+37, 182 forced finishes): 24 of 27 correct forced answers are strings the agent had retrieved in that episode, and the gold was already retrieved in 130 of 182 forced episodes, so the budget typically ran out with the evidence in hand. Nor do they track closed-book recall (s11): forced answers score EM .207 vs .087 closed-book on the same questions (not the overall .186), only 8 of 92 equal the closed-book answer, 15 of 19 correct forced answers are wrong closed-book, and of the 27 answerless s11 episodes that +Complete turns correct, 19 are wrong closed-book. Few forced answers carry a quote we could verify (2 of 19 correct ones), so the revision claims only that +Complete converts answerless episodes into committed answers that beat closed-book on the same questions, and lists every logged forced answer in an appendix.
W7. Post hoc decomposition. No registered, hash-frozen claim depends on it, and the paper labels it post hoc. Since submission it has held in two further with-bank sessions: same session, mechanics +3.7 [+2.3,+5.2] (s23+37; pooled with the paper's s11, +4.7 [+3.3,+6.1] vs an instruction residual of +1.1 [-0.3,+2.6]); a third session, +4.8 to +5.3 on the original and two freshly distilled banks. The paper's A6000 block agrees (+3.8 vs +0.4, Table 9). On gpt-oss and Qwen3-32B the same decomposition returns small mechanics (+1.2, +0.4), as the thesis predicts. The claim stands as scoped in Sec. 4.2: on Qwen3.5-27B with a bank, mechanics carry most of the +Cite arm's gain.
W8. Matched-mechanics memory effect. The residual is now bounded by a plan fixed and hashed before launch (fourth session, 8 fresh seeds). Primary, overlap-free (n=3,768): +0.77 [-0.56,+2.07], p=.165, inside the +/-4pt margin the paper registered for the bare memory contrast; the upper bound misses the stricter +/-2pt criterion we pre-specified for this test by 0.07pt, so by the plan's rule the primary is unresolved at +/-2. The pre-specified secondary, all 13 within-session pairs (4 sessions, 11 seeds, overlap-free, n=6,123), gives +1.01 [+0.10,+1.93], which the plan's rule labels a positive residual within +/-2pt. On all questions the residual is +2.4 [+1.0,+3.8], about two thirds of it on the 6% of questions whose gold is a retrieved stored answer. At matched mechanics the bank helps Qwen3.5 by about 2.4pt, mostly through stored answers, and by about 1pt on overlap-free questions, far below the +5.0 to +6.4 instruction-arm gains. The revision replaces the "neither established nor ruled out" wording (Sec. 4.2, 6, 7) with these numbers.
Answer-level overlap (found by us). Beyond the title-level check (Table 8), the gold is among retrieved stored answers for 5.8% (Qwen3.5) / 5.0% (gpt-oss) of questions. It can only favor Mem, so we re-estimated the bare and matched memory contrasts without them (same session; the Qwen3.5 matched residual is superseded by the pre-planned test in W8). Qwen3.5: bare +0.6 -> -1.1 [-2.8,+0.7], inside +/-4. gpt-oss keeps its gain: bare +6.4 -> +4.0 [+1.9,+6.2], matched +5.8 -> +3.5 [+1.1,+6.1]. Qwen3-32B: bare +10.7 -> +7.9 [+5.6,+10.2], matched +11.0 -> +8.5 [+6.2,+10.9]. Echo: 0 of 59 (Qwen3.5) and 0 of 25 (gpt-oss) test-set breaks echo a stored answer. The revision reports every memory contrast both ways.
W9/Q4. Pipeline figure. It will be in the revised PDF: S_bank -> solved episodes -> distiller -> INSIGHT/PROCEDURE/OUTCOME items -> pinned top-7 retrieval -> 8-call ReAct loop, with A6 (conditional +3 calls), A8 (terminal forced finish) and the {instruction} slot (+Cite/+Enforce/+Tools) marked on the loop, plus an arm-difference matrix.
Q2. Scope. The per-system findings (the Qwen3.5 null, answer reuse) are scoped to static distilled-text banks, multi-hop QA and an 8-call budget. The method transfers: report memory's answerless delta; if it is non-zero, add the matched-completion arm and an answer-overlap check. On the agents analyzed in full it discriminated correctly: answerless excess +11.8 / +1.9 / +0.3 -> mechanics +3.7 to +6.2 / +1.2 / +0.4; bare vs matched memory gain +0.6 vs +2.4 (8 pre-planned seeds; +0.8 overlap-free) / +6.4 vs +5.8 / +10.7 vs +11.0. On a fourth agent, Qwen3.8-27B (FP8): answerless +4.7 -> mechanics +5.4 [+3.8,+7.1]; bare vs matched gain +2.5 vs +4.4 (overlap-free bare +0.5).
Q3. Trajectory-distilled memories. See W4: they did not change the conclusions (-0.3 bare, -0.4 with +Complete; answerless rise and mechanics gain intact).
Response to Reviewer ruwm (Part 1/2)
Thank you for the close reading. In one line: a memory vs no-memory comparison also changes how episodes end, and on the three agents analyzed in full, the size of that confound tracked one cheap number, memory's excess in answerless termination. Delta-EM in pt, 90% CIs; post-submission runs are post hoc (logged deviations); "overlap-free" = excluding questions whose gold is a retrieved stored answer; Qwen3-32B = new third agent (n=1000).
W1. Execution-level, not a new memory mechanism. The practical corollary first: on Qwen3.5-27B, a no-bank baseline with the same mechanics and one instruction (NoMem+Cite) trails no memory configuration by more than 3pt, at 10.4k vs 16.3k tokens per episode (Sec. 5). The paper claims no mechanism, only the confound and two cheap arms that measure it. On Qwen3.5-27B with a bank, mechanics alone (Mem->Mem+Complete) give +4.7 [+3.3,+6.1] against an instruction residual of +1.1 [-0.3,+2.6] (three A100 seeds, mixed session), and +3.7 [+2.3,+5.2] same session: most of the +5.0 to +6.4 the instruction-bearing arms gain over Mem (Table 10). Where memory leaves termination alone, bare and matched contrasts agree: gpt-oss-120b +6.4 [+4.1,+8.9] vs +5.8 [+3.2,+8.6] (same session); Qwen3-32B +10.7 [+8.3,+13.2] vs +11.0 [+8.5,+13.6].
W2. Generality of the confound. It generalizes as a pattern, not as a constant. Memory's answerless excess is +11.8pt on Qwen3.5-27B, +1.9 [-0.4,+4.2] on gpt-oss-120b and +0.3 on Qwen3-32B; the mechanics share follows it: +3.7 to +6.2, +1.2 [-1.0,+3.4], +0.4 [-0.7,+1.5]. gpt-oss's answerless episodes end in repeated format failures (s11 transcripts: 3.2 per episode vs 1.3 in answered ones; 4.8 searches), not in searches to the cap, so a forced finish has little to convert; Qwen3.5's run the searches to the cap (7.9 searches; 13/137 parse failures). Qwen3-32B is the bank's own distiller (Sec. 3.1), which may inflate the size of its memory gain (it survives overlap exclusion at +7.9 [+5.6,+10.2]) but not the diagnostic, which uses only its near-zero answerless excess and its matching bare and matched contrasts. Not an 8-call artifact (s11): at 12 and 16 calls the bare Qwen3.5 contrast is +0.4 and -0.2, answerless excess intact (+12.6, +10.8pt). A fourth, newer agent, Qwen3.8-27B (FP8), repeats the Qwen3.5 pattern: answerless +4.7, mechanics +5.4 [+3.8,+7.1], instruction beyond them +1.3 [-0.3,+3.0], overlap-free bare gain +0.5 [-1.4,+2.3]. One environment family, and the revision says so.
W3. A small static bank from solved episodes. We varied the bank and the budget; the channel did not move, because it lives in the loop. Third session, three banks (original; fresh re-distillation, same recipe; trajectory-distilled): bare NoMem->bank +0.4 to +1.2, all inside the registered +/-4; answerless 13.4% -> 24.9-26.4% on every bank; mechanics +4.8 to +5.3 (all p<1e-10). Transcripts (s11): all 137 answerless Mem episodes run to the cap, 96 (70%) with a gold-bearing passage already retrieved. Any memory that adds context under a call budget can open this channel; the revision scopes the tested class (see W5).
W4. PROCEDURE is reconstructed. Trajectory grounding changes nothing detectable. A fresh run on S_bank logged executed trajectories; the same distiller built two banks from the same episodes, one with the original recipe and one from the executed trace (every query, top-5 titles per step, Finish line). Reconstructed -> trajectory: -0.3 [-1.8,+1.2] bare (p=.81), -0.4 [-1.9,+1.1] with +Complete (n=1000); the answerless rise (26.4%) and the mechanics gain (+5.2 [+3.7,+6.7]) persist on the trajectory bank. Deleting procedures outright does not lower EM (+2.2 [+0.2,+4.2], Holm ns; Mem-AnsOnly, Table 2), consistent with procedures that describe behavior this agent already performs (Sec. 5).
W5. No mainstream memory system. The audit's object is the comparison, not a system. We did not run ReasoningBank, MemRL or SEEM and claim nothing about them; reimplementing them is outside the rebuttal window, and the title, abstract and claims will name the tested class: static distilled-text banks. What transfers is the instrument: a confound where termination moved (Qwen3.5), none where it did not (gpt-oss, Qwen3-32B), and again on Qwen3.8 (above). On the systems named: utility-aware retrieval changes which items arrive, but on Qwen3.5 retrieval relevance does not move EM (Mem-Decoy 0.0, Mem-BM25 -0.2; Tables 2, 4); the termination channel sits upstream of it.