acceptodds
Under review as a conference paper at ICLR 2027

Experience Memory and Episode Completion: A Pre-Registered Audit on Two LLM Agents

Abstract

Experience memory, a bank of distilled past episodes retrieved into an agent's context, is widely reported to improve LLM agents, but outcome-bearing memories may also induce premature commitment. We pre-registered a 33-arm dissection of an ExpeL-style memory bank on three multi-hop QA benchmarks (about 29,700 logged episodes, plus 3,000 in two later blocks): the design was hash-frozen before any test episode, four rounds of numeric go/no-go gates ran without a threshold change, and every deviation is reported (Appendix A). For Qwen3.5-27B, the pre-registered NoMem–Mem contrast is equivalent within the registered margin (paired ΔEM +0.6pt; TOST 90% CI [−1.2, +2.4] within ±4pt). None of six content manipulations produces a significant change after Holm correction, and memory does not shorten search: unconditional searches rise, and episodes that reach an answer (a set the treatment itself shifts) use the same number with and without the bank. Memory instead changes how episodes end: the share that exhausts the call budget without answering rises from 14.0% to 25.2%. A post-hoc, single-seed ablation shows that episode-completion mechanics alone (a budget extension and a terminal forced finish) reproduce most of the gain of the evidence-citation instruction arms in three of four decompositions: mechanics recover +6.2pt and the instruction adds +2.0pt beyond them (90% CI [−0.2, +4.2]). Two blocks on a second hardware setup (4×A6000) give, for mechanics and then instruction, +3.8 and +0.4pt with the bank and +2.2 and +2.6pt without it. At matched mechanics the memory contrast is +2.4pt on both hardware setups, too imprecise to establish either a nonzero effect or equivalence. Hard enforcement does not outperform the instruction, a content-free bookkeeping instruction reaches a similar EM point estimate, and optimizer-evolved instructions (GEPA, MIPROv2) do not significantly outperform the hand-written instruction on the full test set; their minibatch gains do not transfer. Two conclusions reverse on gpt-oss-120b, a mixture-of-experts agent with lower base EM on the matched subset: memory helps (+8.5pt, p = 3.7 × 10⁻⁷, two seeds) and stored answers carry the value (redaction −5.3pt). Memory-bank effects differed across the two agents studied here and should be reported per agent. For Qwen3.5-27B, completion mechanics plus one instruction, without a bank, trails no memory configuration by more than 3pt (pooled point estimates), at under two-thirds of the tokens per episode of Mem+Cite.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, experience memory, pre-registration, equivalence testing, multi-hop question answering, prompt optimization, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ruwmOpenReview ↗
Rating
4
Confidence
3
Informativeness
3(+2)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This project presents a pre-registered empirical audit of experience memory in LLM agents. Using a 33-arm dissection of an ExpeL-style memory bank, the authors study memory presence, memory content, retrieval relevance, evidence instructions, enforcement, and episode-completion mechanics. On Qwen3.5-27B, the paper finds little evidence that the memory bank itself improves EM or induces the hypothesized search shortcut. A post-hoc analysis instead suggests that budget extension and a terminal forced-finish mechanism account for a substantial fraction of the gains observed in instruction-bearing arms. On gpt-oss-120b, however, memory provides a clear benefit and stored answers account for much of this gain.

Critical strengths
  1. The paper studies an important attribution problem in agent-memory evaluation: whether measured gains actually come from memory content or from other execution and evaluation choices.

  2. The experimental protocol is unusually careful, including pre-registration, paired tests, equivalence testing, multiple-comparison correction, seed-based null controls, and a second-agent replication.

  3. The paper transparently reports null results and findings that contradict its original hypotheses.

Critical weaknesses
  1. The strongest empirical effect comes from lightweight execution-level interventions (extra calls and a natural-language forced-finish prompt), rather than from a new memory mechanism. Therefore, the contribution depends heavily on whether this completion-mechanics confound generalizes beyond the specific experimental setting.

  2. The evaluated memory system is quite narrow: a small static ExpeL-style text bank built only from successful episodes. Moreover, the PROCEDURE field is not distilled from the actual execution trajectory, since the distiller only observes the question, search count, and final answer. This makes the studied memory closer to distilled textual advice plus a historical outcome than to a full episodic trajectory memory.

  3. More importantly, the paper does not empirically consider representative mainstream memory systems. The related-work section discusses Reflexion, ReasoningBank, MemRL, SEEM, and other recent utility-aware, adaptive, or structured memory approaches, but none of them is evaluated as a baseline or replication setting. As a result, it is unclear whether the reported content-level null results and completion-mechanics confound persist for modern dynamic retrieval, memory-writing/retention, structured episodic memory, or self-evolving procedural memory. This limits the generality of the paper's broader framing around "experience memory."

  4. The source of the +Complete improvement remains ambiguous. The terminal forced-finish answer is not evidence-checked, and the model already achieves non-trivial closed-book EM. Thus, part of the gain may come from forcing the model to guess using parametric knowledge or benchmark memorization rather than from recovering unfinished evidence-grounded reasoning. Since retrieved passages and model outputs were not retained, this alternative explanation cannot currently be ruled out.

  5. The key completion-mechanics decomposition is post-hoc and was motivated by an asymmetry introduced during gating, where A6/A8 were applied only to instruction-bearing arms. The matched-mechanics memory effect remains statistically inconclusive, so the residual value of memory is still unresolved.

  6. The presentation would benefit substantially from a clear architecture/experimental-pipeline figure. The current figures mainly show result plots, but there is no single figure explaining memory construction, retrieval, the ReAct loop, A6/A8 completion mechanics, and how the major experimental arms differ. For a paper whose main goal is to disentangle these components, this makes the central story unnecessarily difficult to follow.

Questions for authors
  1. Can the authors separate forced-finish gains that are supported by evidence retrieved during the episode from gains that come from parametric knowledge or unsupported guessing?

  2. Do the same completion-mechanics effects persist for representative modern memory systems, such as ReasoningBank, MemRL, structured episodic memory (e.g., SEEM), or other adaptive memory-writing/retrieval methods? Without such experiments, how broadly should the paper's conclusions about experience memory be interpreted?

  3. How would the conclusions change if memories were distilled from the actual executed trajectories rather than from the question, search count, and final answer?

  4. Could the authors add a clear pipeline figure showing memory construction/retrieval, the agent execution loop, and the exact locations of the +Complete, +Cite, and +Enforce interventions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ruwm (Part 1/2)

Thank you for the close reading. In one line: a memory vs no-memory comparison also changes how episodes end, and on the three agents analyzed in full, the size of that confound tracked one cheap number, memory's excess in answerless termination. Delta-EM in pt, 90% CIs; post-submission runs are post hoc (logged deviations); "overlap-free" = excluding questions whose gold is a retrieved stored answer; Qwen3-32B = new third agent (n=1000).

W1. Execution-level, not a new memory mechanism. The practical corollary first: on Qwen3.5-27B, a no-bank baseline with the same mechanics and one instruction (NoMem+Cite) trails no memory configuration by more than 3pt, at 10.4k vs 16.3k tokens per episode (Sec. 5). The paper claims no mechanism, only the confound and two cheap arms that measure it. On Qwen3.5-27B with a bank, mechanics alone (Mem->Mem+Complete) give +4.7 [+3.3,+6.1] against an instruction residual of +1.1 [-0.3,+2.6] (three A100 seeds, mixed session), and +3.7 [+2.3,+5.2] same session: most of the +5.0 to +6.4 the instruction-bearing arms gain over Mem (Table 10). Where memory leaves termination alone, bare and matched contrasts agree: gpt-oss-120b +6.4 [+4.1,+8.9] vs +5.8 [+3.2,+8.6] (same session); Qwen3-32B +10.7 [+8.3,+13.2] vs +11.0 [+8.5,+13.6].

W2. Generality of the confound. It generalizes as a pattern, not as a constant. Memory's answerless excess is +11.8pt on Qwen3.5-27B, +1.9 [-0.4,+4.2] on gpt-oss-120b and +0.3 on Qwen3-32B; the mechanics share follows it: +3.7 to +6.2, +1.2 [-1.0,+3.4], +0.4 [-0.7,+1.5]. gpt-oss's answerless episodes end in repeated format failures (s11 transcripts: 3.2 per episode vs 1.3 in answered ones; 4.8 searches), not in searches to the cap, so a forced finish has little to convert; Qwen3.5's run the searches to the cap (7.9 searches; 13/137 parse failures). Qwen3-32B is the bank's own distiller (Sec. 3.1), which may inflate the size of its memory gain (it survives overlap exclusion at +7.9 [+5.6,+10.2]) but not the diagnostic, which uses only its near-zero answerless excess and its matching bare and matched contrasts. Not an 8-call artifact (s11): at 12 and 16 calls the bare Qwen3.5 contrast is +0.4 and -0.2, answerless excess intact (+12.6, +10.8pt). A fourth, newer agent, Qwen3.8-27B (FP8), repeats the Qwen3.5 pattern: answerless +4.7, mechanics +5.4 [+3.8,+7.1], instruction beyond them +1.3 [-0.3,+3.0], overlap-free bare gain +0.5 [-1.4,+2.3]. One environment family, and the revision says so.

W3. A small static bank from solved episodes. We varied the bank and the budget; the channel did not move, because it lives in the loop. Third session, three banks (original; fresh re-distillation, same recipe; trajectory-distilled): bare NoMem->bank +0.4 to +1.2, all inside the registered +/-4; answerless 13.4% -> 24.9-26.4% on every bank; mechanics +4.8 to +5.3 (all p<1e-10). Transcripts (s11): all 137 answerless Mem episodes run to the cap, 96 (70%) with a gold-bearing passage already retrieved. Any memory that adds context under a call budget can open this channel; the revision scopes the tested class (see W5).

W4. PROCEDURE is reconstructed. Trajectory grounding changes nothing detectable. A fresh run on S_bank logged executed trajectories; the same distiller built two banks from the same episodes, one with the original recipe and one from the executed trace (every query, top-5 titles per step, Finish line). Reconstructed -> trajectory: -0.3 [-1.8,+1.2] bare (p=.81), -0.4 [-1.9,+1.1] with +Complete (n=1000); the answerless rise (26.4%) and the mechanics gain (+5.2 [+3.7,+6.7]) persist on the trajectory bank. Deleting procedures outright does not lower EM (+2.2 [+0.2,+4.2], Holm ns; Mem-AnsOnly, Table 2), consistent with procedures that describe behavior this agent already performs (Sec. 5).

W5. No mainstream memory system. The audit's object is the comparison, not a system. We did not run ReasoningBank, MemRL or SEEM and claim nothing about them; reimplementing them is outside the rebuttal window, and the title, abstract and claims will name the tested class: static distilled-text banks. What transfers is the instrument: a confound where termination moved (Qwen3.5), none where it did not (gpt-oss, Qwen3-32B), and again on Qwen3.8 (above). On the systems named: utility-aware retrieval changes which items arrive, but on Qwen3.5 retrieval relevance does not move EM (Mem-Decoy 0.0, Mem-BM25 -0.2; Tables 2, 4); the termination channel sits upstream of it.

AuthorsOpenReview ↗

Response to Reviewer ruwm (Part 2/2)

W6/Q1. Evidence-grounded or parametric? Correct forced answers are overwhelmingly strings the agent retrieved in that episode, not closed-book guesses. We re-ran Mem+Complete with passage logging (s23+37, 182 forced finishes): 24 of 27 correct forced answers are strings the agent had retrieved in that episode, and the gold was already retrieved in 130 of 182 forced episodes, so the budget typically ran out with the evidence in hand. Nor do they track closed-book recall (s11): forced answers score EM .207 vs .087 closed-book on the same questions (not the overall .186), only 8 of 92 equal the closed-book answer, 15 of 19 correct forced answers are wrong closed-book, and of the 27 answerless s11 episodes that +Complete turns correct, 19 are wrong closed-book. Few forced answers carry a quote we could verify (2 of 19 correct ones), so the revision claims only that +Complete converts answerless episodes into committed answers that beat closed-book on the same questions, and lists every logged forced answer in an appendix.

W7. Post hoc decomposition. No registered, hash-frozen claim depends on it, and the paper labels it post hoc. Since submission it has held in two further with-bank sessions: same session, mechanics +3.7 [+2.3,+5.2] (s23+37; pooled with the paper's s11, +4.7 [+3.3,+6.1] vs an instruction residual of +1.1 [-0.3,+2.6]); a third session, +4.8 to +5.3 on the original and two freshly distilled banks. The paper's A6000 block agrees (+3.8 vs +0.4, Table 9). On gpt-oss and Qwen3-32B the same decomposition returns small mechanics (+1.2, +0.4), as the thesis predicts. The claim stands as scoped in Sec. 4.2: on Qwen3.5-27B with a bank, mechanics carry most of the +Cite arm's gain.

W8. Matched-mechanics memory effect. The residual is now bounded by a plan fixed and hashed before launch (fourth session, 8 fresh seeds). Primary, overlap-free (n=3,768): +0.77 [-0.56,+2.07], p=.165, inside the +/-4pt margin the paper registered for the bare memory contrast; the upper bound misses the stricter +/-2pt criterion we pre-specified for this test by 0.07pt, so by the plan's rule the primary is unresolved at +/-2. The pre-specified secondary, all 13 within-session pairs (4 sessions, 11 seeds, overlap-free, n=6,123), gives +1.01 [+0.10,+1.93], which the plan's rule labels a positive residual within +/-2pt. On all questions the residual is +2.4 [+1.0,+3.8], about two thirds of it on the 6% of questions whose gold is a retrieved stored answer. At matched mechanics the bank helps Qwen3.5 by about 2.4pt, mostly through stored answers, and by about 1pt on overlap-free questions, far below the +5.0 to +6.4 instruction-arm gains. The revision replaces the "neither established nor ruled out" wording (Sec. 4.2, 6, 7) with these numbers.

Answer-level overlap (found by us). Beyond the title-level check (Table 8), the gold is among retrieved stored answers for 5.8% (Qwen3.5) / 5.0% (gpt-oss) of questions. It can only favor Mem, so we re-estimated the bare and matched memory contrasts without them (same session; the Qwen3.5 matched residual is superseded by the pre-planned test in W8). Qwen3.5: bare +0.6 -> -1.1 [-2.8,+0.7], inside +/-4. gpt-oss keeps its gain: bare +6.4 -> +4.0 [+1.9,+6.2], matched +5.8 -> +3.5 [+1.1,+6.1]. Qwen3-32B: bare +10.7 -> +7.9 [+5.6,+10.2], matched +11.0 -> +8.5 [+6.2,+10.9]. Echo: 0 of 59 (Qwen3.5) and 0 of 25 (gpt-oss) test-set breaks echo a stored answer. The revision reports every memory contrast both ways.

W9/Q4. Pipeline figure. It will be in the revised PDF: S_bank -> solved episodes -> distiller -> INSIGHT/PROCEDURE/OUTCOME items -> pinned top-7 retrieval -> 8-call ReAct loop, with A6 (conditional +3 calls), A8 (terminal forced finish) and the {instruction} slot (+Cite/+Enforce/+Tools) marked on the loop, plus an arm-difference matrix.

Q2. Scope. The per-system findings (the Qwen3.5 null, answer reuse) are scoped to static distilled-text banks, multi-hop QA and an 8-call budget. The method transfers: report memory's answerless delta; if it is non-zero, add the matched-completion arm and an answer-overlap check. On the agents analyzed in full it discriminated correctly: answerless excess +11.8 / +1.9 / +0.3 -> mechanics +3.7 to +6.2 / +1.2 / +0.4; bare vs matched memory gain +0.6 vs +2.4 (8 pre-planned seeds; +0.8 overlap-free) / +6.4 vs +5.8 / +10.7 vs +11.0. On a fourth agent, Qwen3.8-27B (FP8): answerless +4.7 -> mechanics +5.4 [+3.8,+7.1]; bare vs matched gain +2.5 vs +4.4 (overlap-free bare +0.5).

Q3. Trajectory-distilled memories. See W4: they did not change the conclusions (-0.3 bare, -0.4 with +Complete; answerless rise and mechanics gain intact).

Reviewer_zfk3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether memories of past tasks help LLM agents or encourage premature answers. Memory barely changes qwen-3.5-27B’s accuracy but improves gpt-oss-120b’s by 8.5%, showing that its value depends on the agent. For qwen, much of the improvement associated with citation instructions comes from extra opportunities to finish and submit an answer. The main lesson is to evaluate memory with matched instructions, call budgets, and completion rules.

Critical strengths

None

Critical weaknesses

The paper offers a transparent audit, but its main findings remain specific to the experimental setup. It is difficult to grasp the main idea of the paper. The abstract simply gives a bunch of numbers without connecting them into a generalizable insight.

Questions for authors
  1. why does qwen fails to improve performance while gpt-oss improves?
2 comments
AuthorsOpenReview ↗

Response to Reviewer zfk3 (Part 1/2)

Thank you. Your summary is our thesis; below we put it first, answer your question, and show the rewrite. Runs marked "new" are post hoc and logged as deviations; full tables are in the general response.

The main idea in one sentence. A memory bank changes how an agent's episodes end, not only what it knows, so a bare memory-vs-no-memory comparison mixes termination, answer overlap and content in agent-dependent proportions. In plain terms: on Qwen3.5-27B memory made the agent keep searching instead of stopping, and most of the gain credited to a citation prompt came from a chance to stop; on gpt-oss-120b and on a third agent, memory left stopping alone and helped.

"Findings remain specific to the experimental setup." We respectfully disagree about what the finding is. What transfers is not an effect size but a confound every memory evaluation inherits: adding context under a call budget can change when the agent stops, and whether it does is a measurable property of the agent. Three agents, same bank, questions and harness (Delta-EM pt, 90% CI; same-session arms; new unless noted; gpt-oss on the registered 300-question subset):

AgentBase EMAnswerless shiftBare memoryMatched stopping rulesOverlap-free bare
Qwen3.5-27B (3 seeds, n=1,500).476+11.8pt+0.6 [-1.3,+2.5]+2.4 [+1.0,+3.8] (8 fresh seeds, n=4,000; about +1 overlap-free)-1.1 [-2.8,+0.7]
gpt-oss-120b (3 seeds, n=900).324+1.9pt+6.4 [+4.1,+8.9]+5.8 [+3.2,+8.6]+4.0 [+1.9,+6.2]
Qwen3-32B (2 seeds, n=1,000).333+0.3pt+10.7 [+8.3,+13.2]+11.0 [+8.5,+13.6]+7.9 [+5.6,+10.2]

Memory helps the two agents whose stopping behavior it leaves alone; on the one it pushes to the cap, the bare effect is null and the matched residual is small (+2.4 on all questions, about +1 on overlap-free ones). gpt-oss's same-session +6.4 sits inside the paper's CI for +8.5 [+5.7,+11.5]. Qwen3-32B is the bank's distiller model, so part of its gain may reflect self-match; against a pure style effect, the gain survives overlap exclusion (+7.9). The completion fix tells the same story in reverse: added to the memory arm it gives +3.7 [+2.3,+5.2] on Qwen3.5, whose stopping memory disrupted, but +1.2 [-1.0,+3.4] on gpt-oss and +0.4 [-0.7,+1.5] on Qwen3-32B. This is the pattern the paper's mechanism predicts, on the three agents above: a hypothesis supported, not a proof. A fourth, newer agent, Qwen3.8-27B, falls between them on answerless excess (+4.7pt) and bare memory gain (+2.5pt); its completion-fix gain (+5.4) is within Qwen3.5's range (+3.7 to +6.2).

Within Qwen3.5 the confound held on every axis we varied (every session and hardware setup, four bank constructions, 8/12/16-call budgets): the completion control adds +4.8 to +6.2 with every bank, trajectory distillation changes nothing (-0.3 [-1.8,+1.2]), and the answerless excess persists at 12 and 16 calls (+12.6, +10.8pt). One environment family, we grant; the channel lives in the call budget and stopping rule, which every tool-using agent loop has, and the controls (same budget and forced finish on both arms, filled instruction slot, overlap check) are cheap.

Q1. Why does Qwen3.5 not improve while gpt-oss does? Qwen3.5 reads the memory, keeps searching and runs out of calls, often with the answer already retrieved; the other two agents use it and stop. Two measurable agent properties account for it.

(1) Termination. Memory pushes Qwen3.5 to search until the cap (answerless 14% -> 25%) but barely moves the other two. Transcripts (seed 11; new): Qwen3.5's answerless memory episodes spend 7.9 of 8 calls on searches, with the gold string already retrieved in 70% (96/137). gpt-oss's answerless episodes differ in kind: every one contains malformed actions (3.2 per episode on average, with 4.8 searches), against 9.5% of Qwen3.5's.

(2) Headroom and use of stored answers. gpt-oss and Qwen3-32B start about 15 points lower (Qwen3-32B on the same 500 questions). gpt-oss leans on the bank's answer lines: redacting them costs -5.3 vs -0.8 on Qwen3.5 (Fig. 4). Within Qwen3.5 the dataset with the largest memory point estimate (MuSiQue, +2.0; no per-corpus test) is the only one where redaction hurts (-3.3; Table 15), the gpt-oss signature.

Termination explains why the bare contrast is null on Qwen3.5; headroom and stored-answer use explain why the matched residual is small there and large on the two lower-baseline agents. On these agents the two properties move together; an agent with a different base EM and family would be needed to separate them.

AuthorsOpenReview ↗

Response to Reviewer zfk3 (Part 2/2)

"I learned little or nothing new" / no strengths. We respectfully disagree. The paper delivers three findings a reader of the memory literature would not expect, and the third agent bears on each (Part 1).

  1. Memory made Qwen3.5 fail to stop, not commit early, the opposite of the registered shortcutting worry. The answerless excess (+11.2pt in the paper) persists across sessions, all four banks and 12- and 16-call budgets, and the transcripts in Part 1 show the agent holding the evidence and searching on. Stored answers misled neither agent: 0 of 59 Qwen3.5 and 0 of 25 gpt-oss episodes that memory broke echo a stored answer.

  2. A gain credited to a prompt was mostly a termination effect. Wherever both were measured on Qwen3.5 with a bank, the completion control exceeds the instruction: +6.2 vs +2.0 (Table 5), +3.8 vs +0.4 (A6000, Table 9), +4.7 [+3.3,+6.1] vs +1.1 [-0.3,+2.6] (three A100 seeds, mixed session); the completion control alone adds +3.7 [+2.3,+5.2] same session. Forced answers beat closed-book on the same questions, and correct ones appear in that episode's retrieved text: forced-finish EM .207 vs .087 closed-book on the same questions (seed 11), and 24 of 27 correct forced answers in two further seeds appear in that episode's retrieved passages (new).

  3. A title-level contamination check misses answer leakage. For 5.8% (Qwen3.5) and 5.0% (gpt-oss) of questions the gold answer is among the retrieved stored answers (new). We found and disclosed it ourselves; excluding those questions leaves the direction of every conclusion intact, and the revision reports overlap-excluded estimates throughout.

What you learn:

  • The same bank, questions and harness give a null on one agent and +10.7 on another; without the termination check a reader credits or blames the bank for the agent's stopping behavior. That applies to every memory paper that reports one agent.
  • The audit is a tool, not only a null: it separates a memory gain that survives both controls (Qwen3-32B: +10.7, still +7.9 overlap-free, answerless unchanged) from a termination artifact (Qwen3.5).
  • Cost: on Qwen3.5 a no-bank arm with completion rules and an instruction trails every memory configuration by at most 3 points (point estimates) at 36% fewer tokens than the memory arm with the citation instruction (Sec. 5); on the other two agents the bank pays.

The memory contrast and its +/-4pt equivalence test were registered before any result; the termination shift is the registered shortcutting prediction observed in reverse. The matched-rules residual was then re-tested on 8 fresh pre-planned seeds: +0.77 [-0.56,+2.07] overlap-free, inside the registered +/-4 margin; the stricter +/-2 bound we pre-specified for this test is missed by 0.07pt, so we do not claim equivalence at +/-2.

"Difficult to grasp the main idea" / "abstract is a bunch of numbers." Your summary is the main idea; the submission buried it under arm codes and intervals. The revised PDF, to be uploaded during the discussion period, states it in sentence one, replaces arm codes with names, adds a pipeline figure and a schematic of the mechanism (memory -> more answerless terminations -> completion rules recover EM), moves CIs from the abstract to Sec. 4.2, and rewrites the contributions as findings. Draft abstract:

"Experience memory, past episodes distilled and retrieved into an agent's context, is reported to help LLM agents. In a pre-registered audit of a static ExpeL-style bank on multi-hop QA, we find that a memory bank changes how an agent's episodes end, not only what it knows. On Qwen3.5-27B, memory left accuracy within the registered +/-4-point margin but raised the share of episodes that exhausted the call budget without answering from 14% to 25%, often with the answer already retrieved; most of the accuracy credited to an evidence-citation instruction came from a budget extension and a forced finish, and with stopping rules matched the bank's residual is small. On gpt-oss-120b and Qwen3-32B (the bank's distiller), whose stopping behavior memory leaves alone, memory helps, and the gain survives matched stopping rules and exclusion of questions whose gold answer is among the stored answers. Memory helped where it left stopping alone; where it did not, a bare comparison misattributed termination to memory. Memory evaluations should match completion rules and the instruction slot, check stored answers for overlap, and report effects per agent."

We hope the thesis up front, the answer to your question and the rewrite give reason to reconsider your score.

Reviewer_KkbAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether experience memory (a bank of lessons distilled from past episodes, as in ExpeL) actually helps LLM agents on multi-hop QA. The authors pre-register a large set of conditions on HotpotQA, MuSiQue, and 2WikiMultiHopQA, using a ReAct agent with search. The conditions vary the memory content (e.g., removing stored answers or adding decoys) and add instructions or optimized prompts. For Qwen3.5-27B, memory has no measurable effect on accuracy and mainly makes the agent run out of its step budget more often. A follow-up analysis suggests that the gains from the instruction conditions come mostly from completion mechanics (an extra step plus a forced final answer). For gpt-oss-120b the result is different: memory helps.

Critical strengths
  • Unusually rigorous and transparent protocol. The hash-frozen design, unchanged gate thresholds, paired McNemar/TOST tests, Holm families, a treatment-free null floor, and a full deviation log (Appendix A) make the null results credible. That kind of rigor is rare in agent-memory work.

  • A useful warning for memory evaluations. The paper shows that a memory arm can differ from its baseline in how episodes end, not only in what is in the context (Figure 2, Table 6). This gives a concrete reason to add mechanics-matched and answer-redacted controls.

Critical weaknesses
  • The headline mechanism finding is post hoc, underpowered, and comes from a confound the design created.** Amendments A6/A8 gave completion mechanics only to instruction-bearing arms (Section 3.3). So every registered slot contrast (F-A4, F-A2; Tables 10, 13) compares mechanics plus instruction against a bare arm. The repair (Section 4.2, Tables 5 and 9) uses one seed per hardware setup. Its key estimates (+2.0pt instruction residual, +2.4pt matched-mechanics memory effect) are inside the 2.8pt null floor and the seed spread (Mem s11/s23 differ by 3.2pt; the instruction gain is +8.2 vs. +3.0 by seed, Table 7). The decomposition also flips in one of four blocks. In practice, the most interesting claim is still unresolved.

  • The primary equivalence depends on a fixed 8-call budget and may not mean "memory is useless".** Memory raises answerless termination by +11pt (Figure 2), and with matched mechanics the memory effect is +2.4pt on both setups, which is neither shown nor ruled out (Section 6). The registered null therefore partly reflects budget exhaustion under one cap. Logs do not keep model text or retrieved passages (Section 6, Appendix F), so the paper cannot say why memory causes budget exhaustion. The authors also could not classify test-set breaks for answer echo (Section 4.1).

  • Generality is narrow, and the bank is weak by design.** The study uses one environment family and a 199-item bank. PROCEDURE lines are reconstructed without trajectories (Section 3.2), so the bank is weaker than ExpeL's. The two agents give opposite conclusions (Section 4.5), and the mechanics and optimizer arms were not run on gpt-oss-120b. As a result, the main takeaway ("report per agent") is mostly a caution, not a finding that generalizes.

Questions for authors
  1. Can you run a second A100 seed for the +Complete arms, plus +Complete arms on gpt-oss-120b? This would show whether "mechanics carry most of the gain" and the +2.4pt matched-mechanics memory effect hold up, and whether gpt-oss's +8.5pt memory gain survives matched mechanics.
  2. What causes the higher answerless rate under memory? Do longer contexts (9.6k vs. 5.4k tokens) interact with the 420-token output cap or with parse failures, or do agents really keep searching until the cap? Any small re-run that logs model text would help.
  3. How sensitive is the primary contrast to the call budget (for example, 12 or 16 calls)? Equivalence that holds only at one cap would limit the claim.
  4. Would a bank distilled from real executed trajectories (as in ExpeL) change the Qwen null? Can you rule out that the reconstructed procedures explain it?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KkbA (Part 1/2)

Thank you for naming concrete runs. Every experiment you asked for (Q1-Q4) has been run, plus a pre-planned 8-seed test of the matched-mechanics residual and a third agent (Qwen3-32B). Paired EM differences in pt, 90% question-cluster CIs; new runs are post hoc, logged as deviations. "Same session" = both arms in one new session; "overlap-free" = minus questions whose gold equals a retrieved stored answer (29/500 Qwen, 15/300 gpt-oss; W2d).

W1a (a confound "the design created"). We found and disclosed the A6/A8 asymmetry ourselves. F-A4 (Table 10) is confounded; F-A2 (Table 13) is not: NoMem+Cite->Mem+Cite +2.0 [+0.2,+3.8] (Holm ns) compares two instruction-bearing arms, so both carry A6/A8 and the contrast is already mechanics-matched, as are the enforcement and optimizer contrasts (Sec. 4.2). The multi-seed decomposition now quantifies F-A4's confound: with the bank, mechanics +3.7 to +4.7 vs instruction +1.1 (W1b-c), so mechanics alone account for most of the +5.6pt Mem->Mem+Cite gain; the other A6/A8 arms (+5.0 to +6.4) carry the same mechanics.

W1b (one seed per setup). Mem->Mem+Complete with the bank, every CI excluding zero:

BlockdEM90% CIn
A100 s11 (Table 5)+6.2[+4.4,+8.0]500
A6000 s23 (Table 9)+3.8[+2.4,+5.4]500
A100 s23+s37, same session, new+3.7[+2.3,+5.2]1,000
A100 3 seeds pooled, mixed session (s23/s37 +Complete arms new)+4.7[+3.3,+6.1]1,500
3 bank constructions, third session, s11+s23+4.8 to +5.3all p<1e-101,000 each

Forced answers beat closed-book, and the correct ones appear in retrieved text: forced-answer EM .207 vs .087 closed-book on the same questions (s11), and 24/27 correct forced answers appear in a passage retrieved in that episode (s23+s37).

W1c (inside the 2.8pt floor). The term inside the floor is the instruction residual, which is what "mechanics carry most of the gain" predicts. The mechanics term is not: same session +3.7 [+2.3,+5.2]; pooled over three A100 seeds +4.7 [+3.3,+6.1], whole CI above the floor. On the same 1,500 pairs the instruction residual (Mem+Complete->Mem+Cite) is +1.1 [-0.3,+2.6], p=.178, inside the floor; point difference +3.6. A6000: +3.8 vs +0.4 [-1.8,+2.6].

W1d (the flip). The flip you cite is the A6000 no-bank block (+2.2 vs +2.6). With the bank, mechanics lead in 4/4 decompositions: A100 s11 +6.2 vs +2.0, s23 +3.0 vs 0.0, s37 +4.8 vs +1.4 (s23/s37 mixed session); A6000 +3.8 vs +0.4. Without a bank the mechanics term is smaller (NoMem has fewer answerless episodes to convert: 14.3% vs 26.2%) and the order varies by block. Sec. 4.2's heading will read "(post hoc; with the bank, Qwen3.5-27B)".

W1e ("still unresolved"). Two claims, both now multi-seed: (1) mechanics, not the instruction, account for the instruction arms' gain over Mem; (2) the matched-mechanics residual has a pre-planned 8-seed interval (W2b). We adopt your framing: the registered 8-call equivalence is the headline, Sec. 4.2 is tagged post hoc and scoped, and the abstract scopes claims to static distilled-text banks.

W2b (+2.4 "neither shown nor ruled out"). Now bounded by a pre-planned test: one session, 8 fresh seeds, NoMem+Complete->Mem+Complete, n=4,000 pairs, plan and decision rule hashed before launch. Primary, overlap-free (n=3,768): +0.77 [-0.56,+2.07], p=.165. By the pre-specified rule it is not resolved at the stricter +/-2 bound we set for this test (upper bound 2.07, 0.07pt over), so we do not claim +/-2 from it; it lies inside the +/-4 margin the paper registered for the bare contrast. The pre-specified secondary (13 within-session pairs, 4 sessions, 11 seeds; overlap-free, n=6,123) is +1.01 [+0.10,+1.93]: positive and within +/-2. Full set (n=4,000): +2.38 [+0.97,+3.82], about two thirds of it on the 6% of questions whose gold is a retrieved stored answer. Memory helps a little at matched mechanics, mostly via stored answers, hence the paper's answer-redacted or overlap-checked control (and a leakage caveat in Limitations). A residual of about 1pt overlap-free cannot produce the +5.6pt Mem+Cite gain that mechanics carry.

W2a/Q3 (8-call conditionality). Raising the cap uncovers no memory gain. Sweep (s11, bare arms, n=500, post hoc), NoMem->Mem: +2.4 at 8 calls, +0.4 [-2.2,+3.2] at 12, -0.2 [-2.8,+2.6] at 16; at 12 and 16 both CIs lie inside +/-4 (one seed), and NoMem gains more from the extra budget than Mem (8->16: +5.2 vs +2.6). Memory's answerless excess persists at every cap tested (+11.2, +12.6, +10.8pt). The 8-call claim is multi-seed: same session s11+s23+s37 +0.6 [-1.3,+2.5], n=1,500; bank-construction session +1.2 [-1.0,+3.4]. The registered claim is equivalence within +/-4 on Qwen3.5, not "memory is useless" (W3d).

AuthorsOpenReview ↗

Response to Reviewer KkbA (Part 2/2)

W2c/Q2 (why answerless rises). Text-logged re-run (s11; string-match heuristics): agents really do keep searching until the cap. Mem answerless episodes (n=137): 7.89 searches over 8 turns (NoMem 7.57); parse failure in 13 (9.5%) vs 28/81 (35%) for NoMem; 1,056 assistant characters over 8 turns, about 130 per turn, far below the 420-token cap (NoMem 2,114); gold string already in a retrieved passage in 96/137 (70%). The longer context (9.6k vs 5.4k tokens) does not push outputs toward the cap, since Mem's answerless outputs are the shorter ones; what differs is whether the agent stops. +Complete removes the channel (answerless 24.6% -> 1.9%, same session).

W2d (echo; answer overlap). Echo classified: 0 of Qwen's 59 test-set breaks echo a retrieved stored answer, 0 of gpt-oss's 25 (fixes 18/65, 14/76; s11+s23). The same pass found an answer-level overlap: gold among retrieved stored answers for 5.8%/5.0% of test questions. It can only favor memory, so the registered null is conservative. Overlap-free, same session: Qwen bare +0.6 -> -1.1 [-2.8,+0.7] (n=1,413), inside +/-4; gpt-oss bare +6.4 -> +4.0 [+1.9,+6.2] (n=855, p=.002), matched +5.8 -> +3.5 [+1.1,+6.1] (p=.009); Qwen3-32B bare +10.7 -> +7.9 [+5.6,+10.2]. Memory gains survive where memory helps; the Qwen null stands.

W3a-c/Q4 (narrow; bank weak by design; ExpeL). Bank construction does not explain the null. New text-logged run (third session; s11+s23, n=1,000): two banks distilled from the same 193 solved episodes by the same distiller, differing only in whether the executed trajectory is shown (ExpeL-style vs original recipe). Minimal pair: recon->traj bare -0.3 [-1.8,+1.2], with +Complete -0.4 [-1.9,+1.1]. Bare NoMem->bank is inside +/-4 for the original, rebuilt and trajectory banks (+1.2, +0.7, +0.4); mechanics are +4.8, +5.3, +5.2 on them (all p<1e-10). Capacity: the same 199-item bank gives +6.4 [+4.1,+8.9] on gpt-oss and +10.7 [+8.3,+13.2] on Qwen3-32B, so it is not too small to matter. The confound lives in the agent loop and call budget, not in the bank; any system that adds context under a call budget is exposed to this channel; whether it opens depends on the agent (W3d), which is why the controls are agent-agnostic.

W3e/Q1b (+Complete on gpt-oss). Run (3 seeds, same session, n=900). Memory survives matched mechanics: NoMem+Complete->Mem+Complete +5.8 [+3.2,+8.6], p=2.0e-05; overlap-free +3.5 [+1.1,+6.1]. Mechanics add only +1.2 [-1.0,+3.4] on Mem and +1.9 [-0.3,+4.2] on NoMem although answerless falls 18.9% -> 0.7%: gpt-oss's answerless episodes are format failures, not budget exhaustion (s11 transcripts: 3.2 to 3.5 parse failures per answerless episode vs 1.3 to 1.4 per answered one; 4.5 to 4.8 of 8 searches used). Same-session bare +6.4 [+4.1,+8.9] is inside the paper's CI for +8.5 [+5.7,+11.5] (arms drift across sessions; we infer within session).

W3d ("opposite conclusions"). Agent-dependence is the paper's thesis, not a weakness of it, and with a third agent it is a consistent pattern:

Agentanswerless excessbare NoMem->Memmechanics Mem->Mem+Completematched, overlap-free
Qwen3.5-27B+11.8 [+9.7,+13.9]+0.6 [-1.3,+2.5]+3.7 [+2.3,+5.2]+0.77 [-0.56,+2.07] (8 seeds)
gpt-oss-120b+1.9 [-0.4,+4.2]+6.4 [+4.1,+8.9]+1.2 [-1.0,+3.4]+3.5 [+1.1,+6.1] (3 seeds)
Qwen3-32B+0.3 [-0.2,+0.9]+10.7 [+8.3,+13.2]+0.4 [-0.7,+1.5]+8.5 [+6.2,+10.9] (2 seeds)

(Same-session arms; Qwen3.5 mechanics pooled over 3 A100 seeds, mixed session: +4.7 [+3.3,+6.1]. Qwen3-32B: n=1,000; it is the bank's distiller model.) Across the three agents, memory helps where it does not raise answerless termination and is null where it does, and the mechanics gain follows the same order (+3.7 / +1.2 / +0.4). A fourth, newer agent, Qwen3.8-27B (FP8), falls in line on answerless excess and bare gain: answerless +4.7, mechanics +5.4 [+3.8,+7.1], instruction beyond mechanics +1.3 [-0.3,+3.0] (n=1,000).

W3f ("a caution, not a finding"). The takeaway is quantitative and cross-agent, with a prescription: how much memory raises answerless termination tracks how far a memory comparison is driven by episode completion, and two cheap controls (matched completion rule and instruction slot; overlap-checked bank) separate real memory gains from completion artifacts on the three agents analysed here, which your Strength 2 credits. Limits (one task family; optimizer arms Qwen-only) go to Limitations. Revised abstract, Sec. 4.2 and 7 will be uploaded if PDF revision is permitted.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Thesis. A memory-vs-no-memory comparison also changes how episodes end, and that change gets credited to memory; our cheap, system-agnostic controls detect when it does. Across four agents, how much a bank raises answerless termination (budget exhausted, no answer) tracks whether memory helps. Delta-EM pt, 90% CI, same-session arms:

Agent (base EM)answerless, NoMem->Memmemory, NoMem->Memmechanicsmatched memory
Qwen3.5-27B (.476)+11.8 [+9.7,+13.9]+0.6 [-1.3,+2.5]+3.7 [+2.3,+5.2]+2.4 [+1.0,+3.8]; overlap-free +0.8, see (b)
Qwen3.8-27B, FP8 (.472)+4.7 [+2.9,+6.5]+2.5 [+0.5,+4.5]+5.4 [+3.8,+7.1]+4.4 [+2.4,+6.4]
gpt-oss-120b (.324)+1.9 [-0.4,+4.2]+6.4 [+4.1,+8.9]+1.2 [-1.0,+3.4]+5.8 [+3.2,+8.6]
Qwen3-32B (.333)+0.3 [-0.2,+0.9]+10.7 [+8.3,+13.2]+0.4 [-0.7,+1.5]+11.0 [+8.5,+13.6]

(n=1,500/1,000/900/1,000; Qwen3.5 mechanics n=1,000, matched n=4,000. +Complete = conditional +3 calls, then forced finish, no instruction; mechanics = Mem->Mem+Complete; matched = both arms +Complete. New runs are post hoc, logged as deviations. gpt-oss +6.4 sits inside the paper's +8.5 [+5.7,+11.5]; NoMem drifted across sessions, hence same-session arms only.) The order is monotone: the more answerless episodes the bank adds (+11.8 > +4.7 > +1.9 > +0.3), the smaller the memory gain (+0.6 < +2.5 < +6.4 < +10.7). The newer Qwen3.8 reproduces the paper's decomposition: mechanics +5.4, instruction beyond them +1.3 [-0.3,+3.0], overlap-free bare gain +0.5 [-1.4,+2.3]. In short: on Qwen3.5 memory did not shorten search, it lengthened it until the calls ran out, so gains credited to citation prompts came mostly from a chance to stop; gpt-oss and Qwen3-32B stop, answer, and gain.

(a) Every requested control has been run (same-session unless labeled).

  1. On Qwen3.5, mechanics exceed the instruction in every with-bank decomposition: 3 A100 seeds pooled +4.7 [+3.3,+6.1] vs +1.1 [-0.3,+2.6] (mixed session); A6000 +3.8 vs +0.4. Mechanics alone add +3.7 same-session (answerless 24.6->1.9%) and +4.8, +5.3, +5.2 on the original, rebuilt and trajectory-distilled banks in a third session.
  2. Not a cap artifact: 12 calls +0.4 [-2.2,+3.2], 16 calls -0.2 [-2.8,+2.6] (one seed, n=500 each); memory's answerless excess persists (+12.6, +10.8pt).
  3. The registered Qwen3.5 null replicates in three sessions (+0.6, +0.6, +1.2); in the third, the original, rebuilt and trajectory banks give +1.2, +0.7, +0.4, all inside +/-4. Distilling from executed trajectories changes nothing: -0.3 [-1.8,+1.2].
  4. Why answerless rises (transcripts): with memory Qwen3.5 searches to the cap with the gold string already retrieved (70% of answerless episodes); parse failures 9.5%; outputs far below the 420-token cap.
  5. Forced finishes beat closed-book on the same questions (EM .207 vs .087, seed 11); in two further seeds 24/27 correct forced answers appear in passages retrieved in that episode: +Complete turns answerless episodes into answers whose strings appear in that episode's retrieved text.
  6. Answer-level overlap, found and disclosed by us (gold among retrieved stored answers: 5.8% Qwen, 5.0% gpt-oss), can only favor memory. Excluding it: Qwen3.5 +0.6 -> -1.1 [-2.8,+0.7]; gpt-oss +6.4 -> +4.0 [+1.9,+6.2], matched +5.8 -> +3.5 [+1.1,+6.1]; Qwen3-32B +10.7 -> +7.9 [+5.6,+10.2]. Qwen3-32B is the distiller's own model (Sec. 3.1; Qwen3.5-27B shares only the family), and its gain survives. 0/59 and 0/25 test-set breaks echo a stored answer.

(b) Pre-planned test of the matched-mechanics residual (fourth session, 8 fresh seeds, plan and decision rule hashed in advance). Primary, overlap-free, n=3,768: +0.77 [-0.56,+2.07], inside the registered +/-4 margin (set for the bare contrast); the stricter +/-2 bound we pre-specified is missed by 0.07pt (label: unresolved), so we do not claim it from the primary. Secondary (pre-specified; all 13 within-session pairs, 4 sessions, 11 seeds, overlap-free, n=6,123): +1.01 [+0.10,+1.93], both pre-specified labels: positive, and within +/-2pt. All questions: +2.38 [+0.97,+3.82], two thirds of it on the ~6% overlap questions. "Neither established nor ruled out" becomes "small": about +1pt overlap-free (primary unresolved at the +/-2 bound; pooled secondary within it), +2.4pt on all questions, far below the +5.0 to +6.4pt instruction-arm gains (Table 10).

Prescription. Matched completion rule, matched instruction slot, redacted or overlap-checked stored answers, per-agent reporting. Any memory that adds context under a call budget can open this channel. Scope: one environment family; MemRL/SEEM-style systems and gpt-oss optimizer arms are outside the window.

Revision (PDF during discussion): pipeline figure, arm matrix, abstract/intro rewritten around the per-agent pattern; "mechanics carry most of the gain" scoped to Qwen3.5 with a bank; 8-16-call equivalence; Sec. 4.2/6/7 give the small residual.

AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised PDF. New or changed text is in blue; the OpenReview diff against the submission is also available. Because a pipeline figure is now Figure 1, the submission's Figures 1-4 are now Figures 2-5 (figure numbers in our earlier responses refer to the submission).

  • Title and abstract: scoped to static experience-memory banks; the abstract now leads with the thesis (a memory-vs-no-memory comparison also changes how episodes end) and the four-agent pattern.
  • Sec. 3: pipeline figure (bank construction, retrieval, the ReAct loop, the instruction slot and the A6/A8 completion mechanics) and an arm-difference matrix (Table 19).
  • Sec. 4.1-4.2: test-set echo classification (0/59 breaks), the multi-seed mechanics decomposition, and the pre-planned eight-seed matched-mechanics test with its pre-specified decision rule.
  • Sec. 4.5-7: the four-agent comparison (Qwen3.5-27B, Qwen3.8-27B, gpt-oss-120b, Qwen3-32B), the termination channel, and the reporting controls.
  • Appendix I (new): every discussion-period experiment with full tables, including the call-budget sweep, transcript diagnostics, answer-overlap exclusion and the trajectory-distilled bank.

All new runs are post hoc and logged as deviations; the numbers match our responses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.