acceptodds
Under review as a conference paper at ICLR 2027

MemLadder: Controlled Auditing of Agent Memory

Abstract

Agent memory is commonly evaluated by comparing endpoint success with and without retrieved experience, but that comparison is not an attribution test for the retrieved text. It jointly changes the prompt channel, control text, action policy, exposure duration, and realized continuation. We introduce MemLadder, a controlled audit that separates behavioral rerouting, single-decision downstream value, and remaining-episode exposure-policy return. Its five mutually exclusive conditions range from bare context to retrieved source-grounded memory, including structural, length- and format-matched generic, and mismatched-real controls. From the same restored ALFWorld state, MemLadder executes each induced action, removes exposure text, and estimates return with five repeated memory-free executions under the frozen no-memory serving configuration. In a contemporaneous, human-verified audit, retrieved memory versus an actionable domain-general control produced actor-state-averaged substantive rerouting rates of 25.6% and 28.7%. On a fresh episode-disjoint 180-state panel, however, the one-step value contrast was −0.67 percentage points for both Qwen actors (episode 95% CI [−1.78, +0.45]). Under the binary K = 5 instrument, 94–98% of fresh-panel states were at an empirical text-arm floor or ceiling, so this small average contrast is a low-resolution estimate rather than evidence that memory is generally valueless. An exhaustive environment-only audit of all 5,231 admissible state–action choices finds a controller-relative outcome boundary in 50/180 states (27.8%), yet retrieved memory and the actionable control fall on opposite sides in only 2/180 and 5/180 states for 35B and 9B, respectively; mean preservation is 90.2% on controller-solvable states. Thus both weak-continuation failure and strong-continuation recovery can compress observed one-step separation. Behavioral rerouting, one-step downstream value, and remaining-episode exposure-policy return therefore require separate controls and should be reported as distinct estimands.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, LLM agents, agent evaluation, memory attribution, action-value estimation, long-horizon decision making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j4HvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that comparing agent success with and without memory can't show what the memory contributes because adding memory also changes the prompt format, its length, and how the rest of the episode unfolds. It proposes MemLadder, an evaluation protocol in ALFWorld with two Qwen3.5 models. From the same saved state, the agent takes one action under several conditions: no memory, generic text controls, mismatched memory and correctly retrieved memory. The method then measures whether the action changes and what that action is worth when the rest of the episode is run without memory.

Contributions include:

  • Separating rerouting, one-step value and return with memory kept in the prompt. This is a conceptual distinction, not a new method.
  • A same-state test with matched controls and repeated memory-free continuations. The protocol works and the intervals are calibrated, but it can't detect even an expert action's one-step value.
  • Memory changes behavior, but it shows no value advantage.
Critical strengths

The paper shows that a with/without-memory comparison mixes the memory's content with other effects, such as prompt format and length, and it separates whether memory changes behavior from whether that change adds value.

The controls (length-matched generic text, mismatched real memory, an actionable generic procedure) are well designed

Critical weaknesses

In most states every condition always fails or always succeeds. Even a single expert action can't be distinguished from a random one (Fig. 3). The null result for memory is therefore uninformative: the paper can't separate "memory doesn't help" from "this method can't tell".

The paper isn't clearly written. Key terms ("arm", "post-primary", "continuation identity") are undefined or used before they're defined. The results section is mostly strings of numbers, and the main text points to appendix tables that aren't included. With no clear story, it's hard to tell what was planned in advance and what the main claim is.

Questions for authors

A single expert action can't be distinguished from a random one (Fig. 3). What size of one-step memory effect could this method actually detect?

Why use binary success with near-deterministic repeated runs instead of a graded measure such as progress toward the goal?

Why is one-step value under a memory-free continuation the right thing to measure, when deployed memory stays in the prompt across decisions?

What should a practitioner building agent memory do differently because of these findings?

1 comment
AuthorsOpenReview ↗

Detectable Single-Decision Value, Outcome Resolution, and Practical Use

Thank you for the specific questions. We address them in revised Section 5 and Appendices F and M.

  1. Can MemLadder detect single-decision value, and at what scale?

Your central concern is that a small retrieved-memory contrast may reflect an insensitive audit. The original one-action expert-versus-random comparison was inconclusive. We therefore selected 50 new episode-disjoint ALFWorld states with favorable and unfavorable actions identified by a separate fixed controller before observing Actor returns.

Under memory-free continuation, the prespecified three-Actor pooled action-value gap was +14.07 percentage points (95% CI [+6.27, +22.53], fixed-sequence p=.00152). Constructed, oracle-informed guidance expressing the correct action intent without its exact command exceeded broad category guidance by +12.00 points ([+4.67, +20.13], p=.00510).

The direct-action contrast survived Actor-specific Holm correction for 35B and Gemma, but not 9B. The guidance result relies on the prespecified pooled test; none of its three Actor-specific contrasts survived Holm correction (Appendix F, Table 13).

On these same states, H4−HP was +2.27 points (95% CI [−0.40, +5.33]), and correct-intent guidance exceeded H4 by +10.00 points ([+2.53, +18.27]); both are prespecified secondary comparisons with unadjusted inference (Section 5.2C). Thus, the audit detected effects of 12–14 points here. The retrieved-memory interval does not support advantages above about +5.3 points, while smaller benefits remain uncertain. This is conditional sensitivity, not a universal detection threshold or evidence of zero memory value.

  1. Why binary success and repeated greedy continuations rather than graded outcomes?

Binary success was the frozen primary endpoint. Only 2.1–7.1% of continuation identities mixed success and failure among five suffixes under the greedy serving configuration (Section 3.3), so repetition adds limited resolution to a saturated binary measure.

We also reconstructed graded residual progress. On Fresh180, 35B H4−HP was −2.67 points (state 95% CI [−5.04, −0.29]); the equal-candidate interval crossed zero. Graded outcomes reduced some ties but did not reveal a stable positive H4 benefit. In a 48-state sensitivity test, temperature-0.7 continuation changed H4−HP by −0.42 and −1.04 points for 35B and 9B (Appendix M.4).

  1. Why remove memory after one decision?

A common memory-free continuation measures the downstream value of a single memory-induced action. It does not measure a system that keeps memory available throughout an episode. The same-state contrast is a local advantage difference; persistent return additionally depends on the states visited and later actions of each exposure policy (Section 6). We evaluate persistent exposure separately in RQ3.

  1. What should practitioners do differently?

First compare retrieval with an actionable generic control, rather than relying only on memory/no-memory success. Then check whether consequential action differences are detectable, estimate single-decision value under a declared continuation policy, and separately evaluate persistent return. Indeed, HP alone changes approximately 15–21% of first actions or successor states relative to an empty channel (Section 5.1), illustrating why generic guidance effects should not be credited to retrieved history.

  1. Presentation.

We now define arm (Section 3.1), post-primary (Introduction), and continuation identity (Section 3.3) at first use. Section 4 distinguishes original, pre-submission supplementary, and post-submission analyses; Table 2 presents their value results, and Appendix B documents panel chronology. The complete appendix is now included in the same PDF as the main text. The new single-decision validation appears in the main text (Section 5.2, Figure 2).

Reviewer_mkDBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that comparing agent success with and without memory does not attribute an effect to the retrieved text. Exposure also changes the prompt, the control text, the policy and the continuation. MemLadder is an evaluation protocol that separates three quantities. These are behavioral rerouting, the value of one memory-induced decision under a frozen no-memory continuation policy, and remaining-episode exposure return.

All text conditions branch from one restored ALFWorld state. They range from bare context (H0) to naturally retrieved memory (H4), with structural, generic and mismatched-memory controls in between. A token-matched actionable control (HP) was added later. Each condition's first action is executed and then valued with five memory-free suffix rollouts (K = 5). The actors are Qwen3.5-35B-A3B and 9B, with greedy decoding and a 30-step budget.

Relative to HP, retrieved memory changes behavior on 25.55% and 28.65% of states, against 15.33% and 20.62% for HP versus H1. On a fresh 180-state panel (Fresh180) the H4-HP value contrast is -0.67 pp. Most of those states sit at a floor or ceiling (94.4% and 98.3%). The 23-page appendix adds official-panel intervals, sensitivity analyses, a high-K panel, adapted memory-system components and full calibration records.

Critical strengths
  1. Separating rerouting, one-step value and exposure return is well motivated, and the control ladder is strong. All conditions start from the same state, and the controls include matched generic text, mismatched real memory and a token-matched actionable control. With these, the paper shows that retrieved memory changes behavior well beyond a strong control. The appendix also documents the inference in unusual detail, with calibration grids, leave-one-candidate-out tables and an actor-independence audit.
Critical weaknesses
  1. The value results do not line up across panels, and the main text covers the primary endpoint in one sentence of §5.2 without intervals. In appendix Table 7, the original primary endpoint, official H4-H2, is +2.34 pp for 9B with a 95% interval of [+0.47, +4.20]. In appendix Table 15, official H4-HP for 9B is +1.90 pp, with episode interval [+0.27, +3.53] and Holm p = 0.0397. Its candidate interval [-0.92, +4.71] crosses zero. Descriptively, the 9B effect sits in the seen split (+3.86 seen, -0.15 unseen), and 35B shows the reverse. On Fresh180 the same contrast is -0.67 pp. A post-primary graded-progress estimate for 35B is -2.67 pp [-5.04, -0.29] at the state level, while its equal-candidate version crosses zero. The abstract reports only the Fresh180 value result. The paper's conclusion of "no stable H4 advantage" is fair, but from the main text alone a reader cannot see that the prespecified endpoint's interval excludes zero for one actor.

  2. The paper is very hard to read. The main text relies on provenance labels such as "post-primary" and "contemporaneous". It uses several similarly named panels, including two different 180-state sets, and leaves many of the deciding numbers to the appendix. The scope is also narrow. Results come from one environment and one 94-item memory library, and the writer model is not named. The Synapse-, ReasoningBank- and AdaMEM-style components are compared with their own generic controls on 48 states rather than with HP, and no ranking among them survives Holm correction. Three of four additional model families fail the capability gate, which leaves Gemma as the only cross-family check.

Questions for authors
  1. How should a reader reconcile the positive official 9B contrasts with the Fresh180 result and the 35B graded estimate? Do you read the 9B effect as specific to the seen split?
  2. Could the adapted memory components be compared with HP, so that they bear on the paper's main contrast?
1 comment
AuthorsOpenReview ↗

Reconciling the Value Results, Common Controls, and Presentation

Thank you for identifying the need to reconcile the positive official 9B result with the subsequent findings and to improve the presentation of the experimental evidence. We now report the original and later results together in Section 5.2A, Table 2.

  1. Why do the official and Fresh180 results differ?

On the original official heldout, the prespecified H4−H2 contrast was +2.34 percentage points for 9B (episode 95% CI [+0.47, +4.20]; candidate-CR1 CI [+0.11, +4.56]) and +0.44 points for 35B.

In the later contemporaneous evaluation, 9B H4−HP was +1.90 points (episode CI [+0.27, +3.53]; candidate CI [−0.92, +4.71]). On the episode-disjoint Fresh180 panel, H4−HP was −0.67 points for both Qwen Actors (episode CI [−1.78, +0.45]).

These estimates use different controls, state populations, and dependence assumptions. H2 is the original length-matched generic control, whereas HP provides actionable generic guidance. We retain the positive official 9B finding rather than treating the subsequent near-zero contrasts as replacements.

Regarding the seen split specifically: the contemporaneous 9B estimate is descriptively larger on seen states (+3.86 points) than unseen states (−0.15 points). However, these are descriptive subgroup estimates without multiplicity control or a dedicated interaction test. We therefore cannot conclude that the positive 9B effect is confined to the seen split.

The 35B graded-progress estimate is negative under state weighting, whereas its equal-candidate sensitivity interval crosses zero. Together, the findings indicate heterogeneity across the tested settings, without establishing a uniform direction or a statistically confirmed panel interaction.

  1. Common HP comparisons for adapted memory components.

We performed the requested comparisons for Synapse-style, ReasoningBank-style, and AdaMEM-low adaptations against the common actionable control HP, while retaining their original method-specific generic controls (Appendix L, Tables 28–29).

All six Native−HP point estimates were positive, ranging from +2.08 to +6.67 percentage points, but none survived Holm correction. The evidence is therefore inconclusive and does not support a statistical ranking among the components. These are common-harness adaptations, not official reproductions of the complete systems.

  1. Clarified panel identities and improved readability.

The previously evaluated common 180-state subset is a deterministic subset of the 274 official held-out states. Fresh180 instead consists of 180 previously unused train episodes, with no overlap with the prior targets or memory-source episodes (Appendix B, Table 4).

Section 5.2A and Table 2 now place the original and later estimates together. Figure 2 presents the new single-decision sensitivity evidence, and Appendix F details the post-submission tests and their statistical status.

We also explicitly identify the original memory writer as Qwen3.5-35B-A3B (Section 4; Appendix F). Gemma provides additional cross-family evidence in the new validation studies, although the broader conclusions remain limited to ALFWorld and the frozen 94-item memory bank.

Reviewer_CeEROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MemLadder, a controlled audit protocol for assessing and attributing the effects of LLM-based agent memory. The protocol distinguishes behavioral rerouting, single-decision downstream value, and remaining-episode exposure-policy return as three distinct estimands, and controls for factors such as the prompt channel, generic text, source-grounded memory text, and retrieval match through five mutually exclusive input conditions and a supplementary actionable domain-general control. In the single-decision audit, each condition starts from the same restored ALFWorld state, executes its induced first action, then removes the exposure text and estimates the action’s return under a frozen no-memory continuation policy through repeated memory-free continuations. Experiments show that retrieved memory induces substantial behavioral rerouting relative to the actionable domain-general control, but does not exhibit a stable one-step return advantage on a fresh, episode-disjoint 180-state panel. Further diagnostics indicate that binary endpoint saturation, weak-continuation failure, and strong-continuation recovery can compress observable one-step value differences.

Critical strengths
  1. This paper clearly distinguishes between behavioral rerouting, single-decision downstream value, and remaining-episode exposure-policy return. Final success rates can be affected simultaneously by the prompt channel, generic control text, memory content, exposure duration, and subsequent execution, so the memory/no-memory contrast alone is insufficient to attribute gains to retrieved historical content. By separating these evaluation targets, the paper addresses an important attribution problem in agent memory research.

  2. H0–H4 cover bare context, a structural null, length- and format-matched generic text, mismatched real memory, and naturally retrieved memory. The subsequent addition of HP, an actionable domain-general control, also directly addresses the concern that the original H2 may not be sufficiently actionable. Additionally, same-state execution, exposure-text removal, complete continuation identity checking, and actual environment execution make the comparisons more closely aligned with explicit local interventions. The subsequent interleaving of different conditions within the same service window also improves interpretability relative to the earlier cross-runtime comparisons.

Critical weaknesses
  1. In Fresh180, 170/180 and 177/180 states for 35B and 9B, respectively, are at an empirical all-zero floor or all-one ceiling across H0, HP, and H4. Although H4 and HP differ in complete continuation identity on 54/180 states for each model, only 2/180 states for each model exhibit nonzero empirical value contrasts. Therefore, the current experiments clearly demonstrate that value signals are difficult to distinguish under this setting, but do not sufficiently demonstrate that MemLadder can reliably identify beneficial and harmful memory interventions in realistic agent settings. The additional diagnostics still do not establish a real-task validation setting with a clear single-decision value signal. The expert-minus-random contrasts in Figure 3 exclude zero only for the three- and five-action interventions, which cannot substitute for validating the sensitivity of the single-decision audit.

  2. H4−HP is a more convincing contrast, but H4 and HP may still differ simultaneously in task specificity, entity information, the strength of action cues, and semantic relevance. Length, format, and template matching do not guarantee that these dimensions are also controlled. Likewise, H3−H2 changes both source-grounded content and semantic information, while the interpretation of H4−H3 depends on whether the mismatched memory is indeed less relevant. The authors explicitly acknowledge that these contrasts do not isolate provenance as an intrinsic property or constitute a causal decomposition. This qualification is appropriate, but it also indicates that the granularity of attribution remains limited. In Section 5.1, both H3 and H4 induce substantial behavioral rerouting, but fixed-expert first-action agreement does not show a consistent retrieval-specific advantage across actors. Thus, the existing controls support the conclusion that different text conditions change behavior, but do not yet fully isolate the informational contribution specific to retrieved memory.

  3. The experiments are primarily conducted in ALFWorld, using enumerable admissible textual actions, a fixed history window, an action-selection configuration with thinking disabled, and a limited execution budget. The two primary actors also belong to the same model family. Consequently, the dependence on the audited state distribution and the frozen no-memory continuation policy limits the applicability of the findings to practical memory systems.

Questions for authors

Please see Weaknesses

1 comment
AuthorsOpenReview ↗

Single-Decision Sensitivity, Guidance Specificity, and Attribution

Thank you for highlighting the need to validate whether MemLadder can detect beneficial and harmful single-decision effects in realistic agent tasks. We added targeted experiments while preserving the original held-out analysis (Section 5.2; Appendix F).

  1. Single-decision sensitivity and beneficial versus adverse guidance.

We selected 50 new episode-disjoint ALFWorld states with two admissible first actions that produced different outcomes under a fixed expert controller. The reference actions were selected before observing Actor returns, which were independently estimated under their memory-free continuation policies.

The prespecified three-Actor pooled action-value difference was +14.07 percentage points (95% CI [+6.27, +22.53], fixed-sequence p=.00152).

On the same states, constructed, oracle-informed guidance expressing the correct action intent without its exact executable command exceeded broad action-category guidance by +12.00 points ([+4.67, +20.13], p=.00510).

A prespecified secondary comparison also found that explicit correct-action guidance exceeded misleading guidance by +13.33 points ([+5.60, +21.73], unadjusted p=.00275). However, misleading guidance did not show a confirmed disadvantage relative to HP. Thus, the audit distinguishes higher- and lower-value constructed interventions, without establishing harm relative to every baseline.

  1. Which aspects of guidance carry value?

The information-specificity experiments provide further evidence (Section 5.2C; Appendix F). Correct-action intent exceeded broad category guidance by +12.00 points in the prespecified pooled analysis. Providing the exact command did not show a reliable additional advantage over correct-action intent (−0.27 points, 95% CI [−1.60, +0.80]; secondary analysis).

Oracle-selected visible facts yielded a +6.13-point difference relative to category guidance ([+0.80, +12.13]; unadjusted secondary analysis). An earlier diagnostic did not establish a reliable difference between category guidance and abstract correct rules.

These results indicate that state-specific action intent can convey measurable value without reproducing the exact command. They do not isolate information specificity as an independent causal factor, since the guidance conditions also differ in what they communicate about the current decision.

  1. Retrieved and historical-memory value.

On the same consequential states, H4−HP was +2.27 points (95% CI [−0.40, +5.33]), while oracle-informed correct-intent guidance exceeded H4 by +10.00 points ([+2.53, +18.27]). Both are prespecified secondary comparisons with unadjusted inference (Section 5.2C).

On 120 independent unscreened episodes, H4 changed 29–41% of actions or successor states across Qwen and Gemma, but H4−HP was −0.83, +1.33, and +0.83 points, without robust positive evidence.

We also evaluated unchanged source-grounded memories from the frozen bank. Human-verified relevant memory versus its matched generic control yielded +1.26 points ([-0.89, +3.70], p=.3388) across 90 eligible states. The planned relevant-versus-mismatched study stopped before Actor execution because only 4/120 proposed pairs passed its predefined quality gate.

  1. Attribution and scope.

We agree that H4−HP does not independently isolate historical provenance, semantic relevance, entity information, and instruction strength. H3 mismatch also does not guarantee irrelevance. The new guidance experiments validate conditional measurement sensitivity, rather than pure historical-provenance attribution.

The additional Gemma evaluation provides cross-family evidence, but the experiments remain limited to ALFWorld, the audited memory bank, and the tested continuation policies. These limitations are stated explicitly in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and New Validation Evidence

We thank the reviewers for their detailed feedback. We have uploaded a focused revision that preserves the original three estimands, five-arm protocol, official held-out results, and statistical identities.

The main changes are:

Clearer presentation of the original results. Section 5.2A and Table 2 now present the positive official 9B result alongside the subsequent actionable-control and independent-panel findings, without pooling different controls or state populations.

Real-task single-decision validation. On 50 new episode-disjoint consequential states, the prespecified pooled tests detect a +14.07-point difference between independently selected first actions (95% CI [+6.27, +22.53]) and a +12.00-point advantage for constructed, oracle-informed correct-action guidance over broad action-category guidance ([+4.67, +20.13]). On the same states, H4−HP is +2.27 points ([-0.40, +5.33]; prespecified secondary, unadjusted). These findings are reported in Section 5.2B–C and Appendix F.

Additional memory and component evaluations. Independent unscreened episodes and human-verified historical-memory tests show substantial behavioral changes without a robust mean value advantage. We also added common-HP comparisons for three adapted memory components (Appendix L).

Improved readability and documentation. Key terminology, panel chronology, model configurations, and evidence status are now clearer. The supplementary material is included in the same PDF.

The post-submission diagnostics supplement rather than replace the original primary analysis. Unsuccessful tests and experiments stopped at predefined quality gates are also documented.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.