- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper argues that comparing agent success with and without memory can't show what the memory contributes because adding memory also changes the prompt format, its length, and how the rest of the episode unfolds. It proposes MemLadder, an evaluation protocol in ALFWorld with two Qwen3.5 models. From the same saved state, the agent takes one action under several conditions: no memory, generic text controls, mismatched memory and correctly retrieved memory. The method then measures whether the action changes and what that action is worth when the rest of the episode is run without memory.
Contributions include:
- Separating rerouting, one-step value and return with memory kept in the prompt. This is a conceptual distinction, not a new method.
- A same-state test with matched controls and repeated memory-free continuations. The protocol works and the intervals are calibrated, but it can't detect even an expert action's one-step value.
- Memory changes behavior, but it shows no value advantage.
Critical strengths
The paper shows that a with/without-memory comparison mixes the memory's content with other effects, such as prompt format and length, and it separates whether memory changes behavior from whether that change adds value.
The controls (length-matched generic text, mismatched real memory, an actionable generic procedure) are well designed
Critical weaknesses
In most states every condition always fails or always succeeds. Even a single expert action can't be distinguished from a random one (Fig. 3). The null result for memory is therefore uninformative: the paper can't separate "memory doesn't help" from "this method can't tell".
The paper isn't clearly written. Key terms ("arm", "post-primary", "continuation identity") are undefined or used before they're defined. The results section is mostly strings of numbers, and the main text points to appendix tables that aren't included. With no clear story, it's hard to tell what was planned in advance and what the main claim is.
Questions for authors
A single expert action can't be distinguished from a random one (Fig. 3). What size of one-step memory effect could this method actually detect?
Why use binary success with near-deterministic repeated runs instead of a graded measure such as progress toward the goal?
Why is one-step value under a memory-free continuation the right thing to measure, when deployed memory stays in the prompt across decisions?
What should a practitioner building agent memory do differently because of these findings?
Detectable Single-Decision Value, Outcome Resolution, and Practical Use
Thank you for the specific questions. We address them in revised Section 5 and Appendices F and M.
Your central concern is that a small retrieved-memory contrast may reflect an insensitive audit. The original one-action expert-versus-random comparison was inconclusive. We therefore selected 50 new episode-disjoint ALFWorld states with favorable and unfavorable actions identified by a separate fixed controller before observing Actor returns.
Under memory-free continuation, the prespecified three-Actor pooled action-value gap was +14.07 percentage points (95% CI [+6.27, +22.53], fixed-sequence p=.00152). Constructed, oracle-informed guidance expressing the correct action intent without its exact command exceeded broad category guidance by +12.00 points ([+4.67, +20.13], p=.00510).
The direct-action contrast survived Actor-specific Holm correction for 35B and Gemma, but not 9B. The guidance result relies on the prespecified pooled test; none of its three Actor-specific contrasts survived Holm correction (Appendix F, Table 13).
On these same states, H4−HP was +2.27 points (95% CI [−0.40, +5.33]), and correct-intent guidance exceeded H4 by +10.00 points ([+2.53, +18.27]); both are prespecified secondary comparisons with unadjusted inference (Section 5.2C). Thus, the audit detected effects of 12–14 points here. The retrieved-memory interval does not support advantages above about +5.3 points, while smaller benefits remain uncertain. This is conditional sensitivity, not a universal detection threshold or evidence of zero memory value.
Binary success was the frozen primary endpoint. Only 2.1–7.1% of continuation identities mixed success and failure among five suffixes under the greedy serving configuration (Section 3.3), so repetition adds limited resolution to a saturated binary measure.
We also reconstructed graded residual progress. On Fresh180, 35B H4−HP was −2.67 points (state 95% CI [−5.04, −0.29]); the equal-candidate interval crossed zero. Graded outcomes reduced some ties but did not reveal a stable positive H4 benefit. In a 48-state sensitivity test, temperature-0.7 continuation changed H4−HP by −0.42 and −1.04 points for 35B and 9B (Appendix M.4).
A common memory-free continuation measures the downstream value of a single memory-induced action. It does not measure a system that keeps memory available throughout an episode. The same-state contrast is a local advantage difference; persistent return additionally depends on the states visited and later actions of each exposure policy (Section 6). We evaluate persistent exposure separately in RQ3.
First compare retrieval with an actionable generic control, rather than relying only on memory/no-memory success. Then check whether consequential action differences are detectable, estimate single-decision value under a declared continuation policy, and separately evaluate persistent return. Indeed, HP alone changes approximately 15–21% of first actions or successor states relative to an empty channel (Section 5.1), illustrating why generic guidance effects should not be credited to retrieved history.
We now define arm (Section 3.1), post-primary (Introduction), and continuation identity (Section 3.3) at first use. Section 4 distinguishes original, pre-submission supplementary, and post-submission analyses; Table 2 presents their value results, and Appendix B documents panel chronology. The complete appendix is now included in the same PDF as the main text. The new single-decision validation appears in the main text (Section 5.2, Figure 2).