- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies experience-derived textual memory for LLM agents. It compares raw trajectories, reflections, rules, and procedural skills across six benchmarks, and examines memory writers, experience sources, evolution budgets, and maintenance policies on benchmark subsets. The acting model is held fixed, and memory is frozen during held-out evaluation. The reported findings include benchmark-dependent representation rankings, strong performance from raw trajectories, non-uniform benefits from different writers and maintenance policies, and advantages of success-derived memory. The intended contribution is a set of empirical design principles for agent memory.
Critical strengths
-
The paper addresses a consequential design question through a useful comparative framework. Organizing the study around memory content, writer, and experience source makes the design space easier to analyze. The six-benchmark representation comparison, shared acting model, and frozen-memory evaluation provide practical information for researchers choosing memory designs.
-
Several observed effects are substantial. Skill memory improves ALFWorld success from 58% to 79%, Rule memory improves AppWorld from 20% to 34%, and success-only versus failure-only Rule memory differs by 38 percentage points on ALFWorld. These results show that memory design and source selection can materially affect downstream performance and motivate more controlled investigation.
Critical weaknesses
-
The protocol does not fully isolate the factors that the paper aims to disentangle. Section 3.1 describes an online loop in which memory influences subsequent rollouts, which then update memory. Changing the representation or writer can therefore also change the experience used to construct later memory. This supports comparison of complete online configurations, while isolating the transformation of experience requires holding the underlying trajectories fixed. Similarly, collecting 100 qualifying episodes for success-only and failure-only conditions does not ensure equal total interaction budgets or matched source-task distributions. These distinctions matter because the intended conclusions concern the effects of representation, writer, and outcome source themselves.
-
Some comparative conclusions exceed the numerical and statistical evidence. Lines 375–377 state that writer changes produce a smaller spread than representation changes on all three benchmarks. In Figure 4, however, ALFWorld Rule memory ranges from 57% to 83% across writers, a 26-point spread, while the largest representation spread within an individual writer is 13 points. The stated comparison therefore needs correction or a different, explicit definition. More broadly, repeated runs are reported but the main figures and tables omit uncertainty estimates, making smaller rankings, maintenance effects, and saturation around 50–100 episodes difficult to assess. My weak-rejection recommendation is based on these limitations in the reliability and interpretation of the proposed design principles.
Questions for authors
-
Were the same rollout traces supplied to different representations and writers, or were traces regenerated under each evolving memory? Please clarify what is fixed in the evolution-data protocol and what constitutes an episode. For Table 3, can the existing logs provide total rollout attempts and source-task distributions for the success-only and failure-only conditions? A comparison using successful and failed trajectories from matched tasks, if available, would help assess the outcome-source interpretation without requiring a complete rerun.
-
How was the spread comparison in lines 375–377 calculated, and how does it reconcile with Figure 4's 26-point writer spread for ALFWorld Rule memory? Please also provide existing per-run results or uncertainty estimates for the principal comparisons, preferably using the shared evaluation tasks. Clarifying which rankings, maintenance gains, and saturation claims remain supported would be useful.