- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Previous comment
AMIE adapts the index structure of an agent's memory online as the query distribution evolves (merging, splitting, and reorganizing entries) instead of using a static index. The method comes with a clear algorithmic description and theoretical analysis (complete proofs in the appendix). Experiments across dialogue-memory benchmarks and backbones show consistent but small gains (mostly <2pp, roughly 0.7–3.4pp), supported by significance tests and ablations.
Updated:
AMIE maintains persistent per-memory suppression state to adjust future retrieval scores without changing stored content. It is evaluated across five state-of-=the-art memory systems and three benchmarks. Therefore, my original description of merging and splitting entries was incorrect.
Critical strengths
Separating memory content from retrieval adaptation is very useful. The motivation is sound. The added cost measurements and future-query evaluations protocol make the practical benefits clearer, particularly for the limited LLM budget.
Critical weaknesses
Small effect sizes:Most gains are under 2pp (≈0.7–3.4pp); the paper does not sufficiently argue when the evolution overhead is worth paying. Overhead vs. assumptions:Maintenance and extra LLM-call costs are not systematically reported; the stationarity assumption behind the theory is not validated on real agent workloads.
Update:
However, the benefits remain dependent on the retrieval setting. The decay reduces but does not eliminate this count loss (1.71 Recall@10pp). Besiodes, the semantic policies provide the better selectivity at additional inference cost.
Questions for authors
- How do gains and overhead scale when the memory store grows 10×?
- What is the full cost budget of index evolution (LLM calls, latency, storage)?
- Do the guarantees and gains hold under non-stationary query distributions (bursty topics)?
- Are there settings with larger effect sizes (long-horizon personalization, cross-session tasks)?
2 comments
Thanks authors for the very detailed responses. The updated experiments about scaling, cost, and future-query results address my main questions and concerns. I also correct mistakes in my review, like AMIE does not merge or split memories. However, the related near-domain failures and semantic-policy costs still limite the performance. Therefore, I will keep my score at 3.
Follow-up on Rebuttal
We thank the reviewer again for the thoughtful feedback. As the discussion phase is approaching its end, we would like to briefly follow up. We have provided additional experiments addressing your concerns about the practical significance of AMIE's gains, computational and memory overhead, robustness under non-stationary query distributions, and benefits in longer-horizon and larger-scale memory settings. Specifically, we report detailed latency, token, and storage measurements; evaluate memory stores expanded up to 21× and multiple non-stationary query schedules; and demonstrate larger retrieval gains on held-out future queries, cross-session tasks, and previously unseen evidence.
We hope these additional results and clarifications satisfactorily address your concerns. If any questions remain, we would be very happy to provide further clarification or discussion before the discussion phase concludes.