- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper benchmark different memory mechanism on 3 different tasks. Their result claim that there are still a huge headroom between the best memory routers and the best performance achievable using an oracle.
Critical strengths
- The overall question itself is timely; The question of the effect of memory structure with respect to the tasks at hand is under-explored, especially with the rising number of memory paper/method in recent literatures.
- This does not endorse the three main question of the paper or their methodology
Critical weaknesses
- I struggle to see what is the novelty or contribution of this paper
- While I don't take issue with the negative result that they showed, the novelty of the research is weak; They paper describes either methodology from the literature or flawed methodology (see Review 2).
- The novelty itself is incremental. See [1], which is one of the paper they cited and use a very similar methodology with almost no discussion on how this work differ.
- The paper feels AI generated with both weak readability and flawed methodology
- The language is really bad and almost unreadable: dressing up normal language with academic sounding language (e.g. profile eligibility, fixed references), unclear definitions and structure in both the methodology, and results.
- The methodology for similarity is weak: Weak model (all-MiniLM-L6-v2), normalized result (which again I dont think is defensible as routing model will use full result) so similar requests ended up favoring different memories is unsurprising.
- Memory-Substrate-Based Routing by itself is I think not defensible. We see multiple memory routing based works (the authors mentioned quite a lot in the related work section), that performs better than the method in this paper (substrates).
- Given that full context may be more than 32k, the performance (which is less than no memory and compressed KV) reduces my confidence of their results, where they just do blanket zero score for excluded request above 32k context. Also 32k context limit is arbitrary and I think it's lacking especially for long benchmarks.
- Profile eligibility is not defined in main text. It also introduces many confounder (result is biased towards lower context methods, etc). It's really bad that the author manually set the benchmark questions that can be answered. Compressed KV and A-Mem are excluded based on heuristic which distort the result.
- Offline selection is flawed. They run all substrates on prompts, and store it in a "matrix". However benchmark like ALFWorld is online and need actual rollout.
- The baseline itself is weak because of ALFworld issue (see above). Most of the gain can just be traced back to ALFworld.
- Memory routing policy discussed in section 5 is weak. Learned policies are weak and/or elementary without any discussion on SoTA methods.
- Lookup is strong, but its trivial as it gets task metadata (which doesn't make sense for real world usage) and outcomes. "The Aggregate Gain Is Concentrated in the Task-Source Distinction"
- They reported "Held-Out Groups Test More Than Unseen Memory." This is again a failure of the authors to establish a clear train-test split and incredibly confounded by TTL.
- Cost discussion are insufficient. It took an unreasonable amount of time to setup all 12 substrates, a Pareto analysis is warranted to see the gains and losses.
- More flaws are hidden in the appendix, e.g. Compressed KV does not compress at all, it re-prefill some arbitrary selected episodes instead of compressing the actual state from the attention. Furthermore the appendix is unreadable and further increases my concern of this being a fully AI written paper with minimal human supervision.
- Overall they did not distinguish memory mechanism from their implementation and scoring choices, and use a blanked statement of evaluation of memory mechanisms. They included ReasoningBank where its only eligible for ALFWorld, etc.
[1] Huang, et al. Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents. https://arxiv.org/pdf/2608.15008
Questions for authors
- "giving equal weight to the three benchmarks and, within MemoryAgent-Bench, to its four competencies. This prevents LoCoMo’s 76% share of requests from dominating the aggregate." Why not report each bench separately from the start?
- Is there any discussion on confounders introduced (task mix, length bias, etc)?
- Why do the authors think offline selection for ALFworld is defensible?
Comment: Given the severity of the issue of the paper, the probablity of the score being raised barring a huge rewrite and/or methodology convincing during rebuttal will be close to none.
5 comments
W2-d: Thank you for the careful check. We clarify the full-context results as follows:
(1) The 32K limit is the effective context window of the backbone Qwen3.5-9B (Sec. 3), and it applies to every substrate.
(2) The low aggregate score of full context mainly comes from the zero-credit convention. Full context is excluded on ALFWorld (one third of the weight) and on histories beyond 32K, which gives the aggregate 5.74. On LoCoMo, it scores 16.47, above no memory (3.11) and Compressed KV (9.15) (Table B.3), and on the requests the oracle routes to it, it scores 62.37 (Table 3).
(3) This convention does not affect the main results. LightRAG (best fixed, 36.82) and the summary tree (36.11) have no excluded cells, so is unaffected by any eligibility rule, and the oracle only maximizes over observed cells.
In the revised version, we will report scores on covered requests beside the zero-credit scores, and add a truncation fallback so that full context is also evaluated on longer histories.
W2-e: Thanks for the comment. We explain the eligibility rules as follows:
(1) The eligibility mask consists of four rules of the evaluation profile (App. A), and no benchmark question is selected manually. Each rule applies to a whole task source or follows from the context window: ReasoningBank is restricted to ALFWorld (2,346 cells), full context must fit the context window and is excluded on ALFWorld (1,550 cells), and A-Mem and Compressed KV are excluded on test-time learning (100 cells each).
(2) These rules do not change which substrate is the best fixed one. LightRAG and the summary tree have no excluded cells. For A-Mem and Compressed KV, even a perfect score on all 200 excluded cells would give at most and , both below 36.82.
(3) ReasoningBank is restricted to ALFWorld because it distills strategies from task trajectories, which the QA benchmarks do not provide. Its fixed score of 12.94 therefore mainly reflects this scope.
In the revised version, we will define profile eligibility in Sec. 5.1 and list the four rules there as a table.
W2-f & Q3: Thank you for this important comment. We would like to provide responses by separating three cases:
(1) Half of the ALFWorld requests read a static memory. The 134 offline-bank requests use a fixed bank of expert trajectories, so their outcomes do not depend on which substrate served earlier requests, and offline selection is valid for them.
(2) The task-source lookup is scored on an actual rollout. It assigns every ALFWorld request to ReasoningBank, so its ALFWorld score (38.81) is ReasoningBank's own executed column, whose online memory was updated with its own trajectories. No columns are combined, and this gives its ALFWorld contribution of .
(3) Per-request switching under online accumulation needs actual rollouts. and the oracle switch substrates within ALFWorld, so on the 134 online requests their scores are offline estimates. We will add router-in-the-loop rollouts in the revised version, and report offline-bank and online requests separately.
Regarding the baseline, the gain on ALFWorld reflects that the globally best substrate is not the locally best one there (LightRAG 18.28 vs. ReasoningBank 38.81), and a task-source policy makes use of this difference. The QA-only comparison is reported in Sec. 6.2: on the 2,346 QA requests, the lookup and score 45.69 and 46.04 against 46.08 for fixed LightRAG.
W2-g: Thanks for the comment. We have the following discussions:
(1) The lookup is intended as a simple reference, and the finding is that more complex policies do not improve on it. The best of the 23 variants differs from it by with (Table E.1).
(2) The policies follow the main designs of current routers. They cover category classification as in vLLM Semantic Router (), nearest-neighbor retrieval (), per-candidate score regression (), and prompted selection with eight LLMs (). We will add adapted versions of more recent routers (e.g., RouteLLM, RouterDC, GraphRouter) in the revised version.
(3) Policies without the task-source label reach nearly the same score. predicts the category from the request text only. It reaches 43.31 and makes the same choice as the lookup on 98.89% of requests (Table B.14). Removing the source and request-type inputs from changes its score by (Table B.17). As noted in Sec. 6.1, every policy also receives the eligibility mask, which separates ALFWorld from QA requests.
(4) All learned policies, including the lookup, are fitted on training-fold cells only (Sec. 6.1), so no policy uses test outcomes.
W2-h: Thank you for pointing this out. Please kindly notice that both protocols use explicit train-test splits, and the effect of test-time learning is separated from the rest. We provide the details as follows:
(1) For the split, the first protocol evaluates new requests under known memory and keeps identical request texts in one fold. The second holds out whole groups from 29 clusters and also excludes training requests whose text occurs in the test fold (Sec. 6.1, App. B.6).
(2) For test-time learning, Table B.18 decomposes the 7.29-point drop of the grouped protocol by source: test-time learning accounts for 5.58 because its entire competency is held out, whereas LoCoMo and long-range understanding, where a memory unit is held out, account for 0.57 and 0.00.
(3) We also report an unseen-memory test without test-time learning. Leave-one-memory-unit-out on the 20 LoCoMo and long-range units gives 38.47 for against 39.22 for the fixed comparator ( with , Table B.19).
W2-i: Thanks for the constructive suggestion. (1) A gain-versus-cost analysis over substrate subsets is reported in Sec. 6.3, Fig. 3(b) and App. E.3. We grow the substrate set greedily, and report both the oracle and the best policy at each size against the construction time. (2) A small subset already covers most of the headroom. The first three substrates raise the oracle from 36.82 to 51.92, and six substrates reach 61.15 at 82.4 of the 112.1 construction hours, i.e., 94% of the full oracle at about 73.5% of the cost. The last six substrates add only 3.59 points. (3) The policies need even fewer substrates. The best policy scores 43.29 with six substrates against 44.00 with all 12. The benchmark-level rule needs only LightRAG and ReasoningBank, i.e., construction hours (Table B.11), for 43.66 in-sample. In the revised version, we will present these results as an explicit score-versus-construction-hours Pareto frontier in Sec. 6.3.
W2-j: Thank you for the careful reading of the appendix. Our explanations are three-fold as follows:
(1) The construction of is stated in both the main text and App. A. Table 2 lists its readout as "episode re-prefill", and App. A states that utterances are clustered into episodes and the selected episodes are re-prefilled through the backbone's own attention path, so it is not an implementation of a published key-value compression method. App. A also states that and are built for this evaluation rather than reproduced.
(2) has little influence on the results. It is not the best fixed substrate (12.36 vs. 36.82 for LightRAG), and removing it lowers the oracle by at most 2.15 points (Table 3).
(3) Our conclusions are about the substrates as configured in this evaluation. App. A states that the evaluation profile specifies the scope of the comparison rather than a substrate's intrinsic ability, and App. F.2 states that underperformance may reflect the evaluation setup as well as the substrate design.
In the revised version, we will move these statements from the appendix into Sec. 4 next to Table 2, state the scope of the conclusions in the main text, and reorganize the appendix for readability.
We sincerely appreciate Reviewer dwSZ for the detailed and professional feedback. We have carefully considered every point and respond below, grouped by theme in the order of the review. We hope these clarifications address the concerns.
W1: Thank you for pointing this out. We would like to clarify the difference from [1]. [1] compares memory substrates as fixed design choices and reports that none is uniformly best across benchmarks. Our paper studies a follow-up question that [1] leaves open, namely whether this observation can be turned into a gain by choosing a substrate per request. To keep the two studies comparable, we adopt the substrate families and reference configurations of [1], and our contributions are the following three parts:
(1) A per-request outcome matrix for all 12 substrates on one mixed pool of 2,614 requests, with eligibility and observation masks.
(2) A decomposition of the oracle gap by granularity (from 43.66 at the benchmark level to 45.88 at the memory-unit level, leaving 18.87 points at the request level, Fig. 2(b)).
(3) A comparison of 23 routing-policy variants in five families, showing that none improves on a six-row lookup table with an interval excluding zero.
In the revised version, we will add a dedicated paragraph and a comparison table against [1] in Sec. 2.
W2-a: Thank you for the comment. For the terms, "profile eligibility" indicates whether the evaluation profile admits a substrate on a request (App. A). For example, ReasoningBank is eligible only on ALFWorld. A "fixed reference" is the score of using one single substrate for every request, i.e., without routing. We will add these explanations to Sec. 5.1 in the revised version. We will also carefully rewrite the main text and the appendix with simpler language and a clearer structure to improve the readability.
W2-b: Thank you for the comment. We would like to offer the following clarifications:
(1) The similarity analysis in Sec. 4 is a motivating probe, and the main results do not depend on it. The oracle, the headroom and all policy comparisons in Secs. 5-6 are computed from the outcome matrix.
(2) L2 normalization is the standard output of all-MiniLM-L6-v2. It only fixes the vector length, so that the inner product equals the cosine similarity used in Table 1. The direction of the embedding, which carries the semantic information, is unchanged.
(3) The observation that requests favor different substrates does not rely on the encoder. It is also measured directly on the outcome matrix: even granting the best substrate for each memory unit leaves 18.87 of the 27.93 points (Sec. 5.2).
(4) Regarding stronger models, the LLM judges (π5) read the request directly without this encoder, and seven of eight score below the lookup (Sec. 6.2). We will further repeat the analysis of Table 1 with stronger sentence encoders in the revised version.
W2-c: Thank you for raising this point. We would like to offer the following explanations:
(1) These routers and substrate-based routing address different decisions. The routers cited in Sec. 2 (e.g., Self-RAG, Adaptive-RAG, MemFlow) decide whether or how deeply to retrieve within one memory design, whereas we choose among heterogeneous memory designs, such as a knowledge graph, a summary tree or a strategy bank. The two are therefore complementary.
(2) We choose substrates as the action space because they are the unit at which [1] reports no uniformly best choice, which motivates choosing among them per request.
(3) The results reported by these works are obtained with different backbones and benchmarks. For a direct comparison, we will add representative ones as reference columns of our matrix in the revised version.