- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether a single episodic-memory retrieval topology can serve all query types in long-term LLM agents. The authors introduce CTEG, a typed memory graph containing temporal, causal, and contradiction edges, and evaluate a collection of component ablations on LongMemEval-S.
The main empirical observation is that different query categories prefer different retrieval configurations. Based on this observation, the paper further introduces a three-way query router that selects among associative, precise, and latest-oriented retrieval policies. The study includes per-type analyses, significance tests, an oracle-routing upper bound, and a second evaluation on LoCoMo.
Critical strengths
-
I appreciate the paper's diagnostic perspective. Rather than only proposing another graph-memory architecture, it asks when individual memory components help or hurt, and reports negative results rather than hiding them.
-
The empirical analysis is relatively careful. The paper includes multiple ablations, per-question-type results, bootstrap confidence intervals, McNemar tests, multiple-comparison correction, and a second benchmark. I also appreciate that the authors explicitly report when effects are underpowered or fail to replicate.
Critical weaknesses
- The query-aware routing claim is not validated on a held-out routing setting.
The routing hypothesis and per-type policy preferences are established on the same 500-question LongMemEval-S evaluation set on which the router is subsequently reported. The paper itself notes that this is the full evaluation set rather than a held-out routing split.
This makes the routing result difficult to interpret as generalization. The oracle result is naturally post-hoc, but the practical router also lacks an independent test set for evaluating whether its query-to-policy decisions transfer beyond the data used to identify the per-type retrieval patterns.
This concern is amplified by the fact that the actual router reaches 84.0%, which is slightly below the best fixed policy at 84.6%. Thus, the current experiments demonstrate useful per-type heterogeneity more clearly than they demonstrate the effectiveness of query-aware routing itself.
- The central "no universal memory topology" conclusion is stronger than the evidence currently supports.
The numerical best variant differs across question types, but most of these differences are not statistically established. The paper states that temporal reasoning is the only adequately powered category with a detectable per-type CTEG effect; several other categories are underpowered or statistically indistinguishable.
More importantly, the topology effect does not replicate on LoCoMo, where all CTEG variants are within roughly 0.5 percentage points of one another. This suggests that query-type-dependent topology may be benchmark- or regime-dependent rather than a general property of episodic memory retrieval.
I also find the headline gain over flat RAG somewhat difficult to attribute to graph topology. On LongMemEval-S, removing BFS entirely still reaches 83.8%, compared with 70.2% for flat RAG, while the best temporal-only BFS system reaches 84.6%. The paper itself notes that most of this gain comes from the shared retrieval/reranking pipeline. Therefore, the evidence more strongly supports the value of the overall structured retrieval pipeline than the specific graph-topology claim.
Questions for authors
-
How should the query-aware routing result be interpreted without a held-out routing split? Was the mapping between query types and retrieval policies fixed before inspecting the LongMemEval-S per-type results, or was it informed by those results?
-
Do the authors have any independent evaluation of the router on a held-out set or another benchmark? This would materially affect my assessment of the routing contribution.
-
Given that most per-type differences are underpowered or statistically non-significant, and that topology differences largely disappear on LoCoMo, would the authors consider narrowing the "no universal memory" claim to a benchmark- or regime-dependent retrieval heterogeneity claim?
-
Could the authors clarify the LATEST policy on preference queries? The paper states that reducing the stale-node multiplier from 0.25 to 0.15 reduces false-staleness suppression, but numerically this appears to impose a stronger penalty on stale memories. I may be misunderstanding the scoring convention.
2 comments
Thank you for the revisions. The held-out results help, but routing does not clearly beat a fixed policy, and most topology effects remain uncertain or fail to replicate on LoCoMo. I maintain my score of 2
Response to Reviewer 2u8S
We thank the reviewer for the careful, sympathetic reading and for valuing the diagnostic, negative-result framing. Your two bolded concerns - the held-out routing split and the strength of the "no universal memory" claim - are exactly right, and we have addressed both directly. A revised PDF is uploaded; changes are summarized in our top-level common response.
W1 / Q1–Q2 - No held-out routing split.
You are correct that the per-type policy mapping and the router were reported on the same n=500 set. We add a held-out evaluation (Section 4.5, Table 4): we stratify the 500 questions by type, learn the per-type best-variant mapping on the training split only, and evaluate on the disjoint test split, over 200 random splits (50/50 and 60/40). On 60/40, held-out test accuracy is: best fixed system (selected on train) 83.8±2.0; train-derived routing 84.8±2.2; label-free router cm_router 84.1±1.8; test-set oracle (ceiling) 87.6±1.8.
On magnitude we are candid: out of sample the advantage over the best fixed system is small - about +1 pp, within the CIs - so we no longer claim routing as a performance contribution. But train-derived routing wins 80.5% of paired held-out splits, which is exactly the evidence W1 asked for: the per-type structure is real and reproducible out of sample. Specifically:
We therefore reposition routing as a diagnostic instrument, and lead instead with the per-type characterization, which does not depend on the ≈1 pp margin: the optimal topology inverts across query types - CONTRADICTS helps knowledge-update (93.6%) but harms preference (56.7%, below flat RAG); recency (cm_win) beats every graph variant on assistant queries (87.5%); CAUSAL BFS hurts temporal-reasoning while temporal-only BFS helps it.
W2 / Q3 - "No universal memory" is stronger than the evidence.
Agreed, and we now quantify it with a per-type power analysis (Appendix B, Table 6). Only temporal-reasoning (+5.3 pp, p=0.039, power 0.56) and preference (+26.7 pp, p=0.008, power 0.85) reach p<0.05; the other four are directional but underpowered (p≥0.34, needing n≈400–1150). Per Q3 we therefore narrow the claim to a benchmark/regime-dependent statement: on sparse, many-session LongMemEval-S no single topology is optimal across query types; on dense-coverage LoCoMo the effect attenuates (the seven graph variants cluster within ≈1.2 pp) - even as structured memory still beats flat RAG by +12.8 pp (95% CI [10.7, 15.6], p<0.001). This regime-dependence is itself a measured, reproducible result, not a convenient account of a non-replication.
On attribution (removing BFS still reaches 83.8% vs 70.2% for flat RAG): we agree and now state it explicitly. Both cm_ng and flat RAG already include the reranker, so the +13.6 pp gap is the structured candidate pool and blended scoring - not the reranker, and not a single edge type. The gain over flat RAG is the structured-retrieval pipeline as a whole (Section 4.3), with topology re-allocating where it helps per type.
Q4 - LATEST contradiction-penalty direction.
Thank you - you read the numbers correctly; this was a presentation ambiguity. The penalty is a multiplier on a stale node's retrieval score (s←s×penalty), so a lower value suppresses the stale node more strongly. LATEST tightens it from 0.25 to 0.15 = stronger suppression. We now define this convention at first use (Sections 3.2 and 3.5).
With the held-out split and per-type power analysis now in hand - the two soundness concerns you bolded - we believe the central claim is on firm, correctly-scoped footing, backed by reproducible evidence. If these changes address your soundness concern, we would be grateful if you would consider raising your score; we are also happy to run the LongMemEval-M scaling study during the discussion period if you would find it decisive.