- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper addresses the temporal mismatch problem in long-term dialogue memory and proposes the idea of 'Deferred Evidence Organization', and the Chain on Demand (COD) framework. The existing methods construct memory representations such as summaries and graph structures in advance during the conversation phase, fixing the information organization without knowing subsequent queries, which can easily cause information loss. COD keeps the original dialogue records unchanged and defers evidence construction until query time.
Critical strengths
1.The matched-control design is clearly where real effort went. When the allocation rule is changed, the memory, candidate pool, 60-anchor budget, reconstruction pipeline, answer model, and judge are all held fixed—this is cleaner than most of what is done in A-MEM, MemoryOS, LightMem, and SimpleMem. The paired question-level design, conversation-cluster bootstrap, continuity-corrected McNemar, and explicit CIs are all pluses. 2.The choice of controls is well justified. The token-matched random expansion (68.4%) vs. temporal reconstruction (73.0%) directly addresses the question of whether this is merely feeding more text; the protected-prefix variant in B.5 (72.0 / 73.0 vs. full-budget 77.0) shows that the entire budget must remain under set-aware control. This is a valuable mechanistic observation.
Critical weaknesses
1.Tt seems this paper overclaims a lot. Its stated thesis is that “deferred organization is superior to eager organization,” but what is actually tested is whether “set-aware allocation is superior to independent relevance-only allocation.” These are not the same claim. The control condition for the latter—relevance-only allocation—is still deferred. Nowhere in the paper is deferred organization compared against eager organization under controlled conditions. The sentence in the abstract, “These results show that long-term memory can preserve the original conversation while deferring evidence organization…,” is not supported by the experimental evidence that follows. 2.Mechanistically, there is almost nothing new. Decomposed, every component is existing work:
- R + λ·diversity is simply MMR (Carbonell & Goldstein, 1998), which the paper itself cites, except that cosine is replaced by Jaccard; there is no ablation or justification for why Jaccard is used.
- The coverage term (the proportion of uncovered query terms) is standard practice in the IR aspect-coverage/xQuAD line of work.
- Greedy set-aware selection is the textbook solution to submodular maximization.
- The ±2-turn same-session window is exactly the sentence-window retrieval from the 2019-era work and has long been a default operation in conversational QA.
- 'Deferring organization until query time' is, by definition, RAG.
Questions for authors
1.The abstract reports 82.2%, 76.3%, and 86.6%. The actual controlled, matched experiment yields +1.86 points (74.5% → 76.3%). Could you explain/clarify the discrepancy here? 2.In table 2, Mem0 is 36.49%, LangMem is 55.45%, and LangMem on LongMemEval-S is 37.20%—are these numbers consistent with the values reported in the corresponding papers/technical reports? 3.Was the reliability of the judge tested? Both the answer model and the judge are GPT-4o-mini; have you evaluated on newer and/or additional LLMs?
3 comments
thans for your solid experiments. i will raise my socre and wish you gook luck
thanks for revisiting our rebuttal and additional experiments. I sincerely appreciate your feedback and willingness to reconsider your assessment
Rebuttal by Authors
We thank the reviewer for recognizing our controls and analysis. We added MMR, evidence, cross-benchmark, judging, and sensitivity tests.
W1 — Deferred Organization versus Set-Aware Allocation
Set-aware allocation outperforming relevance-only selection does not establish deferred organization as superior to eager memory. CoD preserves original conversations and constructs query-conditioned evidence at inference time; this establishes feasibility, not an eager-vs-deferred advantage.
On 1,986 LoCoMo questions, holding memory, Top-200 candidates, 60 anchors, ±2-turn reconstruction, answer model, and judge fixed, CoD improves accuracy from 74.5% to 76.3%, with 135 wins and 98 losses. We will clarify this distinction in the abstract.
W2 — Novelty Relative to MMR, Coverage Retrieval, and RAG
Diversity, coverage, greedy selection, and temporal windows are established techniques. CoD couples set-aware anchor allocation with same-session reconstruction, potentially recovering turns outside the Top-200 pool. It selects source locations rather than final passages.
W2a. Tuned MMR
On LoCoMo dev100, we tested 12 weights each for Jaccard and cosine MMR with matched anchor budgets and reconstruction. Jaccard reaches 77/100 at α=0.575, matching CoD; cosine reaches 74/100 at α=0.650. We do not claim to outperform tuned MMR.
W2b. Full-Benchmark Ablations and Evidence
We evaluated R/D/C on all 1,986 LoCoMo questions, measuring complete annotated gold-evidence coverage after reconstruction.
Diversity alone hurts, coverage alone helps modestly, and their combination performs best. CoD raises complete gold coverage from 78.10% to 84.39% (+125 questions; +6.29 pp). This is separate from the original paired run.
On dev100, mean recovered gold turns after 20 anchors are 0.90/0.98/0.98 for R/R+C/R+D+C; after 60, 1.12/1.17/1.21. At anchor 10, diversity scores are 0.8279/0.8335/0.8356 for R/R+D/R+D+C. Coverage may recover gold earlier, while diversity alters anchor selection. In
conv-26::qa_105, R misses Caroline's answer-bearing turn 12; CoD anchors at turn 10 and recovers it via ±2 reconstruction, yielding a correct historical judgment. On dev100, 69/84 fully covered questions are correct versus 8/16 without full coverage; this is correlational.W2c. Reconstruction, Parameters, and Budgets
On dev100 (Top-200, 60 anchors), full-gold coverage at windows ±0/±1/±2/±4 is 58/78/81/87% for R versus 59/81/84/89% for CoD; these are coverage, not answer-accuracy, results. We tested 30 (λ,β) settings for selection and three prespecified end-to-end points: R 73/100, CoD (0.06, 0.10) 77/100, and (0.12, 0.20) 76/100. Conversation-disjoint five-fold dev100 validation gives CoD 77/100 versus R 73/100.
Fixed anchors do not imply equal tokens. At 2k/4k/8k/12k surrogate-token caps, R/CoD scores are 29/28, 43/43, 71/77, and 71/75. Gains disappear under tight caps. Selection/reconstruction averages ~0.31 s/query, excluding answering and judging.
Q1 — Absolute Accuracy versus Controlled Gain
The abstract's 82.2% (1,540 non-adversarial LoCoMo), 76.3% (1,986 LoCoMo), and 86.6% (500 LongMemEval-S) are absolute accuracies. The +1.86 pp matched gain compares R 1,479/1,986 with CoD 1,516/1,986.
On matched LongMemEval-S, with Top-200 candidates, 60 anchors, ±2 turns, answer model, and judge fixed, CoD scores 420/500 (84.0%) versus R 392/500 (78.4%), +5.6 pp. This is distinct from the original 433/500 (86.6%) run. Descriptive LoCoMo gains vary by type: open-domain +4.17 pp, temporal +3.12 pp, and single-hop +0.24 pp.
Q2 — Mem0 and LangMem Provenance
The reported Mem0 36.49% and LangMem 55.45% on LoCoMo, and LangMem 37.20% on LongMemEval, come from Table 1 of MemPro: Agentic Memory Systems as Evolvable Programs (Liu et al., EMNLP 2026), under GPT-4o-mini. They are third-party results, not our reproductions. We will clarify their provenance.
Q3 — Judge Reliability and Other LLMs
A blinded Claude Sonnet 5 reassessment of cached dev100 answers yields CoD 76/100 versus R 69/100, preserving the trend under an independent judge; this does not replace full-benchmark or human assessment. With GPT-4o-mini judging, DeepSeek Flash (thinking disabled) scores 1,499/1,986 (75.5%) on LoCoMo and 403/500 (80.6%) on LongMemEval-S; Claude Sonnet 5 as answer model scores 1,538/1,986 (77.4%) and 434/500 (86.8%). These show model compatibility, not allocation gains without matched R runs. We will report model IDs, prompts, decoding, and scoring.
Summary
We will (i) distinguish architectural claims from allocation gains, (ii) clarify retrieval novelty, and (iii) add evidence, sensitivity, and benchmark analyses.
We thank the reviewer for the constructive feedback.