acceptodds
Under review as a conference paper at ICLR 2027

Chain on Demand: Deferred Evidence Organization for Long-Term Conversational Memory

Abstract

Long-term conversational memory faces a timing mismatch: write-time summaries and graphs organize history before a future query is known, while independent retrieval can allocate a limited evidence budget to redundant or fragmented turns. We introduce Chain on Demand (COD), a query-time evidence organization framework over an unchanged turn-level memory. Given a broad candidate pool, COD constructs a temporary evidence chain by allocating a fixed anchor budget according to query relevance, lexical diversity relative to selected anchors, and newly covered query terms, followed by deterministic restoration of each selected anchor’s same-session neighborhood. The resulting chain is used only for the current query, while the underlying conversational memory remains un- changed. In a paired comparison over all 1,986 LoCoMo questions, COD achieves 82.2% accuracy on 1,540 non-adversarial questions and 76.3% on all 1,986 questions. On LongMemEval-S, one fixed configuration that indexes both user and assistant turns obtains 86.6% across all 500 questions. These results show that long-term memory can preserve the original conversation while deferring evidence organization until the information need is known. Our code is available at https://anonymous.4open.science/r/acso3n3A4C4/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, LLM memory, agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mzdzOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses the temporal mismatch problem in long-term dialogue memory and proposes the idea of 'Deferred Evidence Organization', and the Chain on Demand (COD) framework. The existing methods construct memory representations such as summaries and graph structures in advance during the conversation phase, fixing the information organization without knowing subsequent queries, which can easily cause information loss. COD keeps the original dialogue records unchanged and defers evidence construction until query time.

Critical strengths

1.The matched-control design is clearly where real effort went. When the allocation rule is changed, the memory, candidate pool, 60-anchor budget, reconstruction pipeline, answer model, and judge are all held fixed—this is cleaner than most of what is done in A-MEM, MemoryOS, LightMem, and SimpleMem. The paired question-level design, conversation-cluster bootstrap, continuity-corrected McNemar, and explicit CIs are all pluses. 2.The choice of controls is well justified. The token-matched random expansion (68.4%) vs. temporal reconstruction (73.0%) directly addresses the question of whether this is merely feeding more text; the protected-prefix variant in B.5 (72.0 / 73.0 vs. full-budget 77.0) shows that the entire budget must remain under set-aware control. This is a valuable mechanistic observation.

Critical weaknesses

1.Tt seems this paper overclaims a lot. Its stated thesis is that “deferred organization is superior to eager organization,” but what is actually tested is whether “set-aware allocation is superior to independent relevance-only allocation.” These are not the same claim. The control condition for the latter—relevance-only allocation—is still deferred. Nowhere in the paper is deferred organization compared against eager organization under controlled conditions. The sentence in the abstract, “These results show that long-term memory can preserve the original conversation while deferring evidence organization…,” is not supported by the experimental evidence that follows. 2.Mechanistically, there is almost nothing new. Decomposed, every component is existing work:

  • R + λ·diversity is simply MMR (Carbonell & Goldstein, 1998), which the paper itself cites, except that cosine is replaced by Jaccard; there is no ablation or justification for why Jaccard is used.
  • The coverage term (the proportion of uncovered query terms) is standard practice in the IR aspect-coverage/xQuAD line of work.
  • Greedy set-aware selection is the textbook solution to submodular maximization.
  • The ±2-turn same-session window is exactly the sentence-window retrieval from the 2019-era work and has long been a default operation in conversational QA.
  • 'Deferring organization until query time' is, by definition, RAG.
Questions for authors

1.The abstract reports 82.2%, 76.3%, and 86.6%. The actual controlled, matched experiment yields +1.86 points (74.5% → 76.3%). Could you explain/clarify the discrepancy here? 2.In table 2, Mem0 is 36.49%, LangMem is 55.45%, and LangMem on LongMemEval-S is 37.20%—are these numbers consistent with the values reported in the corresponding papers/technical reports? 3.Was the reliability of the judge tested? Both the answer model and the judge are GPT-4o-mini; have you evaluated on newer and/or additional LLMs?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing our controls and analysis. We added MMR, evidence, cross-benchmark, judging, and sensitivity tests.

W1 — Deferred Organization versus Set-Aware Allocation

Set-aware allocation outperforming relevance-only selection does not establish deferred organization as superior to eager memory. CoD preserves original conversations and constructs query-conditioned evidence at inference time; this establishes feasibility, not an eager-vs-deferred advantage.

On 1,986 LoCoMo questions, holding memory, Top-200 candidates, 60 anchors, ±2-turn reconstruction, answer model, and judge fixed, CoD improves accuracy from 74.5% to 76.3%, with 135 wins and 98 losses. We will clarify this distinction in the abstract.

W2 — Novelty Relative to MMR, Coverage Retrieval, and RAG

Diversity, coverage, greedy selection, and temporal windows are established techniques. CoD couples set-aware anchor allocation with same-session reconstruction, potentially recovering turns outside the Top-200 pool. It selects source locations rather than final passages.

W2a. Tuned MMR

On LoCoMo dev100, we tested 12 weights each for Jaccard and cosine MMR with matched anchor budgets and reconstruction. Jaccard reaches 77/100 at α=0.575\alpha=0.575, matching CoD; cosine reaches 74/100 at α=0.650\alpha=0.650. We do not claim to outperform tuned MMR.

W2b. Full-Benchmark Ablations and Evidence

We evaluated R/D/C on all 1,986 LoCoMo questions, measuring complete annotated gold-evidence coverage after reconstruction.

AllocationAccuracyFull-Gold Coverage
R (Relevance)74.57%1,551/1,986
R+D (Diversity)70.64%1,532/1,986
R+C (Coverage)74.97%1,603/1,986
R+D+C76.28%1,676/1,986

Diversity alone hurts, coverage alone helps modestly, and their combination performs best. CoD raises complete gold coverage from 78.10% to 84.39% (+125 questions; +6.29 pp). This is separate from the original paired run.

On dev100, mean recovered gold turns after 20 anchors are 0.90/0.98/0.98 for R/R+C/R+D+C; after 60, 1.12/1.17/1.21. At anchor 10, diversity scores are 0.8279/0.8335/0.8356 for R/R+D/R+D+C. Coverage may recover gold earlier, while diversity alters anchor selection. In conv-26::qa_105, R misses Caroline's answer-bearing turn 12; CoD anchors at turn 10 and recovers it via ±2 reconstruction, yielding a correct historical judgment. On dev100, 69/84 fully covered questions are correct versus 8/16 without full coverage; this is correlational.

W2c. Reconstruction, Parameters, and Budgets

On dev100 (Top-200, 60 anchors), full-gold coverage at windows ±0/±1/±2/±4 is 58/78/81/87% for R versus 59/81/84/89% for CoD; these are coverage, not answer-accuracy, results. We tested 30 (λ,β)(\lambda,\beta) settings for selection and three prespecified end-to-end points: R 73/100, CoD (0.06, 0.10) 77/100, and (0.12, 0.20) 76/100. Conversation-disjoint five-fold dev100 validation gives CoD 77/100 versus R 73/100.

Fixed anchors do not imply equal tokens. At 2k/4k/8k/12k surrogate-token caps, R/CoD scores are 29/28, 43/43, 71/77, and 71/75. Gains disappear under tight caps. Selection/reconstruction averages ~0.31 s/query, excluding answering and judging.

Q1 — Absolute Accuracy versus Controlled Gain

The abstract's 82.2% (1,540 non-adversarial LoCoMo), 76.3% (1,986 LoCoMo), and 86.6% (500 LongMemEval-S) are absolute accuracies. The +1.86 pp matched gain compares R 1,479/1,986 with CoD 1,516/1,986.

On matched LongMemEval-S, with Top-200 candidates, 60 anchors, ±2 turns, answer model, and judge fixed, CoD scores 420/500 (84.0%) versus R 392/500 (78.4%), +5.6 pp. This is distinct from the original 433/500 (86.6%) run. Descriptive LoCoMo gains vary by type: open-domain +4.17 pp, temporal +3.12 pp, and single-hop +0.24 pp.

Q2 — Mem0 and LangMem Provenance

The reported Mem0 36.49% and LangMem 55.45% on LoCoMo, and LangMem 37.20% on LongMemEval, come from Table 1 of MemPro: Agentic Memory Systems as Evolvable Programs (Liu et al., EMNLP 2026), under GPT-4o-mini. They are third-party results, not our reproductions. We will clarify their provenance.

Q3 — Judge Reliability and Other LLMs

A blinded Claude Sonnet 5 reassessment of cached dev100 answers yields CoD 76/100 versus R 69/100, preserving the trend under an independent judge; this does not replace full-benchmark or human assessment. With GPT-4o-mini judging, DeepSeek Flash (thinking disabled) scores 1,499/1,986 (75.5%) on LoCoMo and 403/500 (80.6%) on LongMemEval-S; Claude Sonnet 5 as answer model scores 1,538/1,986 (77.4%) and 434/500 (86.8%). These show model compatibility, not allocation gains without matched R runs. We will report model IDs, prompts, decoding, and scoring.

Summary

We will (i) distinguish architectural claims from allocation gains, (ii) clarify retrieval novelty, and (iii) add evidence, sensitivity, and benchmark analyses.

We thank the reviewer for the constructive feedback.

thans for your solid experiments. i will raise my socre and wish you gook luck

thanks for revisiting our rebuttal and additional experiments. I sincerely appreciate your feedback and willingness to reconsider your assessment

Reviewer_d1ZZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies query-time evidence organization for long-term conversational memory. Chain on Demand (CoD) preserves the original conversation turns. After receiving a query, it retrieves candidates and selects anchors using relevance, lexical diversity relative to the selected anchors, and newly covered query terms. It then restores neighboring turns from the same session, deduplicates them, and orders them chronologically for the answer model. The main system uses a fixed pipeline; iterative evidence acquisition is an exploratory appendix variant.

On all 1,986 LoCoMo questions, with candidates, anchor budget, context reconstruction, and answering and judging procedures held fixed, CoD improves accuracy from 74.5% to 76.3% over relevance-only selection. The difference before rounding is 1.86 percentage points, with a conversation-clustered 95% confidence interval of [0.59, 3.15] percentage points. A single configuration obtains 86.6% accuracy on all 500 LongMemEval-S questions. The authors also provide component ablations, approximately token-matched controls, and cost analysis.

Critical strengths
  1. The method is simple, modular, and practically useful. CoD separates persistent memory storage from query-time evidence organization, without requiring continual construction or maintenance of complex semantic memory structures. Selecting locations before restoring their original neighborhoods preserves local context needed to interpret the conversation and makes the method straightforward to integrate into existing retrieval systems.
  2. The key controls are substantial and the main result has paired statistical support. The full LoCoMo comparison changes only the allocation rule, yielding 135 improvements and 98 regressions, or 37 additional correct answers. Conversation-clustered confidence intervals account for dependence among questions from the same conversation. Approximately matched context volume, random expansion, and component ablations also make a simple explanation based on supplying more text insufficient. The limits of development-set diagnostics and comparisons across evaluation protocols are stated candidly.
Critical weaknesses
  1. The incremental contribution over existing set-selection methods and the mechanism of final evidence improvement need clarification. The combination of relevance, diversity, and lexical coverage is close to existing MMR-style selection. The R+D and R+C ablations are useful, but fixed-weight component comparisons do not replace a strong MMR baseline given an equal tuning budget. Moreover, the allocation objective operates on anchor text, whereas the final input is a union of neighborhoods; anchor diversity need not translate into diversity of the reconstructed evidence. I suggest adding a tuned simple baseline and using the existing paired runs to analyze reconstructed evidence coverage, neighborhood overlap, and representative wins and losses, clarifying the contribution of the added design choices.
  2. External validation of the core gain and the robustness of judging remain limited. LongMemEval-S supports the applicability of the pipeline on a second benchmark, but lacks a matched relevance-only allocator and therefore does not independently establish the benefit of set-aware allocation. GPT-4o-mini is used for both answering and judging, and the paper does not fully specify the prompts, exact model version, and scoring rules. Given the net gain of 37 questions on full LoCoMo, I recommend prioritizing a matched comparison on the second benchmark and a blinded human audit or independent-judge reassessment of examples on which the two systems differ in correctness. These limitations do not invalidate the paired result but constrain its generality and precise interpretation.
Questions for authors

The fixed budget primarily constrains the number of anchors, while the amount of text after neighborhood restoration varies by question. Is there also a hard token limit on the final input? If truncation is triggered, what rule is used, and is it identical for CoD and the relevance-only control? This affects the interpretation of fixed-budget comparisons and practical deployment cost.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing CoD's design and matched evaluation. We added tuned MMR controls, evidence analyses, case studies, matched LongMemEval-S evaluation, and budget tests.

W1 — Contribution Beyond MMR and Reconstructed Evidence

W1a. Tuned MMR

On LoCoMo dev100, we tested 12 weights each for Jaccard- and cosine-based MMR, using identical 60-anchor budgets and ±2-turn reconstruction. Jaccard MMR reaches 77/100 at α=0.575\alpha=0.575, matching CoD (77/100); cosine MMR reaches 74/100 at α=0.650\alpha=0.650. We do not claim superiority over tuned MMR. CoD's focus is evidence construction: anchors select source locations whose reconstructed neighborhoods may contain turns outside the initial Top-200 candidates.

W1b. Evidence After Reconstruction

We extended R/D/C ablations to all 1,986 LoCoMo questions, keeping candidate pools, anchor budgets, and reconstruction fixed.

AllocationAccuracyFull-Gold Coverage
R (Relevance)74.57%1,551/1,986
R+D (Diversity)70.64%1,532/1,986
R+C (Coverage)74.97%1,603/1,986
R+D+C76.28%1,676/1,986

CoD increases complete annotated gold-evidence coverage from 78.10% to 84.39% (+125 questions; +6.29 pp). Thus, allocation changes the evidence recovered after neighborhood reconstruction, not just anchor-level diversity. On dev100 (Top-200, 60 anchors), expanding the window from ±0 to ±4 raises coverage from 59% to 89% for CoD and 58% to 87% for relevance-only. These are coverage measurements, not answer accuracies. Neighborhood-overlap analysis shows no consistent redundancy reduction; evidence completeness and diversity are distinct.

W1c. Wins and Losses

The original matched LoCoMo run contains 135 CoD wins and 98 losses. In conv-26::qa_105, relevance-only misses Caroline's statement on self-acceptance and support (session 7, turn 12); CoD anchors at turn 10 and recovers it via ±2-turn reconstruction. In conv-42::qa_18, CoD similarly recovers the reference to Whispering Falls. Conversely, in conv-43::qa_32, both methods recover Tim's Universal Studios plans, but CoD declines to infer California or Florida and is judged incorrect.

These cases distinguish evidence gains from answering failures. On dev100, CoD answers 69/84 questions correctly with complete gold coverage versus 8/16 without it; this is an association, not a causal estimate.

W2 — External Validation and Judge Reliability

W2a. Matched LongMemEval-S

With Top-200 candidates, 60 anchors, ±2-turn reconstruction, answer model, and judge fixed, CoD scores 420/500 (84.0%) versus 392/500 (78.4%) for relevance-only: +28 correct (+5.6 pp). This complements the LoCoMo matched gain of +1.86 pp (conversation-clustered 95% CI [0.59, 3.15]). The new run is distinct from the originally reported LongMemEval-S 433/500 (86.6%); we will report both separately.

W2b. Independent Judge and Model Robustness

A blinded Claude Sonnet 5 reassessment of cached LoCoMo dev100 answers gives CoD 76/100 versus relevance-only 69/100, preserving the improvement direction under a different judge. As alternative answer models (GPT-4o-mini judge), DeepSeek Flash without thinking scores 75.5%/80.6% and Claude Sonnet 5 scores 77.4%/86.8% on LoCoMo/LongMemEval-S. These demonstrate model compatibility, not cross-model allocation gains. We will report recorded model identifiers, prompts, decoding settings, and scoring rules.

Q1 — Input Token Budget and Truncation

The main protocol fixes 60 anchors, not final input tokens. Both methods apply identical ±2-turn reconstruction, deduplication, and chronological ordering. There is no explicit client-side input cap or truncation; max_tokens=512 limits output. Proxy-side truncation is unverified. On dev100, mean reconstructed-evidence lengths are approximately 8,959 tokens (relevance-only) and 8,969 (CoD), though per-question lengths differ.

We added fixed-evidence-budget tests: neighborhoods are admitted in anchor order only if the complete neighborhood fits the remaining cap, then deduplicated and ordered. At 2k/4k/8k/12k tokens, relevance-only scores 29%/43%/71%/71% versus CoD 28%/43%/77%/75%. CoD gains +6 pp at 8k and +4 pp at 12k, but not under tighter caps. Caps use cl100k_base as a surrogate tokenizer. Selection and reconstruction average about 0.31 s/query, excluding model inference and judging; this is not end-to-end cost.

Summary

We will revise the manuscript to (i) clarify CoD's contribution relative to established set-selection methods, (ii) incorporate reconstructed-evidence analyses and representative cases, (iii) strengthen the discussion of cross-benchmark validation and evaluation reliability, and (iv) specify the input-budget, truncation, and reproducibility protocols.

We thank the reviewer for the constructive feedback, which has helped us more precisely characterize the relationship, while clarifying CoD's applicability and empirical contribution.

Dear Reviewer d1ZZ,

We would like to briefly follow up on our rebuttal and the additional experiments addressing your concerns.

(i) For MMR and evidence quality, tuned Jaccard MMR matches CoD at 77/100, while CoD improves full-benchmark gold-evidence coverage from 78.10% to 84.39%. We also analyzed neighborhood overlap and representative wins/losses.

(ii) For external validation, matched LongMemEval-S evaluation shows a +5.6 pp gain (84.0% vs. 78.4%), and independent Claude Sonnet 5 judging preserves the improvement on dev100 (76/100 vs. 69/100).

(iii) For token budgets, we clarified that the original protocol fixes 60 anchors without client-side input truncation. Additional fixed-budget experiments show gains at 8k–12k tokens, but not under tighter limits.

We would appreciate any remaining questions or an updated assessment if you feel these concerns have been addressed.

Thank you for your time and constructive feedback.

Reviewer_5cKqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces COD, a query-time evidence organization framework that constructs a temporary evidence chain through candidate retrieval, set-aware anchor allocation, and temporal reconstruction. The framework enables a dynamic anchor selection based on relevance, diversity, and coverage. Experiments on LoCoMo and LongMemEval-S shows impressive performance.

Critical strengths

The paper focuses on long-term conversational memory for retrieving coherent evidence from conversation history given a specific query. The proposed framework combines set-aware anchor selection and temporal reconstruction that can be integrated with existing retrieval systems. The paper systematically investigates the sensitivity of evidence volume, temporal reconstruction, and set-aware allocation, providing clear improvement over other memory systems.

Critical weaknesses

The most critical concern is the limited evidence of improvement of the main mechanism. The proposed pipeline includes candidate retrieval, set-aware anchor selection, and temporal neighborhood reconstruction, where set-aware anchor allocation is the most significant contribution. However, Table 4 shows that the proposed set-awaare allocation improves accuracy by 4% compared to "Dense + Temporal", while temporal reconstruction brings most improvement from 57% (Dense-200) to 73% (Dense + Temporal). In addition, a full LoCoMo evaluation (L267-268) shows relatively limited improvement from 74.5% to 76.3% (+1.86 pp). These results suggest that most benefits indeed come from temporal reconstruction, which has been explored in other memory systems, rather than the proposed set-aware allocation strategy. The authors are suggested to clarify whether the allocation strategy brings only incremental improvement. Table 2 and Table 3 report impressive results on LoCoMo and LongMemEval-S compared to other memory systems. However, those results are published from other reports and obtained under different evaluation protocols, resulting in unfair comparisons among these methods. The authors should redesign a unified protocol and evaluate those methods to prove the superiority of the proposed CoD. The combination mechanism of set-aware allocation remains unclear. In Sec 5.2, adding diversity along decreases the accuracy by 3%, while adding coverage along brings no increase. However, combining both diversity and coverage would increase accuracy by 4%. This is an interesting result, and the authors should further analyze the underlying mechanism and explain why the combination works.

Questions for authors
  1. Contribution of set-aware allocation. Table 4 shows a 16 percentage-point improvement from temporal reconstruction (57% → 73%), compared with 4 points from set-aware allocation (73% → 77%) on the diagnostic subset. The matched evaluation on the full LoCoMo benchmark reports a smaller, albeit statistically supported, gain of 1.86 points. Could the authors clarify the practical significance of the allocation strategy beyond established source-context reconstruction techniques? In particular, could they identify which question types benefit most and provide a matched relevance-only ablation on LongMemEval-S to establish whether this benefit generalizes?

  2. Comparability with existing memory systems. The paper acknowledges that Tables 2 and 3 contain cross-protocol results, including differences in evaluation splits. Could the authors evaluate representative strong baselines under a unified protocol, controlling the evaluated questions, answer model, judging procedure, accessible conversation history, and evidence-token budget? Such a comparison would help establish whether CoD’s apparent advantage over existing memory systems persists under comparable conditions.

  3. Interaction between diversity and coverage. Section 5.2 shows that adding diversity alone reduces accuracy from 73% to 70%, while adding coverage alone yields no improvement; combining both increases accuracy to 77%. Could the authors explain and empirically substantiate this interaction? For example, an analysis of selected anchors, evidence redundancy, and gold-evidence coverage before and after reconstruction could clarify how the two terms complement each other. Does this pattern persist beyond the 100-question development subset and across reasonable choices of the diversity and coverage weights?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing CoD's design and controls. We agree temporal reconstruction drives most gains, and add experiments isolating allocation.

W1/Q1 — Allocation Contribution and Generalization

On LoCoMo dev100, Dense-200, Dense+Temporal, and CoD achieve 57%, 73%, and 77%, respectively. Reconstruction contributes 16 pp; allocation adds 4 pp. CoD jointly selects reconstruction locations, potentially recovering turns outside the initial Top-200. Tuned Jaccard MMR also achieves 77/100; we do not claim superiority over tuned MMR.

On matched full LoCoMo (1,986 questions), accuracy rises from 74.5% to 76.3% (+1.86 pp; conversation-clustered 95% CI [0.59, 3.15]), with 135 CoD wins and 98 losses. Among 1,540 non-adversarial questions, gains are +4.17 pp for open-domain (n=96), +3.12 for temporal (n=321), +1.42 for multi-hop (n=282), and +0.24 for single-hop (n=841).

On all 500 LongMemEval-S questions, we fixed Top-200 candidates, 60 anchors, ±2-turn reconstruction, answer model, and judge. CoD scores 420/500 (84.0%) versus relevance-only 392/500 (78.4%), a +5.6 pp gain. This matched run is distinct from the originally reported 433/500 (86.6%). A blinded Claude Sonnet 5 reassessment on dev100 also favors CoD (76/100 vs. 69/100), supporting the trend under an independent judge.

W2/Q2 — Comparability with Memory Systems

We agree that Tables 2–3 contain cross-paper rather than matched system comparisons. Mem0's 36.49% and LangMem's 55.45% on LoCoMo, and LangMem's 37.20% on LongMemEval, are quoted from Table 1 of MemPro (Liu et al., EMNLP 2026), under its GPT-4o-mini setting, not reproduced by us. We will clarify provenance and avoid controlled superiority claims.

We also evaluated LightMem on LoCoMo dev100 using the same answer model and judge. Without client-side truncation, LightMem scores 72/100 versus CoD 77/100. Under a nominal 4k-token cap, LightMem scores 70/100 versus CoD 43/100; however, all CoD inputs were truncated, while LightMem's LLMLingua-2-compressed evidence was not. This suggests greater context needs for CoD, not a unified budget comparison. In separate whole-neighborhood admission tests at 2k/4k/8k/12k surrogate-token budgets, relevance-only versus CoD scores are 29/28, 43/43, 71/77, and 71/75 (dev100). CoD needs sufficient context for reconstructed neighborhoods; we do not claim unified-budget superiority over LightMem.

W3/Q3 — Diversity–Coverage Interaction

We extended matched R/D/C ablations to all 1,986 LoCoMo questions.

AllocationAccuracyFull-Gold Coverage
R (Relevance)74.57%1,551/1,986
R+D (Diversity)70.64%1,532/1,986
R+C (Coverage)74.97%1,603/1,986
R+D+C76.28%1,676/1,986

Full-benchmark results reproduce the dev100 trend (R/R+D/R+C/R+D+C: 73/70/73/77%): diversity alone hurts, coverage alone helps little, and their combination performs best. Complete annotated gold-evidence coverage rises from 78.10% (R) to 84.39% (R+D+C). The accuracy interaction contrast between D and C is +5.24 pp.

In dev100 selection traces, mean recovered gold turns after 20 anchors are 0.90/0.91/0.98/0.98 for R/R+D/R+C/R+D+C; after 60 anchors, 1.12/1.10/1.17/1.21. At the tenth anchor, diversity scores are 0.8279 (R), 0.8335 (R+D), and 0.8356 (R+D+C). This suggests that coverage recovers annotated evidence earlier while diversity changes anchor selection. With Top-200 candidates and 60 anchors, full-gold coverage at ±0/±1/±2/±4 windows is 58/78/81/87% for R versus 59/81/84/89% for CoD. At ±2, using 20/40/60/80 anchors gives 64/75/81/86% for R versus 72/79/84/90% for CoD. These are evidence-coverage results, not answer accuracies.

Cases show both benefits and limits: in conv-26::qa_105, CoD reconstructs an answer-bearing turn missed by R; in conv-43::qa_32, both recover relevant evidence but generate different answers. On dev100, CoD answers 69/84 questions correctly with complete gold coverage versus 8/16 without it; this is an association, not a causal claim. For parameter sensitivity, we tested 30 (λ,β)(\lambda,\beta) combinations at selection time. Three prespecified end-to-end configurations score 73/100 (R), 77/100 (CoD: 0.06, 0.10), and 76/100 (0.12, 0.20). Conversation-disjoint five-fold validation gives 77/100 for CoD versus 73/100 for R; the full grid was not evaluated end-to-end.

Summary

We will (i) separate reconstruction and allocation gains, (ii) add question-type and matched cross-benchmark results, (iii) clarify baseline provenance and token-budget limitations, and (iv) report evidence-recovery trajectories and parameter sensitivity.

We thank the reviewer for helping clarify CoD's contribution and scope.

I also keep my score. good luck

thanks for taking the time to review our rebuttal and additional experiments, we appreciate your consideration and constructive feedback

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and Area Chair for their thoughtful feedback. The reviews recognized CoD's modular evidence construction, clarity, and controlled evaluation. We summarize the discussion record and additional experimental evidence.

ReviewerReview RatingPost-rebuttal Discussion
mzdz3Welcomed the additional experiments and raised the score to 3.
5cKq3Reconsidered the rebuttal and explicitly maintained the score.
d1ZZ3Received a detailed rebuttal and targeted follow-up; no further public assessment is recorded.

The central concerns were allocation novelty and scope, reconstructed-evidence quality, external validation, judging, baseline comparability, and token budgets.

Allocation contribution and claim scope (mzdz; 5cKq). On LoCoMo dev100, Dense-200, Dense+Temporal, and CoD score 57%, 73%, and 77%. Temporal reconstruction accounts for most of this diagnostic gain. On matched full LoCoMo (1,986 questions), CoD improves from 1,479 to 1,516 correct (+1.86 pp; conversation-clustered 95% CI [0.59, 3.15]), with 135 wins and 98 losses. Descriptive gains vary by question type: open-domain +4.17 pp, temporal +3.12 pp, multi-hop +1.42 pp, and single-hop +0.24 pp. Allocation gains are incremental, not proof that deferred organization outperforms eager memory.

Novelty, MMR, and evidence mechanism (mzdz; d1ZZ; 5cKq). We tuned Jaccard and cosine MMR over 12 weights each with matched reconstruction on dev100: Jaccard matches CoD at 77/100, while cosine reaches 74/100. We do not claim superiority over tuned MMR or novelty in diversity, coverage, or temporal windows individually. CoD's focus is selecting source locations whose reconstructed neighborhoods can recover turns outside the initial Top-200. In a separate matched four-way full-LoCoMo ablation, R/R+D/R+C/R+D+C achieve 74.57/70.64/74.97/76.28% accuracy; complete annotated gold-evidence coverage rises from 78.10% (R) to 84.39% (R+D+C), +125 questions. Selection trajectories and traceable wins/losses clarify the evidence-recovery process; final-neighborhood overlap does not consistently decrease. In conv-26::qa_105, CoD reconstructs an answer-bearing turn missed by R. These diagnostics describe evidence recovery rather than proving causality.

Cross-benchmark validation and judge reliability (mzdz; d1ZZ; 5cKq). On a new matched LongMemEval-S evaluation (500 questions), holding Top-200 candidates, 60 anchors, ±2 reconstruction, answer model, and judge fixed, CoD achieves 420/500 (84.0%) versus relevance-only 392/500 (78.4%), +5.6 pp. This is distinct from the originally reported full-pipeline 433/500 (86.6%). A blinded Claude Sonnet 5 reassessment on dev100 yields CoD 76/100 versus relevance-only 69/100. CoD-only DeepSeek Flash scores 75.5/80.6% and Claude Sonnet 5 scores 77.4/86.8% on LoCoMo/LongMemEval-S; these are not matched allocation tests.

System comparability and practical budgets (mzdz; d1ZZ; 5cKq). The Mem0/LangMem results in Tables 2–3 are third-party values from MemPro Table 1, not matched reproductions. Our exploratory LightMem dev100 comparison uses the same answer model and judge but does not control memory construction and effective token budgets: without client-side truncation LightMem/CoD score 72/77, whereas nominal 4k settings yield 70/43, with CoD inputs truncated and LightMem's compressed inputs untruncated. We therefore make no unified-budget superiority claim. The original pipeline fixes 60 anchors, with no explicit client-side input cap; max_tokens=512 limits output. Under separate 2k/4k/8k/12k whole-neighborhood evidence caps (surrogate tokenizer), relevance-only/CoD score 29/28, 43/43, 71/77, and 71/75 on dev100. Conversation-disjoint five-fold validation yields 77/100 versus 73/100 on dev100. We tested 30 weight settings for selection only, with three end-to-end checks.

We thank the reviewers again for their constructive comments, which have significantly strengthened our work. We believe that the additional experiments, analyses, and clarifications address the main concerns raised, and we would be grateful if they could be taken into account in the final assessment. We remain happy to discuss any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.