acceptodds
Under review as a conference paper at ICLR 2027

Contract Memory Compiler: Resolve, Then Traverse

Abstract

External memory lets language-model agents answer questions about histories too long for the answer model's context window. Updates create a harder problem than retrieving a recent fact: changing one relation can redirect a multi-hop question to records about an entity absent from the question. We study this update-dependent evidence selection problem and introduce the Contract Memory Compiler (CMC). Before seeing a question, CMC uses a language model to identify relations in the history and record where each one was stated. It applies later updates to determine the current relations, follows them from entities named in the question, and passes the corresponding original records to the answer model in one call. Thus the current state determines which evidence is read, rather than merely refreshing values in a previously selected context. To the best of our knowledge, CMC achieves state-of-the-art multi-hop accuracy on FactConsolidation, reaching 78.25% overall and 61.0% at 262K. With the extracted relations and answer model held fixed, selecting evidence before resolving updates reduces multi-hop accuracy to 21.50%. We also introduce MQuAKE-MemStream, a derived dataset of ordered memory streams built from MQuAKE-Remastered counterfactual cases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, multi-hop question answering, knowledge updates, long-context retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_viu1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Agents, today, have to contend with long running session with long history sizes well beyond the context window of the model. This typically also often involves dealing with evolving facts, facts that get updated over the course of the session. Contract Memory Compiler (CMC) deals with such scenarios of long running sessions with evolving facts and how to handle evidence retrieval for the agent in such scenarios. The paper proposes to extract source-linked relations from the session history. It maintains only the latest version of (object, source position) corresponding to a specific (subject, relation) pair ("keep latest" approach). For each question, it determines the initial subject anchors, and then expands and fetches more information in a breadth first manner using the object to provide evidence for the next subject. The "keep latest" approach is the method of choice for conflict resolution. This simple mechanism is providing good performance improvement for the system.

Critical strengths
  • Simple Mechanism: The approach is roughly session level GraphRAG with additional step to resolve conflicts. This conflict resolution is simply done by maintaining the latest version of (object, source position) corresponding to every (subject, relation) pair. This simple mechanism is showing good performance suggests that there is no complex machinery is needed for performance. However, it also puts a high premium on quality of graph extraction.
  • Fairly Clean Explanation: The paper explains the algorithm fairly well and it was easy to follow the steps. However, the experiment section is very oddly structured.
Critical weaknesses
  • Unclear Choice: The core approach of this paper is very similar to APEX-MEM but substantially simpler. However, paper has not clearly justified why this simplicity is enough.
  • Baseline Comparison: The paper should compare with APEX-MEM and show how it compares against this approach.
  • Baseline Models: The proposed approach seems to be using Hy4 model whereas the reported numbers used in baseline are using a different model (gpt-4o-mini/gpt-4.1-mini,gpt-5-mini), so it is very hard to understand the contributions as the comparison do not exactly have a single confounding factor.
Questions for authors
  • What are the distinguishing points of the proposed approach with prior approaches like APEX-MEM, Zep/Graphiti?
  • Why is "keep the latest" approach sufficient? or are there regimes under which simply "keep the latest" will not be sufficient?
  • How does the proposed approach compare with these pre-existing approaches on the benchmark?
  • If you reproduce some of the SoTA baselines of MemoryAgentBench with the same set of models as used in your approach, how much do they change and how does your proposed approach compare with these methods in that setting?
5 comments
AuthorsOpenReview ↗

Response to Reviewer viu1: Resolving Before Traversal

Thank you for the candid review. Two critiques of our draft were spot-on: the experiment section was hard to follow, and mixing HY4 with published GPT-4o/4.1/5-mini baselines confounded the method with the model. In the revision, Table 4 runs every baseline on HY4-preview, Table 2 isolates reader swaps, and cross-backbone published scores move to Appendix L.

Where we respectfully disagree is on viewing CMC as simply "session-level GraphRAG with keep-latest" or a stripped-down APEX-MEM. Keep-latest (LWW) and BFS are standard primitives, and we claim neither as new. What prior evaluations leave untested is how an update changes which downstream records must be retrieved, not just what value a retrieved key holds.

1. Why "GraphRAG + keep-latest" is not enough (Unclear Choice, Q1, Q2)

If a meeting moves from Room A (floor 2) to Room B (floor 5), selecting before the move gets meeting -> Room A and Room A -> floor 2. Refreshing those keys gives meeting -> Room B and Room A -> floor 2: every fact in context is current, yet Room B -> floor 5 is missing. Much as IRCoT (Trivedi et al., ACL 2023) showed retrieval depends on intermediate reasoning, single-call evidence selection requires resolving updates before traversal.

On all 800 FactConsolidation Qs (400 SH/400 MH) with frozen extraction and HY4:

  • Order of operations (Table 1, App. G): Fixing anchors, 5-hop cap, rendering, and budget, traversing a per-key prior graph then refreshing selected keys gets 97.25% SH but 21.50% MH, vs. 97.25% / 78.25% for CMC (+56.75 pp). When updates do not redirect the path (31 MH items), both get 100%; when they do (369 items), refreshing drops to 14.91% vs. 76.42%.
  • Testing "GraphRAG + keep-latest" directly (Table 21, K.11, K.17): Feeding our keep-latest graph into official GraphRAG Local Search (semantic anchors) lifts MH from 11.00% (historical graph) to 52.00%, still 26.25 pp below paired CMC (78.25%); Graphiti RRF retrieval on the same current graph gets 18.75% MH (CMC 78.50%). Conflict resolution alone misses redirected hops.

2. Plugging CMC's completion step into GraphRAG and Graphiti (Q1, Q3)

Adding CMC's 5-hop current closure to retrieved records (shared rendering and HY4, 800 Qs; Figure 4, K.12) lifts GraphRAG MH from 44.00% to 54.75% (+10.75 pp) and Graphiti from 18.75% to 55.25% (+36.50 pp, also adding semantic anchors). Fresh controls fix extraction, current graph, anchors, base records, and HY4, padding baselines with BM25 to match CMC's per-query length:

Component control (800 Qs)MH (%)Mean tokens
GraphRAG + BM25 vs. + CMC completion43.50 vs. 54.501080.74 vs. 1080.65
Graphiti + 1-hop + BM25 vs. + CMC (5-hop)45.50 vs. 56.50888.13 vs. 888.11

Both gain +11.00 pp at matched length (component-level on dev histories, excluding native ingestion).

3. Why simpler than APEX-MEM & vs. Zep/Graphiti (W1, W2, Q1)

APEX-MEM stores event histories queried at runtime by a multi-tool LLM agent; Zep/Graphiti track temporal validity with hybrid search. While multi-step agent tools can chase updated hops, CMC shows that for single-valued updates query-time LLM planning is unnecessary: deterministic CPU traversal on the resolved graph prepares evidence for one LLM call. APEX-MEM has no public code (unlisted on its author page) for a matched run; we evaluated open-source Graphiti rather than hosted Zep.

4. Same-backbone baselines and reader swaps (Weakness 3, Q3, Q4)

In Table 4, all methods use HY4-preview (FC is the 262K 200-Q split; outside SF, CMC uses window/card/note interfaces, adaptations disclosed):

Same HY4 readerMH-QASumDetQAFC-SHFC-MH
HY4 BM2560.018.977.530.02.0
HippoRAG30.020.257.722.07.0
CAR54.06.069.039.012.0
MAGMA66.036.678.941.03.0
CMC77.034.967.695.061.0

On all 800 FactCon Qs (Table 29), HY4 MAGMA, G-Walk, and MeLLo get 4.00-4.25%, 40.25-57.25%, and 3.75-9.25% MH vs. 78.25-78.50% for CMC. Replacing HY4 in CMC (Table 2) still gives 76.50% MH on Qwen3.5-4B (BM25: 20.50%) and 55.00% on Llama-3.1-8B (BM25: 11.50%), showing the gain is not an HY4 artifact.

5. Where keep-latest breaks & extraction sensitivity (Q2)

Keep-latest suffices only for ordered, single-valued overwrites with post-update traversal. It breaks on concurrent values, conflicts, out-of-order logs, or event aggregation: on Memora, overwriting dated step counts drops CMC to 20.68 FAMA vs. 51.96 for BM25 (K.13). Synthetic 10% omissions, splits, or flips lower MH from 78.25% to 56.00%, 55.50%, or 52.25% (K.5).

We hope these results clarify CMC's contribution and merit a fresh look.

AuthorsOpenReview ↗

Follow-up on Q1 (APEX-MEM), Q2 (why simplicity suffices), and Q4 (baseline shifts)

Dear Reviewer viu1,

Thank you again for your review. While reflecting on your questions -- specifically how CMC differs from APEX-MEM (Q1), why a simple mechanism suffices (Q2), and how much MemoryAgentBench baselines change under HY4 (Q4) -- we wanted to share three brief clarifications that complement our response above:

  1. Why simplicity is the insight, not a shortcut (Q2, and your Critical Strength) In your Strengths, you kindly noted that CMC's simple mechanism shows "no complex machinery is needed for performance." What surprised us, however, is that existing systems still miss this behavior even when given clean current facts. On MQuAKE-MemStream (Table 3, Qwen3.5-4B), passing all resolved current facts to the reader with zero truncation gets 98.75% single-hop but only 53.38% multi-hop, vs. 96.62% (+43.23 pp) after CMC compiles the reachable closure. Over the same keep-latest fact bank on HY4, BGE-M3 dense retrieval reaches only 33.25% MH at 1,535 tokens vs. 78.25% for CMC at 185 tokens (Table 9), and multi-step SH-Conflict/CAR (1,301 LLM calls) reaches 60.75% vs. 78.25% (Table 10). Showing that this lightweight contract is both necessary and transferable (+11.00 pp when added to GraphRAG and Graphiti at matched length, Figure 4) is the central finding of the paper.

  2. Deeper architectural contrast with APEX-MEM (Q1, W1) We see APEX-MEM (Banerjee et al., ACL 2026) as solving a broader, complementary problem. Where tasks need temporal intervals, duration math, or multi-day event aggregation, append-only event graphs and GraphSQL excel -- and CMC's overwrites fall short (as we show on Memora, 20.68 vs. 51.96 for BM25, App. K.13). Conversely, for long histories (>1,000 docs), APEX-MEM's online mode pre-filters documents by query relevance before graph construction and relies on a multi-step ReAct agent (up to 40 tool calls). When an update redirects a multi-hop query to an entity absent from the question, pre-filtering by the question risks omitting the downstream record before traversal starts, whereas CMC compiles the full stream offline (13.8 ms read, zero runtime tool planning). We searched carefully for official APEX-MEM code (including the authors' page) to include a matched run; since none is publicly released yet, we kept the comparison architectural rather than guessing an unverified reimplementation.

  3. How much MemoryAgentBench baselines shift under HY4 (Q4) Comparing Table 30 (published) with Table 4 (HY4-preview, disclosed interfaces) validates your intuition on static tasks while isolating the update bottleneck: moving to HY4 strongly lifts baselines on static recall (BM25 rises 66.0 -> 78.0 on SH-QA and 52.1 -> 77.5 on DetQA; Mem0 rises 32.0 -> 48.0 on MH-QA; Self-RAG rises 31.8 -> 67.8 on EventQA). Yet on evolving multi-hop facts (FC-MH, 262K), every baseline stays near zero: BM25 goes 3.0 -> 2.0, Mem0 2.0 -> 0.0, Self-RAG 3.0 -> 1.0, and RAPTOR 1.0 -> 0.0, vs. 61.0 for CMC (and 12.0 for CAR, 3.0 for MAGMA). Conversely, downgrading CMC to Qwen3.5-4B still gives 76.50% MH (Table 2).

We sincerely appreciate your constructive feedback, which motivated these unified-backbone and cross-retriever controls. We hope these clarifications address your remaining concerns, and we would be grateful if you might reconsider your assessment in light of the revised evidence.

Reviewer_viu1OpenReview ↗

Thanks for taking the effort to address the concerns

If a meeting moves from Room A (floor 2) to Room B (floor 5), selecting before the move gets meeting -> Room A and Room A -> floor 2. Refreshing those keys gives meeting -> Room B and Room A -> floor 2: every fact in context is current, yet Room B -> floor 5 is missing.

I am confused why Room B -> floor 5 is missing. Sorry, but the other arguments on GraphRAG are also quite not fully clear to me.

Adding CMC's 5-hop current closure to retrieved records (shared rendering and HY4, 800 Qs; Figure 4, K.12) lifts GraphRAG MH from 44.00% to 54.75% (+10.75 pp) and Graphiti from 18.75% to 55.25% (+36.50 pp, also adding semantic anchors).

Surprising complementarity here, it might be worth a more deeper note in the paper.

In Table 4, all methods use HY4-preview (FC is the 262K 200-Q split; outside SF, CMC uses window/card/note interfaces, adaptations disclosed):

Thanks for this comparison.

Based on the above, I have adjusted my scores to weak rejection. However, given that some of the aspect above are still not clear, I will reserve moving the score any further up.

Reviewer_W37XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies update-dependent evidence selection in long-term agent memory. It introduces Contract Memory Compiler (CMC), which resolves the current state of extracted relations before traversing the resulting graph to retrieve provenance-linked evidence. Experiments on FactConsolidation and the constructed MQuAKE-MemStream benchmark demonstrate substantial multi-hop accuracy gains over stale-state and retrieval baselines. The central contribution is a precise formulation of how memory updates should affect subsequent evidence selection, supported by controlled experiments.

Critical strengths

Clear problem formulation. The paper identifies an important distinction between updating stored facts and updating which evidence should be retrieved. The proposed read contract formalizes this distinction under explicit assumptions.

Convincing controlled experiments. The main comparison holds extraction, reader, and evidence budget fixed, isolating the effect of resolving updates before evidence selection. The substantial multi-hop improvement supports the central claim.

Useful empirical analysis. Matched-budget retrieval comparisons, evidence coverage diagnostics, and reader substitutions strengthen the findings. The additional MQuAKE-MemStream evaluation provides complementary evidence.

Critical weaknesses

Limited methodological novelty. The core implementation combines last-write-wins resolution with bounded graph traversal, both well-established techniques. The formal coverage property largely follows from the traversal definition. Although the controlled experiments establish the importance of operation ordering, the incremental contribution over existing update-aware graph memory systems remains less clear. More direct comparisons with methods such as APEX-MEM, MAGMA, or the G-Walk and MeLLo variants discussed in the paper would better establish the methodological advance.

Limited independent validation. FactConsolidation was used during development, while MQuAKE-MemStream consists of constructed two-hop cases with supplied supporting facts. These settings provide useful controls but leave generalization to independently sampled, less structured memory histories uncertain. Evaluation on held-out histories with paraphrased relations, entity aliases, and branching dependencies would strengthen the evidence.

Insufficient analysis of semantic extraction errors. The proposed guarantees are conditional on correctly extracted and aligned relations. However, structural validation does not ensure semantic correctness, and exact surface-form anchoring may miss relevant entities. An error analysis distinguishing extraction omissions, relation misalignment, anchoring failures, and reader errors would clarify the practical reliability of the approach.

Gap between graph-level guarantees and rendered evidence. Currency and bounded coverage are guaranteed for selected graph edges, but the reader receives original source records together with unresolved text. These records may contain superseded or conflicting information, while budget-based truncation can remove otherwise selected evidence. The frequency and practical impact of these cases remain unclear.

Questions for authors

How does CMC compare with existing update-aware graph retrieval methods under matched extraction, reader, and computational budgets? In particular, what do the G-Walk and MeLLo comparisons reveal about the incremental contribution beyond standard traversal of a resolved graph?

Which design choices were informed by FactConsolidation development results, and do the improvements persist on independently held-out memory histories with more varied relation structures?

What are the dominant sources of remaining errors—fact extraction, relation alignment, entity anchoring, traversal coverage, or answer generation?

How frequently do rendered evidence packets contain outdated or conflicting assertions, and how does the system preserve relevant evidence when the context budget is exceeded?

3 comments
AuthorsOpenReview ↗

Response to Reviewer W37X

Thank you for the generous review and for noting our formulation, controlled design, and empirical analysis. Your four questions pinpointed where we needed stronger comparisons, held-out stress tests, and error/rendering diagnostics. Below is the new evidence in the revision.

1. Matched comparisons & what G-Walk / MeLLo reveal (W1, Q1)

We agree LWW and bounded BFS are standard primitives and that coverage follows from the definition (independent BFS matches CMC on all 6,400 closures). Our contribution is architectural and empirical: even on a resolved graph, ranking or iterative walkers miss redirected multi-hop dependencies unless deterministic post-update closure precedes answering.

  • Same-extraction, same-graph, same-reader controls (Table 21, K.11, K.17): On our resolved graph, official GraphRAG Local Search (semantic anchors) gets 52.00% MH (vs. 78.25% CMC) and Graphiti RRF gets 18.75% MH (vs. 78.50% CMC). Adding CMC's 5-hop completion lifts GraphRAG MH from 44.00% to 54.75% and Graphiti from 18.75% to 55.25% (with semantic anchors). Fixing extraction, graph, anchors, base records, rendering, and HY4 at matched length (Figure 4, K.12, 800 Qs; component-level, no native ingestion), CMC completion adds +11.00 pp:
Matched control (800 Qs)MH (%)Mean tokens
GraphRAG + BM25 vs. + CMC43.50 vs. 54.501080.74 vs. 1080.65
Graphiti + 1-hop + BM25 vs. + CMC (5-hop)45.50 vs. 56.50888.13 vs. 888.11
  • What G-Walk & MeLLo reveal (Tables 5, 29; B.2): Even on a version-resolved bank, iterative LLM walkers compound per-hop anchoring/selection errors: under Qwen3.5-4B (same grammar bank & reader), G-Walk/MeLLo get 30.5% / 8.0% MH vs. 79.00% CMC; under HY4, G-Walk gets 40.25% (57.25% with surface anchors) and MeLLo rises from 3.75% (raw) to 9.25% (resolved) vs. 78.50% CMC (grammar banks vs. CMC LLM extraction, unequal budgets). HY4 MAGMA gets 4.00% MH (4.25% with CMC's prompt) vs. 78.25% (K.15); CAR gets 60.75% (Table 10). APEX-MEM has no public code (unlisted on its author page; K.9).

2. Development exposure & held-out stress tests (W2, Q2)

FactCon informed development, so we do not claim the 5-hop cap, surface anchoring, unresolved retention, or 60K-char cap were benchmark-blind (K.8). Testing the unchanged pipeline on 30 new seeded histories across your six variations (180 Qs; Table 16, K.2):

New variant (30 Qs)CMCFact BM25Full history
Canonical / Branching / Mixed30 / 30 / 308 / 3 / 1717 / 27 / 11
Paraphrase / Alias / Combined11 / 14 / 172 / 16 / 2520 / 14 / 17
Total (/ 180)132 (73.3%)71 (39.4%)106 (58.9%)

CMC handles branching and mixed records cleanly (30/30) and leads overall, with clear limits on paraphrases and aliases (variants share latent histories). On external Memora histories (150 Qs; K.13), CMC trails BM25/full history (20.68 vs. 51.96 / 63.97 FAMA x100, single-HY4 judge; full history uses a larger window) as LWW overwrites dated events (e.g., daily steps).

3. Extraction, alignment, anchoring & reader errors (W3, Q3)

  • Stress tests (K.2): All 19/19 paraphrase errors miss required support. All 60/60 alias/combined items miss surface anchors and use full-history fallback; 29/60 still fail despite full support in context.
  • FactCon audit (K.3, K.5, K.6): Across 18,336 statements, only 126 mismatch/reverse endpoints or stay unresolved vs. templates. Projecting template relation labels onto 1-triple records lifts MH from 78.25% to 80.00% (+7/0 gains/losses) while entity spans give 78.00%, pointing to relation alignment (plus >5-hop chains and reader errors) over span extraction. Synthetic 10% omissions, splits, or flips drop MH to 56.00%, 55.50%, and 52.25% (sensitivity, not natural rates).

4. Graph guarantees vs. rendered text & budget overflow (W4, Q4)

Across 800 FactCon packets (Table 15, K.1), 0/800 conflict on parsed keys and 800/800 fit the 60K-char cap, though 600/800 include unresolved lines; templates flag outdated assertions in 243/800 packets and key conflicts in 13/800. Serial-order rendering absorbs almost all of this: 186/243 flagged packets still answer right, 30/30 mixed-record items succeed despite stale clauses, and omitting unresolved text shifts MH by +1.00 pp (78.25% -> 79.25%, 6 gains/2 losses). Under overflow, keeping the newest suffix does not protect older hops: a 512-char cap truncates 244/800 packets (losing a selected source in 238) and drops MH to 70.25%. Coverage guarantees thus hold before truncation.

Thank you again for your constructive feedback.

AuthorsOpenReview ↗

Brief follow-up on Q1, Q3, and Q4

Dear Reviewer W37X,

As the discussion window enters its final two days, we wanted to share three brief takeaways on your empirical questions (Q1, Q3, and Q4) to complement the detailed tables above:

  1. What G-Walk and MeLLo show beyond resolved-graph BFS (Q1): Even when given a version-resolved one-value-per-key bank under identical Qwen3.5-4B reader and grammar inputs (Table 5), iterative LLM walkers reach 30.5% and 8.0% MH vs. 79.00% for CMC because per-hop anchoring and selection errors compound. Similarly, running official GraphRAG Local Search or Graphiti on our resolved keep-latest graph gives 52.00% and 18.75% MH vs. 78.25% / 78.50% for CMC, while appending CMC's closure gains +11.00 pp on both at matched token length (Figure 4).

  2. Dominant error sources (Q3): Auditing all 18,336 statements alongside template projections (App. K.3, K.6) shows that relation-slot alignment outweighs entity span errors on FactConsolidation (projecting template relation labels lifts MH from 78.25% to 80.00% with 7 gains / 0 losses, vs. 78.00% for spans). On the 180-question stress suite (132/180 overall; 30/30 on branching and mixed clauses), paraphrase misses (19/19) stem from missing support, whereas alias queries (60/60) miss explicit anchors and trigger full-history fallback.

  3. Practical impact of stale clauses in rendered text (Q4): Although template checks flag outdated assertions in 243/800 packets, serial-order rendering absorbs almost all of them -- 186/243 flagged items still answer correctly, 30/30 mixed-record stress cases succeed, and dropping unresolved text shifts MH by just +1.00 pp (78.25% -> 79.25%). Tight truncation (512 chars) lowers MH to 70.25%, confirming our coverage guarantee holds prior to truncation.

Thanks again for the thoughtful review; we hope these findings address your remaining questions and further strengthen your support for the paper.

Thanks for the author's detailed response. The additional experiments address my experimental concerns and strengthen the paper's empirical claims.

However, the concern about methodological novelty and generalization remains. The core approach relies on established graph operations, while the weaker results on paraphrases, aliases, and Memora highlight its dependence on structured, single-valued relations and reliable entity matching.

Thus, I maintain my original rating.

Reviewer_WAsTOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Contract Memory Compiler (CMC), a memory-reading framework for long-term LLM agents operating over evolving memory streams.

Critical strengths
  • The addressed problem is formulated clearly and a simple example is used throughout the paper to guide the readers.
  • The work is well-motivated and timely, addressing an important open problem in language model agents.
  • Strong quantitative results under the selected benchmarks and assumptions.
  • The proposed CMC approach is conceptually simple.
  • The paper contributes a new benchmark dataset, that is a variant of an existing one.
  • Good breadth of ablation experiments.
Critical weaknesses
  • Moderate novelty of the methodological contribution. Methodology consists of various steps that have been used before in related literature, so the contribution is a bit incremental.
  • The assumptions made about memory structure may not always hold in real use cases, as memories can contain conflicting evidence, ambiguity and complicated relationships across multiple entities.
  • More extensive comparisons against the state of the art would be helpful.
  • The value of the proposed "read contract" is unclear.
  • The discussion of results and limitations should be expanded.
  • Figure 1 was not very helpful in providing an overview of the proposed system.
  • Evaluation was performed using a small number of LLMs (Qwen3.5-4B, Qwen3.5-9B, Llama-3.1-8B, HY4-preview)
Questions for authors
  • How would the proposed CMC be integrated to existing pipelines and what effects would this potentially have?
  • How is performance affected by the underlying LLM? How may the size (in parameters) affect performance?
  • How is performance affected in cases of inaccurate relation extraction?
  • How sensitive is the method to the selected depth of hops?
  • The novelty of the proposed approach should be better clarified in relation to existing approaches
3 comments
AuthorsOpenReview ↗

Response to Reviewer WAsT

Thank you for the constructive review and for noting our motivation, guiding example, MQuAKE-MemStream, and ablations. We reorganized the experiments and expanded Sec. 6 and App. K to address your five questions and related concerns.

1. Novelty, value of the read contract & SOTA baselines (W1, W3, W4, Q5)

We agree LWW and BFS are standard primitives and claim neither as new (ordinary BFS matches CMC on all 6,400 closures). What prior work leaves implicit is update-dependent evidence selection: if a meeting moves from Room A (floor 2) to Room B (floor 5), refreshing an earlier selection gives meeting -> Room B and Room A -> floor 2: both facts are current, yet Room B -> floor 5 is missing.

Why formalize a read contract (Currency + Bounded Coverage)?

  • It separates value currency from dependency coverage. In an 800-Q control (Table 1, App. G), traversing a per-key prior graph then refreshing selected keys returns 100% current parsed facts (Currency holds) yet misses redirected hops (Coverage fails), dropping MH to 21.50% vs. 78.25% for CMC (14.91% vs. 76.42% on 369 redirected items). Even on our resolved graph, official GraphRAG Local Search and Graphiti RRF (Table 21, K.11, K.17) get 52.00% and 18.75% MH vs. 78.25% / 78.50% CMC.
  • Expanded SOTA baselines (W3): Table 4 runs 10 baselines on HY4-preview: on FC-MH (262K, 200 Qs) and MH-QA, CMC gets 61.0% and 77.0%, vs. MAGMA (3.0% / 66.0%), CAR (12.0% / 54.0%), HippoRAG (7.0% / 30.0%), Mem0 (0.0% / 48.0%), Self-RAG (1.0% / 37.0%), and RAPTOR (0.0% / 22.0%) (MAGMA leads on MCC 86.8, Sum 36.6, DetQA 78.9; outside SF, CMC uses window/card/note interfaces, adaptations disclosed). On all 800 FactCon Qs (Table 29), HY4 MAGMA, G-Walk, and MeLLo get 4.00-4.25%, 40.25-57.25%, and 3.75-9.25% MH vs. 78.25-78.50% CMC. (APEX-MEM targets event-history tools and lists no public code on its author page, K.9.)

2. Pipeline integration & practical effects (Q1)

CMC connects a record store to a frozen reader: preparation extracts triples and updates the graph once per write, while query reads run on CPU (13.8 ms median at 262K, App. J; 185 mean tokens vs. 1,528 BM25, Table 1) with no query-time LLM planning. Adding CMC's 5-hop completion to existing selectors at matched length (Figure 4, K.12, 800 Qs; component-level, no native ingestion) adds +11.00 pp to both:

Selector + extension (800 Qs)MH (%)Mean tokens
GraphRAG + BM25 vs. + CMC43.50 vs. 54.501080.74 vs. 1080.65
Graphiti + 1-hop + BM25 vs. + CMC (5-hop)45.50 vs. 56.50888.13 vs. 888.11

Trade-offs: Offline extraction uses 1.40M prep tokens on FactCon (K.15) and schema edits need replay (K.8); concurrent online writes are unmeasured.

3. Underlying LLM & parameter scale (W7, Q2)

Across four models (Table 2, K.7, 800 Qs): with shared LLM extraction, CMC vs. Fact BM25 gets 76.50% vs. 20.50% MH (Qwen3.5-4B), 55.00% vs. 11.50% (Llama-3.1-8B; SH: 90.75% vs. 92.00%), and 78.25% vs. 20.75% (HY4). With grammar extraction (Table 20), scaling Qwen3.5 from 4B to 9B on uncompiled Latest state gives 0.00 pp MH gain (22.50% -> 22.50%), whereas 4B + CMC reaches 79.00% (9B: 80.25%): scale alone cannot recover missing hops, though broader LLM families merit testing.

4. Extraction errors & hop-depth sensitivity (Q3, Q4)

  • Extraction noise (Table 18, K.5): Clean CMC gets 78.25% MH; 5% / 10% synthetic noise drops MH to 64.75% / 56.00% (omissions), 66.50% / 55.50% (splits), and 63.50% / 52.25% (flips). Auditing 18,336 statements (K.3, K.6) shows relation alignment matters most (template labels lift MH to 80.00%).
  • Hop depth (App. H, K.4, 800 Qs):
Hop cap h1235
MH (%) / Tokens5.50 / 115.354.00 / 138.471.75 / 158.478.25 / 184.6

Median 262K CPU read stays ~9.71 ms at 1 and 5 hops (Table 17); h=5 is not universally optimal.

5. Real-world structure, limits & Figure 1 (W2, W5, W6)

  • Limits (Sec. 6, K.1, K.2, K.13): On 180 stress Qs (Table 16), BFS handles multi-entity branching (30/30) and mixed stale/current clauses (30/30), but drops on paraphrases (11/30 vs. 20/30 full history) and aliases (14/30). On Memora (K.13), CMC trails BM25 (20.68 vs. 51.96 FAMA x100, single-HY4 judge) as LWW overwrites dated step counts; a 512-char cap drops MH to 70.25%. Concurrent values, conflicts, and event aggregation need richer rules.
  • Figure 1 & Sec. 3: Revised Figure 1's caption and Sec. 3 map offline [1]-[2] (parse + LWW) vs. query-time [3]-[5] (anchor, BFS, source render).

Thank you again.

Thank you for addressing my comments. I have updated my score.

AuthorsOpenReview ↗

Thank you, Reviewer WAsT

Thank you very much for revisiting the paper and updating your score. Your feedback genuinely improved the work, and we are deeply encouraged that the new experiments have addressed your questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author Summary: Memory Extraction and Context Compilation for Multi-Hop Reasoning

We sincerely thank the Area Chair and Reviewers W37X, WAsT, and viu1 for their time and constructive feedback. We are encouraged that Reviewers W37X and WAsT recognized the clarity of our problem formulation, controlled experimental design, and MQuAKE-MemStream benchmark. Across all three reviews, the central questions were how CMC advances beyond standard graph retrieval with conflict resolution, how methods compare under a unified LLM backbone, and where our assumptions hold in practice. Guided by these suggestions, we have revised the manuscript (marked in red) and summarize three overarching takeaways below.

  1. Why LLM agents need CMC: structured memory extraction and context compilation must work together. Long-horizon agents do not struggle with evolving histories merely because of context-window length. Even when all current facts fit in context without truncation, giving an LLM an uncompiled dump of latest facts leaves multi-hop accuracy at 53.38% on MQuAKE-MemStream (vs. 96.62% with CMC, Table 3) and 27.00% on non-overflow FactConsolidation histories (vs. 85.67%, Table 6). Unlocking reliable LLM reasoning requires two capabilities acting in concert:
  • Memory extraction must organize raw session streams into aligned, source-linked relations so state updates can be resolved cleanly (our source-only LLM extractor reaches 78.25% multi-hop accuracy on FactConsolidation, nearly matching hand-crafted grammar rules at 79.75%, Table 7).
  • Context compilation must resolve those updates before traversing dependencies. Simply keeping retrieved facts current is insufficient when an update redirects a query to a new downstream entity: traversing a prior graph and refreshing selected facts returns 100% current parsed facts yet drops multi-hop accuracy from 78.25% to 21.50% (Table 1). Likewise, feeding our already-resolved keep-latest graph directly into official GraphRAG Local Search or Graphiti retrieval yields only 52.00% and 18.75% multi-hop accuracy, vs. 78.25% / 78.50% for paired CMC (Table 21, Appendix K.11, K.17). Paired together, offline extraction and post-update compilation distill histories up to 262K tokens into compact, noise-free source contexts (185 mean prompt tokens across FactConsolidation; 13.8 ms median CPU read at 262K, with no query-time LLM planning), allowing even a lightweight Qwen3.5-4B reader to reach 76.50% multi-hop accuracy (vs. 20.50% for BM25, Table 2).
  1. Plug-and-play context gains for existing memory retrievers. Rather than competing only as a standalone pipeline, CMC's post-update dependency completion can directly improve context quality for existing selectors. Under strict per-query length-matched controls on 800 questions (fixing extraction, current graphs, anchors, base records, and HY4-preview, while padding baselines with BM25 to the same token count), adding CMC completion lifts multi-hop accuracy by +11.00 percentage points on both GraphRAG (43.50% to 54.50%) and Graphiti (45.50% to 56.50%) at equal context length (Figure 4, Appendix K.12; component-level on development histories). Better dependency structure -- not more tokens -- unlocks the reader.

  2. Unified-backbone SOTA comparisons and clear applicability boundaries. Following the reviewers' advice, Table 4 now evaluates CMC alongside 13 baselines under the same HY4-preview reader (separating reader swaps in Table 2 and cross-backbone published references in Appendix L). On the 262K FactConsolidation multi-hop split and MH-QA, CMC reaches 61.0% and 77.0%, compared with 3.0% / 66.0% for MAGMA, 12.0% / 54.0% for CAR, 7.0% / 30.0% for HippoRAG, and 2.0% / 60.0% for BM25 (while MAGMA leads on MCC, Summary, and DetectiveQA; task interfaces outside SF and system adaptations are disclosed). Finally, this extraction-plus-compilation view clarifies CMC's honest boundaries (Section 6, Appendix K): because compilation relies on aligned extraction and single-valued overwrites, accuracy degrades under synthetic extraction noise (K.5), unaligned paraphrases or aliases in our 180-question stress suite (while scoring 30/30 on multi-entity branching and mixed stale/current clauses, K.2), and multi-day event aggregation on Memora where past events must be accumulated rather than overwritten (K.13).

We are deeply grateful to all three reviewers and the Area Chair; your feedback made both our empirical comparisons and our conceptual framing substantially clearer.

AuthorsOpenReview ↗

Closing Thanks to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

As the discussion wraps up, we genuinely want to thank the Area Chair and Reviewers WAsT, W37X, and viu1. Looking back at our first draft and the version we have now, the paper changed a lot because of your questions, far more than we could have fixed on our own.

We really appreciate Reviewer WAsT's support and careful reading from the start. WAsT caught that Figure 1 and Section 3 mixed up what runs offline versus at query time, and pushed us to test where our assumptions break and how the method scales across models. Rewriting that overview and checking 4B-to-9B models alongside our failure case on Memora made the method and its limits much clearer, and we are very grateful for WAsT's encouragement and follow-up during the discussion.

Reviewer W37X's questions pushed us to look inside our errors: why G-Walk and MeLLo still miss steps on an updated fact bank, what happens under paraphrases, aliases,and branching, and whether old facts left inside raw sentences confuse the reader. That showed clearly where relation matching and exact entity matching still fall short.

We are also very grateful to Reviewer viu1 for staying engaged across multiple rounds. Your first review was tough, but you were completely right that our experiment section was hard to follow and mixed HY4 with published GPT scores. Rerunning every MemoryAgentBench baseline on HY4 fixed that issue and made the whole evaluation much cleaner. Your follow-up mattered just as much. When you said our Room B explanation and GraphRAG numbers were still confusing, it made us realize we had been taking that intuition for granted instead of showing it directly. That pushed us to expand Section 6 and add a full check in Appendix M. Readers can now clearly see why retrievers almost always find the first fact yet miss later facts when those new entities never appear in the question. It also makes clear why extra chunks still lower reading accuracy even when the full chain is in the prompt.

Thank you again for taking the time to read our work so closely. We really appreciate how much clearer this discussion helped make the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.