- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Agents, today, have to contend with long running session with long history sizes well beyond the context window of the model. This typically also often involves dealing with evolving facts, facts that get updated over the course of the session. Contract Memory Compiler (CMC) deals with such scenarios of long running sessions with evolving facts and how to handle evidence retrieval for the agent in such scenarios. The paper proposes to extract source-linked relations from the session history. It maintains only the latest version of (object, source position) corresponding to a specific (subject, relation) pair ("keep latest" approach). For each question, it determines the initial subject anchors, and then expands and fetches more information in a breadth first manner using the object to provide evidence for the next subject. The "keep latest" approach is the method of choice for conflict resolution. This simple mechanism is providing good performance improvement for the system.
Critical strengths
- Simple Mechanism: The approach is roughly session level GraphRAG with additional step to resolve conflicts. This conflict resolution is simply done by maintaining the latest version of (object, source position) corresponding to every (subject, relation) pair. This simple mechanism is showing good performance suggests that there is no complex machinery is needed for performance. However, it also puts a high premium on quality of graph extraction.
- Fairly Clean Explanation: The paper explains the algorithm fairly well and it was easy to follow the steps. However, the experiment section is very oddly structured.
Critical weaknesses
- Unclear Choice: The core approach of this paper is very similar to APEX-MEM but substantially simpler. However, paper has not clearly justified why this simplicity is enough.
- Baseline Comparison: The paper should compare with APEX-MEM and show how it compares against this approach.
- Baseline Models: The proposed approach seems to be using Hy4 model whereas the reported numbers used in baseline are using a different model (gpt-4o-mini/gpt-4.1-mini,gpt-5-mini), so it is very hard to understand the contributions as the comparison do not exactly have a single confounding factor.
Questions for authors
- What are the distinguishing points of the proposed approach with prior approaches like APEX-MEM, Zep/Graphiti?
- Why is "keep the latest" approach sufficient? or are there regimes under which simply "keep the latest" will not be sufficient?
- How does the proposed approach compare with these pre-existing approaches on the benchmark?
- If you reproduce some of the SoTA baselines of MemoryAgentBench with the same set of models as used in your approach, how much do they change and how does your proposed approach compare with these methods in that setting?
5 comments
Follow-up on Q1 (APEX-MEM), Q2 (why simplicity suffices), and Q4 (baseline shifts)
Dear Reviewer viu1,
Thank you again for your review. While reflecting on your questions -- specifically how CMC differs from APEX-MEM (Q1), why a simple mechanism suffices (Q2), and how much MemoryAgentBench baselines change under HY4 (Q4) -- we wanted to share three brief clarifications that complement our response above:
-
Why simplicity is the insight, not a shortcut (Q2, and your Critical Strength) In your Strengths, you kindly noted that CMC's simple mechanism shows "no complex machinery is needed for performance." What surprised us, however, is that existing systems still miss this behavior even when given clean current facts. On MQuAKE-MemStream (Table 3, Qwen3.5-4B), passing all resolved current facts to the reader with zero truncation gets 98.75% single-hop but only 53.38% multi-hop, vs. 96.62% (+43.23 pp) after CMC compiles the reachable closure. Over the same keep-latest fact bank on HY4, BGE-M3 dense retrieval reaches only 33.25% MH at 1,535 tokens vs. 78.25% for CMC at 185 tokens (Table 9), and multi-step SH-Conflict/CAR (1,301 LLM calls) reaches 60.75% vs. 78.25% (Table 10). Showing that this lightweight contract is both necessary and transferable (+11.00 pp when added to GraphRAG and Graphiti at matched length, Figure 4) is the central finding of the paper.
-
Deeper architectural contrast with APEX-MEM (Q1, W1) We see APEX-MEM (Banerjee et al., ACL 2026) as solving a broader, complementary problem. Where tasks need temporal intervals, duration math, or multi-day event aggregation, append-only event graphs and GraphSQL excel -- and CMC's overwrites fall short (as we show on Memora, 20.68 vs. 51.96 for BM25, App. K.13). Conversely, for long histories (>1,000 docs), APEX-MEM's online mode pre-filters documents by query relevance before graph construction and relies on a multi-step ReAct agent (up to 40 tool calls). When an update redirects a multi-hop query to an entity absent from the question, pre-filtering by the question risks omitting the downstream record before traversal starts, whereas CMC compiles the full stream offline (13.8 ms read, zero runtime tool planning). We searched carefully for official APEX-MEM code (including the authors' page) to include a matched run; since none is publicly released yet, we kept the comparison architectural rather than guessing an unverified reimplementation.
-
How much MemoryAgentBench baselines shift under HY4 (Q4) Comparing Table 30 (published) with Table 4 (HY4-preview, disclosed interfaces) validates your intuition on static tasks while isolating the update bottleneck: moving to HY4 strongly lifts baselines on static recall (BM25 rises 66.0 -> 78.0 on SH-QA and 52.1 -> 77.5 on DetQA; Mem0 rises 32.0 -> 48.0 on MH-QA; Self-RAG rises 31.8 -> 67.8 on EventQA). Yet on evolving multi-hop facts (FC-MH, 262K), every baseline stays near zero: BM25 goes 3.0 -> 2.0, Mem0 2.0 -> 0.0, Self-RAG 3.0 -> 1.0, and RAPTOR 1.0 -> 0.0, vs. 61.0 for CMC (and 12.0 for CAR, 3.0 for MAGMA). Conversely, downgrading CMC to Qwen3.5-4B still gives 76.50% MH (Table 2).
We sincerely appreciate your constructive feedback, which motivated these unified-backbone and cross-retriever controls. We hope these clarifications address your remaining concerns, and we would be grateful if you might reconsider your assessment in light of the revised evidence.
Thanks for taking the effort to address the concerns
If a meeting moves from Room A (floor 2) to Room B (floor 5), selecting before the move gets
meeting -> Room AandRoom A -> floor 2. Refreshing those keys givesmeeting -> Room BandRoom A -> floor 2: every fact in context is current, yetRoom B -> floor 5is missing.
I am confused why Room B -> floor 5 is missing. Sorry, but the other arguments on GraphRAG are also quite not fully clear to me.
Adding CMC's 5-hop current closure to retrieved records (shared rendering and HY4, 800 Qs; Figure 4, K.12) lifts GraphRAG MH from 44.00% to 54.75% (+10.75 pp) and Graphiti from 18.75% to 55.25% (+36.50 pp, also adding semantic anchors).
Surprising complementarity here, it might be worth a more deeper note in the paper.
In Table 4, all methods use HY4-preview (FC is the 262K 200-Q split; outside SF, CMC uses window/card/note interfaces, adaptations disclosed):
Thanks for this comparison.
Based on the above, I have adjusted my scores to weak rejection. However, given that some of the aspect above are still not clear, I will reserve moving the score any further up.
Response to Reviewer viu1: Resolving Before Traversal
Thank you for the candid review. Two critiques of our draft were spot-on: the experiment section was hard to follow, and mixing HY4 with published GPT-4o/4.1/5-mini baselines confounded the method with the model. In the revision, Table 4 runs every baseline on HY4-preview, Table 2 isolates reader swaps, and cross-backbone published scores move to Appendix L.
Where we respectfully disagree is on viewing CMC as simply "session-level GraphRAG with keep-latest" or a stripped-down APEX-MEM. Keep-latest (LWW) and BFS are standard primitives, and we claim neither as new. What prior evaluations leave untested is how an update changes which downstream records must be retrieved, not just what value a retrieved key holds.
1. Why "GraphRAG + keep-latest" is not enough (Unclear Choice, Q1, Q2)
If a meeting moves from Room A (floor 2) to Room B (floor 5), selecting before the move gets
meeting -> Room AandRoom A -> floor 2. Refreshing those keys givesmeeting -> Room BandRoom A -> floor 2: every fact in context is current, yetRoom B -> floor 5is missing. Much as IRCoT (Trivedi et al., ACL 2023) showed retrieval depends on intermediate reasoning, single-call evidence selection requires resolving updates before traversal.On all 800 FactConsolidation Qs (400 SH/400 MH) with frozen extraction and HY4:
2. Plugging CMC's completion step into GraphRAG and Graphiti (Q1, Q3)
Adding CMC's 5-hop current closure to retrieved records (shared rendering and HY4, 800 Qs; Figure 4, K.12) lifts GraphRAG MH from 44.00% to 54.75% (+10.75 pp) and Graphiti from 18.75% to 55.25% (+36.50 pp, also adding semantic anchors). Fresh controls fix extraction, current graph, anchors, base records, and HY4, padding baselines with BM25 to match CMC's per-query length:
Both gain +11.00 pp at matched length (component-level on dev histories, excluding native ingestion).
3. Why simpler than APEX-MEM & vs. Zep/Graphiti (W1, W2, Q1)
APEX-MEM stores event histories queried at runtime by a multi-tool LLM agent; Zep/Graphiti track temporal validity with hybrid search. While multi-step agent tools can chase updated hops, CMC shows that for single-valued updates query-time LLM planning is unnecessary: deterministic CPU traversal on the resolved graph prepares evidence for one LLM call. APEX-MEM has no public code (unlisted on its author page) for a matched run; we evaluated open-source Graphiti rather than hosted Zep.
4. Same-backbone baselines and reader swaps (Weakness 3, Q3, Q4)
In Table 4, all methods use HY4-preview (FC is the 262K 200-Q split; outside SF, CMC uses window/card/note interfaces, adaptations disclosed):
On all 800 FactCon Qs (Table 29), HY4 MAGMA, G-Walk, and MeLLo get 4.00-4.25%, 40.25-57.25%, and 3.75-9.25% MH vs. 78.25-78.50% for CMC. Replacing HY4 in CMC (Table 2) still gives 76.50% MH on Qwen3.5-4B (BM25: 20.50%) and 55.00% on Llama-3.1-8B (BM25: 11.50%), showing the gain is not an HY4 artifact.
5. Where keep-latest breaks & extraction sensitivity (Q2)
Keep-latest suffices only for ordered, single-valued overwrites with post-update traversal. It breaks on concurrent values, conflicts, out-of-order logs, or event aggregation: on Memora, overwriting dated step counts drops CMC to 20.68 FAMA vs. 51.96 for BM25 (K.13). Synthetic 10% omissions, splits, or flips lower MH from 78.25% to 56.00%, 55.50%, or 52.25% (K.5).
We hope these results clarify CMC's contribution and merit a fresh look.