- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAUSEWAY, a training-free memory framework for long-horizon LLM agents that combines a source-linked bitemporal causal graph, evidence-intervention scoring, answer-dependence auditing, and non-destructive versioning. Its graph path selects evidence using estimated answer sensitivity, graph relations, and temporal validity, and checks generated answers through applicable node-removal and edge-polarity tests, with source replay or abstention when support is insufficient. Task-specific routes use hierarchical session selection for LoCoMo-Plus and native-task readers for MemoryAgentBench. Instantiated with MiMo v2.5, CAUSEWAY achieves the highest Overall scores among the evaluated methods on the fixed evaluation sets: 55.77 on LoCoMo, 87.78 on LoCoMo-Plus, and 70.11 on MemoryAgentBench. A matched LoCoMo ablation reports a 6.30-point improvement in Answerable F1 from verification, with a 1.29-point decrease in answerable coverage. On LoCoMo-Plus, it outperforms Full Context while using 76.99% fewer query-stage LLM tokens, although memory construction costs are excluded.
Critical strengths
- By integrating source-grounded memory, non-destructive temporal versioning, and an evidence verification mechanism, CAUSEWAY enables the handling of factual updates and historical queries, thereby delivering clear practical value for long-horizon agents.
- The framework achieved the highest overall scores across all three benchmark tests. Furthermore, while the introduction of the verification mechanism resulted in a slight decrease in coverage, it improved the "Answerable F1" score. These results strongly substantiate the framework's practical effectiveness and serve as the basis for its high evaluation.
Critical weaknesses
1, Section 3.9 states that CAUSEWAY uses task-specific reading routes across benchmarks, including hierarchical session selection for LoCoMo-Plus and native-task readers for MAB, rather than uniformly executing CE and deletion. However, the main text does not sufficiently explain the functions of these task adapters or clarify whether the baselines receive the same task information and comparable adaptation. I therefore find it difficult to determine how much of the observed improvement comes from the core memory mechanisms versus additional retrieval, routing, or reading strategies. Task adaptation is reasonable, but differences in available information or processing capabilities could affect both the interpretation and fairness of the comparisons. I suggest clarifying the task adapters and input conditions for each method in the main text and providing controlled comparisons that distinguish the contributions of the core mechanisms from those of the task-specific routes.
2, It remains difficult to distinguish the full system’s performance gains from the contributions of its core memory mechanisms. Appendix A.1 states that LoCoMo-Plus uses hierarchical session selection rather than the unified CE scorer/auditor, while MAB routes other than conflict resolution have no recorded CE/deletion audit. Therefore, the overall gains on these benchmarks cannot directly establish the effectiveness of the core intervention and auditing mechanisms. Moreover, public question categories guide CAUSEWAY’s LoCoMo fallback, whereas the category argument is unset in MAGMA’s LoCoMo run; the appendix also explicitly acknowledges that identical metadata-use policies have not been demonstrated across baselines. The Plus route-replacement comparison simultaneously changes candidate granularity, routing guidance, and evidence-token lengths, so it cannot isolate the contribution of the core mechanisms. It remains unclear to me how much of the reported advantage comes from the proposed memory mechanisms versus task-specific retrieval, reading strategies, or additional category guidance.
Questions for authors
What are the storage footprint and peak RAM usage of CAUSEWAY’s source log, graph, historical versions, and caches, and how do they scale with increasing interaction history? Since CAUSEWAY targets long-horizon agents and retains superseded records through non-destructive versioning, could the authors quantify these costs in addition to query-stage LLM usage?
4 comments
Response to Reviewer vGDL (2/3)
W2 (b). LoCoMo-Plus: separating the core mechanisms from granularity, token length and routing.
You noted that the submitted Plus route-replacement changes candidate granularity, routing guidance and evidence length at once. We now vary one factor at a time on the 200-item panel, with the same reader, top-1 selection and 2,048-token cap (Table 4b, App. A.12.4):
| Control | Matched condition | Baseline | Causeway component |
|---|---|---|---|
| Granularity | identical event-level units | 34.0 (cosine) | 92.0 (selector) |
| Token length, raw text | equal tokens per candidate | 35.0 (dense) | 89.0 (constraint-guided selector) |
| Token length, event quotes | equal tokens per candidate | 43.0 (cosine) | 93.0 (selector + event quotes) |
| Online CE | same dense top-3 session pool | 43.0 (no CE) | 61.5 (support-state CE) |
| Online CE + audit | same 3-session hierarchical pool | 93.0 (selector top-1) | 87.5 (CE); 65.0 with deletion audit |
With granularity and per-candidate tokens equalized, the advantage persists, so it is not an artifact of unit size or evidence length. It comes from Causeway's constraint-guided selector and its structured event-quote representation (IDs, dates and source quotes). A session-level online CE improves a weak dense pool by 18.50 points (CI [11.68, 25.41]) and, with deletion audit, cuts wrong answers from 114 to 61. It does not improve on the hierarchical selector, however, which is why the default Plus route omits it. We now state that Plus results support the ledger and selector, not interventional scoring (Sec. 4.5).
W2 (c). MAB.
All methods use the same task-specific output prompts on MAB. As Table 1 shows, CE and deletion audit are active only on the CR route; the other MAB routes are native-task readers over the ledger. The CR evidence isolates ledger mechanisms under a fixed reader and budget (Table 16, 50 items):
- Removing bitemporal versioning lowers CR accuracy from 84.00 to 34.00.
- Replacing causal entity-chain traversal with BM25 lowers FC-MH from 76.00 to 16.00.
For AR, TTL and LRU, we present the gains as system-level results of the ledger plus native readers, not as evidence for CE.
Response to Reviewer vGDL (3/3)
Q1. Storage footprint and peak RAM as history grows.
We ran 45 standalone profiling runs: the ten real LoCoMo histories plus five synthetic scaling regimes, three independent runs per condition. Each run measures disk footprint before and after 12 incremental update batches and peak resident set size (RSS) during ledger load and update (App. A.11.2, Table 27):
| Condition | Active history | Initial disk (MiB) | Disk after 12 updates (MiB) | Peak RSS (MiB) |
|---|---|---|---|---|
| Ten real LoCoMo histories | all ten ledgers in one SQLite DB | 69.87 | – (static) | profiled per history |
| Token scaling | 1,276,000 current tokens, 1 version | 67.42 | 134.85 | 387.65 |
| Version depth | 127,000 current tokens, 20 versions | 68.23 | 136.46 | 451.62 |
Each ledger is stored as a compact serialized JSON payload in SQLite rather than duplicating a dense-vector index per snapshot. Storage therefore grows approximately linearly across the tested token-count and version-depth regimes. At 1.28M tokens, or at 20 retained versions per record, disk stays below 137 MiB and peak RSS below 452 MiB. Non-destructive versioning does raise retention and deletion questions; the Ethics Statement notes that deployments need consent, access control, retention and deletion mechanisms.
Thank you again. The revised paper now states which mechanisms are active on each benchmark, qualifies the LoCoMo comparison with MAGMA under matched metadata, and isolates the core mechanisms from routing on Plus. If these changes address your concerns, we would appreciate it if you would consider raising your score.
Best regards,
Authors
Thank you for the detailed reply and additional experiments. I recognize the positive verification results, but overall, I am still not convinced that the evidence is strong enough to change my recommendation. I will therefore keep my original score.
Response to Reviewer vGDL (1/3)
Dear Reviewer vGDL,
Thank you for the review and for pressing on fairness and attribution. We ran matched-metadata, matched-granularity and matched-token controls, and profiled storage. All changes are in the revised PDF (in blue); numbers below refer to it.
W1. What do the task adapters do, and do baselines receive the same information?
The adapters play two roles only. They set the candidate unit to match the benchmark's context structure, and they format the final answer for the native scorer (short spans for MAB AR/CR, class labels for TTL, conversational responses for LoCoMo-Plus). All methods receive the same question texts and output-format specifications. The revised Sec. 3.4 adds Table 1, which states for each benchmark which Causeway operations are active and which public metadata Causeway uses:
The table makes your point explicit: the full CE + audit path runs on LoCoMo, and only part of it runs on the other benchmarks. The one input asymmetry across methods is public question categories on LoCoMo, which we test directly below. Sec. 4.5 then holds candidate granularity, token budget, reader and metadata fixed to separate the core mechanisms from the routes.
W2 (a). Question-category metadata on LoCoMo.
We reran Causeway, the official MAGMA implementation and LightMem on all 600 LoCoMo questions with the same reader (GLM-5.3-flash). In one arm, categories were hidden from every method; in the other, the same categories were given to every method (App. A.12.5, Table 32):
We report this plainly. With categories hidden from both, Causeway leads MAGMA by 0.65. With categories given to both, MAGMA leads by 1.50, and both systems benefit. Part of the 7.08-point main-table gap on LoCoMo therefore reflects the asymmetric use of categories. The revised Sec. 4.2 states that the two systems are within 1.5 points on LoCoMo under matched metadata. The margins over MAGMA on LoCoMo-Plus (+70.07) and MAB (+33.38) are not affected by this metadata question.