acceptodds
Desk rejected at ICLR 2027

PAY WHEN ASKED: DEMAND-DRIVEN AGENT MEMORY

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term agents must organize growing archives without knowing which information future queries will need. We view memory as demand-driven computation: retain source evidence while postponing structure until it is useful. Yet query-time construction alone leaves a second cost unresolved: repeatedly rebuilding relations for different questions about the same evidence. We formalize this trade-off by separating global access coverage, local reuse, and answer computation. Our central insight is that sparse global access can coexist with repeated local demand, making selective construction and persistent reuse complementary. Guided by this insight, we introduce SRH-Graph, a demand-driven relation-memory method that constructs source-grounded relations on retrieval, reuses them across distinct questions, and invalidates them when sources change; SRH-Base provides the lightweight retrieval-only variant. On controlled multi-hop chains, SRH-Graph reaches 80.83–100% accuracy where Naive RAG reaches 0–3.33%; reuse reduces repeated construction, while deferral savings vanish near full coverage. On real benchmarks, we find gains on a fixed sparse document stream, but retain Naive RAG's conversational advantage and report higher graph costs. Our results identify when organizing memory on demand is useful, and when simpler retrieval or advance construction remains preferable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Retrieval-Augmented Generation, Demand-Driven Graph Construction, Cross-Query Reuse, Source-Versioned Caching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uZvSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when agent memory should be constructed and reused. It proposes SRH-Graph, which retrieves raw sources, constructs source-grounded relations on demand, caches them across distinct questions, and invalidates them when sources change. SRH-Base provides a retrieval-only alternative. The analysis separates global access coverage, local reuse, and answer computation. Evaluation includes fixed-artifact scheduling, controlled multi-hop streams with traversal and reuse ablations, source updates, high-coverage conditions, and four real benchmarks. The reported results show substantial synthetic multi-hop improvements and savings relative to eager construction or repeated reconstruction, alongside conversational regressions and higher token costs than Naive RAG on real tasks.

Critical strengths
  1. The paper usefully separates avoided preprocessing, persistent relation reuse, and answer computation. The controlled experiments show that these produce different savings: reuse reduces repeated construction, while savings against eager construction diminish near full coverage. The traversal and reuse ablations, source-update tests, and cold-query latency measurements make these operating boundaries informative.

  2. The empirical reporting preserves important negative results and uncertainty. Technical failures remain in the denominators, accuracy and F1 are distinguished, and historical comparisons are explicitly identified. Reporting conversational regressions, scoring sensitivity, and higher costs provides a more useful assessment of the method’s practical limits than aggregate improvements alone.

Critical weaknesses
  1. The distinctive benefit of relation memory over simpler deferred retrieval remains insufficiently established. Deferred construction is already explored by LazyGraphRAG and LazyMem, so the central question is what persistent relation construction and reuse add. In the synthetic study, SRH-Graph performs bounded multi-round expansion, while Naive RAG performs one-pass retrieval. Removing traversal also removes access to the bridging sources; this demonstrates the importance of expansion, but does not establish that graph construction is preferable to iterative text retrieval or query expansion. The paper acknowledges unequal evidence exposure and unavailable matched implementations of some related methods. A budget-matched iterative retrieval control would nevertheless address a consequential alternative explanation for the large synthetic gains.

  2. Real-workload evidence for the proposed operating regime is limited. The clearest positive real-task result comes from one fixed 40-question RepLiQA stream. SRH-Graph regresses on LoCoMo relative to SRH-Base, has no confirmed improvement on ConvoMem, and its small LongMemEval advantage changes direction under the common-Claude scoring sensitivity. Recorded token costs exceed Naive RAG on all four tasks. These limitations are transparently acknowledged, but stronger evidence connecting measured coverage and relation reuse to a useful real-world quality–cost tradeoff is needed to substantiate the practical guidance about when to pay for structure.

Questions for authors
  1. Clarification: What technical or empirical contribution distinguishes the persistent source-versioned relation cache from combining deferred construction with conventional memoization? Please identify the specific benefit beyond the deferred-construction approaches discussed in Section 2.

  2. Evidence request: How does SRH-Graph compare with iterative raw-text retrieval or query expansion under matched retrieval rounds, admitted-source limits, answer-context budgets, and answer models? This comparison would help determine whether explicit relation construction is necessary for the synthetic gains.

  3. Analysis request: On the real query streams, what are the constructed-source coverage, relation-cache reuse rates, and construction-versus-answer cost breakdowns? Can these measurements identify a real workload regime in which persistent relation reuse provides a useful quality–cost advantage, with robustness beyond the single sparse RepLiQA stream?

2 comments

Thank you for asking us to distinguish deferring construction, reusing prior work, and using relations to direct retrieval. We have run additional comparisons and will narrow the paper's claims to the effects those experiments support.

W1 Value of explicit relations

The original traversal ablation removes access to bridge passages along with edge-guided expansion. Its accuracy drop measures reliance on that expansion path. The independent text comparison in Q2 leaves the value of explicit relations open. We will describe the ablation as a mechanism test and narrow the abstract accordingly.

W2 Evidence for a useful real workload

The results on natural tasks are mixed. LoCoMo declines against SRH-Base, ConvoMem yields no confirmed improvement, and the small LongMemEval-S gain reverses under common-Claude scoring. A separate 90-question pilot on HotpotQA, MuSiQue and 2Wiki used fixed candidate pools and included three SRH recoveries at different construction limits. It yielded 46/90 exact matches for SRH-Graph and 55/90 each for Naive RAG and HippoRAG2. Our two new RepLiQA streams also favored cached iterative text end to end. Q3 gives the completion rates and explains how construction failures shaped that result.

Q1 What conventional memoization explains

The cache uses conventional memoization and source-version invalidation, including for empty extractions. A query can follow a cached relation into the raw archive and trigger construction on a previously untouched source. LazyGraphRAG and LazyMem motivate deferral. Our controlled measurements separate the savings from avoiding repeated construction (39.84–40.73% of lifecycle tokens) and from deferral relative to eager construction (0.036% at full coverage).

On 40 synthetic questions, iterative text with and without an excerpt cache gave the same answers (26/40 exact) and identical payloads on all 131 request pairs. Each used 103,232 input and 1,852 output tokens. The 228/283 cache hits saved local extraction work.

Q2 Iterative retrieval with comparable limits

In the four-family bridge pilot, both arms started from raw archives and shared a 397B reader, original-passage packing, and limits of seven lookups, six sources and 4,096 evidence tokens. SRH-Graph scored 7/16 and iterative text 8/16. Both abstained on all four required-source deletions, and the no-context reader abstained on 4/4. SRH recorded 14,315 construction and 4,221 answer tokens. Text recorded 14,807 planning and 5,934 answer tokens. Embeddings were metered separately as calls.

On the original 240 questions, both arms expanded the same SRH context under a common 397B reader, 4,096 evidence tokens and 64 answer tokens. SRH scored 38/240 and iterative text 35/240, including five failures per arm (+1.25 percentage points, descriptive 95% interval −2.08 to +4.58). Final reader inputs matched in 178/234 co-valid pairs. SRH answering used 932,650 tokens and iterative planning plus answering used 3,626,919. Earlier graph construction is accounted for separately. Eight unknown calls reserved 5,056 completion tokens.

Q3 Coverage, reuse and separate costs

A cache hit may reuse an empty extraction. LongMemEval-S and ConvoMem reset per question, so their coverage totals combine separate cache scopes. LongMemEval-S includes 915 smoke-check tokens. ConvoMem had a 63,833-token unknown-usage reservation in addition to the reported costs.

WorkloadHits / admissionsSources cached / availableConstruction tokensAnswer tokens
LoCoMo4,139 / 4,410271 / 272370,1847,822,885
LongMemEval-S0 / 1,1311,131 / 23,8674,258,6712,329,428
Sparse RepLiQA13 / 10996 / 500203,850385,018
ConvoMem0 / 1,5961,596 / 6,5703,104,3682,294,513

In the historical RepLiQA stream, mean set-token F1 was 62.32% for SRH and 47.22% for Naive RAG. The paired gain of +15.09 points stays within +13.15 to +16.53 points when any one document's score is omitted. Eleven of 40 questions saw a cache hit. SRH recorded 588,868 tokens, including 385,018 for answering, versus 121,807 total for the historical Naive configuration. The estimate describes one query stream.

We then froze two disjoint held-out RepLiQA streams with 24 questions from eight documents and 100 candidate documents per stream. Each arm started cold with the same question order, Qwen3.5-397B reader, raw-text answer packing, a 4,096-token evidence budget, up to seven lookups and up to six admitted chunks per question. F1 below is normalized word-overlap F1. Failed slots receive zero in the full 24-question denominator.

SplitArmAnswersMean F1EMDocs cachedCross-question hits / accesses
3SRH-Graph2/246.05%1/244/1002/8
3Cached text24/2462.39%4/2428/10024/132
4SRH-Graph3/244.39%0/247/1002/13
4Cached text24/2455.78%5/2416/10030/110

SRH cached 6/687 and 11/682 chunks in the two splits. Text cached 56/687 and 45/682. In split 3, SRH recorded 24,204 construction and 1,446 answer tokens, and text recorded 36,695 planning and 19,237 answer tokens. In split 4 the corresponding totals were 26,499 and 1,692 for SRH, and 31,644 and 16,322 for text. Counts include failed model requests. Embedding calls were tracked separately.

Of the 43 graph failures, 35 stopped at quote or edge validation, one at malformed JSON, and seven depended on earlier failures. All ten rejected outputs in a diagnostic sample still contained individually source-supported relations. The lower SRH token totals came with far fewer delivered answers. These runs document a reliability limit of this implementation. Retrieval quality with a functioning constructor remains open.

In the controlled streams, median retrieval was 48.86 seconds at the first position and 0.49 seconds at later positions, where questions differed and reused the corpus cache. We will report this latency alongside the savings against rebuilding.

Reviewer_xEwjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes demand-driven construction that reuses the results from structured source required of the query. The core perspective is that both the low global access coverage and high local reuse can be existed. The proposed method SRH-Graph extracts the relation from the retrieved source, and search the additional evidence with edge-guided expansion. The extracted results are saved in source-version-keyed cache, and it runs the cache invalidation when source is changed.

Critical strengths

S1. Clear problem setup and cost decomposition. It is meaningful that the paper analyzes Global access coverage, local reuse, and answer-generation cost. In particular, distinction with cost saving and end-to-end lifecycle cost saving is helpful to understand the system trade-off.

S2. Ablation studies. When removing the traversal, the accuracy of synthetic benchmark drastically dropped, where as the accuracy is maintained while cost increases when removing reuse. It supports that traversal and reuse contribute evidence discovery and construction cost, respectively.

Critical weaknesses

W1. Novelty. Deferred construction is already handled in the previous research, and the authors also agreed. Therefore, the core contribution in this paper is source-version-keyed relation reuse and cache invalidation. However, this paper cannot sufficiently show the somewhat new principle or practical benefits beyond the lazy construction and general caching. Cost model is accounting framework to easily understand, but there is lack of validation of determining good operating point in new workload in advance.

W2. Lack of baseline. Synthetic benchmark should follow the relation chain across various source. Therefore, the big gap of SRH-Graph between single-pass Naive RAG and edge-guided expansion is hard to prove the necessity of relation graph. The most important baseline the authors should add is iterative retrieval which leverages the intermediate retrieval result without explicit relation graph. Additionally, the authors should compare the baseline with the same answer model and similar retrieval and token budget, thereby the authors can claim the benefits of graph-based memory and multi-step retrieval.

W3. Lack of result on real-world benchmark. In LoCoMo, SRH-Graph shows the lower results than SRH-Base and Naive RAG. In ConvoMem and LongMemEval-S, it is uncertain to improve and it requires the additional token cost. Sparse RepLiQA is the potential benchmark to show the good result, but it relies on only 40-question single stream; it is too small set. If the authors should evaluate the proposed method in MuSiQue, 2WikiMultiHopQA, and MemoryAgentBench, those results can support the effectiveness of the proposed method.

W4. Lack of matched comparison with separating construction policy. The core contribution of this paper is not use the graph naively, but when construction and reuse should be run. However, in the real-world experiment, there is no matched-comparison result with eager construction, demand-driven construction with reuse, and demand-driven construction without reuse that use the identical component. Since there is lack of matched comparison with previous lazy construction method, it is hard to validate the effectiveness of proposed reuse policy.

W5. Prompt Length. SRH-Graph keeps the retrieval context of SRH-Base, and adds the source quote induced from relation within 16,000 characters. Hence, it is hard to conclude where the improvement comes from: (1) relation traversal, (2) more context evidence in answer model.

W6. Synthetic benchmark seems to be unfair. The chain depth of synthetic benchmark is three, which is identical to the traversal limit of SRH-Graph. It would be better dataset when chain depth is various and sometimes it is over than the traversal limit. Plus, reporting the statistics of dataset configuration can make the dataset is more reliable.

W7. Lack of discussion of latency. On-demand design moves the computational cost into query time. The first retrieval time in SRH-Graph is about 17--20 seconds, which is much longer than eager (0.6 seconds). When we think about the practical services such as chatbot or agent system, answer latency started from the query might be sensitive.

Questions for authors

See the weaknesses, and here is the minor comment for the authors.

1. Missing references. Here are the missing related works that might be relevant to this manuscript.

[1] Gutiérrez, Bernal J., et al. "Hipporag: Neurobiologically inspired long-term memory for large language models.", NeurIPS 2024.

[2] Guo, Zirui, et al. "LightRAG: Simple and Fast Retrieval-Augmented Generation." EMNLP (Findings). 2025.

[3] Xu, Wujiang, et al. "A-mem: Agentic memory for llm agents." NeurIPS 2026.

[4] Trivedi, Harsh, et al. "Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.", ACL 2023.

2 comments

Thank you for asking us to separate the effects of traversal, additional evidence, and reuse. We have added controls for each comparison where we could and narrowed the claim to the behavior of the tested construction and retrieval policies.

W1 Novelty and choosing an operating point

SRH-Graph couples raw-source search with a versioned relation cache. It extracts relations when retrieval reaches a source and reuses them while that source version remains current. Deferral, caching, and invalidation are established techniques. Section 3.3 accounts for work saved by deferral and by reuse. Choosing an operating point in advance would require separate validation on unseen query streams. In the controlled workloads, reuse saves 39.84–40.73% against per-question reconstruction. The saving against eager construction falls to 0.816% at 75% object coverage and 0.036% at full coverage.

W2 Iterative retrieval baseline

On all 240 questions in the original 10% object-coverage condition, SRH-Graph scored 38/240 and iterative raw text scored 35/240. Five technical failures per arm remain in the denominator. The reader, 64-token answer cap and 4,096-token evidence ceiling were shared. Both arms could expand the original SRH base in the space left over. The paired gap was +1.25 percentage points (descriptive 95% within-seed hotspot bootstrap interval −2.08 to +4.58). This measures expansion given the same base. In the bridge pilot in W6, each retriever starts independently from the same raw-source pool.

We also compared iterative text with and without a deterministic source-excerpt cache on 40 synthetic questions. Both scored 26/40, with identical payloads in all 131 request pairs. The cache hit on 228/283 accesses and saved local re-extraction. Each arm used 103,232 input and 1,852 output tokens.

W3 Evidence on natural questions

In the submitted results, SRH-Graph trails SRH-Base on LoCoMo (995/1,986 vs. 1,030/1,986) and ConvoMem (510/657 vs. 513/657). The LongMemEval-S difference against historical Naive RAG crosses zero and reverses under common-Claude scoring. The positive historical RepLiQA comparison covers one 40-question stream.

We compared SRH-Graph with cached iterative text on two held-out RepLiQA streams, each with 24 questions and 100 candidate documents. Both arms started cold with the same Qwen3.5-397B reader, original-passage packing, and 4,096-token evidence budget. Normalized word-overlap F1 was 6.05% vs. 62.39% on RepLiQA-3 and 4.39% vs. 55.78% on RepLiQA-4. Exact match was 1/24 vs. 4/24 and 0/24 vs. 5/24. SRH produced 2/24 and 3/24 valid answers, and cached text produced 24/24 on both. All 43 graph failures, comprising 36 constructor validation or parse failures and seven dependent slots, count as zero. This measures the tested pipeline end to end. The effect of traversal alone remains open.

Our post-review pilot used fixed candidate pools for 30 questions each from HotpotQA, MuSiQue, and 2WikiMultiHopQA. SRH-Graph, Naive RAG, and HippoRAG2 got 46/90, 55/90, and 55/90 exact matches. On MuSiQue the counts were 4/30, 12/30, and 11/30. On 2WikiMultiHopQA they were 19/30, 22/30, and 21/30. Three SRH recoveries used different construction limits. A comparable MemoryAgentBench run remains open.

W4 Construction policies

Appendix D, Table 8 compares eager construction, demand-driven construction with reuse, and demand-driven construction without reuse under the same graph extraction and packing recipe and a shared reader. At 1% object coverage they used 628,444, 136,110, and 226,264 lifecycle tokens, respectively. At 50% they used 6,561,277, 6,435,179, and 10,781,258. Independent extraction calls can produce different graphs. The fixed-artifact exercise in §5.1 instead charges different schedules on the same artifacts. A matched three-way real-stream experiment and a direct comparison with an earlier lazy system remain open.

W5 Evidence shown to the reader

At 50% synthetic coverage, the median answer input was 5,427 tokens for SRH-Graph and 4,780 for Naive RAG. In W2, the base filled the 4,096-token budget in 179/235 valid calls per arm, and 178/234 pairs with two valid answers had identical final prompts. In the bridge pilot, SRH and iterative text delivered all required passages uncut on 3/12 and 4/12 answerable cases under the same ceiling. In the natural multi-hop pilot, all annotated support IDs reached the reader on 58/90, 67/90, and 64/90 cases for SRH, Naive, and HippoRAG2. The saved record leaves sentence-level clipping unresolved.

In the matched RepLiQA follow-up, at least one chunk from the annotated document reached the reader on 2/24 and 3/24 SRH questions, versus 23/24 and 24/24 with cached text. These are document-presence counts and include cases that stopped before packing.

W6 Depth and workload design

The original chains have depth three, matching the traversal limit, and each hotspot has eight questions. Across the three corpora, target-object coverage of 1%, 5%, 10%, 20%, and 50% yielded construction on 25, 113, 184, 334, and 598 of the 900 source versions, respectively. We will report both coverage measures. In MuSiQue, at 2, 3, and 4 hops (10 questions each), SRH got 1, 2, and 1 exact matches, and Naive RAG got 3, 6, and 3.

Our bridge pilot includes one family at each of 2, 3, 4, and 6 required documents. Each family includes the intact route, removal of a required document, removal of an irrelevant document, and redirection. SRH-Graph got 7/16 and independent iterative text got 8/16. The three-document intact case accounts for the difference. Both systems correctly abstained on all four required-document deletions and missed every answerable variant at four and six documents. The no-context control abstained on 4/4. With one family per depth and route length given in the question, this is a small diagnostic of the fixed policies.

W7 Latency

At 75% object coverage, mean first and later retrieval took 20.24 and 0.63 seconds with deferred construction, versus 0.62 and 0.56 seconds with eager construction (§5.3). At full coverage, the times were 17.40 and 0.57 versus 0.61 and 0.56 seconds. Eager startup construction is accounted for separately. The cold path is the practical cost of deferral here, especially for a service with a tight first-response limit.

Q1 Missing references

We will add HippoRAG (Gutiérrez et al., NeurIPS 2024) for graph-based associative retrieval, LightRAG (Guo et al., Findings of EMNLP 2025) for incremental graph indexing, A-Mem (Xu et al., NeurIPS 2025) for linked memory notes, and IRCoT (Trivedi et al., ACL 2023) for interleaved reasoning and retrieval. The iterative control in W2 was built for the matched test above. The natural pilot used HippoRAG2.

Reviewer_JGWyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks when an agent should pay to build structured memory over its archive. SRH-Graph builds structure only on demand. At query time it extracts relations from the retrieved sources, caches them by source version, and follows them for a few hops to reach bridging sources. A token cost model separates avoided construction, avoided reconstruction and answer cost. On synthetic relation chains, SRH-Graph reaches 80–100% accuracy vs. 0–3% for Naive RAG. On four real benchmarks it is mostly worse than Naive RAG and always more expensive.

Critical strengths
  • The cost decomposition is clear and explains when deferral pays off (Fig. 2).
  • The reporting is candid: regressions are shown in the main text and failures count as zeros.
Critical weaknesses

1. Weak baselines and thin related work. This is my main concern.

  • The only external RAG baseline is Naive RAG. The memory systems (mem0, Graphiti, MemOS) are historical runs that cover only some tasks, and none of them are run on ConvoMem or the synthetic chains.
  • GraphRAG, LazyGraphRAG and LazyMem are cited but not compared, even though the last two are the closest prior work.
  • LightRAG is not cited. It already supports incremental graph updates without a full rebuild, which overlaps directly with this paper's source-level invalidation.
  • HippoRAG is not cited, although the H in SRH stands for "Hippocampus-inspired". It is a natural baseline for the multi-hop experiments.
  • There are no iterative retrieval baselines without a graph (IRCoT, ReAct, self-ask). They can reach bridging documents without any construction cost, so they matter for the paper's own question of when to pay for structure.
  • The paper has only 15 references. Related memory work such as G-Memory, A-Mem and RAPTOR should at least be discussed.

2. The contribution is thin, and the key premise is assumed rather than measured. The paper itself says query-time construction isn't new. What remains is caching relations by source version, which is standard memoization. The method relies on real workloads repeatedly touching the same few sources, but the synthetic study builds this in by fixing eight questions per hotspot. On the real benchmarks, the numbers that would test the premise are not reported: cache hit rates, the fraction of sources constructed, and how often different questions share sources.

3. The experiments are narrow, and the real-data results are weak. The synthetic gain is close to true by construction: single-pass retrieval cannot reach the third source, and eager construction does about as well. So it shows that traversal helps on multi-hop questions, not that demand-driven construction does. On real tasks:

BenchmarkAccuracy vs. Naive RAGTokens vs. Naive RAG
LoCoMo−3.1 pp (significant)1.07x
ConvoMem−2.1 pp2.55x
LongMemEval-S+1.2 pp (not significant)2.94x
RepLiQA sparse+15.1 pp F14.83x

The only clear gain comes from a single 40-question stream. There is also no standard multi-hop QA benchmark such as HotpotQA, 2WikiMultiHopQA or MuSiQue, which is where the method should help most.

4. The comparisons are not matched. The historical runs differ in date, API gateway, judge and answer model. SRH-Graph also sees more evidence than Naive RAG. The mem0, Graphiti and MemOS scores on LoCoMo (31.3%, 26.7%, 39.7%) are far below the published ones: the Mem0 paper reports 66.9% for Mem0 and 66.0% for Zep, and the MemOS paper reports 75.8%. This makes me doubt that these configurations are representative.

5. Presentation. The paper reads like an audit log. It is full of qualifiers and references to earlier versions the reader doesn't have ("the accompanying draft", "legacy SRH", SHA-256 hashes). Meanwhile, basic method details are missing: the extraction prompt, how traversal is triggered, and how entities are matched across sources.

Minor

  • The abstract leads with the synthetic result, which oversells the work.
  • The 96.41% saving in §5.1 is fixed by the ratio of retrieved to total sessions, so it is an accounting identity rather than an experimental finding.
  • Deferral puts construction on the critical path: first retrieval takes 17–20 s vs. about 0.6 s for eager.
Questions for authors

See weaknesses

2 comments

Thank you for pressing us on the baseline and the role of the graph. We agree that the abstract overemphasized the gap against one-pass retrieval. We will lead with the measured savings from avoiding repeated construction and state the quality and latency trade-offs alongside them.

W1 Baselines and related work

We compared SRH-Graph with independently initialized iterative text retrieval using a source excerpt cache on RepLiQA splits 3 and 4. Each held-out stream had 24 questions and 100 candidate documents. Both arms used the same 397B reader, original-passage packer and 4,096-token evidence budget. Word-overlap F1 was 6.05% and 4.39% for SRH, versus 62.39% and 55.78% for iterative text. SRH produced valid answers on 2/24 and 3/24 questions, and iterative text on 24/24 in each stream. We counted all 43 failed SRH slots as zero. Of those, 35 failed quote or edge checks, one returned malformed JSON, and seven depended on a failed construction. This measures end-to-end reliability of the tested configuration. The few successful constructions leave the effect of graph traversal unresolved.

In a four-family bridge pilot, both methods started from the raw archive and shared a reader and passage packer. SRH-Graph scored 7/16 exact matches and iterative raw text 8/16. Both abstained on 4/4 necessary-source removals. Complete supporting evidence reached the reader on 3/12 versus 4/12 answerable variants, and all seven answers were correct. This is a small mechanism check with a stronger comparator than one-pass retrieval.

In the original synthetic 10% coverage condition, SRH scored 38/240 and iterative raw text 35/240 with the same reader and 4,096-token evidence limit. Five failures per arm remain in the denominator. The gap was +1.25 points (descriptive 95% within-seed hotspot bootstrap interval −2.08 to +4.58). Both methods started from SRH-retrieved evidence, and 178/234 valid pairs had identical reader input. This comparison measures expansion from that shared context.

We also tested ordinary excerpt caching on 40 previously undispatched synthetic questions. Independent raw-text retrieval and the excerpt-cache version each answered 26/40 correctly, with identical payloads across all 131 paired model requests (103,232 input and 1,852 output tokens per arm). The 228/283 cache hits avoided local extraction while leaving the model workload unchanged in this stream.

We will discuss IRCoT, ReAct and Self-Ask as multi-step alternatives, and place our implementation alongside GraphRAG, LazyGraphRAG and LazyMem, LightRAG's incremental updates, and the retrieval or memory designs in HippoRAG, RAPTOR, G-Memory and A-Mem. We will distinguish the post-submission HippoRAG2 pilot in W3 from the GraphRAG and lazy-method comparisons that remain open.

W2 What the cache contributes and how often it is used

The cache uses conventional source-version keys and invalidation. It stores source-grounded relations, and a query can follow a relation to an untouched source and trigger construction there. In the controlled streams, cross-question reuse saved 39.84–40.73% of lifecycle tokens relative to rebuilding between questions. In the separate eager-construction comparison, the saving fell to 0.816% at 75% coverage and 0.036% at full coverage (§5.3, Table 8).

The eight questions per synthetic hotspot deliberately create reuse. The first position accounted for 147/184 construction calls and had a 48.86-second median retrieval time, versus 0.49 seconds across positions 2–8. LoCoMo had 4,139/4,410 hits, including 1,700 on empty or rejected extractions, and cached 271/272 sources. Of the cached versions, 270 served multiple questions. Sparse RepLiQA had 13/109 hits, cached 96/500 sources, and reused 12 source versions across questions. LongMemEval and ConvoMem reset state between questions by design. We will report hits alongside coverage, scope and quality.

W3 Synthetic scope and real results

We agree that the real evidence is mixed. LoCoMo and ConvoMem regress against historical Naive RAG, the small LongMemEval lead reverses under the common-Claude judge, and construction-plus-answer tokens exceed the historical baseline on all four tasks. The original RepLiQA stream's 62.32% versus 47.22% set-token F1 includes three failed SRH answers. In that same stream, SRH used 203,850 construction and 385,018 answer tokens, versus 121,807 total for historical Naive. We will show the quality and recorded token counts together and identify the differences in historical settings.

In a separate 90-question HotpotQA, MuSiQue and 2Wiki pilot, SRH scored 46/90 exact matches, and Naive RAG and HippoRAG2 each scored 55/90. Three SRH recoveries used different construction limits. All annotated supporting passage IDs appeared in 58/90 SRH contexts, 67/90 Naive contexts and 64/90 HippoRAG2 contexts. Passage-ID coverage leaves open whether individual supporting sentences survived packing.

W4 Matching and the historical systems

Thank you for flagging the discrepancy with the published Mem0, Zep and MemOS results. Our historical mem0, Graphiti and MemOS workflows used different implementations, gateways, readers and judges. We will move those historical runs out of the headline comparison and document each setting. For the synthetic tests we will specify which variables are controlled by a shared reader and document how the initial and final evidence differed, as in W1.

W5 Method and presentation

We will put the algorithm and extraction prompt in the main text. The extractor receives a single source without the question, requests up to 12 explicit relations, and checks each relation against exact phrases in a contiguous source quotation. Case-insensitive literal subject matches at word boundaries seed traversal. The graph starts from two search hits, follows outgoing objects for up to three rounds, and admits at most six sources to the augmentation layer. The original submitted reader also retains its base context. Entity matching is literal.

The 96.41% scheduling figure in §5.1 uses fixed evidence and answers to account for construction under different schedules. We will label it as an accounting calculation and report the measured latency next to it. At 75% and full coverage, first retrieval averaged roughly 20 and 17 seconds with deferral, versus about 0.6 seconds with eager construction, whose startup is reported separately. Advance construction remains a reasonable choice when first-query latency dominates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.