- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when agent memory should be constructed and reused. It proposes SRH-Graph, which retrieves raw sources, constructs source-grounded relations on demand, caches them across distinct questions, and invalidates them when sources change. SRH-Base provides a retrieval-only alternative. The analysis separates global access coverage, local reuse, and answer computation. Evaluation includes fixed-artifact scheduling, controlled multi-hop streams with traversal and reuse ablations, source updates, high-coverage conditions, and four real benchmarks. The reported results show substantial synthetic multi-hop improvements and savings relative to eager construction or repeated reconstruction, alongside conversational regressions and higher token costs than Naive RAG on real tasks.
Critical strengths
-
The paper usefully separates avoided preprocessing, persistent relation reuse, and answer computation. The controlled experiments show that these produce different savings: reuse reduces repeated construction, while savings against eager construction diminish near full coverage. The traversal and reuse ablations, source-update tests, and cold-query latency measurements make these operating boundaries informative.
-
The empirical reporting preserves important negative results and uncertainty. Technical failures remain in the denominators, accuracy and F1 are distinguished, and historical comparisons are explicitly identified. Reporting conversational regressions, scoring sensitivity, and higher costs provides a more useful assessment of the method’s practical limits than aggregate improvements alone.
Critical weaknesses
-
The distinctive benefit of relation memory over simpler deferred retrieval remains insufficiently established. Deferred construction is already explored by LazyGraphRAG and LazyMem, so the central question is what persistent relation construction and reuse add. In the synthetic study, SRH-Graph performs bounded multi-round expansion, while Naive RAG performs one-pass retrieval. Removing traversal also removes access to the bridging sources; this demonstrates the importance of expansion, but does not establish that graph construction is preferable to iterative text retrieval or query expansion. The paper acknowledges unequal evidence exposure and unavailable matched implementations of some related methods. A budget-matched iterative retrieval control would nevertheless address a consequential alternative explanation for the large synthetic gains.
-
Real-workload evidence for the proposed operating regime is limited. The clearest positive real-task result comes from one fixed 40-question RepLiQA stream. SRH-Graph regresses on LoCoMo relative to SRH-Base, has no confirmed improvement on ConvoMem, and its small LongMemEval advantage changes direction under the common-Claude scoring sensitivity. Recorded token costs exceed Naive RAG on all four tasks. These limitations are transparently acknowledged, but stronger evidence connecting measured coverage and relation reuse to a useful real-world quality–cost tradeoff is needed to substantiate the practical guidance about when to pay for structure.
Questions for authors
-
Clarification: What technical or empirical contribution distinguishes the persistent source-versioned relation cache from combining deferred construction with conventional memoization? Please identify the specific benefit beyond the deferred-construction approaches discussed in Section 2.
-
Evidence request: How does SRH-Graph compare with iterative raw-text retrieval or query expansion under matched retrieval rounds, admitted-source limits, answer-context budgets, and answer models? This comparison would help determine whether explicit relation construction is necessary for the synthetic gains.
-
Analysis request: On the real query streams, what are the constructed-source coverage, relation-cache reuse rates, and construction-versus-answer cost breakdowns? Can these measurements identify a real workload regime in which persistent relation reuse provides a useful quality–cost advantage, with robustness beyond the single sparse RepLiQA stream?
2 comments
Q3 Coverage, reuse and separate costs
A cache hit may reuse an empty extraction. LongMemEval-S and ConvoMem reset per question, so their coverage totals combine separate cache scopes. LongMemEval-S includes 915 smoke-check tokens. ConvoMem had a 63,833-token unknown-usage reservation in addition to the reported costs.
| Workload | Hits / admissions | Sources cached / available | Construction tokens | Answer tokens |
|---|---|---|---|---|
| LoCoMo | 4,139 / 4,410 | 271 / 272 | 370,184 | 7,822,885 |
| LongMemEval-S | 0 / 1,131 | 1,131 / 23,867 | 4,258,671 | 2,329,428 |
| Sparse RepLiQA | 13 / 109 | 96 / 500 | 203,850 | 385,018 |
| ConvoMem | 0 / 1,596 | 1,596 / 6,570 | 3,104,368 | 2,294,513 |
In the historical RepLiQA stream, mean set-token F1 was 62.32% for SRH and 47.22% for Naive RAG. The paired gain of +15.09 points stays within +13.15 to +16.53 points when any one document's score is omitted. Eleven of 40 questions saw a cache hit. SRH recorded 588,868 tokens, including 385,018 for answering, versus 121,807 total for the historical Naive configuration. The estimate describes one query stream.
We then froze two disjoint held-out RepLiQA streams with 24 questions from eight documents and 100 candidate documents per stream. Each arm started cold with the same question order, Qwen3.5-397B reader, raw-text answer packing, a 4,096-token evidence budget, up to seven lookups and up to six admitted chunks per question. F1 below is normalized word-overlap F1. Failed slots receive zero in the full 24-question denominator.
| Split | Arm | Answers | Mean F1 | EM | Docs cached | Cross-question hits / accesses |
|---|---|---|---|---|---|---|
| 3 | SRH-Graph | 2/24 | 6.05% | 1/24 | 4/100 | 2/8 |
| 3 | Cached text | 24/24 | 62.39% | 4/24 | 28/100 | 24/132 |
| 4 | SRH-Graph | 3/24 | 4.39% | 0/24 | 7/100 | 2/13 |
| 4 | Cached text | 24/24 | 55.78% | 5/24 | 16/100 | 30/110 |
SRH cached 6/687 and 11/682 chunks in the two splits. Text cached 56/687 and 45/682. In split 3, SRH recorded 24,204 construction and 1,446 answer tokens, and text recorded 36,695 planning and 19,237 answer tokens. In split 4 the corresponding totals were 26,499 and 1,692 for SRH, and 31,644 and 16,322 for text. Counts include failed model requests. Embedding calls were tracked separately.
Of the 43 graph failures, 35 stopped at quote or edge validation, one at malformed JSON, and seven depended on earlier failures. All ten rejected outputs in a diagnostic sample still contained individually source-supported relations. The lower SRH token totals came with far fewer delivered answers. These runs document a reliability limit of this implementation. Retrieval quality with a functioning constructor remains open.
In the controlled streams, median retrieval was 48.86 seconds at the first position and 0.49 seconds at later positions, where questions differed and reused the corpus cache. We will report this latency alongside the savings against rebuilding.
Thank you for asking us to distinguish deferring construction, reusing prior work, and using relations to direct retrieval. We have run additional comparisons and will narrow the paper's claims to the effects those experiments support.
W1 Value of explicit relations
The original traversal ablation removes access to bridge passages along with edge-guided expansion. Its accuracy drop measures reliance on that expansion path. The independent text comparison in Q2 leaves the value of explicit relations open. We will describe the ablation as a mechanism test and narrow the abstract accordingly.
W2 Evidence for a useful real workload
The results on natural tasks are mixed. LoCoMo declines against SRH-Base, ConvoMem yields no confirmed improvement, and the small LongMemEval-S gain reverses under common-Claude scoring. A separate 90-question pilot on HotpotQA, MuSiQue and 2Wiki used fixed candidate pools and included three SRH recoveries at different construction limits. It yielded 46/90 exact matches for SRH-Graph and 55/90 each for Naive RAG and HippoRAG2. Our two new RepLiQA streams also favored cached iterative text end to end. Q3 gives the completion rates and explains how construction failures shaped that result.
Q1 What conventional memoization explains
The cache uses conventional memoization and source-version invalidation, including for empty extractions. A query can follow a cached relation into the raw archive and trigger construction on a previously untouched source. LazyGraphRAG and LazyMem motivate deferral. Our controlled measurements separate the savings from avoiding repeated construction (39.84–40.73% of lifecycle tokens) and from deferral relative to eager construction (0.036% at full coverage).
On 40 synthetic questions, iterative text with and without an excerpt cache gave the same answers (26/40 exact) and identical payloads on all 131 request pairs. Each used 103,232 input and 1,852 output tokens. The 228/283 cache hits saved local extraction work.
Q2 Iterative retrieval with comparable limits
In the four-family bridge pilot, both arms started from raw archives and shared a 397B reader, original-passage packing, and limits of seven lookups, six sources and 4,096 evidence tokens. SRH-Graph scored 7/16 and iterative text 8/16. Both abstained on all four required-source deletions, and the no-context reader abstained on 4/4. SRH recorded 14,315 construction and 4,221 answer tokens. Text recorded 14,807 planning and 5,934 answer tokens. Embeddings were metered separately as calls.
On the original 240 questions, both arms expanded the same SRH context under a common 397B reader, 4,096 evidence tokens and 64 answer tokens. SRH scored 38/240 and iterative text 35/240, including five failures per arm (+1.25 percentage points, descriptive 95% interval −2.08 to +4.58). Final reader inputs matched in 178/234 co-valid pairs. SRH answering used 932,650 tokens and iterative planning plus answering used 3,626,919. Earlier graph construction is accounted for separately. Eight unknown calls reserved 5,056 completion tokens.