- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CorMem, a long-term memory system for LLM agents that combines a cue–fact bipartite graph with a hierarchical schema layer. The authors report improvements over several baselines on LoCoMo and PersonaMem-v2.
Critical strengths
The paper is well written and generally clear.
Critical weaknesses
1.The structural ablations in the main text change both the graph topology and the types of query seeds, making it difficult to isolate the contribution of the bipartite structure itself. Could the authors clarify whether the bipartite structure still outperforms simpler alternatives when the query information and candidate retrieval conditions are kept as similar as possible? Regarding Table 2, why does the higher recall translate into only a small improvement in answer quality? How much practical benefit does the full structure provide over the simpler structure? 2.The paper does not evaluate whether the hierarchical organization of gists and graph propagation are necessary. Would a comparison between flat retrieval and hierarchical graph-based retrieval demonstrate an independent benefit from the more complex structure? 3.I also have concerns about how the system handles conflicts between new instructions and existing abstractions. The paper aggregates historical evidence through shared cues and hierarchical gists, but the main text does not clearly explain how such conflicts are resolved. In these situations, how does the system ensure that the latest constraint is retrieved and given priority? Are old gists revised, marked as invalid, or assigned lower priority, or does the system simply rely on the answer model to resolve conflicts after seeing the conflicting information? If the new constraint is not included in the final retrieval results, how does the system avoid continuing to use the outdated preference?
Questions for authors
Please see the specific concerns and questions raised in the Weaknesses section.
6 comments
W1-2. Why higher recall yields smaller F1/BLEU-1 gains
Three factors help explain the difference between Recall@3 and F1/BLEU-1 gains in Table 2.
- Lexical scoring noise. F1/BLEU-1 emphasize wording overlap, which can obscure answer correctness (Ru et al., 2024). An incorrect answer can therefore outscore a correct but differently phrased one.
- Non-proportional changes. Retrieval coverage and generation quality are distinct evaluation targets (Ru et al., 2024). Our grouped results show that greater coverage does not correspond to proportional F1/BLEU-1 gains, particularly at partial recall.
- Generation variability. Generated answers introduce sampling variability into evaluation (Zhang et al., 2026). We reduce its impact by averaging three generations per question under a shared answer prompt.
The following example illustrates the first point.
Question
What kind of stress was Sam dealing with in October 2023?Reference answer
Work-related stress.
| Method | Generated answer | Recall@3 (%) | F1 (%) | BLEU-1 (%) |
|---|---|---|---|---|
| CorMem (correct) | “Work stress, family pressure regarding health.” | 100.00 | 25.00 | 16.67 |
| Flat (incorrect) | “Health-related stress.” | 0.00 | 50.00 | 50.00 |
CorMem identifies work stress and adds dialogue-supported detail, yet Flat’s incorrect answer receives higher scores.
For the second point, we group Full CorMem’s questions by gold-session recall. The two higher partial-recall groups have nearly identical answer scores, while complete recall corresponds to substantially higher scores.
| Gold-session Recall@3 | F1 (%) | BLEU-1 (%) |
|---|---|---|
| 22.78 | 16.97 | |
| 23.21 | 18.11 | |
| 33.02 | 22.83 | |
| 32.28 | 22.66 | |
| 62.51 | 54.62 |
Although F1/BLEU-1 depend on reference answers and model-generated wording, they remain widely used metrics for evaluating memory systems. We retain them for comparable end-to-end evaluation, alongside Recall@3 as a core retrieval metric that directly measures evidence coverage independently of answer generation.
- Ru et al. (2024). RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. NeurIPS.
- Zhang et al. (2026). Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation. Findings of ACL.
W2. Benefits of hierarchical gist organization and graph propagation
We sincerely thank the reviewer for suggesting these comparisons. W1-1 shows that the full structure outperforms flat retrieval with the same memory units (LoCoMo F1 53.52% → 55.28%, Holm-adjusted , and PersonaMem-v2 accuracy 66.98% → 70.48%, ). Below, we separately examine the contributions of graph propagation and gist hierarchy.
1. Graph propagation adds 3.12 Recall@3 points with fixed seeds
To isolate the contribution of graph propagation, we disable fact–cue–fact traversal and activate sessions directly from the seeds.
We follow the Table 2 protocol on LoCoMo, keeping extracted memory, queries, and initial seed activations identical.
| Variant | Recall@3 (%) ↑ |
|---|---|
| w/o fact–cue propagation | 83.71 |
| Full CorMem | 86.83† |
† versus w/o fact–cue propagation.
Propagation improves Recall@3 by 3.12 percentage points (). With extraction and initial activation fixed, this gain demonstrates the benefit of traversing shared-cue paths to retrieve supporting evidence.
2. Gist hierarchy improves accuracy by 4.34 points
To examine hierarchical gist organization, we flatten the gists into a single layer by removing gist–gist edges and connecting higher-level gists directly to their supporting facts.
Queries, initial seed activations, and the facts retrieved from the cognitive map remain identical. We focus on the 311 queries whose retrieved gist units change. The remaining queries retrieve the same memory units, so this diagnostic focuses on changes in retrieved content while reducing noise from answer generation. Queries are selected solely by retrieval differences, independently of answer correctness.
| Gist graph | Accuracy (%) ↑ |
|---|---|
| Flattened gist graph | 60.29 |
| Original hierarchy | 64.63† |
† versus the flattened gist graph.
On this diagnostic subset, retaining the hierarchy improves accuracy by 4.34 percentage points (95% cluster-bootstrap CI [0.81, 7.99], ). This supports the hierarchy’s retrieval role, allowing higher-level gists to aggregate PPR activation from multiple descendants and guide the selection of relevant abstractions.
We sincerely thank the reviewer for the careful examination of CorMem’s key mechanisms and the constructive suggestions for strengthening the ablation analysis. Below, we present additional controlled experiments to clarify the contributions of the cue–fact graph and hierarchical schema, and explain how the system handles memory updates.
W1-1. Contribution of the cue–fact structure to retrieval
We thank the reviewer for suggesting controlled comparisons. We examine full versus flat retrieval with the same memory units, cue–fact conversion in HippoRAG 2 (Gutiérrez et al., 2025), and evidence recovery from cue seeds alone.
1. Practical benefits of the full structure over simpler structures
Using the same stored facts and gists, we compare full CorMem with a simpler flat alternative that retrieves these units directly.
† Significant improvement over Flat (p<.05).
LoCoMo F1 improves by 1.76 percentage points, with significant gains in both F1 and BLEU-1 (Holm-adjusted p=.0352 for both). PersonaMem-v2 accuracy improves by 3.50 percentage points (p=3.71×10−6). These gains demonstrate the practical benefit of the full structure over the simpler flat alternative, with memory content held fixed.
2. Cue–fact conversion improves HippoRAG 2 by 3.46 points
To test the transferable benefit of the cue–fact structure, we reuse HippoRAG 2’s frozen OpenIE outputs
(entity1, relation, entity2)and replace the directentity1–entity2connection withentity1–fact–entity2. GPT-4o-mini minimally verbalizes each triple as a fact sentence to express the original relation unambiguously. We retain HippoRAG 2’s models, candidate budget, PPR parameters, and Top-3 budget, while using verbalized fact embeddings and adding a fact-node restart channel. We evaluate the conversion on LoCoMo with GPT-4o-mini.† Holm-adjusted p<.05 versus the native graph.
Recall improves across all four question categories and all ten memory stores, with an overall gain of 3.46 percentage points (95% CI [2.19, 5.25], Holm-adjusted p=.004). These gains demonstrate the benefit of introducing the cue–fact representation into HippoRAG 2 using its frozen OpenIE evidence and existing retrieval framework.
3. Cue–fact propagation achieves 79.23% Recall@3 without fact seeds
Under the Table 2 protocol, we reuse the original query-derived seeds on the unchanged graph and zero only the fact-seed weights before renormalization.
† p=.002 versus w/o fact seeds.
Cue-seeded propagation achieves 79.23% Recall@3, above HippoRAG 2’s 73.93%, showing that supporting evidence remains accessible through cues even without direct query–fact activation. (disabling cue seeds also significantly reduces Recall@3 to 84.08%, p=.004)