- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies derived memory maintenance, where updating a source fact may invalidate previously stored conclusions or plans even when their revised values are not explicitly provided. It introduces DRM-bench, comprising 3,200 two-session scenarios across semantic, episodic, and procedural memory, with model-generated answers written back into memory before subsequent updates. DUNE constructs a proposition graph with similarity, contradiction, derivation, and revision relations, and separately estimates validity and currentness to suppress stale memories during retrieval. Experiments report improved average source-update and derived-update accuracy over the evaluated retrieval and memory baselines.
Critical strengths
-
The paper identifies a useful distinction between updating source facts and maintaining their derived consequences. DRM-bench explicitly stores earlier model-generated answers and evaluates both source and derived questions after an update, making stale-answer reuse visible.
-
DUNE achieves the highest average derived-update accuracy among the evaluated methods, exceeding the strongest baseline by 12.2 percentage points. The component ablations further support the usefulness of dependency tracking and update propagation for maintaining derived memories.
Critical weaknesses
- The graph construction component (Section 4.1) is insufficiently evaluated. How accurately are proposition nodes and dependency edges extracted, particularly for implicit dependencies? An error analysis and controlled perturbations of missing or incorrect edges would help assess how graph construction errors affect downstream updates.
- The benefit of DUNE’s separate validity and currentness scores remains unclear. Could the authors add ablations using a single activity score and simple dependency-based invalidation, while keeping the extracted graph fixed? The existing ablations do not isolate the contribution of this two-score formulation and its optimization.
- The accuracy–cost trade-off under the main experimental setting remains unclear. Accuracy is evaluated with Qwen-Flash, whereas the cost study uses Claude Haiku 4.5. Could the authors report costs under the main setting so that the performance gains and computational overhead can be assessed together?
Questions for authors
Please refer to Weaknesses.
3 comments
Rebuttal2
Question2: Ablations of two score
We further compared two alternative approaches: a Single-activity method that relies on a unified activity score, and a Simple-invalidation method that applies straightforward invalidation based on dependencies. Both baselines followed the same two-stage evaluation protocol as our main model across 200 DRM-SE scenarios, which included writing the true answer from QA1 back into the memory. The results are presented below:
| Update Strategy | Source Update | Derived Update | Overall |
|---|---|---|---|
| Single activity | 61.50% | 45.50% | 53.50% |
| Simple invalidation | 74.00% | 50.50% | 62.25% |
| Full (Ours) | 81.50% | 51.00% | 66.25% |
The Overall score represents the combined accuracy for both Source and Derived QA2 problems, with each approach evaluated on a total of 400 instances. As shown in the table, our comprehensive state update mechanism is crucial for accurately maintaining both direct source facts and derived dependencies, as simplified strategies fail to capture the full complexity of the updating process.
Rebuttal3
Question3: The cost evaluated with Qwen-Flash.
We further measured the paired inference costs of DUNE and BM25 RAG. Both methods were evaluated on the same 50 DRM-SE scenarios using the Qwen-Flash model, following the complete two-stage protocol (Session 1, QA1, writing back the ground-truth answer, appending Session 2, and QA2). Each method generated 200 responses (400 in total), with token usage recorded via the API, no Judge model was invoked during this cost measurement.
As shown in Table 1, while both methods require the same number of LLM calls per scenario, DUNE consumes approximately 3.16× more input tokens and 6.32× more output tokens under the native configuration. Notably, graph construction, reasoning, and index building are performed via local computation.
Table 1: Cost comparison between DUNE and BM25 RAG (averaged per scenario over two stages).
| Metric | DUNE | BM25 RAG |
|---|---|---|
| LLM Calls | 4 | 4 |
| Input Tokens | 11,149.76 | 3,524.76 |
| Output Tokens | 683.86 | 108.22 |
| Graph/Index Build Time (s) | 0.015601 | 0.000130 |
| Two-stage Retrieval Time (s) | 0.000706 | 0.000315 |
| Total HTTP Request Time (s) | 17.912 | 10.667 |
| Peak Process RSS (MiB) | 1,736.08 | 758.44 |
Rebuttal
Thanks for your time and valuable feedback!
Question 1: A direct evaluation of graph construction quality.
We conducted two rounds of manual annotation on a sample of 50 instances, with the following results:
These results provide a direct quantitative evaluation of graph construction quality. As a complementary analysis, we also applied controlled perturbations of 0%, 10%, 25%, and 50% to the constructed graphs and repeatedly ran the propagation solver. All 20 runs converged, and we observed no flips in binary states. This result indicates that, within the tested perturbation range, the propagation process remains stable.