- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether persistent memory in long-term agents is reliable at the moment it is written. The authors view each memory as a derivation from earlier interaction history, so its content should be supported by the history available at write time. Building on this idea, the paper introduces DerivAudit, a framework that looks for supporting evidence beyond the citations provided by the memory writer, checks whether combining individual facts produces meaning the history never established, and examines how write-time verification decisions affect later memory use.
Experiments use natural memory writes from LoCoMo and HaluMem-Medium together with controlled data built by the authors, with Qwen, Gemma, and Llama as verification models. The paper finds that many memories that look unsupported from their citations alone are in fact supported once broader pre-write history is considered. At the same time, plausible premises spread across several interactions can make unsupported compositions harder to detect, which the paper calls the distributed-evidence paradox. Asking a verifier to check more true details can also lead it to reject valid memories, and downstream experiments show that both distorted and missing memories affect later answers.
Critical strengths
CS1. The paper raises the reliability of memory writing as a problem in its own right, which is a valuable perspective. Most prior work focuses on how memories are retrieved and used, whereas this paper asks whether a memory is actually supported by the history at the time it is written, and it distinguishes incomplete provenance from unsupported composition. This framing offers useful guidance for designing the write stage of memory systems.
CS2. The experimental design and reporting are solid. The paper analyzes both natural and controlled data systematically, and it reports non-significant or unfavorable results openly rather than avoiding them. This makes the conclusions more credible and gives future work a reliable basis to build on.
Critical weaknesses
CW1. The evidence for the distributed-evidence paradox comes almost entirely from Qwen, which makes it hard to support the paper's central conclusion.
The distributed-evidence paradox is the paper's main finding, yet its strong form only appears on Qwen3-30B-A3B. Under distributed evidence, holistic and obligation-aware verification detect only about one in ten invalid Qwen cases, while Gemma and Llama in the same setting still detect roughly 70–80%. The mechanism control used to explain this pattern was also run only with the Qwen verifier. Since Qwen3-30B-A3B activates only about 3B parameters per token, it is difficult to tell whether this pattern is a general problem of memory verification or mainly reflects the limitations of a model with a small active parameter count.
CW2. The proposed composition-aware verification views do not show a consistent advantage.
The three composition-aware views involve no training and differ mainly in how the memory is rendered in the prompt. On Qwen in Table 3, none of them matches the holistic judge in unsupported detection, and the gain in multi-span retention amounts to roughly four of the 39 multi-span memories. Compact relations performs best on Qwen, yet on Gemma it rejects all 39 multi-span memories. The appendix itself concludes that no representation is a universal repair, and changing only the paraphrase template and few-shot examples already produces variation larger than most between-interface gaps under default prompts. On the current evidence, much of the difference in Table 3 may come from prompt and rendering choices rather than from the views themselves.
Questions for authors
- The accumulated verification noise experiment adds obligations that are "independently known to be true" to supported memories, but it does not state where these obligations come from or how they are presented in the prompt. Could the authors clarify the exact setup?
- The admission rule only decides whether a new memory is added. When a newly admitted memory conflicts with an earlier stored record, for example after a change of residence, which record is treated as authoritative at read time? Are memory records timestamped, and does DerivAudit consider invalidating or revising earlier records?
- Under distributed evidence, the sharp drop in detection in Figure 3(b) appears mainly on Qwen. Could the controlled evidence distribution experiment be repeated on at least one closed commercial model (such as GPT or Claude) and on different sizes of the open model families used in the paper? If a similar drop is still observed on these stronger models, I would consider the paper's central finding to be well supported and would raise my score accordingly.
- In Table 3, the performance of the composition-aware views differs substantially between Qwen and Gemma, no results are reported for Llama, and multi-span retention is based on only 39 multi-span memories. Could these views be run on all three backbones, with the set of multi-span valid memories expanded to a size that supports significance testing?
5 comments
Response to Cross-Model Generality of the Distributed-Evidence Paradox
Under distributed evidence, the sharp drop in detection in Figure 3(b) appears mainly on Qwen. Could the controlled evidence distribution experiment be repeated on at least one closed commercial model (such as GPT or Claude) and on different sizes of the open model families used in the paper? If a similar drop is still observed on these stronger models, I would consider the paper's central finding to be well supported and would raise my score accordingly.
We sincerely thank the reviewer for raising this concern. To assess cross-model generality, we additionally evaluate GPT-6 Astra and GPT-5.6 on the same 70 matched families (280 cases), preserving the evidence conditions and verification protocol.
| Model | Local Detection ↑ | Distributed Detection ↑ | Drop (pp) |
|---|---|---|---|
| GPT-6 Astra | 98.6 | 82.9 | 15.7 |
| GPT-5.6 | 94.3 | 78.6 | 15.7 |
These results extend the distributed-evidence finding to stronger commercial models, suggesting that the failure mode persists across model families, although its severity varies considerably.
Response to Cross-Backbone Evaluation and Multi-Span Sample Size
In Table 3, the performance of the composition-aware views differs substantially between Qwen and Gemma, no results are reported for Llama, and multi-span retention is based on only 39 multi-span memories. Could these views be run on all three backbones, with the set of multi-span valid memories expanded to a size that supports significance testing?
We sincerely thank the reviewer for raising this concern. To strengthen statistical robustness, we expanded the valid multi-span subset from 39 to 200 independently validated cases, using the same evaluation protocol.
| Qwen Sample | Holistic ↑ | Compact Relations ↑ | Composition Graph ↑ | Per-Obligation ↑ |
|---|---|---|---|---|
| Original (39) | 84.6 | 94.9 | 87.2 | 89.7 |
| Expanded (200) | 80.2 | 91.4 | 85.0 | 86.9 |
All values are retention percentages. The expanded results preserve the original pattern: compact relations achieve the highest retention (91.4%), outperforming holistic verification by 11.2 points.
Hi, just a quick reminder that we’ve posted our rebuttal. Looking forward to your feedback, and thanks again for taking the time to review!
Response to Reviewer wN9P
Thanks for your insightful review. We appreciate your recognition of the value of our work!
Response to Cross-Backbone Generality of the Distributed-Evidence Paradox
We sincerely thank the reviewer for raising this concern. Our distributed-evidence evaluation already covers Qwen, Gemma, and Llama, revealing substantial backbone-dependent differences in unsupported-memory detection. To further assess generality, we will extend the plausible-premise replacement controls to Gemma and Llama, reporting paired confidence intervals.
Response to Consistency of Composition-Aware Verification
We sincerely thank the reviewer for raising this concern. We agree that composition-aware views do not consistently outperform established methods. Our contribution is to characterize the trade-off between unsupported-relation detection and valid multi-span retention, rather than propose a universally superior verifier.
In Table 3, Qwen's compact-relation view improves multi-span retention from 84.6% to 94.9%, with comparable detection (59.5% vs. 60.8%) relative to holistic verification. On Gemma, per-obligation checking improves detection from 93.7% to 98.7% but reduces retention from 92.3% to 76.9%. Appendix G further demonstrates sensitivity to representation and prompting. These findings show that explicit composition checking does not guarantee improved verification reliability, reinforcing the importance of evaluating both detection and retention.
Response to Accumulated Verification Noise Setup
We sincerely thank the reviewer for raising this concern. Additional obligations are drawn from other memory records in the same conversation, restricted to records supported under expanded-history auditing with individually entailed obligations.
For each supported memory, we vary the number of additional checks (k = 0, 2, 4), with k = 2 forming a prefix of k = 4. These obligations are presented as additional verification checks, while the candidate memory, evidence, and reference validity remain fixed.
Response to Temporal Conflicts and Memory Revision
We sincerely thank the reviewer for raising this concern. DerivAudit tracks write-time boundaries and distinguishes temporal updates from contradictions (Appendix B), but does not prescribe read-time precedence or automatic revision. To investigate this, we additionally evaluate 100 temporal-update cases under four conflict-resolution policies.
All values are percentages. Explicit supersession improves both current-state accuracy (81.8% → 85.4%) and historical accuracy (32.1% → 84.7%), whereas latest-record selection sacrifices historical information.