acceptodds
Under review as a conference paper at ICLR 2027

Memory Is a Derivation: The Distributed-Evidence Paradox in Long-Term Agents

Abstract

Long-running LLM agents compress past interactions into persistent memories that may be reused as premises for later tasks. This creates a distinct derivation problem: whether the memory actually follows from what the interaction history supports. Relevant evidence may be scattered across earlier interactions, while compression can introduce relations or event status that the history never established. A valid memory may therefore appear unsupported because its citations omit relevant evidence, while individually supported facts may be composed into a stronger statement the history never established. We characterize this problem through three coupled requirements: (1) Evidence scope; (2) Compositional validity; (3) Admission reliability. We therefore ask whether the interaction history available at write time supports what enters persistent memory. We introduce DerivAudit, a framework for auditing whether a memory is actually supported by the history available when it was written. The audit separates three questions: whether supporting evidence lies beyond writer-provided citations, whether the composed memory introduces unsupported meaning, and how write-time admission decisions affect later memory use. Across two natural memory corpora, audits using broader pre-write history recover support for nearly 60% of memories that appear unsupported from citations alone, while 17–21% remain unsupported after expansion. Yet broader evidence does not by itself make admission reliable: unsupported memories are still frequently admitted (59–89%) across verification models, and evidence expansion alone worsens it on two backbones. Controlled experiments reveal a distributed-evidence paradox: valid memories often require combining evidence across interactions, yet several plausible facts can make unsupported compositions harder to detect. Moreover, requiring a verifier to check more independently true details can itself increase rejection of valid memories. On 391 labelled real memory writes, composition-aware verification with expanded history improves retention of valid memories rescued by broader-history evidence but does not consistently reduce unsupported admission across backbones. Downstream probes further show that both distorted and missing memories can harm later answers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent Memory, Evidence Retrieval, Semantic Derivation Integrity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wN9POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether persistent memory in long-term agents is reliable at the moment it is written. The authors view each memory as a derivation from earlier interaction history, so its content should be supported by the history available at write time. Building on this idea, the paper introduces DerivAudit, a framework that looks for supporting evidence beyond the citations provided by the memory writer, checks whether combining individual facts produces meaning the history never established, and examines how write-time verification decisions affect later memory use.

Experiments use natural memory writes from LoCoMo and HaluMem-Medium together with controlled data built by the authors, with Qwen, Gemma, and Llama as verification models. The paper finds that many memories that look unsupported from their citations alone are in fact supported once broader pre-write history is considered. At the same time, plausible premises spread across several interactions can make unsupported compositions harder to detect, which the paper calls the distributed-evidence paradox. Asking a verifier to check more true details can also lead it to reject valid memories, and downstream experiments show that both distorted and missing memories affect later answers.

Critical strengths

CS1. The paper raises the reliability of memory writing as a problem in its own right, which is a valuable perspective. Most prior work focuses on how memories are retrieved and used, whereas this paper asks whether a memory is actually supported by the history at the time it is written, and it distinguishes incomplete provenance from unsupported composition. This framing offers useful guidance for designing the write stage of memory systems.

CS2. The experimental design and reporting are solid. The paper analyzes both natural and controlled data systematically, and it reports non-significant or unfavorable results openly rather than avoiding them. This makes the conclusions more credible and gives future work a reliable basis to build on.

Critical weaknesses

CW1. The evidence for the distributed-evidence paradox comes almost entirely from Qwen, which makes it hard to support the paper's central conclusion.

The distributed-evidence paradox is the paper's main finding, yet its strong form only appears on Qwen3-30B-A3B. Under distributed evidence, holistic and obligation-aware verification detect only about one in ten invalid Qwen cases, while Gemma and Llama in the same setting still detect roughly 70–80%. The mechanism control used to explain this pattern was also run only with the Qwen verifier. Since Qwen3-30B-A3B activates only about 3B parameters per token, it is difficult to tell whether this pattern is a general problem of memory verification or mainly reflects the limitations of a model with a small active parameter count.

CW2. The proposed composition-aware verification views do not show a consistent advantage.

The three composition-aware views involve no training and differ mainly in how the memory is rendered in the prompt. On Qwen in Table 3, none of them matches the holistic judge in unsupported detection, and the gain in multi-span retention amounts to roughly four of the 39 multi-span memories. Compact relations performs best on Qwen, yet on Gemma it rejects all 39 multi-span memories. The appendix itself concludes that no representation is a universal repair, and changing only the paraphrase template and few-shot examples already produces variation larger than most between-interface gaps under default prompts. On the current evidence, much of the difference in Table 3 may come from prompt and rendering choices rather than from the views themselves.

Questions for authors
  1. The accumulated verification noise experiment adds obligations that are "independently known to be true" to supported memories, but it does not state where these obligations come from or how they are presented in the prompt. Could the authors clarify the exact setup?
  2. The admission rule only decides whether a new memory is added. When a newly admitted memory conflicts with an earlier stored record, for example after a change of residence, which record is treated as authoritative at read time? Are memory records timestamped, and does DerivAudit consider invalidating or revising earlier records?
  3. Under distributed evidence, the sharp drop in detection in Figure 3(b) appears mainly on Qwen. Could the controlled evidence distribution experiment be repeated on at least one closed commercial model (such as GPT or Claude) and on different sizes of the open model families used in the paper? If a similar drop is still observed on these stronger models, I would consider the paper's central finding to be well supported and would raise my score accordingly.
  4. In Table 3, the performance of the composition-aware views differs substantially between Qwen and Gemma, no results are reported for Llama, and multi-span retention is based on only 39 multi-span memories. Could these views be run on all three backbones, with the set of multi-span valid memories expanded to a size that supports significance testing?
5 comments

Response to Reviewer wN9P

Thanks for your insightful review. We appreciate your recognition of the value of our work!


Response to Cross-Backbone Generality of the Distributed-Evidence Paradox

The evidence for the distributed-evidence paradox comes almost entirely from Qwen, which makes it hard to support the paper's central conclusion.

We sincerely thank the reviewer for raising this concern. Our distributed-evidence evaluation already covers Qwen, Gemma, and Llama, revealing substantial backbone-dependent differences in unsupported-memory detection. To further assess generality, we will extend the plausible-premise replacement controls to Gemma and Llama, reporting paired confidence intervals.

ModelHolistic (%)Obligation-aware (%)QA (%)
Qwen8.611.4~46
Gemma~77~76~74
Llama~79~73~73

Response to Consistency of Composition-Aware Verification

The proposed composition-aware verification views do not show a consistent advantage.

We sincerely thank the reviewer for raising this concern. We agree that composition-aware views do not consistently outperform established methods. Our contribution is to characterize the trade-off between unsupported-relation detection and valid multi-span retention, rather than propose a universally superior verifier.

In Table 3, Qwen's compact-relation view improves multi-span retention from 84.6% to 94.9%, with comparable detection (59.5% vs. 60.8%) relative to holistic verification. On Gemma, per-obligation checking improves detection from 93.7% to 98.7% but reduces retention from 92.3% to 76.9%. Appendix G further demonstrates sensitivity to representation and prompting. These findings show that explicit composition checking does not guarantee improved verification reliability, reinforcing the importance of evaluating both detection and retention.


Response to Accumulated Verification Noise Setup

The accumulated verification noise experiment adds obligations that are "independently known to be true" to supported memories, but it does not state where these obligations come from or how they are presented in the prompt. Could the authors clarify the exact setup?

We sincerely thank the reviewer for raising this concern. Additional obligations are drawn from other memory records in the same conversation, restricted to records supported under expanded-history auditing with individually entailed obligations.

For each supported memory, we vary the number of additional checks (k = 0, 2, 4), with k = 2 forming a prefix of k = 4. These obligations are presented as additional verification checks, while the candidate memory, evidence, and reference validity remain fixed.


Response to Temporal Conflicts and Memory Revision

The admission rule only decides whether a new memory is added. When a newly admitted memory conflicts with an earlier stored record, for example after a change of residence, which record is treated as authoritative at read time? Are memory records timestamped, and does DerivAudit consider invalidating or revising earlier records?

We sincerely thank the reviewer for raising this concern. DerivAudit tracks write-time boundaries and distinguishes temporal updates from contradictions (Appendix B), but does not prescribe read-time precedence or automatic revision. To investigate this, we additionally evaluate 100 temporal-update cases under four conflict-resolution policies.

Memory PolicyCurrent Acc. ↑Historical Acc. ↑Stale Errors ↓
No resolution81.832.14.4
Latest-applicable83.216.82.9
Explicit supersession85.484.72.9
Oracle selection83.285.43.6

All values are percentages. Explicit supersession improves both current-state accuracy (81.8% → 85.4%) and historical accuracy (32.1% → 84.7%), whereas latest-record selection sacrifices historical information.


Response to Cross-Model Generality of the Distributed-Evidence Paradox

Under distributed evidence, the sharp drop in detection in Figure 3(b) appears mainly on Qwen. Could the controlled evidence distribution experiment be repeated on at least one closed commercial model (such as GPT or Claude) and on different sizes of the open model families used in the paper? If a similar drop is still observed on these stronger models, I would consider the paper's central finding to be well supported and would raise my score accordingly.

We sincerely thank the reviewer for raising this concern. To assess cross-model generality, we additionally evaluate GPT-6 Astra and GPT-5.6 on the same 70 matched families (280 cases), preserving the evidence conditions and verification protocol.

ModelLocal Detection ↑Distributed Detection ↑Drop (pp)
GPT-6 Astra98.682.915.7
GPT-5.694.378.615.7

These results extend the distributed-evidence finding to stronger commercial models, suggesting that the failure mode persists across model families, although its severity varies considerably.


Response to Cross-Backbone Evaluation and Multi-Span Sample Size

In Table 3, the performance of the composition-aware views differs substantially between Qwen and Gemma, no results are reported for Llama, and multi-span retention is based on only 39 multi-span memories. Could these views be run on all three backbones, with the set of multi-span valid memories expanded to a size that supports significance testing?

We sincerely thank the reviewer for raising this concern. To strengthen statistical robustness, we expanded the valid multi-span subset from 39 to 200 independently validated cases, using the same evaluation protocol.

Qwen SampleHolistic ↑Compact Relations ↑Composition Graph ↑Per-Obligation ↑
Original (39)84.694.987.289.7
Expanded (200)80.291.485.086.9

All values are retention percentages. The expanded results preserve the original pattern: compact relations achieve the highest retention (91.4%), outperforming holistic verification by 11.2 points.

Hi, just a quick reminder that we’ve posted our rebuttal. Looking forward to your feedback, and thanks again for taking the time to review!

Reviewer_TAZVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of long-running LLM agents when writing to persistent memory. The authors conceptualize "memory writing" as a form of inference drawn from historical interactions and propose the criterion of "semantic derivation integrity"—the requirement that any written memory must be supported by the history of interactions that preceded the moment of writing. Based on this, the authors introduce an auditing framework called DerivAudit, which conducts analysis across three dimensions: scope of evidence, compositional validity, and admission reliability. The experimental evaluation utilizes 400 unedited, naturally occurring memory writes from the LoCoMo and HaluMem-Medium datasets, employing three verification models—Qwen3-30B-A3B, Gemma-4-31B, and Llama-3.3-70B—and is further supplemented by controlled, constructed experiments.

Critical strengths
  1. Many agentic systems rely on long-term memory; the risk that "information written into memory is subsequently treated as fact and repeatedly utilized" is very real. Figure 1 illustrates this issue clearly with an example—moving to Boston for a new job—making the problem immediately apparent.
  2. The materials provided for reproducibility are quite comprehensive. The appendix includes the full prompts used for all validation interfaces, the criteria for threshold selection, the annotation workflow, and a detailed breakdown of sample attrition. Furthermore, the reproducibility statement lists the accompanying code, configuration settings, and random seeds.
Critical weaknesses
  1. Some of the conclusions in the paper are based on samples that are too small; while the figures may appear definitive, they could actually be highly incidental and should not be taken as universal principles.
  2. The "harm" query is designed such that the answer shifts in response to the distortion, while the "content" query is designed to be answerable only by the specific record in question. Given this setup, it is almost inevitable that introducing distorted memories will lead to failure, and removing those memories will render the model unable to provide an answer. While these experiments demonstrate that the reader model relies on its memory, they offer limited insight into the actual, practical impact of errors introduced during the "writing" (memory-encoding) phase. The conclusions would be far more compelling if the impact could be estimated within the context of real-world retrieval scenarios and actual task distributions.
  3. The sharp drop in Gemma's pass rate can be explained by the verifier correctly identifying a lack of sufficient evidence—it does not necessarily mean that "more checks lead to a higher chance of error."
Questions for authors
  1. For the 70 matched clusters, what are the retention rate of valid memories and the detection rate of invalid memories under local conditions? Could you provide the full 2×2 contingency table of the results?
  2. What is the sample size for the manual review? What is the inter-rater agreement (or consistency rate) for the category of "initially under-cited but recovered through expansion"?
  3. Which models were used to generate the reference labels, and do they belong to the same family as the models being evaluated? What is the relationship between the 160 doubly-annotated records mentioned in Appendix C and the total set of 400 records?
3 comments

Response to Reviewer TAZV

Thanks for your insightful review. We appreciate your recognition of the value of our work!


Response to Sample Size and Statistical Robustness

Some of the conclusions in the paper are based on samples that are too small; while the figures may appear definitive, they could actually be highly incidental and should not be taken as universal principles.

We sincerely thank the reviewer for raising this concern. Our existing evaluation covers 400 natural writes, 70 matched families (280 cases), and 512 controlled records, providing substantial evidence through systematic paired comparisons. To further assess generality, we additionally evaluate 100 longer-memory writes and 30 longer-memory families.

Additional EvaluationMetricResult95% CI
100 longer writesProvenance repair37.5%[25.6, 50.0]
Unsupported writes26.0%[18.0, 35.0]
30 longer families (k=8)Holistic detection23.3% → 0.0%Drop: 23.3 pp [10.0, 40.0]
30 longer families (k=128)Holistic detection33.3% → 20.0%Drop: 13.3 pp [−10.0, 36.7]

Detection rates compare local → distributed evidence. These additional results confirm that provenance gaps persist in longer memories.


Response to Practical Downstream Impact

The "harm" query is designed such that the answer shifts in response to the distortion, while the "content" query is designed to be answerable only by the specific record in question…

We sincerely thank the reviewer for raising this concern. Our original controlled probes isolate the effects of memory errors, while natural retrieval analysis shows that invalid retrieved memories cause misinformation in 96.2% of Qwen and 100% of Gemma cases. To assess practical impact, we additionally evaluate natural memory writes on benchmark QA tasks under unforced retrieval, comparing admission policies with identical reader and retrieval settings.

Admission PolicyFalse Admit ↓False Reject ↓Misinformation ↓
No gate100.00.014.1
Citation-only84.69.612.8
Expanded history80.89.911.4
+ Obligation-aware76.99.311.8
Oracle0.00.011.8

All values are percentages. Expanded-history verification reduces misinformation from 14.1% to 11.4%, demonstrating practical benefits beyond controlled probes.


Response to Evidence Sufficiency in Additional Verification

The sharp drop in Gemma's pass rate can be explained by the verifier correctly identifying a lack of sufficient evidence—it does not necessarily mean that "more checks lead to a higher chance of error."

We sincerely thank the reviewer for raising this concern. To distinguish verification noise from insufficient evidence, we additionally test obligations drawn directly from supported candidate memories (k=0,2,4), keeping the memory, evidence, and verifier fixed.

ModelOriginal (k=0 → 4)Within-Memory k=0k=2k=4
Qwen99.4% → 98.1%100.0100.099.3
Gemma93.3% → 2.6%97.891.893.3
Llama97.4% → 35.5%97.097.897.8

Unlike the original experiment, within-memory checking largely preserves acceptance, particularly for Gemma (97.8% → 93.3%) and Llama (97.0% → 97.8%).


Response to Local-Condition Results and Full 2×2 Comparison

For the 70 matched clusters, what are the retention rate of valid memories and the detection rate of invalid memories under local conditions? Could you provide the full 2×2 contingency table of the results?

We sincerely thank the reviewer for raising this concern. Our evaluation includes 70 matched families (280 cases) covering local/distributed evidence × valid/invalid memories. We provide the complete 2×2 results for Qwen below, reporting valid-memory retention and invalid-memory detection.

MethodLocal Valid ↑Local Invalid ↑Distributed Valid ↑Distributed Invalid ↑
Holistic100.044.3100.08.6
Obligation-aware100.061.4100.011.4
Predicate–argument QA98.677.1100.0~46

These results further support our finding that distributed plausible premises make unsupported compositions harder to reject, rather than simply reducing valid-memory retention.


Response to Manual Review Sample Size and Agreement

What is the sample size for the manual review? What is the inter-rater agreement (or consistency rate) for the category of "initially under-cited but recovered through expansion"?

We sincerely thank the reviewer for raising this concern. Our original manuscript reports 96.0% human–model agreement on a stratified sample (§5). To specifically validate provenance recovery, we additionally conducted independent human review of all 112 provenance-repaired memories, evaluating both citation-only and expanded-history evidence.

DatasetWritesHuman Agreement
LoCoMo5194.1% (48/51)
HaluMem-Medium6198.4% (60/61)
Total11296.4% (108/112)

The 96.4% agreement provides additional support for the reliability of our provenance-recovery labels.


Response to Reference Labeling Models and Annotation Scope

Which models were used to generate the reference labels, and do they belong to the same family as the models being evaluated? What is the relationship between the 160 doubly-annotated records mentioned in Appendix C and the total set of 400 records?

We sincerely thank the reviewer for raising this concern. Reference labels were generated using GPT-6 Astra, with independent annotation instances and blinded adjudication. This model belongs to a different family from all evaluated backbones (Qwen3-30B-A3B, Gemma-4-31B, and Llama-3.3-70B-Instruct), avoiding direct same-family evaluation bias.

The 160 doubly annotated records in Appendix C are a subset of the 400 natural writes (200 LoCoMo + 200 HaluMem-Medium), used for detailed inter-annotator agreement analysis (643 initial obligations, 631 retained). All 400 writes enter the evidence-scope audit, while 391 with resolved labels enter admission evaluation.

Hi, just a quick reminder that we’ve posted our rebuttal. Looking forward to your feedback, and thanks again for taking the time to review!

Reviewer_qd2FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly focuses on a problem that the persistent memories, which are derived from previous interactions and will be used as a premise for later tasks, may result in a derivation problem: whether the memory actually follows from what the interaction history supports. This problem may mainly be derived from the scattered effective earlier interactions. To study this problem, this paper characterizes the problem into three coupled requirements and proposes DerivAudit to audit whether a memory is actually supported by the history available when it was written. For validation, extensive empirical experiments are conducted.

Critical strengths
  • The problem studied in this paper is important and meaningful. As memory becomes more and more important in agentic frameworks, agentic memory has been faced with a trade-off between preserving as much useful information as possible and reducing unnecessary parts for budget saving. However, this inevitably results in the derivation problem mentioned in this paper. Thus, I appreciate the problem studied in this paper.

  • Authors honestly report the negative results and variations across verification models. Due to the complicated structure of agentic systems, not all strategies work well to facilitate the performance. The negative results are essential to examine the capability of agentic systems in specific tasks.

Critical weaknesses
  1. The abstract is too long to read. A brief but informative abstract is required in a research paper. Note that this is a minor weakness, which does not affect my evaluation of the paper.

  2. This paper should further clearly distinguish itself from general factuality verification. Specifically, the unsupported relations, incomplete evidence, and errors mentioned in the paper are also concerns in topics like summarization and entailment verification. Meanwhile, most experiments in this paper evaluate a static memory-evidence pair instead of self-improved agents. It would be better if the authors could highlight the findings that are specific to persistent memories, instead of just applying the existing verification methods to conversation summaries.

  3. I noted that reference judgements are mainly generated by the model in this paper. In such a case, the failure to retrieve support evidence may resemble an unsupported memory. Although treating those cases as unsolved may potentially help, it does not address relevant passages missed. This uncertainty may influence both the estimated prevalence of unsupported writes and verifier error rates.

  4. For the "distributed-evidence paradox" case in the experiment section (e.g., Appendix G), an important finding here is that replacing seemingly reasonable premises with irrelevant passage support that seemingly relevant evidence encourages the acceptance of unsupported cases. However, this intervention modifies semantic relevance and how available the plausible premise is simultaneously.

Questions for authors
  1. Among the experiments in the paper, are there any findings that are specific to persistent memories and general factual consistency checking, respectively?

  2. The conclusions regarding natural audits are derived from experiments with certain assumptions. So, how about the conclusions under the assumptions where stronger retrieval or human inspection of the complete history?

  3. Whether additional checking remains harmful in the case where each additional obligation is a part of the candidate memory?

  4. Consider a case where false admission and false rejection happen in the same agent workflow. In such a case, whether the admision policies help improve downstream performance?

4 comments

Response to Reviewer qd2F

Thanks for your insightful review. We appreciate your recognition of the value of our work!


Response to Abstract Length

The abstract is too long to read. A brief but informative abstract is required in a research paper. Note that this is a minor weakness, which does not affect my evaluation of the paper.

We sincerely thank the reviewer for this suggestion. We have shortened and streamlined the abstract in the newly uploaded PDF to improve readability while preserving the core motivation, approach, and key findings.


Response to Novelty

It would be better if the authors could highlight the findings that are specific to persistent memories, instead of just applying the existing verification methods to conversation summaries. Among the experiments in the paper, are there any findings that are specific to persistent memories and general factual consistency checking, respectively?

We sincerely thank the reviewer for raising this concern. While these error types overlap with general factuality verification, our contribution is to examine memory formation, write-time admission, and subsequent reuse. Our findings can be distinguished as follows:

ScopeFindingsEvidence
Memory formationProvenance repair; admission trade-offs§4.1–4.2; Table 2
Memory persistenceConsolidation drift; downstream costs§4.3; Fig. 4
General verificationDistributed-evidence paradox; verification noise§4.2; Tables 6–7

Specifically, expanded history recovers support for ~60% of citation-insufficient memories and improves Qwen's repaired-memory retention from 82.1% to 93.8% (p = 0.007), while unsupported admission remains inconsistent across models. Lifecycle experiments further reveal consolidation drift and downstream costs of distorted or missing memories.


Response to Reference Judgments and Retrieval Uncertainty

This uncertainty may influence both the estimated prevalence of unsupported writes and verifier error rates.

We sincerely thank the reviewer for raising this concern. We acknowledge that retrieval failure does not establish historical invalidity. Our protocol supplements writer citations with up to 12 BM25-retrieved pre-write passages (16 total), marks search-limited cases as unresolved, and restricts admission evaluation to 391 decidable writes (Appendix D). Reference labels use two independent model assessments with blinded adjudication and separate entailment/refutation checks, while stratified human review shows 96% agreement.

While these safeguards cannot eliminate missed evidence, our controlled composition experiments establish validity by construction and manipulate evidence through deletion and replacement, independently supporting the compositional findings.


Response to Distributed-Evidence Paradox Controls

However, this intervention modifies semantic relevance and how available the plausible premise is simultaneously.

We sincerely thank the reviewer for raising this concern. We agree that replacing plausible premises with irrelevant passages changes both semantic relevance and premise availability, preventing full isolation of their effects. However, our matched experiments control candidate memory, validity, and evidence budget. Removing sufficient support flips 88–100% of decisions to rejection, while removing redundant support causes no flips. Replacing plausible premises with token- and position-matched irrelevant history further reduces unsupported-memory acceptance from 65–88% to 0–10%. These results support the role of plausible but non-licensing evidence in erroneous acceptance, although the two factors cannot be independently disentangled.


Response to Memory-Specific vs. General Findings

Among the experiments in the paper, are there any findings that are specific to persistent memories and general factual consistency checking, respectively?

We sincerely thank the reviewer for raising this concern. Our general verification findings include the distributed-evidence paradox and accumulated verification noise, highlighting difficulties in detecting unsupported compositions and checking multiple valid claims. In contrast, our memory-specific findings concern formation, admission, and reuse: expanded history recovers support for ~60% of citation-insufficient memories, while unsupported admission remains frequent. Moreover, repeated consolidation exhibits temporal and modality drift, and downstream interventions demonstrate the costs of both retaining distorted memories and discarding valid ones.


Response to Stronger Retrieval and Full-History Validation

The conclusions regarding natural audits are derived from experiments with certain assumptions. So, how about the conclusions under the assumptions where stronger retrieval or human inspection of the complete history?

We sincerely thank the reviewer for raising this concern. To assess robustness, we additionally evaluated all 400 writes using hybrid BM25+dense retrieval with two evidence budgets, and conducted full-history human inspection of 120 stratified cases covering supported, unsupported, provenance-repaired, and unresolved writes.

SettingProvenance Repair (%) LoCoMo / HaluMemUnsupported Writes (%) LoCoMo / HaluMemQwen Unsupported Admission (%)
Original BM25 (12/16)60.0 / 59.817.0 / 21.080.8
Hybrid (12/16)50.6 / 48.024.5 / 29.083.8
Hybrid (24/32)57.6 / 60.823.0 / 25.083.0
Full-history human (120)61.4 / 64.625.6 / 27.082.3

The results reinforce our central findings: substantial provenance recovery (48.0–64.6%) coexists with persistently high unsupported admission (80.8–83.8%).


Response to Within-Memory Verification Obligations

Whether additional checking remains harmful in the case where each additional obligation is a part of the candidate memory?

We sincerely thank the reviewer for raising this concern. To test whether additional checking remains harmful when obligations are intrinsic to the candidate memory, we conduct a new experiment on supported memories with at least four annotated obligations, varying explicitly checked obligations (k=0,2,4) while keeping the memory, evidence, and verifier fixed.

Admission PolicyFalse Admission (%) ↓False Rejection (%) ↓Misinformation (%) ↓
No gate100.00.014.1
Citation-only84.69.612.8
Expanded history80.89.911.4
+ Obligation-aware checking76.99.311.8
Oracle admission0.00.011.8

Unlike the sharp degradation with externally added obligations, within-memory checking largely preserves acceptance (Qwen −0.7 pp; Gemma −4.5 pp; Llama +0.8 pp).


Response to Mixed Admission Errors in Agent Workflows

Consider a case where false admission and false rejection happen in the same agent workflow. In such a case, whether the admision policies help improve downstream performance?

We sincerely thank the reviewer for raising this concern. To examine both error types jointly, we additionally evaluate mixed-error workflows containing supported and unsupported candidate memories, comparing admission policies under identical retrieval and downstream QA settings.

Admission PolicyFalse Admission (%) ↓False Rejection (%) ↓Misinformation (%) ↓
No gate100.00.014.1
Citation-only84.69.612.8
Expanded history80.89.911.4
+ Obligation-aware checking76.99.311.8
Oracle admission0.00.011.8

Expanded-history admission reduces downstream misinformation from 14.1% to 11.4%, despite introducing some false rejections.

Hi, just a quick reminder that we’ve posted our rebuttal. Looking forward to your feedback, and thanks again for taking the time to review!

Sorry for the late reply.

Thanks for the author's efforts in rebuttal. I have checked the responses. Generally, I think my concerns are well addressed in some cases. So far, I prefer to maintain my score.

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank all reviewers and area chairs for their careful evaluation, insightful comments, and constructive feedback. We appreciate their recognition of the importance of our research problem, systematic experimental design, and transparent reporting. Below, we summarize the key strengths of Memory Is a Derivation (DerivAudit) recognized across the reviews:

  • Novel perspective on reliable agent memory. All three reviewers recognized the importance of write-time memory reliability. Reviewer qd2F described the problem as "important and meaningful," TAZV emphasized its practical significance, and wN9P considered it "a valuable perspective."

  • Unified framework for memory reliability. DerivAudit jointly examines evidence scope, compositional validity, and admission reliability, distinguishing incomplete provenance from unsupported composition and connecting write-time decisions to subsequent memory use.

  • Systematic and transparent empirical study. Our evaluation includes 400 natural writes, 70 matched families (280 cases), and 512 controlled records. Reviewers qd2F and wN9P appreciated our transparent reporting of negative results, while TAZV highlighted the comprehensive reproducibility materials.

During the rebuttal period, we addressed reviewer concerns through additional experiments and clarifications, summarized below:

  1. Stronger retrieval and human validation. In response to qd2F and TAZV, we evaluated hybrid retrieval on 400 writes and full-history human inspection on 120 cases, confirming substantial provenance recovery (48.0–64.6%) alongside persistent unsupported admission (80.8–83.8%). Human review of 112 provenance-repaired memories further achieved 96.4% agreement.

  2. Model generality and statistical robustness. In response to wN9P, we evaluated GPT-6 Astra and GPT-5.6, both exhibiting 15.7-point detection drops under distributed evidence. We expanded valid multi-span evaluation from 39 to 200 cases and added 100 longer-memory writes and 30 longer-memory families, further examining the robustness and scope of our findings.

  3. Verification mechanisms and admission trade-offs. Additional within-memory obligation experiments clarify that verification degradation depends on evidence sufficiency, not simply check count. We further clarified the model-dependent severity of distributed-evidence failures and the limitations of composition-aware verification.

  4. Downstream impact and temporal reliability. New mixed-error evaluations show that expanded-history admission reduces misinformation from 14.1% to 11.4%, while temporal-update experiments demonstrate that explicit supersession improves both current-state and historical accuracy.

Overall, these additional results reinforce our central conclusion: reliable persistent memory requires not only recovering historical evidence, but also verifying whether that evidence supports the composed memory and understanding the downstream consequences of admission decisions. DerivAudit provides a systematic foundation for studying these challenges, while acknowledging model-dependent variations and limitations.

We sincerely thank all reviewers and area chairs again for their valuable feedback and consideration.

Best,

All Authors of Submission 19153

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.