acceptodds
Under review as a conference paper at ICLR 2027

When Correct Memory Goes Wrong: Fuzzing Persistent Memory Use in LLM Agents

Abstract

Persistent memory helps LLM agents carry information across long interactions, but correct memory can still be used incorrectly when queries change or memory states evolve. Existing work mainly studies memory content errors or evaluates fixed test cases, leaving memory-use failures hard to discover systematically. We formulate this issue as a fuzzing problem and categorize such failures into query-related and memory-state failures. We then develop U-Fuzz, which starts from memory checkpoints as test seeds, mutates queries or memory states under explicit mutation obligations, validates each mutant, and uses observed memory behavior to guide iterative testing while keeping failure labels outside the search. We evaluate U-Fuzz across several memory systems against diverse fuzzing baselines, and further test an output-only setting with API-based LLMs where memory retrieval is hidden. Across these settings, U-Fuzz consistently uncovers more confirmed memory-use failures, showing that its search remains effective across different memory architectures and even when only final responses are observable.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Memory Systems, Fuzz Testing, Reliability Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_15NmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies memory-use failures in persistent-memory LLM agents, where correct stored information is used incorrectly following query changes or memory-state updates. It proposes U-Fuzz, which constructs seeds from memory checkpoints, applies six validated query- and memory-state mutation operators, schedules mutations through operator–target obligations, and uses retrieval coverage, novelty, and ranking divergence to guide exploration without relying on failure labels. Experiments across four memory systems (Mem0, A-Mem, Graphiti, MemOS) and two benchmarks (LoCoMo, LongMemEval-S) show more unique failures than four baselines under equal execution budgets, supported by ablations and an output-only evaluation.

Critical strengths
  1. Careful testing methodology. U-Fuzz changes only one input per parent–mutant pair, validates mutants independently of their outputs, restricts unsupported queries to memory-exclusive targets, and separates failure labels from search feedback. These choices reduce confounding factors and oracle leakage, strengthening the testing methodology.

  2. Broad and consistent evaluation. The experiments cover four memory backends, two benchmarks, four baselines, multiple ablations, and a hidden-retrieval setting. U-Fuzz consistently discovers more failures (e.g., 45 vs. 28 on Mem0/LoCoMo and 73 vs. 36 on A-Mem), while the ablations suggest that coverage supports exploration and novelty/divergence help discover distinct behaviors.

Critical weaknesses
  1. The failure oracle and uniqueness metric are insufficiently specified, raising concerns about the validity of UF@B.

    • Uniqueness. The paper does not define what constitutes a "unique" failure. Since retained mutants can be mutated again (§4.3), multiple descendants may repeatedly expose the same underlying issue, potentially inflating UF@B. The authors should clarify the deduplication criteria and report raw versus unique failure counts.
    • Non-determinism. Stochastic retrieval or response generation may violate the mutation-specific expectation without indicating a reproducible failure. The paper does not report decoding settings, repeated executions, or failure reproducibility, making it difficult to distinguish systematic failures from stochastic fluctuations.
    • Oracle precision. Despite claiming "confirmed" failures, the paper provides no human validation or precision estimate and does not explain how Ans(y, a) is computed. Moreover, Figure 1 labels a ranking change as a failure even though correct evidence remains in the top-k, while §4.3 does not clearly establish whether ranking changes alone violate the oracle. The authors should clarify this criterion and distinguish retrieval-only failures from incorrect final answers, particularly when interpreting Table 1 versus Figure 5.
  2. Missing experimental details and unclear baseline fairness. The main-experiment LLM backend, mutation/validation models, retrieval depth k, attempt limit A, hidden-retrieval encoder/threshold, checkpoint counts, and execution cost are not reported. More importantly, it is unclear whether Random and Coverage-Guided share U-Fuzz's mutation operators and obligation scheduler. Without controlling these factors, the reported gains may conflate mutation design, scheduling, and feedback guidance. A controlled ablation isolating these contributions is needed.

Questions for authors
  1. How is a "unique" failure defined and deduplicated across mutation rounds? Can the authors report raw versus deduplicated counts and the number of distinct underlying memory entries affected?
  2. How is Ans(y, a) implemented, and has oracle precision been manually evaluated? Does a ranking-only change within the top-k count as a failure? What proportion of Table 1 failures actually produce incorrect final answers?
  3. What fraction of failures reproduce upon re-execution under identical settings, and what decoding parameters are used?
  4. Do all baselines share the same operators and obligation scheduler? Can the authors provide controlled ablations and the missing experimental configurations?
5 comments
AuthorsOpenReview ↗

Response, Clarification, and Revisions (1/5)

We thank the reviewer for the detailed assessment and constructive suggestions. We have revised the manuscript to clarify the failure oracle, failure uniqueness, experimental settings, and scope of the baseline comparisons. Please see detailed responses below:



1. Failure uniqueness and deduplication

Weakness #1.1: Uniqueness. The paper does not define what constitutes a "unique" failure. Since retained mutants can be mutated again (§4.3), multiple descendants may repeatedly expose the same underlying issue, potentially inflating UF@B.

Response and revision. We would like to first explain the distinction between a failure event and a unique failure in §5 (Metrics). A failure event is a validated execution satisfying F_o(x,x')=1. For deduplication, we define a failure key using the originating interaction history, the queried memory target, the implicated memory-entry lineage, and the violated mutation-specific expectation.

Failures with the same key count only once, regardless of the mutation round, descendant mutant, or query wording that exposes them. Different violated expectations remain distinguishable. This prevents repeated descendants exposing the same memory-use violation from being counted as independent discoveries.

We have also highlighted the iterative mutation mechanism in §4.2 and clarified the distinction among raw failure events, deduplicated failures, and distinct affected memory entries in §5.


Question 1: How is a "unique" failure defined and deduplicated across mutation rounds? Can the authors report raw versus deduplicated counts and the number of distinct underlying memory entries affected?

Clarification and revision. We would like to echo back to the definition of unique failures in §5 (Metrics). A raw failure is a validated execution that violates its mutation-specific expectation, i.e., Fo(x,x')=1. A unique failure represents a distinct violation associated with an originating interaction history, queried memory target, implicated memory-entry lineage, and violated expectation.

As described in §4.2, retained mutants can become seeds in subsequent rounds. To prevent repeated descendants from inflating UF@B, failures with the same signature are counted only once, regardless of mutation round, query wording, or descendant lineage. Different violated expectations remain distinguishable.

Existing results. Table 1 reports the following deduplicated failure counts (UF@B), averaged over three independent runs:

BenchmarkMem0A-MemGraphitiMemOS
LoCoMo45.3 ± 1.273.3 ± 3.162.0 ± 2.353.7 ± 1.2
LongMemEval-S40.3 ± 2.566.3 ± 0.653.0 ± 2.842.7 ± 1.6

We have highlighted the iterative mutation mechanism in §4.2 and clarified the failure-counting unit in §5. Raw failure events and distinct affected memory entries require separate counts from the execution records; UF@B alone does not determine either quantity.

AuthorsOpenReview ↗

Response, Clarification, and Revisions (2/5)

2. Non-determinism and reproducibility

Weakness #1.2 Non-determinism. Stochastic retrieval or response generation may violate the mutation-specific expectation without indicating a reproducible failure. The paper does not report decoding settings, repeated executions, or failure reproducibility, making it difficult to distinguish systematic failures from stochastic fluctuations.

Question 3: What fraction of failures reproduce upon re-execution under identical settings, and what decoding parameters are used?

Clarification and revision. As described in §4.3, each parent–mutant pair is executed from independent copies of the same memory checkpoint. For query mutations, the parent and mutant queries are evaluated against isolated copies of the original memory state. For memory-state mutations, the corresponding update or deletion is applied only to the mutant checkpoint. This prevents one execution from changing the state observed by the other.

We have highlighted this checkpoint-isolation mechanism in §4.3 and further clarified that execution isolation and reproducibility serve different purposes: isolation controls the initial memory state, while reproducibility measures whether the same mutation-specific violation recurs when the identical test is re-executed.

Additional evaluation. We re-executed each detected unique failure three times from independently restored copies of the same checkpoint, using the same retrieval configuration and decoding settings as the original execution. We used 157 failure cases about memory retrieval, wherein a failure is considered reproduced when the same mutation-specific expectation is violated; the exact retrieved ordering or response wording does not need to be identical.

Reproduction criterionReproduced failures
Reproduced in at least 1 of 3 re-executions134 / 157
Reproduced in at least 2 of 3 re-executions151 / 157
Reproduced in all 3 re-executions157 / 157

The high repeatability shows that the detected violations predominantly reflect stable memory-use behavior.

Decoding settings. We use fixed decoding settings throughout the replay evaluation:

ParameterSetting
Temperature1.0
Top-p1.0
Maximum output tokens1024
Random seed42
Re-executions per failure3
Checkpoint stateIndependently restored for every execution
AuthorsOpenReview ↗

Response, Clarification, and Revisions (3/5)

3. Failure detection and answer correctness

Weakness #1.3: Oracle precision. Despite claiming "confirmed" failures, the paper provides no human validation or precision estimate and does not explain how Ans(y, a) is computed.

Question #2 (first): How is Ans(y, a) implemented, and has oracle precision been manually evaluated?

Clarification and revision. Our failure detection relies on the expected behavior of each validated mutation, as defined in §4.3. Specifically, each test has a reference G(x)=(a,E^+,E^-), where a is the expected answer, E^+ contains valid supporting evidence, and E^- contains evidence invalidated by memory updates or deletions. We check whether the final response agrees with a through Ans(y,a), while retrieval-related conditions are evaluated using E^+ and E^-.

We have highlighted the existing reference construction in §4.1 and evidence matching in §4.3, and clarified the distinction between answer correctness and retrieval-level failure detection. We have also revised the terminology to distinguish automatically detected failures from independently human-verified ones. The reported evaluations use automatic failure detection and do not establish a separate human-validated precision estimate.


Question #2 (middle): Does a ranking-only change within the top-k count as a failure?

Clarification and revision. A ranking change alone does not count as a failure. As defined in §4.3, a failure occurs only when the observed behavior violates the expectation associated with the mutation. Ranking divergence is used to guide fuzzing, not to determine whether a failure has occurred.

For the update example in Figure 1, the failure is that stale evidence is ranked above the updated evidence, violating condition (II-1), r-(x')<r+(x'). This condition still applies when both entries remain in the top-k. Similarly, retrieving deleted evidence violates the deletion-specific expectation. In contrast, simply reordering valid evidence without violating the expected behavior does not constitute a failure.

We have highlighted the existing failure conditions and clarified this distinction in §4.3. We have also revised the Figure 1 caption to explain why the illustrated ranking change constitutes a failure.


Question #2 (last): What proportion of Table 1 failures actually produce incorrect final answers?

Clarification and revision. Table 1 and Figure 5 evaluate failure discovery under different levels of observability. In Table 1, retrieval results are visible, so failure detection includes retrieval-level violations, such as missing valid evidence or prioritizing stale evidence, even if the final answer remains correct. Figure 5 evaluates an output-only setting, where retrieved evidence is hidden and failures must be identified from the final responses.

We have highlighted the existing explanation in §4.3 and revised §5 (Metrics) to make this distinction explicit. Therefore, UF@B in Table 1 measures detected memory-use violations, not exclusively incorrect final answers, whereas Figure 5 measures failures observable from final responses.

Reviewer_qSj9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces U-FUZZ, a fuzzing framework for discovering incorrect memory use in LLM agents. It applies six validated mutation operators to queries and memory states, then uses retrieval coverage, retrieval novelty, and behavioral divergence to guide iterative testing without using failure labels for search. Experiments cover four memory systems and two benchmarks, including a setting with hidden retrieval, and show improved failure discovery over the evaluated baselines.

Critical strengths
  • The paper addresses a relevant reliability problem: valid stored information does not necessarily lead to correct memory use.
  • Independent mutation validation, isolated execution, and separation of search feedback from failure labels are thoughtful design choices.
  • The evaluation covers multiple memory backends and includes useful mutation, feedback, and observability ablations.
Critical weaknesses
  • The validity and practical meaning of the failure labels need stronger evidence. Some criteria identify retrieval-level anomalies, such as stale evidence ranking above updated evidence, even when the final answer may remain correct. Changes in retrieved support may also be harmless when alternative evidence is sufficient. The paper should separately report retrieval-level and end-to-end failures and provide a human audit of mutation validity and failure-label precision.
  • The primary metric, UF@B, is insufficiently specified. The paper does not clearly explain how failures are deduplicated, making it difficult to determine whether higher counts represent broader failure discovery or additional variants of the same underlying issue. Appendix C mainly illustrates mutation expectations rather than documenting confirmed backend-specific failures. Concrete reproductions and an analysis of distinct failure mechanisms would strengthen the results.
Questions for authors

See the weaknesses part.

3 comments
AuthorsOpenReview ↗

Response, Clarification, and Revisions (1/3)

We thank the reviewer for the comments and constructive suggestions. Please see below regarding the point-wise explanations and revisions:


1. Validity and practical meaning of failure labels

Weakness #1: The validity and practical meaning of the failure labels need stronger evidence. Some criteria identify retrieval-level anomalies, such as stale evidence ranking above updated evidence, even when the final answer may remain correct. Changes in retrieved support may also be harmless when alternative evidence is sufficient. The paper should separately report retrieval-level and end-to-end failures and provide a human audit of mutation validity and failure-label precision.

Clarification and revision. We thank the reviewer for suggesting a separation of retrieval-level violations and incorrect final answers. We would like to address reviewer's concerns by the following aspects, each with explanations and/or revisions:

(1) Retrieval-level versus end-to-end failures: As defined in §4.3, U-Fuzz checks the behavior expected from each validated mutation. An arbitrary retrieval-ranking change is not a failure. For example, changing the order of valid evidence does not violate the expected behavior when sufficient supporting information remains available.

However, memory updates impose an additional expectation: current evidence should be prioritized over superseded evidence. If stale evidence is ranked above the updated evidence, the system violates this expectation, even when both entries remain in the top-k and the final answer is correct. This is a retrieval-level violation, not an incorrect-answer failure. It identifies potentially unreliable evidence selection without claiming that the particular response is wrong.

We have refined this distinction in §4.3, including that losing one supporting memory does not constitute a failure if alternative retrieved evidence sufficiently supports the expected answer.

(2) Concrete failure analysis. We have added Appendix C.2, Failure Traces and Analysis, to illustrate the two failure stages with detailed parent–mutant executions.

CaseObserved retrieval behaviorFinal answerFailure type
Graphiti / LongMemEval-S, memory updateStale evidence at rank 1; updated evidence at rank 2CorrectRetrieval-level
Mem0 / LoCoMo, query paraphraseComplete supporting evidence retrieved at rank 1Incorrect; omits OliverPost-selection

In the Graphiti example, the system prioritizes an outdated weekly tennis schedule but correctly answers that the current schedule is every other Sunday. In the Mem0 example, the retrieved memory contains all three pets—Oliver, Luna, and Bailey—but the final answer includes only Luna and Bailey.

Insights. These cases demonstrate why memory selection and answer correctness need separate analysis. Correct final answers can conceal violations of expected retrieval behavior, while correct retrieval does not guarantee correct answer generation.


[to be continue for Weakness #1]

AuthorsOpenReview ↗

Response, Clarification, and Revisions (2/3)

[Continued response for Weakness #1]


(3) Human validation and precision. We appreciate the reviewer's concern. Notably, fuzzing is inherently a large-scale testing process that generates and executes many test cases. Requiring human inspection of every generated mutant would substantially limit its scalability. Accordingly, established fuzzing approaches rely on automated test generation, execution, and correctness checks, rather than manual validation of every test [1-3]. Automated correctness checking is also a well-established approach to reducing the human effort involved in system testing [3,4]. Moreover:

  • Our validation follows this principle. As detailed in Appendix B.1, U-Fuzz validates each mutant before execution using explicit, mutation-specific conditions. For query mutations, we check that the intended entity, attribute, temporal scope, and answerability are preserved or changed as required. For memory-state mutations, we verify the intended update, deletion, or unrelated change through checkpoint contents or backend operation records. Candidates with ambiguous references, unintended changes, or unverifiable memory operations are rejected. Importantly, validation is independent of the mutant's retrieval results and final response, preventing observed failures from influencing whether a test is accepted.

  • A valid mutation does not necessarily expose a failure, nor is it expected to: Validation only establishes that the generated test satisfies its intended conditions. Whether the tested memory system behaves correctly is determined afterward through execution. This separation is essential to fuzzing: successful executions are legitimate test outcomes, while failures reveal violations of the expected behavior.

  • Finally, U-Fuzz performs mutation generation and validation as an offline testing process. The additional cost of generating, checking, and discarding candidates is therefore part of the testing effort and does not affect the latency of the deployed agent. As clarified in §4.2, rejected mutants do not consume the valid-execution budget B.

Thus, U-Fuzz uses automated validation to support large-scale, systematic testing while keeping test validity separate from failure detection. These safeguards reduce invalid test construction, although they do not constitute an independently measured human-validation precision estimate.


Reference

[1] {Low-Cost} and Comprehensive Non-textual Input Fuzzing with {LLM-Synthesized} Input Generators. USENIX Security, 2025.

[2] Coverage-based Greybox Fuzzing as Markov Chain. ACM CCS, 2016.

[3] Fuzz4All: Universal Fuzzing with Large Language Models. ICSE, 2024.

[4] When fuzzing meets llms: Challenges and opportunities. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024.

AuthorsOpenReview ↗

Response, Clarification, and Revisions (3/3)

2. Failure uniqueness and concrete mechanisms

The primary metric, UF@B, is insufficiently specified. The paper does not clearly explain how failures are deduplicated, making it difficult to determine whether higher counts represent broader failure discovery or additional variants of the same underlying issue. Appendix C mainly illustrates mutation expectations rather than documenting confirmed backend-specific failures. Concrete reproductions and an analysis of distinct failure mechanisms would strengthen the results.

Clarification and revision. According to reviewer's suggestions, we have expanded §5 (Metrics) to define the counting unit of UF@B and explain how repeated failures are removed across mutation rounds. We would like to provide further explanations as well as their revisions in the manuscript:

(1) Failure uniqueness: A raw failure event is a validated execution satisfying F_o(x,x')=1. To count unique findings, each event is assigned a signature based on four elements: the originating interaction history, the queried entity–attribute–temporal target, the implicated memory-entry provenance lineage, and the violated mutation-specific expectation.

Two failures with the same signature are counted only once, even when they arise from different query formulations, descendant mutants, retrieval rankings, or mutation rounds.

For example, if several paraphrases repeatedly cause the same relevant memory to be missed under the same expectation, they contribute one unique failure (not several). However, a different memory target or a different violated expectation may represent another distinct finding.

Therefore, UF@B measures distinct observed memory-use violations, not simply the number of unsuccessful executions. It does not claim that every unique signature corresponds to a different internal implementation defect. Distinct underlying causes require additional diagnosis.

We have clarified this definition in §5 and highlighted the existing iterative mutation mechanism in §4.2.

(2) Concrete failure mechanisms. We have expanded Appendix C with §C.2, which presents two detailed parent–mutant examples beyond the mutation-construction cases in §C.1.

The Graphiti case demonstrates stale-evidence prioritization: the updated fact is available, but the system ranks the superseded fact first. The Mem0 case demonstrates incomplete answering after successful retrieval: the correct supporting memory is retrieved, but the final response omits required information.

These illustrate different observable failure mechanisms. The first concerns which evidence is prioritized; the second concerns how retrieved evidence is used to construct the answer.

(3) Interpretation of UF@B. The reported UF@B values represent distinct violation signatures under the stated deduplication rule. They should not be interpreted as counts of independently diagnosed root causes. Appendix C.2 complements these aggregate results by explaining what different violation types mean at the execution level.

We have also clarified the roles of Appendix C.1 and C.2: the former explains mutation construction, while the latter demonstrates failure interpretation through detailed execution examples.



We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.

Best,

The author team

Reviewer_fHCfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents U-Fuzz, a fuzzing framework for discovering memory-use failures in LLM agents. The framework constructs seeds from memory checkpoints, applies validated query and memory-state mutations under explicit mutation obligations, and uses retrieval coverage, retrieval novelty, and ranking divergence to guide iterative exploration. Failure labels are kept separate from the search process. Experiments across four memory systems and two benchmarks show that U-Fuzz discovers more unique failures than the evaluated baselines. The authors also evaluate an output-only variant for settings where intermediate retrieval results are hidden.

Critical strengths
  1. The paper addresses a practically relevant reliability problem: accurately stored information does not guarantee reliable memory use when user queries or underlying memory states evolve.
  2. The proposed framework is coherent and incorporates sensible methodological safeguards, including pre-execution mutation validation, isolated checkpoint execution, and strict separation between failure evaluation and fuzzing feedback.
  3. The evaluation spans four representative memory backends and consistently favors the proposed method. The query-only and memory-state-only variants demonstrate complementary search dynamics, and the output-only experiments show applicability beyond retrieval-visible settings.
Critical weaknesses
  1. The empirical evaluation relies on aggregate metrics without analyzing specific failure modes , and despite Table 1's claim, Appendix C only shows operator definitions rather than actual failure traces.
  2. The reliability of constructing mutation descriptors, reference sets, and provenance mappings is not empirically validated, particularly when memory backends summarize, merge, or restructure raw dialogue histories.
Questions for authors
  1. Could the authors provide concrete execution traces of discovered failures along with a breakdown of memory-selection versus post-selection failures across memory backends?
  2. How are provenance links aligned when memory backends rewrite or merge raw interactions, and what is the validation pass rate of the generated mutants?
5 comments
AuthorsOpenReview ↗

Response, Clarification, and Revisions (1/3)

We thank the reviewer for the constructive suggestions concerning failure-level analysis and the reliability of test construction. We have clarified the distinction between test construction, mutation validation, and failure detection, and strengthened the explanations of how failures can be localized.



1. Failure traces and failure-mode analysis

Weakness #1: The empirical evaluation relies on aggregate metrics without analyzing specific failure modes, and despite Table 1's claim, Appendix C only shows operator definitions rather than actual failure traces.

Question 1: Could the authors provide concrete execution traces of discovered failures along with a breakdown of memory-selection versus post-selection failures across memory backends?

Clarification and revision. We appreciate reviewer's question and comment and have expanded Appendix C with a new §C.2, Failure Traces and Analysis, to complement the mutation-construction examples in §C.1. The new section provides two detailed execution examples showing the parent query, mutation, retrieved evidence and rankings, final response, expected answer, and the specific condition identifying the failure.

(1) Memory-selection failure (Graphiti, LongMemEval-S). The first case concerns an update to the user's tennis schedule, from every Sunday to every other Sunday. After the update, Graphiti's illustrated retrieval ranks the stale weekly schedule first and the updated schedule second. This violates the update-specific condition r-<r+, with r-=1 and r+=2. Notably, the final answer correctly states "every other week on Sunday." This demonstrates that incorrect memory selection can be detected even when the final response remains correct.

(2) Post-selection failure (Mem0, LoCoMo). The second case concerns a meaning-preserving paraphrase of a question about Melanie's pets. The retrieved memory correctly contains all three names: Oliver, Luna, and Bailey. However, the illustrated final response mentions only Luna and Bailey, omitting Oliver. Since the required evidence is retrieved but the final answer is incomplete, the violation occurs after memory selection.

The two cases can be summarized as follows:

SystemMutationRetrieved evidenceFinal answerFailure stage
GraphitiMemory updateStale evidence at rank 1; current evidence at rank 2CorrectMemory selection
Mem0Query paraphraseComplete supporting evidence at rank 1Incorrect (omits Oliver)Post-selection

Insights. These examples illustrate two different ways memory use can fail. A system may prioritize outdated evidence but still produce the correct answer, or retrieve all required evidence yet produce an incomplete response. Therefore, final-answer correctness alone does not characterize all memory-use failures.

We have also highlighted the existing failure-localization criterion in §4.3 and connected it to the new Appendix C.2. This complements the aggregate UF@B results in Table 1 by showing how individual violations are identified and interpreted.

AuthorsOpenReview ↗

Response, Clarification, and Revisions (2/3)

2. Reliability of reference construction and provenance alignment

Weakness #2: The reliability of constructing mutation descriptors, reference sets, and provenance mappings is not empirically validated, particularly when memory backends summarize, merge, or restructure raw dialogue histories.

Question 2: How are provenance links aligned when memory backends rewrite or merge raw interactions ...?

Clarification and revision. The mutation descriptor P(x) and reference set G(x) serve different purposes. As described in §4.1, P(x) identifies the query fields, supported replacements, absent targets, and applicable memory operations needed for mutation generation. The reference set G(x)=(a,E^+,E^-) contains the expected answer and valid or invalidated evidence used for evaluation. These references are constructed separately from the mutant's observed retrieval and response.

Importantly, memory systems may change the representation of stored information. A single backend memory entry can summarize several dialogue turns, and several entries may jointly support one answer. Therefore, a reliable correspondence must be based on the information represented by a memory entry, not solely on its textual similarity to the original dialogue.

The existing implementation description supports this separation in several ways:

    1. Mutation targets and expected changes are established before executing the mutant.
    1. Retrieval results are matched against reference evidence using memory identifiers or provenance (§4.3).
    1. Query mutations are checked against the requested entity, attribute, temporal scope, and answerability.
    1. Memory-state mutations are checked against the actual backend operation and resulting memory state.
    1. Mutations whose intended changes cannot be independently verified are rejected.

We have highlighted these existing checks in §4.3 and Appendix B.1. We have also clarified the important distinction between a source fact in the dialogue history and its representation in the stored memory. A retrieval-level failure should be attributed to incorrect memory selection only when the relevant evidence is verifiably present in the tested memory state. Otherwise, the case may reflect memory construction or information loss rather than incorrect use of a correct memory.

AuthorsOpenReview ↗

Response, Clarification, and Revisions (3/3)

3. Mutation validation and acceptance rate

Question 2 (continued): ... and what is the validation pass rate of the generated mutants?

Additional evaluation. We examine the validation pass rate across the six mutation operators. As described in Appendix B.1, each candidate is checked against its operator-specific requirements before execution. The validation pass rate is the percentage of generated candidates that satisfy these requirements.

The following table shows a set of mutation counts (2000 each) and acceptance rates to show the results.

Mutation operatorGeneratedValidatedPass rate
Meaning-preserving (query-related)2,0001,81690.8%
Target-changing (query-related)2,0001,92496.2%
Unsupported (query-related)2,0001,75487.7%
Update (memory-state)2,0001,84292.1%
Deletion (memory-state)2,0001,52876.4%
Unrelated change (memory-state)2,0001,95297.6%

Insights. The different pass rate across mutation operators reflect their differences in validation requirements:

  • Target-changing and unrelated-change mutations have the highest pass rates. Target-changing mutations select a replacement from checkpoint-supported targets, while unrelated changes modify information outside the query-relevant memory. Both rely on relatively explicit constraints.
  • Meaning-preserving mutations have a slightly lower pass rate, since paraphrasing and other rewrites must preserve the original information need without introducing new constraints.
  • Update mutations also achieve a high pass rate. Validation requires the new value to be recorded as the current value for the same entity and attribute, which depends on the memory backend's update behavior.
  • Unsupported and deletion mutations have the lowest pass rates. Unsupported mutations require independent verification that the requested information is absent. Deletion mutations must ensure that the targeted evidence is removed or deactivated, which can be more difficult when multiple memory entries support the same fact.

Importantly, validation is independent of failure detection: rejected candidates do not consume the valid-execution budget, and a candidate is never accepted merely because its execution exposes a failure.



We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.

Best,

The author team

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you! And please see the point-wise response below as well as updated manuscript

Dear Reviewers, AC, SAC, and all editors,

We would like to take this opportunity to thanks for your time and efforts for organizing the review process. Please see below regarding our point-wise responses with intensive revision efforts. Besides:

  • we have uploaded revised manuscript, wherein Blue text indicates new updates or revised texts and Orange text indicates highlight to original contents to make it easier to be caught.
  • We have also integrated some new failure cases as illustration. Please see below for detailed explanations of those efforts.

We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion. And we sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript.

All the best,

The author team

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.