- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies memory-use failures in persistent-memory LLM agents, where correct stored information is used incorrectly following query changes or memory-state updates. It proposes U-Fuzz, which constructs seeds from memory checkpoints, applies six validated query- and memory-state mutation operators, schedules mutations through operator–target obligations, and uses retrieval coverage, novelty, and ranking divergence to guide exploration without relying on failure labels. Experiments across four memory systems (Mem0, A-Mem, Graphiti, MemOS) and two benchmarks (LoCoMo, LongMemEval-S) show more unique failures than four baselines under equal execution budgets, supported by ablations and an output-only evaluation.
Critical strengths
-
Careful testing methodology. U-Fuzz changes only one input per parent–mutant pair, validates mutants independently of their outputs, restricts unsupported queries to memory-exclusive targets, and separates failure labels from search feedback. These choices reduce confounding factors and oracle leakage, strengthening the testing methodology.
-
Broad and consistent evaluation. The experiments cover four memory backends, two benchmarks, four baselines, multiple ablations, and a hidden-retrieval setting. U-Fuzz consistently discovers more failures (e.g., 45 vs. 28 on Mem0/LoCoMo and 73 vs. 36 on A-Mem), while the ablations suggest that coverage supports exploration and novelty/divergence help discover distinct behaviors.
Critical weaknesses
-
The failure oracle and uniqueness metric are insufficiently specified, raising concerns about the validity of UF@B.
- Uniqueness. The paper does not define what constitutes a "unique" failure. Since retained mutants can be mutated again (§4.3), multiple descendants may repeatedly expose the same underlying issue, potentially inflating UF@B. The authors should clarify the deduplication criteria and report raw versus unique failure counts.
- Non-determinism. Stochastic retrieval or response generation may violate the mutation-specific expectation without indicating a reproducible failure. The paper does not report decoding settings, repeated executions, or failure reproducibility, making it difficult to distinguish systematic failures from stochastic fluctuations.
- Oracle precision. Despite claiming "confirmed" failures, the paper provides no human validation or precision estimate and does not explain how Ans(y, a) is computed. Moreover, Figure 1 labels a ranking change as a failure even though correct evidence remains in the top-k, while §4.3 does not clearly establish whether ranking changes alone violate the oracle. The authors should clarify this criterion and distinguish retrieval-only failures from incorrect final answers, particularly when interpreting Table 1 versus Figure 5.
-
Missing experimental details and unclear baseline fairness. The main-experiment LLM backend, mutation/validation models, retrieval depth k, attempt limit A, hidden-retrieval encoder/threshold, checkpoint counts, and execution cost are not reported. More importantly, it is unclear whether Random and Coverage-Guided share U-Fuzz's mutation operators and obligation scheduler. Without controlling these factors, the reported gains may conflate mutation design, scheduling, and feedback guidance. A controlled ablation isolating these contributions is needed.
Questions for authors
- How is a "unique" failure defined and deduplicated across mutation rounds? Can the authors report raw versus deduplicated counts and the number of distinct underlying memory entries affected?
- How is Ans(y, a) implemented, and has oracle precision been manually evaluated? Does a ranking-only change within the top-k count as a failure? What proportion of Table 1 failures actually produce incorrect final answers?
- What fraction of failures reproduce upon re-execution under identical settings, and what decoding parameters are used?
- Do all baselines share the same operators and obligation scheduler? Can the authors provide controlled ablations and the missing experimental configurations?
5 comments
Response, Clarification, and Revisions (2/5)
2. Non-determinism and reproducibility
Weakness #1.2 Non-determinism. Stochastic retrieval or response generation may violate the mutation-specific expectation without indicating a reproducible failure. The paper does not report decoding settings, repeated executions, or failure reproducibility, making it difficult to distinguish systematic failures from stochastic fluctuations.
Question 3: What fraction of failures reproduce upon re-execution under identical settings, and what decoding parameters are used?
Clarification and revision. As described in §4.3, each parent–mutant pair is executed from independent copies of the same memory checkpoint. For query mutations, the parent and mutant queries are evaluated against isolated copies of the original memory state. For memory-state mutations, the corresponding update or deletion is applied only to the mutant checkpoint. This prevents one execution from changing the state observed by the other.
We have highlighted this checkpoint-isolation mechanism in §4.3 and further clarified that execution isolation and reproducibility serve different purposes: isolation controls the initial memory state, while reproducibility measures whether the same mutation-specific violation recurs when the identical test is re-executed.
Additional evaluation. We re-executed each detected unique failure three times from independently restored copies of the same checkpoint, using the same retrieval configuration and decoding settings as the original execution. We used 157 failure cases about memory retrieval, wherein a failure is considered reproduced when the same mutation-specific expectation is violated; the exact retrieved ordering or response wording does not need to be identical.
| Reproduction criterion | Reproduced failures |
|---|---|
| Reproduced in at least 1 of 3 re-executions | 134 / 157 |
| Reproduced in at least 2 of 3 re-executions | 151 / 157 |
| Reproduced in all 3 re-executions | 157 / 157 |
The high repeatability shows that the detected violations predominantly reflect stable memory-use behavior.
Decoding settings. We use fixed decoding settings throughout the replay evaluation:
| Parameter | Setting |
|---|---|
| Temperature | 1.0 |
| Top-p | 1.0 |
| Maximum output tokens | 1024 |
| Random seed | 42 |
| Re-executions per failure | 3 |
| Checkpoint state | Independently restored for every execution |
Response, Clarification, and Revisions (3/5)
3. Failure detection and answer correctness
Weakness #1.3: Oracle precision. Despite claiming "confirmed" failures, the paper provides no human validation or precision estimate and does not explain how Ans(y, a) is computed.
Question #2 (first): How is Ans(y, a) implemented, and has oracle precision been manually evaluated?
Clarification and revision. Our failure detection relies on the expected behavior of each validated mutation, as defined in §4.3. Specifically, each test has a reference G(x)=(a,E^+,E^-), where a is the expected answer, E^+ contains valid supporting evidence, and E^- contains evidence invalidated by memory updates or deletions. We check whether the final response agrees with a through Ans(y,a), while retrieval-related conditions are evaluated using E^+ and E^-.
We have highlighted the existing reference construction in §4.1 and evidence matching in §4.3, and clarified the distinction between answer correctness and retrieval-level failure detection. We have also revised the terminology to distinguish automatically detected failures from independently human-verified ones. The reported evaluations use automatic failure detection and do not establish a separate human-validated precision estimate.
Question #2 (middle): Does a ranking-only change within the top-k count as a failure?
Clarification and revision. A ranking change alone does not count as a failure. As defined in §4.3, a failure occurs only when the observed behavior violates the expectation associated with the mutation. Ranking divergence is used to guide fuzzing, not to determine whether a failure has occurred.
For the update example in Figure 1, the failure is that stale evidence is ranked above the updated evidence, violating condition (II-1), r-(x')<r+(x'). This condition still applies when both entries remain in the top-k. Similarly, retrieving deleted evidence violates the deletion-specific expectation. In contrast, simply reordering valid evidence without violating the expected behavior does not constitute a failure.
We have highlighted the existing failure conditions and clarified this distinction in §4.3. We have also revised the Figure 1 caption to explain why the illustrated ranking change constitutes a failure.
Question #2 (last): What proportion of Table 1 failures actually produce incorrect final answers?
Clarification and revision. Table 1 and Figure 5 evaluate failure discovery under different levels of observability. In Table 1, retrieval results are visible, so failure detection includes retrieval-level violations, such as missing valid evidence or prioritizing stale evidence, even if the final answer remains correct. Figure 5 evaluates an output-only setting, where retrieved evidence is hidden and failures must be identified from the final responses.
We have highlighted the existing explanation in §4.3 and revised §5 (Metrics) to make this distinction explicit. Therefore, UF@B in Table 1 measures detected memory-use violations, not exclusively incorrect final answers, whereas Figure 5 measures failures observable from final responses.
Response, Clarification, and Revisions (1/5)
We thank the reviewer for the detailed assessment and constructive suggestions. We have revised the manuscript to clarify the failure oracle, failure uniqueness, experimental settings, and scope of the baseline comparisons. Please see detailed responses below:
1. Failure uniqueness and deduplication
Response and revision. We would like to first explain the distinction between a failure event and a unique failure in §5 (Metrics). A failure event is a validated execution satisfying F_o(x,x')=1. For deduplication, we define a failure key using the originating interaction history, the queried memory target, the implicated memory-entry lineage, and the violated mutation-specific expectation.
Failures with the same key count only once, regardless of the mutation round, descendant mutant, or query wording that exposes them. Different violated expectations remain distinguishable. This prevents repeated descendants exposing the same memory-use violation from being counted as independent discoveries.
We have also highlighted the iterative mutation mechanism in §4.2 and clarified the distinction among raw failure events, deduplicated failures, and distinct affected memory entries in §5.
Clarification and revision. We would like to echo back to the definition of unique failures in §5 (Metrics). A raw failure is a validated execution that violates its mutation-specific expectation, i.e., Fo(x,x')=1. A unique failure represents a distinct violation associated with an originating interaction history, queried memory target, implicated memory-entry lineage, and violated expectation.
As described in §4.2, retained mutants can become seeds in subsequent rounds. To prevent repeated descendants from inflating UF@B, failures with the same signature are counted only once, regardless of mutation round, query wording, or descendant lineage. Different violated expectations remain distinguishable.
Existing results. Table 1 reports the following deduplicated failure counts (UF@B), averaged over three independent runs:
We have highlighted the iterative mutation mechanism in §4.2 and clarified the failure-counting unit in §5. Raw failure events and distinct affected memory entries require separate counts from the execution records; UF@B alone does not determine either quantity.