- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces WTF, a framework for diagnosing failures in memory-based self-evolving LLM agents using execution traces and memory histories. It analyzes 9,544 traces across four methods, five benchmarks, and three models, producing a taxonomy of 25 failure classes. Selected replay experiments demonstrate that removing harmful memory can improve task success, while the broader analysis associates whole-store memory injection with more memory-related failures.
Critical strengths
-
Linking task traces to historical memory snapshots is valuable. The example in Appendix E.7 demonstrates how ignoring update timing can produce an incorrect diagnosis.
-
The controlled replays provide useful evidence for specific harmful-memory mechanisms. Separating behavioral recovery from full task success in Table 2 also makes the intervention results more informative.
Critical weaknesses
-
Diagnostic accuracy is insufficiently validated. Human calibration is useful, but no quantitative pre-reconciliation agreement or held-out diagnosis accuracy is reported. Appendix C only reclassifies existing diagnoses, leaving errors in the underlying explanations untested.
-
The delivery-mode comparison is confounded. Section 6 compares different agent systems rather than isolating memory delivery. Update failures further complicate the comparison: eight AWM runs on DeepSeek never stored a workflow, potentially reducing their exposure to harmful memory.
-
The memory-growth conclusion is not well isolated. Appendix F.1 uses fixed task orders, confounding stream position with task identity. Comparing trends across methods helps, but does not rule out method-specific difficulties on later tasks.
-
The replay evidence has limited coverage. The 14 memory-free comparisons support selected mechanisms, but cases were selected partly for reproducible failures. They cannot establish how reliably the diagnostic framework identifies memory-induced failures across the corpus.
Questions for authors
-
From the existing human annotations, can you report agreement before reconciliation, particularly on memory-induced versus base-model failures? Were any diagnoses independently evaluated on traces excluded from calibration and taxonomy construction?
-
Using the existing logs, do the delivery-mode gap and stream-position trends persist on matched task/model subsets after excluding runs with failed memory updates or empty memory? This would help assess how much of the reported difference is attributable to memory delivery.
There is no rebuttal submitted so far from the authors. I will maintain my score as "weak rejection".