- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MemTrace studies how to find the operations responsible for wrong answers in LLM memory systems. The authors instrument memory pipelines to record how information is extracted, updated, retrieved, and used, and build MemTraceBench with 160 human-annotated failures from four memory systems. They propose both graph exploration and operation-log search for automatic diagnosis, then use the diagnoses to guide prompt optimization.
Critical strengths
- The paper is well written and easy to follow. The motivation makes sense: a wrong answer can come from a memory update many sessions earlier, and inspecting the final response alone will miss that. Figure 2 gives a helpful picture of how the method follows the information through the system.
- I see value in the tracing toolkit and the annotated benchmark. Recording the evolution of individual memories gives developers something more useful than another end-task accuracy score. The examples of information being dropped or timestamps being reassigned also make the debugging problem concrete.
- The prompt optimization experiment provides a useful check that the diagnoses can lead to better behavior. I appreciate that the authors split optimization and testing by user trajectory. The improvement from 59.08% to 66.70% is encouraging.
Critical weaknesses
-
I am not fully convinced by the baseline comparison or by how much of the gain comes from the graph itself. Appendix G says All-at-Once keeps only the latest 600,000 tokens, so it loses exactly the earlier operations this paper aims to diagnose. MemTrace-OBS is a useful comparison, but it changes the representation and search interface together. With GPT-5.4, it actually has higher overall operation identification accuracy than MemTrace, 46.25% versus 38.13%, at much lower token cost. A comparison with matched starting information and budgets would help isolate the value of following dependency edges. Appendix B explains several adaptation difficulties, but a replay-free graph reasoner such as CHIEF, already cited in the paper, is worth considering. The chunked baseline mentioned in Appendix G should also have its accuracy and cost reported, even if it performs poorly.
-
The operation labels need a little more validation. Figure 7b reports annotator disagreement ranging from 3% to 46%, and Appendix D.6 says the interface shows a preliminary automatic diagnosis to annotators. What algorithm produces these suggestions, and how is it related to the evaluated methods? The independent annotation and final adjudication are useful, but I would like to know whether disagreements concern the error category or the exact operation. Appendix D.2 also merges several retrieval steps into one operation when their responsibility is ambiguous, which affects what exact-operation accuracy means. A small blind audit of the final labels, with examples showing how the decisive operation was chosen, would make the benchmark more convincing.
-
The evaluation leaves some uncertainty about generalization and the interpretation of the failure patterns. Both attribution backbones are from OpenAI, and all failures are collected from memory systems using GPT-4.1 mini. One attribution model from another family, such as Qwen or DeepSeek, would make the comparison more informative. There is also a dataset-mix issue: Figure 8 shows that the source distributions differ across systems. This makes it hard to tell how much of the difference in Figure 3b comes from the memory architecture and how much comes from the questions being tested. A per-dataset breakdown would help support the stronger comparisons in Section 5.3.
-
A few presentation details are easy to fix. Table captions should be above the tables. I would also bring the benchmark composition into the main text: Appendix D.2 already gives the collected failure counts for each system, and Figure 8 gives the source proportions, but a single table with exact system-by-dataset counts for both the collected pool and the final benchmark would be much easier to read.
Questions for authors
- How do graph exploration and log search compare when starting information and inference budgets are matched? Could you report the chunked baseline’s results and discuss an adaptation of a replay-free attribution method such as CHIEF?
- What produces the preliminary diagnoses shown during annotation? How often do annotators disagree on error types versus operation identifiers, and how are ambiguous or merged operations handled in the final labels?
- Do the main method rankings hold with an attribution backbone from another model family? Can you provide results by source dataset, along with the exact benchmark counts and repeated-run details?
12 comments
Response to W1 and Q1 (Continued)
Main results. As shown in Table 1, adding directly related operations improves both ETA and OIA for all three backbones. However, the magnitude and cost of this benefit remain model-dependent. For GPT-4.1 mini and Qwen3.6-Flash-0416, the reported token consumption decreases, although wall-clock time does not decrease consistently. DeepSeek-V4-Flash-0731 exhibits a pronounced runtime tail: the mean runtime increases from 2.74 to 7.99 minutes, whereas the median increases more moderately from 2.07 to 2.98 minutes. Its P95 increases from 7.24 to 37.08 minutes. Qualitative inspection shows that, in these slow cases, DeepSeek-V4-Flash-0731 repeatedly verifies whether the selected operation is the earliest decisive error. Thus, the large mean increase is driven by a relatively small number of unusually long reasoning trajectories.
Error-stage analysis. To determine when dependency information is most useful, we further stratify the same Mem0 cases by their gold error type. We classify Extraction, Update, and Deletion errors as memory-construction errors, and Retrieval and Response errors as non-construction errors. As shown in Table 2, adding directly related operations improve OIA by 8.00 percentage points for construction errors and 2.22 points for non-construction errors. After further excluding DeepSeek-V4-Flash-0731 to control for the anomalously long reasoning trajectories observed in some of its cases, the improvement remains 8.00 points for construction errors but drops to 0.00 points for non-construction errors. These results indicate when execution-graph dependencies are most useful. In memory-construction failures, the faulty operation may occur long before the final question, while its effect becomes visible only in later stages. Showing which earlier operations produce an operation’s inputs and which later operations use its outputs helps the model follow information across these distant steps and locate the original fault. In contrast, operations responsible for retrieval and response failures often contain keywords that overlap with the question or golden answer. They are therefore likely to be surfaced early by text-based search and can usually be diagnosed from their local operation content. Consequently, dependency information provides a smaller additional benefit for their OIA.
Changes to the paper: We have added the controlled dependency ablation to Section 5.3 (Table 3b) and a detailed treatment in Appendix H, including the formal definition of directly related operations and the error-stage results in Table 7.
Clarification of Graph-Based Attribution and Additional Baseline Evaluation
First, we would like to clarify that both MemTrace and MemTrace-OBS use starting-point initialization. However, unlike MemTrace, MemTrace-OBS is not required to first inspect operations associated with these starting points. Instead, the starting points are provided as hints in the prompt, indicating that the corresponding messages may contain source evidence relevant to the question.
Additional evaluation of the chunked baseline. We would also like to clarify the performance of the chunked baseline. In our preliminary experiments, we strictly follow the MemAgent procedure and evaluate GPT-4.1 mini on 10 sampled cases. We observe very poor attribution accuracy (OIA close to 0%) and substantial computational overhead, particularly in runtime.
To enable a more practical comparison with MemTrace, we introduce an early-stopping mechanism: after processing each chunk, the model can directly output its attribution result if it considers the available information sufficient, rather than continuing through the entire execution trace. We evaluate this variant on the RAG and Mem0 subsets, whose execution traces contain fewer than tokens, using GPT-4.1 mini and DeepSeek-V4-Flash-0731.
Table 3: Performance and inference cost of the chunked baseline on the Mem0 and RAG subsets of MemTraceBench.
| Backbone | Memory System | ETA | OIA | Tokens | Time |
|---|---|---|---|---|---|
| GPT-4.1 mini | Mem0 | 20.00 | 0.00 | 337.25 | 5.79 |
| RAG | 15.00 | 0.00 | 258.06 | 3.47 | |
| DeepSeek-v4-Flash-0731 | Mem0 | 62.50 | 32.50 | 2,180.78 | 17.83 |
| RAG | 92.50 | 62.50 | 1,681.99 | 11.66 |
As shown in Table 3, GPT-4.1 mini achieves 0% OIA on both subsets. We observe that it tends to stop prematurely, which also explains its relatively low token consumption and runtime. For DeepSeek-V4-Flash-0731, the chunked baseline achieves higher OIA but still underperforms MemTrace on both subsets (see Table 4). Moreover, processing longer execution traces incurs substantial computational overhead. For example, DeepSeek-V4-Flash-0731 requires an average of 17.83 minutes per case on Mem0. Since Long-Context and EverMemOS have even longer execution traces, applying this approach to these systems would likely incur greater costs.
Changes to the paper: Appendix G now clarifies how starting points are provided to MemTrace-OBS and presents the early-stopping variant and its complete results in Table 6.
Challenges in Adapting CHIEF to Our Setting
CHIEF, while avoiding execution replay, also relies on LLMs to transform unstructured agent trajectories into hierarchical dependency graphs consisting of subtask, agent, and step nodes. Similar to GraphTracer, this LLM-based graph construction can become prohibitively expensive for long-horizon, multi-session interaction histories. More importantly, our scenario do not involve explicit task decomposition or multi-agent collaboration, making it difficult to map our execution graphs onto CHIEF's subtask–agent–step hierarchy. Consequently, CHIEF's hierarchical attribution procedure cannot be directly applied to our setting without substantial redesign.
Changes to the paper: We have added this comparison with CHIEF, including the graph-construction cost and structural mismatch with our setting, to Appendix B.
W1 and Q1: Isolating the Contribution of Execution-Graph Dependencies
Response: Thank you for identifying this confounding factor. We agree that the original comparison between MemTrace and MemTrace-OBS changes both the dependency representation and the search interface. We therefore conduct a controlled ablation that keeps the MemTrace-OBS interface fixed and varies only whether operation dependencies are exposed. Let In(o) and Out(o) denote the input and output variables of operation o, respectively. We define its directly related operations as
\\;\middle|\\; \mathrm{Out}(o') \cap \mathrm{In}(o) \neq \emptyset \\;\lor\\; \mathrm{In}(o') \cap \mathrm{Out}(o) \neq \emptyset \right\\}.$$ The first condition identifies upstream operations that produce an input of $o$, while the second identifies downstream operations that consume an output of $o$. Starting from MemTrace-OBS, we augment each operation block with only the identifiers and names of these **directly related operations**. The prompt, search tools, context budget, and iteration budget remain unchanged. Thus, the only experimental variable is the availability of edge-derived dependency information. Given the rebuttal-time compute constraints, we evaluate GPT-4.1 mini, Qwen3.6-Flash-0416, and DeepSeek-V4-Flash-0731 on the RAG, Mem0, and EverMemOS subsets of MemTraceBench. **Table 1: Controlled ablation of directly related operations in MemTrace-OBS.** | Model | Method | ETA | OIA | Tokens | Time | | ---------------------- | ----------------------------- | --------- | --------- | ------------ | -------- | | GPT-4.1 mini | MemTrace-OBS | 23.61 | 11.39 | 914.62 | 2.55 | | | + Directly Related Operations | **35.00** | **13.33** | **487.60** | **2.35** | | Qwen3.6-Flash-0416 | MemTrace-OBS | 65.83 | 48.33 | 663.17 | **4.46** | | | + Directly Related Operations | **66.67** | **49.17** | **480.50** | 5.88 | | DeepSeek-V4-Flash-0731 | MemTrace-OBS | 60.83 | 53.33 | **1,257.89** | **2.74** | | | + Directly Related Operations | **72.50** | **61.67** | 369,117.98 | 7.99 | **Table 2: Paired Mem0 results stratified by the stage of the gold error.** | Models | Method | Construction ETA | Construction OIA | Non-construction ETA | Non-construction OIA | | ------------------- | ----------------------------- | ----------------------------------: | ---------------------------------: | ----------------------------------: | ---------------------------------: | | All three models | MemTrace-OBS | 50.67 | 32.00 | 48.89 | 44.44 | | | + Directly Related Operations | 61.33 $\textcolor{blue}{(+ 10.66)}$ | 40.00 $\textcolor{blue}{(+8.00)}$ | 57.78 $\textcolor{blue}{(+8.89)}$ | 46.67 $\textcolor{blue}{(+ 2.23)}$ | | GPT-4.1 mini + Qwen | MemTrace-OBS | 52.00 | 28.00 | 36.67 | 30.00 | | | + Directly Related Operations | 62.00 $\textcolor{blue}{(+ 10.00)}$ | 36.00 $\textcolor{blue}{(+ 8.00)}$ | 46.67 $\textcolor{blue}{(+ 10.00)}$ | 30.00 $\textcolor{blue}{(+ 0.00)}$ |