acceptodds
Under review as a conference paper at ICLR 2027

MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

10 views

Abstract

Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug. Tracing memory's dynamic evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM memory systems. We propose a novel framework that transforms memory pipelines into executable memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a benchmark collected from representative memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that memory failures are systematic, stemming from operation-level issues like information loss and retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62 percentage points.

Primary area:
datasets and benchmarks
Keywords:
memory, tracing, agent, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pvjoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemTrace studies how to find the operations responsible for wrong answers in LLM memory systems. The authors instrument memory pipelines to record how information is extracted, updated, retrieved, and used, and build MemTraceBench with 160 human-annotated failures from four memory systems. They propose both graph exploration and operation-log search for automatic diagnosis, then use the diagnoses to guide prompt optimization.

Critical strengths
  • The paper is well written and easy to follow. The motivation makes sense: a wrong answer can come from a memory update many sessions earlier, and inspecting the final response alone will miss that. Figure 2 gives a helpful picture of how the method follows the information through the system.
  • I see value in the tracing toolkit and the annotated benchmark. Recording the evolution of individual memories gives developers something more useful than another end-task accuracy score. The examples of information being dropped or timestamps being reassigned also make the debugging problem concrete.
  • The prompt optimization experiment provides a useful check that the diagnoses can lead to better behavior. I appreciate that the authors split optimization and testing by user trajectory. The improvement from 59.08% to 66.70% is encouraging.
Critical weaknesses
  1. I am not fully convinced by the baseline comparison or by how much of the gain comes from the graph itself. Appendix G says All-at-Once keeps only the latest 600,000 tokens, so it loses exactly the earlier operations this paper aims to diagnose. MemTrace-OBS is a useful comparison, but it changes the representation and search interface together. With GPT-5.4, it actually has higher overall operation identification accuracy than MemTrace, 46.25% versus 38.13%, at much lower token cost. A comparison with matched starting information and budgets would help isolate the value of following dependency edges. Appendix B explains several adaptation difficulties, but a replay-free graph reasoner such as CHIEF, already cited in the paper, is worth considering. The chunked baseline mentioned in Appendix G should also have its accuracy and cost reported, even if it performs poorly.

  2. The operation labels need a little more validation. Figure 7b reports annotator disagreement ranging from 3% to 46%, and Appendix D.6 says the interface shows a preliminary automatic diagnosis to annotators. What algorithm produces these suggestions, and how is it related to the evaluated methods? The independent annotation and final adjudication are useful, but I would like to know whether disagreements concern the error category or the exact operation. Appendix D.2 also merges several retrieval steps into one operation when their responsibility is ambiguous, which affects what exact-operation accuracy means. A small blind audit of the final labels, with examples showing how the decisive operation was chosen, would make the benchmark more convincing.

  3. The evaluation leaves some uncertainty about generalization and the interpretation of the failure patterns. Both attribution backbones are from OpenAI, and all failures are collected from memory systems using GPT-4.1 mini. One attribution model from another family, such as Qwen or DeepSeek, would make the comparison more informative. There is also a dataset-mix issue: Figure 8 shows that the source distributions differ across systems. This makes it hard to tell how much of the difference in Figure 3b comes from the memory architecture and how much comes from the questions being tested. A per-dataset breakdown would help support the stronger comparisons in Section 5.3.

  4. A few presentation details are easy to fix. Table captions should be above the tables. I would also bring the benchmark composition into the main text: Appendix D.2 already gives the collected failure counts for each system, and Figure 8 gives the source proportions, but a single table with exact system-by-dataset counts for both the collected pool and the final benchmark would be much easier to read.

Questions for authors
  1. How do graph exploration and log search compare when starting information and inference budgets are matched? Could you report the chunked baseline’s results and discuss an adaptation of a replay-free attribution method such as CHIEF?
  2. What produces the preliminary diagnoses shown during annotation? How often do annotators disagree on error types versus operation identifiers, and how are ambiguous or merged operations handled in the final labels?
  3. Do the main method rankings hold with an attribution backbone from another model family? Can you provide results by source dataset, along with the exact benchmark counts and repeated-run details?
12 comments

W1 and Q1:\textcolor{red}{\mathrm{W1 \space and \space Q1}\text{:}} Isolating the Contribution of Execution-Graph Dependencies

Response:\textcolor{green}{\mathrm{Response}\text{:}} Thank you for identifying this confounding factor. We agree that the original comparison between MemTrace and MemTrace-OBS changes both the dependency representation and the search interface. We therefore conduct a controlled ablation that keeps the MemTrace-OBS interface fixed and varies only whether operation dependencies are exposed. Let In(o)\mathrm{In}(o) and Out(o)\mathrm{Out}(o) denote the input and output variables of operation oo, respectively. We define its directly related operations as

\\;\middle|\\; \mathrm{Out}(o') \cap \mathrm{In}(o) \neq \emptyset \\;\lor\\; \mathrm{In}(o') \cap \mathrm{Out}(o) \neq \emptyset \right\\}.$$ The first condition identifies upstream operations that produce an input of $o$, while the second identifies downstream operations that consume an output of $o$. Starting from MemTrace-OBS, we augment each operation block with only the identifiers and names of these **directly related operations**. The prompt, search tools, context budget, and iteration budget remain unchanged. Thus, the only experimental variable is the availability of edge-derived dependency information. Given the rebuttal-time compute constraints, we evaluate GPT-4.1 mini, Qwen3.6-Flash-0416, and DeepSeek-V4-Flash-0731 on the RAG, Mem0, and EverMemOS subsets of MemTraceBench. **Table 1: Controlled ablation of directly related operations in MemTrace-OBS.** | Model | Method | ETA | OIA | Tokens | Time | | ---------------------- | ----------------------------- | --------- | --------- | ------------ | -------- | | GPT-4.1 mini | MemTrace-OBS | 23.61 | 11.39 | 914.62 | 2.55 | | | + Directly Related Operations | **35.00** | **13.33** | **487.60** | **2.35** | | Qwen3.6-Flash-0416 | MemTrace-OBS | 65.83 | 48.33 | 663.17 | **4.46** | | | + Directly Related Operations | **66.67** | **49.17** | **480.50** | 5.88 | | DeepSeek-V4-Flash-0731 | MemTrace-OBS | 60.83 | 53.33 | **1,257.89** | **2.74** | | | + Directly Related Operations | **72.50** | **61.67** | 369,117.98 | 7.99 | **Table 2: Paired Mem0 results stratified by the stage of the gold error.** | Models | Method | Construction ETA | Construction OIA | Non-construction ETA | Non-construction OIA | | ------------------- | ----------------------------- | ----------------------------------: | ---------------------------------: | ----------------------------------: | ---------------------------------: | | All three models | MemTrace-OBS | 50.67 | 32.00 | 48.89 | 44.44 | | | + Directly Related Operations | 61.33 $\textcolor{blue}{(+ 10.66)}$ | 40.00 $\textcolor{blue}{(+8.00)}$ | 57.78 $\textcolor{blue}{(+8.89)}$ | 46.67 $\textcolor{blue}{(+ 2.23)}$ | | GPT-4.1 mini + Qwen | MemTrace-OBS | 52.00 | 28.00 | 36.67 | 30.00 | | | + Directly Related Operations | 62.00 $\textcolor{blue}{(+ 10.00)}$ | 36.00 $\textcolor{blue}{(+ 8.00)}$ | 46.67 $\textcolor{blue}{(+ 10.00)}$ | 30.00 $\textcolor{blue}{(+ 0.00)}$ |
AuthorsOpenReview ↗

Response to W1 and Q1 (Continued)

Main results. As shown in Table 1, adding directly related operations improves both ETA and OIA for all three backbones. However, the magnitude and cost of this benefit remain model-dependent. For GPT-4.1 mini and Qwen3.6-Flash-0416, the reported token consumption decreases, although wall-clock time does not decrease consistently. DeepSeek-V4-Flash-0731 exhibits a pronounced runtime tail: the mean runtime increases from 2.74 to 7.99 minutes, whereas the median increases more moderately from 2.07 to 2.98 minutes. Its P95 increases from 7.24 to 37.08 minutes. Qualitative inspection shows that, in these slow cases, DeepSeek-V4-Flash-0731 repeatedly verifies whether the selected operation is the earliest decisive error. Thus, the large mean increase is driven by a relatively small number of unusually long reasoning trajectories.

Error-stage analysis. To determine when dependency information is most useful, we further stratify the same Mem0 cases by their gold error type. We classify Extraction, Update, and Deletion errors as memory-construction errors, and Retrieval and Response errors as non-construction errors. As shown in Table 2, adding directly related operations improve OIA by 8.00 percentage points for construction errors and 2.22 points for non-construction errors. After further excluding DeepSeek-V4-Flash-0731 to control for the anomalously long reasoning trajectories observed in some of its cases, the improvement remains 8.00 points for construction errors but drops to 0.00 points for non-construction errors. These results indicate when execution-graph dependencies are most useful. In memory-construction failures, the faulty operation may occur long before the final question, while its effect becomes visible only in later stages. Showing which earlier operations produce an operation’s inputs and which later operations use its outputs helps the model follow information across these distant steps and locate the original fault. In contrast, operations responsible for retrieval and response failures often contain keywords that overlap with the question or golden answer. They are therefore likely to be surfaced early by text-based search and can usually be diagnosed from their local operation content. Consequently, dependency information provides a smaller additional benefit for their OIA.

Changes to the paper: We have added the controlled dependency ablation to Section 5.3 (Table 3b) and a detailed treatment in Appendix H, including the formal definition of directly related operations and the error-stage results in Table 7.

W1 and Q1:\textcolor{red}{\mathrm{W1 \space and \space Q1}\text{:}} Clarification of Graph-Based Attribution and Additional Baseline Evaluation

Response:\textcolor{green}{\mathrm{Response}\text{:}} First, we would like to clarify that both MemTrace and MemTrace-OBS use starting-point initialization. However, unlike MemTrace, MemTrace-OBS is not required to first inspect operations associated with these starting points. Instead, the starting points are provided as hints in the prompt, indicating that the corresponding messages may contain source evidence relevant to the question.

Additional evaluation of the chunked baseline. We would also like to clarify the performance of the chunked baseline. In our preliminary experiments, we strictly follow the MemAgent procedure and evaluate GPT-4.1 mini on 10 sampled cases. We observe very poor attribution accuracy (OIA close to 0%) and substantial computational overhead, particularly in runtime.

To enable a more practical comparison with MemTrace, we introduce an early-stopping mechanism: after processing each chunk, the model can directly output its attribution result if it considers the available information sufficient, rather than continuing through the entire execution trace. We evaluate this variant on the RAG and Mem0 subsets, whose execution traces contain fewer than 10710^7 tokens, using GPT-4.1 mini and DeepSeek-V4-Flash-0731.

Table 3: Performance and inference cost of the chunked baseline on the Mem0 and RAG subsets of MemTraceBench.

BackboneMemory SystemETAOIATokensTime
GPT-4.1 miniMem020.000.00337.255.79
RAG15.000.00258.063.47
DeepSeek-v4-Flash-0731Mem062.5032.502,180.7817.83
RAG92.5062.501,681.9911.66

As shown in Table 3, GPT-4.1 mini achieves 0% OIA on both subsets. We observe that it tends to stop prematurely, which also explains its relatively low token consumption and runtime. For DeepSeek-V4-Flash-0731, the chunked baseline achieves higher OIA but still underperforms MemTrace on both subsets (see Table 4). Moreover, processing longer execution traces incurs substantial computational overhead. For example, DeepSeek-V4-Flash-0731 requires an average of 17.83 minutes per case on Mem0. Since Long-Context and EverMemOS have even longer execution traces, applying this approach to these systems would likely incur greater costs.

Changes to the paper: Appendix G now clarifies how starting points are provided to MemTrace-OBS and presents the early-stopping variant and its complete results in Table 6.

W1 and Q1:\textcolor{red}{\mathrm{W1 \space and \space Q1}\text{:}} Challenges in Adapting CHIEF to Our Setting

Response:\textcolor{green}{\mathrm{Response}\text{:}} CHIEF, while avoiding execution replay, also relies on LLMs to transform unstructured agent trajectories into hierarchical dependency graphs consisting of subtask, agent, and step nodes. Similar to GraphTracer, this LLM-based graph construction can become prohibitively expensive for long-horizon, multi-session interaction histories. More importantly, our scenario do not involve explicit task decomposition or multi-agent collaboration, making it difficult to map our execution graphs onto CHIEF's subtask–agent–step hierarchy. Consequently, CHIEF's hierarchical attribution procedure cannot be directly applied to our setting without substantial redesign.

Changes to the paper: We have added this comparison with CHIEF, including the graph-construction cost and structural mismatch with our setting, to Appendix B.

Reviewer_hbU6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of error tracing and attribution in long-term memory systems. The core motivation is that an incorrect response may stem from memory retrieval, updates, or deletions that occurred much earlier; however, logs captured solely at the final response stage often lack the information necessary to reveal these cross-session state changes. The authors developed an instrumentation tool called "smartcomment," which represents the execution process as an "execution graph"—a structure capturing operations, variables, and their interdependencies. The diagnostic objective is defined as identifying the earliest decisive erroneous operation. Based on this representation, the proposed framework, MemTrace, first retrieves source messages relevant to a given query and then traverses the local operation subgraph to verify whether information was correctly preserved and utilized. The authors constructed "MemTraceBench," a dataset comprising 160 system error cases drawn from Long-Context, RAG, Mem0, and EverMemOS, with data sourced from LoCoMo, LongMemEval, and RealMem. Evaluation metrics include the accuracy of error-type classification and the precision of identifying the specific erroneous operation. The study compares three approaches: full log analysis, log searching without graph-based dependencies, and exploration via execution graphs. Experiments indicate that execution graphs are beneficial for certain systems and when using less capable diagnostic models, though the advantage is not consistent across all scenarios. When using GPT-5.4, MemTrace achieved an overall operation-localization accuracy of 38.13%, compared to 46.25% for MemTrace-OBS. The paper also demonstrates applications in automated diagnostic reporting and prompt optimization; for instance, the performance score for Mem0 on the LoCoMo benchmark improved from 59.08% to 66.70%.

Critical strengths
  1. Transforming state changes in memory—which span across sessions—into inspectable information dependencies is a diagnostically significant design choice. In my view, the most valuable technical contribution of this paper lies in its approach to tracking how variables are created, modified, passed, and utilized, and then linking these processes through explicit dependencies. In long-term memory systems, there is often a significant temporal gap between the moment an error occurs and the moment it is finally exposed; consequently, observing only the final retrieval and response is typically insufficient to pinpoint exactly where the information was lost. Appendix E addresses critical engineering details—such as in-place modifications, object replication, identity tracking post-serialization, and deletion markers—that determine whether the tracing mechanism can effectively capture state transitions, rather than merely generating a visual log. The fact that this tool has been applied to four distinct memory systems demonstrates that its representation is sufficiently flexible to accommodate the heterogeneous implementations discussed in the paper. This capability underscores the paper’s value as a diagnostic infrastructure. Even if the automated attribution methods leave room for improvement, the ability to link early state changes to subsequent failures provides a concrete basis for both manual inspection and future algorithmic research.
  2. The benchmark construction rigorously distinguishes between system errors, data errors, and evaluation errors, while offering an initial demonstration of the actionability of the diagnostic results. Rather than labeling all instances where the evaluation model flagged an error as "system failures," the authors employed a process where each case was reviewed by three annotators, with final labels determined by majority vote or discussion. This resulted in a dataset comprising 160 system errors, alongside 67 annotation errors and 3 evaluation errors. This methodology is significant, as it prevents erroneous reference answers from being misclassified as system-level diagnostic targets. Furthermore, the paper explores the use of operation-level diagnostics to guide prompt modifications, reporting performance gains across seven test trajectories. While the attribution logic behind these improvements could be more robust, the experiment provides a promising initial signal: diagnostic outputs can potentially be translated into concrete system enhancements, rather than serving merely as post-hoc descriptions of failure. Collectively, these elements establish a cohesive research loop, deriving value from inspectable execution traces, human-verified diagnostic targets, and a clear connection to downstream system refinements.
Critical weaknesses

W1. The independent contribution of the execution graph has not been sufficiently demonstrated; the key localization results and computational costs do not favor the core methodology.

Table 1 shows that, when using GPT-5.4, MemTrace achieves an overall error-operation localization accuracy of 38.13%—lower than the 46.25% achieved by MemTrace-OBS—representing a difference of 8.12 percentage points. Meanwhile, the accuracy for identifying error types improves only marginally, from 53.75% to 54.38% (an increase of 0.63 percentage points). Table 2 indicates that MemTrace consumes approximately 6.28 times more tokens and requires about 4.22 times the runtime of MemTrace-OBS. Consequently, when employing a more capable diagnostic model, the exploration of an execution graph has yet to demonstrate a clear advantage in terms of localization quality relative to the associated costs.

I note that the authors acknowledge in Section 5.2 that the utility of the graph structure depends on both model capability and the type of error; indeed, MemTrace does yield better localization results on the "Long-Context" and "Mem0" subsets when using GPT-5.4. While this supports a conclusion of conditional effectiveness, the paper does not sufficiently explain what precisely determines these conditions. Furthermore, since MemTrace-OBS alters both the representation of dependencies and the search interface simultaneously, the current comparison fails to isolate the specific contribution of the graph edges themselves.

This issue concerns the paper’s core technical contribution. If the primary gains stem from the filtering of relevant logs, the search strategy, or the organization of context, then an execution graph might be one viable implementation—but its necessity and the justification for its additional costs remain unproven. It is essential to compare exploration with and without dependency information—while keeping the available information and budget as consistent as possible—and to report whether the observed gains are consistently concentrated in specific error types, such as early-stage state corruption.

W2. The prompt optimization experiments fail to adequately isolate the role of operation-level attribution; the performance decline in the control group could be explained by other factors.

Section 6.2 reports that MemTrace-guided optimization improves the score from 59.08% to 66.70%, whereas the control group—lacking such attribution—sees a decline to 44.73%. However, Appendix I.2 ​​indicates that the differences between the two groups encompass both the subgraphs visible to the optimizer and the scope of prompts permitted for modification. MemTrace provides relevant operational subgraphs based on localization results and restricts modifications to the prompts associated with those specific operations. In contrast, the control group is provided only with the subgraph corresponding to the final response stage, yet it is tasked with updating prompts across all three stages: extraction, update decision-making, and response generation.

While this setup allows for the evaluation of the entire workflow, it precludes isolating performance gains to accurate operation localization. Because the control group is required to modify upstream extraction and update modules without having access to the evidence that informs these processes, its observed performance degradation may stem from an overly broad editing scope—one applied in the absence of sufficient supporting evidence. Furthermore, the MemTrace group utilizes source evidence and system priors; it remains unclear how the control group obtains equivalent auxiliary information.

The authors employ three optimization trajectories and seven independent test trajectories, which ensures that the results are not merely artifacts of evaluating the model on the same trajectories used for optimization. Nevertheless, this train-test split does not resolve the aforementioned confounding of mechanisms. To substantiate that localized attribution is the primary driver of performance gains, it would be necessary to incorporate an optimization baseline guided by MemTrace-OBS, alongside a control condition that more closely aligns in terms of available evidence and the scope of permissible modifications. This would allow for the disentanglement of the individual contributions of localization accuracy, information provision, and editing constraints.

W3. Critical uncertainty remains regarding the reliability of the "earliest decisive error" labels; current evaluation primarily measures alignment with human annotations.

Section 2 defines a "decisive error" as an operation that, if corrected—assuming all subsequent operations are then executed ideally—would enable the system to produce the correct final answer. A footnote clarifies that this counterfactual process is not actually executed during the experiments. While such a definition serves as a useful diagnostic convention, it does not guarantee that correcting only the identified operation will restore the system to correct, functional behavior. Consequently, the accuracy of operation ID matching alone cannot validate the actual remedial value of the localized operation.

Furthermore, the human annotation process is susceptible to "anchoring" effects. Appendix D.6 notes that the annotation interface displays the error types and locations predicted by the automated attribution algorithm. Although the authors explicitly frame these outputs as mere references and employ three independent annotators, such independence does not preclude a shared reliance on the same automated suggestions. Appendix J.2 reports an inter-annotator disagreement rate ranging from 3% to 46%, indicating that some cases are inherently difficult to evaluate.

Furthermore, Appendix D.2 permits the merging of indistinguishable low-level operations into a single operation, which alters the granularity of the "correct" operation IDs. For metrics relying on exact ID matching, the specific method of operation segmentation can itself influence the difficulty of the task. It is necessary to clarify the relationship between the pre-attribution algorithm and the methods being evaluated. Additionally, the validity of the labels—as a foundation for core evaluation—should be further verified through methods such as blind annotation on a subset, review of alternative plausible labels, or small-scale checks against actual system fixes.

W4. The selection process for the benchmark and the composition of data across systems limit the interpretation of fault distribution and the scope of full-lifecycle diagnosis.

Appendix D.4 specifies a quota of 40 system errors collected per system; however, Appendix J.2 reveals that the final error samples for each system differ significantly in their dataset composition. Errors for Mem0 and EverMemOS are predominantly drawn from LoCoMo, whereas those for RAG and Long-Context systems are largely from LongMemEval. Consequently, the differences observed in Figure 3 are influenced by a combination of system architecture, dataset content, query types, and selection criteria.

While this design is effective for constructing a benchmark with a balanced number of systems, the observed error proportions cannot be directly interpreted as stable, intrinsic characteristics of the system architectures themselves. For instance, claims regarding the superior robustness or generalizability of a specific retrieval module would require further validation under consistent data conditions and with appropriate denominators. Simply observing the errors that occurred and were subsequently selected for each system is insufficient to estimate the overall failure probability of an individual module.

The current benchmark does not account for deletion errors, and the attribution of multiple concurrent errors is relegated to an extended discussion in Appendix F. The authors acknowledge these limitations; the issue is not a lack of awareness regarding insufficient coverage, but rather that the abstract’s claim—implying the ability to perform root-cause localization for any failure case—exceeds the scope of the provided validation. The current evidence supports diagnostic conclusions only within the context of the specific systems, configurations, and single-error scenarios examined. ### W5. Reliance on ground truth and source evidence has been partially analyzed, but end-to-end diagnostic validation in the absence of ground truth is missing.

MemTrace’s formal input includes ground truth, which is used alongside the query to retrieve initial evidence. Within this offline diagnostic framework, the use of ground truth is an acceptable experimental condition and should not be automatically classified as data leakage. However, when the paper discusses failure diagnosis in a real-world deployment scenario, the input conditions shift, as the correct answer is typically unavailable.

While Appendix J.1 addresses this issue, it only reports the Recall@8 for source evidence retrieval. The recall is 70.34% when ground truth is provided, 59.50% when only the query is used, and 62.20% when the system’s own predictions are incorporated. These results indicate that alternative queries can provide some retrieval cues; however, they do not clarify how subsequent error-type classification, operational localization, or diagnostic costs are affected. Furthermore, erroneous predictions could lead the system to explore irrelevant starting nodes—an impact that cannot be captured by average retrieval recall alone.

As shown in Table 3, the provision of source evidence and system priors significantly improves attribution performance, underscoring the substantial influence of initial information on the final outcome. While the current evidence strongly supports a diagnostic scenario assisted by ground truth or human-provided source evidence, the system’s full diagnostic capability in the absence of such information remains unverified. It is necessary to report end-to-end ETA, OIA, and diagnostic costs under these conditions, or to explicitly narrow the scope of the conclusions regarding real-world deployment.

Questions for authors
  1. Under what specific conditions do execution graphs provide independent gains? In the case of GPT-5.4, MemTrace exhibits a lower overall OIA (Overall Attribution Accuracy) than MemTrace-OBS, yet incurs significantly higher costs. Is it possible to report paired results for the same error cases, distinguishing between errors that occur during early-stage extraction/updating and those that arise during later-stage retrieval/answering? If the search interface and budget are held constant—varying only the availability of dependency relationships—does the advantage persist?
  2. Can the gains attributed to prompt optimization be disentangled from those resulting from the provision of information and editing constraints? Why does the "no-attribution" control condition—which receives only the final answer subgraph—require modifications to the prompts used across all three stages? Do both groups have access to the same source evidence and system priors? How would performance shift if the system utilized the localization results from MemTrace-OBS, or if the same local editing constraints were applied? These findings will be crucial in determining whether I can accept the claim that "correct attribution is the primary source of the gains."
  3. How do automated attribution suggestions influence human labeling? What exactly is the pre-attribution algorithm described in Appendix D.6, and does it share the same model, prompts, or reasoning pipeline as MemTrace? Are annotators exposed to these suggestions before forming their initial judgments? Could you provide independent re-labeling results—collected without showing the automated suggestions—and report the rate at which labels changed?
  4. What is the empirical relationship between "critical errors" and errors that are actually fixable? Has the study verified—at least for a subset of samples—whether correcting only the identified operation (while keeping the downstream modules unchanged) leads to an improvement in the final answer? How does the current method, which relies on exact ID matching, handle cases where multiple valid operation points exist? For operations that have been merged, could you clarify their granularity and the specific prompts that can be modified?
  5. Does the distribution of failures across systems hold true when the data sources are controlled? Could you report the error distribution broken down by shared datasets or identical execution traces, and provide the denominators (e.g., the total number of questions evaluated)? I am particularly interested in verifying whether the assessment that the EverMemOS extraction module is more robust remains valid when the inputs and questions are held constant.
12 comments

W1 and Q1:\textcolor{red}{\mathrm{W1 \space and \space Q1}\text{:}} Isolating the Contribution of Execution-Graph Dependencies

Response:\textcolor{green}{\mathrm{Response}\text{:}} Thank you for identifying this confounding factor. We agree that the original comparison between MemTrace and MemTrace-OBS changes both the dependency representation and the search interface. We therefore conduct a controlled ablation that keeps the MemTrace-OBS interface fixed and varies only whether operation dependencies are exposed. Let In(o)\mathrm{In}(o) and Out(o)\mathrm{Out}(o) denote the input and output variables of operation oo, respectively. We define its directly related operations as

\\;\middle|\\; \mathrm{Out}(o') \cap \mathrm{In}(o) \neq \emptyset \\;\lor\\; \mathrm{In}(o') \cap \mathrm{Out}(o) \neq \emptyset \right\\}.$$ The first condition identifies upstream operations that produce an input of $o$, while the second identifies downstream operations that consume an output of $o$. Starting from MemTrace-OBS, we augment each operation block with only the identifiers and names of these **directly related operations**. The prompt, search tools, context budget, and iteration budget remain unchanged. Thus, the only experimental variable is the availability of edge-derived dependency information. Given the rebuttal-time compute constraints, we evaluate GPT-4.1 mini, Qwen3.6-Flash-0416, and DeepSeek-V4-Flash-0731 on the RAG, Mem0, and EverMemOS subsets of MemTraceBench. **Table 1: Controlled ablation of directly related operations in MemTrace-OBS.** | Model | Method | ETA | OIA | Tokens | Time | | ---------------------- | ----------------------------- | --------- | --------- | ------------ | -------- | | GPT-4.1 mini | MemTrace-OBS | 23.61 | 11.39 | 914.62 | 2.55 | | | + Directly Related Operations | **35.00** | **13.33** | **487.60** | **2.35** | | Qwen3.6-Flash-0416 | MemTrace-OBS | 65.83 | 48.33 | 663.17 | **4.46** | | | + Directly Related Operations | **66.67** | **49.17** | **480.50** | 5.88 | | DeepSeek-V4-Flash-0731 | MemTrace-OBS | 60.83 | 53.33 | **1,257.89** | **2.74** | | | + Directly Related Operations | **72.50** | **61.67** | 369,117.98 | 7.99 | **Table 2: Paired Mem0 results stratified by the stage of the gold error.** | Models | Method | Construction ETA | Construction OIA | Non-construction ETA | Non-construction OIA | | ------------------- | ----------------------------- | ----------------------------------: | ---------------------------------: | ----------------------------------: | ---------------------------------: | | All three models | MemTrace-OBS | 50.67 | 32.00 | 48.89 | 44.44 | | | + Directly Related Operations | 61.33 $\textcolor{blue}{(+ 10.66)}$ | 40.00 $\textcolor{blue}{(+8.00)}$ | 57.78 $\textcolor{blue}{(+8.89)}$ | 46.67 $\textcolor{blue}{(+ 2.23)}$ | | GPT-4.1 mini + Qwen | MemTrace-OBS | 52.00 | 28.00 | 36.67 | 30.00 | | | + Directly Related Operations | 62.00 $\textcolor{blue}{(+ 10.00)}$ | 36.00 $\textcolor{blue}{(+ 8.00)}$ | 46.67 $\textcolor{blue}{(+ 10.00)}$ | 30.00 $\textcolor{blue}{(+ 0.00)}$ |
AuthorsOpenReview ↗

Response to W1 and Q1 (Continued)

Main results. As shown in Table 1, adding directly related operations improves both ETA and OIA for all three backbones. However, the magnitude and cost of this benefit remain model-dependent. For GPT-4.1 mini and Qwen3.6-Flash-0416, the reported token consumption decreases, although wall-clock time does not decrease consistently. DeepSeek-V4-Flash-0731 exhibits a pronounced runtime tail: the mean runtime increases from 2.74 to 7.99 minutes, whereas the median increases more moderately from 2.07 to 2.98 minutes. Its P95 increases from 7.24 to 37.08 minutes. Qualitative inspection shows that, in these slow cases, DeepSeek-V4-Flash-0731 repeatedly verifies whether the selected operation is the earliest decisive error. Thus, the large mean increase is driven by a relatively small number of unusually long reasoning trajectories.

Error-stage analysis. To determine when dependency information is most useful, we further stratify the same Mem0 cases by their gold error type. We classify Extraction, Update, and Deletion errors as memory-construction errors, and Retrieval and Response errors as non-construction errors. As shown in Table 2, adding directly related operations improve OIA by 8.00 percentage points for construction errors and 2.22 points for non-construction errors. After further excluding DeepSeek-V4-Flash-0731 to control for the anomalously long reasoning trajectories observed in some of its cases, the improvement remains 8.00 points for construction errors but drops to 0.00 points for non-construction errors. These results indicate when execution-graph dependencies are most useful. In memory-construction failures, the faulty operation may occur long before the final question, while its effect becomes visible only in later stages. Showing which earlier operations produce an operation’s inputs and which later operations use its outputs helps the model follow information across these distant steps and locate the original fault. In contrast, operations responsible for retrieval and response failures often contain keywords that overlap with the question or golden answer. They are therefore likely to be surfaced early by text-based search and can usually be diagnosed from their local operation content. Consequently, dependency information provides a smaller additional benefit for their OIA.

Changes to the paper: We have added the controlled dependency ablation to Section 5.3 (Table 3b) and a detailed treatment in Appendix H, including the formal definition of directly related operations and the error-stage results in Table 7.

W2 and Q2:\textcolor{red}{\mathrm{W2 \space and \space Q2}\text{:}} Disentangling the Effects of Failure Attribution on Prompt Optimization

Response:\textcolor{green}{\mathrm{Response}\text{:}} Thank you for your detailed comments. We agree that the original comparison does not fully isolate the effect of failure attribution. We clarify the baseline design and provide additional experiments below.

Clarification of the no-attribution baseline. Without failure attribution, the optimizer cannot identify which operation causes the incorrect answer. Therefore, we treat all three operations (extraction, update, and response) as possible sources of error and allow the optimizer to modify all three prompts. Since no localization is performed, the optimizer only receives the final response subgraph, rather than a localized subgraph of the potentially faulty operation. Regarding auxiliary information, the no-attribution baseline does not use source evidence (SE) or prior knowledge (PK), as these are used to support failure attribution.

Additional experiments with controlled optimization settings. To better understand the role of attribution, we add two settings: MemTrace-OBS and MemTrace without SE or PK. The original MemTrace setting in Figure 4 uses both SE and PK, which we now denote as MemTrace (w/ SE & PK). All three settings follow the same optimization procedure: the optimizer receives the subgraph selected by the attribution method and modifies only the prompts associated with the identified operation. Thus, the editing rule is identical across these settings. In addition, MemTrace-OBS and MemTrace use neither SE nor PK, allowing a more direct comparison of their attribution methods.

Table 3. Impact of failure attribution accuracy on downstream prompt optimization performance.

Attribution MethodOIA (%)Score (%)
MemTrace-OBS55.0052.52
MemTrace59.1756.78
MemTrace (w/ SE & PK)72.5066.70

As shown in the Table 3, better attribution accuracy leads to better prompt optimization results. Without SE and PK, MemTrace achieves a higher OIA than MemTrace-OBS (59.17% vs. 55.00%) and a higher optimization score (56.78% vs. 52.52%), under the same editing rules and auxiliary-information conditions. With SE and PK, MemTrace achieves the highest OIA (72.50%) and optimization score (66.70%). However, both MemTrace-OBS and MemTrace without SE and PK perform worse than the original system (59.08%). Only MemTrace with SE and PK improves performance, by 7.62 percentage points. These results suggest that accurate failure attribution plays an important role in prompt optimization, while inaccurate attribution may lead to ineffective or even harmful updates.

Changes to the paper: We have revised Section 6.2 and Figure 4b to report all three attribution-guided settings.

Reviewer_vYTzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses failure diagnosis and error attribution in LLM memory systems. To overcome the temporal separation between when an error is introduced and when it surfaces downstream, the authors propose an execution graph abstraction that models memory operations and intermediate variables as a directed acyclic bipartite graph via a lightweight instrumentation package. The paper introduces MemTraceBench, a benchmark of 160 human-annotated system failures across three public datasets categorized into a 7-type error taxonomy, and evaluates MemTrace, an agentic graph-traversal method that iteratively traces localized operation subgraphs to pinpoint the earliest decisive faulty operation. Experiments demonstrate that MemTrace outperforms flattened log baselines, improving Mem0 end-task performance by 7.62 percentage points.

Critical strengths

The paper investigates an important problem: errors introduced during memory construction or updates may only affect responses many interactions later. The tracing toolkit and operation-level benchmark provide useful support for investigating these errors.

The authors demonstrate a practical application of failure attribution. Using the attribution results to guide prompt optimisation improves Mem0’s score from 59.08% to 66.70% on held-out LoCoMo trajectories.

Critical weaknesses
  • Only two outdated LLMs studied: GPT-4.1 mini and GPT-5.4. Also no open-sources LLMs are studied.
  • The correctness of the causal attribution labels requires further validation. The labels rely on human judgements without intervention-based validation, and initial pairwise annotator disagreement ranges from 3% to 46%. Although discussion produces consensus labels, this does not establish that the identified operations are causally decisive.
Questions for authors

Under what conditions does graph traversal outperform operation search? Can MemTrace-OBS achieve comparable improvements in downstream prompt optimisation?

Can the authors validate the identified causes through controlled interventions on a subset of cases, and obtain independent annotations without showing automatic attribution suggestions?

9 comments

W1:\textcolor{red}{\mathrm{W1}\text{:}} Evaluation with Recent Open-Source and Non-OpenAI Models

Response:\textcolor{green}{\mathrm{Response}\text{:}} Thank you for this suggestion. We agree that evaluating additional model families would strengthen our conclusions. We extend our experiments to two recent models: Qwen3.6-Flash-0416 (released in April 2026) and DeepSeek-V4-Flash-0731 (released in July 2026). Both models are evaluated with thinking enabled and reasoning effort set to high. The additional attribution accuracy and inference cost results are reported in Tables 1 and 2, respectively.

Table 1: Failure attribution accuracy on MemTraceBench across memory systems.

BackboneMethodLong-Context ETALong-Context OIARAG ETARAG OIAMem0 ETAMem0 OIAEverMemOS ETAEverMemOS OIAOverall ETAOverall OIA
Qwen3.6-Flash-0416All-at-Once25.000.0070.0047.5035.0012.5047.505.0044.3816.25
MemTrace-OBS17.5015.0075.0067.5065.0052.5057.5025.0053.7540.00
MemTrace20.0015.0032.5040.0067.5055.0032.5010.0038.1330.00
DeepSeek-V4-Flash-0731All-at-Once22.500.0057.5042.5035.002.5037.5015.0038.1315.00
MemTrace-OBS10.0010.0077.5075.0057.5052.5047.5032.5048.1342.50
MemTrace42.5037.5065.0065.0077.5062.5045.0020.0057.5046.25

Table 2: Average inference cost on MemTraceBench across memory systems.

BackboneMethodLong-Context TokensLong-Context TimeRAG TokensRAG TimeMem0 TokensMem0 TimeEverMemOS TokensEverMemOS TimeOverall TokensOverall Time
Qwen3.6-Flash-0416All-at-Once1,260.524.771,312.055.141,305.915.501,218.934.501,274.354.98
MemTrace-OBS601.843.461,121.724.20434.062.63433.723.68647.843.49
MemTrace2,746.6512.001,330.334.68918.203.721,598.704.651,648.476.26
DeepSeek-V4-Flash-0731All-at-Once1,178.780.941,219.171.161,226.371.011,138.751.021,190.771.03
MemTrace-OBS1,357.403.271,122.391.70774.481.761,876.794.771,282.762.87
MemTrace1,156.682.06897.912.53997.362.241,583.532.411,158.872.31
AuthorsOpenReview ↗

Response to W1 (Continued)

Attribution performance across model families. The results reveal that the optimal exploration strategy varies across backbone models. On Qwen3.6-Flash-0416, MemTrace-OBS achieves higher overall OIA than MemTrace (40.00% vs. 30.00%), whereas MemTrace performs better on DeepSeek-V4-Flash-0731 (46.25% vs. 42.50%). This suggests that the effectiveness of different exploration strategies depends partly on the backbone model, potentially reflecting differences in their training and reasoning capabilities.

Despite these differences, we observe a consistent pattern across memory-system architectures. MemTrace-OBS achieves higher OIA on both RAG and EverMemOS across the two new backbones, whereas MemTrace performs better on Mem0 and Long-Context. A closer examination of the failure cases suggests that MemTrace is more effective at identifying memory construction errors, where the faulty operation may occur much earlier in the execution history. In contrast, MemTrace-OBS tends to focus on operations closer to retrieval or response generation, making it more effective for these later-stage errors. These findings are consistent with our original observation that graph-based exploration is particularly beneficial for tracing upstream memory errors, while operation-based search is often more effective for retrieval and response errors.

Inference efficiency. Table 2 shows that the efficiency of the two exploration strategies also depends on the backbone model. MemTrace-OBS is more efficient on Qwen3.6-Flash-0416, whereas MemTrace achieves slightly lower overall token consumption and runtime on DeepSeek-V4-Flash-0731. Our examination of the generated reasoning traces suggests that DeepSeek-V4-Flash-0731 tends to produce longer chains of thought during search-based operation exploration, increasing the cost of MemTrace-OBS. Nevertheless, the efficiency gap between the two methods is relatively small on DeepSeek-V4-Flash-0731 compared with the substantial advantage of MemTrace-OBS observed with other backbones.

Importantly, these results also qualify our original conclusion that search-based operation exploration offers an efficient alternative for weakly structured traces. This efficiency advantage is not consistent when reasoning-enabled models are used, as longer reasoning traces can offset the benefits of search-based exploration. We therefore revise this conclusion to emphasize that the efficiency of search-based exploration depends on the backbone model and its reasoning configuration, rather than presenting it as a general advantage.

Changes to the paper: We have added Qwen3.6-Flash-0416 and DeepSeek-V4-Flash-0731 to the main accuracy and cost results in Tables 1 and 2, expanded Section 5.2 with backbone- and error-stage-dependent analyses.

Q1:\textcolor{red}{\mathrm{Q1}\text{:}} Under what conditions does graph traversal outperform operation search?

Response:\textcolor{green}{\mathrm{Response}\text{:}} Based on our response to W1\textcolor{red}{\mathrm{W1}} and the experimental results in the original paper, graph traversal is particularly beneficial in two scenarios:

(1) When using weaker backbone models, the graph structure provides explicit dependency information that helps the model identify faulty operations more accurately.

(2) When diagnosing errors in memory construction, graph traversal leverages operation dependencies to identify faulty operations in the memory construction stage, which may occur long before the final question is asked.

Furthermore, we conduct additional ablation experiments to isolate the contribution of graph structure. Let In(o)\mathrm{In}(o) and Out(o)\mathrm{Out}(o) denote the input and output variables of operation oo, respectively. We define its directly related operations as

\\;\middle|\\; \mathrm{Out}(o') \cap \mathrm{In}(o) \neq \emptyset \\;\lor\\; \mathrm{In}(o') \cap \mathrm{Out}(o) \neq \emptyset \right\\}.$$ The first condition identifies upstream operations that produce an input of $o$, while the second identifies downstream operations that consume an output of $o$. Starting from MemTrace-OBS, we augment each operation block with only the identifiers and names of these **directly related operations**. The prompt, search tools, context budget, and iteration budget remain unchanged. Thus, the only experimental variable is the availability of edge-derived dependency information. Given the rebuttal-time compute constraints, we evaluate GPT-4.1 mini, Qwen3.6-Flash-0416, and DeepSeek-V4-Flash-0731 on the RAG, Mem0, and EverMemOS subsets of MemTraceBench. **Table 3: Controlled ablation of directly related operations in MemTrace-OBS.** | Model | Method | ETA | OIA | Tokens | Time | | ---------------------- | ----------------------------- | --------- | --------- | ------------ | -------- | | GPT-4.1 mini | MemTrace-OBS | 23.61 | 11.39 | 914.62 | 2.55 | | | + Directly Related Operations | **35.00** | **13.33** | **487.60** | **2.35** | | Qwen3.6-Flash-0416 | MemTrace-OBS | 65.83 | 48.33 | 663.17 | **4.46** | | | + Directly Related Operations | **66.67** | **49.17** | **480.50** | 5.88 | | DeepSeek-V4-Flash-0731 | MemTrace-OBS | 60.83 | 53.33 | **1,257.89** | **2.74** | | | + Directly Related Operations | **72.50** | **61.67** | 369,117.98 | 7.99 | **Table 4: Paired Mem0 results stratified by the stage of the gold error.** | Models | Method | Construction ETA | Construction OIA | Non-construction ETA | Non-construction OIA | | ------------------- | ----------------------------- | ----------------------------------: | ---------------------------------: | ----------------------------------: | ---------------------------------: | | All three models | MemTrace-OBS | 50.67 | 32.00 | 48.89 | 44.44 | | | + Directly Related Operations | 61.33 $\textcolor{blue}{(+ 10.66)}$ | 40.00 $\textcolor{blue}{(+8.00)}$ | 57.78 $\textcolor{blue}{(+8.89)}$ | 46.67 $\textcolor{blue}{(+ 2.23)}$ | | GPT-4.1 mini + Qwen | MemTrace-OBS | 52.00 | 28.00 | 36.67 | 30.00 | | | + Directly Related Operations | 62.00 $\textcolor{blue}{(+ 10.00)}$ | 36.00 $\textcolor{blue}{(+ 8.00)}$ | 46.67 $\textcolor{blue}{(+ 10.00)}$ | 30.00 $\textcolor{blue}{(+ 0.00)}$ |

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (1/2)

Dear reviewers, thank you for your thoughtful and constructive feedback!

Our Strengths Summarized by Reviewers

We sincerely appreciate the reviewers for highlighting the following strengths of our work:

  • Important and well-motivated problem. The paper addresses the challenging problem that errors introduced during memory construction may only become visible many interactions later, making final-response inspection insufficient for root-cause diagnosis (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}}).
  • Useful and flexible diagnostic infrastructure. The tracing toolkit transforms cross-session memory-state changes into inspectable information dependencies, captures practically important state transitions, and supports heterogeneous memory-system implementations (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}}).
  • Carefully constructed operation-level benchmark. The benchmark provides concrete diagnostic targets and rigorously separates system errors from annotation and evaluation errors through human verification (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}}).
  • Actionable failure attribution. The prompt-optimization experiment demonstrates that diagnostic results can guide concrete system improvements, increasing Mem0's held-out LoCoMo score from 59.08% to 66.70% with a trajectory-level train-test split (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}}).
  • Clear presentation and intuitive examples. The paper is well written and easy to follow, while its figures and examples make the long-horizon debugging problem concrete (Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}}).
AuthorsOpenReview ↗

Global Response (2/2)

Our Main Revisions and Supplementary Experiments

In response to the reviewers' feedback, we conduct the following new experiments and revise the manuscript accordingly:

  • Evaluation with Recent Open-Source and Non-OpenAI Models (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We add Qwen3.6-Flash-0416 and DeepSeek-V4-Flash-0731 to the main accuracy and cost results in Tables 1 and 2 and expand Section 5.2. The new results establish that the preferred exploration strategy depends on both the backbone and error stage, and we accordingly qualify the efficiency claim for search-based exploration.
  • Isolating the Contribution of Execution-Graph Dependencies (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We add a controlled ablation that holds the search interface fixed and varies only the availability of edge-derived dependencies (Section 5.3, Table 3b; Appendix H, Table 7). Dependency information improves ETA and OIA for all three tested backbones and is especially useful for tracing temporally distant memory-construction errors: OIA increases by 8.00 points for construction errors versus 2.22 points for non-construction errors.
  • Disentangling Failure Attribution from Prompt Optimization (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}})
    • We revise Section 6.2 and Figure 4b to compare MemTrace-OBS, MemTrace without source evidence or prior knowledge, and MemTrace with both. The controlled results show that higher OIA leads to better optimization and that only sufficiently accurate attribution improves over the original system, from 59.08% to 66.70%.
  • Preliminary Attribution and Annotation Reliability (Reviewer vYTz\textcolor{orange}{\mathrm{Reviewer \space vYTz}}, Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We document the preliminary attribution procedure and its limitations in Appendix D.7 and add a blind re-annotation study in Appendix D.8, which achieves 93.33% agreement with the original labels. We also clarify the disagreement statistics in Figure 7b and add Figure 8 in Appendix K.2 to separately report error-type and operation-ID disagreement on system-related cases.
  • Failure-Attribution Scope and Multi-Error Evaluation (Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}})
    • We revise the abstract to state that MemTrace identifies potential root causes rather than guaranteeing the root cause of every failed case. We also add set-level Precision and Recall in Appendix F for evaluating cases with multiple decisive faulty operations.
  • End-to-End Attribution without Ground-Truth Answers (Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}})
    • We add a query-construction ablation with query-only, query-plus-system-prediction, and query-plus-golden-answer settings (Appendix K.1, Table 9). The results show that strong reasoning models can retain competitive attribution accuracy without a golden answer, while revealing a substantial efficiency trade-off.
  • Controlled and Per-Dataset Failure Distributions (Reviewer hbU6\textcolor{orange}{\mathrm{Reviewer \space hbU6}}, Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We add per-dataset error distributions for all four memory systems on LoCoMo, LongMemEval, and RealMem (Appendix K.2, Figure 10). The analysis shows that the principal error patterns are stable across datasets for most systems and are not primarily driven by differences in question sources.
  • Graph-Based Attribution and Additional Baseline Comparisons (Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We expand Appendix G to clarify MemTrace-OBS starting-point initialization and add an early-stopping chunked baseline with complete accuracy and cost results (Table 6). We also added a comparison with CHIEF in Appendix B, explaining its graph-construction cost and structural mismatch with long-term memory systems.
  • Presentation and Benchmark Statistics (Reviewer pvjo\textcolor{orange}{\mathrm{Reviewer \space pvjo}})
    • We move all table captions above their tables and adde exact system-by-dataset counts for both the collected failure pool and final benchmark in Appendix D.2 (Table 5).

We thank all reviewers again for helping us substantially strengthen the empirical evaluation, annotation validation, analysis, and presentation of the paper.

A revised draft of our manuscript has been uploaded, with added content marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.