- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates error propagation in long-term memory-augmented LLMs. It identifies two propagation pathways: memory updating and question feedback. For each question, the authors construct closely matched correct and erroneous responses and use them to generate four counterfactual trajectories: fully clean, memory-only contamination, question-only contamination, and joint contamination. Based on this framework, the paper introduces four error measures—memory-state retention, natural-response errors, controlled-probe errors, and probability-level error preference—and quantifies several causal effects, including the total propagation effect (TPE) and pathway-specific effects.
Critical strengths
- The paper develops a causal model of error propagation in LLMs and constructs four counterfactual trajectories to quantify multiple causal effects.
- The findings are important such as errors may re-emerge under targeted probing even when absent from natural responses, and generation probabilities may shift toward erroneous answers despite correct final outputs. The observed model-specific susceptibility to different error categories is also informative.
Critical weaknesses
The generalizability of the conclusions is limited for the following reasons:
- The dataset contains only 144 synthetic multi-turn interactions, with 24 instances per category.
- The evaluation covers only two 7B models. It remains unclear whether stronger open- or closed-source models, particularly those with self-reflection or verification mechanisms, would exhibit similar memory-error dynamics.
- Differences between synthetic and real-world interactions, such as user language and behavior, further limit external validity. Moreover, Qwen2.5-32B is used for data generation, while one evaluated response model belongs to the same family, potentially introducing shared model-family biases.
Similarly, although memory repair and joint repair reduce the residual effect by 70% and 98.5%, respectively, these results are obtained entirely from synthetic data and have not been validated in real-world settings.
Questions for authors
See Weaknesses.
4 comments
Response to W3: Real User Histories and Model Diversity
Thank you for your suggestion. We have added experiments using real user conversation histories and response models from different families to examine whether error propagation also occurs in these settings.
First, we completed experiments based on MemUse, comprising 50 dialogue cases from 19 real users. Each case starts from a real conversation history and its corresponding user question, followed by controlled error injection and 12 turns of simulated interaction. Both Qwen2.5-7B and OLMo-3-7B completed all four counterfactual trajectories for every case. This introduces real user language and historical context while maintaining comparable experimental conditions.
Second, OLMo-3-7B, already included in the paper, belongs to a different model family from Qwen. We have also added gpt-oss-120b from another model family, using medium reasoning effort. Keeping the original question-generation and evaluation settings unchanged, we completed all four counterfactual trajectories for all 144 candidate cases.
The natural-response results from these additional experiments are summarized below:
| Dataset | Response model | Number of cases | Natural-response TPE [95% CI] |
|---|---|---|---|
| MemUse real user histories | Qwen2.5-7B | 50 | 4.00 [0.30, 9.35] |
| MemUse real user histories | OLMo-3-7B | 50 | 5.67 [2.54, 9.62] |
| Original controlled cases | gpt-oss-120b | 144 | 15.74 [11.86, 19.85] |
Values represent the total propagation effect averaged over post-injection turns 1–12, expressed in percentage points. They measure the change in the proportion of responses containing the target error between the trajectory with errors introduced through both pathways and the clean trajectory. All reported confidence intervals are entirely above zero.
The MemUse results show that injected errors can influence subsequent responses when interactions begin from real user histories. The OLMo and gpt-oss results further show that this phenomenon also occurs in response models outside the Qwen family. We also compared the effects of the two pathways. The average pathway effects for gpt-oss-120b over post-injection turns 1–12 are reported below:
| Metrics | Memory-update pathway effect (M) | Question-feedback pathway effect (Q) | Difference (M − Q) [95% CI] |
|---|---|---|---|
| Natural-response errors | 7.29 | 7.75 | −0.46 [−4.17, 3.30] |
| Memory-state errors | 41.32 | 4.69 | 36.63 [28.65, 44.91] |
| Controlled-probe errors | 25.69 | 3.99 | 21.70 [15.39, 28.01] |
| Probability-level error preference | 0.368 | 0.025 | 0.344 [0.099, 0.597] |
The first three metrics are reported in percentage points, while the final metric is measured in bits/token. For gpt-oss, the memory-update pathway has a larger effect than the question-feedback pathway on memory-state errors, controlled-probe errors, and probability-level error preference. For example, the controlled-probe effect is 25.69 percentage points for the memory-update pathway, compared with 3.99 percentage points for the question-feedback pathway. This supports the original observation that the memory-update pathway has a greater influence on error retention and subsequent probe responses.
These additional experiments extend the paper’s evidence across both data sources and model families. The MemUse results show that error propagation through the two pathways remains observable when using real users’ conversation histories and questions. The OLMo and gpt-oss results show that this phenomenon also occurs in response models outside the Qwen family. Furthermore, the pathway analysis for gpt-oss supports a larger contribution from the memory-update pathway on memory-state, controlled-probe, and probability-level error preference outcomes. We will include these experiments and their complete results in the revised paper to show how the original findings are supported in settings grounded in real user histories and across different model families.
Response to the Concern about Real-World Validation of Repair Results
Thank you for your comment on the real-world applicability of our repair results. To address this concern, we added restoration experiments based on real user histories from MemUse, using 50 dialogue cases from 19 users, and evaluated them on Qwen2.5-7B and OLMo-3-7B. Each case starts from a real dialogue history and user question, undergoes controlled error injection, and then continues with model-generated dialogue. After the end of turn 5, we compare four conditions: No Repair, Question Repair, Memory Repair, and Joint Repair. We then examine performance over turns 6–15 across all 50 cases.
We use the same probe questions to check whether responses under different repair conditions contain the target error and compare them with the clean reference without error injection. A lower MRE indicates that the results after repair are closer to the clean reference.
| Response model | Repair method | No Repair MRE | MRE after repair | MRE reduction [95% CI] |
|---|---|---|---|---|
| Qwen2.5-7B | Question Repair | 0.130 | 0.106 | 18.5% [−11.3%, 46.7%] |
| Qwen2.5-7B | Memory Repair | 0.130 | 0.056 | 56.9% [19.6%, 84.0%] |
| Qwen2.5-7B | Joint Repair | 0.130 | 0.044 | 66.2% [26.7%, 86.6%] |
| OLMo-3-7B | Question Repair | 0.116 | 0.094 | 19.0% [−18.2%, 44.6%] |
| OLMo-3-7B | Memory Repair | 0.116 | 0.096 | 17.2% [−23.3%, 44.9%] |
| OLMo-3-7B | Joint Repair | 0.116 | 0.076 | 34.5% [12.5%, 54.0%] |
All percentage reductions are calculated relative to No Repair. Confidence intervals are computed using 5,000 bootstrap resamples clustered by user.
On Qwen, Memory Repair and Joint Repair reduce MRE by 56.9% and 66.2%, respectively, with confidence intervals supporting improvements in both cases. A direct comparison also shows that Memory Repair outperforms Question Repair. On OLMo, Joint Repair reduces MRE by 34.5%, also supported by the confidence interval.
These additional results show that improvements from repair can also be observed with real user histories, while the benefits vary across models. We will include these experiments and the complete results in the revised manuscript and describe the specific settings for the real user data and model-generated interactions.
Thank you for taking the time to read our paper closely and share your comments. These improvements have strengthened the credibility of our work. We value this exchange and welcome any further questions you may have. We sincerely wish you all the best!
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end, with less than two days remaining, we wanted to ensure that we have addressed all your concerns satisfactorily. If there are any additional points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Response to W1: Dataset Size and Additional MemUse Experiments & Response to W2: Larger Models with Reasoning and Reflection
Response to W1: Dataset Size and Additional MemUse Experiments
Thank you for raising the question of dataset size. The 144 cases form the main balanced benchmark, with 24 cases per category. Section 5.3 and Appendix A.3 also report 84 independently constructed additional candidates, bringing the total to 228 candidates and the valid cohorts to 197 Qwen cases and 185 OLMo cases. The expanded cohorts retain the main error-propagation trends.
To extend our evaluation to settings grounded in real conversations, we added experiments using MemUse [1]. We selected 55 candidate cases with sufficient dialogue context and official fact annotations. Target facts were verified against users’ original utterances to construct correct and erroneous responses differing only in the target fact. Ultimately, 50 cases from 19 users yielded validated response pairs and were included in subsequent experiments.
Starting from these real conversation histories and user questions, we ran experiments with Qwen2.5-7B and OLMo3-7B. Each case begins with two shared dialogue turns. At turn 3, we introduce matched correct or erroneous responses, then continue each of the four counterfactual trajectories for 12 additional turns. Across both models, this yields 4,800 post-intervention interaction turns. Subsequent questions are generated dynamically based on the model’s preceding response, rather than being scripted in advance.
At each turn, we assess whether the target error appears in responses and memory states, as well as the model’s probability-level error preference. Detailed results for these experiments are provided in our response to W3.
Response to W2: Larger Models with Reasoning and Reflection
Thank you for this suggestion. We have expanded the evaluation beyond the original two 7B models to larger models and configurations with reasoning and explicit reflection. We added four experimental settings: gpt-oss-120b with medium reasoning effort, and Qwen3-32B in three configurations—non-thinking, thinking, and thinking with explicit reflection.
All four settings use the original 144 candidate cases, the MemoryBank mechanism, error injection at turn 3, and four counterfactual trajectories, followed for the full 12 post-injection turns. Qwen2.5-32B-Instruct is used consistently for subsequent question generation and proposition-level evaluation. The primary analysis includes all candidate cases, including those in which the target error was not successfully written into memory, and uses 5,000 case-level bootstrap resamples. Comparisons across configurations are paired by candidate case.
The additional results show that error propagation is not limited to smaller models: it also occurs in larger models and configurations with reasoning and reflection. Across all four settings, the natural-response total propagation effect (TPE), averaged over the 12 post-injection turns, is positive:
Across all four settings, the natural-response TPE averaged over post-injection turns 2–12 remains positive, with 95% CIs above zero, showing propagation beyond the first turn. Memory-update effects exceed question-feedback effects on memory-state and controlled-probe metrics in all four settings, with all 95% CIs for the differences above zero. With explicit reflection, the natural-response memory-update and question-feedback effects averaged over post-injection turns 1–12 are 14.06 and 7.29 percentage points, respectively. The difference is 6.77 percentage points [95% CI: 2.84, 10.88]. These findings extend the analysis to reasoning and reflection.
Compared with thinking alone, adding explicit reflection reduces the first-attempt error-writing rate from 54.86% to 18.75%, a decrease of 36.11 percentage points [95% CI: 26.39, 45.83]. With up to five attempts, the final valid injection rate decreases from 64.58% to 48.61%, a reduction of 15.97 percentage points [95% CI: 5.56, 26.39]. However, a clear positive propagation effect remains in subsequent interactions. Thus, the tested reflection configuration reduces erroneous memory writes without eliminating downstream propagation, highlighting the need to evaluate these separately and track both the memory-update and question-feedback pathways.
[1] Ryuichi Sumida, Koji Inoue, and Tatsuya Kawahara. 2026. MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 18719–18749.