- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates whether repeated errors in multi-turn LLM interactions stem from causal error propagation or shared latent question difficulty. The authors propose a randomized shared-prefix intervention and the Causal Self-Excitation Model (CSEM) to estimate a lag-dependent causal error kernel, observing that causal self-excitation diminishes with model scale. Furthermore, the paper combine CSEM estimates as empirical Bayes priors with a dynamic AR(1) latent state-space model to forecast multi-turn recurrent failures, evaluated across 50 open-source LLMs on multiple QA benchmarks.
Critical strengths
- The paper investigates error persistence in multi-turn LLM interactions, attempting to separate causal self-excitation from unobserved latent heterogeneity.
- The experimental evaluation covers 50 open-source LLMs across 56 inference configurations on three multiple-choice QA benchmarks, reporting empirical results across model scales and reasoning modes.
- The paper formulates a discrete-time recurrent survival model (CAR-Surv) using Polya-Gamma augmentation and FFBS sampling for Bayesian posterior inference.
Critical weaknesses
- The paper simulates errors by forcing an incorrect response at the intervention turn. However, artificially injecting an error acts as an exogenous prefix perturbation that alters the model's natural reasoning trajectory, which differs fundamentally from spontaneous errors caused by hallucinations or flawed reasoning. Can artificially induced errors truly represent the spontaneous failures?
- The evaluation is confined to choice questions with fixed option sets. Under adversarial follow-ups, the model naturally flips or retreats to alternative options, which seems to reflect option elimination or compliance rather than genuine cognitive self-correction. The paper lacks justification to clarify whether this reflects model capability, knowledge boundaries, or behavioral alignment.
- The phenomenon that an earlier error persistently influences subsequent responses is interesting. However, it is validated only on choice questions. Can the existence of this causal kernel be verified in open-ended generation tasks like multi-step mathematical reasoning or code generation?
- Table 2 shows that the survival-only variant (using no intervention data) has already achieved the vast majority of the performance gains. In Table 4, adding CSEM causal anchoring improves Recurrent CI by only 0.002, leaves NLL unchanged, and slightly degrades cumulative error MAE (1.453 vs. 1.454). It need more evidence to determine whether CSEM is truly useful.
Questions for authors
Please see the Weaknesses.
2 comments
R3 [W4]: We agree that the original results did not adequately establish the incremental value of CSEM-derived causal information. As also highlighted by Reviewer 1PRj, our original coefficient-level anchoring directly centered CAR-Surv's conditional lag coefficients on CSEM's marginal causal-effect coefficients, despite their different interpretations. Consequently, the original anchoring mechanism may have functioned primarily as an empirical regularizer rather than effectively transferring the causal information estimated by CSEM.
We have addressed this limitation by replacing the original coefficient-level prior with ATE-aligned Causal Anchoring in Section 3.2. Instead of directly matching coefficients, the updated formulation aligns the model-implied marginal ATEs of CAR-Surv, computed through recursive g-computation, with the intervention-derived ATE estimates from CSEM. This aligns the two models at the level of the causal estimand while allowing their conditional parameters to differ. We have rerun all CAR-Surv experiments using this updated formulation.
As shown in Rebuttal Table 10 [https://anonymous.4open.science/r/SurvLLM/Table10.png ], the updated CAR-Surv now improves all seven prediction metrics over the survival-only variant. In particular, Antolini's CI increases from to , recurrent-event CI increases from to , cumulative error MAE decreases from to , and NLL decreases from to .
To further distinguish the contribution of causal information from the regularization effect of anchoring, we introduced a non-causal prior with matched concentration, centered on survival-only model-implied ATEs rather than CSEM-derived estimates. As shown in Rebuttal Table 3 [https://anonymous.4open.science/r/SurvLLM/Table3.png ], CAR-Surv outperforms this non-causal control across all seven metrics. Furthermore, paired bootstrap analysis (), reported in Rebuttal Table 4 [https://anonymous.4open.science/r/SurvLLM/Table4.png ], shows that five of the seven improvements over both the non-causal prior and survival-only have 95% confidence intervals excluding zero. The improvements in recurrent CI and NLL are positive on average, although their confidence intervals include zero.
These results indicate that the limited gains observed in the original submission were partly attributable to the mismatch in the anchoring formulation. The updated results provide stronger evidence that appropriately incorporating CSEM-derived causal information offers predictive benefits beyond non-causal regularization. We have revised Section 3.2 and incorporated the additional ablations and paired uncertainty analyses into the updated manuscript.
Thank you for your careful review and constructive feedback. We greatly appreciate your thoughtful comments and suggestions, which have helped us further clarify and strengthen our work. We will provide point-by-point responses below.
R1 [W1]: Our randomized shared-prefix intervention is designed to identify the causal effect of a committed error by comparing forced-correct and forced-incorrect responses under identical preceding conversational histories and downstream user inputs, rather than merely observing correlations between consecutive errors.
We acknowledge that artificially injected errors may not fully represent spontaneous failures arising from hallucinations or flawed reasoning, as the latter may involve additional internal processes not captured by our intervention protocol.
We have explicitly discussed this limitation in the revised manuscript.
R2 [W2&W3]: We greatly appreciate your insightful comments regarding whether error persistence and self-correction in multiple-choice settings may reflect answer switching among predefined options rather than genuine error propagation. We therefore extended our evaluation to open-ended mathematical reasoning and code generation, using GSM8K and MATH for mathematical reasoning and MBPP and HumanEval for code generation. Unlike multiple-choice interactions, paired interventions use the models' own generated correct and incorrect complete solutions, with identical downstream user inputs across both branches. Correctness is verified through numerical answer matching or executable unit tests, eliminating dependence on predefined answer options.
As shown in Rebuttal Table 7 [https://anonymous.4open.science/r/SurvLLM/Table7.png ], the one-step causal effects remain substantial across three models, ranging from 0.163 to 0.250 for mathematical reasoning and from 0.152 to 0.286 for code generation. These effects persist at longer lags, and CSEM closely matches the direct paired ATE estimates, with MAE ranging from 0.002 to 0.003. Furthermore, Rebuttal Tables 8 [https://anonymous.4open.science/r/SurvLLM/Table8.png ] and 9 [https://anonymous.4open.science/r/SurvLLM/Table9.png ] show that CAR-Surv achieves the best mean performance across all seven prediction metrics on both tasks, further supporting the applicability of our framework beyond multiple-choice benchmarks.
These findings demonstrate that causal error propagation is not confined to multiple-choice interactions and cannot be explained solely by switching among predefined options. Nevertheless, our experiments do not isolate the relative contributions of reasoning capability, knowledge uncertainty, and behavioral alignment, nor do they establish a specific internal mechanism of self-correction. Our framework characterizes the causal influence of previously committed errors on subsequent responses rather than attributing that influence to a particular cognitive process.
We have incorporated these open-ended experiments and further discussed the generalizability and limitations of our framework in Case Study IV in Appendix D.6 of the updated manuscript.