- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CausalCoT to address the unreliability of reasoning traces by decomposing the input into a DAG of sub-questions, each solved locally using only its direct causal dependencies, thereby preventing superficial justifications in traditional chain-of-thought prompting.
Critical strengths
Unfaithful reasoning is a meaningful problem that the paper tries to solve. The method is tested on both open-source and closed-source models of different sizes. The paper presents 3 well-motivated research questions and reports results honestly.
Critical weaknesses
The scope is narrow, as the input needs to be decomposable into a DAG. Building a correct DAG also requires substantial iterative attempts with the LLM, and malformed DAGs can lead to degradation in performance
Performance is weak, as some questions might require the full context to answer and CausalCoT only focuses on local reasoning. The claim that multiple attempts might improve CausalCoT's performance is reasonable, but it still underperforms existing methods while introducing substantial computational overhead.
The faithfulness measurement based on answer changes and prediction changes is not fully reliable. Following the injected text does not necessarily mean the reasoning is faithful, it only shows that the model is responsive to the given text rather than its own reasoning. It also penalizes self-correction, where the model tries to correct the injected error.
Factored Decomp works fine and might achieve a more significant accuracy improvement than CausalCoT when increasing the number of interventions from 0 to 3 on MedQA and Counterfactuals.
There is no comparison of computational overhead or latency, which raises questions about the practical use of the method.
Questions for authors
Do you have a reliable way to construct a DAG from the given input?
Comparison of computation and latency could help a lot in understanding the trade-off between reliability and cost.
Thank you for recognising the importance of the problem, the well-motivated research questions, the breadth of the evaluation and the honest reporting.
Requiring a DAG does not narrow the scope. Standard CoT is itself a DAG in which every step can access the input and all preceding steps; CausalCoT restricts which dependencies are allowed, not which problems can be posed. A step that needs input information receives it from a root node through an explicit edge. We evaluate on discriminative reasoning, following standard practice (Lanham et al., 2023), since faithfulness is measurable there through answer changes. The method targets settings where a model must justify a decision (e.g., clinical diagnosis, legal eligibility), where post-hoc rationalisation matters.
The graph is built in one zero-shot call, with no iteration. Our wording caused this confusion. The same prompt is used for all models and datasets, and Acc in Table 1 is single-attempt. Pass@3 asks whether any of three independent end-to-end runs succeeds; it is computed for every method and is not part of CausalCoT. On it, CausalCoT matches or exceeds CoT on all seven models. We will replace "repeated structural attempts" in Section 5.1 with "independently sampled runs". Malformed graphs average below 0.3% for six models (3.81% for Nano) and are already counted as errors (Table 6). Imperfect plans do reduce accuracy: in 360 analysed failures (H.9) the main factors are missing model knowledge and sub-questions omitting discriminating information; an incorrect node is visible in 354. On your question: structural validity is checked deterministically; plan quality comes from a prompt and has no formal guarantee, as in CoT, least-to-most and Factored Decomposition (FD). Iterative refinement or planner fine-tuning (Appendix E) could improve accuracy at extra cost; our aim is to show that zero-shot planning already suffices to add faithfulness (Table 2).
There is an accuracy cost, a trade-off commonly accepted in the literature for gains in interpretability and safety. The average loss is 1.05-2.60 points for the four stronger models and 4.78-6.38 for the others. Precedents are concept bottleneck models (Koh et al., 2020) and our baseline FD, which also trades accuracy for faithfulness (Radhakrishnan et al., 2023) and is below CoT on every model in Table 1. The gap comes mainly from plan quality, not missing context. With graphs fixed, restoring context access recovers only 2-4.5 point gap for the weaker models (Table 10), but collapses faithfulness (Mini: Answer Change 58.7% to 37.1%, vs 41.0% for CoT; Table 9). A stronger planner's graphs recover more with isolation intact: Gemma 4B on MedQA rises from 44.7% to 54.7% (Table 7), above its CoT accuracy of 49.0%. The trade improves with capability: on GPT-5.4, a 2.2-point loss comes with Answer Change rising from 27% to 48%.
Overall CausalCoT gains more from corrections than FD, and FD's gains support our thesis. You are right that FD gains more in three of six Counterfactuals settings and two of six on MedQA; we will correct H.1, which suggests otherwise. Across the 18 settings with FD results, CausalCoT gains more in 11 and FD in 6, averaging +22.2 points after three corrections against +18.2. FD's gains are expected: it shares decomposition and restricted context, the ingredients introduced to improve faithfulness, and our results show they also make a method respond to corrections. CoT largely does not for stronger models (GPT-5.4 on Counterfactuals: 50.0% to 51.5%).
Answer Change is the standard metric in this literature (Lanham et al., 2023); its limits are why we proposed Predictable Change. We agree that sensitivity to an edited step is necessary, not sufficient, for faithfulness; Predictable Change adds a check that the answer moves in the direction the edit implies. On self-correction: in an audit of 270 unchanged CoT answers, 20 (7%) contained an explicit, supported correction of the edit; crediting them leaves CausalCoT ahead on average for all three audited models (new appendix). The correction experiments are free of this issue, since ignoring a correct edit is not self-correction, and there CausalCoT gains more than CoT in all 21 settings (Tables 3-5). H.5 further tests standard CoT without injecting an error: the model is given its own unedited reasoning for one input together with a minimally different input, and for GPT-5.4 the final answer follows the input, not the reasoning, in 47% of conflict cases, where by definition CausalCoT scores 0%.
Overhead is already reported in Appendix H.11; we will move the numbers to the limitations. On 50 MedQA questions (GPT-5.4 Mini), CoT takes 1.31 minutes and 50 calls, CausalCoT 3.43 minutes and 237 calls, FD 3.77 minutes and 300 calls. The overhead relative to CoT is real (2.6x runtime) and lower than FD's. Independent nodes can run in parallel, so latency scales with graph depth.