- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes evaluating chain-of-thought faithfulness against an exact computation DAG generated by a predefined solver. It introduces Order Fidelity, Path-Monotonic Fidelity, and Propagation Fidelity, and evaluates them on five synthetic computational task families and seven open-weight LLMs. The results suggest that models may react to injected intermediate errors while allowing those errors to propagate beyond the nodes that should depend on them.
Critical strengths
- The paper clearly formalizes several graph-based metrics for evaluating CoT faithfulness.
- Using solver-generated computation graphs provides a more explicit reference than comparing a CoT only with modified versions of itself.
- The paper includes multiple task families, models, graph statistics, and controlled corruption experiments.
Critical weaknesses
-
Limited novelty. The central message—that a correct final answer does not necessarily imply a faithful CoT—is important, but has been widely discussed in previous work. The graph formulation is a useful reframing, yet the conceptual advance over existing faithfulness studies appears limited.
-
Potential false positives from a single reference graph. If a problem admits multiple valid solution strategies, the model may produce a coherent CoT corresponding to computation graph A, while the paper evaluates it against solver graph B. Such a CoT could be incorrectly classified as unfaithful simply because it does not match the authors’ chosen implementation.
-
The empirical evaluation is not sufficiently comprehensive. The DTMI experiments are limited to two task families and three models, while the benchmark consists largely of synthetic computational problems under greedy decoding. The paper does not adequately evaluate alternative valid solvers, broader reasoning tasks, or stronger recent faithfulness baselines. In addition, several figures and tables are visually dense and could be presented more clearly.
Questions for authors
same as weakness