- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces CoT-Interpretability Alignment (CIA), a metric that measures how closely an LLM's verbalized chain-of-thought reasoning matches its internal computational strategies, using interpretability techniques such as linear probes. Experiments done on three reasoning tasks and three language models reveal substantial differences between the verbalized and inferred internal reasoning of the LLM, but show that post-training with rejection sampling, DPO, or GRPO can improve alignment. Further experiments suggest that these improvements arise through two mechanisms: models either become better at reporting their existing internal computations or modify their computations to follow their verbalized reasoning steps better.
Critical strengths
- CIA provides a helpful framework for measuring and improving reasoning faithfulness by connecting interpretability techniques with post-training objectives
- Experiments across three model families, three reasoning tasks, and three post-training approaches demonstrate substantial improvements in CIA while largely preserving task accuracy
- The authors conduct causal intervention experiments that complement the probing results, providing better evidence that observed improvements are a result of genuine changes in model behavior
Critical weaknesses
- Experiments primarily focus on relatively small models and simple/highly structured reasoning tasks, leaving generalization to frontier reasoning models and complex, long-horizon reasoning unclear
Questions for authors
- Would it be possible to evaluate if the method works on larger models too?
- Can you provide a more detailed assessment on reasoning tasks that have more complex and long horizon nature?
2 comments
Thank you for the clarification and follow-up. I maintain my positive score.
Response to Reviewer Q55Z
We thank the reviewer for the positive assessment, and for noting that CIA “provides a helpful framework for measuring and improving reasoning faithfulness” and that the causal interventions “complement the probing results”.
Yes. We have provided experiments in Appendix E: “Extending to Larger and Reasoning Models”, which evaluates Qwen3-14B on 2-digit multiplication with the same pipeline. For the rebuttal, we also evaluated Qwen3-14B on MMLU-Hint: it follows the misleading hint less often than Qwen3-8B (13.4% vs. 17.8%) and is more accurate (0.847 vs. 0.808).
We have provided experimental evidence on Reasoning Models in Appendix E, where Qwen3-8B in thinking mode and DeepSeek-R1-Distill-Llama-8B write a long thinking trace (up to 8,192 new tokens, compared with 512 in the main experiments) in the MMLU-Hint task. Our method also works on these long traces. After we retrain the probe on thinking-mode outputs, it still detects whether the hint drove the answer (macro-F1 0.894 for Qwen3-8B and 0.858 for DeepSeek-R1-Distill).
And the method of training with the reward of CIA still helps: CIA rises by +0.090 (RS) and +0.083 (DPO) for Qwen3-8B and by +0.078 (RS) for R1-Distill, while DPO does not change it significantly for R1-Distill. Each example is still a single multiple-choice question in this task. For tasks with many dependent steps, the same corruption test can be applied step by step: we change one written intermediate result at a time and check whether the answer follows.