- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a unified information-theoretic framework for evaluating chain-of-thought reasoning along two dimensions: faithfulness and efficiency. It represents a CoT as a semantic reasoning graph, measures whether the model’s predictive dependence is explained by the graph’s semantic support, and distinguishes dependence allocated to answer-reaching versus off-path reasoning. The resulting global and step-level metrics are validated through controlled interventions, analyzed in relation to uncertainty and correctness, and further used to construct graph-aware RFT objectives intended to improve reasoning faithfulness and efficiency.
Critical strengths
- This paper proposes a unified framework.
- It discusses a important issue.
- A new method is proposed to solve the problem.
Critical weaknesses
-
Limited scientific and mechanistic insight. The paper mainly defines quantitative measures of faithfulness and efficiency and studies their associations with uncertainty and correctness, but provides little explanation of why unfaithful or inefficient reasoning arises or what internal mechanisms produce these behaviors.
-
The technical novelty appears limited and lies mainly in integration and reformulation. Semantic dependency graphs, CoT faithfulness analysis, behavioral dependence measures, redundant-step detection, reasoning pruning, and faithfulness-oriented fine-tuning have all been explored in prior work. The main contribution is to combine these ingredients into a unified two-dimensional decomposition rather than introduce a fundamentally new reasoning mechanism or learning method.
-
Empirical generalization is limited. The evaluation uses only two Qwen3 variants from the same model family and three relatively structured reasoning benchmarks. It remains unclear whether the proposed metrics and observed relationships generalize to other reasoning model families, frontier systems, or more diverse settings such as coding, planning, and agentic tasks.
-
The framework critically depends on an LLM-generated semantic reasoning graph whose reliability is insufficiently validated. The semantic graph determines which dependencies are considered faithful, answer-reaching, or off-path, yet it is extracted by a single external LLM. The paper does not provide strong human validation, inter-annotator agreement, or robustness analysis across alternative graph extractors, making it difficult to separate properties of the target model from judgments introduced by the graph annotator.
-
The framework is evaluated on heavily structured CoTs rather than natural reasoning traces. Models are required to produce explicitly segmented reasoning outputs, which greatly simplifies graph construction. It is unclear whether the same semantic graph extraction and metrics remain reliable for naturally generated free-form CoTs containing implicit dependencies, self-corrections, or more complex reasoning structure.
Questions for authors
-
What substantive insight does the proposed framework provide about why unfaithful or inefficient reasoning arises, beyond measuring these behaviors?
-
What fundamentally new capability does the two-dimensional formulation enable compared with existing work on CoT faithfulness, dependency graphs, behavioral importance, and reasoning pruning?
-
Do the proposed metrics and empirical findings generalize beyond Qwen3 to other model families, frontier reasoning models, or more diverse reasoning tasks?
-
How robust are the results to errors or variation in the semantic reasoning graph? In particular, would the conclusions remain stable with human annotations or alternative graph-extraction models?
-
How well does the framework extend to naturally generated free-form CoTs without the imposed reasoning structure?
2 comments
Keep my score
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. However, I still have concerns about the validity of the extracted semantic graphs and the limited improvements achieved by faithfulness-aware RFT. While the rebuttal addresses several of my concerns, I remain unconvinced about the overall contribution and maintain my score.
Reply to Reviewer Kdyb
Dear Reviewer Kdyb,
Thank you very much for your time and detailed comments. We address your concerns by clarifying the contributions of our work and several potential misunderstandings.
[Weakness#1&2 & Question#1&2. The scope, novelty, and key contributions of our framework.] We would like to clarify the scope of our work. Explaining “why unfaithful or inefficient reasoning arises or what internal mechanisms produce these behaviors,” or introducing a “fundamentally new reasoning mechanism or learning method,” is beyond the stated objectives of this paper. Our focus is to introduce a novel framework for jointly quantifying these properties and demonstrate its reliability and usefulness, which constitutes a research contribution in its own right, as reflected in most of the literature reviewed in Section 2 and Appendix C.
More importantly, we disagree with your comment that our work lacks novelty. Our contribution is not merely to “combine these ingredients into a unified two-dimensional decomposition.” Our mutual information decomposition, grounded in a semantic reasoning graph, provides conceptually and methodologically distinct formulations of both CoT faithfulness and efficiency relative to prior approaches. Please refer to our General Response for details.
[Weakness#3 & Question#3. Experiments covering a wider range of models and tasks.] We thank the reviewer for suggesting experiments across more model–task combinations. Our expanded experiments span MATH, Omni-MATH-2, and 2WikiMultiHopQA, and include Qwen3-30B-A3B-Instruct-2507, Qwen3-4B-Instruct-2507, Ministral-3-8B-Instruct-2512, Ministral-3-3B-Instruct-2512, and Llama-3.1-8B-Instruct.
Given the space constraints, we refer the reviewer to the updated Tables 3–4 and Figure 2, and especially the expanded experimental results in Appendix D.6, Tables 30–52 of the revised manuscript. Overall, these additional results across these diverse settings support our main findings and conclusions from the original submission:
Please refer to the above tables for detailed numerical results.
[Weakness#4 & Question#4. Robustness to graph extraction.] We thank the reviewer for raising this question. We have added Appendix B.2 to the revised manuscript to examine the sensitivity of our framework to the semantic graph extractor. For Qwen3-30B-A3B-Instruct-2507 on MATH, in addition to DeepSeek V4 Pro, we use GPT-6.1 Terra and Gemini 3.8 Flash to extract graphs from the same responses using the same extraction prompt. Equation (4) defines edge F1 and Jaccard similarity, and Table 12 reports values around 0.9 for both measures, which indicates high agreement across extractors. Table 13 further shows that unfaithfulness and inefficiency scores remain broadly comparable across the three graph annotations. More importantly, Tables 14–20 report the corresponding analysis from Sections 4 and 5 using GPT-6.1 Terra, and the results support all of our main conclusions. Together, these results support the robustness of our framework to the choice of semantic graph extraction model.
[Weakness#5 & Question#5. Extension of our framework to free-form CoTs.] We have further added Appendix B.3 to examine free-form CoTs. In this setting, the model is not required to generate numbered steps or separate Subgoal, Reasoning, and Result fields. Our framework accommodates this setting through a separate annotation stage, in which DeepSeek V4 Pro segments the generated reasoning into steps. Tables 21–27 report the corresponding analyses from Sections 4 and 5 using Qwen3-30B-A3B-Instruct-2507 on MATH, and the results support all of our main conclusions from the original submission. Together, these results show that our framework extends naturally to free-form CoTs and that our main conclusions remain robust in this setting.
We are happy to answer any further questions you may have. If our responses above have addressed your concerns, we would truly appreciate a re-evaluation accordingly.