- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Disagreement Graph Inference, a training-free framework that improves upon Self-Consistency by incorporating intermediate reasoning information from multiple Chain-of-Thought samples. The method extracts and aligns intermediate sub-answers across reasoning chains, estimates their statistical dependencies using mutual information, and constructs a Chow–Liu tree to represent these relationships. It then applies belief propagation to compute the marginal distribution of final answers, aiming to leverage intermediate consensus to correct erroneous majority-vote predictions. Experiments across six reasoning benchmarks and three language models report improvements over Self-Consistency and other aggregation baselines, along with gains in calibration and limited additional inference overhead.
Critical strengths
-
This paper tries to tackle a limitation of self-consistency. The idea that uses the middle reasoning steps is meaningful and has potential.
-
This paper reports experiments across six reasoning benchmarks and three LLMs, including one closed-source LLM, and compares against multiple aggregation baselines. Besides, the results show that the proposed method shows consistent improvement.
Critical weaknesses
-
The proposed rescue effect seems weird in theory. According to the formulation in Eq. (6)–(9), DGI constructs a Chow–Liu tree from empirical distributions and applies belief propagation to compute the marginal probability of the final answer. However, under the standard Chow–Liu factorization with consistent empirical distributions, marginalizing over the intermediate variables should recover exactly the original final-answer distribution used by Self-Consistency. In other words, DGI should produce the same prediction as SC with the current rescue framework written in the paper.
-
The central idea of leveraging intermediate reasoning consensus and graph structures across multiple CoT trajectories has been explored in prior work [1,2,3,4]. Though the proposed methods are different, this paper should compare with the existing works to show the differences.
[1] Liu, Xiaoou, et al. Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution. ICML 2026.
[2] Cao L. Graphreason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. ACL Workshop 2024
[3] Shi H, Liu Y, Pang B, et al. Ssr: Socratic self-refine for large language model reasoning. Arxiv 2025.
[4] Parulekar A, Lee J, Hakkani-Tür D, et al. Reasoning consensus: Structural ensembling of LLM reasoning via weighted DAG aggregation. arxiv 2026.
Questions for authors
-
Could the authors address the problem mentioned above?
-
Since the middle steps were already extracted by the LLM, why is cosine similarity required?
-
Could the authors provide the computational overhead for pilot chain generation, sub-question enumeration, and potentially LLM-based sub-answer extraction for each reasoning chain?
-
Could the authors improve the quality of Figure 1?