- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes RECCON (Reasoning-Chain Consistency), a consistency-based framework for mitigating LLM hallucinations. The authors identify limitations in two existing approaches: self-consistency methods aggregate multiple sampled answers but may overlook inconsistencies at individual reasoning steps, while belief-tree-based methods analyze atomic statements but are argued to have limited capacity to capture complex logical dependencies. RECCON aims to bridge these limitations through a four-step pipeline: sampling, contrasting, verifying, and reweighting.
Critical strengths
-
Relevant and important problem. The paper tries to addresses LLM hallucination where individual reasoning steps may contain errors that are overlooked by answer-level aggregation.
-
Intuitive overall framework. The four-step pipeline of sampling, contrasting, verifying, and reweighting provides a clear high-level workflow for exploiting inconsistencies across multiple reasoning chains. The paper is easy to follow.
-
Fine-grained credibility assessment. Rather than treating each sampled answer as an indivisible unit, RECCON attempts to assess the truthfulness of individual sub-claims and use this information to estimate answer credibility.
Critical weaknesses
-
The identification of inconsistencies is insufficiently justified. In the contrast step, an LLM-based inspection module identifies inconsistent sub-claims across sampled reasoning chains. However, the paper does not adequately clarify how these inconsistencies should be interpreted when all sampled chains contain the same erroneous claim, or when only a small subset of claims is incorrect while the majority are correct. This raises questions about the reliability and coverage of the proposed inconsistency identification mechanism.
-
The reliability of the LLM-based verifier remains questionable. In the verify step, the framework relies on an LLM-based module to assign truthfulness scores. However, the verifier itself may hallucinate, misinterpret retrieved evidence, or make incorrect judgments about the validity of a claim. The paper needs to clarify how verifier errors are controlled. Otherwise, the proposed framework may potentially introduces additional errors into the reweighting process.
-
The treatment of 'complex logical dependencies' is unclear. The paper claims that RECCON can accommodate 'complex' logical dependencies among reasoning steps. How does RECCON represent relationships between sub-claims, particularly when the validity of one step depends on multiple preceding steps? Alternatively, are these dependencies captured simply by retaining the original reasoning chains? If so, the main contribution appears to lie in leveraging reasoning chains rather than in the proposed sub-claim decomposition and verification, and the paper should clarify what methodological novelty goes beyond this design choice.
-
The novelty and complementarity of the framework are limited. The proposed workflow appears to be a combination of 'self-consistency' and 'belief-tree-based' methods. However, it remains unclear whether integrating these two approaches actually addresses their respective limitations, rather than simply combining their existing components.
Questions for authors
See Weaknesses
4 comments
[W3] Representing Logical Dependencies Between Sub-Claims.
By design, ReCCon does not construct an explicit formal dependency graph over sub-claims. Instead, logical dependencies are preserved by retaining each complete reasoning chain as the unit of evaluation and reweighting. Decomposing a complex multi-hop or counterfactual chain into isolated atomic claims often strips away critical context. For example, evaluating the consequent of a counterfactual statement in IfQA against real-world facts makes a logically valid deduction appear false out of context.
To avoid this pitfall, ReCCon only extracts disputed sub-claims to query external evidence, and then feeds the verified findings back into the context of the full reasoning chains so the judge can assess how each finding impacts the chain's overall conclusion. Empirical results strongly support this design. Atomic decomposition baselines suffer severe degradation on counterfactual reasoning, as seen on IfQA in Table 1. Two aggregation rules that ignore the dependency also do worse than the judge on the exact same chains and verification findings. A drop rule discards any chain containing a low-scoring flagged claim, and an average weighting weights each chain by the mean truthfulness score of its sub-claims:
[Table 2]. Aggregation methods using identical chains and verification findings. Divergence set, small setting, AUROC.
| Method | FactCHD | HoVer | FutureX | IfQA | Avg |
|---|---|---|---|---|---|
| RAG + SC | 61.2 | 66.0 | 55.8 | 73.4 | 64.1 |
| Drop rule | 64.8 | 64.6 | 60.3 | 72.2 | 65.5 |
| Average weighting | 64.4 | 67.7 | 57.4 | 74.1 | 65.9 |
| ReCCon (LLM judge) | 66.6 | 73.2 | 59.2 | 80.4 | 69.8 |
The drop rule and average weighting recover 1.4 and 1.8 of the 5.7 AUROC points gained over RAG + SC, and the judge adds the rest. In the large setting, the judge is 1.9 AUROC above the drop rule and 3.5 above average weighting (Table 4 of the revised paper).
[W4] Novelty and Complementarity of the Integrated Framework.
While sub-claim verification, self-consistency (SC), and belief trees are existing primitives, ReCCon's novelty lies in bridging answer-level aggregation and claim-level verification via cross-chain disagreement: it contrasts chains to selectively decide what to verify, and reweights complete chains based on whether their load-bearing premises survive verification.
This integration directly resolves the core limitation of each individual paradigm.
Overcoming the failure mode of self-consistency: the naive averaging of RAG + SC breaks down when sampled chains strongly disagree, as Table 2 of the paper shows. ReCCon resolves these disputes through targeted verification and reweighting, improving divergence-set AUROC by +5.7 and +5.3. Sampling more chains does not close this gap. We sampled six additional chains per statement, so that RAG + SC matched ReCCon's average output-token budget:
[Table 3]. Equal-budget comparison on the divergence set, small setting, AUROC.
| Method | FactCHD | HoVer | FutureX | IfQA | Avg |
|---|---|---|---|---|---|
| RAG + SC, 3 chains, Table 2 of the paper | 61.2 | 66.0 | 55.8 | 73.4 | 64.1 |
| RAG + SC at ReCCon's budget | 65.6 | 64.8 | 56.6 | 79.6 | 66.7 |
| ReCCon | 66.6 | 73.2 | 59.2 | 80.4 | 69.8 |
| ReCCon − RAG + SC at ReCCon's budget | +0.9 [−4.3, 6.1] | +8.4 [4.0, 13.0] | +2.6 [−4.4, 9.7] | +0.8 [−3.2, 4.7] | +3.2 [0.4, 5.8] |
At the same budget, ReCCon remains +3.2 AUROC ahead (95% CI [0.4, 5.8], p = 0.01).
Overcoming the context loss of belief trees and atomic verification: ReCCon keeps the full chain as the unit of reweighting, and [W3] shows with [Table 2] that rules ignoring the dependencies between a chain's claims do worse on the same findings.
Two further results confirm that this synergy goes well beyond a simple sum of parts. Cross-chain contrast makes verification significantly more accurate at less than half the retrieval cost, as shown in [Table 1]. The gains also persist as we scale the ensemble. We added a fourth chain sampled from gpt-oss, so that each model contributes two chains, and compared ReCCon with the average of the same four chains:
[Table 4]. AUROC gain of ReCCon over the 4-chain average, small setting.
| Set | FactCHD | HoVer | FutureX | IfQA | Avg |
|---|---|---|---|---|---|
| Divergence set | +1.9 | +8.2 | +0.7 | +4.4 | +3.8 [2.0, 5.7] |
| Full set | +0.3 | +1.2 | +0.3 | +0.6 | +0.6 [0.2, 1.0] |
Finally, ReCCon makes ensemble predictions transparent by returning the exact points of inter-chain contention, the retrieved evidence settling each dispute, and the resulting chain weights (Appendix H, Table 24 of the revised paper).
Thank you again for your valuable feedback. We hope these clarifications and new experiments address your concerns, and we are happy to discuss further!
Dear Reviewer sze7,
We have uploaded a revised version of the paper. The main results and the ablations (Tables 1–6) now come from the corrected implementation, and the revision includes the new experiments from our responses:
- Aggregation rules: Table 4 and Appendix G.9
- Verification with and without cross-chain contrast: Table 6 and Appendix G.10
- Equal-budget comparison: Table 5 and Appendix G.4
- Scaling to four chains: Appendix G.5
We have also updated our responses above to match the revised paper. Thank you again for your time and feedback.
Thanks for the rebuttal, I will keep my score.
We thank Reviewer sze7 for the thoughtful and constructive review! During the rebuttal period, we added a continuous deterministic weighting baseline using identical verification findings, an ablation quantifying how cross-chain contrast improves verification accuracy and search efficiency, and a scaling experiment with a fourth reasoning chain. Below, we explain how these results and ReCCon's design address each of your questions.
[W1] Handling Shared Errors Across All Chains and Minor Flaws Within Chains.
Thank you for raising these two critical edge cases. ReCCon is explicitly designed to handle both.
Case 1: all sampled chains share the same erroneous premise. Because a shared error produces no inter-chain disagreement, pure cross-chain contrast would miss it. To cover this, for every divergent statement entering the pipeline, the inspector also generates crux questions (Section 3.3). These are fundamental factual questions that must hold true for the overall statement to be valid, regardless of whether the chains disagree on them. Thus, a flawed premise accepted by all chains is still routed to verification with the chains' shared assumption as the candidate answer. If retrieved evidence refutes it, the judge downweights every chain whose verdict relies on that premise (Appendix I.6).
Case 2: a chain contains minor erroneous sub-claims while its core reasoning is sound. ReCCon does not weight chains simply by counting the proportion of correct sub-claims. Instead, the judge evaluates logical dependency: it identifies which specific premises a chain's final verdict actually rests upon, ensuring that a refuted peripheral detail only minimally affects the chain's weight (Appendix I.6). Comparing alternative aggregation rules on the exact same verification findings confirms this design:
[W2] Controlling Errors and Hallucinations in the LLM Verifier.
ReCCon incorporates two structural safeguards to prevent verifier errors from corrupting the final prediction:
Furthermore, cross-chain contrast directly improves verifier reliability. When a question is framed by contrasting chains, it supplies the competing candidate answers upfront, informing the verifier exactly which detail is in dispute so it can execute targeted searches and score competing answers against shared evidence. Verifying an isolated sub-claim gives no hint of what is contested. To quantify this, we ran an ablation verifying the exact same sub-claims individually without candidate answers from cross-chain contrast:
[Table 1]. Verification with vs. without cross-chain contrast, small setting.
Supplying contrastive candidates significantly boosts verification accuracy while cutting search queries by more than half, since one contrastive question resolves 2.2 sub-claims on average. While running the rebuttal experiments, we fixed an implementation bug in our initial submission where certain judge credibility scores were inadvertently floored or reset, preventing Eq. 2 from being applied exactly as formulated. All new results in this response use this corrected implementation. The four-stage methodology remains completely unchanged, and we will update all tables in the revised manuscript.