acceptodds
Under review as a conference paper at ICLR 2027

ReCCon: Mitigating LLM Hallucinations via Fine-Grained Cross-Chain Consistency

Abstract

In this paper, we propose ReCCon, an LLM hallucination mitigation framework that combines retrieval-augmented generation (RAG) with consistency analysis across multiple sampled answers and their reasoning chains. Existing self-consistency methods operate at a coarse-grained level, simply aggregating sampled final answers into a single prediction without pinpointing and inspecting the specific reasoning steps at which inconsistencies arise, thereby discarding valuable information for assessing the trustworthiness of each answer. Yet, performing fine-grained consistency analysis at the reasoning-step level requires reasoning over complex logical dependencies among the reasoning steps. ReCCon addresses this challenge by leveraging the increasingly strong capabilities of LLMs for analyzing complex logical relationships. Specifically, it employs a multi-step, LLM-driven procedure in which inconsistent reasoning steps are first identified by an LLM, and their truthfulness is then assessed through a dual-layer verification strategy combining RAG and self-consistency. Each sampled answer is subsequently reweighted based on the estimated truthfulness of its sub-claims before the answers are aggregated into a single prediction. Our evaluation shows that ReCCon outperforms RAG- and consistency-based baselines in assessing the truthfulness of both factual and predictive statements, while providing better interpretability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, hallucination mitigation, self-consistency, retrieval-augmented generation, chain-of-thought reasoning, claim verification, belief trees, uncertainty calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sze7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes RECCON (Reasoning-Chain Consistency), a consistency-based framework for mitigating LLM hallucinations. The authors identify limitations in two existing approaches: self-consistency methods aggregate multiple sampled answers but may overlook inconsistencies at individual reasoning steps, while belief-tree-based methods analyze atomic statements but are argued to have limited capacity to capture complex logical dependencies. RECCON aims to bridge these limitations through a four-step pipeline: sampling, contrasting, verifying, and reweighting.

Critical strengths
  1. Relevant and important problem. The paper tries to addresses LLM hallucination where individual reasoning steps may contain errors that are overlooked by answer-level aggregation.

  2. Intuitive overall framework. The four-step pipeline of sampling, contrasting, verifying, and reweighting provides a clear high-level workflow for exploiting inconsistencies across multiple reasoning chains. The paper is easy to follow.

  3. Fine-grained credibility assessment. Rather than treating each sampled answer as an indivisible unit, RECCON attempts to assess the truthfulness of individual sub-claims and use this information to estimate answer credibility.

Critical weaknesses
  1. The identification of inconsistencies is insufficiently justified. In the contrast step, an LLM-based inspection module identifies inconsistent sub-claims across sampled reasoning chains. However, the paper does not adequately clarify how these inconsistencies should be interpreted when all sampled chains contain the same erroneous claim, or when only a small subset of claims is incorrect while the majority are correct. This raises questions about the reliability and coverage of the proposed inconsistency identification mechanism.

  2. The reliability of the LLM-based verifier remains questionable. In the verify step, the framework relies on an LLM-based module to assign truthfulness scores. However, the verifier itself may hallucinate, misinterpret retrieved evidence, or make incorrect judgments about the validity of a claim. The paper needs to clarify how verifier errors are controlled. Otherwise, the proposed framework may potentially introduces additional errors into the reweighting process.

  3. The treatment of 'complex logical dependencies' is unclear. The paper claims that RECCON can accommodate 'complex' logical dependencies among reasoning steps. How does RECCON represent relationships between sub-claims, particularly when the validity of one step depends on multiple preceding steps? Alternatively, are these dependencies captured simply by retaining the original reasoning chains? If so, the main contribution appears to lie in leveraging reasoning chains rather than in the proposed sub-claim decomposition and verification, and the paper should clarify what methodological novelty goes beyond this design choice.

  4. The novelty and complementarity of the framework are limited. The proposed workflow appears to be a combination of 'self-consistency' and 'belief-tree-based' methods. However, it remains unclear whether integrating these two approaches actually addresses their respective limitations, rather than simply combining their existing components.

Questions for authors

See Weaknesses

4 comments

We thank Reviewer sze7 for the thoughtful and constructive review! During the rebuttal period, we added a continuous deterministic weighting baseline using identical verification findings, an ablation quantifying how cross-chain contrast improves verification accuracy and search efficiency, and a scaling experiment with a fourth reasoning chain. Below, we explain how these results and ReCCon's design address each of your questions.

[W1] Handling Shared Errors Across All Chains and Minor Flaws Within Chains.

Thank you for raising these two critical edge cases. ReCCon is explicitly designed to handle both.

Case 1: all sampled chains share the same erroneous premise. Because a shared error produces no inter-chain disagreement, pure cross-chain contrast would miss it. To cover this, for every divergent statement entering the pipeline, the inspector also generates crux questions (Section 3.3). These are fundamental factual questions that must hold true for the overall statement to be valid, regardless of whether the chains disagree on them. Thus, a flawed premise accepted by all chains is still routed to verification with the chains' shared assumption as the candidate answer. If retrieved evidence refutes it, the judge downweights every chain whose verdict relies on that premise (Appendix I.6).

Case 2: a chain contains minor erroneous sub-claims while its core reasoning is sound. ReCCon does not weight chains simply by counting the proportion of correct sub-claims. Instead, the judge evaluates logical dependency: it identifies which specific premises a chain's final verdict actually rests upon, ensuring that a refuted peripheral detail only minimally affects the chain's weight (Appendix I.6). Comparing alternative aggregation rules on the exact same verification findings confirms this design:

  • A heuristic drop rule, which discards any chain containing a low-scoring flagged claim and so penalizes minor and major errors equally, falls 4.3 and 1.9 AUROC behind ReCCon on the divergence set (Table 4).
  • A continuous deterministic weighting baseline, which weights each chain by the mean truthfulness score of its sub-claims, achieves 65.9 AUROC on the small divergence set and trails ReCCon's dependency-aware judge at 69.8. Table 4 of the revised paper includes this baseline as average weighting.

[W2] Controlling Errors and Hallucinations in the LLM Verifier.

ReCCon incorporates two structural safeguards to prevent verifier errors from corrupting the final prediction:

  1. Abstention over hallucination: if retrieved passages do not clearly resolve a question, the verifier returns UNSETTLED. Absence of evidence is never treated as refutation, and every verified finding must cite the specific supporting passage (Appendix I.4).
  2. Conservative judge treatment: the judge treats UNSETTLED findings and belief-tree estimates below 0.5 as open questions rather than negative evidence, ensuring unverified claims never overturn a chain (Appendix I.6).

Furthermore, cross-chain contrast directly improves verifier reliability. When a question is framed by contrasting chains, it supplies the competing candidate answers upfront, informing the verifier exactly which detail is in dispute so it can execute targeted searches and score competing answers against shared evidence. Verifying an isolated sub-claim gives no hint of what is contested. To quantify this, we ran an ablation verifying the exact same sub-claims individually without candidate answers from cross-chain contrast:

[Table 1]. Verification with vs. without cross-chain contrast, small setting.

MetricLabeled questionsContrast (ReCCon)Independent Sub-claim Verification
Estimated accuracy across all questions23676.0%72.1%
Accuracy when the two pick different answers14458.3%41.7%
Searches for the covered sub-claims9,70021,151

Supplying contrastive candidates significantly boosts verification accuracy while cutting search queries by more than half, since one contrastive question resolves 2.2 sub-claims on average. While running the rebuttal experiments, we fixed an implementation bug in our initial submission where certain judge credibility scores were inadvertently floored or reset, preventing Eq. 2 from being applied exactly as formulated. All new results in this response use this corrected implementation. The four-stage methodology remains completely unchanged, and we will update all tables in the revised manuscript.

[W3] Representing Logical Dependencies Between Sub-Claims.

By design, ReCCon does not construct an explicit formal dependency graph over sub-claims. Instead, logical dependencies are preserved by retaining each complete reasoning chain as the unit of evaluation and reweighting. Decomposing a complex multi-hop or counterfactual chain into isolated atomic claims often strips away critical context. For example, evaluating the consequent of a counterfactual statement in IfQA against real-world facts makes a logically valid deduction appear false out of context.

To avoid this pitfall, ReCCon only extracts disputed sub-claims to query external evidence, and then feeds the verified findings back into the context of the full reasoning chains so the judge can assess how each finding impacts the chain's overall conclusion. Empirical results strongly support this design. Atomic decomposition baselines suffer severe degradation on counterfactual reasoning, as seen on IfQA in Table 1. Two aggregation rules that ignore the dependency also do worse than the judge on the exact same chains and verification findings. A drop rule discards any chain containing a low-scoring flagged claim, and an average weighting weights each chain by the mean truthfulness score of its sub-claims:

[Table 2]. Aggregation methods using identical chains and verification findings. Divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC61.266.055.873.464.1
Drop rule64.864.660.372.265.5
Average weighting64.467.757.474.165.9
ReCCon (LLM judge)66.673.259.280.469.8

The drop rule and average weighting recover 1.4 and 1.8 of the 5.7 AUROC points gained over RAG + SC, and the judge adds the rest. In the large setting, the judge is 1.9 AUROC above the drop rule and 3.5 above average weighting (Table 4 of the revised paper).

[W4] Novelty and Complementarity of the Integrated Framework.

While sub-claim verification, self-consistency (SC), and belief trees are existing primitives, ReCCon's novelty lies in bridging answer-level aggregation and claim-level verification via cross-chain disagreement: it contrasts chains to selectively decide what to verify, and reweights complete chains based on whether their load-bearing premises survive verification.

This integration directly resolves the core limitation of each individual paradigm.

Overcoming the failure mode of self-consistency: the naive averaging of RAG + SC breaks down when sampled chains strongly disagree, as Table 2 of the paper shows. ReCCon resolves these disputes through targeted verification and reweighting, improving divergence-set AUROC by +5.7 and +5.3. Sampling more chains does not close this gap. We sampled six additional chains per statement, so that RAG + SC matched ReCCon's average output-token budget:

[Table 3]. Equal-budget comparison on the divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC, 3 chains, Table 2 of the paper61.266.055.873.464.1
RAG + SC at ReCCon's budget65.664.856.679.666.7
ReCCon66.673.259.280.469.8
ReCCon − RAG + SC at ReCCon's budget+0.9 [−4.3, 6.1]+8.4 [4.0, 13.0]+2.6 [−4.4, 9.7]+0.8 [−3.2, 4.7]+3.2 [0.4, 5.8]

At the same budget, ReCCon remains +3.2 AUROC ahead (95% CI [0.4, 5.8], p = 0.01).

Overcoming the context loss of belief trees and atomic verification: ReCCon keeps the full chain as the unit of reweighting, and [W3] shows with [Table 2] that rules ignoring the dependencies between a chain's claims do worse on the same findings.

Two further results confirm that this synergy goes well beyond a simple sum of parts. Cross-chain contrast makes verification significantly more accurate at less than half the retrieval cost, as shown in [Table 1]. The gains also persist as we scale the ensemble. We added a fourth chain sampled from gpt-oss, so that each model contributes two chains, and compared ReCCon with the average of the same four chains:

[Table 4]. AUROC gain of ReCCon over the 4-chain average, small setting.

SetFactCHDHoVerFutureXIfQAAvg
Divergence set+1.9+8.2+0.7+4.4+3.8 [2.0, 5.7]
Full set+0.3+1.2+0.3+0.6+0.6 [0.2, 1.0]

Finally, ReCCon makes ensemble predictions transparent by returning the exact points of inter-chain contention, the retrieved evidence settling each dispute, and the resulting chain weights (Appendix H, Table 24 of the revised paper).

Thank you again for your valuable feedback. We hope these clarifications and new experiments address your concerns, and we are happy to discuss further!

Dear Reviewer sze7,

We have uploaded a revised version of the paper. The main results and the ablations (Tables 1–6) now come from the corrected implementation, and the revision includes the new experiments from our responses:

  • Aggregation rules: Table 4 and Appendix G.9
  • Verification with and without cross-chain contrast: Table 6 and Appendix G.10
  • Equal-budget comparison: Table 5 and Appendix G.4
  • Scaling to four chains: Appendix G.5

We have also updated our responses above to match the revised paper. Thank you again for your time and feedback.

Thanks for the rebuttal, I will keep my score.

Reviewer_WACrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ReCCon identifies disagreements across sampled reasoning chains, verifies disputed sub-claims using RAG and belief trees, and reweights predictions with an LLM judge. Across four datasets and two model settings, it reports consistent AUROC improvements, especially on examples where sampled chains disagree.

Critical strengths
  1. Good motivation. Examining why reasoning chains disagree may be more informative than aggregating final answers alone.
  2. Relevant evaluation. Four datasets, two model settings, and several RAG- and consistency-based baselines are included.
  3. Fair conditional gains. On divergence subsets, AUROC improves by 3.9 and 5.2 points over RAG + self-consistency using the same initial chains.
Critical weaknesses
  1. Small overall gains relative to complexity. Full-set improvements over RAG + self-consistency are only 0.6 and 0.5 AUROC points, despite additional extraction, inspection, verification, and judging stages. Computational cost is not adequately quantified. This is inefficient and uneconomical.
  2. Core novelty is not isolated. The method combines claim extraction, RAG verification, BTProp, and LLM-based weighting. A baseline that verifies claims without first contrasting chains is missing, so the unique benefit of cross-chain inconsistency analysis remains unclear.
  3. Questionable inconsistency definition. Section 3.3 treats a claim omitted by another chain as an “asymmetry.” Omission is not contradiction; valid chains often use different evidence. The utility and precision of this selection rule need validation.
  4. Divergence evaluation needs qualification. The subset is selected using the same score-spread threshold (>0.25) that activates the method. The larger conditional gains are useful but cannot replace full-set results; threshold sensitivity is not reported.
  5. Weak component justification. In Table 3, belief trees add only 0.4–0.7 macro-AUROC points, and RL training adds roughly 0.1–0.2 points over the base generator. The necessity of these components is not convincing.
  6. LLM judge remains a confound. The paper compares against a crude drop rule but not simple continuous, deterministic weighting of the same verified claim scores. Improvements may stem from extra judge inference rather than the proposed consistency mechanism.
Questions for authors
  1. Does cross-chain contrast outperform directly verifying and reweighting extracted claims under identical budgets?
  2. Why is a claim’s omission by another chain considered informative inconsistency? What is the inspector’s precision?
  3. How sensitive are results to the 0.25 divergence threshold?
  4. What does RL training contribute beyond the base belief-tree generator, given the small measured gain?
  5. Can deterministic continuous weighting match the LLM judge using identical verification findings?
  6. What are the latency, token, retrieval, and monetary costs under matched budgets?
5 comments

We thank Reviewer WACr for the thorough and insightful review! During the rebuttal period, we added a comprehensive breakdown of token, retrieval, and latency costs alongside an equal-budget comparison with RAG + SC, an ablation that verifies sub-claims independently without cross-chain contrast, a precision analysis of the inspector's contradiction and asymmetry flags, a threshold sensitivity sweep, and a continuous deterministic weighting baseline. Below, we detail how these results address your concerns.

[W1 & Q6] Computational Cost, Latency, and Equal-Budget Comparison.

ReCCon only modifies predictions when initial chains disagree, which happens for 28% of statements in the small setting and 24% in the large setting, so full-set improvements naturally reflect an average across both triggered and non-triggered statements. Nevertheless, full-set gains are statistically significant in both settings: +0.7 AUROC in small (95% CI [0.3, 1.1]) and +0.5 AUROC in large (95% CI [0.1, 0.8]). Gains on the targeted divergence subset are substantial: +5.7 [3.5, 8.0] and +5.3 [2.8, 7.8] AUROC. On IfQA, full-set performance is already near ceiling at 93–94 AUROC. While running the rebuttal experiments, we fixed an implementation bug in our initial submission where certain judge credibility scores were inadvertently floored or reset, preventing Eq. 2 from being applied exactly as formulated. All new results in this response use this corrected implementation. The four-stage methodology remains completely unchanged, and we will update all tables in the revised manuscript.

Amortized across the entire dataset, ReCCon uses 2.6–3.3x the output tokens of 3-chain RAG + SC. To conduct an equal-budget comparison, we sampled six additional chains per statement from the same two models and allocated RAG + SC the exact same average output-token budget as ReCCon. We evaluate on the divergence set, where ReCCon expends all of its additional compute:

[Table 1]. Output tokens per divergent statement, small setting.

StageOutput Tokens
Sub-claim extraction10k
Verification33k–41k
Belief tree16k–27k
Judge7k–8k
Total70k–84k

[Table 2]. Wall-clock latency per divergent statement, small setting.

Per divergent statementValue
Verification input tokens, primarily retrieved passages66k–83k
Amortized latency of extraction, verification and the judge16–18 s

Latency is amortized across batched execution on a single node with 4x H100 GPUs. Monetary cost scales linearly with token counts under standard API pricing.

To evaluate performance under an identical compute budget, we sampled six additional chains per statement from the same two models using new random seeds, so that RAG + SC matched ReCCon's average output-token budget. On the divergence set, where ReCCon spends all of its additional budget, ReCCon outperforms equal-budget RAG + SC across all four datasets by an average of +3.2 AUROC (95% CI [0.4, 5.8], p = 0.01):

[Table 3]. Equal-budget comparison on the divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC, 3 chains, Table 2 of the paper61.266.055.873.464.1
RAG + SC at ReCCon's budget65.664.856.679.666.7
ReCCon66.673.259.280.469.8
ReCCon − RAG + SC at ReCCon's budget+0.9 [−4.3, 6.1]+8.4 [4.0, 13.0]+2.6 [−4.4, 9.7]+0.8 [−3.2, 4.7]+3.2 [0.4, 5.8]

These tables are in the revised manuscript (Table 5 and Appendix G.4).

[W2 & Q1] Ablating Cross-Chain Contrast by Verifying Sub-Claims Independently.

Following your suggestion, we implemented a baseline in the small setting that bypasses the cross-chain inspector and instead verifies every extracted sub-claim from all three chains independently as a standalone question. All other components remain identical: sub-claim extraction, the verifier, the belief tree, the judge, and the atomic fact checks.

To directly measure verification quality, we evaluated how accurately each approach identifies the correct answer across inspector-flagged contradictions:

[Table 4]. Verification with vs. without cross-chain contrast, small setting.

MetricLabeled questionsContrast (ReCCon)Independent Sub-claim Verification
Estimated accuracy across all questions23676.0%72.1%
Accuracy when the two pick different answers14458.3%41.7%
Searches for the covered sub-claims9,70021,151

Cross-chain contrast achieves higher verification accuracy while requiring fewer than half the search queries. This is because a contrastive question explicitly presents the competing candidate answers from different chains, allowing the verifier to pinpoint the exact detail in dispute, formulate targeted search queries, and evaluate competing claims against the same retrieved evidence. Verifying an isolated sub-claim provides no signal about what is contested and requires separate searches for every sub-claim, whereas one contrastive question consolidates 2.2 sub-claims on average.

Together with [W6 & Q5], which isolates our reweighting mechanism, this confirms both the efficiency and effectiveness of cross-chain contrast. Finally, contrasting chains yields explicit interpretability: for 78–99% of divergent statements, ReCCon surfaces at least one evidence-settled point of contention explaining why specific chains were trusted or downweighted (Appendix H, Table 24 of the revised paper).

[W3 & Q2] Role of Omission, Asymmetry Flags, and Inspector Precision.

We agree that omitting a claim is not equivalent to contradicting it, and ReCCon never treats omission itself as evidence against a chain. An asymmetry flag, raised when one chain relies on a premise that other chains do not mention, merely nominates that premise for verification. If retrieval cannot verify or refute the premise, it remains UNSETTLED and never penalizes any chain. However, checking asymmetric premises is highly informative because an unstated or unique premise used by only one chain is frequently the exact pivot driving the chains' divergent conclusions.

To evaluate the precision of the inspector's flags, we measured how often a flag successfully partitions the sampled chains into correct chains, meaning those matching the gold statement label, and incorrect chains. For instance, if chains A and B reach the correct verdict while chain C is incorrect, a flag that pits {A, B} against {C} is counted as precise, whereas one pitting {A, C} against {B} is not. For asymmetries, the partition separates chains that assert the claim from those that omit it. Chance denotes random partitioning of the same size.

[Table 5]. Precision of inspector flags in separating correct vs. incorrect chains, divergence sets.

SettingContradiction FlagsRandom ChanceAsymmetry FlagsRandom Chance
Large66.4%45%35.0%26%
Small74.4%49%40.6%27%

Both flag types exceed random chance by a wide margin, confirming that the inspector reliably isolates the specific reasoning divergences separating correct from incorrect chains. Moreover, asymmetry questions prove highly actionable in practice: in the small setting, 73% of asymmetry questions are successfully resolved by retrieved evidence, and across the 244 divergent statements where the judge's final weighting relies on an asymmetry finding, ReCCon outperforms RAG + SC by +6.8 AUROC (95% CI [2.9, 10.8]) (Appendix G.8 of the revised paper).

[W5 & Q4] Necessity of the Belief Tree and RL Training.

The belief tree is designed specifically as a fallback mechanism for verification questions that direct retrieval cannot settle. Because retrieval successfully resolves 49–79% of questions in the small setting, the belief tree only operates on the remaining unresolved fraction. With the corrected implementation, it changes the macro AUROC on the divergence set by +0.3 and −0.2 in the small and large settings (Table 3 of the revised paper). Meanwhile, RL training teaches the child-claim generator to propose sub-claims that shift belief estimates toward the ground-truth label (Appendix D), as a one-time offline training step with zero inference overhead. ReCCon's gains stem from contrastive verification and chain reweighting, as isolated in [W2 & Q1] and [W6 & Q5]: without the belief tree, ReCCon still improves over RAG + SC by +5.4 and +5.5 AUROC on the divergence set.

[W4 & Q3] Divergence Subset Selection and Threshold Sensitivity.

The gains on the divergence subset complement the full-set metrics, which are reported across all statements in Table 1. Because the divergence set is defined strictly from the spread of initial chain scores prior to verification and without label access, it serves to show where the full-set improvements originate.

We set the 0.25 threshold a priori because reweighting chains whose initial scores are nearly identical has minimal effect on the final prediction (Section 3.3). Following your suggestion, we swept the activation threshold from 0.25 to 0.50:

[Table 6]. Full-set AUROC across divergence thresholds, with the percentage of triggered statements in parentheses.

Setting & DatasetRAG + SC>0.25>0.30>0.40>0.50
Small, FactCHD82.082.3 (24%)82.3 (22%)82.3 (20%)82.3 (19%)
Small, HoVer75.176.2 (32%)76.2 (32%)76.2 (29%)76.1 (29%)
Small, FutureX65.866.5 (30%)66.5 (25%)66.4 (17%)66.2 (12%)
Small, IfQA93.293.8 (28%)93.8 (27%)93.8 (24%)93.8 (23%)
Small, Avg79.079.7 (28%)79.7 (27%)79.7 (24%)79.6 (22%)
Large, FactCHD83.483.6 (19%)83.6 (17%)83.5 (14%)83.5 (10%)
Large, HoVer77.778.3 (28%)78.3 (26%)78.3 (21%)78.3 (17%)
Large, FutureX63.063.6 (25%)63.7 (17%)63.2 (9%)63.0 (5%)
Large, IfQA94.094.5 (24%)94.5 (23%)94.5 (18%)94.4 (14%)
Large, Avg79.580.0 (24%)80.0 (21%)79.9 (17%)79.8 (13%)

ReCCon's improvement over RAG + SC remains statistically significant across all tested thresholds. Higher thresholds offer a smooth compute-accuracy trade-off: triggering on fewer statements reduces cost while retaining most of the gain. At >0.50, average full-set gains shift from +0.70 to +0.60 in small and from +0.48 to +0.29 in large.

[W6 & Q5] Continuous Deterministic Weighting vs. LLM Judge.

Following your suggestion, we implemented a continuous deterministic weighting rule using the exact same chains and verification findings. Specifically, we weight each chain continuously by the mean truthfulness score of its sub-claims, with no constants to tune:

[Table 7]. Aggregation methods using identical chains and verification findings. Divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC61.266.055.873.464.1
Drop rule64.864.660.372.265.5
Average weighting64.467.757.474.165.9
ReCCon (LLM judge)66.673.259.280.469.8

Without the LLM judge, average weighting recovers 1.8 of the 5.7 AUROC points gained over RAG + SC in the small setting and 1.8 of 5.3 in the large setting. The judge adds the remaining 3.9 and 3.5 by telling whether a refuted sub-claim is one the chain's conclusion depends on. Table 4 of the revised paper includes this baseline for both settings, with intervals in Appendix G.9.

Thank you again for your constructive feedback. We hope these new experiments and analyses thoroughly address your concerns, and we welcome any further questions.

Reviewer_xPZ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCCon, a framework that estimates whether a statement is true by examining where several sampled reasoning chains disagree. It samples three chains (two from one model, one from another), extracts sub-claims from each chain, and uses an LLM inspector to flag contradicting or asymmetric sub-claims and turn them into verification questions. The questions are verified with Wikipedia retrieval, and the unsettled ones with a belief-tree method (BTProp) whose paraphrase generator is trained with RL. A judge LLM then gives each chain a credibility weight, and the final score is the weighted mean of the chains' initial truthfulness scores. The authors test it on FactCHD, HoVer, FutureX and IfQA with two model settings, and report the best AUROC on all datasets, with larger gains on the "divergence set" (statements where the initial scores of the chains differ by more than 0.25).

Critical strengths
  1. The idea of using disagreement between chains to decide what to verify is reasonable, and the divergence-set analysis (Section 4.3) is a useful way to show where averaging does not help (e.g., RAG + self-consistency is below RAG on the large-model divergence set).
  2. The paper is clearly written, the pipeline is well described with a running example (Figures 1 and 2), and the method returns the verification questions and findings behind each prediction.
Critical weaknesses
  1. The novelty is limited. The main contribution is to verify at the sub-claim level instead of the answer level, which is not new (FActScore, SAFE, FIRE, Chain-of-Verification, FactReasoner and VeriScore are all cited in the paper). What is new is using the disagreement between chains to select which sub-claims to verify, but the verification itself is existing work put together (retrieval, BTProp and an LLM judge). The pipeline seems reasonable but is also expensive, and the paper does not analyze the cost: it adds an extractor, an inspector, verifier agents with retrieval, a relation labeler and a judge on top of the three chains, plus RL training of the generator, but I did not find any efficiency or token-cost comparison in the main paper.
  2. The gain over a simple baseline is small, and it is unclear whether it is worth the cost. On the full evaluation sets (Table 1), ReCCon improves RAG + self-consistency by 0.6 and 0.5 AUROC on average (79.6 vs 79.0, 80.0 vs 79.5), and by 0.1 to 0.3 on FactCHD and IfQA. The larger gains in the abstract (3.9 and 5.2) are on the divergence set, which is selected with the same threshold (0.25) that ReCCon uses to decide when to run, and outside this set ReCCon returns exactly the mean of the initial scores, i.e., the same as RAG + self-consistency. The divergence sets are also small (104 to 352 statements per dataset), the components add little (the belief tree gives +0.5 and +0.3, and the RL-trained generator +0.2 and +0.1 over the base generator in Table 3), and the baseline uses only k = 3 chains. A baseline with the same budget (e.g., more sampled chains for RAG + self-consistency) is needed to know whether the extra cost really pays off.

Besides, the fine-grained baselines look strange. FactReasoner, VeriScore and FAVA are 8 to 23 points below Direct ask in several columns of Table 1 (e.g., VeriScore gets 52.4 on IfQA vs 78.9 for Direct ask), so I doubt whether they were adapted properly to this statement-level setting, which makes the comparison with prior fine-grained methods less informative.

Overall, the method is reasonable but the novelty is limited and the evidence for the extra cost is weak. I lean toward rejection, and I would reconsider if the authors show the gain at equal cost.

Questions for authors
  • What is the total number of LLM calls and tokens per statement for ReCCon compared with RAG + self-consistency, and how does the accuracy change if the baseline gets the same budget?
  • How was the 0.25 threshold chosen? How many statements trigger the full pipeline in each setting?
  • How were FactReasoner, VeriScore and FAVA adapted to statement-level truthfulness, and why are they below Direct ask?
  • The paired bootstrap resamples statements; do the conclusions hold across different sampling seeds of the chains?
5 comments

We thank Reviewer xPZ3 for the constructive feedback! During the rebuttal period, we added (1) an equal-budget comparison with RAG + SC, (2) an ablation that verifies the same sub-claims independently without cross-chain contrast, (3) a scaling experiment with a fourth reasoning chain, (4) a divergence threshold sweep, and (5) evaluations across two additional chain-sampling seeds. Below, we detail how these results address your concerns.

[W1] Novelty and Component Contributions.

While sub-claim verification, self-consistency (SC), and belief trees are established techniques, ReCCon's core contribution lies in how it leverages cross-chain disagreement. It contrasts sampled chains to selectively pinpoint disputed sub-claims for verification, and it reweights each reasoning chain as a cohesive unit based on whether the core premises supporting its verdict survive verification. Four empirical findings demonstrate what this formulation adds beyond its individual components.

First, context-aware chain reweighting is essential. In Table 4 of the main paper, we compare our LLM judge against a heuristic drop rule, which simply discards any chain containing a low-scoring flagged sub-claim. We also evaluated an average weighting baseline, which weights each chain by the mean truthfulness score of its sub-claims and has no constants to tune.

[Table 1]. Aggregation methods using identical chains and verification findings. Divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC61.266.055.873.464.1
Drop rule64.864.660.372.265.5
Average weighting64.467.757.474.165.9
ReCCon (LLM judge)66.673.259.280.469.8

ReCCon's judge exceeds the drop rule by +4.3 and +1.9 AUROC and average weighting by +3.9 and +3.5 in the small and large settings (Table 4 of the revised paper; intervals in Appendix G.9).

Second, cross-chain contrast improves both verification accuracy and efficiency. Formulating a verification query by contrasting chains provides the verifier with the competing candidate answers upfront. This informs the verifier precisely which detail is in dispute, enabling targeted retrieval and direct comparison against shared evidence. By contrast, verifying an isolated sub-claim gives no signal regarding what specific detail is contested. To quantify this benefit, we evaluated an ablation that verifies the exact same sub-claims individually without cross-chain contrast, measuring how often each approach identifies the ground-truth answer across inspector-flagged contradictions. While running the rebuttal experiments, we fixed an implementation bug in our initial submission where certain judge credibility scores were inadvertently floored or reset, preventing Eq. 2 from being applied exactly as formulated. All new results in this response use this corrected implementation. The four-stage methodology remains completely unchanged, and we will update all tables in the revised manuscript.

[Table 2]. Verification with vs. without cross-chain contrast, small setting.

MetricLabeled questionsContrast (ReCCon)Independent Sub-claim Verification
Estimated accuracy across all questions23676.0%72.1%
Accuracy when the two pick different answers14458.3%41.7%
Searches for the covered sub-claims9,70021,151

Cross-chain contrast not only selects the correct answer substantially more often, but also requires fewer than half as many search queries, as a single contrastive question resolves an average of 2.2 sub-claims simultaneously.

Third, the gains hold when scaling the number of chains. We added a fourth chain sampled from gpt-oss, so that each model contributes two chains, and compared ReCCon with the average of the same four chains.

[Table 3]. ReCCon with four chains vs. the 4-chain average, small setting

SetMethodFactCHDHoVerFutureXIfQAAvg
Divergence setRAG + SC, 4 chains64.362.162.276.966.4
Divergence setReCCon, 4 chains66.370.362.881.370.2
Divergence setReCCon − RAG + SC+1.9 [−1.8, 5.6]+8.2 [4.3, 12.4]+0.7 [−3.4, 4.7]+4.4 [1.7, 7.2]+3.8 [2.0, 5.7]
Full setRAG + SC, 4 chains81.975.466.693.579.4
Full setReCCon, 4 chains82.276.766.994.180.0
Full setReCCon − RAG + SC+0.3 [−0.1, 0.7]+1.2 [0.6, 1.9]+0.3 [−0.9, 1.5]+0.6 [0.2, 1.0]+0.6 [0.2, 1.0]

ReCCon still outperforms the 4-chain average by +3.8 AUROC on the divergence set and +0.6 AUROC on the full set, and the gain is positive on every dataset.

Fourth, ReCCon is interpretable. RAG + SC outputs a simple average that offers no insight into which chain to trust when models disagree, whereas ReCCon provides an explicit audit trail. For every divergent statement, it outputs the specific points of disagreement, the verified verdict for each candidate answer backed by retrieved passages, and the resulting chain weights. In the small setting, 49–79% of verification questions are directly resolved by retrieved evidence, and 78–99% of divergent statements have at least one evidence-settled question. Appendix H, Table 24 of the revised paper gives a qualitative example.

[W2 & Q1] Computational Cost and Equal-Budget Comparison.

Thank you for raising this important point. Below, we break down the token cost per stage and compare ReCCon against RAG + SC under an equal output-token budget.

[Table 4]. Output tokens per divergent statement by stage, small setting.

StageOutput Tokens
Sub-claim extraction10k
Verification33k–41k
Belief tree16k–27k
Judge7k–8k
Total70k–84k

Amortized across the entire dataset, ReCCon uses 2.6–3.3x the output tokens of 3-chain RAG + SC. To conduct an equal-budget comparison, we sampled six additional chains per statement from the same two models and allocated RAG + SC the exact same average output-token budget as ReCCon. We evaluate on the divergence set, where ReCCon expends all of its additional compute:

[Table 5]. Equal-budget comparison on the divergence set, small setting, AUROC.

MethodFactCHDHoVerFutureXIfQAAvg
RAG + SC, 3 chains, Table 2 of the paper61.266.055.873.464.1
RAG + SC at ReCCon's budget65.664.856.679.666.7
ReCCon66.673.259.280.469.8
ReCCon − RAG + SC at ReCCon's budget+0.9 [−4.3, 6.1]+8.4 [4.0, 13.0]+2.6 [−4.4, 9.7]+0.8 [−3.2, 4.7]+3.2 [0.4, 5.8]

Under an identical token budget, ReCCon outperforms RAG + SC across all four datasets, achieving a statistically significant average gain of +3.2 AUROC (95% CI [0.4, 5.8], p = 0.01). We will incorporate both tables into the revised paper.

[W3] Magnitude of Gains over RAG + SC.

ReCCon only modifies predictions when initial chains disagree, which happens for 28% of statements in the small setting and 24% in the large setting, so full-set improvements naturally reflect a diluted average. Even so, the full-set gains are statistically significant in both settings: +0.7 AUROC in small (95% CI [0.3, 1.1]) and +0.5 AUROC in large (95% CI [0.1, 0.8]). The gains on the targeted divergence subset are substantial: +5.7 [3.5, 8.0] and +5.3 [2.8, 7.8] AUROC, respectively. The gain is positive on every dataset in both settings. On IfQA, full-set performance is already near ceiling at 93–94 AUROC.

Crucially, the divergence set is not cherry-picked in ReCCon's favor: it is determined purely from the spread of the initial chains' scores prior to verification and without access to ground-truth labels, and both ReCCon and RAG + SC are evaluated on the exact same split.

Regarding the belief tree and RL training: the belief tree serves strictly as a fallback mechanism for sub-claims that retrieval cannot resolve, so its overall impact is bounded by design. With the corrected implementation, it changes the macro AUROC on the divergence set by +0.3 and −0.2 in the small and large settings (Table 3 of the revised paper), and ReCCon's gain over RAG + SC does not depend on it (+5.4 and +5.5 without the belief tree). The RL training of the generator is a one-time offline cost with zero inference overhead.

[W4 & Q3] Adaptation and Performance of Fine-Grained Baselines.

Appendix G.1 describes how each baseline was adapted to produce a single statement-level truthfulness score. They fall below Direct Ask because atomic verification against the corpus does not fit two of the datasets. On FutureX, the target events occur after the corpus cutoff, so no passage can verify the outcome, and FR1 and VeriScore stay near chance at 48.8–50.9 AUROC. On IfQA, decomposing a counterfactual statement into atomic claims strips away the counterfactual premise, and checking those atoms against real-world facts makes VeriScore drop to 52.4 and 57.7. On HoVer, where the claims are grounded in the Wikipedia corpus, the gap to Direct Ask is smallest.

[Q2] Threshold Selection and Sensitivity Analysis.

We set the divergence threshold to 0.25 a priori because when initial chain scores have a small spread, reweighting them has a negligible effect on the final weighted average (Section 3.3). The statements exceeding this threshold form the divergence set (Table 2):

[Table 6]. Number and proportion of statements triggering the verification pipeline.

SettingFactCHDHoVerFutureXIfQAOverall Share
Small23432412835228%
Large18527810431024%

To evaluate sensitivity, we swept the threshold from 0.25 to 0.50. ReCCon's gain over RAG + SC remains statistically significant across all thresholds, confirming that our conclusions do not hinge on the specific choice of 0.25. Increasing the threshold reduces compute by triggering the pipeline less frequently, trading off a modest fraction of the overall AUROC gain. At 0.50, average full-set gains decrease from +0.70 to +0.60 in the small setting and from +0.48 to +0.29 in the large setting:

[Table 7]. Full-set AUROC across divergence thresholds, with the percentage of triggered statements in parentheses.

Setting & DatasetRAG + SC>0.25>0.30>0.40>0.50
Small, FactCHD82.082.3 (24%)82.3 (22%)82.3 (20%)82.3 (19%)
Small, HoVer75.176.2 (32%)76.2 (32%)76.2 (29%)76.1 (29%)
Small, FutureX65.866.5 (30%)66.5 (25%)66.4 (17%)66.2 (12%)
Small, IfQA93.293.8 (28%)93.8 (27%)93.8 (24%)93.8 (23%)
Small, Avg79.079.7 (28%)79.7 (27%)79.7 (24%)79.6 (22%)
Large, FactCHD83.483.6 (19%)83.6 (17%)83.5 (14%)83.5 (10%)
Large, HoVer77.778.3 (28%)78.3 (26%)78.3 (21%)78.3 (17%)
Large, FutureX63.063.6 (25%)63.7 (17%)63.2 (9%)63.0 (5%)
Large, IfQA94.094.5 (24%)94.5 (23%)94.5 (18%)94.4 (14%)
Large, Avg79.580.0 (24%)80.0 (21%)79.9 (17%)79.8 (13%)

[Q4] Robustness Across Chain-Sampling Seeds.

To verify stability across random seeds, we re-ran the small-setting evaluation on two additional sets of reasoning chains sampled with new random seeds, each inducing its own divergence subset. Across all three independent runs, ReCCon consistently achieves positive average gains on every dataset for both the divergence set and the full set:

[Table 8]. AUROC gain over RAG + SC across three chain-sampling seeds, divergence set.

Chain SampleFactCHDHoVerFutureXIfQA
Original paper chains+5.4+7.2+3.4+7.0
New Seed 1+0.4+3.5+3.3+4.6
New Seed 2+6.2+8.2−0.3+8.4
Average+4.0+6.3+2.1+6.7

[Table 9]. AUROC gain over RAG + SC across three chain-sampling seeds, full set.

Chain SampleFactCHDHoVerFutureXIfQA
Original paper chains+0.3+1.1+0.7+0.6
New Seed 1+0.1+0.6+0.6+0.6
New Seed 2+0.6+1.2−0.1+0.8
Average+0.3+1.0+0.4+0.7

Thank you again for your valuable time and feedback. If our equal-budget comparisons and new ablations address your primary concerns, we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.