acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Aware Bayesian Self-Consistency for Evidence Aggregation and Risk-Controlled Abstention

Abstract

Self-consistency improves language-model reasoning by aggregating multiple sampled solutions, but standard aggregation treats observed support as a proxy for confidence without explicitly accounting for whether different evidence sources are trustworthy for a particular question. We introduce Reliability-Aware Bayesian Self-Consistency (RA-BSC), a probabilistic framework for aggregating evidence from multiple generators and verifiers while representing question-dependent uncertainty in source trust. RA-BSC combines a categorical candidate model with structured source-reliability functions and propagates parameter uncertainty through approximate Bayesian inference. We further separate posterior uncertainty from decision-time guarantees by coupling the resulting predictive and selective scores with a held-out Learn-Then-Test procedure for risk-controlled abstention. Controlled comparisons with fixed-reliability, adaptive deterministic, and conventional self-consistency baselines show that structured reliability modeling is particularly useful when evidence sources disagree, while Bayesian marginalization primarily contributes uncertainty-aware selective information rather than universally improving point prediction. Across reasoning benchmarks, our results highlight that answer aggregation, confidence calibration, and certifiable abstention are distinct objectives, and that explicitly representing uncertainty about evidence quality provides a principled way to connect them.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian self-consistency, uncertainty quantification, evidence aggregation, source reliability, selective prediction, risk-controlled abstention, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tgW3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines answers from several language models with judgments from an answer checker and a reasoning checker. It learns how much to trust each source on a question, then uses an approximate posterior over those trust parameters to represent uncertainty about the fit. A separate Learn-Then-Test rule decides when to return an answer. The main experiments suggest that learned trust helps answer selection, while the extra Bayesian step has mixed benefits. Its clearest positive result is answering more ARC questions after one generator is replaced with Gemma.

Critical strengths
  • The comparisons help separate what is doing the work. The paper tests constant source trust, question-dependent trust, and Bayesian averaging using the same evidence (Table 2). This makes it easier to see whether a gain comes from learning source weights or from representing uncertainty about them.
  • The mixed-family ARC result is promising. With one Qwen generator replaced by Gemma, RA-BSC q10 answers about 92 out of 100 questions, compared with 86 for fixed trust, under the same calibrated error-control procedure (Table 16). That could be useful even though overall accuracy is slightly lower.
Critical weaknesses
  • I’m not yet convinced the Bayesian step adds enough beyond simpler weighting. Simpler trust models choose answers at least as accurately in the main experiments, while the coverage gains are mixed. Since the claimed benefit is deciding when to answer, the strongest simple scores should be tested under the same abstention procedure. Verifier-weighted voting is missing from that main comparison despite its strong ARC ranking result (Tables 1 and 7).
  • It is unclear how much the benefit depends on the chosen sources. Copying a generator raises confidence without adding information (Table 14), and the strongest positive coverage result comes from one changed generator mixture. This makes it hard to tell how well the reported benefit carries over to another source setup. More broadly, it seems the result depends quite a bit on how sources are selected - some recommendations and/or acknowledgement of this limitation would be helpful.
Questions for authors
  • How does it compare with the strongest simple scores under the same answer-or-abstain rule? Consider adding verifier-weighted voting and the scalar-calibrated voting scores to the main LTT comparison, with paired uncertainty for coverage differences.
Reviewer_ERbZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian framework for aggregating sampled answers from multiple generators and verifiers. It models question-dependent source reliability and uses a Laplace approximation to estimate posterior uncertainty. A separate Learn-Then-Test procedure provides risk-controlled abstention. Experiments on three QA benchmarks show improvements over Bayesian softmax in some settings, but mixed results compared with simpler reliability models.

Critical strengths
  1. The paper distinguishes evidence aggregation, uncertainty estimation, and LTT-based risk guarantees, and the controlled ablations help clarify the contribution of each component.

  2. The paper includes competitive baselines, conflict analysis, and source-reliability diagnostics, also it reports cases where simpler methods outperform RA-BSC.

Critical weaknesses
  • Fixed reliability or adaptive MAP often achieves better accuracy and ranking performance , in Tables 2 and 12. The lower-quantile score improves LTT coverage on ARC but reduces it on CommonsenseQA, while no method certifies a nonempty rule on StrategyQA, the added Bayesian complexity is therefore not sufficiently justified.

  • Eq.10 marginalizes the likelihood before normalization, whereas Eqs.11–12 appear to average normalized conditional posteriors, these operations are generally not equivalent, this distinction matters because posterior scores are central to the proposed method.

Questions for authors
  1. Which posterior computation is actually implemented: the formulation in Eq.10or the posterior averaging in Eq.11, does this choice affect the reported results?

2.report paired uncertainty estimates for the LTT coverage differences against fixed reliability and evaluate stricter risk targets, such as α\alpha = 0.1?

Reviewer_w4NAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RA-BSC, a post-hoc aggregator for self-consistency traces. For each question, the distinct extracted answers and an "explicit null state" compete in a categorical model; the null state represents the estimated probability that the correct answer "was missed by candidate extraction". The prior on the null state is estimated from training-split candidate recall (Eq. 1).

There are five evidence sources: per-generator vote fractions from three Qwen2.5 models, plus normalized answer-verifier and process-verifier scores. Each source enters through a mixture likelihood: with probability r_j it points at the true answer, otherwise it is uniform background (Eq. 5). The reliability r_j = σ(μ_j + γ_jᵀh(E)) (Eq. 4) depends on a three-dimensional question-level conflict summary h(E) (Eqs. 2-3).

The 20 parameters (5 sources × (1 + 3)) are fit by MAP on a 20% training split. A diagonal Laplace approximation (Eq. 9) gives posterior draws. The selective score is either the posterior-mean probability of the winning answer or its 10% lower quantile (Eqs. 11-12). A Bonferroni/Clopper-Pearson Learn-Then-Test (LTT) threshold, chosen on a disjoint 30% calibration split over a fixed 21-point grid, provides risk control at α = 0.2 (Eq. 13).

Experiments:

  • 77,940 traces: 20 samples from each of Qwen2.5-1.5B/3B/7B on 299 ARC-Challenge questions and the first 500 questions of CommonsenseQA and StrategyQA.
  • Phi-3-mini serves as both verifiers.
  • Results are averaged over 20 random splits.
  • A staircase ablation compares fixed reliability, adaptive MAP and full RA-BSC (Table 2).
  • High-conflict analyses come in three forms: a fixed pooled top-25% set (Table 5), a question-clustered bootstrap (Table 6), and split-wise conflict quartiles (Table 12).
  • Audits cover the diagonal vs. full Hessian, source duplication and likelihood tempering (Tables 13-15).
  • An ARC run replaces Qwen2.5-1.5B with Gemma-2-2B (Table 16).

The paper itself concludes that point-prediction gains come from structured reliability rather than Bayesian marginalization (Sec. 5.2). It positions marginalization as contributing "uncertainty-aware selective information" (abstract). LTT certificates are nonempty on ARC and CSQA, but no method certifies StrategyQA (Table 7).

Critical strengths
  • Correct risk-control protocol and candid ablations. The LTT setup is valid (Sec. 3.3, Eq. 13). The staircase ablation (Table 2), the question-clustered bootstrap (Table 6) and the Hessian, duplication and tempering audits (Tables 13-15) are the right experiments for separating structured reliability from Bayesian integration. The paper reports them even when they go against the method: on high-conflict questions, RA-BSC is significantly worse than fixed reliability on ARC (−0.020, CI [−0.034, −0.007]) and than adaptive MAP on StrategyQA (−0.035, CI [−0.060, −0.010]) (Table 6). This makes the empirical picture trustworthy, even though that picture is mixed.
  • Structured source reliability helps under generator conflict. On top-quartile conflict questions, RA-BSC beats Bayesian softmax by +0.073 (CI [0.018, 0.129]) on ARC and +0.067 (CI [0.018, 0.117]) on CSQA (Table 6). Against fixed reliability and adaptive MAP, the same table shows differences that are zero or negative. As Sec. 5.3 states, the gain therefore comes from the reliability parameterization, not from the Bayesian layer. This is the paper's most solid positive finding.
Critical weaknesses
  • The defining Bayesian component is not shown to add consistent value, and the selective-ranking evidence is internally inconsistent.

    • Point prediction. By the paper's own account, the MAP variants match or beat posterior marginalization (Sec. 5.2, Tables 2 and 6).
    • AURC. RA-BSC (mean) never beats pooled majority or verifier-weighted SC on a main benchmark. On ARC it scores 0.054 vs 0.040 / 0.038, on CSQA 0.107 vs 0.074 / 0.071, and on StrategyQA 0.254 vs 0.252 / 0.248 (Table 1). It beats fixed reliability only on ARC (0.054 vs 0.060), and loses to it on CSQA (0.066) and StrategyQA (0.238) (Table 2). Its AURC wins are over the softmax baselines on StrategyQA only (0.272 / 0.270; on ARC both softmaxes are better at 0.046, Table 1), deterministic fusion on ARC and StrategyQA (0.083 / 0.287), and adaptive MAP on ARC (0.061) (Table 2).
    • AURC vs. LTT coverage. The two metrics tell different stories.
      • RA-BSC (mean) far exceeds vote fraction on LTT coverage (0.882 / 0.611 vs 0.791 / 0.421 on ARC / CSQA; Table 9). It is still not the best reliability variant: adaptive MAP reaches 0.885 on ARC and fixed reliability 0.632 on CSQA. Only RA-BSC q10 is highest on ARC, at 0.898 (Table 7).
      • The q10 score moves the two metrics in opposite directions (Table 9). On ARC it raises LTT coverage (0.882 to 0.898) while AURC (lower is better) worsens slightly (0.054 to 0.055). On CSQA it improves AURC (0.107 to 0.091) while LTT coverage drops (0.611 to 0.546).
      • The two metrics may weight different parts of the curve. In Fig. 2 (left), RA-BSC has the lowest risk only for coverage of roughly 0.6-1.0 (all curves are tied near 0.55-0.6), which is where the α = 0.2 LTT thresholds operate on ARC (mean coverage 0.79-0.90, Tables 7 and 9). AURC weights the whole curve, including the low-coverage region where RA-BSC is not the best. This explanation is tentative, because the plotted baselines do not match Table 1 (see Soundness).
      • Mean coverage is also confounded by certificate counts. The averages appear to include zero-coverage splits, as StrategyQA's 0/20 certificates and 0.000 mean coverage suggest. On CSQA the certificate counts differ by method (16, 16, 15, 15 and 13 of 20; Table 7).
    • The figure and the table disagree. The only risk-coverage plot does not match the tabulated AURCs (see Soundness). In that plot, a small number of the top-ranked RA-BSC predictions are wrong: the curve spikes at coverage below about 0.05 and is clipped at 0.25 (Fig. 2, left). It is unclear whether this holds across splits. The Hessian, duplication, tempering and Gemma audits (Tables 13-16; see Soundness) add that RA-BSC's ranking is sensitive to how the posterior is approximated and how the likelihood is scaled.

    The title, the method name and the conclusion all rest on the Bayesian layer, so this is the main reason for my rating.

  • No positioning against source-reliability label models, and the key baselines are undefined.

    • Jointly inferring the true label and item-dependent source reliability is the core of Dawid-Skene (1979), GLAD (Whitehill et al., 2009), Raykar et al. (2010) and Snorkel-style label models (Ratner et al., 2016; 2019). None of these appears among the paper's 17 references. LLM verifier-weighted voting work such as DiVeRSe (Li et al., 2023) and Universal Self-Consistency (Chen et al., 2023) is also missing. Sec. 2 mentions "related Bayesian SC and stopping formulations" without citations.
    • The comparators behind the headline gains, MAP and Bayesian softmax, are never defined. Sec. 4 only names them, and p. 4 (after Eq. 9) hints at a "feature-based candidate softmax".
    • Without a Dawid-Skene/GLAD-style baseline on the same five sources, the novelty claim in Sec. 2 cannot be assessed. The scope also limits generality: 299-500 closed-form questions per benchmark, generators of 7B or smaller, and a single Phi-3-mini model as both verifiers, so the two verifier channels are likely dependent.
Questions for authors

please see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.