acceptodds
Under review as a conference paper at ICLR 2027

Empirical-Bayes Truth Tilting for Conformal Inference in Large Language Model Question Answering

Abstract

Large language models (LLMs) are increasingly used as answer engines, yet their tendency to hallucinate makes it important to quantify uncertainty in their answers. Open-ended question answering provides a tractable yet challenging setting for such uncertainty quantification (UQ): sampled responses can be organized into semantic answer clusters, but these clusters are prompt-specific rather than shared across inputs. Each prompt therefore induces a local semantic alphabet of observed answer clusters and an “everything else” atom for unseen answers. Existing query-only conformal methods use missing-mass estimators to assess whether the true answer lies beyond the sampled responses, but they largely equate sampling frequency with truth probability and estimate each instance in isolation. We propose a nonparametric empirical Bayes approach that borrows strength across these apparently unrelated local alphabets. Within each prompt, a Pitman–Yor species model estimates the semantic sampling law; across labeled prompts, an empirical-Bayes truth-tilting layer maps sampling behavior and prompt features to posterior truth mass. The resulting posterior over semantic clusters defines a high-probability nonconformity score, and conformal calibration restores marginal validity. Extensive experiments demonstrate that our Bayesian modeling substantially improves both semantic uncertainty estimation and the downstream conditional coverage of the conformal predictive set for LLM question answering.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, large language model, empirical bayes, conditional coverage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z5YaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conformal prediction for LLM question answering where the correct answer may remain unseen. The proposed BAYES-STATE method combines a Pitman–Yor sampling reference with a shared truth-tilting correction learned from labeled historical questions. The proposed method constructs scores over observed answer clusters and an everything-else category, followed by split conformal calibration. The authors establish finite-sample marginal coverage under exchangeability and bound excess mean squared conditional coverage error in terms of posterior approximation error under a margin condition. Experiments on short-answer QA benchmarks evaluate score quality and conformal performance under matched supervision, while a separate cold-start study examines uncertainty in future sampling counts.

Critical strengths
  1. The paper considers several practical aspects of LLM question answering, including prompt-specific answer spaces, frequent but incorrect responses, and the possibility that the correct answer is absent from the sampled candidates. These considerations inform the local semantic alphabet, the everything-else category, and the learned truth correction. The resulting formulation captures difficulties that arise in the LLM question answering problem.

  2. The experiments cover two QA benchmarks, two LLMs, and different query budgets. Comparisons under matched supervision and component ablations help assess the contributions of truth learning and the sampling reference.

Critical weaknesses
  1. My main concern is whether the reported improvements lead to more useful answer sets. At the primary coverage levels, all observed answer clusters are retained, and the full model produces the same sets as the gate-only version. The lower groupwise coverage error is also accompanied by more fallback use and larger displayed sets. Although these tradeoffs are acknowledged, the results leave unclear how much the full truth-tilting model improves answer selection beyond learning when to abstain.
Questions for authors
  1. Could the authors clarify the finite-sample coverage guarantee given the complex score construction and prompt-specific semantic clustering? A brief explanation of which components are fixed after fitting and why the calibration and test scores remain exchangeable would help readers understand why the standard conformal result applies.

  2. At the primary coverage levels, the full model and the gate-only version return the same prediction sets. When learned ranking changes the sets at lower coverage levels, does it reduce the number of displayed answers or improve coverage at similar fallback rates?

  3. GT+Tilt and Direct-State perform comparably to BAYES-STATE under matched supervision. In what QA settings would the PYP reference improve the final answer sets? It would be helpful to connect its cold-start sampling benefits to answer coverage or fallback behavior.

1 comment

Thank you for the detailed response. I'll keep my positive score.

Reviewer_BueTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces an empirical-Bayes framework for conformal prediction in open-ended QA, combining a Pitman-Yor reference over observed and unseen answer clusters with a truth-tilting model trained on labeled prompts. Experiments show that truth supervision improves answer reliability over CPQ, while the Pitman-Yor component mainly helps future-count prediction when historical data are scarce.

Critical strengths

1. Clear problem formulation. The paper cleanly separates where the LLM tends to sample from where the true answer lies, and learns this correction across prompts without needing shared answer identities.

2. Honest ablations. The matched-supervision comparison (Table 1) reports results that weaken the paper's own PYP component, which makes the empirical claims easy to trust.

Critical weaknesses

1. The contribution is learned abstention, and this literature is missing. Sec. 4.3 shows that at the primary targets, the learned seen-answer ranking changes no decision and every method includes all K observed clusters, so CPQ and BAYES-STATE output the same literal set and differ only in whether EE is included. The method is therefore a learned "abstain or not" classifier and that is then conformalized. This is selective prediction and conformal abstention, e.g. Geifman & El-Yaniv (2017) and Yadkori et al. (2024). LLM confidence estimation such as Kadavath et al. (2022), whose p(True) probe is close to the absence head, is also missing. The paper should be reframed as learned abstention for conformal QA and positioned against this work.

2. Baselines are too weak. The only external baseline is CPQ, whose score uses no truth labels, while BAYES-STATE trains its score on 250 labels, so the results mainly show that supervision beats no supervision. Three standard alternatives are missing: semantic entropy (Kuhn et al., 2023; cited but unused), a p(True)-style correctness score (Kadavath et al., 2022), calibrated on the same labels; and sequence-likelihood confidence where log-probabilities are available. Each can replace the learned score within the same conformal pipeline.

3. The MSCE gain may come from abstaining more. BAYES-STATE abstains more and displays larger sets in all eight settings (Sec. 4.3), and since a fallback is counted as covered, abstaining more on hard groups mechanically pulls their coverage toward the target and lowers MSCE. Results should be reported at matched abstention rates or along an abstention–MSCE curve, together with coverage on non-abstained prompts.

4. PYP is unnecessary for the main results yet dominates the method. Under truth supervision (Table 1), Direct-State without any reference has the lowest NLL, PYP+Tilt is worse than GT+Tilt, and MSCE is essentially equal. Its main advantage appears in Sec. 4.4, which predicts how many future responses fall outside the seen clusters, and even there it holds only when very few historical prompts are available. Since this task uses no correctness labels and does not affect the conformal sets, the case for building the method around PYP is unclear.

5. The theory is mostly standard and does not support the method. Prop. 3.1 states that, with an unrestricted tilt, any change of reference can be exactly offset by changing the tilt, and implies that truth labels cannot determine the reference, which does not supports the role of PYP. Theorem 3.2 is the textbook split-conformal argument.

6. Presentation is hard to follow. The text reads like a revision log ("archived", "retained"), with caveats in almost every paragraph. Notation collides: G_x denotes the PYP measure and G_T the admission set, σ denotes both the PYP discount and a sigma-algebra, and α is used before it is defined. Sections 2 and 3 overlap, key equations are unnumbered, and tables do not highlight the best results. The evaluation is also limited to two short-answer datasets of 1,000 prompts each, with the clustering judge being one of the two generators.

Questions for authors

See Weakness.

1 comment

We sincerely thank Reviewer BueT for the thoughtful follow-up. We apologize for replying later than expected: we spent additional time conducting and auditing matched-reference experiments on the original NQ-Open/PopQA QA tasks. The central question is whether PYP contributes to truth-risk estimation and QA decisions, beyond sampling forecasts. We identify a positive regime: limited truth supervision with equal fallback budgets.

1. Why the new ablation tests PYP's role directly.

Our earlier fresh-QA reference swap fixed the EE/absence gate across PYP, GT, and uniform references. It tested seen-answer offsets, but not whether the reference helps learn the fallback gate. In the new experiment, each reference enters the model and the complete truth head is refitted. PYP, GT, and reference-free Direct share the architecture, features, admission loss, labels, and fitting-only CV.

PYP uses 250 unlabeled occupancy histories per cell to estimate its species parameters; at M=50M=50, only 50 labeled fitting families per dataset train the truth head. Evaluation covers two datasets, two generators, and T=10,20T=10,20: 2,000 test families / 8,000 correlated views. No new generation or truth labels were collected.

2. PYP improves decisions when fallback resources are matched.

We reserve exactly 15% of questions for fallback in every test cell. All seen atoms are displayed unless fallback is chosen, isolating fallback allocation rather than seen ranking. Operational error means a non-deferred question with no acceptable atom in the displayed set; fallback contributes zero displayed atoms.

M=50M=50; 15% fallbackDirectGTPYPPYP − Direct [95% CI]
Operational error41.60%39.84%39.50%−2.10 pp [−3.075, −1.175]
EE Brier0.234870.253370.22184−0.01304 [−0.01921, −0.00643]
Displayed atoms/question2.293251.9371.88950−0.40375 [−0.48413, −0.33000]
Admission NLL0.786160.937360.76390−0.02226 [−0.03945, −0.00407]

All methods use the same 1,200 fallback slots across 8,000 views. Compared with Direct, PYP makes 168 fewer erroneous non-fallback decisions and displays 17.6% fewer atoms per question on average. All eight cell-level error differences favor PYP; NQ-Open and PopQA each show −2.10 pp when aggregated separately. These are results on the original semantic QA task.

3. GT comparison and stronger regularization.

At M=50M=50, PYP also improves probability estimation against the matched GT arm: admission NLL 0.76390 versus 0.93736 (PYP−GT −0.17346, 95% CI [−0.19482, −0.15318]), and EE Brier 0.22184 versus 0.25337 (difference −0.03153 [−0.03600, −0.02697]). PYP−GT operational error remains unresolved (−0.3375 pp [−0.8628, +0.2000]).

Because the initial ridge choices reached the upper grid boundary, we extended the same fitting-only CV grid for all three methods from {0.01,0.1,1}\{0.01,0.1,1\} to {0.01,0.1,1,4,16,64}\{0.01,0.1,1,4,16,64\}.

The decision benefit remains: PYP error 39.0875% versus Direct 41.7000%, a difference of −2.6125 pp [−3.6003, −1.6375]. EE Brier improves by −0.00991 [−0.01687, −0.00249], and displayed atoms by −0.44913 [−0.53352, −0.37313]. PYP's NLL/Brier gains over GT persist, while its NLL contrast against Direct becomes unresolved. Our strongest claim concerns budget-matched error, display burden, and EE Brier.

4. Interpretation and scope.

The shared-gate ablation missed the mechanism identified here: an occupancy reference learned without truth labels provides a useful inductive bias when labeled fitting data are scarce.

The practical benefit operates through learning the EE-risk gate and reallocating a fixed number of deferrals, not through filtering seen candidates. This also explains why unrestricted reference–tilt equivalence does not imply equal finite-data performance under restricted heads and regularization.

The advantage is specific to this regime. At M=250M=250, PYP−Direct error at the same quota is −0.80 pp [−1.60, +0.0125]; we do not claim uniform superiority or an improvement in the separately evaluated high-label all-atoms conformal policy.

A fixed 15% fallback quota is a decision evaluation, not a new conformal coverage guarantee, and the outcome measures alias-based set admission, not verified correctness of a displayed representative or fallback response. These results are exploratory analyses of a previously examined QA bank; paired intervals resample question families conditional on fitted models, rather than providing a fresh independent confirmation.

We hope this directly addresses the requested practical role of PYP: with limited truth labels, the learned sampling reference improves EE-risk estimation and the allocation of the same fallback budget on the original QA tasks.

Reviewer_zoTTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BAYES-STATE for conformal prediction over prompt-specific semantic answer clusters. A Pitman–Yor process estimates sampling mass on observed clusters and an unseen remainder. A supervised “truth tilt” then adjusts these masses using historical prompts, and disjoint split-conformal calibration supplies marginal coverage.

Experiments examine truth-score quality, conformal performance, and short-prefix sampling forecasts. The results indicate that supervised truth correction drives the main fixed-budget improvements, while the PYP reference has a narrower role in cold-start uncertainty propagation.

Critical strengths
  1. Strong component controls support a clear, qualified empirical conclusion - The reference-by-tilt comparison matches supervision and separates truth learning from occupancy estimation. Across eight settings, admission-mass NLL falls from 2.000 for PYP occupancy to 1.527 for PYP+Tilt. The authors also report that GT+Tilt and Direct-State perform slightly better, avoiding an unsupported claim of Bayesian superiority.

  2. The statistical targets and guarantees are distinguished carefully - The paper separates unseen sampling mass from truth absence, marginal coverage from conditional reliability, and fallback from a correct textual answer. The cold-start experiment also compares integrated and plug-in forecasts with identical means, isolating uncertainty propagation rather than attributing the gain to better point prediction.

Critical weaknesses
  1. The distinctive benefit of the empirical-Bayes construction remains limited - Under matched supervision, PYP+Tilt has NLL 1.527 versus 1.515 for GT+Tilt and 1.498 for Direct-State; Brier scores and ranking metrics are also close. The authors acknowledge this, but it leaves the main contribution close to supervised correctness modeling followed by standard conformal calibration. The cold-start result does not fully resolve the issue: its advantage over the occupancy beta-binomial reverses by M=25M=25, and its target is future sampling counts rather than truth or conformal efficiency.

  2. The primary conformal operating points do not demonstrate useful candidate filtering - At the primary targets, every method includes all observed clusters in its literal threshold set, and full versus gate-only prediction produces identical sets in all 480 audited comparisons. The practical distinction is therefore chiefly fallback selection. Relative to CPQ, BAYES-STATE increases both fallback frequency and displayed answer count. Reduced group-MSCE is useful, but does not establish more informative answer sets or a favorable coverage–cost tradeoff.

  3. The implemented learner is not the posterior-fitting procedure used to motivate the framework - The fixed-budget experiments use supervised preliminary tuning and two class-balanced binary heads rather than optimizing Eq. 6. As Appendix A.2 acknowledges, class weighting changes the fitting population, and classifier odds are not automatically truth-to-sampling density ratios. Conformal validity survives this mismatch, but the interpretation as an estimated truth posterior—and its connection to the posterior-error theorem—needs stronger empirical support. A likelihood-based implementation would be an important comparison.

  4. The conditional-reliability evidence has a restricted scope - Some retained groupings and worst-slab diagnostics use method-dependent features, limiting direct comparisons of subpopulations. The common frozen-group audit is a useful correction, but it reuses historical cohorts, and its intervals condition on saved fits rather than including refitting uncertainty. Moreover, operational coverage credits fallback as the full alphabet. These results should therefore be interpreted as evidence about within-population operational reliability, not verified answer correctness or a conditional-coverage guarantee.

Overall, the paper provides useful analysis and unusually candid attribution experiments. Nevertheless, its strongest results establish the value of shared correctness supervision, while the distinctive PYP component does not improve the main truth-estimation comparisons. The primary conformal decisions are also dominated by fallback gating rather than selective answer-set construction. Stronger evidence connecting the proposed statistical formulation to practical prediction-set utility would strengthen the case for acceptance.

Questions for authors
  1. In what practically relevant regime does PYP+Tilt improve on GT+Tilt or Direct-State beyond the scarce-history sampling-count task?

  2. Can you demonstrate better coverage–cost tradeoffs where seen-answer ranking changes the prediction set, ideally across a broader range of coverage targets?

  3. How does direct optimization of the normalized admission objective compare with the class-balanced two-head implementation?

  4. Can the main results be confirmed on a frozen, previously unused cohort, with common groups and uncertainty estimates that include refitting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.