acceptodds
Under review as a conference paper at ICLR 2027

Beyond Binary Correctness: Confidence Estimation for Continuously Graded Responses in Large Language Models

Abstract

Confidence estimation predicts the correctness of generated outputs of LLMs. Most existing approaches typically use an arbitrary threshold over a continuous quality metric, such as syntactic or semantic similarity against ground truth, to determine whether an LLM response is correct. In this paper, we propose modeling confidence as a probability distribution over continuous correctness scores, thereby remaining agnostic to threshold choice, avoiding information loss from binarization, and enabling rich uncertainty estimates such as confidence variance and credible intervals for robust decision making. We propose Beta regression and piecewise uniform regression approaches as distributional methods, showing theoretically that Beta regression achieves superior sample efficiency and faster convergence for smooth distributions with at most one interior mode. We also conduct an empirical study on several datasets across diverse tasks to analyze and compare methods with each other, as well as traditional point-based confidence estimation baselines, demonstrating their relative suitability and advantages.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
confidence estimation, uncertainty quantification, large language models, distributional regression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AMeVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors study confidence estimation for LLMs, and specifically for tasks where model outputs are continuously graded.

Instead of treating correctness as binary (labels yy are either 00 or 11) it is continuous (y∈[0,1]y \in [0,1]), and the authors thus formulate confidence estimation as a probabilistic regression problem.

They evaluate different regression methods on 7 datasets across 4 different tasks, with 2 LLMs per task.

Critical strengths
  • The studied problem is interesting, I have never seen anyone formulating confidence estimation as probabilistic regression (predicting p(y∣x)p(y|x)) for continuous correctness labels y∈[0,1]y \in [0,1] before. This is relevant both for someone interested in LLM confidence estimation/evaluation, and someone interested in probabilistic regression.

  • The paper is well written and easy to follow overall, I think the proposed approach makes sense, and the experimental evaluation covers multiple different datasets/tasks and two different LLMs per task.

Critical weaknesses
  • The paper evaluates different models/methods within the proposed approach of probabilistic regression for continuous correctness labels, but the practical utility of this approach itself hasn't really been demonstrated. It remains unclear what this actually adds in practice compared to the common approach of treating correctness as binary.
Questions for authors

Questions/suggestions:

  • Can you directly compare your proposed continuous approach with the standard binary approach in some way, to demonstrate what this adds in practice? Binarize the correctness labels and implement some common method, directly compare with this as a baseline?

  • The paper evaluates different methods (Beta Reg, Quantile Reg, etc) within the proposed continuous approach for confidence estimation, but how would you actually apply this in practice? For example, to determine whether a given LLM output should be trusted or if the model should abstain? Or, to compare different LLMs in terms of reliability/trustworthiness? You have results for both LLM #1 and LLM #2 in the paper, can you use your approach to evaluate which of these is the best?

  • Also, if using your approach e.g. to evaluate and compare different LLMs, does the choice of regression method matter? Could e.g. Beta Reg give that LLM #2 is better than #1, while Quantile Reg gives that #1 is better than #2? If so, how do you decide which regression method to use?

  • Section 6, "validating that Beta regression methods achieve superior performance with fewer samples": Has this really been evaluated properly though? Could do a controlled evaluation, by creating subsets of the data (e.g. 10%, 25%, 50%, 100%) and comparing how the performance scales with more samples?

  • Section 5.2 - Selective Prediction: Why specifically this Selective-MAE@50% metric (why not e.g. 25% or 10% instead)? Could it not make more sense to evaluate using sparsification plots and/or the AUSE metric, instead of arbitrarily picking one threshold?

  • Section 5.2 - Selective Prediction: You write that "Table 3 shows Hurdle-Beta achieves lower selective error on XSum and HotpotQA" and "on multimodal BigCodeBench, Random Forest is comparable". Perhaps not quite fairly worded? 0.051 vs 0.053 on XSum --> "Beta achieves lower", but 0.322 vs 0.333 on BigCodeBench --> "Random Forest is comparable"? If Beta is better on XSum, then Random Forest is better on BigCodeBench?

Minor things:

  • The caption for Table 2 should probably be above the table instead.
  • Section 2, "Kadavath et al. (2022)elicit verbalized": formatting.
  • Section 4, "is empirically validated in Zhang et al. (2020)": use \citep instead of \citet here?


Clarification of rating:

I found this paper interesting and quite enjoyed reading it. I think the studied problem and proposed approach are interesting and overall make sense. I want to be able to recommend accept for this paper. But, right now I'm quite borderline on it, and I think some clarifications will be required during the rebuttal/discussion.

4 comments

We thank the reviewer for valuable feedback and the careful reading, including the formatting points, and for finding the problem interesting. We respond to specific questions below and describe edits in our revised document to incorporate the feedback.

Q1 (comparison with the binary approach): This comparison is already in the paper. Multi-threshold binary regression is the standard binary approach: logistic regression on correctness labels binarized at a threshold, applied here at five thresholds (0.1, 0.3, 0.5, 0.7, 0.9) with the same features and splits (Section 3.3). CRPS is the Brier score of the threshold event averaged over all thresholds, so Table 1 compares how well each method predicts P(y≥τ∣x)P(y\ge\tau \mid x). Beta and Hurdle-Beta have lower CRPS than multi-threshold on both LLMs for all three datasets (within one standard deviation on BigCodeBench and for HotpotQA with LLM 2), with one model for all thresholds instead of one per threshold. Calibrated data-free confidence scores are further standard baselines (Appendix Tables 4 to 10). We will also include a per-threshold AUROC and Brier comparison in an appendix.

Q2 / Q3 combined (use in practice, choice of method): To decide whether to trust a response, one computes its predicted distribution and abstains or escalates when P(y≥τ∣x)P(y\ge\tau \mid x) falls below a cutoff or the predicted interval is wide. Both τ\tau and the cutoff can be chosen after deployment, per user or task, without retraining, which a binary classifier cannot offer. Selective-MAE (Section 5.2) shows that predicted uncertainty identifies where predictions are reliable.

The choice of regression method is determined mainly by the task's score distribution, not by the LLM (Fig. 2): Hurdle-Beta when mass sits near 1, plain Beta for smooth unimodal scores, and piecewise methods for multimodal or discrete scores, which can be checked on training scores before fitting. Consistently, within each dataset, the same methods are best on ECE, IS and CRPS for both LLMs in Table 1, up to differences within one standard deviation. Since methods differ mainly in their uncertainty estimates and much less in their predicted means on smooth data (Section 5.1), we expect comparisons of LLMs by predicted mean correctness to agree across methods there. Because the regressors are supervised, one can also fit the same regression for each LLM on the same queries and compare the resulting distributions, which could inform the choice of LLM when that is a consideration. We have expanded the discussion of practical use in Section 6.

Q4 (Section 6 claim): We agree that the evidence for this statement is limited, so we have removed it from the Conclusion. The controlled evaluation that the reviewer suggests is Fig. 4(A) and Appendix D.1: training subsets of 100 to 4000 generations on HotpotQA, now referenced in Section 5.2. All methods converge by about 1000 generations (about 200 queries), and Hurdle-Beta is worse than the others at the smallest sizes. The theoretical statement is restricted to plain Beta versus an equal-width histogram (Theorem 1).

Q5 (retention level): The 50% level is a median-retention summary, not a tuned choice, and the metric is defined for any retention rate. The reviewer is correct that a retention curve is more informative, and we will add Selective-MAE at other retention levels (e.g., 25% and 75%) or a sparsification curve as suggested.

Q6 (Table 3 wording): The reviewer is correct. Table 3 reports LLM 1, where the gaps are 0.051 vs. 0.053 on XSum and 0.333 vs. 0.322 on BigCodeBench, both within one standard deviation, while HotpotQA shows a larger gap (0.166 vs. 0.181). With LLM 2, Random Forest is comparable or better (Appendix Tables 4, 8, 10), so the evidence does not support a consistent advantage for Hurdle-Beta in selective prediction, and we have reworded the text accordingly.

We are glad the reviewer found the problem and approach interesting. If these clarifications resolve the open points, we would be grateful for the reviewer supporting the paper in the discussion. We thank them again for helpful comments and suggestions.

Thank you for the response.

I have read the other reviews and all author responses.

My view of the paper remains similar to my original review. I think my main concern ("the practical utility of this approach itself hasn't really been demonstrated. It remains unclear what this actually adds in practice compared to the common approach of treating correctness as binary") mostly remains.

"Or, to compare different LLMs in terms of reliability/trustworthiness? You have results for both LLM #1 and LLM #2 in the paper, can you use your approach to evaluate which of these is the best?", I don't really think this question from my review has been adressed.

"The reviewer is correct that a retention curve is more informative, and we will add Selective-MAE at other retention levels (e.g., 25% and 75%) or a sparsification curve as suggested", I do think some new result for this should be added already in the rebuttal.

I will keep my "3: Weak acceptance" score for now, but I remain quite borderline on the paper.

We thank the reviewer for the follow-up. We were unable to run new experiments during this period as we focused heavily on reworking some theoretical statements, but we appreciate the suggestions (such as around Selective-MAE) and will include them as mentioned previously. Here we clarify two points.

Practical utility compared with binary correctness. In our approach, a single fitted model replaces a set of binary classifiers, each tied to a threshold chosen in advance. This model can answer whether a response clears any quality threshold, including one chosen after deployment, and also gives interval estimates for flagging uncertain cases. Because it is one distribution, its answers for different thresholds are consistent with each other, which separately fitted classifiers do not guarantee. We see this as a practical benefit in itself, in addition to the richer representation of a distribution for continuous correctness.

Comparing LLMs. It may help to clarify the paper's scope. Our primary goal is per-query confidence estimation for a given LLM, not ranking LLMs. The estimators are fit separately for each LLM and each fit is evaluated for calibration. The point is to make the approach agnostic to the choice of LLM, so that the method can be deployed for any LLM. As the reviewer inferred, since each LLM gets its own predicted distribution over correctness, the framework also lends itself to comparing LLMs on the same queries.

We are grateful for the reviewer's feedback and hope they will continue to support the paper in the final discussion.

Thank you again.

I will keep my "3: Weak acceptance" score but remain quite borderline on the paper.

Reviewer_mS2HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLM confidence estimation usually binarizes a continuous quality score (BERTScore, ROUGE, unit-test pass rate) at an arbitrary threshold. It proposes instead to predict the conditional distribution p(y∣x)p(y\mid x) of a correctness score y∈[0,1]y\in[0,1] from black/gray-box features: query and response statistics, mean token log-probability, and Jaccard similarities across 5 sampled generations (Sec. 3.1). Four models are compared: Beta regression with logit-linear mean and constant precision ϕ\phi; Hurdle-Beta, which adds a point mass on [0.99,1][0.99,1]; multi-threshold logistic regression at τ∈{0.1,0.3,0.5,0.7,0.9}\tau\in\{0.1,0.3,0.5,0.7,0.9\}; and quantile regression (Sec. 3.2–3.3). Theorem 1 (Sec. 3.4, App. B) compares Beta with kk-bin histogram estimators and concludes that Beta is preferable for smooth unimodal distributions at moderate nn, and piecewise methods for multimodal ones. Experiments use 7 datasets (HotpotQA, NQ, TriviaQA, CNN/DM, XSum, Biographies, BigCodeBench) with two LLMs each, five metrics (MAE, continuous ECE, Interval Score@90%, CRPS, Selective-MAE@50%) and 5 random 80/20 splits. Hurdle-Beta receives 55% of the win points in Table 2(a) (ties within 1% are split; Hurdle-Beta coincides with Beta on XSum, CNN/DM and Biographies).

Critical strengths
  1. A useful reframing of the problem. Predicting a distribution for a continuous correctness score, so that P(y≥τ)P(y\ge\tau) is available for any τ\tau without retraining, addresses a real practice (thresholding continuous metrics, Sec. 1, Fig. 1A). This is the paper's contribution: the four estimators are standard methods (Beta regression, one-inflated mixtures, per-threshold logistic regression, quantile regression).

  2. Broad, transparent evaluation. Seven datasets spanning QA, summarization, long-form generation and code, proper scoring rules (CRPS, IS) next to MAE/ECE, mean ±\pm std over 5 splits, and full appendix tables (Tables 4–10) that let readers check the summary claims.

Critical weaknesses
  1. The main claimed benefit, richer uncertainty-aware confidence, is not demonstrated, and, for plain Beta, the model as specified cannot deliver it.

    • ϕ\phi is constant (Sec. 3.2; the authors report numerical instability with variable precision), so the predictive variance μ(1−μ)/(1+ϕ)\mu(1-\mu)/(1+\phi) of the Beta component is a function of the predicted mean alone. Inputs with similar μ\mu get similar interval widths, which is the case Fig. 1(B) uses as motivation. Hurdle-Beta adds a mixture weight π1(x)\pi_1(x) and so is not subject to this exactly, but on XSum, CNN/DM and Biographies it coincides with Beta.
    • In the only test of whether predicted uncertainty ranks errors (Selective-MAE@50%), plain Beta is worse than its full MAE: XSum/Granite 0.057 vs 0.054 (Table 8, a gap about one std); HotpotQA 0.232 vs 0.223 and 0.226 vs 0.219 (Table 4). Hurdle-Beta does improve on its own MAE on HotpotQA and BigCodeBench (Tables 4, 10). Table 3 compares Hurdle-Beta only with Random Forest, and on XSum/Llama Quantile (0.048) and multi-threshold (0.050) beat Hurdle-Beta (0.051, Table 8).
    • On XSum and CNN/DM, linear regression is within 0.001 of Beta on MAE and ECE (Tables 1, 7, 8). No binary classifier at a fixed τ\tau is ever evaluated, although the paper argues against that practice, so "threshold-agnostic without loss" is untested.

    This is decisive because it leaves the advantage over existing practice, the paper's central motivation, without empirical support.

  2. The theory is internally inconsistent, and its sample-efficiency message is not supported by the paper's own ablation.

    • The statement and proof of Theorem 1 disagree: the piecewise approximation error is O(k−4)O(k^{-4}) in Part A but O(k−2)O(k^{-2}) in Part B and the proof; the Beta regime is k≤7k\le 7 vs k≤9k\le 9; the multimodal regime is n≥Ck3n\ge Ck^3 (B.1) vs Ck2Ck^2 (proof, Part C(ii)), and the δ\delta threshold in Part C is 1/k41/k^4 in the statement vs 1/k21/k^2 in the proof; Sec. 3.4/B.2 say Beta has 2d2d parameters, but the parenthetical in B.2 (mean weights in Rd\mathbb R^d plus a scalar ϕ\phi) gives d+1d+1. The proof cites an undefined "(A4)" and "Theorem ??" (App. B). The explicit range n∈[100,10000]n\in[100,10000] cannot follow from bounds with unspecified constants.
    • The analysis is for an unconditional equal-width histogram, not for the estimators compared, and the conditional extension (B.2) states rates without proof.
    • Fig. 4(A) (HotpotQA only) does not support the sample-efficiency prediction: at n=100n=100, Hurdle-Beta is above the plot range (>0.20>0.20) while multi-threshold is about 0.175, and at n≈250n\approx 250 it is still worse (about 0.161 vs 0.153). Theorem 1 does not cover the Hurdle-Beta mixture, so this is not a formal refutation, but the Conclusion's claim that Beta methods "achieve superior performance with fewer samples" is not supported.

    Because the theory is presented as one of three main contributions (Sec. 1), this weighs heavily.

Questions for authors
  1. [Clarification] With constant ϕ\phi, the Beta variance depends only on μ\mu. How is the Selective-MAE ranking computed, and why is plain Beta's Selective-MAE@50% higher than its full MAE (Tables 4, 8)? Results with heteroscedastic ϕ(x)\phi(x), or a heteroscedastic Gaussian or point-plus-residual baseline, would directly test whether distributional modeling adds value. This could change my assessment.
  2. [Clarification] The binary methods the paper argues against are never evaluated. How does P(y≥τ)P(y\ge\tau) from Hurdle-Beta or Quantile compare with a logistic classifier trained directly at τ∈{0.3,0.5,0.7}\tau\in\{0.3,0.5,0.7\} on AUROC and Brier score, with the same features?
  3. [Clarification] Please reconcile Theorem 1 with App. B (O(k−4)O(k^{-4}) vs O(k−2)O(k^{-2}), k≤7k\le 7 vs k≤9k\le 9, n≥Ck3n\ge Ck^3 vs Ck2Ck^2, 2d2d vs d+1d+1, assumption (A4)), explain how n∈[100,10000]n\in[100,10000] is derived without constants, and reconcile the sample-efficiency claim with Fig. 4(A).
  4. [Clarification] Sec. 4 uses 1000 HotpotQA queries with an 80/20 split (about 800 training points), but Fig. 4(A) varies nn up to 4000 (App. D.1). Where do the extra samples come from, and could generations of the same query leak between train and test? Table 1 says "temperature zero using stochastic decoding": at temperature 0, are the 5 samples behind the Jaccard features distinct?
  5. [Clarification] In the Hurdle-Beta density (Sec. 3.2), the Beta part on (0,τh)(0,\tau_h) is multiplied by 1−π1(x)1-\pi_1(x), but it is not said whether it is renormalized to (0,τh)(0,\tau_h). If not, the total probability is below 1 and P(y≥τ)P(y\ge\tau) for τ<τh\tau<\tau_h is slightly biased (possibly negligible when the fitted Beta has little mass above τh\tau_h, but this is not stated). Is it renormalized in the code and in the IS, CRPS and ECE computations? Please also report sensitivity to τh\tau_h (e.g., 0.95, 0.99, 0.999).
  6. [Suggestion / future work] Table 2a appears to use one LLM per dataset and awards points only to methods within 1% of the best, regardless of the std over splits (e.g., HotpotQA Llama MAE: multi-threshold 0.206±.0080.206\pm.008 vs Hurdle-Beta 0.198±.0070.198\pm.007 gets 0 points, Table 4). Please report both LLMs and paired tests or CIs over splits.
2 comments

We thank the reviewer for valuable feedback and the detailed reading, particularly around Section 3.2 and Appendix B. We respond to specific questions below and describe edits in our revised document to incorporate the feedback.

W1 / Q1 (uncertainty benefit): The reviewer is correct that with constant precision, the variance of plain Beta depends only on predicted mean μ\mu, so its uncertainty ranking is a ranking by ∣μ−0.5∣\vert \mu-0.5\vert. This explains why plain Beta's Selective-MAE does not improve on its MAE (Tables 4, 8). The uncertainty benefit comes from Hurdle-Beta, whose boundary probability varies with xx, and from the piecewise methods. On HotpotQA (LLM 1), Hurdle-Beta has interval score 0.762 and CRPS 0.127, against 0.824 and 0.154 for plain Beta, and Random Forest has interval score 1.018 (Table 1). We now state in Sections 3.2 and 5.2 that plain Beta is mean-driven. Quantile and multi-threshold are slightly better on XSum (Selective-MAE 0.048 and 0.050 vs. 0.051), and we do not claim Hurdle-Beta wins everywhere. Linear regression matches Beta on MAE and ECE for XSum and CNN/DM but cannot produce intervals or CRPS. Heteroscedastic precision was numerically unstable (Section 3.2); we leave stable heteroscedastic baselines to future work.

Q2 (binary classifiers): The requested comparison is already included in proper-scoring form. Multi-threshold regression is exactly a bank of logistic classifiers, each trained directly at one threshold (τ=0.1,0.3,0.5,0.7,0.9\tau=0.1, 0.3, 0.5, 0.7, 0.9) on the same features, splits and training data, with a monotonicity correction across thresholds. The Brier score of the event y≥τy\ge\tau is the integrand of CRPS, so CRPS is the threshold-averaged Brier score of each method's P(y≥τ∣x)P(y\ge\tau \mid x). Beta and Hurdle-Beta have lower CRPS than multi-threshold on both LLMs for all three datasets. They do so with one model for all thresholds, not one per threshold. AUROC would add a view that ignores calibration and measures ranking at a single threshold; we agree it is complementary and will include it in an appendix.

Q3 / W2 (theory, Fig. 4A): The reviewer is correct on each inconsistency, and we have corrected them in the revised PDF (changes in blue). Lemma 2 is now corrected - the error of piecewise-constant approximation is governed by the first derivative, so the rate is k−2k^{-2}, not k−4k^{-4}, which now agrees with Part B and the proof. We fixed the Beta parameter count, the undefined assumption and the broken reference. The ranges k≤7k\le 7 and n∈[100,10000]n\in[100,10000] could not follow from bounds with unspecified constants, so we replaced the regime conditions by explicit ones. The scope is now clear that the theorem is unconditional and compares plain Beta with an equal-width histogram; Hurdle-Beta is outside scope, and Appendix B.2 provides a heuristic sketch for the conditional case. The reviewer is also correct about Fig. 4(A): on HotpotQA, Hurdle-Beta is worse than the other methods at small nn (about 0.23 vs. 0.175 at n=100n=100), and all methods converge by n≈1000n\approx 1000. The hurdle mixture adds a logistic classifier for the boundary probability, which may be data-hungry (about 28 positives among 100 samples). We removed the claim of superior performance with fewer samples from the Conclusion.

Q4 (data, leakage): There is no leakage as splits are made over queries, so all generations of a query fall entirely in the training or the test set, and the similarity features of a query use only its own generations. Table 1 and Fig. 4(A) both use temperature zero with five generations per query, so the 1000 HotpotQA queries give 5000 generations, and the 80/20 split by query gives 800 training queries and 4000 training generations. This is why Fig. 4(A) goes up to n=4000n=4000. Here nn counts generations, which are correlated within a query, not distinct queries. Note that generations at temperature zero with stochastic decoding (as opposed to greedy decoding) yield differing generations for a query.

Q5 (Hurdle-Beta): The reviewer is correct that this was not clear previously. The Beta component is truncated and renormalized to the interval below the hurdle threshold, and the boundary probability is the mass above it, so the predictive distribution is proper in the IS, CRPS and ECE computations. We now state this in Section 3.2, and we plan to report sensitivity to the hurdle threshold (0.95, 0.99, 0.999).

Q6 (Table 2a): Table 2(a) does indeed use one LLM per dataset and the 1% tie rule ignores variability over splits. Tables 1 and 4 to 10 give mean and standard deviation over five splits for both LLMs. We will state the rule clearly and add both LLMs and paired tests or confidence intervals over splits.

We hope the corrected analysis and the clarifications above address the soundness concerns. If so, we would appreciate the reviewer reconsidering their rating. We thank the reviewer again for helpful comments and suggestions.

As the discussion period is closing, we wanted to remind you that we have uploaded a revised PDF with the changes described in our response. Most of the concerns you raised are addressed there, with the changes highlighted in blue for easy reference. In particular, the revised Section 3.4 and Appendix B, the added scope statements, and the updated Section 5.2 respond to the points raised in your review.

If the revision and our response resolve your main concerns, we would be grateful if you would consider updating your assessment and score. Thank you again for your time and careful review.

Reviewer_2XpCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper models continuous LLM response quality as a probability distribution using Beta, Hurdle-Beta, multi-threshold, and quantile regression. It presents theoretical comparisons and experiments on seven datasets across four task types, showing differences in calibration and distributional fit across tasks.

Critical strengths
  1. Preserving continuous quality captures partial correctness and allows different quality thresholds to be queried without retraining.

  2. The experiments cover diverse tasks and show useful differences between methods, including Hurdle-Beta’s advantage when scores concentrate near the maximum.

Critical weaknesses
  1. Appendix B’s Lemma 2 fails for the density p(y)=1/2+yp(y)=1/2+y on [0,1][0,1]: its second derivative is zero, but its histogram approximation error is nonzero. This invalidates the stated bound and undermines the O(k−4)O(k^{-4}) approximation rate claimed in Theorem 1.

  2. The practical benefit of predicting full distributions remains weakly demonstrated. Table 3 measures errors in predicted quality rather than the actual quality or failure risk of accepted answers, leaving the claims about improved decision making insufficiently supported.

Questions for authors
  1. Can you correct Lemma 2 and clarify which theoretical comparisons in Theorem 1 still hold?

  2. At meaningful quality thresholds, does the full distribution improve calibration or acceptance decisions over classifiers trained directly for those thresholds? A comparison of the quality or failure risk of accepted answers would help establish the practical benefit.

  3. Are train/test splits grouped by query, keeping all responses to the same question in one split?

2 comments

We thank the reviewer for excellent feedback and the careful reading, particularly around Appendix B. We respond to specific questions below and describe the changes we made in our revised document to incorporate the feedback.

W1 / Q1 (Lemma 2): The reviewer is correct - for piecewise-constant approximation, the error is governed by the first derivative and not the second. For p(y)=12+yp(y)=\tfrac12+y, the bin-average error is exactly 1/(12k2)1/(12k^2) although p′′=0p^{\prime\prime}=0. The corrected lemma states that if pp is LL-Lipschitz, then the squared L2L^2 error of its average qq over kk equal bins satisfies ∥p−q∥2≤L2/(12k2)\Vert p-q\Vert^2 \le L^2/(12k^2), which this example attains, and if also p≥c>0p\ge c>0, then KL(p∥q)≤L2/(12ck2)\mathrm{KL}(p\Vert q)\le L^2/(12 c k^2). The approximation rate is therefore O(k−2)O(k^{-2}), not O(k−4)O(k^{-4}).

We rechecked all of Section 3.4 and Appendix B and uploaded a revised PDF with changes in blue. Besides Lemma 2, we modified Lemma 3 (exact decomposition, estimation term k/nk/n), gave a complete proof of Lemma 4, and restated Theorem 1 and its proof so that rates, constants and assumptions agree between the main text and the appendix. The numeric ranges in the old Part C could not follow from bounds with unspecified constants, so we removed them. The conclusions that remain are:

  • piecewise uniform approximates any smooth density arbitrarily well, at rate k−2k^{-2}
  • Beta has irreducible error under multimodality
  • Beta has the smaller estimation term (≈1/n\approx 1/n vs. (k−1)/(2n)(k-1)/(2n)), so it is preferred when pp is close to a Beta density, while piecewise uniform is preferred for multimodal pp and asymptotically

We now state the scope more clearly, which is that the theorem is unconditional and compares plain Beta with an equal-width histogram. Hurdle-Beta and the conditional case are not covered, but Appendix B.2 includes a sketch argument for the latter. Empirical results and the broader conclusions remain unchanged.

W2 / Q2 (practical benefit): Per-threshold classifiers are already a baseline: multi-threshold regression trains logistic classifiers at the thresholds τ=0.1,0.3,0.5,0.7,0.9\tau=0.1, 0.3, 0.5, 0.7, 0.9 on the same features. CRPS averages the Brier score of the event y≤ty\le t over all tt, so Table 1 compares how well each method predicts P(y≥τ∣x)P(y\ge\tau \mid x) across all thresholds. Beta and Hurdle-Beta have lower mean CRPS than multi-threshold on both LLMs for all three datasets.

Beyond this, the distributional approach has practical advantages that a classifier at a fixed τ\tau cannot offer:

  • No threshold chosen in advance. One fitted model answers P(y≥τ∣x)P(y\ge\tau \mid x) for any τ\tau, including thresholds set after deployment or differing across users and tasks. A classifier must be retrained whenever τ\tau changes, and the multi-threshold approach requires fixing a grid beforehand.
  • Richer uncertainty. The same model gives means, variances, quantiles and credible intervals, so responses with similar P(y≥τ∣x)P(y\ge\tau \mid x) but different spread (Fig. 1B) can be distinguished, for example to route wide-interval responses to review.
  • Triage. Selective-MAE@50% (Section 5.2) shows that predicted uncertainty identifies reliable predictions: for Hurdle-Beta, it is below the full MAE for both LLMs on HotpotQA and BigCodeBench (Tables 4 and 10).

We agree that Selective-MAE measures error in predicted quality, not the quality of accepted answers, and we revised Section 5.2 and Appendix C to make this clear.

Q3 (splits): Yes, splits are made at the query level, so all responses to a query fall in one split. We now state this in Section 5.

We hope the corrected analysis and our revision addresses the soundness concern, since the qualitative conclusions remain intact and no empirical result changed. If so, we would appreciate the reviewer reconsidering their rating. We thank the reviewer again for helpful comments and suggestions.

As the discussion period is closing, we wanted to remind you that we have uploaded a revised PDF with the changes described in our response. Most of the concerns you raised are addressed there, with the changes highlighted in blue for easy reference. In particular, the corrected statement and proof of Lemma 2 and the revised theory in Section 3.4 and Appendix B respond directly to your soundness concerns.

If the revision and our response resolve your main concerns, we would be grateful if you would consider updating your assessment and score. Thank you again for your time and careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.