acceptodds
Under review as a conference paper at ICLR 2027

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

Abstract

Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood and outcome , with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
forecasting, uncertainty calibration, uncertainty estimation, llm

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C1XaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on the challenge of improving the calibration and uncertainty estimation of LLM-based probabilistic forecasting. Existing approaches typically rely on binary event outcomes to improve point predictions, while overlooking the richer information contained in human forecasts, particularly the distribution of beliefs and the degree of agreement among forecasters.

To address this limitation, the authors propose the Beta-Bernoulli Calibrator (BBC), model-agnostic framework that transforms an LLM's initial probability estimate into a mixture of Beta distributions. Experiments across multiple LLMs and forecasting datasets demonstrate improved forecasting accuracy, calibration, and uncertainty estimation compared with existing baselines.

Critical strengths

Strengths:

1 The paper addresses an important problem in probabilistic forecasting. Incorporating human forecast distributions and knowledge as supervisory signals is an interesting and potentially valuable direction.

2 The proposed framework is flexible and does not require access to the parameters or internal representations of the input LLMs which allows it to be easy applied to current existing models.

3 The experimental evaluation is comprehensive, covering multiple LLM settings, forecasting-specialized baselines, an external dataset, and ablation studies. The experiment results are convincing.

Critical weaknesses

Major Comments:

1 The paper states that initial forecasts are generated at each question’s opening date. However, some evaluated LLMs were released after certain questions had already been resolved. This raises concerns about the temporal consistency of the evaluation and potential information leakage. Thus, these forecasts cannot have been generated prospectively with the evaluated model. How do authors deal with potential temporal leakage in the historical input forecasts?

2 Section 6.1 evaluates uncertainty quality by examining the relationship between predicted Beta variance and Brier score. While this is a useful diagnostic, the observed correlation does not necessarily establish that BBC captures epistemic uncertainty. I suggest to compare BBC variance against simple uncertainty baselines.

3 The interpretation of the learned variance as epistemic uncertainty needs stronger support. The supervision signals are not directly comparable: Metaculus provides individual forecasts recorded over an event’s lifetime, whereas Polymarket provides a time series of market prices. Additional explanation would be good.

Minor Comments:

1 Please clarify the KL-divergence implementation in Section 4.3. The human forecasts are represented as a discrete histogram, whereas the predicted Beta mixture is continuous.

2 Typo mistake in Figure 2(c). “Calibartor” should be “Calibrator.”

Questions for authors

1 Could the authors clarify when and how the initial forecasts for historical questions were generated? In particular, were they recorded prospectively at the time each question opened, or generated retrospectively?

4 comments

We thank the reviewer for the constructive comments and for recognizing the importance of the problem, the flexibility of the framework, and the comprehensive evaluation. We address each concern below. We have also uploaded a revised paper, with changes marked in blue.


Major 1 & Q1: Retrospective forecasts and temporal leakage

The forecasts were generated retrospectively, with the prompt conditioned on each question's open date. We now clarify this in the revised paper (Section 5.1 and Appendix E.3). To prevent leakage, we split the data temporally by resolution date so that test outcomes are not part of the models' parametric knowledge. That is, all test questions resolve from Aug 2025 onward, beyond the knowledge cutoff of every model we evaluate. Question open dates, however, span Feb 2016 to Dec 2025 (median Jun 2025). Therefore, we prevent outcome leakage but not partial knowledge: for questions that open before a model's cutoff, the model may know some intermediate developments, though never the final answer. The table below reports the share of test questions that open after each model's cutoff.

Table D. Share of the 1,614 test questions that open after each model's knowledge cutoff.

ModelKnowledge cutoffOpened after cutoff
Claude-Sonnet-4Mar 202573.0%
Llama-3.3-70BDec 202397.8%
Qwen3-32B / 8B*Apr 202564.3%
BBC backbone (Llama-3.2-1B)Dec 202397.8%

*Qwen3 has no official knowledge cutoff, so we use its release date.


Major 2: Evaluating the predicted variance

We agree that the correlation in Section 6.1 does not establish that BBC captures epistemic uncertainty. For a calibrated forecast, the expected Brier score is E[(p^−y)2]=p^(1−p^)\mathbb{E}[(\hat p-y)^2]=\hat p(1-\hat p), which does not depend on the variance u^=p^(1−p^)/(α^+β^+1)\hat u=\hat p(1-\hat p)/(\hat\alpha+\hat\beta+1). The variance is largest for forecasts near 0.5, which also have the largest expected Brier score, so the trend can arise from the predicted mean alone. That is, for a fixed p^\hat p, changing u^\hat u does not change the probability of the outcome, so no outcome-based metric can test u^\hat u, whatever methods or baselines produce it.

Testing the accuracy of epistemic uncertainty requires the true event probability, which cannot be observed in real events. We therefore add a synthetic setting with known latent distributions, where regimes share a mean but differ in variance. In Table A (shown as Figure 3 in the revised paper), we see that a model trained on binary outcomes alone predicts nearly the same variance for both regimes of a pair. Adding human supervision separates them and approximately recovers the true values. Human forecasts thus provide additional supervision about variance that outcomes alone cannot.

Table A. True and predicted variance of pp in the synthetic setting, averaged over the test questions of each regime.

RegimeTrue varianceBinary onlyBinary + human
Beta(5,5), mean 0.50.02270.10560.0238
Beta(50,50), mean 0.50.00250.10550.0056
Beta(5,1), mean 0.830.01980.03370.0209
Beta(50,10), mean 0.830.00230.03480.0037

We have revised Section 6.1 accordingly in the updated pdf. We now present Figure 2 as showing that BBC's uncertainty is consistent with its forecasting errors, and we state the caveat above explicitly. We also revised the statements in introduction and conclusion to avoid overclaiming.


Major 3: Human forecast signals on Metaculus and Polymarket

Our goal is to utilize human forecasts, an underexplored source of supervision, to assist training, although they are noisy proxies of a ground truth that we cannot observe. On Metaculus, these are the beliefs of individual forecasters. On Polymarket, each price is already an aggregate over the traders active at that time, and individual beliefs are not observed. It is true that due to the nature of the two platforms, the levels of aggregation differ, but both collect human beliefs about the probability of the event over the question's active period.

For the interpretation of the learned variance, on both platforms the model is trained to predict the distribution of later, better-informed human beliefs for a question, so the learned variance reflects how far these beliefs may end up from the current forecast. The Polymarket signal understates the part of this spread that comes from individual disagreement. We agree that the two signals are not equivalent in this respect, and we note both this difference and the mixing of disagreement and belief changes in the limitations (Appendix A). For the evidence supporting the interpretation of the variance, please see our response to Major 2.

​​Minor 1: KL-divergence implementation

We discretize the predicted density on the bins of the human histogram. hi\mathbf h_i has B=100B=100 equal-width bins on [0,1][0,1] with centers cb=(b−0.5)/Bc_b=(b-0.5)/B. We smooth it with a Gaussian kernel (standard deviation of one bin) and normalize it to sum to one. We then evaluate the density of the predicted mixture at the bin centers, fi(cb)=∑kw^ikBeta(cb;α^ik,β^ik)f_i(c_b)=\sum_{k}\hat w_{ik} \mathrm{Beta}(c_b;\hat\alpha_{ik},\hat\beta_{ik}), and minimize

Lhuman,i=−∑b=1Bhiblog⁡fi(cb).\mathcal L_{\text{human},i}=-\sum_{b=1}^{B}h_{ib} \log f_i(c_b).

This is the cross-entropy between the histogram and the predicted density evaluated at the bin centers, which equals KL(hi∥fi)\mathrm{KL}(\mathbf h_i \Vert f_i) up to the entropy of the histogram, a constant that does not depend on the model.


Minor 2: Typo

We thank the reviewer for catching this. We have fixed it in the revised paper.

Thank you to the authors for the additional experiments and detailed responses. I believe they have adequately addressed my concerns, particularly regarding potential data leakage. I will therefore raise my rating.

We appreciate the reviewer's recognition of our work and the time and effort spent on the review. We believe the constructive feedback has improved our paper.

Reviewer_zxxGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Beta-Bernoulli Calibrator (BBC), a second-stage model that takes a forecasting question and an initial probability from an LLM and predicts a mixture of Beta distributions over the event probability. A small language model with an MLP head is trained using binary-outcome cross-entropy and distributional supervision from human forecast histograms. The mixture mean provides the calibrated forecast, while its variance is interpreted as epistemic uncertainty. Experiments on temporally split Metaculus and Polymarket data, together with cross-platform evaluation on Kalshi, demonstrate improvements over raw forecasts and several calibration baselines. Ablations investigate calibrator backbones, input content, and the contribution of human distributional supervision.

Critical strengths
  1. Practical compatibility with black-box forecasters, supported by empirical improvements. BBC requires neither access to the input model’s internal representations nor updates to its parameters. In Table 1, calibrating Claude-Sonnet-4 reduces Brier score from 0.146 to 0.125 and improves AUC from 0.723 to 0.742. The Kalshi evaluation provides useful evidence that the approach can also improve predictions under a platform shift. These results support the practical value of learning a separate, task-specific correction model.
  2. The ablations meaningfully examine the sources of improvement. Table 5 shows that removing the initial forecast reduces AUC from 0.744 to 0.650 in the Claude-based setting, indicating that BBC benefits from the input forecaster rather than simply replacing it. Appendix H.3 includes matched-backbone crowd-mean distillation and fixed-spread histogram controls. These experiments provide evidence of additional discrimination benefits from question-specific distributional supervision, while also revealing its limitations. Table 4 further demonstrates that the approach is not restricted to a single calibrator backbone.
Critical weaknesses
  1. The epistemic-uncertainty interpretation is not sufficiently validated. The supervision combines lifetime individual forecasts on Metaculus with market-price trajectories on Polymarket. These distributions mix disagreement with belief changes caused by new information, so their variance need not represent epistemic uncertainty under a fixed information set. Figure 2 shows that predicted variance correlates with Brier error, but this does not establish a separation from aleatoric uncertainty. This leaves a central claim insufficiently supported.
  2. The added value of the Beta-mixture formulation is only partially established. In Table 15, matched crowd-mean-plus-binary distillation nearly matches BBC in Brier score (difference: 0.001, not statistically significant). BBC provides a significant AUC gain of 0.008, but this comparison covers only Claude-Sonnet-4. Without broader matched controls or alternative distributional heads, the evidence more strongly supports learning from human forecasts than the specific benefit of the Beta-Bernoulli formulation.
Questions for authors
  • What precise information set is the predicted distribution conditioned on, and under what assumptions should a lifetime crowd-belief histogram be interpreted as a distribution over event probability at the forecast timestamp? How are changes caused by new information distinguished from uncertainty under fixed information?

  • Beyond the variance-versus-Brier association, what evidence distinguishes BBC’s variance from a measure of crowd disagreement or event difficulty? In particular, is there an analysis that controls for the predicted mean, since binary-outcome variability itself depends on the event probability?

  • Do the gains over the matched crowd-mean-plus-binary baseline persist across other input LLMs and on Kalshi? Is there evidence that the Beta-mixture head offers an advantage over a directly predicted histogram or another distributional head when the backbone, supervision, and training budget are held fixed? These comparisons would directly affect my assessment of the distributional contribution.

2 comments

We thank the reviewer for the detailed review and for recognizing BBC's practical compatibility with black-box forecasters and the value of our ablations. We address each concern below. We have also uploaded a revised paper, with changes marked in blue.


W1: Validity of the epistemic-uncertainty interpretation

The supervision distribution. We agree that our current construction of human distributions mixes disagreement with belief changes caused by new information, so the variance is not epistemic uncertainty under a fixed information set in the strict sense. That notion would require forecasts made at the forecast date, and neither platform provides enough of them. On Metaculus, forecasts arrive gradually: a training question receives a median of 125 forecasts from 58 forecasters, at a median rate of about 4 per day, which is typically too few to form a distribution on a single day. On Polymarket, the market price reflects aggregated beliefs, not individual ones. We therefore use the forecasts made over a question's active period, which we treat as noisy samples of pp. What the model learns is the spread of these later, better-informed beliefs around the current forecast. This spread is the uncertainty that gets resolved as information arrives over that period, which is the sense in which we call it epistemic. We now note this in the limitations (Appendix A).

Validation of the variance. We also agree with the reviewer that Figure 2 cannot separate epistemic from aleatoric uncertainty. For a calibrated forecast, the expected Brier score is E[(p^−y)2]=p^(1−p^)\mathbb{E}[(\hat p-y)^2]=\hat p(1-\hat p), which does not depend on the variance u^=p^(1−p^)/(α^+β^+1)\hat u=\hat p(1-\hat p)/(\hat\alpha+\hat\beta+1). The variance is largest for forecasts near 0.5, which also have the largest expected Brier score, so the trend can arise from the predicted mean alone.

We have revised Section 6.1 accordingly in the updated pdf.

Figure 2. We now present it as showing that BBC's uncertainty is consistent with its forecasting errors, and we state the caveat above explicitly. We also revised the statements in introduction and conclusion to avoid overclaiming.

New synthetic experiment (Figure 3). Testing the accuracy of epistemic uncertainty requires the true event probability, which cannot be observed in real events. We therefore add a synthetic setting with known latent distributions, where regimes share a mean but differ in variance. In Table A (shown as Figure 3 in the revised paper), we see that a model trained on binary outcomes alone predicts nearly the same variance for both regimes of a pair. Adding human supervision separates them and approximately recovers the true values. Human forecasts thus provide additional supervision about variance that outcomes alone cannot.

Table A. True and predicted variance of pp in the synthetic setting, averaged over the test questions of each regime.

RegimeTrue varianceBinary onlyBinary + human
Beta(5,5), mean 0.50.02270.10560.0238
Beta(50,50), mean 0.50.00250.10550.0056
Beta(5,1), mean 0.830.01980.03370.0209
Beta(50,10), mean 0.830.00230.03480.0037

Q1: Information set of the predicted distribution

Information set. The predicted distribution is conditioned on the calibrator input xix_i (Section 4.2): the question text and the input LLM's initial forecast, which is generated as of the question's open date.

Assumption. We treat the human forecasts made over a question's active period as noisy samples of pip_i given xix_i. These forecasts are made after the forecast timestamp, so the learned distribution describes later, better-informed beliefs given xix_i. It is not a distribution under the information available at the forecast timestamp, as we discuss in our response to W1.

New information vs. fixed information. We do not distinguish the two. The histogram pools disagreement among forecasters with belief changes caused by new information, and neither platform provides enough forecasts at a single time point to separate them (see our response to W1). The predicted variance should therefore be read as their combined spread, which we now state in the limitations (Appendix A).


Q2: What the predicted variance captures

Theoretically, for a calibrated forecast, the expected Brier score is E[(p^−y)2]=p^(1−p^)\mathbb{E}[(\hat p-y)^2]=\hat p(1-\hat p), which does not depend on the variance u^=p^(1−p^)/(α^+β^+1)\hat u=\hat p(1-\hat p)/(\hat\alpha+\hat\beta+1). For a fixed p^\hat p, changing u^\hat u does not change the probability of the outcome, so no outcome-based metric can test u^\hat u, whatever method produces it. We therefore show the synthetic experiment with known ground truth pp (see our response to W1), where we do see such evidence: for regimes with the same mean (the same event difficulty), human supervision recovers the difference in variance, and binary outcomes alone do not.

W2 & Q3: Benefit of the Beta-Bernoulli formulation

Matched control across input LLMs and Kalshi. As the reviewer suggested, we extend the crowd-mean-plus-binary baseline to the other input LLMs and to Kalshi. From Table B, we see that the pattern of Table 15 holds for every input LLM: Brier is matched, with differences of at most 0.002 in either direction, and BBC has higher AUC by 0.008 to 0.010 (p=0.007p=0.007 averaged over the four). On Kalshi (Table C), the comparison is mixed: for Qwen3-8B, BBC is better in Brier and AUC, while for Qwen3-32B the baseline is better in Brier and AUC is not significantly different.

Table B. Δ\Delta = BBC minus the crowd-mean-plus-binary baseline on the test set, with the same setup as Table 15. One-sided paired bootstrap (B=1,000B=1{,}000); ∗p<0.05^{\ast}p<0.05, ∗∗p<0.01^{\ast\ast}p<0.01.

Input LLMΔBrier ↓ΔAcc ↑ΔAUC ↑ΔECE ↓
Claude-Sonnet-4 (Table 15)-0.001+0.003+0.008*+0.003
Llama-3.3-70B+0.002-0.001+0.009+0.017
Qwen3-32B+0.001-0.007+0.009*+0.009
Qwen3-8B-0.000+0.002+0.010+0.005
Mean over the four+0.000-0.001+0.009**+0.009

Table C. BBC vs. the crowd-mean-plus-binary baseline comparison on OOD Kalshi for the two Qwen3 models. Δ\Delta and tests as in Table B; ∗∗∗p<0.001^{\ast\ast\ast}p<0.001.

Input LLMΔBrier ↓ΔAcc ↑ΔAUC ↑ΔECE ↓
Qwen3-32B+0.007-0.007-0.006+0.025
Qwen3-8B-0.002*+0.001+0.021***+0.026

Alternative distributional head. We also compare our mixture-of-Beta head with a directly predicted histogram. The same MLP head outputs a softmax over the 100 bins of the human histogram, and the forecast is the histogram mean. All other settings are unchanged. In Table E, we see that BBC has higher AUC for both input LLMs. For Claude-Sonnet-4 it is also better in Brier and accuracy (p=0.039p=0.039 and p=0.022p=0.022); for Llama-3.3-70B these are tied. The histogram head has a lower KL to the human histograms, as expected from its 100 free bins against the 15 parameters of the mixture.

Table E. Beta-mixture head vs. histogram head on the test set, under the setup of Table 15 (3 seeds each). KL is computed against the human histograms as in Table 1.

Input LLMHeadBrier ↓Acc ↑AUC ↑ECE ↓KL ↓
Claude-Sonnet-4Beta mixture (BBC)0.12550.8360.7440.0368.95
Histogram0.12690.8300.7330.0267.56
Llama-3.3-70BBeta mixture (BBC)0.13560.8300.6790.0609.97
Histogram0.13610.8300.6680.0508.95

The benefit of the Beta-Bernoulli formulation. We agree that the crowd mean explains much of the improvement in Brier score. This is expected: the crowd mean is the first-order signal that human forecasts add beyond the binary outcome. Learning the distributional shape adds a small but consistent gain in discrimination, and it gives a distribution over p, which a point forecast cannot provide. In the synthetic experiment with known ground truth (see our response to W1), this distributional supervision is what allows the variance of pp to be learned. Under the same budget, BBC also has higher AUC than a directly predicted histogram for both input LLMs. We therefore see the Beta-Bernoulli formulation as a natural and compact way to model this distribution. Independently of this choice, we believe the pipeline is itself a contribution: post-hoc calibration with human forecasts as supervision is an easy and lightweight method that substantially improves on the input LLM.

Reviewer_3zMJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Beta-Bernoulli Calibrator for LLM forecasting. Given a question and an initial LLM probability estimate, a lightweight language model predicts a mixture of Beta distributions, whose mean serves as the calibrated forecast and whose variance is intended to capture epistemic uncertainty. BBC is trained using binary outcomes and human forecast distributions. Experiments on Metaculus and Polymarket report improvements in forecasting and calibration, with additional evaluation on Kalshi showing cross-platform generalization.

Critical strengths
  1. The paper addresses a practically relevant problem. Improving the calibration of LLM forecasts is important, and using human forecasts as additional supervision is a reasonable direction.

  2. The approach supports black-box forecasting models. BBC requires only the question and an initial probability estimate, without access to the input model’s parameters. A relatively small 1B calibrator already provides useful improvements.

  3. BBC improves Brier scores across several input models and retains benefits on different platform. Ablations on human supervision, calibrator size, and input content help clarify which components contribute to performance.

Critical weaknesses
  1. The human supervision is temporally misaligned with the forecast. The LLM predicts at question opening, whereas the human distribution covers the question’s entire lifetime and incorporates later information. The calibrator may therefore learn to compensate for this information gap rather than capture uncertainty at the forecast date. This is particularly concerning for Polymarket price histories, whose variation reflects changing information over time.

  2. The uncertainty evaluation does not distinguish epistemic from outcome uncertainty. Figure 2 shows that predicted variance correlates with Brier loss. However, events with probabilities near 0.5 naturally have higher expected Brier loss, even when their probabilities are known perfectly.

  3. The additional value of modeling the human distribution is limited. Human supervision provides relatively small gains over binary-only BBC. Appendix H.3 also shows that crowd-mean distillation with binary supervision closely matches BBC in Brier score, although BBC improves AUC. The paper should more clearly establish what practical benefit comes from learning distributional shape beyond the crowd mean.

  4. The temporal leakage controls need clarification. The paper states that test questions resolve after the models’ knowledge cutoffs, but forecasts are made for the question opening date. Please clarify whether these opening dates also fall after the corresponding models’ knowledge cutoffs.

Questions for authors

N/A

4 comments

We thank the reviewer for the thoughtful review and for recognizing the practical relevance of the problem, the black-box compatibility of BBC, and the ablation studies. We provide further clarifications and additional analyses below. We have also uploaded a revised paper, with changes marked in blue.


W1: Temporal alignment of human supervision

We agree that the human supervision is not aligned with the forecast date: the human distributions are collected during a question's active period and incorporate later information. The learned distribution is therefore not the distribution of beliefs at the forecast date in the strict sense. That would require forecasts made at the forecast date, and neither platform provides enough of them. On Metaculus, forecasts arrive gradually: a training question receives a median of 125 forecasts from 58 forecasters, at a median rate of about 4 per day, which is typically too few to form a distribution on a single day. On Polymarket, the market price reflects aggregated beliefs, not individual ones, so the histogram captures only how the price evolves after the forecast date, as the reviewer points out. We therefore use the forecasts made over a question's active period, which we treat as noisy samples from the distribution over p. What the model learns is the distribution of these later, better-informed beliefs around the current forecast, which reflects the information gap between the forecast date and later beliefs. This is not the same as uncertainty defined only by what is known at the forecast date, and we now state in the limitations how the variance should be interpreted (Appendix A).

Still, this distribution is useful supervision, and the two signals trade off bias and variance. The binary outcome is an unbiased target for the event probability, but it is a single draw per question with high variance. Later human forecasts have much lower variance, although they can be biased. Learning from both improves the Brier score and AUC over training on outcomes alone (Table 1).


W2: Epistemic vs. outcome uncertainty

We thank the reviewer for this point, and we agree. Since the variance of a Beta distribution is p^(1−p^)/(α^+β^+1)\hat p(1-\hat p)/(\hat\alpha+\hat\beta+1), it is largest for forecasts near 0.5, which also have the largest expected Brier score. The trend in Figure 2 can therefore arise from the predicted mean alone, and it does not separate epistemic from outcome uncertainty.

We have revised Section 6.1 accordingly in the updated pdf.

Figure 2. We now present it as showing that BBC's uncertainty is consistent with its forecasting errors, and we state the caveat above explicitly. We also revised the statements in introduction and conclusion to avoid overclaiming.

New synthetic experiment (Figure 3). Testing the accuracy of epistemic uncertainty requires the true event probability, which cannot be observed in real events. We therefore add a synthetic setting with known latent distributions, where regimes share a mean but differ in variance. In Table A (shown as Figure 3 in the revised paper), we see that a model trained on binary outcomes alone predicts nearly the same variance for both regimes of a pair. Adding human supervision separates them and approximately recovers the true values. Human forecasts thus provide additional supervision about epistemic uncertainty that outcomes alone cannot.

Table A. True and predicted variance of p in the synthetic setting, averaged over the test questions of each regime.

RegimeTrue varianceBinary onlyBinary + human
Beta(5,5), mean 0.50.02270.10560.0238
Beta(50,50), mean 0.50.00250.10550.0056
Beta(5,1), mean 0.830.01980.03370.0209
Beta(50,10), mean 0.830.00230.03480.0037

W3: The benefit of learning the distributional shape

We agree that the crowd mean explains much of the improvement in Brier score. This is expected: the crowd mean is the first-order signal that human forecasts add beyond the binary outcome. Compared with the crowd-mean-plus-binary baseline, learning the distributional shape adds a small but consistent gain in discrimination (Table B extends this comparison to the other input LLMs), and it gives a distribution over pp, which a point forecast cannot provide. In the synthetic experiment with known ground truth (see our response to W2), this distributional supervision is what allows the variance of pp to be learned. We therefore see the Beta-Bernoulli formulation as a natural and compact way to model this distribution. Independently of this choice, we believe the pipeline is itself a contribution: post-hoc calibration with human forecasts as supervision is an easy and lightweight method that substantially improves on the input LLM.

Table B. Δ\Delta = BBC minus the crowd-mean-plus-binary baseline on the test set, with the same setup as Table 15. One-sided paired bootstrap (B=1,000B=1{,}000); ∗p<0.05^{\ast}p<0.05, ∗∗p<0.01^{\ast\ast}p<0.01.

Input LLMΔBrier ↓ΔAcc ↑ΔAUC ↑ΔECE ↓
Claude-Sonnet-4 (Table 15)-0.001+0.003+0.008*+0.003
Llama-3.3-70B+0.002-0.001+0.009+0.017
Qwen3-32B+0.001-0.007+0.009*+0.009
Qwen3-8B-0.000+0.002+0.010+0.005
Mean over the four+0.000-0.001+0.009**+0.009

W4: Temporal leakage control

We thank the reviewer for raising this point and clarify below. The forecasts were generated retrospectively, with the prompt conditioned on each question's open date. We split the data temporally by resolution date so that test outcomes are not part of the models' parametric knowledge. That is, all test questions resolve from Aug 2025 onward, beyond the knowledge cutoff of every model we evaluate. Question open dates, however, span Feb 2016 to Dec 2025 (median Jun 2025). Therefore, we prevent outcome leakage but not partial knowledge: for questions that open before a model's cutoff, the model may know some intermediate developments, though never the final answer. The table below reports the share of test questions that open after each model's cutoff. We have clarified this in the revised paper (Section 5.1 and Appendix E.3).

Table D. Share of the 1,614 test questions that open after each model's knowledge cutoff.

ModelKnowledge cutoffOpened after cutoff
Claude-Sonnet-4Mar 202573.0%
Llama-3.3-70BDec 202397.8%
Qwen3-32B / 8B*Apr 202564.3%
BBC backbone (Llama-3.2-1B)Dec 202397.8%

*Qwen3 has no official knowledge cutoff, so we use its release date.

I appreciate the response and additional experiments, which partially address my concerns. I think the temporal mismatch still limits the interpretation of the learned uncertainty, and distributional supervision mainly improves AUC, without consistent gains in Brier score or calibration. Therefore, I maintain my original score. I thank the authors for their efforts and look forward further refinement of the work.

We sincerely thank the reviewer for the follow-up and for the careful evaluation of our work. The comments have helped us state the scope of our claims more clearly.

We believe the pipeline itself is a useful contribution: an easy and effective method for post-hoc calibration, which improves on the input LLM across models and generalizes to Kalshi. On the remaining points, our aim is to make use of human forecasts, an underexplored source of supervision for LLM forecasting, which we treat as noisy samples of the event probability. We agree that they are not aligned with the forecast date. Neither platform provides enough forecasts made at the forecast date, so forecasts over the question's active period are the closest signal available, and training on them improves both Brier score and AUC over training on binary outcomes alone. The crowd mean accounts for most of this gain, and the distributional shape adds further discrimination on top of it. We therefore hope the reviewer will see the temporal mismatch as a limitation of the current data, which we now state in the paper, while the method and its empirical gains still stand.

Thank you again for the constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.