- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focus on the challenge of improving the calibration and uncertainty estimation of LLM-based probabilistic forecasting. Existing approaches typically rely on binary event outcomes to improve point predictions, while overlooking the richer information contained in human forecasts, particularly the distribution of beliefs and the degree of agreement among forecasters.
To address this limitation, the authors propose the Beta-Bernoulli Calibrator (BBC), model-agnostic framework that transforms an LLM's initial probability estimate into a mixture of Beta distributions. Experiments across multiple LLMs and forecasting datasets demonstrate improved forecasting accuracy, calibration, and uncertainty estimation compared with existing baselines.
Critical strengths
Strengths:
1 The paper addresses an important problem in probabilistic forecasting. Incorporating human forecast distributions and knowledge as supervisory signals is an interesting and potentially valuable direction.
2 The proposed framework is flexible and does not require access to the parameters or internal representations of the input LLMs which allows it to be easy applied to current existing models.
3 The experimental evaluation is comprehensive, covering multiple LLM settings, forecasting-specialized baselines, an external dataset, and ablation studies. The experiment results are convincing.
Critical weaknesses
Major Comments:
1 The paper states that initial forecasts are generated at each question’s opening date. However, some evaluated LLMs were released after certain questions had already been resolved. This raises concerns about the temporal consistency of the evaluation and potential information leakage. Thus, these forecasts cannot have been generated prospectively with the evaluated model. How do authors deal with potential temporal leakage in the historical input forecasts?
2 Section 6.1 evaluates uncertainty quality by examining the relationship between predicted Beta variance and Brier score. While this is a useful diagnostic, the observed correlation does not necessarily establish that BBC captures epistemic uncertainty. I suggest to compare BBC variance against simple uncertainty baselines.
3 The interpretation of the learned variance as epistemic uncertainty needs stronger support. The supervision signals are not directly comparable: Metaculus provides individual forecasts recorded over an event’s lifetime, whereas Polymarket provides a time series of market prices. Additional explanation would be good.
Minor Comments:
1 Please clarify the KL-divergence implementation in Section 4.3. The human forecasts are represented as a discrete histogram, whereas the predicted Beta mixture is continuous.
2 Typo mistake in Figure 2(c). “Calibartor” should be “Calibrator.”
Questions for authors
1 Could the authors clarify when and how the initial forecasts for historical questions were generated? In particular, were they recorded prospectively at the time each question opened, or generated retrospectively?
4 comments
Minor 1: KL-divergence implementation
We discretize the predicted density on the bins of the human histogram. has equal-width bins on with centers . We smooth it with a Gaussian kernel (standard deviation of one bin) and normalize it to sum to one. We then evaluate the density of the predicted mixture at the bin centers, , and minimize
This is the cross-entropy between the histogram and the predicted density evaluated at the bin centers, which equals up to the entropy of the histogram, a constant that does not depend on the model.
Minor 2: Typo
We thank the reviewer for catching this. We have fixed it in the revised paper.
Thank you to the authors for the additional experiments and detailed responses. I believe they have adequately addressed my concerns, particularly regarding potential data leakage. I will therefore raise my rating.
We appreciate the reviewer's recognition of our work and the time and effort spent on the review. We believe the constructive feedback has improved our paper.
We thank the reviewer for the constructive comments and for recognizing the importance of the problem, the flexibility of the framework, and the comprehensive evaluation. We address each concern below. We have also uploaded a revised paper, with changes marked in blue.
Major 1 & Q1: Retrospective forecasts and temporal leakage
The forecasts were generated retrospectively, with the prompt conditioned on each question's open date. We now clarify this in the revised paper (Section 5.1 and Appendix E.3). To prevent leakage, we split the data temporally by resolution date so that test outcomes are not part of the models' parametric knowledge. That is, all test questions resolve from Aug 2025 onward, beyond the knowledge cutoff of every model we evaluate. Question open dates, however, span Feb 2016 to Dec 2025 (median Jun 2025). Therefore, we prevent outcome leakage but not partial knowledge: for questions that open before a model's cutoff, the model may know some intermediate developments, though never the final answer. The table below reports the share of test questions that open after each model's cutoff.
Table D. Share of the 1,614 test questions that open after each model's knowledge cutoff.
*Qwen3 has no official knowledge cutoff, so we use its release date.
Major 2: Evaluating the predicted variance
We agree that the correlation in Section 6.1 does not establish that BBC captures epistemic uncertainty. For a calibrated forecast, the expected Brier score is E[(p^−y)2]=p^(1−p^), which does not depend on the variance u^=p^(1−p^)/(α^+β^+1). The variance is largest for forecasts near 0.5, which also have the largest expected Brier score, so the trend can arise from the predicted mean alone. That is, for a fixed p^, changing u^ does not change the probability of the outcome, so no outcome-based metric can test u^, whatever methods or baselines produce it.
Testing the accuracy of epistemic uncertainty requires the true event probability, which cannot be observed in real events. We therefore add a synthetic setting with known latent distributions, where regimes share a mean but differ in variance. In Table A (shown as Figure 3 in the revised paper), we see that a model trained on binary outcomes alone predicts nearly the same variance for both regimes of a pair. Adding human supervision separates them and approximately recovers the true values. Human forecasts thus provide additional supervision about variance that outcomes alone cannot.
Table A. True and predicted variance of p in the synthetic setting, averaged over the test questions of each regime.
We have revised Section 6.1 accordingly in the updated pdf. We now present Figure 2 as showing that BBC's uncertainty is consistent with its forecasting errors, and we state the caveat above explicitly. We also revised the statements in introduction and conclusion to avoid overclaiming.
Major 3: Human forecast signals on Metaculus and Polymarket
Our goal is to utilize human forecasts, an underexplored source of supervision, to assist training, although they are noisy proxies of a ground truth that we cannot observe. On Metaculus, these are the beliefs of individual forecasters. On Polymarket, each price is already an aggregate over the traders active at that time, and individual beliefs are not observed. It is true that due to the nature of the two platforms, the levels of aggregation differ, but both collect human beliefs about the probability of the event over the question's active period.
For the interpretation of the learned variance, on both platforms the model is trained to predict the distribution of later, better-informed human beliefs for a question, so the learned variance reflects how far these beliefs may end up from the current forecast. The Polymarket signal understates the part of this spread that comes from individual disagreement. We agree that the two signals are not equivalent in this respect, and we note both this difference and the mixing of disagreement and belief changes in the limitations (Appendix A). For the evidence supporting the interpretation of the variance, please see our response to Major 2.