- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the coherence of LLM probability forecasts through a bounded-stake Dutch-book linear program. Its optimal value equals the minimum sup-norm distance to the set of coherent forecasts, enabling evaluation without realized outcomes. Experiments cover 15 models and 100 stock-days, with detailed interventions conducted on GPT-OSS-120B. The results document substantial variation across models, additional inconsistencies in joint-event forecasts, and strong dependence on elicitation protocol. Full-panel elicitation nearly eliminates the workhorse’s incoherence, while some contextual additions substantially increase it.
Critical strengths
-
An exact diagnostic with an interpretable certificate. For the specified event space, the LP determines whether the reported probabilities admit a common distribution and produces a portfolio witnessing any incompatibility. Its dual quantifies the smallest maximum probability adjustment needed to restore coherence. This is a useful evaluation criterion when outcomes remain unresolved.
-
Informative elicitation comparisons with relevant controls. The study includes date-clustered uncertainty estimates, repeated elicitations, complementary and less-related question groupings, and fixed-size linked/dispersed panels. The contrast between isolated and full-panel elicitation demonstrates that measured coherence can depend substantially on how related forecasts are requested.
Critical weaknesses
-
The paper does not establish the empirical value of its principal methodological distinction. Paleka et al. (2025) already formulate consistency through worst-case arbitrage, although with a different scoring geometry, and Andrews (2026) supplies the bounded-stake formulation used here. The contribution therefore rests on what exhaustive evaluation reveals empirically. That case is missing. In the one-day setting, all 14 probabilities are determined by four bin masses with only three degrees of freedom; normalization and consistency of unions with those masses already characterize coherence. Figure 1 illustrates an ordinary three-event additivity failure. The paper does not show whether the full LP reveals consequential failures beyond local checks in the richer settings, or whether its additional coverage changes any substantive conclusion.
-
The strongest findings remain tied to an unexplained, unusually coherent baseline. All detailed interventions concern GPT-OSS-120B, selected partly for its favorable preliminary coherence. The study does not establish whether its baseline forecasts largely reproduce empirical bin frequencies, which are coherent by construction, or whether the interventions cause inconsistent departures from that procedure. This distinction matters for interpreting what the experiments teach about probabilistic forecasting. The baseline comparison across 15 models does not resolve the issue because those models are not evaluated under the same interventions. Moreover, the approximately 100-fold range in incoherence provides limited evidence of diagnostic superiority by itself: raw Brier scores include outcome uncertainty, so their relative spread is not directly comparable.
Questions for authors
- Under the same total-stake normalization, how much additional profit does the full LP identify beyond portfolios restricted to complement, additivity, and marginalization relations?