- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to post-process LLM-generated marginal and pairwise-intersection probabilities so that they are jointly realizable by a single probability distribution. The method represents coherent probabilities as mixtures of binary outer products and solves a globally rescaled projection problem through column generation, combining exact binary quadratic optimization with nonnegative least squares. The output includes an explicit joint-distribution certificate, together with a tolerance-dependent optimality guarantee for the stated objective. Experiments evaluate four LLMs on coin and dice problems with known ground truth, reporting reduced RMSE in 35 of 40 model–premise settings.
Critical strengths
-
The method provides global realizability of the supplied marginal and pairwise probabilities, an explicit joint-distribution certificate, and a certified optimality gap for the stated projection objective with global rescaling. These guarantees provide a principled advantage over local consistency repairs and require neither model retraining nor ground-truth labels.
-
The controlled experiments provide evidence that coherence enforcement can improve probability accuracy. RMSE decreases in 35 of 40 settings, with the average falling from 0.307 to 0.185. Using problems with exact ground truth allows the evaluation to distinguish improvements in coherence from improvements in accuracy.
Critical weaknesses
-
The practical benefit for recent frontier models remains unclear. The evaluation covers Gemma 2, Qwen 2.5, GLM-4.5, and GPT-OSS-120B, but does not establish how much incoherence remains in more recent frontier models or whether the proposed correction improves their probability estimates. This matters because projection increases RMSE in all five GPT-OSS coin-flip settings, where the raw estimates are already comparatively accurate. The current evidence therefore leaves the method’s practical benefit with stronger initial estimates insufficiently established.
-
The empirical comparison lacks recent LLM-specific probability-repair methods. Relevant alternatives include Joint-Coherent Decoding (JCD) [1], which projects LLM probabilities onto coherent polytopes defined by small logical cliques, and ArbitrageForecaster [2], which repairs forecasts through arbitrage-based consistency checks. The paper discusses ArbitrageForecaster but does not evaluate it, and does not discuss or compare against JCD. These methods enforce different consistency constraints from the full pairwise coherence considered here. A direct empirical comparison with these methods would be helpful, or clarification of why such a comparison is not applicable.
References
[1] Arbitrage-Free Forecasts from Language Models via Coherence Projection. Forecast@ICML 2026. https://openreview.net/pdf?id=Tqos7VqQhH
[2] Paleka et al. Consistency Checks for Language Model Forecasters. ICLR 2025. https://arxiv.org/abs/2412.18544
Questions for authors
-
How do the degree of incoherence and the accuracy gains from projection change when using more recent frontier models?
-
How does the proposed method compare empirically with JCD and ArbitrageForecaster under comparable inputs and inference budgets, or what differences prevent a meaningful comparison?