- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies weak-to-strong aggregation for probabilistic event forecasting: whether forecasts from two individually weaker LLM forecasters can be combined to match or outperform the strongest individual in a comparison group. Using ForecastBench, the authors construct shared-coverage groups and evaluate learned linear and log-odds pooling against several fixed aggregation rules. The main analysis covers 70 forecasters, 16 groups, and 1121 eligible pair occurrences. Learned linear pooling produces at least one pair matching or improving on the strongest individual in 11 groups and coming within 5% in all 16. Additional experiments examine calibration, larger pools, question categories, individual recalibration, and cost/access constraints. The benchmark and successful pairs are identified retrospectively using test performance.
Critical strengths
-
A clear, practically motivated question on real data at reasonable scale. Asking whether cheaper, open-weight, or otherwise accessible models can stand in for an inaccessible frontier forecaster matters to practitioners. The study uses real, resolved ForecastBench data across 70 forecasters and 1121 pairs.
-
Broad comparisons and transparent reporting of the experimental setup. The authors construct maximal groups with explicit coverage thresholds and report results at both group and pair level. They include many fixed baselines, including robust-aggregation rules that are rarely tested empirically, and report bootstrap CIs and fitted coefficients for every best pair. This transparency is what allows the concerns below to be checked from the paper's own tables.
Critical weaknesses
- The learned aggregates receive supervised post-processing on historical data, whereas the strongest individual benchmark is evaluated without comparable recalibration. This supports the paper’s stated comparison against raw individual forecasts, but does not establish how much of the improvement requires combining complementary information. The reported linear coefficients suggest that probability rescaling may contribute substantially: for example, the weights sum to 0.6646 in overall G3 and 0.6819 in G13. Moreover, in Finance G4 (Table 17), the reported weights are (0.5378, 0.0000), so the best “pair” effectively reduces to a rescaled single model at the reported precision.
Appendix D.2 partially addresses this concern through per-model Platt scaling, but only for weaker forecasters, using log loss rather than the aggregation objective of Brier score, and a different transformation family. A more directly matched control would fit single-model probability scaling using the same training Brier objective for both weaker models and the benchmark. Comparing these controls with the learned pairs, alongside normalized pooling and training-estimated base-rate baselines, would help quantify the contribution of aggregation beyond recalibration. Without such comparisons, the aggregation-specific interpretation remains insufficiently supported.
- The benchmark, weaker-model eligibility, and reported best pair are all determined using test-set Brier scores (section 3.2 and Table 8). In the largest group, the group-level success criterion searches over up to 136 eligible pairs on the same test sample. Although the authors explicitly frame the comparison as retrospective, these results do not establish how reliably a pair selected without test outcomes would outperform a strong individual on new data.
Several reported successes also have small point-estimate improvements, e.g., G4 (0.118 vs. 0.120), G7 (0.146 vs. 0.147), and G9 (0.128 vs. 0.129); G12 is indistinguishable at the displayed precision. This does not itself establish statistical insignificance, but the reported marginal confidence intervals do not resolve the issue: they condition on test-selected pairs and do not provide selection-adjusted inference for the paired Brier-score differences. The “within 5%” group-level criterion is also insufficiently discriminative on its own, since the harmonic mean achieves 16/16 groups under the same criterion.
Questions for authors
-
Does aggregation improve beyond matched individual recalibration? Appendix D.2 partially addresses recalibration, but it calibrates only weaker forecasters using log loss, whereas the aggregation weights are optimized for Brier score. Could you fit a matched single-model, separately for both the benchmark and each weaker forecaster, using the same training data and Brier objective as the aggregator?
-
Do model configurations differ in information condition (for example, whether crowd or market forecasts or news are included in the prompt)? If so, how much of the ranking (e.g., Llama-3.3-70B as benchmark ahead of o3 and R1) comes from information condition rather than model capability?
-
Footnote 1 states that the same boundary convention applies to all aggregation rules. Taken literally, this makes arithmetic pooling return 0 for forecasts (0,0.8), rather than the arithmetic mean of 0.4. Was this convention implemented for arithmetic and learned linear pooling as well? How frequently is it triggered, and are the conclusions robust to using each rule’s ordinary boundary behavior?
8 comments
Q2: Do model configurations differ in information condition?
Yes. All forecasts used in our experiments are taken directly from ForecastBench (Karger et al., 2025). We do not modify the original model prompts or information conditions, nor do we generate new forecasts. As defined in Section 2.1, each forecaster corresponds to a model configuration specified by its model version, prompt, and information condition. Within each model family, we select the configuration with the largest forecast coverage as its representative (Section 3.1), without using forecasting outcomes or performance. Consequently, the representative forecasters retain their original information conditions from ForecastBench, which may differ across models.
We also note that the effects of different information conditions on LLM forecasting performance have been studied in prior work. In particular, ForecastBench (Karger et al., 2025, Section 5) evaluates different prompting and information settings, including access to crowd forecasts and retrieved news. Similarly, Prophet Arena (Yang et al., 2026, Section 4.2.2, Figure 5) systematically examines how access to market data and news sources affects forecasting performance.
We focus on a different but complementary question: given the forecasts produced by existing LLM configurations, can two individually weaker forecasters be aggregated to match or outperform a stronger one without using its forecasts? Our objective is therefore not to isolate the effects of model capability and information conditions, but to investigate the benefits of aggregating available forecasts. Accordingly, rankings such as Llama-3.3-70B relative to o3 and DeepSeek-R1 should be interpreted as comparisons of their observed forecasting configurations on the same questions, rather than comparisons of their intrinsic model capabilities under identical information conditions.
We have made this clear in the revision, adding a footnote (see footnote 2) to emphasize this setup.
Q3: Boundary conventions for forecasts of exactly 0 or 1.
Thank you for pointing this out, we have corrected the footnote. Please see details in Global Response, Point 4.
We hope these additions address the reviewer’s concerns, and we would be glad to clarify any remaining points.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. However, a few follow-up questions remain.
-
Why Figures 15 and 16 only show a single "Validation-selected pair" bar per group alongside all three rescaling methods?
-
The test scores for the "Validation-selected pair" in Figures 15–16 do not align with the results presented in Figure 11 (for instance, the score for the G11 group shifts from approximately 0.863 in Figure 11 to about 0.884 in Figures 15–16). Could you clarify whether these experiments were conducted using different data splits?
Response to Reviewer KMZA
We sincerely thank you for the thoughtful and constructive feedback. We particularly appreciate your recognition of the practical motivation behind weak-to-strong forecast aggregation and the breadth and transparency of our experimental evaluation. Your suggestions have motivated us to conduct several additional experiments and substantially strengthen the manuscript. We address your concerns below, with further details provided in our Global Response.
W1, W2, and Q1: Improvements beyond individual recalibration/rescaling.
In the revision, we introduce three single-model controls, Platt scaling, multiplicative scaling, and affine calibration, all fitted using the same training data and Brier-score objective as learned linear pooling. Importantly, we extend these controls to all forecasters, including the group benchmark, as the reviewer suggested (Global Response, Point 2; revised Section 5.2, Table 3).
Under validation-based selection, learned linear pooling matches or improves on the best rescaled weaker forecaster in 14–15 of 16 groups, and on the best rescaled individual from the entire group in 10 of 16 groups across all three transformations. These additional comparisons provide evidence that aggregation can achieve improvements beyond individual recalibration.
Regarding the specific examples highlighted by the reviewer, we also clarify that the best learned-linear pair in every overall comparison group has two strictly positive weights, including G3 and G13 (Table 7). We agree that the Finance G4 pair effectively reduces to a rescaled single forecaster, but this example belongs to our category-specific analysis and does not contribute to the main group-level results.
W3 and W4: Validation-based selection and statistical significance.
We appreciate the reviewer's concerns regarding test-set selection and the statistical interpretation of small improvements. As detailed in Global Response, Points 1 and 3, we introduce a separate validation set for selecting both the benchmark and the best weaker-model pair, followed by evaluation on held-out test questions.
Under this protocol, learned linear pooling matches or improves on the benchmark in 12 of 16 groups. We further report paired bootstrap confidence intervals for test Brier-score differences (Appendix I.1, Figure 12). The 95% intervals lie entirely in favor of aggregation in five groups, entirely against it in one group, and include zero in the remaining ten groups.
We also agree that the "within 5%" criterion alone is not sufficient evidence of aggregation improvements, especially since some fixed aggregation rules also perform well under this criterion. We therefore emphasize the match-or-improve results and paired Brier-score differences, using the within-5% criterion as a complementary measure of how closely an aggregate approaches the benchmark.
We also kindly clarify that our main focus is to investigate whether two individually weaker forecasters can match or outperform a stronger individual without using its forecasts, rather than to claim statistically significant improvements in every group. We have revised the presentation accordingly.