acceptodds
Under review as a conference paper at ICLR 2027

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

Abstract

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench KBYJ-25, we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
forecast aggregation, probabilistic forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KMZAOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies weak-to-strong aggregation for probabilistic event forecasting: whether forecasts from two individually weaker LLM forecasters can be combined to match or outperform the strongest individual in a comparison group. Using ForecastBench, the authors construct shared-coverage groups and evaluate learned linear and log-odds pooling against several fixed aggregation rules. The main analysis covers 70 forecasters, 16 groups, and 1121 eligible pair occurrences. Learned linear pooling produces at least one pair matching or improving on the strongest individual in 11 groups and coming within 5% in all 16. Additional experiments examine calibration, larger pools, question categories, individual recalibration, and cost/access constraints. The benchmark and successful pairs are identified retrospectively using test performance.

Critical strengths
  1. A clear, practically motivated question on real data at reasonable scale. Asking whether cheaper, open-weight, or otherwise accessible models can stand in for an inaccessible frontier forecaster matters to practitioners. The study uses real, resolved ForecastBench data across 70 forecasters and 1121 pairs.

  2. Broad comparisons and transparent reporting of the experimental setup. The authors construct maximal groups with explicit coverage thresholds and report results at both group and pair level. They include many fixed baselines, including robust-aggregation rules that are rarely tested empirically, and report bootstrap CIs and fitted coefficients for every best pair. This transparency is what allows the concerns below to be checked from the paper's own tables.

Critical weaknesses
  1. The learned aggregates receive supervised post-processing on historical data, whereas the strongest individual benchmark is evaluated without comparable recalibration. This supports the paper’s stated comparison against raw individual forecasts, but does not establish how much of the improvement requires combining complementary information. The reported linear coefficients suggest that probability rescaling may contribute substantially: for example, the weights sum to 0.6646 in overall G3 and 0.6819 in G13. Moreover, in Finance G4 (Table 17), the reported weights are (0.5378, 0.0000), so the best “pair” effectively reduces to a rescaled single model at the reported precision.

Appendix D.2 partially addresses this concern through per-model Platt scaling, but only for weaker forecasters, using log loss rather than the aggregation objective of Brier score, and a different transformation family. A more directly matched control would fit single-model probability scaling using the same training Brier objective for both weaker models and the benchmark. Comparing these controls with the learned pairs, alongside normalized pooling and training-estimated base-rate baselines, would help quantify the contribution of aggregation beyond recalibration. Without such comparisons, the aggregation-specific interpretation remains insufficiently supported.

  1. The benchmark, weaker-model eligibility, and reported best pair are all determined using test-set Brier scores (section 3.2 and Table 8). In the largest group, the group-level success criterion searches over up to 136 eligible pairs on the same test sample. Although the authors explicitly frame the comparison as retrospective, these results do not establish how reliably a pair selected without test outcomes would outperform a strong individual on new data.

Several reported successes also have small point-estimate improvements, e.g., G4 (0.118 vs. 0.120), G7 (0.146 vs. 0.147), and G9 (0.128 vs. 0.129); G12 is indistinguishable at the displayed precision. This does not itself establish statistical insignificance, but the reported marginal confidence intervals do not resolve the issue: they condition on test-selected pairs and do not provide selection-adjusted inference for the paired Brier-score differences. The “within 5%” group-level criterion is also insufficiently discriminative on its own, since the harmonic mean achieves 16/16 groups under the same criterion.

Questions for authors
  1. Does aggregation improve beyond matched individual recalibration? Appendix D.2 partially addresses recalibration, but it calibrates only weaker forecasters using log loss, whereas the aggregation weights are optimized for Brier score. Could you fit a matched single-model, separately for both the benchmark and each weaker forecaster, using the same training data and Brier objective as the aggregator?

  2. Do model configurations differ in information condition (for example, whether crowd or market forecasts or news are included in the prompt)? If so, how much of the ranking (e.g., Llama-3.3-70B as benchmark ahead of o3 and R1) comes from information condition rather than model capability?

  3. Footnote 1 states that the same boundary convention applies to all aggregation rules. Taken literally, this makes arithmetic pooling return 0 for forecasts (0,0.8), rather than the arithmetic mean of 0.4. Was this convention implemented for arithmetic and learned linear pooling as well? How frequently is it triggered, and are the conclusions robust to using each rule’s ordinary boundary behavior?

8 comments

Response to Reviewer KMZA

We sincerely thank you for the thoughtful and constructive feedback. We particularly appreciate your recognition of the practical motivation behind weak-to-strong forecast aggregation and the breadth and transparency of our experimental evaluation. Your suggestions have motivated us to conduct several additional experiments and substantially strengthen the manuscript. We address your concerns below, with further details provided in our Global Response.

W1, W2, and Q1: Improvements beyond individual recalibration/rescaling.

In the revision, we introduce three single-model controls, Platt scaling, multiplicative scaling, and affine calibration, all fitted using the same training data and Brier-score objective as learned linear pooling. Importantly, we extend these controls to all forecasters, including the group benchmark, as the reviewer suggested (Global Response, Point 2; revised Section 5.2, Table 3).

Under validation-based selection, learned linear pooling matches or improves on the best rescaled weaker forecaster in 14–15 of 16 groups, and on the best rescaled individual from the entire group in 10 of 16 groups across all three transformations. These additional comparisons provide evidence that aggregation can achieve improvements beyond individual recalibration.

Regarding the specific examples highlighted by the reviewer, we also clarify that the best learned-linear pair in every overall comparison group has two strictly positive weights, including G3 and G13 (Table 7). We agree that the Finance G4 pair effectively reduces to a rescaled single forecaster, but this example belongs to our category-specific analysis and does not contribute to the main group-level results.

W3 and W4: Validation-based selection and statistical significance.

We appreciate the reviewer's concerns regarding test-set selection and the statistical interpretation of small improvements. As detailed in Global Response, Points 1 and 3, we introduce a separate validation set for selecting both the benchmark and the best weaker-model pair, followed by evaluation on held-out test questions.

Under this protocol, learned linear pooling matches or improves on the benchmark in 12 of 16 groups. We further report paired bootstrap confidence intervals for test Brier-score differences (Appendix I.1, Figure 12). The 95% intervals lie entirely in favor of aggregation in five groups, entirely against it in one group, and include zero in the remaining ten groups.

We also agree that the "within 5%" criterion alone is not sufficient evidence of aggregation improvements, especially since some fixed aggregation rules also perform well under this criterion. We therefore emphasize the match-or-improve results and paired Brier-score differences, using the within-5% criterion as a complementary measure of how closely an aggregate approaches the benchmark.

We also kindly clarify that our main focus is to investigate whether two individually weaker forecasters can match or outperform a stronger individual without using its forecasts, rather than to claim statistically significant improvements in every group. We have revised the presentation accordingly.

Q2: Do model configurations differ in information condition?

Yes. All forecasts used in our experiments are taken directly from ForecastBench (Karger et al., 2025). We do not modify the original model prompts or information conditions, nor do we generate new forecasts. As defined in Section 2.1, each forecaster corresponds to a model configuration specified by its model version, prompt, and information condition. Within each model family, we select the configuration with the largest forecast coverage as its representative (Section 3.1), without using forecasting outcomes or performance. Consequently, the representative forecasters retain their original information conditions from ForecastBench, which may differ across models.

We also note that the effects of different information conditions on LLM forecasting performance have been studied in prior work. In particular, ForecastBench (Karger et al., 2025, Section 5) evaluates different prompting and information settings, including access to crowd forecasts and retrieved news. Similarly, Prophet Arena (Yang et al., 2026, Section 4.2.2, Figure 5) systematically examines how access to market data and news sources affects forecasting performance.

We focus on a different but complementary question: given the forecasts produced by existing LLM configurations, can two individually weaker forecasters be aggregated to match or outperform a stronger one without using its forecasts? Our objective is therefore not to isolate the effects of model capability and information conditions, but to investigate the benefits of aggregating available forecasts. Accordingly, rankings such as Llama-3.3-70B relative to o3 and DeepSeek-R1 should be interpreted as comparisons of their observed forecasting configurations on the same questions, rather than comparisons of their intrinsic model capabilities under identical information conditions.

We have made this clear in the revision, adding a footnote (see footnote 2) to emphasize this setup.

Q3: Boundary conventions for forecasts of exactly 0 or 1.

Thank you for pointing this out, we have corrected the footnote. Please see details in Global Response, Point 4.

We hope these additions address the reviewer’s concerns, and we would be glad to clarify any remaining points.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. However, a few follow-up questions remain.

  1. Why Figures 15 and 16 only show a single "Validation-selected pair" bar per group alongside all three rescaling methods?

  2. The test scores for the "Validation-selected pair" in Figures 15–16 do not align with the results presented in Figure 11 (for instance, the score for the G11 group shifts from approximately 0.863 in Figure 11 to about 0.884 in Figures 15–16). Could you clarify whether these experiments were conducted using different data splits?

Reviewer_hMdGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors find that combining the predictions of weaker LLM forecasters can sometimes outperform or match the performance of stronger ones.

Critical strengths
  • The problem is clearly explained alongside prior and related work.
  • The main experiments and the experiments evaluating how useful adding more predictors affects performance are interesting, as well as the findings, such that the best model doesn't necessarily drive performance.
Critical weaknesses
  • The authors talk motivate the use of weaker predictors due to the cost of stronger ones, but the cost evidence is confined to the appendix and is weak.
  • The benefits of combining predictions aren't novel. The new element is the framing: exclude the best model and ask whether weaker ones can replace it.
Questions for authors

Comments

  • The related work section is in the appendix. I don't really mind it given the limited space, but this is usually not good practice. You have a lot of text describing the order in which you will present things in the paper. These can be removed due to repetition, e.g., paragraph on line 336.
5 comments

Response to Reviewer hMdG

We sincerely thank you for the thoughtful feedback and encouraging comments. We particularly appreciate your recognition of our research question, main experimental findings, and analysis of how aggregation performance changes with the number of forecasters. We address your concerns below.

W1: Cost evidence as practical motivation.

We thank the reviewer for this comment. We would like to kindly emphasize that our primary focus is to study whether two individually weaker LLM forecasters can be aggregated to match or outperform a stronger forecaster, without using its forecasts. We believe this is a novel and practically meaningful research question in its own right, as it challenges the conventional approach of relying on the strongest available individual forecaster and explores whether weaker forecasters can collectively provide a competitive alternative.

Importantly, our motivation is not limited to monetary cost savings. In practice, access to stronger forecasters may be restricted by cost, proprietary APIs, deployment requirements, or other practical considerations. These scenarios motivate our investigation, but our central contribution is to systematically study the possibility and effectiveness of weak-to-strong forecast aggregation, rather than to establish that aggregation always provides a cheaper alternative.

W2: The benefits of combining predictions are not novel.

We agree that the general benefit of forecast combination is well established, and we do not claim it as our contribution. As the reviewer notes, the new element is the question itself: we exclude the strongest forecaster and ask whether weaker forecasters alone can replace it. Prior studies of LLM forecast aggregation, including Schoenegger et al. (2024), Aitchison et al. (2026), and Douven (2026), evaluate ensembles that may contain the strongest model, so they do not answer this question. We would like to highlight what the paper contributes beyond the framing:

  • Systematic evidence at scale. We evaluate 70 forecasters, 16 comparison groups, and 1,121 weaker-model pairs on shared questions within each group.
  • Findings specific to the weak-to-strong setting. The gains do not require a constituent that is already close to the benchmark: under learned linear pooling, 15 of the 16 groups contain a pair within 5% of the benchmark in which neither member is near-best (Table 9).
  • New validation in the revision. The finding persists when both the pair and the benchmark are selected on a separate validation set (12 of 16 groups), when individual forecasters are rescaled with the same training data and objective (10 of 16 groups against the best rescaled forecaster in the entire group), and in 12 disjoint groups that share neither forecasters nor questions (9 of 12 groups). Please see Global Response, Points 1–3.

Thus, we view our contribution as primarily an empirical study and evaluation framework for weak-to-strong forecast aggregation, rather than a new aggregation algorithm.

Q1: Placement of related work.

Thank you for the helpful suggestion. We have included a condensed related-work section in Introduction section.

We thank the reviewer again for the constructive suggestions, which have helped us better articulate the paper's motivation, contribution, and empirical scope.

Dear Reviewer hMdG,

Thank you again for taking the time to review our paper and for your suggestions on clarifying its motivation and contribution. We wanted to follow up on our rebuttal. If there are any points you would like us to address further, we would be happy to discuss them or provide additional clarification.

We appreciate your time and consideration.

AuthorsOpenReview ↗

Thank you again for your comments/suggestions

Dear Reviewer hMdG,

We again sincerely appreciate your thoughtful feedback, which has been very helpful in improving the motivation and presentation of our paper. We have carefully addressed your comments in our rebuttal and revised manuscript, where we have clarified the practical motivation and added a condensed related work para into the Introduction. We believe these revisions have addressed your main concerns.

We completely understand that you may have a busy schedule. However, as the discussion period is drawing to a close, we would be very grateful if you could find some time to review our responses. If any concerns remain, we are happy to provide further clarification before the discussion period ends.

Thank you again for your time and consideration!

Reviewer_p3fiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether combinations of individually weaker LLM forecasters can match or outperform the strongest individual. It evaluates learned and fixed aggregation rules across 70 forecasters, 16 overlapping shared-coverage groups, and 1121 eligible pair occurrences. Coefficients are fitted on training data and benchmarks / successful pairs are identified using test outcomes. Learned linear aggregation matches or beats the benchmark in 11/16 groups, with a pair within 5% in every group. Further analyses examine calibration, larger pools, higher coverage, categories, and practical constraints. The contribution is primarily empirical.

Critical strengths
  • I primarily value this contribution for its useful comparison on shared questions. By excluding the strongest forecaster from candidate pairs and evaluating models on the same questions, the paper directly examines whether weaker forecasters can outperform a stronger one while controlling for differences in question difficulty (Section 3).
  • The experimental reporting is substantive. The authors report group membership, learned coefficients, per group scores, bootstrap intervals, and results at a higher coverage threshold (Tables 3, 6, 8, and 17). This makes the results easier to inspect and helps identify where further controls are needed.
Critical weaknesses
  • The test-set selection presents as complicating the main results. The benchmark is selected from (n) individuals, while the best pair is selected from up to (\binom{n-1}{2}) candidates using the same test outcomes. This makes the reported success rates difficult to interpret, particularly relative to calibrated singles.
  • The overlapping groups and evaluation split limit generalization. The 16 groups share models and questions, so they are not independent replications. The random subquestion split can also place related questions in both training and test sets. The conditional bootstrap intervals do not account for pair selection, making it unclear to me whether the improvements hold for pairs selected independently on validation data.
  • An open question for me is if the results suggest single-model rescaling rather than aggregation. In Group 4 if the finance experiments, learned linear pooling achieves Brier 0.201 versus 0.247 with coefficients (0.5378, 0.0000), effectively using one forecaster (Table 17). Overall G3's coefficients sum to 0.6646 (demonstrating the rescaling). The Platt-scaling control beats the benchmark in 7/16 groups but beats the best pair in only 2/16 (Table 7). However, it uses log loss rather than the aggregation method's Brier objective, and its training weights are somewhat unclear. I would suggest that a matched single-model scaling baseline is needed to establish how much improvement actually comes from combining forecasts.
Questions for authors
  • How does the best weaker model perform when its predictions are rescaled using the same Brier objective as linear pooling? If models and pairs are selected on a validation set, do the improvements hold on a separate test set?
  • How often do models predict exactly 0 or 1? Is the endpoint rule applied to linear pooling and arithmetic averaging as well? How do the results change without it?
6 comments

Response to Reviewer p3fi

We sincerely thank you for the careful evaluation and constructive feedback. We particularly appreciate the recognition of our shared-question comparison framework and the detailed experimental reporting. Your comments have helped us identify several important ways to strengthen our evaluation, and we have conducted additional experiments and substantially revised the manuscript accordingly. We address your concerns below, with detailed results provided in our Global Response.

W1. Test-set selection of the benchmark and the best pair.

We have added an experiment in which the benchmark, pair eligibility, and the best pair are all selected on a separate validation set (Section 5.1, Table 2). The validation-selected pair matches or improves on the group benchmark in 12 of 16 groups on held-out test questions and comes within 5% in 15 groups. Please see Point 1 of the global response for the protocol and the full results.

The reviewer also raised the comparison with calibrated individual forecasters. The new rescaling comparison applies the same selection protocol to the pair and to the rescaled individuals, so neither side benefits from a larger search over test outcomes (Section 5.2, Table 3b). Please see Global Response, Point 2.

W2. Overlapping groups, the data split, and bootstrap intervals.

Overlapping groups. We have constructed 12 disjoint groups that share neither forecasters nor questions (Section I.3, Table 17). The validation-selected pair matches or improves on the group benchmark in 9 of these 12 groups and comes within 5% in 10. Please see point 3 of the global response.

Data split. We apologize for the imprecise wording in the original submission. In all of our experiments, including the main experiments of the original submission, we split the data by question and assign all subquestions of a given question to the same set. Subquestions of the same question therefore never appear in both the training and the test set. We have corrected the description in Section 3.1. The main results are unchanged because the experimental procedure itself has not changed.

Bootstrap intervals. We now report paired bootstrap intervals for the Brier-score difference between the pair and the benchmark under validation selection, where the selection is independent of the test questions (Section I.1, Figure 12). The pair is significantly better than the benchmark in five groups and significantly worse in one, and the two are statistically indistinguishable in the remaining ten. Please see Global Response, Point 3, which also clarifies that our goal is to match or outperform the benchmark, not to outperform it significantly.

W3. Single-model rescaling versus aggregation.

We have enriched the experiments on the single-model control (Section 5.2, Section I.2). We now consider Platt scaling, multiplicative scaling, and affine calibration, each fitted by minimizing the Brier score on the same training questions used for learned linear pooling. Please see Global Response, Point 2 for the results, and our answer to Q1 below for the numbers most relevant to the reviewer’s question.

Regarding the specific cases the reviewer cited, the best linear pair in overall group G3 has weights (0.5320, 0.1326), so both forecasters contribute, and the same holds for every overall group in Table 7. The Finance G4 pair in Table 16 does have a second weight of 0.0000 at the reported precision, and we agree that this case is a rescaled single forecaster. This case belongs to the category-specific analysis and does not enter the main counts. Multiplicative scaling, which returns min⁡1,cx\min\\{1, cx\\} for a nonnegative scale c, is exactly this one-forecaster special case of learned linear pooling, so Table 3 measures directly how much the second forecaster adds.

Regarding the training weights of the earlier Platt control, the instance weights in the original log loss were the per-question weights used in our Brier score, under which each question receives equal weight regardless of its number of subquestions. The new controls remove this ambiguity: all three transformations and the aggregation weights minimize the same training Brier score defined in Section 2.2.

Q1. Rescaling under the Brier objective, and selection on a validation set.

Rescaled best weaker model. Against the best rescaled weaker forecaster, the pair matches or improves in 15, 16, and 16 of 16 groups under Platt scaling, multiplicative scaling, and affine calibration under test selection, and in 14, 15, and 14 of 16 groups under validation selection (Table 3). The pair is within 5% of the best rescaled weaker forecaster in all 16 groups in every case. Even when we also rescale the benchmark and compare with the best rescaled forecaster in the entire group, the validation-selected pair matches or improves in 10 of 16 groups under each transformation.

Validation selection. The weak-to-strong aggregation do hold on a separate test set. The validation-selected pair matches or improves on the validation-selected benchmark in 12 of 16 groups, with 11 strict improvements and one tie (Table 2). Please see Global Response, Point 1.

Q2. Forecasts of exactly 0 or 1 and the endpoint rule.

Thank you for pointing this out, we have corrected the footnote. Please see details in Global Response, Point 4.

We hope these additions address the reviewer’s concerns, and we would be glad to clarify any remaining points.

Dear Reviewer p3fi,

Thank you again for your careful review and constructive feedback on our experimental evaluation. We wanted to follow up on our rebuttal. If you have any remaining questions or concerns about the additional analyses or our response, we would be happy to discuss them.

We appreciate your time and consideration.

Comments on the paper

3 comments

Global Response

We thank all reviewers for their careful reading and constructive feedback. Reviewers p3fi and KMZA raised several closely related comments, and we address them jointly here. We respond to the remaining points in the individual responses. The revision adds a new Section 5 (“Further Validation of Weak-to-Strong Aggregation”) and a new Appendix I (“Robustness Checks and Controls”). We summarize the main additions as follows and detail them below.

Summary of changes. The revision adds (i) validation-based selection of both the pair and the benchmark (Section 5.1, Section I.1), (ii) Brier-matched single-model rescaling controls (Section 5.2, Section I.2), (iii) paired bootstrap intervals (Figure 12), (iv) disjoint comparison groups (Section I.3), and (v) a boundary-convention sensitivity analysis (Section I.5). The abstract, introduction, and conclusion now state these results explicitly.

1. Validation-based pair selection (p3fi W1, W2, Q1; KMZA W3).

Both reviewers observed that, in our main experiment, the group benchmark, pair eligibility, and the best pair are all determined from test Brier scores. This may introduce selection effects, which may complicate the interpretation of our original results. We agree, and to address this concern, we have added an experiment in which every selection step is independent of the test set (Section 5.1; Table 2; Appendix I.1, Figures 11-12).

Specifically, we divide the questions in each comparison group into three disjoint sets of approximately equal size: training, validation, and test. We first learn the aggregation weights using the training set. We then use the validation set to identify the group benchmark, determine eligible weaker-model pairs, and select the pair with the lowest validation Brier score. Finally, we evaluate the selected pair and benchmark on the held-out test set, without further selection or refitting. Our new results are as follows:

  • Our new results show that the validation-selected weaker-model pair matches or improves on the group benchmark in 12 of 16 groups (including 11 strict improvements and one tie) and comes within 5% of its Brier score in 15 of 16 groups (Section 5.1, Table 2). These results complement our original results and demonstrate that competitive weaker-model pairs can be identified without using test outcomes for selection.

Table 2: Learned linear pooling against the group benchmark.

Pair and benchmark selectionMatch or improve benchmarkWithin 5% of benchmark
Test selection11/1616/16
Validation selection12/1615/16
  • To further assess statistical uncertainty, we conduct a paired bootstrap analysis using 5,000 resamples of whole test questions, with the selected models and fitted aggregation weights held fixed. Under validation-based selection, the 95% confidence intervals for the paired test Brier-score improvements lie entirely in favor of aggregation in five comparison groups (Appendix I.1, Figure 12).

2. Comparison with recalibrated/rescaled individual forecasters (p3fi W3, Q1; KMZA W1, W2, Q1).

Both reviewers noted that the learned coefficients need not sum to one, so part of the gain could come from rescaling individual forecasts, not from combining multiple forecasts. Both reviewers also pointed out that our earlier Platt-scaling control was fitted with log loss and was therefore not matched to the Brier objective used for aggregation. Reviewer KMZA further noted that the control applied only to weaker forecasters and used a different transformation family.

To address these concerns, we substantially expand our single-model control experiments by considering three transformations: Platt scaling, multiplicative scaling, and affine calibration. We fit each of them by minimizing the Brier score on the same training questions used for learned linear pooling. As Reviewer KMZA requested, we apply these transformations to every forecaster in the group, including the benchmark. We then compare the pair with the best rescaled individual among (i) the eligible weaker forecasters and (ii) all forecasters in the group. We evaluate these controls under both test-based and validation-based selection protocols (Table 3):

Table 3: Group-level BS results for learned linear pooling against rescaled individual forecasts under (a) test selection and (b) validation selection.

(a) Test selection.

Single-model controlEligible weak forecasters (Match or improve)Eligible weak forecasters (Within 5%)All forecasters (Match or improve)All forecasters (Within 5%)
No rescaling16/1616/16−−
Platt scaling15/1616/1610/1615/16
Multiplicative scaling16/1616/1612/1615/16
Affine calibration16/1616/1612/1615/16

(b) Validation selection.

Single-model controlEligible weak forecasters (Match or improve)Eligible weak forecasters (Within 5%)All forecasters (Match or improve)All forecasters (Within 5%)
No rescaling16/1616/16−−
Platt scaling14/1616/1610/1614/16
Multiplicative scaling15/1616/1610/1614/16
Affine calibration14/1616/1610/1614/16
  • Against the best rescaled weaker forecaster, the pair matches or improves in 15–16 of 16 groups under test selection and in 14–15 of 16 groups under validation selection. The pair is within 5% in all 16 groups in every case.
  • Against the best rescaled forecaster from the entire group, the pair matches or improves in 10–12 of 16 groups under test selection and in 10 of 16 groups under each transformation under validation selection. The pair is within 5% in at least 14 groups.

These comparisons provide additional evidence that combining weaker forecasters can yield improvements beyond those achieved by the single-model transformations considered here. Regarding the specific coefficients the reviewers cited, the Finance G4 pair in Table 16 does have a second weight of 0.0000 at the reported precision, and we agree that this case is a rescaled single forecaster. But this case belongs to the category-specific analysis and does not enter the main counts. In the main counts, the best linear pair in every overall group in Table 7 has two strictly positive weights, including G3 and G13.

3. Uncertainty for paired differences and overlap across groups (p3fi W2; KMZA W4).

The reviewers noted that the earlier intervals were marginal and conditioned on test-selected pairs. We now report paired bootstrap intervals for the Brier-score difference between the pair and the benchmark, resampling whole test questions 5,000 times (Section I.1, Figure 12). Under validation selection, where the selection is independent of the test questions, the pair is significantly better than the benchmark in five groups (G1, G3, G6, G8, G16) and significantly worse in only one (G10); the two are statistically indistinguishable in the remaining ten. The pair is also within 5% of the benchmark in 15 of 16 groups.

We agree with Reviewer KMZA that several of the reported improvements are small. We would like to clarify that our goal is not to show that aggregating weaker forecasters significantly outperforms the strongest individual, but that a pair of weaker forecasters can match or outperform it without using its forecasts. This is the relevant criterion when the strongest forecaster is inaccessible, and a pair that performs on par with the benchmark already meets it. We have clarified this interpretation in the revision.

Reviewer p3fi also noted that the 16 groups share forecasters and questions and are not independent replications. We have therefore constructed 12 disjoint groups from 53 forecasters that share neither forecasters nor questions, each with at least 1,000 shared subquestions (Section I.3). Encouragingly, the results remain consistent with our main findings. The pair matches or improves on the group benchmark in 8 of 12 groups under test selection and in 9 of 12 under validation selection, and comes within 5% in 10 groups under either protocol (Table 17, Figure 17).

4. Boundary convention for forecasts of exactly 0 or 1 (p3fi Q2; KMZA Q3)

We thank both reviewers for pointing out the ambiguity in the original footnote. We kindly clarify that the boundary convention applies only to rules whose formulas can be undefined at the endpoints: learned log-odds pooling, the log-odds mean, and the robust aggregation rules. The simple mean and learned linear pooling use their ordinary formulas in all main experiments, so the simple mean of the forecasts (0, 0.8) is 0.4. We have corrected footnote 1 accordingly.

We have also added a sensitivity analysis (Section I.5, Table 20). Forecasts of exactly 0 or 1 account for 6,890 of 464,752 individual forecast records in the group test sets (1.483%). Applying the convention to learned linear pooling leaves both group counts unchanged at 11 of 16 and 16 of 16. Applying it to the simple mean changes the match-or-improve count from 8 to 7 groups and leaves the within-5% count at 15.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.