- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a new benchmark for forecasting that uses data from sports events.
Critical strengths
I appreciate that the evaluation goes beyond outcome accuracy and include the trajectory of the forecast. The focus is matches played between sides in which points are scored by each.
Critical weaknesses
- While I appreciate the effort to evaluate the uncertainty of the forecast, I do believe that the using the difference between the points scored by each team to evaluate the uncertainty is not an appropriate method. The reason is that one can be very confident that a match will be tight, however this cannot be captured here. From an ML perspective it is the reason that a linear regressor is not considered a probabilistic method, but a Gaussian Process regressor is; the latter gives a probability distribution over the prediction. Previous work (https://arxiv.org/abs/2502.19676) has proposed to use crowdsourced uncertainty estimates over outcomes, which genenarlises to tasks not involving two sides scoring points, e.g. timeframe forecasts.
- How was avoiding test data leakage avoided, i.e. how was it ensured that the sport events used in the dataset hadn't appeared in the training data of the LLMs? My concern is that some of the LLMs are very recent
- As far as I can see, the events included in the benchmark are likely to be included in the training data of future LLM. Thus my concern is that it will not be very useful to future research, as it will not be assessing genuine forecasting abilities
- A lot of important information is not in the main text but in the appendix. E.g. the human validation mentioned in the workflow view is mentioned without any numbers. These are in the appendix but they should be in the main paper
Questions for authors
None
3 comments
I appreciate the revisions
-
The quantile-based approach is interesting but it raises the question how these are obtained from LLMs, as they are not part of the standard output; is eliciting the quantiles part of the benchmark or something the systems participating have to produce somehow?
-
The limitation of relying on score differences remains, as events not characterised in this way can't be handled.
-
The rolling perspective is useful, but it needs to be acknowledged that this means that comparisons across different points in time become harder
I maintain my score
Additional Clarifications
Clarification on Task Interface, Cross-Sport Scope, and Rolling Evaluation
We thank Reviewer 8ckf for the follow-up comment and for acknowledging the interest of our quantile-based uncertainty formulation and the usefulness of our rolling prospective design. To ensure complete clarity for the reviewers and Area Chair, we briefly address the three points:
- Structured quantile elicitation as a standard benchmark interface: FIELD explicitly defines five numerical predictive quantiles () for Margin and Total in its task prompt and output schema. Defining task-specific output structures is standard practice in widely adopted LLM benchmarks: SWE-bench [1] requires systems to generate code patches, while MMLU [2] evaluates predictions within a predefined multiple-choice space, without relying on these artifacts being native output modalities of the underlying LLM. Similarly, FIELD requires participating systems to directly submit five numerical predictive quantiles, without requiring native continuous-distribution logits. These submitted quantiles are evaluated directly via the strictly proper Weighted Interval Score (WIS) without post-hoc numerical repair or quantile reordering (Section 3.1, Appendices A.5 and C.1).
- Intentional cross-sport scope: FIELD is designed as a cross-sport forecasting benchmark for two-sided competitive matches with verifiable scoring dynamics, rather than a universal benchmark for arbitrary event types. While currently instantiated on football, basketball, and baseball, the framework is designed to accommodate additional sports with compatible scoring structures and well-defined sport-specific target mappings (such as American football or ice hockey; Section 3.1). We do not claim coverage of non-score forecasting tasks in the present study, consistent with the scope and transferability limitations discussed in Section 5.3.
- Cross-time comparability in rolling benchmarks: We agree that differences in event difficulty and model versions require careful treatment in cross-time comparisons. FIELD currently reports its original 232-event evaluation and subsequent 62-event prospective cohort separately, using shared valid support for within-cohort comparisons (Section 5.2, Appendix F). As additional prospective batches accumulate, we plan to introduce cumulative leaderboards alongside cohort-specific analyses. This follows established precedents in dynamic prospective evaluation: FutureX [3] maintains both weekly and overall leaderboards, while ForecastBench [4] continuously updates its evaluation and employs difficulty-adjusted Brier scoring [5] to compare forecasters evaluated on different question sets.
We appreciate the discussion and hope these clarifications are helpful.
References
- [1] Jimenez, C. E., et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024.
- [2] Hendrycks, D., et al. Measuring Massive Multitask Language Understanding. ICLR, 2021.
- [3] Zeng, Z., et al. FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction. ICLR, 2026.
- [4] Karger, E., et al. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. ICLR, 2025.
- [5] Kucinskas, S., Bastani, H., & Karger, E. ForecastBench: Updated Ranking Methodology. Technical Report, Forecasting Research Institute, 2025.
Response to Reviewer 8ckf
We thank Reviewer 8ckf for taking the time to review our submission. We apologize for replying close to the discussion deadline—we spent the rebuttal period collecting an additional 62-event prospective cohort and conducting new human audits. We have also uploaded a revised manuscript, with all additions and updates highlighted in blue for ease of review. Below, we clarify three key design aspects of FIELD and summarize our revisions.
1. Capturing Uncertainty and Tight Matches via Predictive Quantiles (Weakness 1)
2. Pre-Event Prospective Freezing and Long-Term Benchmark Utility (Weaknesses 2 & 3)
Clarification:
Strict pre-event temporal boundary: All 232 primary events took place in September 2026 and were registered before kickoff (Appendices A.1 and C.2). Forecasts were collected and archived within fixed pre-event windows at H1=T−24h and H2=T−2h, with no post-event regeneration or backfilling, strictly preserving the temporal boundary between forecasting and outcome resolution.
Long-term utility via rolling prospective evaluation: Rather than a static dataset, FIELD is a continuously runnable prospective protocol. To demonstrate this directly, we collected and evaluated a new 62-event prospective cohort (22 football, 20 basketball, 20 baseball) after the initial benchmark (Section 5.2, Appendix F), replicating our core trajectory–outcome finding on newly scheduled events. Any future LLM release can be evaluated on newly registered upcoming matches under the exact same pipeline.
Paper Revisions: Highlighted FIELD’s rolling prospective registration in Sections 1, 3.1, and 5.3, and incorporated the new 62-event cohort in Sections 4.3, 5.2, and Appendix F.
3. Human Validation Statistics in the Main Text (Weakness 4)