- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces FluForecastBench, an influenza forecasting benchmark with 90 surveillance series from the United States, Japan, Canada and Germany, grouped into eight tasks. The authors argue that retrospective evaluation hides the delays, gaps and revisions in surveillance data that real forecasters face.
Each forecast is made under two input conditions and scored against the same targets. Condition A uses the history available under a task-specific publication rule, and condition B uses a fixed later edition of the data.
Twelve baselines are compared on seven per-task leaderboards using MAE relative to a seasonal naive model (SN52). A controlled experiment on US-HHS and Japan over two seasons uses separate periods for model selection and interval calibration, with intervals from a pooled residual adapter. Further diagnostics try to separate the effect of where the history ends from the effect of revised values.
No method performs best across tasks, and on the German state panel every fitted method is worse than SN52. In the controlled experiment, the fitted models beat SN52 on point accuracy. However, their 80% intervals cover only 39–54% of targets in the United States and 15–40% in Japan, and coverage is near zero at epidemic peaks. In Japan, the A/B difference comes mainly from how recent the available history is rather than from revised values.
Critical strengths
-
Evaluation tied to publication time across four surveillance systems. Pairing data available at forecast time with a later edition, scored on the same targets, is the right design, and covering four countries’ systems in native units is unusual. Effective lead is reported separately from nominal horizon, and the paper is open about how incomplete the archives are.
-
Clear evidence that interval calibration is the bottleneck. Point forecasts beat SN52, while 80% intervals reach only 15–54% coverage and almost none near peaks. This is a useful negative result for practitioners and argues against reporting point accuracy alone.
Critical weaknesses
-
Unclear use case and limited contribution to the forecasting community. The introduction does not clearly state what problem the benchmark solves or who would use it. Epidemic forecasting hubs, such as the COVID-19 Forecast Hub and CDC FluSight, already collect forecasts made in real time. Those forecasts were produced under the same reporting delays, gaps and revisions that this paper describes, so hub performance comparisons already reflect these constraints. The paper does not explain what its retrospective reconstruction adds beyond the hubs. Its main findings are also already known: revisions and delays change forecast scores (Brooks et al., 2018; Back2Future), no architecture wins across regions, and intervals under-cover near peaks. Because scores depend on both the model and its input data, the A/B comparison mainly shows that different inputs give different scores. It does not lead to guidance on model choice, revision handling or calibration. Revision-aware models, which motivate the work, are not evaluated (Sec. 6).
-
The work does not yet function as a benchmark, and it is hard to follow. The seven leaderboards use procedure-specific recipes, seasons and supports, and several cover only one season. The controlled protocol covers only two tasks. The executable supplement covers only US-HHS and Japan, and the German virology data cannot be redistributed (App. G). There is no standard protocol, hosted leaderboard or submission process that would let a new method obtain a comparable score. The text relies on undefined terms such as “calibration states”, “coordinates”, “editions” and “adjudicated”, and it uses SN52 before defining it. Several diagnostics were also designed after the results were inspected (l. 402–403, 462), which weakens the confirmatory claims.
Questions for authors
- Horizons: h2 means two calendar positions after the origin, so effective lead differs between A and B and across countries (l. 155–158). Are horizons comparable across tasks? Should results be reported by effective lead instead?
- Standard protocol: what exact protocol must a new method follow to appear on each leaderboard (data splits, tuning budget, calibration policy)?