acceptodds
Withdrawn from ICLR 2027

FluForecastBench: Influenza Forecasting with Publication-Time Information

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable influenza forecasting requires accurate predictions and useful uncertainty estimates from surveillance reports that are delayed, incomplete and revised. Retrospective evaluation using later reports can conceal the information constraints faced by a forecaster. Although prior studies establish the importance of reporting revisions, comparisons must also account for missing observations and changes in the data used to calibrate predictive uncertainty. To make these constraints explicit in evaluation, we introduce FluForecastBench, covering four countries, eight tasks and 90 surveillance series. It pairs histories selected under explicit publication rules with specified later editions and evaluates both forecasts against the same targets. The US condition uses archived early reports. Other tasks follow their recorded publication or first-seen rules. We evaluate twelve baseline procedures across seven per-task leaderboards. Relative performance varies across tasks: seven fitted procedures have higher error than a seasonal naive forecast on the German state panel. A separate five-baseline experiment in the United States and Japan separates model selection from interval calibration over two seasons. All four fitted baselines improve mean point accuracy under publication-constrained inputs, but nominal 80% intervals cover only 39–54% of US targets and 15–40% of Japanese targets under a shared preceding-season residual calibration scheme. Further analyses distinguish revised values from differences in available history and calibration data. These results identify incomplete histories and the transfer of uncertainty calibration across seasons as concrete challenges for influenza forecasting.

Primary area:
datasets and benchmarks
Keywords:
influenza forecasting, time series forecasting, benchmarks, delayed observations, data revisions, uncertainty calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_t8LfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces FluForecastBench, an influenza forecasting benchmark with 90 surveillance series from the United States, Japan, Canada and Germany, grouped into eight tasks. The authors argue that retrospective evaluation hides the delays, gaps and revisions in surveillance data that real forecasters face.

Each forecast is made under two input conditions and scored against the same targets. Condition A uses the history available under a task-specific publication rule, and condition B uses a fixed later edition of the data.

Twelve baselines are compared on seven per-task leaderboards using MAE relative to a seasonal naive model (SN52). A controlled experiment on US-HHS and Japan over two seasons uses separate periods for model selection and interval calibration, with intervals from a pooled residual adapter. Further diagnostics try to separate the effect of where the history ends from the effect of revised values.

No method performs best across tasks, and on the German state panel every fitted method is worse than SN52. In the controlled experiment, the fitted models beat SN52 on point accuracy. However, their 80% intervals cover only 39–54% of targets in the United States and 15–40% in Japan, and coverage is near zero at epidemic peaks. In Japan, the A/B difference comes mainly from how recent the available history is rather than from revised values.

Critical strengths
  1. Evaluation tied to publication time across four surveillance systems. Pairing data available at forecast time with a later edition, scored on the same targets, is the right design, and covering four countries’ systems in native units is unusual. Effective lead is reported separately from nominal horizon, and the paper is open about how incomplete the archives are.

  2. Clear evidence that interval calibration is the bottleneck. Point forecasts beat SN52, while 80% intervals reach only 15–54% coverage and almost none near peaks. This is a useful negative result for practitioners and argues against reporting point accuracy alone.

Critical weaknesses
  1. Unclear use case and limited contribution to the forecasting community. The introduction does not clearly state what problem the benchmark solves or who would use it. Epidemic forecasting hubs, such as the COVID-19 Forecast Hub and CDC FluSight, already collect forecasts made in real time. Those forecasts were produced under the same reporting delays, gaps and revisions that this paper describes, so hub performance comparisons already reflect these constraints. The paper does not explain what its retrospective reconstruction adds beyond the hubs. Its main findings are also already known: revisions and delays change forecast scores (Brooks et al., 2018; Back2Future), no architecture wins across regions, and intervals under-cover near peaks. Because scores depend on both the model and its input data, the A/B comparison mainly shows that different inputs give different scores. It does not lead to guidance on model choice, revision handling or calibration. Revision-aware models, which motivate the work, are not evaluated (Sec. 6).

  2. The work does not yet function as a benchmark, and it is hard to follow. The seven leaderboards use procedure-specific recipes, seasons and supports, and several cover only one season. The controlled protocol covers only two tasks. The executable supplement covers only US-HHS and Japan, and the German virology data cannot be redistributed (App. G). There is no standard protocol, hosted leaderboard or submission process that would let a new method obtain a comparable score. The text relies on undefined terms such as “calibration states”, “coordinates”, “editions” and “adjudicated”, and it uses SN52 before defining it. Several diagnostics were also designed after the results were inspected (l. 402–403, 462), which weakens the confirmatory claims.

Questions for authors
  1. Horizons: h2 means two calendar positions after the origin, so effective lead differs between A and B and across countries (l. 155–158). Are horizons comparable across tasks? Should results be reported by effective lead instead?
  2. Standard protocol: what exact protocol must a new method follow to appear on each leaderboard (data splits, tuning budget, calibration policy)?
Reviewer_ZCq3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an influenza forecasting benchmark with eight tasks and 90 surveillance series across four countries. It compares histories selected under publication rules with later data editions, using identical evaluation targets. Twelve baseline procedures are evaluated, with a separate controlled US/Japan experiment. The analyses examine how available history, revised observations, and calibration data affect forecast accuracy and uncertainty.

Critical strengths
  1. The paired evaluations use identical targets and include controls separating revised values, available history, and calibration effects. These analyses provide more insight than a simple leaderboard.

  2. The controlled experiment separates model selection and calibration chronologically and reports seed variability and uncertainty estimates. These are meaningful safeguards, and the authors clearly acknowledge several limitations.

Critical weaknesses
  1. The operational information setting is not consistently represented. US condition A excludes intermediate revisions that were already public at issuance, while some large gaps elsewhere reflect incomplete archives rather than reporting delays. The experiments therefore mix genuine publication constraints with restrictions introduced by data collection and selection. This weakens the connection between the benchmark and operational forecasting.

The additional value over established resources is insufficiently demonstrated. FluSight already provides shared forecasting evaluation, and historical influenza data versions and revision-aware evaluation are established. Appendix D.4 identifies multi-country integration and controlled comparisons as the added contribution, but the executable supplement described covers only US-HHS and Japan. For a benchmark paper, this gap between the advertised scope and the usable resource is consequential.

Questions for authors

Beyond the functional comparison in Table 14, which additional capabilities are reproducibly available across all eight tasks? Please clarify exactly which histories, source manifests, readers, and evaluation code are included in the submitted release.

Do the main conclusions hold when condition A includes all revisions actually available at issuance and archive gaps are distinguished from genuine publication delays?

How much of the reported undercoverage is specific to the fixed, pooled residual adapter? A matched comparison with rolling or season-conditioned calibration would help establish whether this is a broader benchmark challenge.

Reviewer_yg3DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces FluForecastBench, a benchmark for influenza forecasting that accounts for the information available when predictions are made. The authors compile surveillance data across four countries, covering eight forecasting tasks and 90 time series. They compare existing forecasting models using publication-constrained historical data and subsequently updated data, allowing them to examine how reporting delays and data revisions affect forecasting performance.

The authors evaluate twelve forecasting procedures across seven task-specific leaderboards and conduct a more controlled experiment in the United States and Japan. They find that forecasting performance varies substantially across settings, that missing recent observations can significantly affect prediction accuracy, and that models with relatively accurate point predictions often produce poorly calibrated uncertainty estimates. The paper also provides additional analyses separating the effects of observation availability, revised values, and uncertainty calibration.

Critical strengths

I think this is an interesting and important topic. Predicting influenza outbreaks, especially when and where they will occur, is clearly important for public health. I also agree with the authors that forecasting algorithms should be evaluated using the information that would actually have been available at the time of prediction. This is particularly important for infectious disease forecasting, where reporting delays and subsequent data revisions can significantly affect the inputs available to a model.

I appreciate the amount of work that went into assembling the benchmark across four countries and evaluating a relatively large collection of existing forecasting methods. The authors make a reasonable effort to ensure that methods are compared using common evaluation targets, and I particularly appreciate the additional experiments that distinguish missing observations from revisions to previously reported values.

I also found the results on uncertainty calibration interesting. In Table 3, the authors show that several models achieve meaningful improvements in point prediction accuracy while their nominal 80% prediction intervals cover only 39–54% of observations in the United States and 15–40% in Japan. Coverage is even worse near epidemic peaks. This is an important practical observation because uncertainty estimates are particularly valuable when public-health decisions need to be made under rapidly changing conditions.

Critical weaknesses

My main reservation concerns the novelty and significance of the contribution. I think this is generally good work, but I struggle to identify what fundamentally new insight we learn from the paper.

The importance of reporting delays and retrospective data revisions is already well established in the infectious disease forecasting literature. The authors themselves cite several relevant papers, including Brooks et al. (2018) on reporting revisions and Back2Future (Kamarthi et al., 2022) on learning backfill dynamics. Existing forecasting initiatives have also emphasized the importance of evaluating predictions against information available in real time. The contribution of this paper therefore appears to lie primarily in assembling these ideas into a multi-country benchmark.

While this is potentially useful, I am not convinced that the empirical findings are sufficiently novel. For example, the observation that missing recent data negatively affects forecasting accuracy seems intuitive, as does the finding that historical uncertainty calibration may fail when epidemic patterns change between seasons. I would have liked to see a much clearer explanation of what we learn from these experiments that was previously unknown, and how these findings should change the way we develop forecasting algorithms.

I also think the evidence that retrospective evaluation meaningfully changes the conclusions we draw about forecasting models is weaker than the motivation might suggest. In Section 5.3, the authors report that only seven of 42 primary comparisons exhibit a supported change in the performance gap between methods, and none establishes a statistically supported reversal in their relative ranking. This is an important result, but it leaves me wondering how consequential the proposed evaluation framework is for existing model comparisons. The authors should explain more clearly whether the main contribution is identifying cases where conventional evaluations lead to incorrect conclusions, or simply providing a more comprehensive evaluation protocol.

My second concern is that the benchmark itself does not always reconstruct the information that would actually have been available to a forecaster. In particular, the authors acknowledge that Condition A for the United States uses only archived early reports and excludes intermediate revisions that might already have been available at the forecast date. Thus, the main publication-constrained condition is not necessarily equivalent to a genuine real-time information set. Some other tasks also have substantial gaps in their historical archives, and the authors acknowledge that publication-time availability for certain auxiliary inputs cannot be fully verified. These are understandable data limitations, but they matter because accurately reconstructing historical information availability is the central motivation of the paper.

Furthermore, the experimental evidence is less comprehensive than the headline coverage of four countries and 90 time series might suggest. The most carefully controlled experiment in Section 5.2 covers only the United States and Japan over two held-out seasons. Several other task-specific evaluations cover only one season. This is particularly concerning for influenza forecasting because epidemic dynamics can differ substantially across seasons. The reported uncertainty intervals condition on the observed seasons, so it is unclear how confidently the findings would generalize to future outbreaks.

I also have some reservations about the interpretation of the uncertainty-calibration results. The controlled experiment uses a relatively simple residual-calibration procedure based on a short period from the preceding season. For the first Japanese evaluation fold, the authors have only 38 common calibration coordinates. While the resulting undercoverage is striking, I am not sure how much it tells us about the broader difficulty of calibrated influenza forecasting, as opposed to the limitations of this particular calibration procedure. A comparison against stronger adaptive or season-aware calibration methods would help establish whether this is a fundamental challenge.

Finally, for a paper whose primary contribution is a benchmark, reproducibility is particularly important. Although the authors provide substantial documentation and verification, they acknowledge that the executable supplement primarily covers the controlled US/Japan experiments, while parts of the broader four-country dataset cannot be redistributed. The full GPU experiments have also not been independently reproduced. I appreciate the authors' transparency, but these limitations make it more difficult to assess how readily other researchers could adopt the proposed benchmark.

Overall, I think the paper addresses an important problem and represents a meaningful amount of work. However, I remain uncertain whether it provides sufficiently new scientific insights or a sufficiently complete benchmark to warrant acceptance at ICLR. I would have liked the authors to make a much stronger case for what this benchmark enables that existing real-time forecasting datasets and evaluation approaches do not.

Questions for authors

My main question is about the novelty of the contribution. Given the existing literature on reporting delays, data revisions, and real-time influenza forecasting, what do the authors consider the fundamentally new scientific insight of this paper? Beyond assembling a larger benchmark, what have we learned about the behavior of forecasting models that was not previously understood? I would be particularly interested in how the authors position their contribution relative to Brooks et al. (2018), Back2Future, and existing influenza forecasting evaluation initiatives.

Relatedly, how should we interpret the finding in Section 5.3 that only seven of 42 primary comparisons exhibit a supported change in relative performance, with no statistically supported reversal in model rankings? Does this suggest that retrospective data revisions have a relatively limited effect on the conclusions of existing forecasting comparisons, or do the authors believe the evaluation is insufficiently powered to detect such changes?

I would also appreciate clarification on the operational interpretation of Condition A. Since the US data do not include all revisions that would have been available at the forecast date, and some other settings have substantial archival gaps, to what extent does Condition A capture the information constraints faced by actual forecasters? How would the main results change if the authors could reconstruct complete publication-time information rather than relying on the available historical records?

Finally, could the authors clarify how much of the severe undercoverage in Section 5.2 is attributable to their particular residual-calibration procedure? Would stronger adaptive calibration methods substantially improve coverage, especially around epidemic peaks? I think this would help determine whether the finding represents a general challenge in influenza forecasting or a limitation of the specific experimental setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.