- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper evaluates ten publicly available time-series foundation models on 26 water-demand series from five collections. The main setting uses univariate zero-shot forecasting with a 48-hour horizon and statistical baselines. Additional experiments vary context and horizon length, introduce weather/calendar covariates, fine-tune Chronos-2 globally or locally, and compare zero-shot models with published BWDF challenge entries. Several models outperform Seasonal Naïve on average; the analysis also reports practical equivalence among leading models, inference costs, and unresolved pretraining-overlap limitations.
Critical strengths
-
A useful domain evaluation with an operationally relevant question. The paper evaluates whether pretrained models can provide effective forecasts without maintaining a separately fitted model for every local use case. The breadth of model families, forecasting settings, and water-demand series makes the results practically informative. The reduction in mean wMAPE from 23.8% for Seasonal Naïve to 17.3% for Chronos-2 is meaningful because the seasonal baseline is competitive in this application.
-
The statistical interpretation and limitations are unusually careful. The analysis distinguishes statistical differences from a predefined region of practical equivalence, considers inference cost, and audits documented pretraining overlap. It explicitly avoids equating zero-shot evaluation with proven absence of pretraining exposure. It also acknowledges that the BWDF comparison uses heterogeneous challenge procedures. These qualifications improve the usefulness of the model-selection conclusions and should be retained.
Critical weaknesses
-
Essential evaluation and metric details remain insufficiently specified in the manuscript. Section 3.4 states that forecast windows are shared, and fine-tuning uses held-out windows, but does not explicitly give the per-series training/evaluation cutoffs, forecast-origin stride, or window counts. These definitions are essential for interpreting global/local adaptation. Table 13 also needs a precise metric explanation. Appendix C.1 states that each series has a fixed wMAPE denominator, so MAE and wMAPE should scale proportionally within a series when evaluated on the same targets. This holds for the 15-minute SOWEKI, YW, and Mauth series, where the MAE/wMAPE ratio is nearly unchanged between zero-shot and fine-tuned Chronos-2. It does not hold for the hourly BWDF and ReCycle series: for BWDF 01, global fine-tuning changes MAE from 3.77 to 0.56 but wMAPE only from 12.64% to 9.03%, and the ratio differs by roughly 3–6× across these eleven series. A different unit, temporal resolution, or set of evaluation targets in the fine-tuning pipeline could explain this pattern. This is a request to establish the computation, not evidence of leakage or an arithmetic error, but it affects how the reported fine-tuning gains on these series should be read.
-
The breadth of independent evidence and the resulting general insight are limited. The 26 series come from five collections, with BWDF and SOWEKI contributing 21 of them, so a series-level macro-average gives those two collections most of the weight. The BWDF ranking is a useful case study but cannot replace a controlled comparison with locally trained alternatives, as the paper itself recognizes. The covariate result is also configuration-dependent, and its mechanism is not isolated. Consequently, the current evidence supports strong starting baselines for the sampled setting more clearly than a broader understanding of when foundation models transfer successfully to water demand.
Questions for authors
-
Protocol clarification: Please give the chronological cutoffs, forecast-origin stride/counts, and exact configuration or code locations defining the zero-shot and fine-tuning windows. How are all local and global adaptation examples separated from evaluation targets?
-
Metric clarification: What are the exact formulas, aggregation order, units, and temporal resolution used for MAE and wMAPE in Table 13? Why does the MAE/wMAPE ratio change between zero-shot and fine-tuned configurations for the hourly BWDF and ReCycle series but not for the 15-minute series, and do all configurations use identical evaluation targets?
-
Collection-level robustness: Averaging the collection-level wMAPE values in Table 3 with equal weight gives Timer-S1 22.8%, Chronos-2 23.0%, and Seasonal Naïve 33.6%, so the advantage over the seasonal baseline appears robust while the top ordering changes. Do the Friedman/Conover and Bayesian signed-rank conclusions also hold when collections, rather than individual series, are treated as the units of analysis?
-
Presentation of the main results: The main text contains no figures; all four figures are in the appendix, and the central evidence is conveyed only through dense tables. In particular, Table 3 squeezes 13 methods × 2 metrics into the page width at a font size far below the body text, making the primary result difficult to read. For an empirical evaluation whose conclusions concern accuracy–cost trade-offs, practical equivalence, and sensitivity to context and horizon, this substantially weakens accessibility. Could the authors restructure the main results, for example, by splitting or transposing Table 3, adding an accuracy-versus-inference-time plot, and summarizing the Friedman/Conover results with a critical-difference diagram in the main text?
2 comments
Thank you for the response and the revised manuscript.
My protocol and metric questions have been addressed. The matched evaluation clarifies the fine-tuning comparison, and the revised aggregation description is consistent with the reported results. I also appreciate the collection-level analysis, the qualifications added to the main claims, and the improved readability of the split tables.
The corrected fine-tuning comparison reports median-wMAPE reductions of approximately 0.44 pp and 0.62 pp for global and local adaptation. These gains are useful but modest, and do not substantially strengthen the contribution beyond the zero-shot benchmark. Presentation has improved, and I have raised my presentation score accordingly, although the main text still lacks a visual summary of the accuracy–cost trade-off.
However, my main concern about the breadth of evidence and the resulting insight remains. Twenty-one of the 26 series come from two collections, so the evidence for broader generalization is limited. The BWDF comparison is useful, but its heterogeneous procedures do not provide a controlled comparison with locally trained alternatives. Overall, the paper establishes that pretrained TSFMs are useful baselines in the sampled setting, but provides limited additional understanding of when they transfer successfully to water demand.
I therefore maintain my original rating. The revision resolves my evaluation concerns, but in my view the paper would benefit from further development before publication.
Thank you very much for your feedback and questions. We aim to keep our responses concise and refer to the revised manuscript for further details. Major changes are marked blue in the text.
CW 1
Your observation about the MAE/wMAPE ratio prompted us to rerun the Chronos-2 comparison. Zero-shot, global, and local forecasts now use identical held-out targets, resolution, and scale. We independently recomputed the errors and confirmed that the ratio agrees across configurations within each series. Section 4.3 and Table 6 report the five-seed results, and Appendix F (Tables 15–16) gives the per-series values.
CW 2
We screened water-demand datasets from the literature for public access, temporal resolution, record length, and data quality.
dataset_selection.mddocuments inclusion and exclusion decisions. BWDF and SOWEKI supply 21 of the 26 series across our five collections (Section 3.3, Appendix A). QfA 3 reports how the results change when each collection receives equal weight.We chose BWDF because its participants developed 31 methods specifically for the same forecasting task. Against these specialist methods, zero-shot TSFMs were competitive without local training (Section 4.4). This comparison addresses practical performance on BWDF. A systematic comparison of conventional architectures and tuning choices would require a separate study.
Prior water-demand studies guided our choice of weather and calendar inputs. On the two collections with weather data, adding these inputs increased inference cost without consistent accuracy gains. Thus, inputs used by conventional water-demand models did not reliably improve zero-shot forecasts here. Section 5 discusses whether demand history already captures some of this information and identifies this explanation as a hypothesis (see also Section 4.2).
QfA 1
Each series was split chronologically into approximately 70% training, 15% validation, and 15% test data, with cutoffs rounded to local midnight in Europe/Berlin. Local adapters used only their own training partition. The global adapter pooled those partitions, each with its own cutoff. Validation loss determined checkpoint selection. Rolling forecasts could use earlier test observations as context. Test targets were excluded from fitting and checkpoint selection.
With two weeks of context and a 48-hour horizon and stride, all variants shared 1,397 complete windows and 268,224 targets (Section 3.4.1, Appendix F). Under
results/chronos2_matched_comparison/,series_protocol.csvrecords cutoffs and window counts per series,forecast_windows.csvlists the windows, and each seed'spaired_metrics.csvrecords the errors. The shared configuration is defined inbuild_steps()insrc/tsfm_water_demand_forecasting/run_chronos2_matched_comparison.pyand documented inchronos2_matched_comparison.md.QfA 2
For each series, let E be the sum of absolute errors over all scored targets, N their count, and D total observed demand. MAE = E/N in the series' original unit, and wMAPE = 100 × E/D in percent. Thus, identical targets give the same MAE/wMAPE ratio across configurations.
The earlier fine-tuning pipeline divided hourly BWDF and ReCycle values by four when converting to 15-minute steps. This changed their MAE scale. The zero-shot reference also covered a different evaluation period. In the revised comparison, all variants use the same 15-minute targets and repeat each hourly value unchanged across four steps.
Tables 15–16 give the mean and sample standard deviation across seeds for each series and fine-tuning variant. Table 6 first takes the median across series per seed, then reports its mean and sample standard deviation across seeds. Zero-shot is computed once. Section 3.4.2 distinguishes these summaries from the observation-weighted main benchmark.
QfA 3
With equal collection weights, Timer-S1 has the lowest mean wMAPE (22.84%), followed by Chronos-2 (23.02%). Seven TSFMs still outperform Seasonal Naïve (33.59%), and Chronos-2 retains the best mean rank. With the five collections as blocks, the permutation-calibrated Friedman test still indicates differences among the ten TSFMs (p < 10⁻⁵). However, 13 of 45 Conover/Holm decisions and 21 of 45 Bayesian classifications change. All four series-level equivalence findings involving Chronos-2 become inconclusive at the 95% posterior threshold (Section 4.1, Appendix C.2).
QfA 4
We split the original Table 3 into separate MAE and wMAPE tables (Tables 3 and 4), increased the font size, and kept both on the same page. Section 4.1 discusses accuracy, inference time, and practical equivalence. Appendices C.3 and D provide the detailed runtime and context/horizon figures.