- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies ensemble forecasting: given the forecasts of experts, how to combine them into one forecast when the relative accuracy of the experts changes over time. It builds on an offline regularized ensemble forecasting criterion and proposes an online version that, in each period, minimizes three terms: the dispersion of the ensemble around the contemporaneous average of the experts, a shrinkage toward a prior built from historical errors, and a proximal term that keeps the weights close to the previous period. Only one scalar, the common-correlation parameter of the prior, is updated online, and large expert pools are handled with an entmax parameterization for sparsity. The paper proves an regret bound against the best fixed combination, and reports first place on M5 (154 series, 50 team forecasts) and on the SPF survey.
Critical strengths
Forecast combination is a practically important problem and the formulation is easy to read: each of the three terms has a clear meaning, and none of them is a black box. The experiments on real data are visible. There are two real forecasting settings, 50 real team forecasts in M5 and real professional forecasters in SPF, a broad baseline set, and a consistent first place, and the improvement over the simple mean is not trivial (about 8% on M5).
Critical weaknesses
I do not think there is online policy learning here. No state-to-action function and no value function are learned, and each period's weights are just the solution of a convex program, while the prior is a rolling-window statistic and the only learned object is the scalar correlation parameter. In substance this is a sequence of proximally regularized convex programs, i.e. online convex optimization with a shrinkage-plus-proximal regularizer, so the title overstates the contribution.
The theory is also mismatched to the motivation and weak relative to the online-learning literature. The regret is against the best fixed combination, and the assumptions (a time-invariant error second-moment matrix, a convergence of the prior weights) essentially impose stationarity, which is the regime the paper claims to improve upon. The rate is the generic online convex optimization rate, so the proximal term, which is the only new structural ingredient, is never shown to help. The standard adaptive aggregation baselines (BOA, MLpol, discounted Hedge) are not compared against, and there is no dynamic-oracle regret, which would be the right target here.
The gains over the closest baseline are modest and not significance-tested (about 2.6% over offline REF on M5 and 0.7% on SPF GDP, over a 7-day evaluation window on M5), and there is no ablation that removes the proximal term.
Questions for authors
- What is learned here, beyond the scalar correlation ? Please state precisely the state, action, and objective that make the problem an instance of policy learning rather than online convex optimization.
- Can you compare against BOA, MLpol, and discounted Hedge/EWA? These are the standard online-aggregation baselines; a comparison on M5/SPF would directly test the "online" claim.
3 comments
Thanks author for the revision. The theoretical concerns (W2) are now adequately addressed, and I appreciate the additional baselines, the significance tests and the ablation. However, the new results confirm that the gain over the closest baseline is modest and not significant on the SPF GDP comparison, and that the proximal term contributes only marginally, while the central framing of online policy learning is restated rather than resolved. I therefore keep my rating at 2, but I have raised my soundness score to 3 in recognition of the improved theory and experiments.
Thanks a lot for your reply and for raising the soundness score. We would like to provide a few clarifications.
- Although the numerical differences in forecast errors between comparable methods look small, which is very common in the M5 and SPF applications in the ensemble forecasting literature (e.g., Makridakis et al., 2022; Jose and Winkler, 2008; Su et al., 2026), the improvement of Online REF over the closest baseline, Offline REF, is statistically significant at the 0.05 level in most of the comparisons, including in the M5 data, SPF CPI inflation, and SPF change in the unemployment rate. SPF nominal GDP growth is the only exception where Online REF is not significantly better than Offline REF, but it still slightly reduces RMSSE. This demonstrates that overall, Online REF substantially improves over Offline REF. Moreover, across all 12 benchmarks under both M5 and SPF, the benefit of Online REF is statistically significant in 47 of 48 comparisons at the 0.05 significance level.
We have also performed statistical tests on the improvement of adding the proximal term. The contribution of the proximal term is significant at the 0.05 level in M5, SPF CPI inflation, and SPF change in the unemployment rate, as shown below:
| Dataset | Indicator | Online REF | Online REF () | p-value |
|---|---|---|---|---|
| M5 | — | 0.337 | 0.341 | <0.001* |
| SPF | GDP | 1.033 | 1.040 | 0.282 |
| SPF | CPI | 0.892 | 0.927 | 0.033* |
| SPF | Unemployment rate | 1.468 | 1.533 | 0.018* |
We will add the new results in the finalized version.
- As noted in our reply to W1 & Q1. Online policy learning, we have revised Sections 2.1–2.3 to clarify the state, action, objective, and online update. At period , let denote the state, consisting of current forecasts, prior weights based on historical forecasting performance, and preceding optimal weights. The decision maker selects the action , where is the policy that maps the current state to the selected weights (Hazan and Singh, 2021). The policy is obtained by Eq. (1), specifying how forecast weights are selected in each period by solving an optimization problem. Optimization-based policy has been used in the policy learning literature, e.g., Agrawal et al. (2020).
Thank you for reviewing our paper and providing valuable feedback. We have made several revisions to the manuscript, with the changes highlighted in blue. Below, we address the critical weaknesses (W1, W2, W3) and questions (Q1, Q2).
Thank you for raising this point. We use “policy” as a context-to-action mapping (Hazan & Singh, 2021). At period t, the state is St=(yt,st,wt−1), consisting of current forecasts, prior weights based on historical forecasting performance, and preceding weights. The action is the selected weight vector wt=πt(St)∈ΔN, and the objective is to minimize cumulative squared forecasting loss.
Under this formulation, Eq.(1) defines the weight-selection policy through an optimization problem. The online learning component updates the common-correlation parameter ρt in Eq.(3) using the realized aggregate forecasting loss; this parameter enters the CCR prior st in Eq.(2). The prior weights still reflect individual experts' historical performance and change over time. Updating ρt changes the CCR prior and, consequently, the weights selected by Eq.(1). As discussed by Agrawal et al. (2020), optimization-based policies can be learned through their parameters. We agree that the within-period weight selection is an optimization problem; our use of “online policy learning” refers to the online adaptation of this parameterized weighting rule, without requiring a learned value function.
We have revised Sections 2.1–2.3 to clarify the state, action, objective, and online update.
Thank you for your comments. Theorem 1 compares online REF with the best fixed convex combination of expert forecasts selected in hindsight. This is a standard regret in online learning and forecast aggregation (e.g., Zinkevich, 2003; Shalev-Shwartz, 2012; Wintenberger, 2017; Gaillard et al., 2014). We have clarified this connection in Remark 1.
In this revision, we have relaxed both the second-moment and prior-weight assumptions (Assumptions 2(ii) and 4 in Appendix A). The conditional error second-moment matrix may now vary over time. For the projected prior weights, we replace the 1/t mean-squared convergence requirement with a weaker bound on their cumulative expected distance from the population-optimal weights.
We agree that dynamic-oracle regret is also a relevant theoretical target (e.g., Zhao et al., 2020; Jacobsen and Orabona, 2024). Establishing such a bound for online REF would require substantial additional theoretical work, and we leave this extension for future work, as discussed in the Discussion section.
Thank you for the suggestion! We have added BOA, MLpol, and discounted Hedge/EWA in both M5 and SPF applications. The results are given in Tables 1 and 2. Online REF achieves the best performance over all benchmarks.
We also report statistical significance tests for improvements in these tables. The improvements over the compared benchmarks are statistically significant in most comparisons.
We have also added Online REF with κ=0, which removes the preceding weights penalty. The results are given in Tables A.4 and A.5 of Appendix G.2. Removing the preceding weights penalty from Online REF generally increases forecast errors relative to the full Online REF, but it is generally better than Offline REF.