acceptodds
Under review as a conference paper at ICLR 2027

Learning Observation-Guided Probabilistic Forecast Mixtures with Online Regret Guarantees

Abstract

Data-driven weather models have achieved similar overall performance to physics-based methods, but no single model wins benchmarks. A common next step is blending them, but existing methods are usually simple averages over coarse boxes with less online adaptation and no theoretical relative error guarantees. We look at historical forecast ground station pairs and learn the best mixed model through supervised learning. We then adapt online to streamed observations with expert advice algorithms, and develop novel relative error bounds in terms of probabilistic ensemble metrics. We get 30% + improvements in evaluation metrics relative to standard methods and lower error relative to estimates of the best possible forecast in hindsight, in evaluations over multiple variables and regions. This is extensible to any observation set which can help create bespoke forecasts customized to different uses with formal guarantees.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
weather forecasting, online learning, regret bounds, probabilistic forecasts, ensembles, learning from stations, expert advice

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dcyEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first trains a spatial combiner on historical forecasts and station observations. An online layer then combines raw forecasts, the learned combiner, and calibration baselines to produce station corrected probabilistic forecasts. The authors also derive a regret bound against the best fixed mixture in hindsight under the continuous ranked probability score and evaluate forecast quality across several station networks.

Critical strengths

The station level evaluation across regions is a concrete strength. Beyond average scores, the paper examines gains by terrain relief and freezes components of the online layer to separate weight adaptation from affine correction. These results make the improvements at mountain stations and their likely sources easier to assess than an aggregate score alone.

Critical weaknesses

In “Problem Formulation & Theoretical Results,” both the ideal prediction formula and its pseudocode put cumulative historical score in the exponent with a positive sign. The numerical algorithm instead uses a negative sign, as does the posterior update in the appendix proof. Since a larger cumulative score means a worse forecast, the positive sign gives greater weight to historically worse mixtures, opposite to the update required by the proof. This is more than a numerical integration difference: the stated ideal algorithm and the algorithm analyzed in the proof disagree, so the claimed regret guarantee cannot yet be verified for the algorithm as defined.

The table freezing components in “Ablations” shows that updating the affine correction alone captures most of the full online layer's gain, and the paper acknowledges the smaller effect of updating mixture weights alone. Meanwhile, “EMOS” describes fitting a Gaussian to consensus mean and spread, but does not establish whether that baseline receives the same station and lead specific online correction and feedback timing as the proposed method. The ablation helps explain the components within the proposed method, yet the comparison with this baseline could still mainly reflect different capacities for local correction. It therefore does not isolate the learned mixture's advantage over conventional postprocessing.

Questions for authors

Could the authors make the direction of the historical loss weight consistent across the ideal algorithm, numerical algorithm, and proof, and clarify which update the theoretical guarantee and reported experiments use?

Could the authors clarify how frozen weights are initialized in the ablation and provide a simple postprocessing comparison with the same station and lead specific feedback and online correction opportunities, so that local correction can be separated from learned mixing?

Reviewer_vRqcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies probabilistic weather forecast aggregation. It first trains an offline model to combine forecasts using historical observations. It then uses new station observations to update the forecast combination online. The paper reports improvements in CRPS and presents regret bounds against the best fixed mixture of experts in hindsight.

Critical strengths
  1. The paper addresses a useful practical problem. Adapting existing weather forecasts to local station observations could help users who cannot retrain large weather models.
  2. The experiments cover several forecast models, regions, lead times, and weather variables. The regional and seasonal results, together with the component ablations, provide a broad view of the reported gains.
Critical weaknesses
  1. The paper needs more careful editing. The proof of Theorem 1 refers to “Algorithm 7” (p. 17, line 896), but the paper only contains Algorithms 1 and 2. The caption of Figure 10 reads “The body’s Figure 1, on two more continents” (p. 14, line 748), although Figure 1 shows the framework. The definition beginning with “Let Di,t” is left unfinished before the next sentence starts (p. 16, lines 820–822). Figure 3(c) also has no descriptive title (p. 6, lines 273–286).
  2. The central algorithm is stated inconsistently. The paper presents Algorithm 2 as a Monte Carlo approximation of Algorithm 1. However, Equation (5) and Algorithm 1 use a positive sign for St−1(p) (p. 3, lines 137–140 and 149–151), while Algorithm 2 uses a negative sign (p. 4, lines 176–177). The proof also uses a negative sign in Eqs. (15) and (43). The authors should make these formulas consistent and clarify which sign was used in the experiments.
  3. The contribution seems not clear enough. The framework mainly combines existing components, including U-Net, the CRPS aggregation method of V’yugin and Trunov (2021), exponentiated-gradient updates, and AdaGrad, and thus its algorithmic novelty appears limited.
Questions for authors

see above comments.

Reviewer_i7s1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper post-processes probabilistic weather forecasts against station observations. An offline U-Net is trained on forecast–station pairs, then adapted online. The theoretical contribution is VT-MoE, a CRPS aggregation algorithm with regret measured against the best fixed convex mixture of expert CDFs. The strongest empirical method is different: it averages expert quantiles and applies per-station location and scale corrections. Temperature results cover India, Europe, and CONUS; precipitation results cover India.

Critical strengths
  1. Useful application. Adapting global forecasts to local observations without retraining the underlying models addresses a real gap.

  2. Substantial evaluation. Gains over the implemented EMOS baseline are 39.4% (India), 25.4% (Europe), and 15.0% (CONUS), across several forecasting systems and terrain types. The online procedure respects verification delay.

  3. Informative ablations. Table 2 separates weight adaptation from affine adaptation and shows weight adaptation still helps after affine adaptation. The station-joining experiments support the intended use case.

Critical weaknesses
  1. The theory is inconsistent as written.

    Equation (5) and Algorithm 1 put a positive sign on cumulative loss in the exponent; Algorithm 2 and the proof use a negative sign. The positive sign up-weights worse mixtures and gives linear regret for two constant experts with constant outcomes. The intended algorithm must be stated once, consistently.

    Theorem 1 and Equation (28) take square roots of regret, but regret as defined in (4) can be negative. This happens for the substitution algorithm itself, with no affine correction, since its output is not a fixed mixture. No nonnegativity assumption or positive-part convention is given. Equation (28) also substitutes the comparator loss L(p*) where the triangle-inequality expansion produces the algorithm's own loss.

    The proof of Lemma 4 omits a factor of 1/2 in the pairwise identity (the lemma statement is correct). The proof sets the ambient gradient to zero at an interior simplex minimizer; only its projection onto the tangent space vanishes, though the first-order term still drops. These are repairable and the logarithmic rate looks recoverable, but a corrected statement is required. The two-expert Freund example does not establish minimax optimality of the (N−1) constant for general N.

    The Monte Carlo failure bound carries a factor T² exp(4T)/M, so it is vacuous at any realistic horizon. This follows from the crude bound (48) and does not show exponential cost is necessary, but the paper gives no usable guarantee and never reports the M used.

  2. The guarantee does not cover the strongest method, and the scoring needs clarification.

    CDF mixing, quantile averaging, and affine correction are distinct operations. Appendix D claims VT-MoE predicts inside the simplex of CDF mixtures and that the two arms differ only by the affine pair. Neither holds: the substitution output is a nonlinear function of the expert CDFs, and the deployed layer also pools quantiles and uses exponentiated gradient. Figure 3(b) does show bare VT-MoE near zero regret and the affine variants below zero, which is suggestive, but the caption's argument that negative regret follows from the affine term is wrong. Please define each plotted variant, including "VT-MoE + affine."

    The theorem uses exact CDF CRPS; Appendix B scores with the fair M(M−1) estimator. That estimator is unbiased only for i.i.d. samples, which pooled quantile levels are not, and its gap from the empirical-CDF form scales with spread/(M−1). Ensemble sizes are referred to in Appendix A but never given. Please state how each forecast is represented and scored, and confirm the conclusions under exact CDF CRPS or quantile integration.

  3. The comparisons do not isolate learned aggregation from recalibration.

    In Table 2, affine adaptation alone recovers 79.4% of the total gain (1.037 → 0.906); weight adaptation adds a further 3.75% (0.906 → 0.872). The latter is real evidence. But the frozen-weight arm still contains the learned offline combiner as an expert, so the combiner's own contribution is never measured. EMOS uses a consensus mean and mean spread by design, which makes it a restricted baseline. Needed: an equal-weight pool and the best single system, each with the same online affine correction; an ablation removing the offline expert; and uncertainty on the Table 2 increments (Figures 3c and 11 have error bars, Table 2 does not).

  4. Novelty is not positioned.

    Neural postprocessing with station information: Rasp and Lerch (2018). CRPS mixability and online aggregation: V'yugin and Trunov (2019), which the paper's own proof of Theorem 3 follows. Adaptive quantile aggregation: Berrisch and Ziel (2021). Regret against fixed convex combinations of forecast CDFs: Thorey et al. (2017), Zamo et al. (2021). The mixture comparator alone is not new. State what is new relative to these.

Questions for authors
  1. Provide one consistent algorithm and a corrected regret theorem that handles negative regret. Which constants and optimality claims survive?

  2. Which implementation produces each VT-MoE curve? Give M, support bounds, clipping, delay handling, and sensitivity to M.

  3. What are the ensemble sizes and quantile levels, and why is fair CRPS appropriate for pooled quantiles? Do the results hold under exact CDF CRPS?

  4. How do equally recalibrated single-model and equal-weight baselines perform, and what do the offline expert and adaptive weights add, with uncertainty?

Comments on the paper

4 comments

Thank you for fixing many of the mathematical issues I pointed out (sign flip, minimax, etc.). I think there are still a few issues that remain though:

  • Regarding ``The proof sets the ambient gradient to zero\ldots{} is required'': I was pointing to Theorem 1 not Lemma 5. I don't think this affects overall correctness, but the reasoning has issues.
  • Can you please share effective sample size vs t for real runs? If close to M, then the post. distribution is nearly uniform.
  • My listed 3rd weakness and 4th question did not get responses. I think the question of adequate baselines is still important.
  • I'm still unsure of novelty. V'yugin and Trunov (2019, footnote 5) show how an expert set can be made with integrals. Algorithm 1 is this construction but with a uniform prior. Dzhamtyrova and Kalnishkan (COPA 2019) analyze a similar setting with a pool of experts and CRPS and obtain an n(B−A)/2 · ln T bound. Zamo et al. (2021) compete with a convex combination of CDFs, and Thorey et al. (2017) with weighted empirical CDF.

I plan to maintain my score, as I think a few of my key concerns still stand.

AuthorsOpenReview ↗

Response to Critical Weakness 4

Our response to critical weakness 4 follows. We accept the assertion that our novelty has not been positioned accurately. To address this, we have edited our introduction, and replaced the paragraph “Our extensible framework… (CONUS)” by a more accurate description of our contribution.

There are a number of desirable features that one expects from an algorithm for expert combination. For instance, recent works that have focused on using neural networks for combination (Rasp2018), or statistical estimation such as the aforementioned EMOS(Gneiting2005). However, these methods do not come with guarantees of optimality within their subclass, prompting the search for such algorithms within the field of expert advice. More importantly, since algorithms require implementation, one desires analysis of their numerical versions that come with correctness guarantees. For instance, a number of works (Berrisch, Thorey, Zamo) discuss regret bounds for expert algorithms and assert their minimaxity in specific situations. In particular, the latter three works analyze Algorithm 1 of this paper, and prove results such as minimaxity over a restricted class, as well as a number of alternatives to Algorithm 1 that perform better in alternative situations. Nevertheless, these algorithms involve integration over high-dimensional spaces, or the computation of quantities that do not possess explicit formulas.

In light of these observations, we now list our contributions in this article.

  • We use a particular algorithm from the theory of expert advice to combine theoretical forecasting experts using an extension of the framework of (Vyugin2021) to the mixture-of-experts setting. We remark that this algorithm (see Algorithm1) is not new and its analysis has been done before (Freund1996, Berrisch, Thorey, Zamo). Our novelty lies in the analysis of a Monte Carlo version, namely Algorithm 2 (see Theorem 1), for which we provide error bounds which are consistent with the best-known regret bounds for the vanilla Algorithm 1.

  • Subsequently, we test out this algorithm in practice, in conjunction with an offline combiner. Specifically, we develop two online adaptation methods, one having a formal bound that gives a guarantee on the error of the final forecast relative to the best forecast or possible forecast combination in hindsight, and the second which is a learned recent window calibration. Both have better performance than a standard baseline used, with the learned model empirically outperforming the former.

  • We also perform a first evaluation of neural models on the Automated Weather Stations (AWS) network of the Indian Meteorological Department (IMD). Our best model observes 40% Continuous Ranked Probability Score (CRPS) improvements are observed in India, with more than 50% for challenging terrains in the hills, and even around 15-25% for relatively more assimilated and data-rich regions like Europe and Continental United States (CONUS).

In particular, our remarks on each of the papers you have mentioned follow.

“Neural postprocessing with station information: Rasp and Lerch (2018).” : This article, while using neural networks to capture relationships, does not assert minimax optimality in any form. We sought an algorithm that, at least in a restricted scenario, did make such an assertion, leading us to the field of expert advice.

CRPS mixability and online aggregation: V'yugin and Trunov (2019) : Note that this article primarily confines itself to finitely many experts, and we are able to extend its results to a mixture of experts.

Adaptive quantile aggregation: Berrisch and Ziel (2021). Regret against fixed convex combinations of forecast CDFs: Thorey et al. (2017), Zamo et al. (2021). : All these articles contain the analysis of regret for algorithm 1, which we ensure to not place as a novelty in our paper anymore. However, since we are implementing the algorithm in practice, the Monte Carlo Algorithm 2's regret bounds in Theorem 1 can be considered a novelty.

AuthorsOpenReview ↗

Response to official comment

Respected Reviewer,

Thank you for your additional queries. We proceed to address them now.

a) Thank you for elaborating on your concern. Indeed, we made an error while reasoning that A−Bp∗=0A-Bp^* = 0 for an interior minimizer $p^* $. The answer, instead, is that at an interior minimum $p^* $ we have A−Bp∗=λ1A-Bp^* = \lambda \mathbf{1} for some λ∈R\lambda \in \mathbb R. However, observing that 1Tδ=0\mathbf{1}^T\delta = 0, one still has (A−Bp∗)Tδ=0(A-Bp^* )^T\delta = 0 at an interior minimum $p^* $, demonstrating that the rest of the inequalities and the proof goes through as written. Specifically, we replace "For the proof of (9),... $ A-Bp^* = 0$", with

"For the proof of (9), notes that at an interior minimum $p^* $, we have A−Bp∗=λ1A-Bp^* = \lambda \mathbf{1} for some λ∈R\lambda \in \mathbb R. However, observing that 1Tδ=0\mathbf{1}^T\delta = 0, one still has (A−Bp∗)Tδ=0(A-Bp^* )^T\delta = 0 at an interior minimum p∗p^*". The rest of the proof now follows as written.

b) Unfortunately, we did not track these statistics, but merely noted them during our runs. We will attempt to provide these by the deadline.

c) We have now discussed the other critical weaknesses and questions below.

d) Our novelty stands on the combination of expert advice (which comes with minimaxity guarantees even for mixture-of-experts), a Monte Carlo version of the algorithm with consistent regret bounds (which none of the cited papers contain), and the offline combiner, which is specific to our application. This is something we have emphasized in our edited review as well, in the response to Critical weakness 1.

We thank you for your follow-up questions.

AuthorsOpenReview ↗

Responses to questions

We shall now respond to the question posed.

Question 1: This is now covered by our responses to the critical weaknesses. To be precise, we provide a regret bound for Algorithm 2, which is Theorem 1, and we do not have any optimality associated to the Monte Carlo version of the algorithm, although the bound is expected to be tight as stated. Negative regret has been handled, and all constants as stated survive, apart from those whose value holds no significance e.g. C1,C2C_1,C_2.

Question 2: We have now added these details in Appendix B.1.

Question 3: The ensemble sizes and quantile levels have been included in Appendix B.1. We are unsure precisely what is meant by "exact CDF CRPS", since the implementation of the algorithm takes into account the CRPS distance between discrete distributions, which is an exact formula. Regarding the fairness of CRPS for pooled quantiles, we note that various papers have alluded to the importance of the CRPS distance (the aforementioned Berrisch and Vyugin papers are among these, with the Berrisch paper also using quantiles in their estimation, but in a different manner). Therefore, we borrowed this methodology from previous literature.

Question 4: This has been addressed in Figures 3 and 10. Here, for a variety of combiners including equal-weight and best single system, we have provided evidence of over-performance of our model. Furthermore, we will add a table that reflects precisely the improvement of the best model over the static weights, and the model without the offline combiner; nevertheless, the improvements are visually apparent from Figure 3 (Also see Figure 6 for a similar comparison). We are currently running experiments so that we can provide uncertainty baselines, and hope to include them in the camera ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.