acceptodds
Under review as a conference paper at ICLR 2027

Cautious Adaptation under Persistent–Transient Ambiguity for Online Forecasting

Abstract

Online forecasting methods increasingly emphasize rapid adaptation to concept drift and equate model adaptation with immediate deployment, yet newly observed behavior may indicate either the beginning of a persistent shift or a transient episode caused by temporary events, sensor failure, or malicious poisoning. At the moment the data is received, these alternatives may be indistinguishable, although they imply opposite decisions for future deployment. We therefore formulate online adaptation as an operational deployment decision problem under persistent–transient ambiguity. A newly updated model is promoted only if its out-of-sample forecasts demonstrate statistically credible improvements over the previous model. We prove that any model promotion rule using only update-time information incurs a nonzero worst-case excess risk across indistinguishable persistent and transient continuations. The resulting framework shifts the emphasis from unconditional rapid adaptation toward cautious updating. Our method hedges the frozen models while out-of-sample evidence accumulates during the subsequent interval, then uses a Diebold–Mariano test to determine promotion. We further establish a pathwise within-cycle sublinear regret bound relative to the better frozen model. Experiments on five forecasting benchmarks show that our method is competitive with strong recent online forecasting baselines, while operational diagnostics characterize its deployment behavior beyond predictive MSE.

Primary area:
learning on time series and dynamical systems
Keywords:
Online time series forecasting, Persistent–transient ambiguity, Cautious deployment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SaXsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies cautious adaptation in online forecasting. Intuitively, to have robust online updates in forecasting we need to distinguish true persistent changes from transient episodes (e.g. noise and outliers). The paper proposes a rule to evaluate the recently updated model with its frozen counterpart, to decide whether the update should be kept or rejected. The proposed approach is evaluated on standard forecasting benchmarks.

Critical strengths
  • I like the framing of "cautious deployment". It's a nice practical framing that highlights the main limitation of online forecasters (i.e. their very poor robustness to transients)
  • the proposed method is simple and practical
Critical weaknesses

writing:

  • some parts of the paper excessively rely on ambiguous and unexplained jargon. As an example, at the start of Sec. 3: "We then formulate the incumbent-candidate deployment decision and show how it gives rise to persistent–transient ambiguity and an unavoidable cost of immediate model promotion.". At this point, the incumbent, the ambiguity, and the "unavoidable cost", "immediate promotion" are all undefined. I can guess what it means, but the jargon makes reading it harder than necessary, without adding any specific information.
  • more in general, the paper proposes a relatively simple method that is made harder to understand by the opaque description

experiments are a bit limited:

  • only the MSE metric is shown
  • limited baselines: the paper should compare against methods designed to protect against forgetting in online forecasting, such as [1, 2]. These methods are often explicitly designed to adapt to the transient and then quickly recover once it's finished.
  • the proposed method in general does not seem to consistently improve the performance
  • The paper remarks the importance of robustness to attacks several times. However, this is never tested.

[1] Pham, Quang, et al. "Learning fast and slow for online time series forecasting." ICLR 2023 [2] Wen, Qingsong, et al. "Onenet: Enhancing time series forecasting models under concept drift by online ensembling." NeurIPS 2023

Questions for authors
  • the method seems dependent on the relationship between the update window size, horizon, and the length of the transient. I did not see an analysis of this in the paper
  • In table 1, I understand why HAP is better in the transient case. Can you motivate why it's much better also in the persistent case? Are the other methods slower to adapt?
8 comments
AuthorsOpenReview ↗

Response to Q1 (duration relationship)

[Q1]. the method seems dependent on the relationship between the update window size, horizon, and the length of the transient. I did not see an analysis of this in the paper

We thank the reviewer for raising this point, and we agree that the relationship between these time scales should affect HAP’s performance. We have added a timing-sensitivity experiment on the transient case of semi-synthetic ETTm1. We vary the window size U∈9,15,21U\in\\{9,15,21\\} (days) and the transient length D/U∈1/3,2/3,1,4/3,5/3,2,7/3,8/3D/U\in\\{1/3,2/3,1,4/3,5/3,2,7/3,8/3\\}. The intervention magnitude is fixed at Δ=6\Delta=6. and the forecasting horizon is fixed at H=48H=48 (12 hours).

The intervention starts at the same time ss and ends at s+Ds+D. ss is aligned with an update boundary. For each fixed UU, all transient length D/U are evaluated over the same evaluation interval [s+U,s+3U)[s+U,s+3U). This correponds to the two intervals following the first update after ss. During the evaluation interval, HAP continues its normal operations. The MSE are reported below (See a more visual figure of the same results).

UU (days)D/U=1/3D/U=1/32/32/33/33/34/34/35/35/36/36/37/37/38/38/3
90.5735 ± 0.01340.5779 ± 0.01130.9901 ± 0.03921.1629 ± 0.06391.1715 ± 0.06351.2218 ± 0.06281.1367 ± 0.05451.0631 ± 0.0837
150.5783 ± 0.02400.5809 ± 0.02630.8197 ± 0.01420.8670 ± 0.01540.9330 ± 0.02450.9659 ± 0.00860.9166 ± 0.01440.9087 ± 0.0208
210.6827 ± 0.01040.6765 ± 0.01510.8439 ± 0.00730.9235 ± 0.01070.9868 ± 0.02791.0121 ± 0.03080.9647 ± 0.01860.9513 ± 0.0105

These results show that performance depends on D/UD/U as suggeted by the reviewer. More precisely:

1. HAP is robust to transient event when transient length is smaller than window size: when the transient lasts less than window size, the candidate is not fully trained on transient behaviour and it receives smaller weights than the incumbent during hedged prediction on the subsequent interval.

2. HAP performs worst when the transient event spans exactly both the candidate training interval and the subsequent comparison interval: the largest MSE occurs at D=2UD=2U for all tested window size. At s+2Us+2U, the DM test would promote a model adapted on the transient, just as the transient disappears.

3. For transient length longer than two intervals, HAP's error decreases.: in this case, more of the evaluation interval is covered by the transient event, where adaptation can be more useful than harmful. In fact, with the transient length being longer (in terms of number of intervals), the transient event starts to be "persistent".

We thank the reviewer again for this interesting question. The results are quite aligned with HAP's learning and deployment mechanism. We hope that this experiemental analysis has responded to your question.

AuthorsOpenReview ↗

Response to Q2 (persistent case explanation)

[Q2]. In table 1, I understand why HAP is better in the transient case. Can you motivate why it's much better also in the persistent case? Are the other methods slower to adapt?

We thank the reviewer for this interesting question, we have investigated on temporal losses (daily and six hours) across all runs to understand why HAP beats baselines in persistant case. It turns out that HAP is worse for the first 6 hours during the 30 days evalution interval, then becomes better for the rest of the interval.

At the branching boundary, HAP's candidate has already been trained on the intervention just as baselines. HAP is actually worse immediately after branching: all four baselines outperform it during the first 6 hours. However, the candidate is already substantially better than the incumbent (2~3 times lower MSE). Actually, HAP initially assigns equal weight, so its early disadvantage comes from the incumbent. The first complete forecast feedback arrives 12 hours after branching. At that hedge update, the candidate weights reach 0.986–1.000 across all runs, and it dominates deployment hereafter.

In fact, the baselines are not necessarily slow to adapt. Rather, HAP has a useful candidate that performs better than baselines in this experiment. The fixed candidate available at branching is held fixed for the entire 30-day evaluation, its persistent MSE averaged over Δ=3,6,9\Delta=3,6,9 is 0.6663. Even without further adaptation, the candidate is better than all baselines:

0.6398⏟HAP<0.6663⏟fixed candidate<0.6852⏟best baseline (DSOF).\underbrace{0.6398}_{\text{HAP}} < \underbrace{0.6663}_{\text{fixed candidate}} < \underbrace{0.6852}_{\text{best baseline (DSOF)}}.

A plausible explanation for this is updating per batch can be more stable than continuouly updating when dealing with streaming data. HAP updates each candidate over the completed intervals of forecasting examples. In this experiment, that batch contains repeated observations of the modified daily profile. Fitting over this broader temporal window can help to learn than reacting to local fluctuations. As Read and Žliobaitė (2025, Sections 5.3 and 5.6) emphasize, continuously issuing predictions does not require updating model parameters after every observation, the appropriate update frequency depends on the stability of the predictive structure [1]. HAP's advantage over baselines in this experiemnt is the learning of a strong candidate from the batch and rapidly increasing its predictive influence during hedged deployment.

Reference

[1] Jesse Read and Indrė Žliobaitė. Supervised Learning from Data Streams: An Overview and Update. ACM Computing Surveys, 2025. See Sections 5.3 and 5.6. https://arxiv.org/html/2212.14720v3

AuthorsOpenReview ↗

Response to W1 and W2 (writing)

[W1, W2]. some parts of the paper excessively rely on ambiguous and unexplained jargon. As an example, at the start of Sec. 3: "We then formulate the incumbent-candidate deployment decision and show how it gives rise to persistent–transient ambiguity and an unavoidable cost of immediate model promotion.". At this point, the incumbent, the ambiguity, and the "unavoidable cost", "immediate promotion" are all undefined. I can guess what it means, but the jargon makes reading it harder than necessary, without adding any specific information. - more in general, the paper proposes a relatively simple method that is made harder to understand by the opaque description

Thank you for highlighting the passages that were difficult to follow. We would first like to clarify that the concepts mentioned at the start of Sec. 3 are introduced earlier in the paper.

In the Introduction, the paragraph “A motivating paired example” explicitly defines the two model roles (lines 078–079):

“We call the currently retained model the incumbent and the model adapted on B4\mathcal B_4 the candidate.”

The same example explains persistent–transient ambiguity: two continuations share the same history but favor different models afterward (lines 075–082). The following paragraph motivates delaying immediate replacement while subsequent observations provide evidence about the update’s usefulness (lines 083–085). Then, in the first contribution ("In Section 3, we formulate..."), we state the strictly positive worst-case excess risk of committing immediately under indistinguishable continuation (lines 088–091).

Nevertheless, we agree that the start of the Sec. 3 compresses these concepts into one sentence and can be presented more clearly. We propose the following formulation:

“We then study whether to keep the current model (incumbent) or replace it with an adapted model (candidate). When the same observed history admits two future continuations, any rule forced to select one model incurs additional expected loss in at least one continuation.”

More generally, we will make the formal setting more clear and provide reminders of the concepts where needed.

Reviewer_ExkZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Research paper "CAUTIOUS ADAPTATION UNDER PERSISTENT–TRANSIENT AMBIGUITY FOR ONLINE FORECASTING" is devited to online forecasting when new arriving data samples can be signal of lasting change or temporary episode. Surely, two considered cases require different online adaptation decisions. Authors demonstrate that every rule that should decide instantly has nonzero worst-case risk. They propose HAP - method that keeps old model parameters and adapted candidate model, then combines them with delayed-feedback hedge rule. It promotes candidate only if Diebold–Mariano test on new data favors it. Experiments use synthetic ETTm1 benchmark and five standard forecasting datasets.

Critical strengths
  1. Clear mathematical descritpion. Research paper provides clear mathematical description of rule to choose between keeping and replacing model during online forecasting.

  2. Specific theoretical results. Research paper provides clear results with full proofs including lower bound on risk of doing instant decision and regret bound for hedged running.

Critical weaknesses
  1. Problem statement is not novel. Online forecasting under concept drift is well studied research area. Many existing methods address this problem. The idea of keeping old model and using instead new model after it shows better results on new data samples is common idea. For ICLR conference contribution is weak.

  2. Theory relies on standard methods. Proposition 1: if two situations look same but need opposite actions, then any rule should lose in one of them. Theorem 1: combines known results for hedge with delayed feedback with basic property of squared loss and promotion step is standard Diebold–Mariano test.

  3. Theoretical part covers only part of method. Regret bound compares deployed forecasting with better of two fixed models within one loop. It does not explain whether promotion decisions are right or how method performs over whole benchmark interval.

  4. On standard benchmarks gains compared with recent methods are small and not consistent.

Questions for authors
  1. How HAP is different from champion-challenger deployment or from background model replacement in drift detection methods?

  2. Which parts of theoretical results authors see as new compared with known bounds for hedge with delayed feedback?

13 comments
AuthorsOpenReview ↗

Response to Q1 (difference to other methods)

[Q1]. How HAP is different from champion-challenger deployment or from background model replacement in drift detection methods?

We thank the reviewer for asking this interesting question. HAP differs from background replacement by preparing and evaluating candidates without using drift alarms, and from shadow-only champion–challenger deployment by allowing the candidate to participate in deployment before promotion.

1. background model replacement in drift detection methods

Representative approaches include Adaptive Random Forest (ARF; Gomes et al., 2017) [1] and Hoeffding Adaptive Trees (HAT; Bifet and Gavaldà, 2009) [2]. Like HAP, they preserve an incumbent (active model) while preparing an candidate (background model). But they significantly differ in the timing for initiation and promotion of the background model.

AspectARFHATHAP
Preparing an alternativeA warning triggers training.A detected local change triggers an alternate subtree.A candidate is trained at scheduled update boundaries and no drift warning is required.
Replacement/promotion triggerA drift alarm triggers replacement.Evidence that the alternative is more accurateA DM rule compares the frozen pair's subsequent losses and no drift alarm is required.
Training during comparisonActive and background trees continue learning.Active and alternate subtrees continue learning.Both models remain frozen and only their deployment weights change, training are performed per interval

These differences matter in online forecasting because detecting a change and deciding whether a fitted update is useful are distinct questions. Read and Žliobaitė (2025, Section 5.5) emphasize that concept drift detection is neither mandatory for adaptive learning nor necessarily beneficial: false drift alarms can deteriorate performance, and even correctly detected changes doesn't necessarily increase forecasting error or justify replacing the current model [3]. This motivates our design, HAP therefore does not require a drift alarm. Also following [3], HAP's training is per interval for stability.

2 champion-challenger deployment

Champion–challenger is a broad model-management principle: an existing model is in use while alternatives are evaluated for replacement. HAP shares this principle, but applying it to a particular task requires specifying how challengers are trained, how predictions are produced during evaluation, which feedback is available, and when to replace the model in use.

We distinguish HAP from shadow-only deployment [4] and ChaCha [5]. The following table highlights how they produce predictions while promotion/replacement remains undecided, Let Y^tI\widehat Y_t^{\mathrm I} and Y^tC\widehat Y_t^{\mathrm C} denote the incumbent and candidate forecasts.

ApproachPrediction before promotionDeployment implication
Shadow-only [4]Y^t=Y^tI\widehat Y_t=\widehat Y_t^{\mathrm I}Candidate forecasts are evaluated but not used for prediction.
ChaCha (Wu et al., 2021) [5]Y^t∈{Y^tI,Y^tC}\widehat Y_t\in\{\widehat Y_t^{\mathrm I},\widehat Y_t^{\mathrm C}\}Hard selection using uncertainty-adjusted losses.
HAPY^t=(1−λb,t)Y^tI+λb,tY^tC\widehat Y_t=(1-\lambda_{b,t})\widehat Y_t^{\mathrm I}+\lambda_{b,t}\widehat Y_t^{\mathrm C}Aggregation of a frozen pair through delayed Hedge.

Holding everything else fixed, the difference from shadow-only deployment reduces to λb,t≡0\lambda_{b,t}\equiv0. This can postpone using an useful adaptation before promotion. While ChaCha already permits challenger use before promotion, it selects one predictor at a time. HAP instead hedges models before promotion

Our contribution makes the champion–challenger principle operational for the persistent–transient ambiguity studied in the paper. In addition, applying the champion–challenger principle also distinguishes HAP from the strong online forecasting baselines compared in the paper.

References

[1] Heitor M. Gomes et al. Adaptive Random Forests for Evolving Data Stream Classification. Machine Learning, 2017. https://doi.org/10.1007/s10994-017-5642-8

[2] Albert Bifet and Ricard Gavaldà. Adaptive Learning from Evolving Data Streams. Advances in Intelligent Data Analysis VIII, 2009. https://doi.org/10.1007/978-3-642-03915-7_22

[3] Jesse Read and Indrė Žliobaitė. Supervised Learning from Data Streams: An Overview and Update. ACM Computing Surveys, 2025. See Section 5.5. https://arxiv.org/html/2212.14720v3

[4] Amazon Web Services. Shadow tests. Amazon SageMaker AI Developer Guide. https://docs.aws.amazon.com/sagemaker/latest/dg/shadow-tests.html

[5] Qingyun Wu, Chi Wang, John Langford, Paul Mineiro, and Marco Rossi. ChaCha for Online AutoML. Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11263–11273, 2021. See Section 3.2 and Algorithm 1. https://proceedings.mlr.press/v139/wu21d.html

AuthorsOpenReview ↗

Response to Q2, W2 and W3 (theoritical analysis and novelty, 1/4) – General response

[Q2, W2 and W3]. Which parts of theoretical results authors see as new compared with known bounds for hedge with delayed feedback? - Theory relies on ... is standard Diebold–Mariano test. - Theoretical part covers ... how method performs over whole benchmark interval.

We agree that our analysis builds on existing theoretical results and that our regret theorem concerns a single adaptation cycle. In our defense, we intend to use these results to clarify the identified problem and justify our method: the minimax result formalizes the cost of immediate commitment under persistent–transient ambiguity and motivates our method, while the regret bounds justify the use of hedge algorithm.

The additional theoretical contribution analyses what happens after the comparison interval. As pointed out by the reviewer, a within-interval regret bound does not establish whether decisions are beneficial for subsequent forecasting. To address this limitation, we first quantify the future cost of a promotion/retention decision, then combine that cost with the Hedge guarantee across successive intervals to establish an overall bound. We explain these results below before presenting their formal statements and proofs.

1. Cost of the promotion. The relevant question is not whether the DM test selects the winner between two models on the data used for comparison, but whether the selected model leads to more or less loss after the selection. The frozon models are compared on Ib+1I_{b+1} and the decision is made at τb+1\tau_{b+1}, so the seleted model will be deployed on Ib+2I_{b+2}. Therefore, we assess that decision on Ib+2I_{b+2} using the same frozen pair. Let CbC_b be the conditional average excess risk of the selected model relative to the better one of that pair on Ib+2I_{b+2}. Under reasonable assumptions, we obtain

E[Cb∣Ob]≤estimation-error bound+δ threshold contribution+ρb⏟change in conditional relative risk.\mathbb E[C_b\mid\mathcal O_b]\leq {}\text{estimation-error bound}+\delta~ \text{threshold contribution}+\underbrace{\rho_b}_{\text{change in conditional relative risk}}.

The first term comes from the uncertainty in estimating the models' relative performance and the second term is for the magnitude of the threshold of DM test. ρb\rho_b bounds how much the conditional relative risk on the next interval Ib+2I_{b+2} differs from the comparison interval Ib+1I_{b+1}. This result reflects that a promotion decision has a small expected cost when the comparison is accurate and remains informative on the next interval.

2. Comparator and regret. We introduce a rolling replacement oracle. The oracle knows conditional risks as the next deployment interval after decision. There, at each boundary, it selects whichever model of the compared incumbent–candidate pair has lower conditional expected loss over that interval. This comparator is appropriate for assessing model promotion: it can favor a model that HAP discarded, which the previous pair-oracle comparison would overlook. The selected model becomes the incumbent in the next pair. Consequently, the next-interval excess loss can be decomposed as

L(HAP)−L(oracle)⏟total excess loss=L(HAP)−L(selectedincumbent)⏟controlled by delayed Hedge+L(selectedincumbent)−L(oracle)⏟controlled by promotion risk,\underbrace{L(\mathrm{HAP})-L(\mathrm{oracle})}_{\text{total excess loss}} = \underbrace{L(\mathrm{HAP})-L(\mathrm{selected\\ incumbent})}_{\text{controlled by delayed Hedge}} + \underbrace{L(\mathrm{selected\\ incumbent})-L(\mathrm{oracle})}_{\text{controlled by promotion risk}},

where LL denotes cumulative loss over the next-interval. These new results explain how promotion decision affect subsequent forecasting and how their costs accumulate alongside the cost of hedged deployment. The following are the theorems and the proofs.

AuthorsOpenReview ↗

Response to Q2, W2 and W3 (theoritical analysis and novelty, 2/4) – Post-decision risk

Let τ0<⋯<τB\tau_0<\cdots<\tau_B, B≥2B\geq2, be deterministic integer update times, with τb+1−τb>H\tau_{b+1}-\tau_b>H. For b=0,…,B−1b=0,\ldots,B-1, write

Db+1=τb,…,τb+1−1,\Ib+1=τb,…,τb+1−H,nb+1=∣Ib+1∣,Qb+1=Db+1∖Ib+1.\begin{aligned}\mathcal D_{b+1}&=\\{\tau_b,\ldots,\tau_{b+1}-1\\},\\\I_{b+1}&=\\{\tau_b,\ldots,\tau_{b+1}-H\\},\qquad n_{b+1}=|I_{b+1}|,\\\\\mathcal Q_{b+1}&=\mathcal D_{b+1}\setminus I_{b+1}.\end{aligned}

The set Db+1\mathcal D_{b+1} contains all forecast origins in the interval. The H−1H-1 origins in Qb+1\mathcal Q_{b+1} have targets that are not yet fully observed at τb+1\tau_{b+1}. Their losses are scored once those targets become available.

We condition throughout on the pre-test information used to select η>0\eta>0 and δ∈(0,1)\delta\in(0,1), which are then fixed during testing. Let Ob\mathcal O_b contain the observations and training randomness available at τb\tau_b. The pair (fbI,fbC)(f_b^{\mathrm I},f_b^{\mathrm C}) remains fixed on Db+1\mathcal D_{b+1}. With the paper's normalized squared loss, set

Lt(f)=1Hd∣f(Xt)−Yt∣F2,Zb,t=Lt(fbC)−Lt(fbI).L_t(f)=\frac{1}{Hd}\\|f(X_t)-Y_t\\|_F^2,\qquad Z_{b,t}=L_t(f_b^{\mathrm C})-L_t(f_b^{\mathrm I}).

For analysis, the same definition of Zb,tZ_{b,t} is used on the following interval, with the old pair kept fixed. The deployed predictor is

fb,tdep=(1−λb,t)fbI+λb,tfbC,λb,t=σ(−η∑u=τbt−HZb,u),σ(x)=(1+e−x)−1,\begin{aligned}f_{b,t}^{\mathrm{dep}}&=(1-\lambda_{b,t})f_b^{\mathrm I}+\lambda_{b,t}f_b^{\mathrm C},\\\\\lambda_{b,t}&=\sigma(-\eta\sum_{u=\tau_b}^{t-H}Z_{b,u}),\qquad\sigma(x)=(1+e^{-x})^{-1},\end{aligned}

where an empty sum is zero. For b=0,…,B−2b=0,\ldots,B-2, define

Z‾b+1=1nb+1∑t∈Ib+1Zb,t,μb=E[Z‾b+1∣Ob].\overline Z_{b+1}=\frac{1}{n_{b+1}}\sum_{t\in I_{b+1}}Z_{b,t},\qquad\mu_b=\mathbb E[\overline Z_{b+1}\mid\mathcal O_b].

Thus, μb\mu_b is the paper's μb,n\mu_{b,n} at n=nb+1n=n_{b+1}. Let Ω^b≥0\widehat\Omega_b\geq0 denote the HAC estimate from Ib+1I_{b+1}. We consider the promotion rule

cb=z1−δΩ^bnb+1,ϕb+1=1Z‾b+1+cb<0,c_b=z_{1-\delta}\sqrt{\frac{\widehat\Omega_b}{n_{b+1}}},\qquad\phi_{b+1}=\mathbf1\\{\overline Z_{b+1}+c_b<0\\},

where z1−δz_{1-\delta} is the standard normal quantile. The retained model is fb+1I=fbCf_{b+1}^{\mathrm I}=f_b^{\mathrm C} when ϕb+1=1\phi_{b+1}=1, and fbIf_b^{\mathrm I} otherwise. It initializes the next candidate.

Post-decision risk

For the pair formed at τb\tau_b, define

rbg=1nb+2E[∑t∈Ib+2Lt(fbg),∣,Ob+1],g∈I,C,νb=rbC−rbI.r_b^g=\frac{1}{n_{b+2}}\mathbb E[\sum_{t\in I_{b+2}}L_t(f_b^g)\\,|\\,\mathcal O_{b+1}],\qquad g\in\\{\mathrm I,\mathrm C\\},\qquad\nu_b=r_b^{\mathrm C}-r_b^{\mathrm I}.

The conditional excess risk of the retention decision is

Cb=(1−ϕb+1)rbI+ϕb+1rbC−min⁡rbI,rbC=ϕb+1[νb]++(1−ϕb+1)[−νb]+,\begin{aligned}C_b&=(1-\phi_{b+1})r_b^{\mathrm I}+\phi_{b+1}r_b^{\mathrm C}-\min\\{r_b^{\mathrm I},r_b^{\mathrm C}\\}\\\\&=\phi_{b+1}[\nu_b]_++(1-\phi_{b+1})[-\nu_b]_+,\end{aligned}

where [x]+=max⁡x,0[x]_+=\max\\{x,0\\}.

Assumption A1 (Conditional moment bounds). For each b=0,…,B−2b=0,\ldots,B-2, there exist nonnegative vb,wbv_b,w_b such that,

Var⁡(Z‾b+1∣Ob)≤vb2nb+1,E[Ω^b∣Ob]≤wb2.\operatorname{Var}(\overline Z_{b+1}\mid\mathcal O_b)\leq\frac{v_b^2}{n_{b+1}},\qquad\mathbb E[\widehat\Omega_b\mid\mathcal O_b]\leq w_b^2.

Assumption A2 (Near-future representativeness). For each b=0,…,B−2b=0,\ldots,B-2, there exists a nonnegative ρb\rho_b such that E[∣νb−μb∣∣Ob]≤ρb.\mathbb E[|\nu_b-\mu_b|\mid\mathcal O_b]\leq\rho_b.

These assumptions are imposed on the pairs generated by HAP and they supplement the paper's local comparison assumption.

Theorem 2 (Post-decision risk)

Under Assumptions A1-A2, for every b=0,…,B−2b=0,\ldots,B-2,

E[Cb∣Ob]≤εbsel:=vb+∣z1−δ∣wbnb+1+ρb.\mathbb E[C_b\mid\mathcal O_b]\leq\varepsilon_b^{\mathrm{sel}}:=\frac{v_b+|z_{1-\delta}|w_b}{\sqrt{n_{b+1}}}+\rho_b.

In particular, for every γ>0\gamma>0,

Pr⁡(ϕb+1≠1νb<0,;∣νb∣≥γ,∣,Ob)≤min⁡1,εbselγ.\Pr(\phi_{b+1}\neq\mathbf1\\{\nu_b<0\\},\\;|\nu_b|\geq\gamma\\,|\\,\mathcal O_b)\leq\min\\{1,\frac{\varepsilon_b^{\mathrm{sel}}}{\gamma}\\}.

Proof. For p∈0,1p\in\\{0,1\\}, let

Fp(x)=p[x]++(1−p)[−x]+.F_p(x)=p[x]_++(1-p)[-x]_+.

The map FpF_p is 11-Lipschitz and positively homogeneous. If ϕb+1=1\phi_{b+1}=1, then Z‾b+1<−cb\overline Z_{b+1}<-c_b and Fϕb+1(Z‾b+1)≤[−cb]+F_{\phi_{b+1}}(\overline Z_{b+1})\leq[-c_b]_+. If ϕb+1=0\phi_{b+1}=0, then Z‾b+1≥−cb\overline Z_{b+1}\geq-c_b and Fϕb+1(Z‾b+1)≤[cb]+F_{\phi_{b+1}}(\overline Z_{b+1})\leq[c_b]_+. Hence, for either sign of cbc_b,

Cb=Fϕb+1(νb)≤∣cb∣+∣νb−Z‾b+1∣≤∣cb∣+∣Z‾b+1−μb∣+∣νb−μb∣.\begin{aligned}C_b&=F_{\phi_{b+1}}(\nu_b)\\\\&\leq|c_b|+|\nu_b-\overline Z_{b+1}|\\\\&\leq|c_b|+|\overline Z_{b+1}-\mu_b|+|\nu_b-\mu_b|.\end{aligned}

By conditional Cauchy-Schwarz, Jensen's inequality, and Assumption A1,

E[∣Z‾b+1−μb∣∣Ob]≤Var⁡(Z‾b+1∣Ob)≤vbnb+1,E[∣cb∣∣Ob]≤∣z1−δ∣nb+1E[Ω^b∣Ob]≤∣z1−δ∣wbnb+1.\begin{aligned}\mathbb E[|\overline Z_{b+1}-\mu_b|\mid\mathcal O_b]&\leq\sqrt{\operatorname{Var}(\overline Z_{b+1}\mid\mathcal O_b)}\leq\frac{v_b}{\sqrt{n_{b+1}}},\\\\\mathbb E[|c_b|\mid\mathcal O_b]&\leq\frac{|z_{1-\delta}|}{\sqrt{n_{b+1}}}\sqrt{\mathbb E[\widehat\Omega_b\mid\mathcal O_b]}\leq\frac{|z_{1-\delta}|w_b}{\sqrt{n_{b+1}}}.\end{aligned}

Taking conditional expectations and applying Assumption A2 proves the risk bound. On the event in the second statement, Cb=∣νb∣≥γC_b=|\nu_b|\geq\gamma. The probability bound follows from conditional Markov's inequality. □\square

Reviewer_inGvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Hedge and Promote (HAP), an online forecasting framework. Specifically, it maintains two models: the Incumbent and the Candidate models. It uses a Hedge-based ensemble to combine these two models. It then applies one-sided Diebold-Mariano (DM) test to determine if the candidate should permanently replace the incumbent model.

Critical strengths
  1. This is a well-motivated paper. It addresses an important problem in real-world forecasting applications.
  2. The proposed framework is conceptually simple and interpretable. It is model-agnostic and can be combined with almost any forecasting algorithms.
  3. Competitive experimental results. HAP achieves the lowest MSE on 6 out of 15 forecasting tasks.
Critical weaknesses
  1. Limited methodology novelty. The proposed framework basically combines Hedge-based ensemble learning with statistical model comparison using the Diebold-Mariano test.
  2. The theoretical analysis is also based on well-established results. It provides two theoretical results: a minimax bound and a sublinear within-cycle regret bound. In particular, the established regret bound only guarantee within a single adaptation cycle, and it does not establish an overall regret bound.
  3. Insufficient ablation studies. Some important missing baselines include (but not limited) 1) Hedge without statistical promotion; 2) DM-based promotion without Hedge; 3) Fixed-weight incumbent-candidate averaging.
Questions for authors
  1. Can you perform additional ablation studies to justify the designed framework?
10 comments
AuthorsOpenReview ↗

Response to Q1 and W3 (ablations)

[Q1 and W3]: Can you perform additional ablation studies to justify the designed framework? - Insufficient ablation studies. Some important missing baselines include (but not limited) 1) Hedge without statistical promotion; 2) DM-based promotion without Hedge; 3) Fixed-weight incumbent-candidate averaging.

Thank you for requesting these comparisons. We have evaluated three ablations as suggested to examine the roles of Hedge prediction and statistical promotion. Let λb,t\lambda_{b,t} denote the candidate's weight in the deployed forecast.

AblationsPrediction during comparisonPromotion at the next boundary
Hedge without statistical promotionKeep adaptive Hedge weightsSelect the model with lower empirical mean loss
DM-based promotion without Hedgeλb,t=1\lambda_{b,t}=1 use only the candidate's forecastKeep DM-based promotion
Fixed-weight incumbent-candidate averagingλb,t=1/2\lambda_{b,t}=1/2 equally average the two forecastsKeep DM-based promotion

For DM-based promotion, instead of fixing δ=0.2\delta=0.2 we select δ∈0.2,0.4,0.5,0.6,0.8\delta\in \\{0.2,0.4,0.5,0.6,0.8 \\} separately for each experiment using the validation stream before online testing. The Hedge without statistical promotion ablation is equivalent to fixing δ=0.5\delta=0.5.

We report the test MSE in the following table. The ranks are computed separately within each task and then averaged.

DatasetHHHedge without statistical promotionDM-based promotion without HedgeFixed averaging + DMHAP
ETTm1240.4742 ± 0.05760.4916 ± 0.05810.4784 ± 0.06900.4711 ± 0.0563
ETTm1480.6029 ± 0.00270.6247 ± 0.00510.5948 ± 0.00240.5979 ± 0.0072
ETTm1960.7687 ± 0.06390.7901 ± 0.06170.7870 ± 0.06350.7588 ± 0.0668
ETTh2242.0530 ± 0.06472.0428 ± 0.05882.3513 ± 0.03341.9882 ± 0.0590
ETTh2483.6612 ± 0.26103.4748 ± 0.10803.8306 ± 0.06773.6061 ± 0.2395
ETTh2966.1043 ± 0.01436.1373 ± 0.01186.2749 ± 0.04776.0924 ± 0.0223
Electricity244.1027 ± 0.00664.1166 ± 0.02034.0952 ± 0.01584.0927 ± 0.0098
Electricity484.8169 ± 0.01734.8709 ± 0.02004.8195 ± 0.01734.8118 ± 0.0160
Electricity966.0177 ± 0.01496.1072 ± 0.01066.0184 ± 0.01006.0040 ± 0.0121
Weather240.7364 ± 0.00550.7400 ± 0.00470.7353 ± 0.00640.7344 ± 0.0049
Weather481.0108 ± 0.00271.0204 ± 0.00701.0105 ± 0.00361.0080 ± 0.0059
Weather961.3521 ± 0.00351.3695 ± 0.00401.3509 ± 0.01081.3449 ± 0.0064
Traffic240.3663 ± 0.00040.3666 ± 0.00020.3665 ± 0.00020.3662 ± 0.0003
Traffic480.3843 ± 0.00050.3846 ± 0.00040.3845 ± 0.00050.3841 ± 0.0005
Traffic960.3979 ± 0.00020.3987 ± 0.00070.3980 ± 0.00020.3977 ± 0.0002
Average rank ↓2.473.602.801.13

Even though several improvements are small, HAP achieves the lowest mean MSE on 13 of 15 tasks. So these results support the Hedge then Promote framework over the tested ablations.

AuthorsOpenReview ↗

Response to W1 (methodology novelty)

[W1]: Limited methodology novelty. The proposed framework basically combines Hedge-based ensemble learning with statistical model comparison using the Diebold-Mariano test.

We agree that Hedge algorithm and Diebold–Mariano test are established tools, and we didn't invent new Hedge algorithm or statistical test. Our contribution instead addresses an important practical problem in online time series forecasting: learning newly observed behavior can improve future predictions when that behavior persists, but can be harmful when it disappears. This ambiguity is intrinsic to real-world application.

Although previous works have recognized this difficulty, identifying it does not by itself specify how updated model should be deployed and retained. HAP provides a concrete response by separating the opportunity to learn from the decision to retain an adaptation, allowing updates to influence predictions provisionally while their usefulness is assessed on subsequent observations. The methodological contribution we claim is this problem-driven framework and its demonstrated operational value, rather than novelty of its ingredients.

AuthorsOpenReview ↗

Response to W2 (theoretical analysis, 1/4) – General response

[W2]: The theoretical analysis is also based on well-established results. It provides two theoretical results: a minimax bound and a sublinear within-cycle regret bound. In particular, the established regret bound only guarantee within a single adaptation cycle, and it does not establish an overall regret bound.

We agree that our analysis builds on existing theoretical results and that our regret theorem concerns a single adaptation cycle. In our defense, we intend to use these results to clarify the identified problem and justify our method: the minimax result formalizes the cost of immediate commitment under persistent–transient ambiguity and motivates our method, while the regret bounds justify the use of hedge algorithm.

To address the limitation about the regret bound, we extend the analysis. First, we quantify the future cost of a promotion/retention decision. Then we combine that cost with the Hedge guarantee across successive intervals to establish an overall bound. Before giving the new theorems and proofs, we first explain the meaning of these two results.

1. Cost of the promotion. The relevant question is not whether the DM test selects the winner between two models on the data used for comparison, but whether the selected model leads to more or less loss after the selection. The frozon models are compared on Ib+1I_{b+1} and the decision is made at τb+1\tau_{b+1}, so the seleted model will be deployed on Ib+2I_{b+2}. Therefore, we assess that decision on Ib+2I_{b+2} using the same frozen pair. Let CbC_b be the conditional average excess risk of the selected model relative to the better one of that pair on Ib+2I_{b+2}. Under reasonable assumptions, we obtain

E[Cb∣Ob]≤estimation-error bound+δ threshold contribution+ρb⏟change in conditional relative risk.\mathbb E[C_b\mid\mathcal O_b] \leq {} \text{estimation-error bound} +\delta~ \text{threshold contribution}+\underbrace{\rho_b}_{\text{change in conditional relative risk}}.

The first term comes from the uncertainty in estimating the models' relative performance and the second term is for the magnitude of the threshold of DM test. ρb\rho_b bounds how much the conditional relative risk on the next interval Ib+2I_{b+2} differs from the comparison interval Ib+1I_{b+1}. This result reflects that a promotion decision has a small expected cost when the comparison is accurate and remains informative on the next interval.

2. Comparator and regret. We introduce a rolling replacement oracle. The oracle knows conditional risks as the next deployment interval after decision. There, at each boundary, it selects whichever model of the compared incumbent–candidate pair has lower conditional expected loss over that interval. This comparator is appropriate for assessing model promotion: it can favor a model that HAP discarded, which the previous pair-oracle comparison would overlook. The selected model becomes the incumbent in the next pair. Consequently, the next-interval excess loss can be decomposed as

L(HAP)−L(oracle)⏟total excess loss=L(HAP)−L(selectedincumbent)⏟controlled by delayed Hedge+L(selectedincumbent)−L(oracle)⏟controlled by promotion risk,\underbrace{L(\mathrm{HAP})-L(\mathrm{oracle})}_{\text{total excess loss}} = \underbrace{L(\mathrm{HAP})-L(\mathrm{selected\\ incumbent})}_{\text{controlled by delayed Hedge}} + \underbrace{L(\mathrm{selected\\ incumbent})-L(\mathrm{oracle})}_{\text{controlled by promotion risk}}, where LL denotes cumulative loss over the next-interval. These new results explain how promotion decision affect subsequent forecasting and how their costs accumulate alongside the cost of hedged deployment. The following are the theorems and the proofs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.