- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies cautious adaptation in online forecasting. Intuitively, to have robust online updates in forecasting we need to distinguish true persistent changes from transient episodes (e.g. noise and outliers). The paper proposes a rule to evaluate the recently updated model with its frozen counterpart, to decide whether the update should be kept or rejected. The proposed approach is evaluated on standard forecasting benchmarks.
Critical strengths
- I like the framing of "cautious deployment". It's a nice practical framing that highlights the main limitation of online forecasters (i.e. their very poor robustness to transients)
- the proposed method is simple and practical
Critical weaknesses
writing:
- some parts of the paper excessively rely on ambiguous and unexplained jargon. As an example, at the start of Sec. 3: "We then formulate the incumbent-candidate deployment decision and show how it gives rise to persistent–transient ambiguity and an unavoidable cost of immediate model promotion.". At this point, the incumbent, the ambiguity, and the "unavoidable cost", "immediate promotion" are all undefined. I can guess what it means, but the jargon makes reading it harder than necessary, without adding any specific information.
- more in general, the paper proposes a relatively simple method that is made harder to understand by the opaque description
experiments are a bit limited:
- only the MSE metric is shown
- limited baselines: the paper should compare against methods designed to protect against forgetting in online forecasting, such as [1, 2]. These methods are often explicitly designed to adapt to the transient and then quickly recover once it's finished.
- the proposed method in general does not seem to consistently improve the performance
- The paper remarks the importance of robustness to attacks several times. However, this is never tested.
[1] Pham, Quang, et al. "Learning fast and slow for online time series forecasting." ICLR 2023 [2] Wen, Qingsong, et al. "Onenet: Enhancing time series forecasting models under concept drift by online ensembling." NeurIPS 2023
Questions for authors
- the method seems dependent on the relationship between the update window size, horizon, and the length of the transient. I did not see an analysis of this in the paper
- In table 1, I understand why HAP is better in the transient case. Can you motivate why it's much better also in the persistent case? Are the other methods slower to adapt?
8 comments
Response to Q2 (persistent case explanation)
[Q2]. In table 1, I understand why HAP is better in the transient case. Can you motivate why it's much better also in the persistent case? Are the other methods slower to adapt?
We thank the reviewer for this interesting question, we have investigated on temporal losses (daily and six hours) across all runs to understand why HAP beats baselines in persistant case. It turns out that HAP is worse for the first 6 hours during the 30 days evalution interval, then becomes better for the rest of the interval.
At the branching boundary, HAP's candidate has already been trained on the intervention just as baselines. HAP is actually worse immediately after branching: all four baselines outperform it during the first 6 hours. However, the candidate is already substantially better than the incumbent (2~3 times lower MSE). Actually, HAP initially assigns equal weight, so its early disadvantage comes from the incumbent. The first complete forecast feedback arrives 12 hours after branching. At that hedge update, the candidate weights reach 0.986–1.000 across all runs, and it dominates deployment hereafter.
In fact, the baselines are not necessarily slow to adapt. Rather, HAP has a useful candidate that performs better than baselines in this experiment. The fixed candidate available at branching is held fixed for the entire 30-day evaluation, its persistent MSE averaged over is 0.6663. Even without further adaptation, the candidate is better than all baselines:
A plausible explanation for this is updating per batch can be more stable than continuouly updating when dealing with streaming data. HAP updates each candidate over the completed intervals of forecasting examples. In this experiment, that batch contains repeated observations of the modified daily profile. Fitting over this broader temporal window can help to learn than reacting to local fluctuations. As Read and Žliobaitė (2025, Sections 5.3 and 5.6) emphasize, continuously issuing predictions does not require updating model parameters after every observation, the appropriate update frequency depends on the stability of the predictive structure [1]. HAP's advantage over baselines in this experiemnt is the learning of a strong candidate from the batch and rapidly increasing its predictive influence during hedged deployment.
Reference
[1] Jesse Read and Indrė Žliobaitė. Supervised Learning from Data Streams: An Overview and Update. ACM Computing Surveys, 2025. See Sections 5.3 and 5.6. https://arxiv.org/html/2212.14720v3
Response to W1 and W2 (writing)
[W1, W2]. some parts of the paper excessively rely on ambiguous and unexplained jargon. As an example, at the start of Sec. 3: "We then formulate the incumbent-candidate deployment decision and show how it gives rise to persistent–transient ambiguity and an unavoidable cost of immediate model promotion.". At this point, the incumbent, the ambiguity, and the "unavoidable cost", "immediate promotion" are all undefined. I can guess what it means, but the jargon makes reading it harder than necessary, without adding any specific information. - more in general, the paper proposes a relatively simple method that is made harder to understand by the opaque description
Thank you for highlighting the passages that were difficult to follow. We would first like to clarify that the concepts mentioned at the start of Sec. 3 are introduced earlier in the paper.
In the Introduction, the paragraph “A motivating paired example” explicitly defines the two model roles (lines 078–079):
“We call the currently retained model the incumbent and the model adapted on the candidate.”
The same example explains persistent–transient ambiguity: two continuations share the same history but favor different models afterward (lines 075–082). The following paragraph motivates delaying immediate replacement while subsequent observations provide evidence about the update’s usefulness (lines 083–085). Then, in the first contribution ("In Section 3, we formulate..."), we state the strictly positive worst-case excess risk of committing immediately under indistinguishable continuation (lines 088–091).
Nevertheless, we agree that the start of the Sec. 3 compresses these concepts into one sentence and can be presented more clearly. We propose the following formulation:
“We then study whether to keep the current model (incumbent) or replace it with an adapted model (candidate). When the same observed history admits two future continuations, any rule forced to select one model incurs additional expected loss in at least one continuation.”
More generally, we will make the formal setting more clear and provide reminders of the concepts where needed.
Response to Q1 (duration relationship)
[Q1]. the method seems dependent on the relationship between the update window size, horizon, and the length of the transient. I did not see an analysis of this in the paper
We thank the reviewer for raising this point, and we agree that the relationship between these time scales should affect HAP’s performance. We have added a timing-sensitivity experiment on the transient case of semi-synthetic ETTm1. We vary the window size U∈9,15,21 (days) and the transient length D/U∈1/3,2/3,1,4/3,5/3,2,7/3,8/3. The intervention magnitude is fixed at Δ=6. and the forecasting horizon is fixed at H=48 (12 hours).
The intervention starts at the same time s and ends at s+D. s is aligned with an update boundary. For each fixed U, all transient length D/U are evaluated over the same evaluation interval [s+U,s+3U). This correponds to the two intervals following the first update after s. During the evaluation interval, HAP continues its normal operations. The MSE are reported below (See a more visual figure of the same results).
These results show that performance depends on D/U as suggeted by the reviewer. More precisely:
1. HAP is robust to transient event when transient length is smaller than window size: when the transient lasts less than window size, the candidate is not fully trained on transient behaviour and it receives smaller weights than the incumbent during hedged prediction on the subsequent interval.
2. HAP performs worst when the transient event spans exactly both the candidate training interval and the subsequent comparison interval: the largest MSE occurs at D=2U for all tested window size. At s+2U, the DM test would promote a model adapted on the transient, just as the transient disappears.
3. For transient length longer than two intervals, HAP's error decreases.: in this case, more of the evaluation interval is covered by the transient event, where adaptation can be more useful than harmful. In fact, with the transient length being longer (in terms of number of intervals), the transient event starts to be "persistent".
We thank the reviewer again for this interesting question. The results are quite aligned with HAP's learning and deployment mechanism. We hope that this experiemental analysis has responded to your question.