acceptodds
Under review as a conference paper at ICLR 2027

Local Risk Structure for Adaptive Retraining in Streaming Time Series

Abstract

Predictors for streaming time series must decide how much new data to collect before updating, even though the future benefit of an update is unknown. We propose RETICLE (Risk Evaluation Through Interval Comparisons of Local Errors), a data-only audit of accumulated-data quality for selecting an update size without retraining the downstream predictor at each candidate size. RETICLE fits causal local affine probes in a frozen representation and assesses their risk across bandwidths. We show that approximate quasiconvexity of the risk profile on a finite bandwidth grid is necessary for finite-sample local-learning regularity; monotone profiles are included as special cases. Using this necessary consequence as a data-quality criterion, RETICLE certifies the risk-shape condition with simultaneous confidence bounds and selects the first adequately supported checkpoint that passes. Under causal auditing assumptions, we control the probability of false shape certification simultaneously over the declared checkpoints and representations, including the selected checkpoint, without requiring independent observations. Experiments examine checkpoint selection for six downstream learners on synthetic and real time series, as well as continuous fine-tuning on three forecasting datasets with two drift detectors. RETICLE achieves the lowest test error in 12 of 18 online settings, ties for the lowest in two more, and attains an overall mean rank of 1.28 among the compared baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
Streaming time series, Update data size selection, Concept drift, Adaptive retraining, Local regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iHfLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much post drift data should be collected before retraining a streaming time series predictor. It proposes RETICLE, a learner independent audit that evaluates causal local affine probes in a frozen predictive representation and selects the first checkpoint whose risk profile across bandwidths is certified as approximately quasiconvex. The paper establishes simultaneous control of false shape certification under temporal dependence and checkpoint selection. Experiments evaluate the selected checkpoints across six downstream learners and continuous adaptation on three forecasting streams with two drift detectors, where RETICLE generally outperforms CALIPER and validation tuned fixed waiting.

Critical strengths
  1. Principled and statistically valid formulation. RETICLE derives approximate quasiconvexity from the local bias variance structure and provides simultaneous confidence guarantees over declared checkpoints and representations under temporal dependence. This gives the selection rule a substantially stronger statistical foundation than a purely heuristic update trigger.

  2. Careful validation across learners and online settings. The checkpoint experiments apply a single learner independent selection to six heterogeneous downstream models, while the online study evaluates three forecasters with two drift detectors and validation tuned baselines. The consistent downstream gains across these settings provide meaningful evidence that the audited local risk structure is practically useful.

Critical weaknesses
  1. The certificate is only indirectly connected to retraining utility. A PASS certifies approximate quasiconvexity of the local probe risk profile, but it does not establish low absolute risk, full local learning regularity, improvement after retraining, or optimality of the selected checkpoint. The theoretical guarantee therefore validates the proxy decision rather than the main downstream objective.

  2. Learner independence and generalization are only partially demonstrated. In the online study, the tolerance coefficient is selected separately for each dataset, detector, and downstream model using validation MSE. Moreover, the reported 18 conditions are generated from only three individual streams. It therefore remains unclear whether a shared learner independent configuration would transfer reliably across models and unseen streams.

Questions for authors
  1. Is RETICLE intended to be learner independent only after the coefficient cc has been tuned for a particular downstream model? How would cc be selected for a new learner or stream when downstream validation replay is unavailable?

  2. Across the checkpoint experiments, how often does a PASS correspond to improved downstream performance relative to both updating at the preceding checkpoint and waiting until the next checkpoint? This would clarify whether the certified risk shape is a reliable proxy for practical update timing.

Reviewer_z37YOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RETICLE, a method for selecting retraining data sizes in streaming time series. It computes quasiconvexity of risk across bandwidths, or weights across past datapoints, with error control.

Critical strengths
  • Advances the tradeoff between data size for update vs. model accuracy one step further.
  • Provides statistical guarantees for temporal dependence and checkpoint selection.
Critical weaknesses

The paper was an enjoyable read, but I found the following potential concerns. Note that the following represents my initial impression of the paper, and I am open to discussion. I welcome any corrections to my potential misunderstandings.

Reproducibility cocnern

  • The code is inaccessible. The provided anonymous github link (App. K) shows "This content can't be shown" that prevents reproduction of the results.
  • Main online results lack uncertainty reports. Table 1 provides no variability (e.g., standard deviation, confidence intervals) across repeated runs. Without repeated runs with uncertainty estimates, readers cannot tell if the performance gain is stable across trials or just one champion data. Ideally the gain should be analyzed with a proper statistical test.
  • Missing comparison with continual, incremental updating. A straightforward alternative to the delayed updating is to update the model each time step. The manuscript does not fully motivate why we need to concern more complex delayed update over the simpler continual updating (perhaps due to computational intensity, but one can reduce learning rate to run small dataset for lightweight training), and does not conduct comparison experiment either. CALIPER paper does have comparison against the incremental updating, but CALIPER paper uses different datasets and forcasting horizons, and this paper replaces CAQLIPER's original full retraining procedure with one-epoch finetuning.
  • Insufficient ablation. Experiment vs. CALIPER changes many experimental conditions such as representation, probe regularization, shape criterion, and predicted varables. Thus the performance gain cannot automatically be attributed to their proposed framework. Controlled ablations are needed to establish the advantage of the quasiconvexity monitoring.

Gap between risk certification and checkpoint performance guarantees

  • How does the quasiconvexity guarantee the checkpoint performance? If I understand correctly, the presented theory that audits quasiconvexity of the risk curve with mathematical rigor, while the bridge between the theory and practical performance guarantee of the checkpoint is terse. If so, this paper can be reduced to nearly independent theory and experiment, each of which can be useful standalone but together much weaker than the paper suggests.
  • When does the auditing end? Given that the risk auditing is a sequential task with optional stopping, existence of termination guarantee is practically necessary but not provided. For example, the classical sequential hypothesis test on i.i.d. stream automatically guarantees termination in finite hitting time (e.g., Wald 1947), and non-i.i.d. stream requires additional assumptions to guarantee the same. What would be a minimal assumption set for guarantee for PASS criterion?
Questions for authors

Please provide point-by-point clarifications to my concerns above.

Reviewer_FCTMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RETICLE as a novel approach to determine how much data to collect after a drift before retraining a forecaster. Once configured, RETICLE is data-only and does not need to query or retrain the downstream predictor. After a regime shift happens in the data, RETICLE fits local linear probes in a learned (and then frozen) representation of the data, at different bandwidths, and measures their prediction error across bandwidths. The authors showed that approximate quasiconvexity of the error profile across bandwidths is a necessary condition for local-learning regularity. RETICLE checks this shape with confidence bounds that remain valid at the selected checkpoint. The first checkpoint that passes this check (and has enough local data support) is selected for retraining the forecaster. The authors supported their approach with experiments on both checkpoint selection and continuous fine-tuning.

Critical strengths
  1. Very relevant problem. The authors tackled a real-world problem faced by professional forecasters: after a change is detected, how much new data to wait for before retraining the model.

  2. The experiments cover many settings: three synthetic systems and 16 real cases with six different learners for checkpoint selection, and three forecasting datasets with two drift detectors for online fine-tuning.

  3. Analyzing the shape of the local probe error as a function of the bandwidth is an interesting idea. It follows the bias-variance trade-off of local estimators: large bandwidths give more bias, and small bandwidths give more variance.

Critical weaknesses
  1. The paper is very difficult to follow and understand. The notation is extremely heavy, and many fundamental words are used long before a clear definition is given in this context. For example, the bandwidth is only defined through the kernel in Eq. 5, and checkpoints only in Section 3.2. The intuition behind the method (the bias-variance trade-off in Figure 1b) is given in only a few sentences and is quickly buried under the notation. This makes it very difficult to actually learn something useful from this work.

  2. I find it very difficult to understand the reasons behind Definition 1. This is particularly important because Proposition 1, which justifies using quasiconvexity as the selection criterion, is strictly based on this Definition. It is possible that I missed something, but from my understanding Proposition 1 follows almost directly from your own Definition 1, so I find it difficult to see its value.

Questions for authors
  1. Can you please add an intuitive and more general justification of the reasons behind the construction of Definition 1, and explain what general value Proposition 1 brings?

  2. Can you please report how many checkpoints are classified as “VIOLATED” or “UNRESOLVED” before one is selected (“PASS”)?

  3. Your method can potentially add a significant computational burden. Appendix H gives the asymptotic cost (quadratic in the number of samples in the worst case), but no actual running times are reported. Can you please report the computation time of RETICLE compared to the other approaches you tested and to the fine-tuning itself?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.