acceptodds
Under review as a conference paper at ICLR 2027

Anytime-Valid Detection of Scale-Regime Drift for Conformal Time-Series Forecasting

Abstract

Conformal prediction intervals for time series can silently lose coverage when the data-generating process switches between scale regimes. Fixed calibration windows then underestimate uncertainty, while continuously adaptive methods often recover coverage only by substantially widening the intervals. We propose CREST (Conformal Recalibration via an Anytime-valid Scale-drift detector), a method that detects scale changes online and uses the resulting alarms to trigger conformal recalibration. CREST applies a blockwise e-process to seasonally whitened and locally standardized residuals, which yields anytime-valid false-alarm control under a local-scale null. To preserve this guarantee across repeated detector restarts, we introduce a budgeted restart scheme that allocates the false-alarm budget harmonically across segments. Experiments on real financial and weather time series show that CREST restores coverage close to the nominal level while maintaining interval widths comparable to a rolling baseline and requiring no trained state-space model. On car-index and weather data, CREST improves empirical coverage from 64.9% to 88.7% and from 77.1% to 89.8%, respectively, bringing both close to the nominal 90% target.

Primary area:
learning on time series and dynamical systems
Keywords:
conformal prediction, anytime-valid inference, e-process, change-point detection, time-series forecasting, scale-regime drift, non-stationarity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mNL5OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces CREST (Conformal Recalibration via an Anytime-valid Scale-drift detector), an online framework for event-triggered conformal recalibration under changes in the scale of time-series residuals. Fixed calibration can severely undercover after variance increases, while continuously adaptive calibration can produce wider intervals.

Critical strengths
  1. Principled Sequential Detection with Budgeted False-Alarm Control: Under the exact conditional null, the blockwise e-statistics, geometric mixture, and Ville bound provide a coherent detector, while the harmonic schedule preserves the false-alarm budget across an unbounded number of restarts. This guarantee concerns alarms and is distinct from conformal coverage. Table 10 supports the value of budgeting empirically: over five independent null segments, fixed-level restarts yield global FAR 0.093 versus 0.023 with harmonic levels. The construction offers an interpretable alarm stream and a computationally inexpensive control mechanism for recalibration.

  2. Useful Coverage–Width Trade-Off and Measured Post-Alarm Recovery: The main results preserve the empirical strength identified in the initial review: CREST substantially improves over Fixed on CarIndex and Weather, with widths 2.03 and 2.33, much narrower than ACI's 3.16 and 5.18. Table 3 locates the benefit more precisely: in the synthetic delay window CREST and Rolling both cover 0.591, while recovery-window coverage is 0.853 versus 0.785. Table 16 additionally shows unchanged coverage to three decimals on Weather and ETTm2 for misspecified lags ℓ∈{1,12,24,48}\ell\in\{1,12,24,48\}, with alarm-rate changes below 0.04 per 1,000 steps. This supports practical robustness to the tested lag choices, although it does not establish null calibration under arbitrary remaining dependence.

Critical weaknesses
  1. Remaining Gap Between Idealized Alarm Guarantees and Raw Plug-In Deployment: The rebuttal resolves which detector generated the main results: the raw chi-square tilt. The reported Gaussian-null FAR 0.044 at nominal 0.05 is reassuring finite-horizon evidence, already present in the original manuscript; it supplies no evidence of significant Gaussian miscalibration, although it cannot establish exact uniform-in-time validity of the estimated procedure. The substantive remaining concern is heavy-tail sensitivity: Table 12 reports raw-tilt FAR 0.736 under t(5)t(5) and 0.988 under t(3)t(3), versus 0.028 in that table's Gaussian setting. Appendix E explains the divergent positive moment-generating function of squared heavy-tail innovations. Robustification is a meaningful mitigation: Table 13 reports H13 FAR 0.000 and 0.008, with detection rates 1.000 and 0.988 and median delays 105 and 85 steps under t(5)t(5) and t(3)t(3), respectively. W05 detects faster at a median 45 steps, but its t(3)t(3) FAR is 0.088; self-normalized FAR is 0.096–0.208. Appendix E explicitly leaves conservative analytic normalization for future work. The lag sweep addresses a particular whitening error, not general dependence or heavy tails. These synthetic failures do not establish the real datasets' false-alarm probabilities, but leave the calibration of the main raw detector in non-Gaussian deployments unresolved.

  2. Coverage Benefits Are Local and Depend on the Forecast-Error Stream: Table 3 answers the original transition question: CREST improves recovery coverage from Rolling's 0.785 to 0.853, but does not improve the 0.591 delay-window coverage; settled coverage reaches 0.898. This is useful recovery evidence, with a remaining period of practical undercoverage rather than a violation of Proposition 1. The initial review's claim of seamless near-nominal transfer must also be corrected: the original manuscript already reported CarIndex coverage of 0.778–0.790 for WPMixer and the TimeMixer variants, unchanged in updated Tables 5/11. Appendix I now offers an explanation: their forecast errors can diverge from the AR residual stream monitored by the detector. This limits transferability without establishing new deterioration after revision. Aggregate comparisons provide context for the local benefit: Table 2's coverage gain over Rolling is 0.001 on Weather, Exchange, and ETTm2, and Table 17's mean normalized Winkler scores are 0.863 for CREST, 0.862 for Rolling, and 0.861 for CM. Table 4 likewise finds similar coverage and widths for locally normalized baselines. These results support competitive prediction and an additional alarm interface; the small aggregate differences do not negate the measured recovery gain. Marginal channel coverage is accepted as the stated scope, and the α/G\alpha/G allocation controls alarm budgets rather than establishing joint predictive coverage.

Questions for authors
  1. Robust Deployment Evidence: The main-variant clarification resolves which implementation was used. If a raw-versus-W05/H13 comparison or a robustification-threshold sweep already exists on Future, CarIndex, or Exchange, what does it show about downstream coverage, width, and alarm frequency? This follows the original question about sensitivity to robustification and would help assess whether the synthetic mitigation preserves the main empirical benefits. Real-data alarm frequency alone would not identify false-alarm probability without known change labels.

  2. Detection After Many Restarts: The harmonic schedule gives log⁡(1/αj)=constant+2log⁡j\log(1/\alpha_j)=\mathrm{constant}+2\log j, and Table 10 verifies budgeting over five independent null segments. Is there existing detection-delay or recovery-coverage evidence after many genuine regime changes? This is the empirical part of the original long-horizon question: the log threshold grows logarithmically, while the evidence threshold 1/αj1/\alpha_j grows quadratically. The false-alarm guarantee is accepted under its stated assumptions; late-segment detection performance remains a secondary uncertainty, not an additional experiment required for acceptance.

Reviewer_v4XhOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on conformal methods for time series forecasting under scale-regime drift. In this setting the noise variance changes abruptly while point forecasts stay roughly accurate, so the conformal calibration set becomes stale. The proposed method (CREST) consists of three modules. A block-energy e-process computes SkS_k over non-overlapping blocks of size BB and forms the chi-square tilted e-statistic. It mixes over candidate change starts with geometric weights, in the style of a Shiryaev–Roberts procedure, and raises an alarm via Ville’s inequality. Proposition 1 provides a post-alarm split-conformal coverage guarantee under conditional exchangeability. Experiments cover five real datasets with several underlying algorithms, the CPTC benchmark suite and synthetic data.

Critical strengths
  1. Point forecasts stay accurate while uncertainty estimates become outdated. This failure mode is common in finance and environmental monitoring, and it is worth studying in its own right;
  2. Correct core construction in the idealized model. Under i.i.d. N(0,1)N(0,1) residuals with known scale, the martingale construction in Lemma 1 and the Ville argument are correct.
Critical weaknesses
  1. Missing scale-adaptive baselines. Since the paper focuses specifically on scale-regime drift, a natural alternative is to use locally normalized conformity scores, such as ∣yt−y^t∣/σ^t|y_t-\hat y_t|/\hat\sigma_t, where σ^t\hat\sigma_t is estimated using GARCH, EWMA, or rolling volatility estimators. These methods can adapt to changing residual scales without explicit change-point detection. The scorecasting component of Conformal PID provides another relevant approach to modeling predictable changes in conformity scores. A comparison with such methods is important for establishing the incremental value of detection-triggered recalibration.
  2. Novelty is overstated. For example, Prior detection-triggered adaptation [1] already used conformal test martingales to trigger retraining. SAOCP [2] studies strongly adaptive online conformal prediction and provides guarantees over arbitrary time intervals, making it particularly relevant to nonstationary settings with abrupt distribution changes. NexCP [3] provides a general framework for weighted conformal prediction beyond exchangeability and is relevant to understanding coverage degradation under distribution drift;
  3. The null hypothesis is specified too loosely. Lemma 1 requires Sk∣Fk−1∼χB2S_k\mid\mathcal F_{k-1}\sim\chi^2_B exactly, which amounts to i.i.d. Gaussian residuals. The paper only asks for residuals that are “approximately white”. The asymptotic convergence of standardized block energies to a chi-square distribution does not by itself establish uniform-in-time false-alarm control. A stronger sequential argument is needed to justify the claimed anytime-validity when whitening parameters and local scales are estimated.
  4. Block aggregation does not intrinsically provide robustness to heavy-tailed outliers. Because ek=∏t∈k(1−2λ)1/2eλrt2e_k=\prod_{t\in k}(1-2\lambda)^{1/2}e^{\lambda r_t^2}, the block e-value is simply the product of per-step e-values. A single outlier therefore contributes the same factor whether or not the data are blocked;
  5. Gains over Rolling are negligible. The headline improvement (64.9% → 88.7%) is measured against Fixed calibration, but Rolling already reaches 87.0%. In Table 2, CREST improves coverage over Rolling by only +0.4, +1.7, +0.1, +0.1 and +0.1 pp, while its intervals are 0.5–8% wider. Furthermore, should highlight the results in Table 2 to make it easy to follow;
  6. No local coverage evaluation. The motivation is transient undercoverage after scale jumps. The main results, however, report only global average coverage, which is the metric that hides such failures. The paper should report coverage in windows after alarms or known change points, worst-window coverage, and recovery time. Conformal PID reported consecutive miscoverage. It would be better to report such metric or something new to show the motivation. Some possible options are (1) Post-change coverage; (2) Worst-window coverage; (3) Consecutive miscoverage;
  7. Statements in Introduction need references to support. It would be better to cite some relevant papers to support claims in Introduction, e.g. “Such changes are common in ….” in Line 39.

[1] Vovk, Vladimir, et al. "Retrain or not retrain: conformal test martingales for change-point detection." Conformal and Probabilistic Prediction and Applications. PMLR, 2021.

[2] Bhatnagar, Aadyot, et al. "Improved online conformal prediction via strongly adaptive online learning." International Conference on Machine Learning. PMLR, 2023.

[3] Barber, Rina Foygel, et al. "Conformal prediction beyond exchangeability." The Annals of Statistics 51.2 (2023): 816-845.

Questions for authors
  1. Which mixture weights are implemented: (1−ρ)ρt(1-\rho)\rho^t (Eq. 1) or (1−ρ)ρn−t(1-\rho)\rho^{n-t} (Fig. 2, Remark (i))? If the former, how does the detection delay scale with the location of the change within a segment?
  2. The experimental settings are not clear. How are these conformal baselines designed? And how to choose the hyper-parameters for?
  3. What exactly does “the local null holds within each segment” mean in Lemma 2? How is a false alarm defined when earlier alarms are true detections, and how are the blind window RR and delayed detections handled?
  4. Are CREST’s gains over Rolling statistically significant at the per-channel level? How does CREST compare to Rolling with a tuned WW on the coverage–width frontier and on the interval score?
  5. How does CREST compare to normalized / locally adaptive conformal scores, which handle scale drift without any detection step?
Reviewer_w2vWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CREST is an event-triggered conformal recalibration framework targeting residual-scale increases in time-series forecasting. It combines seasonal-AR whitening and local standardization with a block-energy detector, resetting the calibration buffer after alarms and allocating false-alarm budgets across restarts. The paper claims anytime-valid false-alarm control and post-reset conformal coverage under conditional exchangeability. Validation includes synthetic detection experiments and real-data comparisons of coverage and interval width across several datasets and forecasters.

Critical strengths

The paper addresses a relevant failure mode: residual uncertainty can change sufficiently that historical calibration scores become inappropriate. The explicit alarm-and-reset interface provides an actionable response without requiring a learned latent-state model or retraining the forecaster after every alarm. The mixture construction and summable restart allocation have a sound underlying principle when the block statistics satisfy the required conditional e-value property. Proposition 1 also distinguishes local post-reset coverage from coverage throughout an arbitrary nonstationary sequence. The main contribution is the integration of these components into a lightweight forecasting procedure.

The synthetic evaluation examines false alarms, detection probability, delay, scale-versus-level sensitivity, and component choices.

The results provide a credible proof of concept for recalibration triggered by residual-scale increases. The inclusion of heavy-tail stress tests and alternative forecasters is informative, even though their outcomes also expose important limitations.

Critical weaknesses

The argument in Section 4.2 and Appendix H requires a conditional e-value property such as E[ek∣Fk−1]≤1\mathbb{E}[e_k \mid \mathcal{F}_{k-1}] \leq 1. Approximately white, unit-variance residuals do not by themselves imply the chi-square block-energy distribution used for normalization. Estimating both the whitening model and the local scale introduces additional uncertainty that also needs to be accounted for.

The authors acknowledge that the chi-square null is approximate, but the implications for the stated anytime-valid guarantees remain insufficiently specified. In particular, the paper does not establish the required conditional moment bound or quantify the effect of parameter estimation for the implemented detector. The proposed robust variants are assessed empirically, while a conservative analytic normalization is left to future work (Appendix E). Since the restart argument depends on valid false-alarm control within each segment, the authors should clarify the assumptions under which the guarantee holds and explain how those assumptions relate to the implementation used in the experiments.

The main empirical evidence concerns downstream coverage and interval width, yet Table 2 shows that a generic conformal-martingale trigger nearly reproduces CREST’s results: coverage differs by at most 0.4 percentage points, with very similar widths. Rolling calibration already accounts for most of the improvement over Fixed; CREST adds only 0.1 percentage points of coverage on Exchange, ETTm2, and Weather. The paper itself attributes the similar CM performance to the trigger interface. This leaves the additional practical value of CREST’s particular scale-detection mechanism unclear.

I acknowledge the synthetic false-alarm, power, delay, and scale-specificity experiments, as well as the squared-residual CUSUM comparison. However, these do not establish a convincing comparative advantage across realistic residual distributions and repeated changes. The real-data alarm interpretation relies mainly on illustrative examples and the proposed entanglement diagnostic, which provides limited independent evidence that alarms identify meaningful scale-regime boundaries. Moreover, the conservative heavy-tail calibration used for CUSUM is difficult to interpret as a fair comparison unless CREST’s false-alarm control is demonstrated under the same conditions.

The downstream uncertainty-quantification claims also need stronger comparative support. Since the primary empirical outcomes are coverage and interval width, the relevant comparison extends to methods that maintain predictive uncertainty under nonstationarity, including approaches that adapt calibration without explicitly identifying regime boundaries. A more complete evaluation of the relevant approaches discussed in Table 1, under a common forecasting and calibration protocol, would help establish the added value of scale-specific triggering. Coverage and median width are useful diagnostics, but differences in empirical coverage make width comparisons insufficient to establish an overall efficiency advantage. A mean proper interval score, such as the Winkler score, is needed to assess interval quality; the coverage–width sweeps in Appendix F do not replace this evaluation.

Finally, coverage restoration does not transfer reliably across forecasters. On CarIndex, CREST achieves only 79.0%, 78.3%, and 77.8% coverage with TimeMixer, TimeMixer++, and WPMixer, respectively, against a 90% target (Tables 3 and 9). With PatchTST, it falls below Rolling, at 86.4% versus 87.5%. DLinear transfers more successfully, so the concern is the breadth of the claimed reliability. Compatibility with multiple forecasters is demonstrated, but reliable calibration across them is not.

Questions for authors
  • Which detector variant—raw chi-square tilt, Winsorized, Huberized, or self-normalized—produced the principal real-data results? What precise conditional distributional or exponential-moment assumptions justify its e-value property after estimating the whitening model and local scale? Please distinguish the guarantees proved for the idealized model from those established for the evaluated implementation.

  • Holding the recalibration policy fixed, what advantage does CREST provide over CM and a competitive scale-change detector under a common false-alarm criterion? Comparisons of power, detection delay, and localization across repeated changes and different changepoint locations would directly address this question. What independent evidence supports interpreting the real-data alarms as meaningful scale-regime boundaries?

  • Do the conclusions persist when evaluated using mean interval/Winkler score under a common protocol, including relevant adaptive methods such as ResCP and the already evaluated CPTC? Monitoring methods from Table 1, such as WATCH or e-CPD, could be assessed through the same reset interface where applicable. Uncertainty estimates would also help determine whether the small differences from Rolling and CM are meaningful.

  • What explains the substantial undercoverage with TimeMixer, TimeMixer++, and WPMixer? Section 5.1 states that the detector remains driven by AR-whitened residuals when the point forecaster changes. Does a mismatch between the monitored residual process and the deployed forecaster’s errors contribute to these results, or does the limitation arise primarily from recalibration within the detected segments? A targeted diagnosis would materially affect my assessment of the claimed forecaster generality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.