- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces CREST (Conformal Recalibration via an Anytime-valid Scale-drift detector), an online framework for event-triggered conformal recalibration under changes in the scale of time-series residuals. Fixed calibration can severely undercover after variance increases, while continuously adaptive calibration can produce wider intervals.
Critical strengths
-
Principled Sequential Detection with Budgeted False-Alarm Control: Under the exact conditional null, the blockwise e-statistics, geometric mixture, and Ville bound provide a coherent detector, while the harmonic schedule preserves the false-alarm budget across an unbounded number of restarts. This guarantee concerns alarms and is distinct from conformal coverage. Table 10 supports the value of budgeting empirically: over five independent null segments, fixed-level restarts yield global FAR 0.093 versus 0.023 with harmonic levels. The construction offers an interpretable alarm stream and a computationally inexpensive control mechanism for recalibration.
-
Useful Coverage–Width Trade-Off and Measured Post-Alarm Recovery: The main results preserve the empirical strength identified in the initial review: CREST substantially improves over Fixed on CarIndex and Weather, with widths 2.03 and 2.33, much narrower than ACI's 3.16 and 5.18. Table 3 locates the benefit more precisely: in the synthetic delay window CREST and Rolling both cover 0.591, while recovery-window coverage is 0.853 versus 0.785. Table 16 additionally shows unchanged coverage to three decimals on Weather and ETTm2 for misspecified lags , with alarm-rate changes below 0.04 per 1,000 steps. This supports practical robustness to the tested lag choices, although it does not establish null calibration under arbitrary remaining dependence.
Critical weaknesses
-
Remaining Gap Between Idealized Alarm Guarantees and Raw Plug-In Deployment: The rebuttal resolves which detector generated the main results: the raw chi-square tilt. The reported Gaussian-null FAR 0.044 at nominal 0.05 is reassuring finite-horizon evidence, already present in the original manuscript; it supplies no evidence of significant Gaussian miscalibration, although it cannot establish exact uniform-in-time validity of the estimated procedure. The substantive remaining concern is heavy-tail sensitivity: Table 12 reports raw-tilt FAR 0.736 under and 0.988 under , versus 0.028 in that table's Gaussian setting. Appendix E explains the divergent positive moment-generating function of squared heavy-tail innovations. Robustification is a meaningful mitigation: Table 13 reports H13 FAR 0.000 and 0.008, with detection rates 1.000 and 0.988 and median delays 105 and 85 steps under and , respectively. W05 detects faster at a median 45 steps, but its FAR is 0.088; self-normalized FAR is 0.096–0.208. Appendix E explicitly leaves conservative analytic normalization for future work. The lag sweep addresses a particular whitening error, not general dependence or heavy tails. These synthetic failures do not establish the real datasets' false-alarm probabilities, but leave the calibration of the main raw detector in non-Gaussian deployments unresolved.
-
Coverage Benefits Are Local and Depend on the Forecast-Error Stream: Table 3 answers the original transition question: CREST improves recovery coverage from Rolling's 0.785 to 0.853, but does not improve the 0.591 delay-window coverage; settled coverage reaches 0.898. This is useful recovery evidence, with a remaining period of practical undercoverage rather than a violation of Proposition 1. The initial review's claim of seamless near-nominal transfer must also be corrected: the original manuscript already reported CarIndex coverage of 0.778–0.790 for WPMixer and the TimeMixer variants, unchanged in updated Tables 5/11. Appendix I now offers an explanation: their forecast errors can diverge from the AR residual stream monitored by the detector. This limits transferability without establishing new deterioration after revision. Aggregate comparisons provide context for the local benefit: Table 2's coverage gain over Rolling is 0.001 on Weather, Exchange, and ETTm2, and Table 17's mean normalized Winkler scores are 0.863 for CREST, 0.862 for Rolling, and 0.861 for CM. Table 4 likewise finds similar coverage and widths for locally normalized baselines. These results support competitive prediction and an additional alarm interface; the small aggregate differences do not negate the measured recovery gain. Marginal channel coverage is accepted as the stated scope, and the allocation controls alarm budgets rather than establishing joint predictive coverage.
Questions for authors
-
Robust Deployment Evidence: The main-variant clarification resolves which implementation was used. If a raw-versus-W05/H13 comparison or a robustification-threshold sweep already exists on Future, CarIndex, or Exchange, what does it show about downstream coverage, width, and alarm frequency? This follows the original question about sensitivity to robustification and would help assess whether the synthetic mitigation preserves the main empirical benefits. Real-data alarm frequency alone would not identify false-alarm probability without known change labels.
-
Detection After Many Restarts: The harmonic schedule gives , and Table 10 verifies budgeting over five independent null segments. Is there existing detection-delay or recovery-coverage evidence after many genuine regime changes? This is the empirical part of the original long-horizon question: the log threshold grows logarithmically, while the evidence threshold grows quadratically. The false-alarm guarantee is accepted under its stated assumptions; late-segment detection performance remains a secondary uncertainty, not an additional experiment required for acceptance.