- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies split conformal prediction for time series, where the calibration and test data are consecutive and dependent. It looks at whether coverage is close to the target and whether an observed difference is due to miscalibration or normal variation. The authors use functional dependence measures (FDM) to derive coverage bounds and develop methods to estimate uncertainty in realized coverage. They also study long-memory data, where strong dependence can change the distribution of coverage errors, and propose a block-sampling method for inference. The methods are tested on simulated data and hourly electricity-demand forecasts, where the standard exchangeability-based test rejects nominal coverage, but the proposed block-sampling method does not.
Critical strengths
- The paper connects theory to a practical test. The paper connects the Bahadur representation (Theorem 2) to a central limit theorem for realized coverage (Theorem 3), then develops a block-based standard-error estimator (Proposition 4). Unlike concentration bounds such as Oliveira et al. (2024), this approach provides a reference distribution for testing an observed departure from nominal coverage. The estimated standard error is Under the additional conditions in Proposition 4, the authors also establish that the limiting variance is positive and the variance estimator is consistent. The paper also studies a Student-t correction for settings with fewer blocks (Equation 24). In Table 3, at , the 5% rejection rates for ARCH and stochastic-volatility data decrease from 10.54% and 12.92% with normal critical values to 4.75% and 6.19% with the corrected t reference. This suggests that the correction can improve test calibration in the settings studied.
- The experiments show why temporal dependence matters. In Study 2, at , the simulated standard deviations are 1.40 times (ARCH) and 1.97 times (stochastic volatility) the exchangeable standard error. Using the exchangeability-based standard error gives rejection rates of 16.04% and 32.10%, despite a nominal 5% test level. In Study 3, for and , block sampling gives a rejection rate of 5.65%, compared with 44.35% for the moving-block bootstrap and 43.77% for HAC-based Gaussian inference. In the strong-memory case, even at , the 2.5% and 97.5% quantiles of realized coverage are 84.29% and 94.04% for a nominal 90% level. The electricity-demand example is also useful: the forecasts and prediction intervals remain the same, and only the reference distribution changes, yet the test decision changes. These results illustrate the risk of ignoring temporal dependence in the settings studied.
Critical weaknesses
-
Theorems 5-7 focus on Gaussian linear residuals with absolute-error scores and coefficients that follow a specific power-law decay, . Theorem 7 also requires a more specific condition on these coefficients. Appendix D discusses possible extensions to non-Gaussian innovations but states that the extension is not established.
-
Appendix B shows that the indicator dependence measure is not summable () for the long-memory model when . Therefore, Theorem 3 and Proposition 4 from Section 3 do not apply to that model. The general results in Section 3 and the long-memory results in Section 4 cover different classes of processes.
-
The paper does not provide a unified result or practical guidance for deciding which assumptions apply to a given time series. In particular, the long-memory guarantees are not established for general non-Gaussian long-memory residual processes. This limits the range of settings covered by the theory.
-
The realized-coverage inference experiments mainly study rejection rates when nominal coverage holds. The paper does not systematically test cases where true coverage differs from the target, for example, 85% or 95% coverage when the nominal level is 90%. As a result, the experiments show how well the methods control false rejections in the settings studied, but do not establish how effectively the proposed test detects genuine miscalibration. This is particularly important under strong temporal dependence.
-
In Table 8, for and , the median estimated 95% interval width for block sampling is about 8.1 percentage points, while the reference interval width is about 9.75 percentage points. Thus, coverage uncertainty remains substantial even with around one million observations in each window. This may be an inherent effect of strong dependence, but a power study is needed to assess how often the test detects genuine departures from nominal coverage.
-
The same concern applies to the electricity-demand example, where the block-sampling reference interval is wide and the assumptions of the Gaussian linear model have not been established for the real data. Without a power study, it is difficult to interpret a non-rejection as evidence that coverage is close to nominal.
Questions for authors
-
Evaluate the power of the proposed test when actual coverage differs from the nominal 90%, such as when true coverage is 85% or 95%? This would help establish whether the test can detect genuine miscalibration while maintaining the intended false-rejection rate.
-
In the electricity-demand experiment, what evidence supports the Gaussian linear-process approximation for the corrected residuals? How should readers interpret the results when the full assumptions of the long-memory theory may not hold for the real data?
-
How should practitioners interpret the method when the estimated memory parameter is close to , given that the normalization result in Appendix C.2 is pointwise rather than uniform over sequences of memory parameters approaching this boundary?
Thank you for the detailed responses and additional analyses. The power study addresses my main concern by showing how detection changes with the size of the coverage departure and the sample size. The revised discussion also clarifies which method applies under short versus long memory, narrows the uncertainty claims, and explains the electricity-demand result as a dependence-aware test rather than evidence that coverage is nominal. Therefore, I raise my score.