acceptodds
Under review as a conference paper at ICLR 2027

Conformal Coverage for Time Series: Validity and Inference

Abstract

Conformal prediction provides marginal coverage guarantees, yet practitioners may wonder if the observed coverage is truly abnormal or consistent with sampling variation. Inference for realized coverage has received comparatively little attention, especially for time series. We study split conformal prediction with adjacent calibration and test sets of temporally dependent data. Using the functional dependence measure, we derive non-asymptotic bounds on marginal coverage error without mixing assumptions, which can be difficult to verify and may fail even for simple short-memory models. We establish a Bahadur representation to derive, to our knowledge, the first central limit theorem for realized coverage of split conformal prediction under temporal dependence. A consistent block-based estimator of the standard error yields an asymptotically justified test. We further study long-memory time series, which remain understudied in conformal prediction. For Gaussian linear processes, we show how very strong temporal dependence can lead to a non-Gaussian limiting law of realized coverage and establish block-sampling inference with an estimated normalization. The resulting theory explains how temporal dependence changes coverage uncertainty.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Time series, Functional dependence, Split conformal prediction, Realized coverage, Long-range dependence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iT15OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies split conformal prediction for time series, where the calibration and test data are consecutive and dependent. It looks at whether coverage is close to the target and whether an observed difference is due to miscalibration or normal variation. The authors use functional dependence measures (FDM) to derive coverage bounds and develop methods to estimate uncertainty in realized coverage. They also study long-memory data, where strong dependence can change the distribution of coverage errors, and propose a block-sampling method for inference. The methods are tested on simulated data and hourly electricity-demand forecasts, where the standard exchangeability-based test rejects nominal coverage, but the proposed block-sampling method does not.

Critical strengths
  • The paper connects theory to a practical test. The paper connects the Bahadur representation (Theorem 2) to a central limit theorem for realized coverage (Theorem 3), then develops a block-based standard-error estimator (Proposition 4). Unlike concentration bounds such as Oliveira et al. (2024), this approach provides a reference distribution for testing an observed departure from nominal coverage. The estimated standard error is σ^cov2(1n+1m).\sqrt{\widehat{\sigma}_{\mathrm{cov}}^2\left(\frac{1}{n}+\frac{1}{m}\right)}. Under the additional conditions in Proposition 4, the authors also establish that the limiting variance is positive and the variance estimator is consistent. The paper also studies a Student-t correction for settings with fewer blocks (Equation 24). In Table 3, at n=29n=2^9, the 5% rejection rates for ARCH and stochastic-volatility data decrease from 10.54% and 12.92% with normal critical values to 4.75% and 6.19% with the corrected t reference. This suggests that the correction can improve test calibration in the settings studied.
  1. The experiments show why temporal dependence matters. In Study 2, at n=219n=2^{19}, the simulated standard deviations are 1.40 times (ARCH) and 1.97 times (stochastic volatility) the exchangeable standard error. Using the exchangeability-based standard error gives rejection rates of 16.04% and 32.10%, despite a nominal 5% test level. In Study 3, for β=0.6\beta=0.6 and n=220n=2^{20}, block sampling gives a rejection rate of 5.65%, compared with 44.35% for the moving-block bootstrap and 43.77% for HAC-based Gaussian inference. In the strong-memory case, even at n=220n=2^{20}, the 2.5% and 97.5% quantiles of realized coverage are 84.29% and 94.04% for a nominal 90% level. The electricity-demand example is also useful: the forecasts and prediction intervals remain the same, and only the reference distribution changes, yet the test decision changes. These results illustrate the risk of ignoring temporal dependence in the settings studied.
Critical weaknesses
  • Theorems 5-7 focus on Gaussian linear residuals with absolute-error scores and coefficients that follow a specific power-law decay, aj∼cj−βa_j\sim c j^{-\beta}. Theorem 7 also requires a more specific condition on these coefficients. Appendix D discusses possible extensions to non-Gaussian innovations but states that the extension is not established.

  • Appendix B shows that the indicator dependence measure is not summable (Θ=∞\Theta=\infty) for the long-memory model when 1/2<β<11/2<\beta<1. Therefore, Theorem 3 and Proposition 4 from Section 3 do not apply to that model. The general results in Section 3 and the long-memory results in Section 4 cover different classes of processes.

  • The paper does not provide a unified result or practical guidance for deciding which assumptions apply to a given time series. In particular, the long-memory guarantees are not established for general non-Gaussian long-memory residual processes. This limits the range of settings covered by the theory.

  • The realized-coverage inference experiments mainly study rejection rates when nominal coverage holds. The paper does not systematically test cases where true coverage differs from the target, for example, 85% or 95% coverage when the nominal level is 90%. As a result, the experiments show how well the methods control false rejections in the settings studied, but do not establish how effectively the proposed test detects genuine miscalibration. This is particularly important under strong temporal dependence.

  • In Table 8, for β=0.6\beta=0.6 and n=m=220n=m=2^{20}, the median estimated 95% interval width for block sampling is about 8.1 percentage points, while the reference interval width is about 9.75 percentage points. Thus, coverage uncertainty remains substantial even with around one million observations in each window. This may be an inherent effect of strong dependence, but a power study is needed to assess how often the test detects genuine departures from nominal coverage.

  • The same concern applies to the electricity-demand example, where the block-sampling reference interval is wide and the assumptions of the Gaussian linear model have not been established for the real data. Without a power study, it is difficult to interpret a non-rejection as evidence that coverage is close to nominal.

Questions for authors
  • Evaluate the power of the proposed test when actual coverage differs from the nominal 90%, such as when true coverage is 85% or 95%? This would help establish whether the test can detect genuine miscalibration while maintaining the intended false-rejection rate.

  • In the electricity-demand experiment, what evidence supports the Gaussian linear-process approximation for the corrected residuals? How should readers interpret the results when the full assumptions of the long-memory theory may not hold for the real data?

  • How should practitioners interpret the method when the estimated memory parameter is close to β=3/4\beta=3/4, given that the normalization result in Appendix C.2 is pointwise rather than uniform over sequences of memory parameters approaching this boundary?

1 comment

Thank you for the detailed responses and additional analyses. The power study addresses my main concern by showing how detection changes with the size of the coverage departure and the sample size. The revised discussion also clarifies which method applies under short versus long memory, narrows the uncertainty claims, and explains the electricity-demand result as a dependence-aware test rather than evidence that coverage is nominal. Therefore, I raise my score.

Reviewer_rYmiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the theoretical properties of split conformal prediction under temporally dependent data, including non-asymptotic bounds and asymptotic distributions. The main assumption used is the functional dependence measure (FDM) from Wu (2005), which is weaker than β\beta-mixing assumptions in previous work.

Critical strengths
  1. Use FDM to analyze the coverage bound and asymptotic distribution of split conformal prediction under dependence.

  2. Build the first central limit theorem for the realized coverage of split conformal prediction under temporal dependence.

Critical weaknesses
  1. This paper only provides artificial examples that satisfy the assumptions, e.g., Lines 162-175. It lacks examples of practical prediction models. Since conformal prediction is a model-free inference tool, this may limit the application of the theories in this paper. The author should provide examples that satisfy Assumption 1 for commonly used models, at least for linear regression.

  2. Also, the Gaussian linear residual in Assumption 2 is also very strong, and the residual of practical pretrained models hardly satisfy this decomposition.

  3. As a very closed work, this paper lacks detailed discussion with Barber & Pananjady (2026), e.g., the relation between FDM and the switch coefficient in Barber & Pananjady (2026), and the coverage error bound.

Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rYmi, Part 1

R2-W1

We thank the reviewer for this suggestion. We would like to clarify that the assumptions in Section 3 are mild. Section 3 does not assume any model for the data or for the predictor. Its main requirement, Assumption 1, is that the dependence between the scores decays fast enough over time (Θ<∞\Theta < \infty, i.e. short memory). Such conditions are easily verifiable and is the standard setting in time series analysis and is common in real data (Wu, 2005, cited in the paper). And they hold for commonly used time series models such as ARMA, GARCH and nonlinear autoregressions (Shao & Wu, 2007, see below). The most restrictive part is short memory. And that was the reason why we considered long memory in Section 4.

That being said, we agree with the reviewer that an example with a practical prediction model is helpful. We add the following example in the main text, after Assumption 1.

Linear regression. Let μ^(x)=x⊤θ^\hat\mu(x)=x^\top\hat\theta, where θ^\hat\theta is fitted on separate training data by any method (least squares, ridge, lasso); the linear model does not need to be correct. A linear predictor is Lipschitz, so the scores inherit the decay of dependence from the data (Lemma 8 in Appendix A.1). Specifically, if the FDM of the data Zt=(Xt,Yt)Z_t=(X_t,Y_t) satisfies δ2Z(k)=O(ρk)\delta_2^Z(k)=O(\rho^k) for some 0<ρ<10 < \rho < 1 and the score has a bounded density, then θk=O(ρk/3)\theta_k=O(\rho^{k/3}), so Θ<∞\Theta < \infty and Assumption 1 holds. The same argument applies to other Lipschitz predictors, such as neural networks with Lipschitz activations. Note that these conditions are mild. The geometric rate holds when the data follow the standard models mentioned above (e.g. ARMA or GARCH), where ρ\rho is determined by the model coefficients and can be checked on real data from a fitted model. The score density is bounded whenever YtY_t has a bounded conditional density given XtX_t. Moreover, no particular model is required, only short memory.

The example in Lines 162–175 has a different purpose and is not meant as a practical model. It is a simple autoregression that is not even α\alpha-mixing, so guarantees based on mixing conditions (e.g. Oliveira et al., 2024, cited in the paper) do not cover it, yet Assumption 1 holds. It shows that our conditions apply beyond mixing. The assumptions are also verified for the ARCH and stochastic volatility models of Study 2 (Appendix E.2).

Shao & Wu (2007), “Asymptotic spectral theory for nonlinear time series”, Annals of Statistics.

Zhou & Wu (2009), “Local linear quantile estimation for nonstationary time series”, Annals of Statistics.

AuthorsOpenReview ↗

Response to Reviewer rYmi, Part 2

R2-W2

We thank the reviewer for this comment. We would like to clarify why Section 4 uses a model, and why the Gaussian linear model (Assumption 3) is less restrictive than it may appear.

Why Section 4 needs a model. Sections 3 and 4 serve different purposes. Section 3 assumes no model but requires short memory, under which realized coverage is asymptotically normal at rate n−1/2n^{-1/2} regardless of the process type. Section 4 treats long memory, which Section 3 cannot handle. There the n−1/2n^{-1/2} rate no longer always holds: when the memory is strong enough (β<3/4\beta < 3/4, smaller β\beta means longer memory), the coverage error shrinks at the slower rate n1−2βn^{1-2\beta} and the limit is no longer Gaussian (Theorem 6). Since the rate and the limit depend on how long the memory is, we have to model the memory. This is common practice in the long-memory literature (Taqqu, 1975; Dehling & Taqqu, 1989; Wu, 2003, all cited in the paper), and the Gaussian linear process is one such choice.

Gaussianity is not needed. We have updated the discussion in Appendix D by a rigorous proposition and proof. For iid symmetric innovations with a smooth density and a finite fourth moment, Theorems 5-7 hold for every fixed 1/2<β<11/2 < \beta < 1, including the boundary β=3/4\beta=3/4, with the block-sampling procedure unchanged. Two details change: −f′(q)-f'(q) replaces qf(q)/σW2qf(q)/\sigma_W^2 in Theorem 6(ii)-(iii), and Theorem 7 needs f′(q)≠0f'(q)\ne0 when β≤3/4\beta\le3/4. Apart from the updated theoretical results, we have added an extended simulation with heavy-tail innovations like t5t_5. in the design of Study 3, the 5%-level rejection rate at n=216n=2^{16} is 5.55% (β=0.9\beta=0.9) and 6.12% (β=0.7\beta=0.7), compared to 5.48% and 5.79% with Gaussian innovations.

The linear form is representative. It is motivated by Wold's decomposition: every centered, purely nondeterministic stationary process has a linear representation with uncorrelated innovations. Therefore, the linear process in Section 4 is quite general and representative.

For the residuals in our real-data analysis, the standard deviation of block coverage errors, measured directly without the model, decays with the block length at log-log slope −0.27-0.27, far from the −0.5-0.5 of short memory. So the n−1/2n^{-1/2} rate indeed fails for a practical pretrained model. Meanwhile, based on the linear process we can estimate β^=0.63\hat\beta=0.63, which predicts the rate n1−2β^=n−0.26n^{1-2\hat\beta}=n^{-0.26}, quite close to -0.27. This close agreement supports the linear process as a good model for these data.

AuthorsOpenReview ↗

Response to Reviewer rYmi, Part 3

R2-W3

We thank the reviewer for this valuable question. We agree that it is very useful to discuss Barber & Pananjady more thoroughly and state the difference more clearly.

Marginal coverage error vs realized coverage inference. The two papers have different targets. The results of Barber & Pananjady (2026) bound the marginal coverage error, i.e. how far the probability of covering a single test point is from 1−α1-\alpha. Our main results (Theorems 2–7) concern the distribution of realized coverage and the resulting tests, which they do not study. The overlap is our Theorem 1, which also bounds the marginal coverage error, but with a different dependence measure. The following paragraph that we have added to the revised draft might be helpful in clarifying this:

Realized coverage itself is a random variable, and its expectation is the average marginal coverage over the test set. A marginal guarantee only controls the mean coverage, but not how far a single observed value may deviate from 1−α1-\alpha. Judging whether an observed departure is truly significant therefore requires the distribution of realized coverage. It has been studied for exchangeable data (Gazin 2024 cited in the paper). But fore temporal dependence, only concentration bounds are available (Oliveira et al. 2024 cited in the paper). Our results provide this distribution under temporal dependence.

Switch coefficient vs FDM. The switch coefficient Ψk,τ\Psi_{k,\tau} of Barber & Pananjady (2026) is the total variation distance between two rearrangements of the score sequence. Beyond exchangeable data, their paper controls it through β\beta-mixing: for stationary β\beta-mixing sequences it is at most 2β(τ)2\beta(\tau). However, β\beta-mixing is a strong condition. It may fail even for elementary short-memory models with discrete innovations (Andrews, 1984), and even for a Gaussian VAR(1) process the β\beta-mixing coefficient at a fixed lag tends to one as the dimension grows (Han & Wu, 2023; both cited in the paper). Our FDM measures how much replacing one past innovation changes later scores, and it does not rely on mixing. A summable FDM does not imply a small switch coefficient: the process in Lines 162–175 has Θ<∞\Theta < \infty but Ψk,τ=1\Psi_{k,\tau}=1 for all 1≤k≤n−τ1\le k\le n-\tau, because each score is a deterministic function of the next one.

Marginal coverage error bound. Their bound on the coverage loss is min⁡ττ/(n+1)+Ψˉτ\min_\tau\\{\tau/(n+1)+\bar\Psi_\tau\\}, which is of order log⁡n/n\log n/n under geometric β\beta-mixing. Our contribution in this part is applicability beyond mixing, at the cost of a slower rate: n−1/2n^{-1/2} (Theorem 1(ii)) or n−1/3n^{-1/3} (Theorem 1(i)). For example, the process in Lines 162–175 is not mixing: their lower bound on coverage equals 1/(n+1)−α1/(n+1)-\alpha, negative for n≥1/αn\ge1/\alpha, while our bound (Theorem 1(i)) tends to zero.

I thank the authors for their response, which has addressed my concerns. I have increased my rating.

Reviewer_chA3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies theoretical guarantees for marginal coverage and develops statistical inference methods for realized coverage under temporal dependence.

Critical strengths

The paper addresses a well-motivated and clearly formulated problem, supported by detailed theoretical analysis and comprehensive numerical experiments that supports the theoretical findings.

Critical weaknesses

Some methodological choices require further justification, and the paper would benefit from a more extensive discussion of practical limitations, as most numerical studies rely on synthetic data generated under the theoretical assumptions.

Questions for authors
  1. Realized coverage appears to resemble empirical coverage evaluation on a test set, as commonly reported in time-series conformal prediction studies. Therefore, I believe it would be useful to discuss the significance of studying realized coverage beyond conventional marginal coverage guarantees when introducing its definition.

  2. Study 1 provides useful empirical insights that support the theoretical analysis. However, I believe the results also imply that a relatively large calibration set may be needed to achieve coverage consistently close to the nominal level, even for a synthetic process specifically constructed to satisfy the theoretical assumptions. It would be helpful for the authors to discuss this practical limitation.

  3. In Studies 2 and 3, the authors use block lengths of ⌊n2/3⌋\lfloor n^{2/3}\rfloor. Could authors specify how the block length ll is chosen? It would be useful to investigate the sensitivity of the empirical sensitivity results to this choice or rationale in choosing ⌊n2/3⌋\lfloor n^{2/3}\rfloor.

  4. The real-data analysis provides a useful illustration of the proposed inference framework. However, it relies solely on the existing day-ahead electricity-demand forecasts provided by the dataset, with an additional calendar-based correction. The details of those predictions were not specified. Since different forecasting models may produce nonconformity scores with substantially different temporal dependence structures, it would be helpful to consider additional forecasting methods for real data analysis.

5 comments
AuthorsOpenReview ↗

Response to Reviewer chA3, Part 1

R1-W1

We thank the reviewer for these suggestions. We answer the two points in turn.

Methodological choices and limitation. The method involves the following choices.

  • What to test. We study realized coverage instead of marginal coverage; see Q1.
  • Two procedures. Under short memory, realized coverage is asymptotically normal at rate n−1/2n^{-1/2} regardless of the process type, so we choose to estimate the unknown variance (Section 3). Under long memory, the rate and the limit depend on how long the memory is, and asymptotic Gaussianity may not even hold. We model memory with a linear process, and estimate the whole distribution by block sampling (Section 4).
  • Why block sampling. Standard alternatives underestimate the uncertainty under long memory. In simulation, the 5% test substantially overrejects with the moving-block bootstrap or HAC inference, and approaches the nominal level with block sampling. See the comparison in Table 2.
  • Block length. We use n2/3n^{2/3}. The justification and a sensitivity study are given in Q3.

If there are other choices that the reviewer would like us to justify, we would be glad to do so. One more practical limitation: for temporal dependence, a larger calibration set is needed than for exchangeable data. We have added this discussion to the paper; see Q2.

Real data vs theoretical assumptions. We would like to clarify that the linear process in Section 4 is an appropriate model for our real data. First we would like to update that Gaussianity is not required. We have updated the discussion in Appendix D by a rigorous proposition and proof. For iid symmetric innovations with a smooth density and a finite fourth moment, Theorems 5-7 hold for every fixed 1/2<β<11/2 < \beta < 1, including the boundary β=3/4\beta=3/4, with the block-sampling procedure unchanged. Two details change: −f′(q)-f'(q) replaces qf(q)/σW2qf(q)/\sigma_W^2 in Theorem 6(ii)-(iii), and Theorem 7 needs f′(q)≠0f'(q)\ne0 when β≤3/4\beta\le3/4. Apart from the updated theoretical results, we have added an extended simulation with heavy-tail innovations like t5t_5. In the design of Study 3, the 5%-level rejection rate at n=216n=2^{16} is 5.55% (β=0.9\beta=0.9) and 6.12% (β=0.7\beta=0.7), compared to 5.48% and 5.79% with Gaussian innovations. Second, the linear form is representative. It is motivated by Wold's decomposition: every centered, purely nondeterministic stationary process has a linear representation with uncorrelated innovations. Therefore, the linear process in Section 4 is quite general. For the residuals in our real-data analysis, the standard deviation of block coverage errors, measured directly without the model, decays with the block length at log-log slope −0.27-0.27, far from the −0.5-0.5 of short memory. So the n−1/2n^{-1/2} rate does not appear to hold for a practical pretrained model. Meanwhile, based on the linear process we can estimate β^=0.63\hat\beta=0.63, which predicts the rate n1−2β^=n−0.26n^{1-2\hat\beta}=n^{-0.26}, quite close to −0.27-0.27. This close agreement supports the linear process as a good model for these data. We also repeat the real-data analysis with three additional forecasting models (Q4).

R1-Q1

We appreciate the reviewer for raising this good question. Indeed, we agree that it would be useful to discuss the significance of studying realized coverage, beyond of conventional marginal coverage. We add the following discussion in the main text, after introducing the definition of realized coverage.

Realized coverage itself is a random variable, and its expectation is the average marginal coverage over the test set. A marginal guarantee only controls the mean coverage, but not how far a single observed value may deviate from 1−α1-\alpha. Judging whether an observed departure is truly significant therefore requires the distribution of realized coverage. It has been studied for exchangeable data (Gazin 2024 cited in the paper). But fore temporal dependence, only concentration bounds are available (Oliveira et al. 2024 cited in the paper). Our results provide this distribution under temporal dependence.

R1-Q2

We thank the reviewer for pointing this out. Indeed, we agree that this practical limitation should be discussed. We add the following discussion in the main text, after the results of Study 1.

For exchangeable data, split conformal prediction guarantees coverage of at least 1−α1-\alpha regardless of nn. Under temporal dependence, this guarantee holds only approximately: the coverage error is bounded as in Theorem 1, which vanishes as nn grows. The bound increases with Θ\Theta, so stronger dependence requires a larger calibration set.

AuthorsOpenReview ↗

Response to Reviewer chA3, Part 2

R1-Q3

The choice n2/3n^{2/3} has a precedent in the block bootstrap literature for long-memory time series. Under short memory, the block length that is optimal for variance estimation is of order n1/3n^{1/3} (Hall et al., 1995, see below; Flegal & Jones, 2010, cited in the paper), and long memory requires longer blocks: Zhai et al. (2026, Theorem 2 and Remark 4) derive the order that minimizes their error bound, which lies between n2/3n^{2/3} and nn, and adopt the simple choice n2/3n^{2/3}. Our theory allows any exponent in (0,1)(0,1) (Theorem 7), so we follow this choice.

We have also done the sensitivity study below, which supports the choice of n2/3n^{2/3} in our setting.

Study 2. We vary the block length as ℓ=⌊c,n2/3⌋\ell=\lfloor c\\,n^{2/3}\rfloor with c∈0.5,1,2c\in\\{0.5,1,2\\}; the paper uses c=1c=1. Rejection rates (%) at the 5% level, with the corrected-tt reference:

ARCHSV
nnc=0.5c=0.5c=1c=1c=2c=2c=0.5c=0.5c=1c=1c=2c=2
292^{9}5.04.84.98.06.25.2
2112^{11}5.25.45.25.85.45.1
2132^{13}5.25.35.25.75.45.2
2152^{15}5.25.14.85.04.94.7
2172^{17}5.25.35.45.04.84.8
2192^{19}5.15.05.15.25.25.1

The rejection rate is between 4.7% and 5.8% in 34 of the 36 cases; the two exceptions are SV at the smallest sample size. With normal critical values, the rejection rate is more sensitive to cc when there are few blocks; the corrected-tt reference accounts for this.

Study 3. We vary the block length as b=⌊nζ⌋b=\lfloor n^{\zeta}\rfloor with ζ∈1/2,3/5,2/3,3/4\zeta\in\\{1/2,3/5,2/3,3/4\\}; the paper uses ζ=2/3\zeta=2/3. Rejection rates (%) at the 5% level:

n=216n=2^{16}n=220n=2^{20}
ζ\zeta1/21/23/53/52/32/33/43/41/21/23/53/52/32/33/43/4
β=0.9\beta=0.97.75.75.56.36.45.45.25.6
β=0.75\beta=0.759.36.45.97.16.44.44.35.0
β=0.7\beta=0.79.06.25.86.96.14.74.65.3
β=0.6\beta=0.611.78.57.98.87.25.75.76.5

The exponent 2/32/3 gives the smallest rejection rate in all 20 designs (four values of β\beta, five sample sizes). The exponents 3/53/5 and 3/43/4 are close to it, 1/21/2 rejects more often. And again, the differences shrink as nn grows.

Hall, Horowitz & Jing (1995), “On blocking rules for the bootstrap with dependent data”

AuthorsOpenReview ↗

Response to Reviewer chA3, Part 3

R1-Q4

We thank the reviewer for these suggestions. We would like to clarify that the one-day forecast in the dataset is made by ISO New England and reported to the US-EIA daily. According to public description, it combines several models driven by weather forecasts and calendar information. The model itself is not public, so we treat it as a fixed black-box predictor, which is all that our theory requires.

That being said, we do agree with the reviewer that it may be beneficial to try some other models and see if the results change a lot. We add three of them:

  • Seasonal naive: the demand at the same hour one week earlier, y^t=yt−168\hat y_t=y_{t-168}.
  • Linear regression: least squares on hour-of-day by weekday/weekend indicators, two annual sine/cosine pairs, and yt−48y_{t-48}, yt−168y_{t-168}, yt−336y_{t-336}.
  • Gradient boosting: boosted regression trees on the same calendar information and lags (300 trees, learning rate 0.05, no tuning).
ForecastCalibration RMSE (MW)Realized coverage (%)Block-sampling 95% range (%)pp
Corrected operator forecast (paper)33791.2[84.4, 93.5]0.56
Seasonal naive192186.9[83.3, 92.3]0.24
Linear regression133584.5[84.4, 92.1]0.05–0.06
Gradient boosting149987.6[84.1, 92.2]0.31

On the side note: under exchangeability, the 95% range is [89.1%, 90.9%] for every forecast. In contrast, the block-sampling range, which accounts for temporal dependence, is four to five times wider and contains the realized coverage in every case. The additional forecast methods are not as accurate as the ISO forecast in terms of RMSE. But the dependence does change across different methods. We shall add these methods, the comparison, and these discussion into the real data analysis appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You – General Summary of Rebuttal

We thank the reviewers for their careful and constructive comments. We have uploaded a revised manuscript in which all new text is marked in blue. This comment lists the changes to the paper and the main points of our responses.

Changes to the paper

  • Non-Gaussian long memory (Section 4, Appendix D, Appendix E.4). Appendix D (Propositions 12–13) now extends Theorems 5–7, with proofs, from Gaussian innovations to iid symmetric innovations with a smooth density and a finite fourth moment, for every fixed 1/2<β<11/2<\beta<1; the block sampling procedure is unchanged. A new simulation with t5t_5 innovations supports it (Table 10). [iT15: W1–W2; rYmi: W2; chA3: W1]
  • Power study (Appendix E.5, Tables 11–12). We added a power study of both tests, with true coverage between 85% and 95% against the nominal 90%, for the short-memory models and for long-memory processes with β\beta from 0.9 to 0.6. [iT15: W4, Q1]
  • Real data (Section 5, Appendix F). The electricity example is now presented in Section 5, with text moved from Appendix F. We describe the forecast used (Appendix F.1), repeat the analysis with three additional forecasting models, and explain why their realized coverage differs (Appendix F.4, Table 13). [chA3: Q4]
  • Practical guidance (Appendix F.2). We added a rule of thumb for choosing between the two tests: if the standard deviation of block coverage errors decays with the block length at log-log slope near −1/2-1/2, use the test of Section 3; otherwise use block sampling. [iT15: W3]
  • Example for Assumption 1 (Section 3, Appendix A.2). For linear regression, and for any Lipschitz predictor, Assumption 1 follows from geometrically decaying dependence of the data and a bounded score density. [rYmi: W1]
  • Realized coverage and calibration size (Sections 2 and 5). We added a discussion of why realized coverage is studied beyond marginal coverage, and of the calibration-set size under temporal dependence. [chA3: Q1, Q2; rYmi: W3]
  • Minor. The comparison with the moving-block bootstrap and HAC inference now also covers β=0.7\beta=0.7 (Appendix E.3.2), and some tables and figures were rearranged or simplified.

Main clarifications

  • Sections 3 and 4 are separate by design. Under short memory, realized coverage is asymptotically normal at rate n−1/2n^{-1/2} regardless of the process type, and Section 3 assumes no model. Under long memory, the rate and the limit depend on how long the memory is (Theorem 6), so the memory has to be modeled, as is common in the long-memory literature. [iT15: W1–W2; rYmi: W2]
  • The linear model of Section 4 is less restrictive than it may appear. Gaussianity is not needed, and the linear form is motivated by Wold's decomposition. For the electricity data, the observed scaling of coverage fluctuations (log-log slope −0.27-0.27) agrees with the prediction of the fitted linear model (−0.26-0.26). [rYmi: W2; iT15: Q2; chA3: W1]
  • The assumptions of Section 3 are mild. Section 3 imposes no model on the data or on the predictor, and its conditions hold for commonly used models such as ARMA, GARCH and nonlinear autoregressions. The example following Assumption 1 shows that the conditions apply beyond mixing. [rYmi: W1]
  • Relation to Barber & Pananjady (2026). Their results bound the marginal coverage error of a single test point. Our main results (Theorems 2–7) concern the distribution of realized coverage and the resulting tests, which they do not study. The overlap is Theorem 1, where our contribution is applicability beyond mixing, at the cost of a slower rate. [rYmi: W3]
  • Block length. The choice n2/3n^{2/3} follows the block bootstrap literature for long-memory series. A sensitivity study for Studies 2 and 3, reported in our response, supports it. [chA3: Q3]
  • Strong long memory. At β=0.6\beta=0.6 the coverage error shrinks only at rate n−0.2n^{-0.2}, so a wide reference range is an inherent effect of the dependence. Block sampling recovers most of the actual spread of realized coverage, while the moving-block bootstrap and HAC inference give ranges that are far too narrow. [iT15: W5]
  • Electricity example. The example asks whether an observed departure from 90% is unusual, and shows that the answer depends on whether temporal dependence is taken into account: for all four forecasts, the observed coverage lies outside the exchangeable 95% reference range and inside the block sampling range (Table 3; Table 13). The example does not rely on interpreting a non-rejection as evidence that coverage is close to nominal. [iT15: W6; chA3: Q4]
  • The boundary β=3/4\beta=3/4. The theory is stated for each fixed β\beta, as is common in the literature. The procedure needs no regime selection, and in simulations at and just below the boundary the size is close to 5% for n≥216n\ge2^{16}. [iT15: Q3]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.