- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2(−1)
- Presentation
- 2
- Soundness
- 2(−1)
Read review
Paper summary
This paper proposes a diffusion-based conformal prediction method that uses denoising errors as nonconformity scores to construct flexible prediction regions for multivariate time series. The authors establish finite-sample approximate marginal and conditional coverage guarantees under stationary -mixing assumptions.
Critical strengths
-
The proposed method is novel, well-motivated and supported by a theoretically grounded framework.
-
The experimental results are promising, particularly in demonstrating the potential of the proposed method to construct flexible prediction regions.
Critical weaknesses
-
The paper lacks sufficient validation and discussion of its theoretical assumptions, including the -mixing assumption for time-series data and the assumptions concerning diffusion model estimation (see Questions for Authors).
-
Several important methodological and experimental details are insufficiently explained, limiting clarity (see Questions for Authors).
-
The experimental evaluation is somewhat limited in terms of baseline selection and response dimensions , making it difficult to show broader advantages of the proposed method.
Questions for authors
-
Assumption 2 introduces an independent-block learning condition with a deterministic approximation radius . However, the authors may need to discuss whether this assumption holds for the diffusion model used in the experiments, or to provide an explicit convergence assumption for it. Although Remark 4.1 shows that the dependence penalty under geometric (\beta)-mixing vanishes as the training sample size increases, this does not guarantee that . Consequently, Corollary 4.2 does not establish convergence of the proposed nonconformity score to the oracle score, which still depends on . It would strengthen the theoretical analysis if the authors could provide sufficient conditions under which vanishes, along with an explicit convergence rate or a justification of Assumption 2 for their neural diffusion estimator.
-
Theorem 4.4 establishes conditional coverage over neighborhoods satisfying , rather than the standard conditional coverage guarantee conditioned on . While this distinction is acknowledged in the theoretical formulation, the resulting guarantee is substantially weaker, and the coverage error may become large for small . Furthermore, it looks like the theorem relies on neighborhood-specific calibration, which differs from the global calibration procedure in Algorithm 1. Additionally, it is unclear how these neighborhoods are selected in practice, and the experiments do not appear to evaluate the proposed localized conditional coverage. The authors should clarify the practical implications of Theorem 4.4 and provide empirical evidence supporting the conditional coverage claims.
-
The authors should provide further motivation and explanation for the use of antithetic sampling and common random numbers (CRN). Their roles in the proposed conformal prediction framework are not immediately clear from the current presentation. In particular, it would be helpful to clarify how these are beneficial.
-
It would be helpful to visually distinguish methods that achieve the target coverage level in Table 1. In its current form, it is difficult to identify which methods meet the target coverage, particularly when comparing prediction-region volumes.
-
The authors specify the deterministic predictors used by the baseline methods (LSTM and LOO bootstrap), but do not clearly identify the deterministic predictor used in the proposed residual-based method (Algorithm 2) for the main experiments in Table 1. Could the authors clarify which predictor is used and provide its implementation details?
-
Could the authors report the computational cost of each method, such as wall-clock training and inference times? If detailed runtime measurements are not currently available, approximate estimates would still be helpful for understanding the computational tradeoffs among the methods.
-
In Section 5, the authors state that "all volumes are measured using the common importance-sampling procedure." Could the authors confirm whether this procedure is applied consistently to all methods, including CONTRA and FlowCP? Additionally, could the authors explain the rationale for choosing importance sampling over uniform Monte Carlo or quasi-Monte Carlo approaches used in prior evaluations of these methods?
-
In Appendix B.4, the authors state that 16,384 samples are used for volume estimation when . However, to my understanding, the experiments only consider . Could the authors clarify whether any experiments with were conducted? Furthermore, I believe the experimental evaluation would be strengthened by considering higher response dimensions, as explored in existing studies [1,2,3]. Such experiments would provide additional evidence of the proposed method's effectiveness and scalability.
-
Another active research direction for constructing flexible prediction regions involves optimal transport [4,5]. While I am not necessarily requesting additional experiments with these methods, I believe the current baselines represent a relatively limited subset of existing approaches. A broader discussion of related methods, particularly those based on optimal transport, would help better contextualize the contributions and advantages of the proposed method.
-
In prior studies [2,3], Traffic and Solar have primarily been used as multivariate forecasting benchmarks , rather than datasets with explicitly demonstrated multimodal conditional distributions. Could the authors clarify whether these datasets exhibit multimodality or, how the datasets were processed to have multimodality Additionally, visualizations of the prediction regions for on these real-world datasets would be valuable for understanding the proposed regions and comparisons with the baseline methods.
-
Based on Algorithm 2, the authors used training set to train deterministic predictors and diffusion model. Can authors clarify how data is split for each baseline method?
[1] CONTRA: Conformal prediction region via normalizing flow transformation
[2] Flow-based Conformal Prediction for Multi-dimensional Time Series
[3] Conformal prediction for multi-dimensional time series by ellipsoidal sets
[4] Multivariate Conformal Prediction using Optimal Transport
[5] Optimal Transport-based Conformal Prediction
7 comments
- Higher response dimensions
We evaluated dy=2,4. Traffic and Solar contain only 15 and 9 locations, respectively, so eight-location subsets would overlap substantially across trials. We therefore limited this evaluation to smaller dimensions and acknowledge that higher-dimensional scalability remains untested. Appendix B.4 now lists the correct sampling budgets for the reported dimensions.
- Optimal transport methods
Thank you for the interesting direction! These are relevant alternatives, and we have included MultiTransport and OpTransport in the 90% comparison in Section 5 and the 95% comparison in Appendix B.6, highlighted in blue. Appendix B.1 describes their implementation sources, and Appendix B.5 now includes their training-time bounds.
- Multimodality and real-data visualizations
We have added a figure for 90% two-dimensional prediction regions on Traffic and Solar in Section 5, with the addition highlighted in blue. These visualizations illustrate region geometry and individual membership decisions. We have not established conditional multimodality in either dataset, and our preprocessing does not introduce it. The synthetic mixtures provide the controlled evaluation of multimodal geometry.
- Data splitting for each method
The revised experiments use dataset-specific chronological splits: 80% pre-calibration / 10% calibration / 10% test for Traffic, and 60% pre-calibration / 20% calibration / 20% test for Solar. For DisCo, 20% of the pre-calibration block is reserved for validation, giving approximately 64/16/10/10 for Traffic and 48/12/20/20 for Solar, after constructing and aligning histories.
Algorithm 2 fits its predictor and residual denoiser on the fitting block and computes calibration residuals using the frozen predictor on the separate calibration block. The main comparison aligns methods at target index 54 and uses the same test interval within each dataset. Baselines have method-specific internal fitting and calibration allocations, which we will document explicitly in Appendix B.1.
The revised 90% results are included in Section 5 and the 95% results in Appendix B.6, highlighted in blue. These show competitive volumes at nominal-or-higher mean coverage in several settings. However, in the main 95% comparison in Appendix B.6, Algorithm 1 has below-nominal mean coverage on Traffic, so we do not interpret its smaller volumes there as evidence of superior efficiency.
Thank you for the response. Some of my concerns and questions have addressed and answered. I still have some remaining questions and concerns:
- Localized VS pointwise conditional coverage
I see the authors clarified this point more prominent in the revised manuscript. But, the experiment results only report marginal average coverage and no empirical evidence for this claim. Please correct me if I overlook something.
- Deterministic predictor used by Algorithm 2
I need further clarification of deterministic predictor. In author's response, The real-data implementation uses ordinary least squares with an intercept and no ridge penalty. Then how does LSTM, LOO estimator used for FlowCP, MultiDimSPCI, and and CONTRA? Did they use different predictor from the proposed methods in Table 1?
- Higher response dimensions
I believe experiments with are too low dimension to claim argument about varying dimensions.
- Multimodality and real-data visualizations
Thank you for updating the figures. While figures are clear in visualizing the prediction sets, I am not convinced if real-data experiments support the proposed method's claim. Figure 3 on synthetic data show multimodality, but real data do not look like they have multimodality, which also align with other studies that use the same datasets.
Thank you for the sufficiently detailed review! We have clarified the learning assumption and localized coverage, moved Algorithm 2 into the main text, updated the coverage-aware tables, and added training-time estimates and optimal-transport comparisons. The 95% comparison in Appendix B.6 now includes all five Traffic subsets: Algorithm 1 undercovers on Traffic, whereas Algorithm 2 attains above-target mean coverage. We have also generated the requested real-data visualizations. Below, we distinguish completed revisions from remaining work. We address every point one by one below.
We agree that a vanishing mixing penalty does not imply that the denoiser approximation radius vanishes. We have updated Assumption 2 in Section 4.1, highlighted in blue, to separate the finite-sample learning condition from an explicit asymptotic assumption requiring rm,a→0. The revised remark gives a convergence rate conditional on an assumed rate for $r_{m,a} $, and clarifies the additional learning condition needed for oracle-score convergence. We do not establish this learning condition for the neural estimator used in our experiments.
We have clarified the scope and implementation of Theorem 4.4 in Section 4.3, highlighted in blue. It provides neighborhood-level coverage conditional on X∈A, where P(X∈A)≥γ, using a calibration quantile computed from observations within A. This is not pointwise coverage conditional on X=x. Neighborhoods can be defined from training data, subject to the theorem’s assumptions. Smaller neighborhoods provide finer localization at the cost of a larger finite-sample error bound. The current experiments evaluate global calibration. Evaluating neighborhood-level coverage requires a separate experiment using the localized calibration procedure.
These techniques control different aspects of Monte Carlo variability.
Neither technique automatically improves coverage. Our validity argument treats the bank as frozen and uses the stated temporal-dependence bounds. We have updated the real-data ablations in Section 5, highlighted in blue, with details in Appendix C.
We have updated Table 1 in Section 5, highlighted in blue, to report the 90% results, with entries below nominal mean coverage grayed out and excluded from the highlighted volume ranking. The corresponding 95% results, including all five Traffic location subsets, are reported in Appendix B.6, also highlighted in blue.
The real-data implementation uses ordinary least squares with an intercept and no ridge penalty. It’s fitted on a flattened 50-step histories from the proper training block. The predictor is then frozen, and an unconditional diffusion model is trained on its residuals. We have moved the general residual formulation into Section 3.3, highlighted in blue, and will add these implementation details to Appendix B.1 to distinguish the evaluated specialization from the general conditional formulation.
We have added approximate wall-clock training-time bounds, in hours, to Appendix B.5, highlighted in blue.
The implementation uses importance sampling to estimate real-data prediction-region volumes for DISCO, FlowCP, CONTRA, MultiTransport, OpTransport, and the global Mahalanobis baseline. MultiDimSPCI is an exception: its ellipsoidal-shell volume is computed analytically.
Our prediction regions are evaluated through membership tests and may contain disconnected components. Uniform Monte Carlo or Sobol QMC over a finite box estimates only the volume inside that box. A small box may omit components, while a large box may waste evaluations outside the region. For real-data experiments, we therefore use importance sampling with a Gaussian kernel-density proposal and a heavy-tailed Student-t component. This provides full support and concentrates evaluations in relevant areas, although numerical integration uncertainty remains. Appendix B.2 already describes randomized QMC volume estimation for the synthetic experiments, and Appendix B.4 describes importance sampling for the real-data experiments.