acceptodds
Under review as a conference paper at ICLR 2027

Conformal Prediction for Multivariate Time Series Forecasting Under Heterogeneous Distribution Shifts

Abstract

provides intervals that cover true values with high probability, supporting reliable decision-making under uncertainty. Existing sequential conformal methods typically target marginal coverage across all components and rely on the assumption that calibration and future forecast data have a joint distribution invariant under reordering. However, distribution shifts are common in time series, violating this invariance assumption, and can furthermore be heterogeneous across components in multivariate settings, leading to component-specific coverage failures. In this paper, we propose (BaCR), which combines two complementary mechanisms: that periodically adapts the quantile estimates used to construct prediction intervals, and that provide immediate online interval correction within the finite range supported by these quantile estimates and report a signal when additional expansion is needed. Theoretically, we derive an exact decomposition of component-wise empirical coverage error that characterizes tracker correction and the boundary limitations from the conditional quantile estimates. Experiments on a synthetic benchmark and four real-world datasets show that BaCR substantially reduces component-wise undercoverage, especially on large-shift components, and produces more efficient intervals compared to baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Uncertainty Quantification, Multi-variate time-series forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_P6PAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses heterogeneous distribution shifts across components in conformal prediction for multivariate time-series forecasting. It proposes Boundary-aware Conformal Calibration (BaCR), which combines a unified recalibrator with component-wise trackers. Its main contributions include the calibration framework, theoretical analysis, and extensive experiments on datasets of varying scales using coverage and interval-quality metrics. Overall, I consider this a strong paper with clear methodological contributions and solid empirical validation.

Critical strengths
  1. The challenges are clear, and the methods to address the challenges are novel and easy but simple to understand and follow. Meanwhile, the experiments are solid. They cover both simulation and real-world datasets, and the evaluation metrics are not merely coverage but also the Winkler score. There are performance evaluations, ablation studies, and parameter sensitivity analysis to fully verify the effectiveness of the proposed method.
Critical weaknesses
  1. The paper mentions the heterogeneity of different components, but there are no tests on this part, and also I am wondering if there is heterogeneity in the dataset used in the paper. Also, the evaluation metrics include coverage and the Winkler score, but I think the interval width is more important to see if the method really improves the coverage instead of making a sacrifice to efficiency of the interval width to get better coverage. A lower Winkler score does not necessarily mean narrower intervals.
Questions for authors
  1. What is the performance when there is no heterogeneity in the component signal? Also, when a less severe or more severe distribution shift occurs, what will the performance of the proposed method be?
  2. Could you provide the interval-width measurements across different baselines and the proposed method?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We sincerely appreciate your efforts to reviewing our work and thanks for pointing out these issues. We have addressed each of your concerns below and will add all results into the revised manuscript. We believe that the revised version has been significantly improved as a result of your valuable suggestions. Please let us know if there are any remaining issues where further improvement is needed.

W1 (Datasets' heterogeneity):

Heterogeneity is common in spatiotemporal data. Prior work treats heterogeneity across locations as a central property of traffic data, including LA, BAY and SD [1,2], and the source paper of the AIR [3] shows that air quality differs across stations.

We also measured this on our data: Components shift differently between calibration and test, some towards under-coverage and others towards over-coverage. For each component, we built a 90% SCP interval from its own calibration residuals and measured its coverage on the test set. Without shift, every component would cover about 90%. The table shows the distribution of these component-level coverage rates: the gap between the 5th and 95th percentiles is 7–12 percentage points (pp).

Component coverage (%)AIRLABAYSD
5th percentile87.580.885.784.9
25th percentile91.485.488.687.2
75th percentile95.690.591.190.1
95th percentile98.192.893.492.2

[1]Xu, Jiyuan, et al. "Cpiri: Channel permutation-invariant relational interaction for multivariate time series forecasting." ICLR 2026; [2]Dong, Zheng, et al. "Heterogeneity-informed meta-parameter learning for spatiotemporal time series forecasting." KDD 2024; [3]Zheng, Yu, et al. "Forecasting fine-grained air quality based on big data." KDD 2015.

Q1 (Heterogeneity & Shift severity):

Both experiments use the paper's synthetic settings in Section 6.2. Bold marks the lowest width and Winkler among methods with |ΔCov| ≤ 1 pp.

Heterogeneity. In None, components share AR coefficients, scale, volatility-cycle phase, regime change points/levels and tail parameters. Static adds component-specific fixed scales AiA_i, so components differ in level but shift identically. Hetero is the original paper setting.

BaCR keeps coverage error within the band in all settings, with the lowest width and Winkler score among methods with |ΔCov| ≤ 1 pp.

SettingMetricOracleSCPNexCPHopCPTSPCICoRelACIPIDBaCR
NoneΔCov(%)+0.0-4.1+0.4-2.2-4.1-0.2-0.1-2.9-0.4
Width1.851.982.281.831.951.992.232.091.95
Winkler2.784.723.624.144.522.883.443.222.85
StaticΔCov(%)+0.0-4.5+0.5-1.9-4.5-0.4-0.1-2.5-0.3
Width2.282.712.762.472.612.412.712.552.38
Winkler3.435.434.304.685.253.664.153.923.56
HeteroΔCov(%)+0.0-4.9+0.5-0.5-4.6-0.6-0.1-2.4-0.4
Width2.562.993.082.992.982.913.032.882.77
Winkler3.916.214.835.216.024.354.694.464.18

Shift severity. We vary regime strength d∈{0,0.8,1.6,2.4}d\in\lbrace 0,0.8,1.6,2.4\rbrace with levels eζe^{\zeta}, ζ∼N(0,(0.6d)2)\zeta\sim\mathcal{N}(0,(0.6d)^2) (d=1.6d=1.6 in the paper). At d=0d=0, periodic volatility remains but regime shifts disappear.

Across severities, BaCR's coverage error stays within 0.4 pp. Its width rises from 2.31 to 3.14, tracking the oracle (2.19 to 2.87), and it achieves the lowest coverage-valid Winkler score. The only exception is width at d=0d=0, where CoRel is marginally narrower but has a higher Winkler score.

dMetricOracleSCPNexCPHopCPTSPCICoRelACIPIDBaCR
d=0ΔCov(%)+0.0+0.2+0.5+0.4-1.4-0.6-0.0-2.6-0.3
Width2.192.592.652.482.522.302.602.462.31
Winkler3.334.194.173.984.313.534.043.823.46
d=0.8ΔCov(%)+0.0-1.8+0.5-0.2-2.5-0.2-0.1-2.6-0.3
Width2.312.712.802.652.682.552.742.602.47
Winkler3.524.844.394.464.873.894.264.043.72
d=1.6ΔCov(%)+0.0-4.9+0.5-0.5-4.6-0.6-0.1-2.4-0.4
Width2.562.993.082.992.982.913.032.882.77
Winkler3.916.214.835.216.024.354.694.464.18
d=2.4ΔCov(%)+0.0-7.5+0.5-2.0-6.6-0.2-0.0-2.3-0.1
Width2.873.363.413.223.353.273.373.233.14
Winkler4.387.725.336.377.364.945.204.994.70
AuthorsOpenReview ↗

Rebuttal (2/2)

Q2 (Width):

We agree that lower Winkler scores do not necessarily imply narrower intervals. We report widths with ΔCov below. Among methods with |ΔCov| ≤ 1 pp, BaCR has the lowest width in 6 of 8 settings: AIR, LA, and BAY overall, and AIR/LA/SD (S+S^+). On BAY S+S^+, BaCR exceeds the coverage range (−1.2 pp), while HopCPT is inside and narrower. The BAY S+S^+ gap is from five of 32 S+S^+ components (−5.7 pp, versus −0.4 pp for the other 27). Their historical records show repeated weekly profiles or a newly emerging congestion pattern, suggesting that limited historical representativeness may contribute to BaCR’s remaining under-coverage.

DatasetMetricSCPNexCPHopCPTSPCICoRelACIPIDBaCR
AIRΔCov(%)+3.3+0.8+2.3-1.4-2.2-1.2-0.5+0.3
Width75.466.773.767.354.375.073.662.7
ΔCov$_{S^+}$(%)-1.4+0.7-0.1-4.3-5.7-1.6-0.7-0.6
Width$_{S^+}$53.061.568.754.844.769.967.654.4
LAΔCov(%)-2.3-0.0-0.1-5.4-1.1-1.9-0.0+0.4
Width14.718.015.612.614.416.729.715.0
ΔCov$_{S^+}$(%)-6.0-0.3-1.5-6.2-1.5-2.4-0.6-0.1
Width$_{S^+}$12.621.116.413.115.318.952.315.7
BAYΔCov(%)-0.6+0.3+0.1-0.5-1.1-1.3-2.1+0.4
Width7.17.46.76.96.37.119.76.6
ΔCov$_{S^+}$(%)-2.9+0.3-0.8-2.7-4.5-1.3-2.0-1.2
Width$_{S^+}$7.710.28.37.87.49.423.48.6
SDΔCov(%)-2.4+0.6-0.1-4.4-2.8-1.2-0.1+0.2
Width96.3106.990.186.996.1103.9114.097.4
ΔCov$_{S^+}$(%)-4.7+0.6-0.4-5.3-4.7-1.3-0.1-0.5
Width$_{S^+}$98.4122.3111.791.698.6116.8131.7105.0

Thank you to the authors for the rebuttal, which addresses my concerns. I will maintain my positive assessment and current score.

Thank you for acknowledging our contribution and maintaining this positive assessment. We sincerely appreciate it!

Reviewer_vq6eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers online conformal prediction for multivariate time series forecasting under heterogeneous distribution shifts, where different components may experience different levels of undercoverage. The authors propose Boundary-aware Conformal Recalibration (BaCR), which combines component-wise trackers that adapt the lower and upper quantile levels using sequential coverage feedback with a periodic recalibrator that adjusts the location and scale of pretrained conditional residual quantile estimates. Theoretically, the authors derive an exact decomposition of component-wise empirical coverage error into tracker-state changes and boundary-clipping terms, along with sufficient conditions for convergence to nominal coverage. Finally, numerical experiments on synthetic and real-world multivariate time series datasets demonstrate improvements in coverage and interval efficiency compared with several existing conformal prediction methods, particularly for components experiencing larger distribution shifts.

Critical strengths
  1. The paper is well written.
  2. The idea of combining a component-wise tracker and unified recalibrator is interesting and has practical implications.
Critical weaknesses
  1. It seems that some important papers related to the proposed methodology are missing from the literature review. For example, [1] considered temporal quantile adjustment for conformal prediction, [2] studied online recalibration of predictive quantiles. Moreover, post-hoc calibration of regression models has been extensively studied (e.g., [3] and [4]). Since the proposed method essentially combines online quantile-level adjustment with periodic recalibration of conditional quantile estimates, I think these papers are quite relevant. It would be helpful if the authors could discuss these works and clarify what new methodological or theoretical insights their approach provides beyond the existing literature, particularly regarding the proposed boundary-aware saturation mechanism.

  2. Can the authors identify a concrete, reasonably general class of distribution shifts under which the proposed saturation-dependent recalibrator provably produces the interior restoring points and sufficiently small replay errors required by Remark 3?

  3. It seems that the numerical experiments use an idealized feedback protocol that may not be feasible in practice. As mentioned in Appendix B.1, for multistep forecasting, the true outcomes for all horizons are provided before the next forecast is issued. However, for h>1h>1, the corresponding outcome is only available after hh time steps. Although all competing methods use the same protocol, this could affect performance, particularly under distribution shifts. I think it would be helpful if the authors could repeat the experiments using a realistic feedback schedule, where forecast errors are used only after the corresponding outcomes become available. It would also be useful to report coverage and interval efficiency separately for different forecast horizons.

[1] Lin, Z., Trivedi, S., & Sun, J. (2022). Conformal prediction with temporal quantile adjustments. Advances in Neural Information Processing Systems, 35, 31017-31030.

[2] Deshpande, S., Marx, C., & Kuleshov, V. (2024, April). Online calibrated and conformal prediction improves Bayesian optimization. In International conference on artificial intelligence and statistics (pp. 1450-1458). PMLR.

[3] Kuleshov, V., Fenner, N., & Ermon, S. (2018, July). Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning (pp. 2796-2804). PMLR.

[4] Dheur, V., & Taieb, S. B. (2023, July). A large-scale study of probabilistic calibration in neural network regression. In International Conference on Machine Learning (pp. 7813-7836). PMLR.

Questions for authors

Kindly see the weakness section.

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We sincerely appreciate your efforts to reviewing our work and thanks for pointing out these issues. We have addressed each of your concerns below and will add all results into the revised manuscript. We believe that the revised version has been significantly improved as a result of your valuable suggestions. Please let us know if there are any remaining issues where further improvement is needed.

W1 (Related work):

We will discuss these works: [1] adjusts per-series levels of cross-sectional conformal scores for exchangeable series; [2] recalibrates predictive quantiles online, with guarantees for arbitrary sequences; [3] fits an offline CDF calibration map; [4] is a large-scale empirical study of recalibration, conformal prediction and regularization. When the required level leaves [0,1][0,1], TQA-E [1] and [2] return infinite endpoints.

Methodological insights. BaCR keeps intervals finite by tracking levels within a finite quantile range. It measures how persistently each tracker uses its outward range and passes this saturation to a component-wise affine recalibration of the quantile values through λ(log⁡s−κsat)2\lambda(\log s-\kappa \mathrm{sat})^2, which raises the scale anchor to eκ sate^{\kappa\,\mathrm{sat}} for components near their boundaries. The table below shows that keeping both modules but removing the coupling lowers coverage by 1.6–3.3 percentage point (pp) under causal feedback.

ΔCov (%), all componentsAIRLABAYSD
BaCR+0.3+0.4+0.4+0.2
λ=0\lambda=0 (no penalty)-1.3-2.1-2.1-1.4
κ=0\kappa=0 (no saturation dependence)-1.6-2.1-2.3-3.1

Theory. Thm. 1 and Lemma 4 show that, for every realized sequence, the only boundary term that lowers coverage is the widening-clipping term, bounded by the period-averaged saturation. This is why outward saturation is the feedback signal.

W2 (Shift class):

The response to this weakness is given in the next block due to character limitation.

W3 (Causal multi-step feedback):

We reran all experiments under causal multi-step feedback: each method uses the target of horizon hh only after it is observed. The table reports selected horizons for AIR (h=1,2,3) and LA(h=1,6,12), and horizon averages (Avg.) for all four datasets. Bold: lowest Winkler score among methods with |ΔCov| ≤ 1 pp.

Causal feedback does affect performance: ACI, within 1 pp in the paper, now under-covers on all four datasets; BaCR stays within 1 pp on average, with the lowest Winkler score among methods within 1 pp at every reported horizon beyond h=1h=1 and on average.

At h=1h=1 BaCR over-covers by 1.1–1.3 pp, yet has the lowest Winkler score of all methods. BaCR uses one recalibration scale per component, shared across horizons, so expansion driven by long horizons also widens short-horizon intervals. With a separate scale per horizon on LA, the error at h=1h=1 falls from +1.3+1.3 to +0.6+0.6 pp, but at h=12h=12 it moves from −0.1-0.1 to −1.1-1.1 pp.

DatasethMetricSCPNexCPHopCPTSPCICoRelACIPIDBaCR
AIR1ΔCov(%)+3.3+1.0+2.4+1.4-1.6-0.6-0.6+1.1
Winkler84.178.584.385.968.779.576.768.3
2ΔCov(%)+3.2+0.8+2.3-0.7-2.3-1.2-0.5+0.3
Winkler115.5108.9118.2119.692.2115.4112.394.0
3ΔCov(%)+3.3+0.7+2.3-4.9-2.6-1.9-0.5-0.4
Winkler135.7128.4137.6149.7108.1141.8137.5111.8
Avg.ΔCov(%)+3.3+0.8+2.3-1.4-2.2-1.2-0.5+0.3
Winkler111.8105.3113.4118.489.7112.3108.991.4
LA1ΔCov(%)-2.3+0.7-0.1-2.2-0.9-0.3-0.7+1.3
Winkler21.720.619.721.617.418.921.016.5
6ΔCov(%)-2.2+0.0-0.1-5.5-1.0-1.8-0.1+0.3
Winkler33.332.528.434.622.930.439.422.8
12ΔCov(%)-2.3-0.8-0.2-7.7-1.2-3.5+0.7-0.1
Winkler37.037.530.741.725.637.461.225.3
Avg.ΔCov(%)-2.3-0.0-0.1-5.4-1.1-1.9-0.0+0.4
Winkler32.331.727.634.222.630.241.922.4
BAYAvg.ΔCov(%)-0.6+0.3+0.1-0.5-1.1-1.3-2.1+0.4
Winkler17.113.210.413.79.711.624.89.4
SDAvg.ΔCov(%)-2.4+0.6-0.1-4.4-2.8-1.2-0.1+0.2
Winkler174.9160.1132.3165.0141.1161.0163.2130.6
AuthorsOpenReview ↗

Rebuttal (2/2)

W2 (Shift class):

Yes. A concrete class is component-wise location–scale shifts.

Let Rj,tR_{j,t} be the forecast residual of component jj and Q^j,tp\widehat Q^p_{j,t} the pretrained estimate of its level-pp quantile. The pretrained quantiles have the correct shape, but possibly the wrong centre and the wrong width,

Q^j,tp=Q^j,t1/2+σ^j,tF−1(p),Rj,t=Q^j,t1/2+νj,t+ρj,tσ^j,tεj,t.\widehat Q^p_{j,t} =\widehat Q^{1/2}_{j,t}+\hat\sigma_{j,t}F^{-1}(p), R_{j,t} =\widehat Q^{1/2}_{j,t}+\nu_{j,t} +\rho_{j,t}\hat\sigma_{j,t}\varepsilon_{j,t}.

Here FF is a fixed symmetric distribution, σ^j,t\hat\sigma_{j,t} is the predicted width, and εj,t∼F\varepsilon_{j,t}\sim F is i.i.d. noise, independent of the information available at forecast time. νj,t\nu_{j,t} is the location shift (error in the centre) and ρj,t\rho_{j,t} the scale shift (true width over predicted width). Both drift slowly (Lipschitz in t/Tt/T, uniformly in TT), except for at most NTN_T jumps per component. As regularity conditions, FF has a positive continuous density and a finite mean, σ^j,t\hat\sigma_{j,t} is bounded above and away from zero, and the quantile grid is symmetric around 1/21/2.

After recalibration with location correction bb and scale multiplier ss, the remaining scale mismatch is ρ/s\rho/s. A restoring point is an interior tracker state towards which the expected update pushes from either side.

r=F−1(qmax⁡)/F−1(1−α/2)>1r=F^{-1}(q_{\max})/F^{-1}(1-\alpha/2)>1 is the scale mismatch below which the trackers alone can restore nominal coverage once the centre is correct. ωj=2λ/(2λ+γj)∈(0,1)\omega_j=2\lambda/(2\lambda+\gamma_j)\in(0,1) bounds the fraction of an upward log scale shift that the refit leaves uncorrected, because the penalty λ(log⁡s−κsat)2\lambda(\log s-\kappa\mathrm{sat})^2 competes with the pinball fit. γj>0\gamma_j>0, guaranteed by the regularity conditions, is a lower bound on how strongly the pinball loss pulls log⁡s\log s towards log⁡ρ\log\rho.

Corollary. Consider one forecast horizon, with learning rate η\eta, recalibration period PP and buffer size BB. Suppose

\qquad (N_T+1)(B+P)=o(T),$$ and the recalibrator's optimization gap vanishes. Then $\mathbb E|\widehat{\mathrm{Cov}}_{j,T}-(1-\alpha)|\to0$ as $T,B,P\to\infty$, $\eta\to0$ and $\eta P\to\infty$. That is, the scale mismatch left after recalibration must stay within the trackers' reach $r$, and the periods and buffers affected by jumps must be a vanishing fraction of the run. **Proof sketch.** *Step 1 (local refit and restoring points).* Away from jumps, shifts vary by only $O((B+P)/T)$ over a window of $B+P$ steps, so freeze them at $(\nu,\rho)$ and let $(b^\circ,s^\circ)$ be any population optimum of the frozen objective. Symmetry gives $b^\circ=\nu$, and $s^\circ$ lies between $\rho$ and $e^{\kappa\ \mathrm{sat}}\le e^{\kappa}$. The first-order condition gives $\log(\rho/s^\circ)\le\omega_j(\log\rho-\kappa\mathrm{sat})_+$, hence $$\min \lbrace1,\rho_-e^{-\kappa}\rbrace \le \rho/s^\circ \le \max\lbrace1,\rho_+^{\omega_j}\rbrace<r.$$ Both endpoints therefore have restoring points with a uniform interior margin. *Step 2 (actual fit and clipping).* Under the stated limits, the finite-buffer error, the within-window drift and the optimization gap vanish, so the actual fit satisfies the bounds of Step 1 and the tracker updates keep their restoring property, both up to $o(1)$. The updates then contract the expected squared distance to the restoring point, so the average within-period tracking error vanishes (as in Lemma 3). A state that stays close to a restoring point with an interior margin rarely reaches a boundary, so both clipping terms vanish (as in Lemma 5). *Step 3 (replay and jumps).* The reset and replay at each refit keep the tracker state within its bounded range, so their contribution and $|\mathcal G_{j,T}|$ are $O(1/(\eta P))\to0$. Periods affected by jumps are a vanishing fraction by $(N_T+1)(B+P)=o(T)$. All terms in Theorem 1 therefore vanish. **Scope.** The corollary also holds for $\kappa=0$: it establishes the benefit of recalibration, not an additional benefit from saturation. At a refit where $\gamma_j$ remains valid, observed saturation raises the admissible scale shift from $r^{1/\omega_j}$ to $r^{1/\omega_j}e^{\kappa\,\mathrm{sat}}$, but only for the saturation observed at that refit. The result excludes shape changes and fast drift, and does not certify the one-pass Adam fit or the fixed hyperparameters of our experiments.

I thank the authors for their detailed rebuttal. I will maintain my positive score.

Thank you for acknowledging our contribution and maintaining this positive assessment. We sincerely appreciate it!

Reviewer_b67UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Boundary-aware Conformal Recalibration (BaCR) to address component-wise coverage degradation in multivariate time series forecasting under heterogeneous distribution shifts. BaCR combines component-wise online trackers with a periodically updated quantile recalibrator, using a saturation signal to identify components requiring additional interval expansion. The authors derive an exact decomposition of empirical coverage error into tracker-state variation and boundary-clipping terms. Experiments on synthetic and four real-world datasets demonstrate promising improvements in coverage and interval efficiency compared with existing conformal prediction methods.

Critical strengths
  1. The paper addresses a practically relevant limitation of existing conformal prediction methods: satisfactory aggregate coverage can conceal substantial undercoverage for individual components under heterogeneous distribution shifts.

  2. The authors investigate the contributions of the tracker and recalibrator through component ablations and stratification by local tail pressure.

  3. BaCR generally achieves favorable coverage–efficiency tradeoffs compared with seven baselines.

Critical weaknesses
  1. The theoretical results do not establish an end-to-end coverage guarantee for the complete algorithm under general distribution shifts.

  2. The incremental contribution of the saturation-aware mechanism is not sufficiently isolated. The central novelty of BaCR is the feedback from tracker saturation to quantile recalibration. However, the existing ablations primarily compare the complete method with versions removing either the tracker or the recalibrator.

  3. The experimental comparison omits many recent online/adaptive conformal baselines, including:

[1]: Wu J, Hu D, Bao Y, et al. Error-quantified conformal inference for time series[C]//ICLR 2025.

[2]: Areces F, Mohri C, Hashimoto T, et al. Online conformal prediction via online optimization[C]//ICML 2025.

[3]: Hu D, Wu J, Xia S T, et al. Distribution-informed online conformal prediction[C]//ICLR 2026.

Questions for authors
  1. How does BaCR perform under strictly causal multi-step feedback? Could the authors repeat the main experiments using updates only after each target becomes observable, rather than revealing all forecasting horizons immediately?

  2. How much of the performance improvement is specifically attributable to saturation-aware coupling? Could the authors report a comprehensive comparison between full BaCR and a version retaining both the tracker and pinball-based recalibrator but removing the saturation-dependent penalty?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We sincerely appreciate your efforts to reviewing our work and thanks for pointing out these issues. We have addressed each of your concerns below and will add all results into the revised manuscript. We believe that the revised version has been significantly improved as a result of your valuable suggestions. Please let us know if there are any remaining issues where further improvement is needed.

Q1 (Causal multi-step feedback) and W3 (Baselines):

We added ECI [1], LQT [2] and COP [3] and reran all methods under causal multi-step feedback: each horizon's target is used only after it is observed (Transformer, mean over 10 seeds). Bold: lowest Winkler score among methods with |ΔCov| ≤ 1 percentage point (pp).

Q1: BaCR stays within 1 pp of nominal coverage on all four datasets, with the lowest Winkler score among methods within 1 pp. ACI and CoRel under-cover on all four when using causal feedback. On S+S^+, BaCR is within 1 pp on AIR, LA and SD with the lowest Winkler score, and 1.2 pp below target on BAY. (The BAY S+S^+ coverage gap of BaCR comes from five of its 32 components (−5.7 pp, versus −0.4 pp for the other 27): four have records that largely repeat week to week, and one changes regime shortly before testing, so the required expansion exceeds BaCR's bounded correction.)

W3: LQT is the best of the three newly added baselines, yet it underperforms BaCR, with Winkler scores 13–32% higher across all components and 14–40% higher on S+S^+. ECI and COP are within 1 pp except on BAY, with higher Winkler scores than LQT. We will add the three baselines to the revised comparison.

DatasetMetricSCPNexCPHopCPTSPCICoRelACIPIDECILQTCOPBaCR
AIRΔCov(%)+3.3+0.8+2.3-1.4-2.2-1.2-0.5-0.5-0.1-0.7+0.3
Winkler111.8105.3113.4118.489.7112.3108.9143.2104.2130.791.4
ΔCov$_{S^+}$(%)-1.4+0.7-0.1-4.3-5.7-1.6-0.7-0.5-0.0-0.7-0.6
Winkler$_{S^+}$110.4105.5153.6116.291.0111.5106.8146.4100.9131.088.7
LAΔCov(%)-2.3-0.0-0.1-5.4-1.1-1.9-0.0-0.4-0.2-0.5+0.4
Winkler32.331.727.634.222.630.241.932.029.632.022.4
ΔCov$_{S^+}$(%)-6.0-0.3-1.5-6.2-1.5-2.4-0.6-0.2-0.2-0.3-0.1
Winkler$_{S^+}$41.739.734.442.826.537.167.739.936.540.026.0
BAYΔCov(%)-0.6+0.3+0.1-0.5-1.1-1.3-2.1+1.6+0.3+1.6+0.4
Winkler17.113.210.413.79.711.624.814.411.214.49.4
ΔCov$_{S^+}$(%)-2.9+0.3-0.8-2.7-4.5-1.3-2.0+1.3+0.2+1.3-1.2
Winkler$_{S^+}$24.617.814.519.113.415.730.019.515.219.512.8
SDΔCov(%)-2.4+0.6-0.1-4.4-2.8-1.2-0.1-0.4+0.1-0.4+0.2
Winkler174.9160.1132.3165.0141.1161.0163.2162.4148.1162.4130.6
ΔCov$_{S^+}$(%)-4.7+0.6-0.4-5.3-4.7-1.3-0.1-0.2+0.1-0.2-0.5
Winkler$_{S^+}$220.6189.4153.9197.4166.2188.8191.5194.5171.7194.4150.3

Q2 and W2 (Ablation):

The table below isolates saturation coupling under causal multi-step feedback while retaining both the tracker and the recalibrator: λ=0\lambda=0 removes the penalty, while κ=0\kappa=0 keeps it but anchors the scale at s=1s=1 (Transformer, mean over 10 seeds).

  • The tracker-only variant (w/o recal.) under-covers by 2.1–3.2 pp averaged over all components. It uses ACI-style feedback to update lower and upper quantile levels within the quantile model’s finite grid;
  • Both variants (λ=0\lambda=0 and κ=0\kappa=0) under-cover on all four datasets. Compared with λ=0\lambda=0, BaCR improves coverage by 1.6–2.5 pp on all components and 0.9–2.2 pp on S+S^+. Compared with κ=0\kappa=0, by 1.9–3.3 pp and 2.7–4.2 pp, with the larger gain on S+S^+ in every dataset. A penalty without saturation dependence (κ=0\kappa=0) is 0.5–2.8 pp worse on S+S^+ than no penalty, so the gain comes from the saturation dependence rather than from regularizing the scale;
  • Winkler scores of BaCR, λ=0\lambda=0 and κ=0\kappa=0 differ by at most 1.5%: coupling restores coverage at comparable Winkler scores;
  • On BAY S+S^+, BaCR reduces the gap from 2.7 pp (λ=0\lambda=0) and 5.4 pp (κ=0\kappa=0) to 1.2 pp.
DatasetMetricBaCRλ=0κ=0w/o trackerw/o recal.
AIRΔCov(%)+0.3-1.3-1.6-1.7-2.1
Winkler91.491.090.889.090.6
ΔCov$_{S^+}$(%)-0.6-1.5-3.5-5.9-3.6
Winkler$_{S^+}$88.788.688.789.388.9
LAΔCov(%)+0.4-2.1-2.1-1.3-2.4
Winkler22.422.522.221.622.3
ΔCov$_{S^+}$(%)-0.1-2.3-2.8-2.2-3.0
Winkler$_{S^+}$26.026.225.925.226.0
BAYΔCov(%)+0.4-2.1-2.3-2.3-2.4
Winkler9.49.49.39.29.3
ΔCov$_{S^+}$(%)-1.2-2.7-5.4-8.4-5.0
Winkler$_{S^+}$12.812.912.913.312.8
SDΔCov(%)+0.2-1.4-3.1-3.6-3.2
Winkler130.6130.6130.4130.6130.5
ΔCov$_{S^+}$(%)-0.5-1.4-4.2-5.8-4.4
Winkler$_{S^+}$150.3150.7152.5157.6153.5
AuthorsOpenReview ↗

Rebuttal (2/2)

W1 (Theory):

We agree that the current results give no end-to-end coverage guarantee under general shifts, and will state this explicitly. ACI can issue unbounded intervals. BaCR tracks a finite quantile range and relies on recalibration when that range is insufficient.

For the complete algorithm and any realized sequence, Thm. 1 decomposes the coverage error into three observable terms: tracker-state movement, widening clipping and narrowing clipping. Lemma 4 bounds widening clipping by the observed saturation. These results require neither exchangeability nor accurate quantiles, but do not ensure that the three terms are small. We therefore give a class of distribution shifts under which they vanish and coverage converges to the nominal level.

Setting. Let Rj,tR_{j,t} be the forecast residual of component jj and Q^j,tp\widehat Q^p_{j,t} the pretrained estimate of its level-pp quantile. We consider location–scale shifts: the pretrained quantiles have the correct shape, but possibly the wrong centre and the wrong width,

Q^j,tp=Q^j,t1/2+σ^j,tF−1(p),Rj,t=Q^j,t1/2+νj,t+ρj,tσ^j,tεj,t.\widehat Q^p_{j,t}=\widehat Q^{1/2}_{j,t} + \hat\sigma_{j,t}F^{-1}(p), R_{j,t} =\widehat Q^{1/2}_{j,t}+\nu_{j,t} +\rho_{j,t}\hat\sigma_{j,t}\varepsilon_{j,t}.

Here FF is a fixed symmetric distribution, σ^j,t\hat\sigma_{j,t} is the predicted width, and εj,t∼F\varepsilon_{j,t}\sim F is i.i.d. noise, independent of the information available at forecast time. νj,t\nu_{j,t} is the location shift (error in the centre) and ρj,t\rho_{j,t} the scale shift (true width over predicted width). Both drift slowly (Lipschitz in t/Tt/T, uniformly in TT), except for at most NTN_T jumps per component. As regularity conditions, FF has a positive continuous density and a finite mean, σ^j,t\hat\sigma_{j,t} is bounded above and away from zero, and the quantile grid is symmetric around 1/21/2.

After recalibration with location correction bb and scale multiplier ss, the remaining scale mismatch is ρ/s\rho/s. A restoring point is an interior tracker state towards which the expected update pushes from either side.

r=F−1(qmax⁡)/F−1(1−α/2)>1r=F^{-1}(q_{\max})/F^{-1}(1-\alpha/2)>1 is the scale mismatch below which the trackers alone can restore nominal coverage once the centre is correct. ωj=2λ/(2λ+γj)∈(0,1)\omega_j=2\lambda/(2\lambda+\gamma_j)\in(0,1) bounds the fraction of an upward log scale shift that the refit leaves uncorrected, because the penalty λ(log⁡s−κsat)2\lambda(\log s-\kappa\mathrm{sat})^2 competes with the pinball fit. γj>0\gamma_j>0, guaranteed by the regularity conditions, is a lower bound on how strongly the pinball loss pulls log⁡s\log s towards log⁡ρ\log\rho.

Corollary. Consider one forecast horizon, with learning rate η\eta, recalibration period PP and buffer size BB. Suppose

0<ρ−≤ρj,t≤ρ+<r1/ωj,(NT+1)(B+P)=o(T),0<\rho_-\le\rho_{j,t}\le\rho_+<r^{1/\omega_j}, (N_T+1)(B+P)=o(T),

and the recalibrator's optimization gap vanishes. Then E∣Cov^j,T−(1−α)∣→0\mathbb E|\widehat{\mathrm{Cov}}_{j,T}-(1-\alpha)|\to0 as T,B,P→∞T,B,P\to\infty, η→0\eta\to0 and ηP→∞\eta P\to\infty.

Proof sketch.

Step 1 (local refit and restoring points). Away from jumps, shifts vary by only O((B+P)/T)O((B+P)/T) over a window of B+PB+P steps, so freeze them at (ν,ρ)(\nu,\rho) and let (b∘,s∘)(b^\circ,s^\circ) be any population optimum of the frozen objective. Symmetry gives b∘=νb^\circ=\nu, and s∘s^\circ lies between ρ\rho and eκsat≤eκe^{\kappa\mathrm{sat}}\le e^{\kappa}. The first-order condition gives log⁡(ρ/s∘)≤ωj(log⁡ρ−κsat)+\log(\rho/s^\circ)\le\omega_j(\log\rho-\kappa\mathrm{sat})_+, hence

\le\max\lbrace1,\rho_+^{\omega_j}\rbrace<r.$$ Both endpoints therefore have restoring points with a uniform interior margin. *Step 2 (actual fit and clipping).* Under the stated limits, the finite-buffer error, the within-window drift and the optimization gap vanish, so the actual fit satisfies the bounds of Step 1 and the tracker updates keep their restoring property, both up to $o(1)$. The updates then contract the expected squared distance to the restoring point, so the average within-period tracking error vanishes (as in Lemma 3). A state that stays close to a restoring point with an interior margin rarely reaches a boundary, so both clipping terms vanish (as in Lemma 5). *Step 3 (replay and jumps).* The reset and replay at each refit keep the tracker state within its bounded range, so their contribution and $|\mathcal G_{j,T}|$ are $O(1/(\eta P))\to0$. Periods affected by jumps are a vanishing fraction by $(N_T+1)(B+P)=o(T)$. All terms in Theorem 1 therefore vanish. **Scope.** The corollary also holds for $\kappa=0$: it establishes the benefit of recalibration, not an additional benefit from saturation. At a refit where $\gamma_j$ remains valid, observed saturation raises the admissible scale shift from $r^{1/\omega_j}$ to $r^{1/\omega_j}e^{\kappa\mathrm{sat}}$, but only for the saturation observed at that refit. The result excludes shape changes and fast drift, and does not certify the one-pass Adam fit or the fixed hyperparameters of our experiments.

It seems that the authors need to adjust the learning rates for ECI and COP based on the specific dataset, but that’s not a big deal. So I will improve my score. Good luck.

We sincerely thank the reviewer for taking the time to carefully reconsider our submission and update the evaluation. We highly appreciate your constructive suggestions!

For ECI and COP, we select one step-size coefficient on a held-out validation segment for each dataset, backbone, and seed. The selected coefficient is shared across all components and forecast horizons, without additional component-wise tuning. Following ECI and COP authors’ implementations, each component–horizon pair automatically scales its effective step size by the range of its recent scores.

For BaCR, we use the same online calibration hyperparameter configuration across all four datasets and both backbones, including the tracker learning rate, recalibration period, buffer size, recalibrator learning rate, and saturation-penalty parameters. We will make this distinction explicit.

Thank you again for your valuable feedback and continued engagement with our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revisions to the paper

Dear Reviewers and AC,

We have uploaded the revised manuscript, incorporating the changes described in our rebuttal. Revisions are highlighted in blue for ease of reference. The key modifications are summarized below:

Related work:

  • We discussed ECI, LQT, COP (reviewer b67U W3) and four related works about quantile recalibration (reviewer vq6e W1). See lines 130 and 142 in the revised version.

Theory:

  • We analyzed a class of distribution shifts in which the boundary values tend to zero and the coverage converges to the nominal level (reviewer vq6e W2 and reviewer b67U W1). See line 356 and Appendix A.2.

Experiments:

  • Evaluation metric: We added interval width as a new evaluation metric and reported corresponding results in every table (reviewer P6PA Q2).

  • Baselines: We added ECI, LQT and COP as the latest baselines (reviewer b67U W3) and the implementation details are given in Appendix B.1 (line 1519). The results of LQT and COP are put in the main comparison table (Section 6, Table 1 and Table 2). The results of ECI are given in Appendix B.2.3 (Table 6).

  • Results:

    • Synthetic datasets: We evaluated the performance of the proposed method under different levels of heterogeneity and degrees of shift severity. The relevant results and analysis are presented in Tables 3 and 4 in Appendix B.2.1 (reviewer P6PA Q1).
    • Real datasets:
      • We tested the heterogeneity of used datasets and reported the statistics in Table 5 in Appendix B.2.2 (reviewer P6PA W1);
      • We updated the main results in Table 2 and Table 6 using causal multi-step feedback (reviewer b67U Q1 and reviewer vq6e W3). The results for individual forecast horizons are reported in Table 7 in Appendix B.2.4;
      • We did a comprehensive ablation study using causal multi-step feedback (reviewer b67U Q2 and W2) and presented these results in Figure 2 in Section 6.2. The numerical results are given in Table 8 in Appendix B.3.1;
      • We updated the sensitivity analysis in Section B.4 under causal feedback setting.

Best regards, Authors

AuthorsOpenReview ↗

Rebuttal Summary to AC

Dear Area Chair,

We thank the reviewers for their careful evaluation and constructive feedback. Below, we summarize the recognized strengths and revisions addressing their concerns.

Strengths Recognized by the Reviewers

  • A practically important and clearly motivated problem (reviewer b67U; Reviewer P6PA). Reviewer b67U highlighted that satisfactory aggregate coverage can conceal substantial component-wise undercoverage under heterogeneous distribution shifts. BaCR directly targets this limitation in multivariate forecasting, where individual variables can experience different shifts and require different corrections.

  • An intuitive and meaningful methodological contribution (reviewer vq6e; Reviewer P6PA). Reviewer vq6e found the combination of trackers and a unified recalibrator interesting and practically relevant, and Reviewer P6PA described the method as novel and easy to follow. The trackers adjust quantile levels after feedback arrives, while the recalibrator periodically updates the quantile values. Their interaction is guided by observable saturation, which signals insufficient outward adjustment range and directs component-specific expansion while keeping the underlying models frozen.

  • Substantial empirical support and favorable coverage–efficiency trade-offs (Reviewer b67U; Reviewer P6PA). Reviewer b67U recognized BaCR's favorable trade-offs against seven baselines and the investigation of its modules through ablations and local-tail-pressure stratification. Reviewer P6PA highlighted the solid evaluation spanning synthetic and real-world data, coverage and Winkler scores.

  • Clear presentation and an accessible design (Reviewer vq6e; Reviewer P6PA). Reviewer vq6e explicitly praised the writing, and Reviewer P6PA found the method straightforward to understand and follow. The separation between tracker correction and periodic recalibration makes the roles of the two modules clear, while the coverage decomposition explains how their finite adjustment range can limit coverage and motivates the saturation feedback.

Responses to All Reviewer Concerns

  1. Causal feedback and forecast horizons (Reviewer b67U Q1; Reviewer vq6e W3).

    We reran comparisons using targets only after they become observable and added per-horizon results. Across four datasets, BaCR's overall coverage deviation is +0.2 to +0.4 percentage points (pp), with the lowest Winkler score among methods within 1 pp of nominal coverage.

  2. Recent baselines and parameter selection (Reviewer b67U W3 and follow-up).

    We added ECI, LQT, and COP. Each baseline coefficients are selected on validation data while retaining automatic score-based scaling. BaCR uses one online calibration hyperparameter configuration across all four datasets and both backbones.

  3. Saturation-coupling ablations (Reviewer b67U W2/Q2).

    Keeping both modules, we removed either the penalty (λ=0\lambda=0) or its saturation dependence (κ=0\kappa=0). Full BaCR improves overall coverage by 1.6–3.3 pp, with Winkler scores differing by at most 1.5%, isolating the coupling's empirical contribution.

  4. Concrete shift class and theoretical scope (Reviewer b67U W1; Reviewer vq6e W2).

    We added an analysis of slowly varying location–scale shifts with controlled jumps, linking recalibration to interior restoring points, vanishing clipping terms, and convergence of component-wise time-averaged coverage under the stated conditions.

  5. Related work (Reviewer vq6e W1).

    We discussed all four suggested references and clarified BaCR's distinction: outward saturation within a finite quantile range guides component-specific recalibration, supported by the coverage decomposition and coupling ablations.

  6. Heterogeneity and shift severity (Reviewer P6PA W1/Q1).

    We quantified heterogeneous component coverage in the real datasets, and added heterogeneity settings and shift-severity settings on synthetic data. BaCR maintains coverage deviation within 0.4 pp throughout, including the homogeneous case.

  7. Interval width (Reviewer P6PA W1/Q2).

    We added interval width alongside coverage and Winkler scores, explicitly reporting the coverage–efficiency trade-off in both baseline comparisons and coupling ablations.

Following the rebuttal discussion, all three reviewers have confirmed that their concerns have been addressed. All corresponding experiments and clarifications have been incorporated into the revised manuscript.

Thank you for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.