acceptodds
Withdrawn from ICLR 2027

RareCP: Regime-Aware Retrieval for Efficient Conformal Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conformal prediction has become increasingly popular for uncertainty quantification, providing distribution-free coverage guarantees and extensions that remain applicable to sequential and time-series data. Yet, broad common protocol comparisons between conformal methods and the native uncertainty estimates of recent time series foundation models remain limited. We show that, for one-step prediction intervals, existing conformal approaches can be substantially less efficient than native intervals. To close this gap, we introduce RareCP, a weighted conformal prediction method with a retrieval architecture tailored to learning recurring residual regimes and adapting to the current forecasting context. We characterize conditions under which calibration consistently estimates the fitted model's population quantiles, isolating its remaining learning and approximation error. We additionally use adaptive conformal inference (ACI) for long-run empirical coverage control under distribution drift.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Time Series Forecasting, Conformal Prediction, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GRjEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies conformal prediction for time series and proposes RareCP, a weighted residual-calibration method that wraps any fixed point forecaster. Each of M retrieval experts encodes the query and calibration through a dataset-conditioned affine map generated by a hypernetwork. Each expert retrieves the top-KnK_n most cosine-similar calibration examples and weights them with a softmax within that set. A history-dependent gate mixes the experts’ signed-residual distributions, and the α/2 and 1−α/2 weighted quantiles give an asymmetric interval, which ACI then adjusts online. Empirically, the paper compares 13 CP methods and 9 native TSFM intervals on four backbones. It reports that RareCP attains the lowest nWink among CP methods on every backbone, and that RareCP on TiRex (0.61) beats the strongest native model, Toto2 (0.64).

Critical strengths

RareCP presents a general framework for residual calibration that can be applied to different forecasting backbones. The combination of dataset-conditioned representations, sparse retrieval, and a history-dependent mixture of experts is an interesting approach to modeling heterogeneous residual distributions.

  1. Comprehensive experiments. Many CP methods and native TSFM intervals are evaluated. RareCP achieves competitive Winkler scores across multiple forecasting backbones, outperforming the evaluated CP baselines on all four backbones;
  2. Clear theoretical analysis. The decomposition into sampling error, localization bias and population discrepancy is clean, and the assumptions are stated explicitly.
Critical weaknesses
  1. RareCP has no finite-sample coverage guarantee of its own. The weighted CDF in Eqs. (1)–(3) includes no test-point mass, and query-dependent top-K weights break the conditions required by NexCP. The paper relies on ACI for its stated long-run empirical coverage control, while the learned weighted residual calibration itself has no established finite-sample coverage guarantee. Since ACI can equally wrap native TSFM quantiles, the comparison that supports the paper’s motivation is RareCP+ACI versus native quantiles conformalized with ACI or CQR. This baseline is missing, even though some papers are cited for exactly this approach. The claim of “closing the gap” also fails on the matched Toto2 backbone, where RareCP scores 0.70 against the native 0.64.
  2. The aggregation is fragile. Unweighted means over datasets whose nWink ranges from 0.006 to 4.4 are dominated by a few hard datasets. Several test sets contain only 84–160 targets;
  3. The calibration-pool scope differs across backbones and is not sufficiently justified or ablated. Sec 4.1 says all methods receive the same calibration examples, but Tabs 6–7 show that RareCP retrieves from an all-dataset pool for three of the four backbones. Although Appendix C.2 describes dataset-level target normalization, it remains unclear whether this scaling adequately aligns the residual distributions used in cross-dataset retrieval. A per-dataset versus cross-dataset pooling ablation is needed. It is unclear whether competitors can use the 10% fitting split, and ACI is withheld from the weighted baselines even though it contributes 10% of RareCP’s gain in Fig. 4(a);
  4. The substantial undercoverage of KOWCPI and ResCP raises concerns about the calibration and tuning protocol. It would be useful to verify their implementations and report coverage-matched comparisons. KOWCPI (average coverage 0.68) and ResCP (0.74) under-cover substantially. HopCPT’s configurations are identical across three backbones despite supposedly independent searches. MoE-CP should be interpreted only as an additional end-to-end reference rather than a matched-backbone comparison;
  5. Key mechanistic claims are not supported. Section 4.3 reports a 0.99 cosine similarity between generated maps and their dataset mean, so the “sample-dependent maps” that distinguish RareCP effectively reduce to one affine map per dataset. The use of dataset identifiers makes zero-shot transfer to unseen datasets unclear, and no leave-one-dataset-out evaluation is provided. The ablations use only TiRex, and their starting point (0.7257) already beats every CP competitor, which suggests that much of the gain comes from the shared pipeline rather than top-K retrieval, the DCA or the MoE.
  6. The experiments do not test the stated motivation. There is no evidence that RareCP retrieves distant recurring regimes, such as a lag distribution of retrieved neighbors or a comparison with a recency-restricted variant. Regime adaptivity should be evaluated through regime-conditional coverage and interval efficiency, rather than aggregate coverage alone, but only marginal coverage is reported, and the interpretability claim has no supporting case study;
  7. Some assumptions are questionable and some constants are vacuous. A1 is used only through a covariance inequality, so α-mixing would suffice. More importantly, it is implausible under the source-then-time ordering for cross-sectionally correlated panels such as solar, LOOP_SEATTLE and electricity. Although A2 holds formally, κ = e^{2/τ_e} reaches about 10⁸–10¹³ under the reported β values, which makes bound (31) vacuous. Although A2 ensures asymptotically growing effective sample size for fixed retrieval temperatures, the extremely large worst-case weight-ratio constants may render the finite-sample bound uninformative. Reporting the empirical effective sample size would help assess the practical relevance of the theory;
  8. The theorem does not describe the deployed algorithm, and its interpretation is overstated. Deployment uses a fixed K, a fixed-capacity FIFO pool, data-dependent ACI levels α_t and a validation-selected K_n. The theorem establishes consistency for fixed quantile levels, but does not establish uniform quantile consistency over the range of adaptively selected miscoverage levels used by ACI. A version uniform in p would at least cover ACI. ε_q(h) also absorbs calibration-to-test distribution shift, not just “learning and approximation error”.
Questions for authors
  1. How does RareCP compare with native TSFM quantiles conformalized via ACI, CQR or normalized scores on the matched backbones, and what is the case for RareCP if they perform comparably?
  2. How are residuals standardized before cross-dataset retrieval, and how sensitive are the results to the calibration-pool scope? Could the authors report per-dataset versus cross-dataset pooling results on each backbone?
  3. What fraction of temperature_rain test targets are imputed, and how do the per-dataset and average results change when these targets are excluded?
  4. Can you show that retrieved neighbors are temporally distant recurrences and report regime-conditional coverage? Can they also provide ablations for joint versus separate training, a per-dataset affine map in place of the hypernetwork, leave-one-dataset-out transfer, and a causal buffer in the leave-one-out training episodes?
  5. How are multiple series ordered within one ACI stream, how are α_t ≤ 0 and α_t ≥ 1 handled, and how does RareCP perform at 50%, 90% and 95% nominal levels?
Reviewer_oNsSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RareCP, a retrieval-based framework for constructing adaptive prediction intervals in time-series forecasting. RareCP combines dataset-conditioned affine transformations, sparse residual retrieval experts, and a context-dependent mixture-of-experts gate to estimate asymmetric residual quantiles. Adaptive conformal inference (ACI) is additionally employed to regulate empirical coverage under distribution shift. The authors establish asymptotic consistency of the estimated quantiles for the fitted retrieval mixture under suitable assumptions. Experiments demonstrate competitive interval efficiency against existing conformal prediction methods and native uncertainty estimates from time-series foundation models.

Critical strengths
  1. The paper addresses an problem: obtaining reliable and efficient uncertainty estimates for time-series foundation models under temporal dependence and distribution shift.
  2. The integration of dataset-conditioned affine maps (DCA), sparse top-(K) retrieval, and context-dependent expert mixing provides a flexible mechanism for estimating heterogeneous residual distributions
Critical weaknesses
  1. There is a substantial gap between the theoretical assumptions and the practical algorithm. The consistency theorem relies on independently fitted representations, geometric beta mixing, stable local residual distributions, sufficient local recurrence, and a retrieval schedule satisfying Kn/n→0K_n/n \to 0 and log⁡(en/Kn)/Kn→0\log(en/K_n)/K_n \to 0. In contrast, the practical algorithm maintains a fixed-capacity rolling calibration pool, uses validation-selected retrieval sizes, and continuously updates the nominal miscoverage level through ACI. Moreover, the asymptotic result allows a nonvanishing population approximation error ϵq(h)\epsilon_q(h), with no general guarantee that the learned experts and gate make this error small.
  2. The novelty and necessity of the individual architectural components could be better established. While the proposed framework combines several established techniques, the fundamental methodological novelty relative to existing retrieval-based conformal methods, particularly HopCPT and ResCP, could be articulated more clearly.
  3. The central conformal coverage guarantee remains insufficiently established.
Questions for authors
  1. How exactly are calibration examples and representations constructed? Section 3 describes retrieval from the query dataset, whereas Tables 6–7 in Appendix C.4 indicate that several backbones use calibration pools containing all datasets. Does this mean that residuals from different datasets can be retrieved for a given query? If so, how are residual scales normalized and chronological information constraints enforced?
  2. How does RareCP compare with recalibrated native TSFM intervals using the same calibration data and ACI updates?
  3. Can the authors explain whether any part of Theorem 1 extends to a fixed-capacity rolling calibration pool with continuously updated residuals and an adaptive miscoverage level?
Reviewer_fKBQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RareCP, a learned residual-retrieval method for constructing asymmetric prediction intervals around a fixed forecasting model. Multiple retrieval experts use dataset-conditioned affine maps to identify relevant historical contexts and weight their signed residuals. A history-dependent gate combines the expert distributions before extracting interval endpoints. Experts and the gate are trained in stages using a smoothed Winkler-score objective, while adaptive conformal inference adjusts the miscoverage level during online evaluation. The empirical evaluation covers 22 GIFT-Eval datasets, four forecasting backbones, 13 conformal prediction competitors, and nine native time-series foundation-model uncertainty estimates. The paper also provides component ablations, a controlled retrieval-scaling experiment, and a consistency result for estimating the quantiles of a frozen learned mixture under assumptions on temporal dependence, localization, and neighborhood growth.

Critical strengths

The empirical study addresses a useful comparison and reports meaningful improvements in interval quality. The evaluation brings together a substantial range of online and weighted CP methods and native TSFM intervals using common forecasting targets and, where applicable, cached point forecasts. Reporting Winkler score alongside width and coverage is valuable because several competitors obtain narrow intervals partly through undercoverage. These results suggest practical value beyond simply reducing interval width. The comparison with native uncertainty estimates is also informative: RareCP improves on matched TiRex and TimesFM estimates, but trails native Toto2. This qualifies the scope of the improvement and supports a useful, rather than universal, empirical contribution.

The retrieval-mixture design is coherent and has supporting evidence beyond the final benchmark table. Figure 4 includes cumulative component ablations, comparisons against MLP encoders, and an expert-count/runtime study. These experiments support incremental benefits from sparse retrieval, dataset-conditioned geometry, and expert mixing, although the largest individual gain in the cumulative ablation comes from cosine normalization. The theoretical analysis also carefully separates calibration estimation error from the fitted model’s remaining population error. This provides a defensible explanation of the locality–sample-size trade-off. I view the contribution primarily as a learned calibration architecture with empirical support; the ability to wrap a fixed point forecaster is useful but is shared with existing post-hoc CP methods and should be positioned accordingly.

Critical weaknesses

The calibration-data protocol is internally inconsistent, making the central comparison and attribution of improvements difficult to assess. Section 3 describes calibration using histories from the query’s dataset, and Section 4.1 states that all CP methods receive the same calibration examples. However, Tables 6–7 specify a calibration pool covering “all datasets” for TimesFM, Toto, and ARIMA, with only TiRex using a “per dataset” pool. Moreover, Tables 31–32 specify “per series” calibration for HopCPT. These descriptions imply materially different residual access across methods, even when forecasts and test targets are shared.

Pooling information across datasets or series can be a legitimate contribution, but the current evidence does not clearly distinguish benefits from the retrieval architecture, joint training, broader calibration access, and calibration-pool size. Figure 4 provides useful architectural ablations but does not resolve this issue. A single-dataset RareCP variant, together with controls that hold calibration scope and size fixed while varying joint training, would materially strengthen the comparison. This concern directly affects how much of the reported improvement can be attributed to the proposed method.

The implemented training and retrieval pipeline is not fully reconciled with the formal description. Although Section 3.1 provides a high-level description of the two-stage training procedure, the specific loss functions used to train the retrieval experts and the mixture gate are only presented in Appendix A. Similarly, the sigmoid-based approximation of weighted quantiles, which is essential for making the training objective differentiable, is not formally defined in the main text. Since these components are central to the proposed method, I believe their mathematical formulations should be included in the main body to make the training procedure understandable without relying extensively on the appendix. Furthermore, Equation (23) introduces an anchoring penalty to prefit teacher maps without clearly explaining how the teacher bank is obtained. Tables 6–7 additionally mention backbone-dependent student distillation, but its relationship to the described two-stage training procedure remains unclear. There is also an apparent discrepancy in the retrieval maps: Equations (4)–(5) state that each history is encoded using its own generated affine map, whereas the text following Equation (21) appears to apply the query-generated map to both the query and its candidates. These alternatives define different retrieval mechanisms and have different implications for caching calibration representations.

These ambiguities matter because they concern central components of the method used to obtain the results. The substantial offline training overhead further increases the importance of understanding exactly what is trained and which components justify that complexity.

Questions for authors
  • Which calibration-pool configuration actually produced Table 1? Please reconcile the within-dataset description in Section 3 and the common-calibration statement in Section 4.1 with the “all datasets” settings in Tables 6–7 and HopCPT’s per-series scope. Please specify candidate-pool sizes and the rules governing which residuals are available to each method.

  • How do results change between (i) RareCP trained separately per dataset with within-dataset calibration, (ii) jointly trained RareCP with the same within-dataset calibration pools, and (iii) jointly trained RareCP with pooled calibration? A comparison holding candidate-pool size and online correction fixed, and giving a simpler retrieval baseline comparable residual access, would help determine whether the architectural improvements persist under matched information.

  • How are the teacher maps trained, what exactly does student distillation optimize, and how do these operations correspond to Equation (23)? Does each calibration history use its own generated affine map, or the current query’s map? A unified algorithm description connecting teacher construction, expert fitting, gate fitting, and inference would resolve these uncertainties.

  • The consistency theorem assumes fitting data independent of calibration data and a growing retrieval neighborhood, whereas the benchmark uses chronological splits and fixed selected neighborhood sizes with rolling memory. Which theoretical conclusions are intended to apply directly to that implementation? Please also specify the ACI update and boundary conventions used when the adaptive miscoverage level leaves its nominal range, so the applicable coverage claim is unambiguous.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response on calibration baselines, theoretical assumptions, and coverage

We thank the reviewers for their careful reading and constructive feedback. We will revise and resubmit the paper.

GRjE and oNsS asked how RareCP compares with native uncertainty estimates after recalibration. GRjE also asked for ACI on the weighted conformal baselines. We will evaluate native quantiles with ACI and CQR on matched backbones and apply ACI to the strongest conformal baselines.

fKBQ, oNsS, and GRjE asked how the growing-pool theorem relates to the benchmark protocol. The theorem characterizes when sampling error and localization bias vanish for RareCP’s fitted top-KK retrieval rule. Figure 3 studies that behavior by varying calibration size while holding the fitted model fixed. On GIFT-Eval, we use a fixed-capacity rolling pool to follow the competing methods’ online protocols more closely. Within each backbone, methods share point forecasts and chronological test targets. All three reviewers also raised questions about calibration access, therefore, we will provide RareCP’s performance in the comparable setting only building the calibration pool from samples within the same series.

As GRjE and oNsS noted, the dependence assumptions could better reflect the structure of panel datasets such as solar, traffic, and electricity. Grouping observations by timestamp in the analysis would allow us to model cross-series dependence within each timestamp while keeping the retrieval rule as is. As GRjE points out, the proof uses mixing through a covariance bound, so we will state the weaker dependence condition it needs. We will also report empirical effective sample sizes to show how concentrated the retrieval weights are in practice.

GRjE and oNsS asked what coverage RareCP’s retrieval rule provides. Our theorem shows that, as the calibration pool grows, the maximum endpoint error at a fixed history hh converges in probability to εq(h)\varepsilon_q(h), the discrepancy between the fitted mixture quantiles and the history-conditional target quantiles. Under the assumptions of Appendix B.4, if εq(h)=0\varepsilon_q(h) = 0 and the target CDF is continuous at both endpoints, an independent test residual at that history has limiting coverage 1−α1 - \alpha.

Related methods handle this issue differently. HopCPT’s conditional coverage argument assumes that the correct error regime is identified for the test input. ResCP establishes coverage conditional on its reservoir state, which can group together histories with different residual quantiles. RareCP retains the resulting endpoint discrepancy explicitly, allowing the analysis to cover imperfect retrieval. Finite-sample validity for RareCP’s retrieval rule remains open. ACI separately controls long-run empirical miscoverage under its update and boundary conventions, which we will document as requested by fKBQ and GRjE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.