- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies conformal prediction for time series and proposes RareCP, a weighted residual-calibration method that wraps any fixed point forecaster. Each of M retrieval experts encodes the query and calibration through a dataset-conditioned affine map generated by a hypernetwork. Each expert retrieves the top- most cosine-similar calibration examples and weights them with a softmax within that set. A history-dependent gate mixes the experts’ signed-residual distributions, and the α/2 and 1−α/2 weighted quantiles give an asymmetric interval, which ACI then adjusts online. Empirically, the paper compares 13 CP methods and 9 native TSFM intervals on four backbones. It reports that RareCP attains the lowest nWink among CP methods on every backbone, and that RareCP on TiRex (0.61) beats the strongest native model, Toto2 (0.64).
Critical strengths
RareCP presents a general framework for residual calibration that can be applied to different forecasting backbones. The combination of dataset-conditioned representations, sparse retrieval, and a history-dependent mixture of experts is an interesting approach to modeling heterogeneous residual distributions.
- Comprehensive experiments. Many CP methods and native TSFM intervals are evaluated. RareCP achieves competitive Winkler scores across multiple forecasting backbones, outperforming the evaluated CP baselines on all four backbones;
- Clear theoretical analysis. The decomposition into sampling error, localization bias and population discrepancy is clean, and the assumptions are stated explicitly.
Critical weaknesses
- RareCP has no finite-sample coverage guarantee of its own. The weighted CDF in Eqs. (1)–(3) includes no test-point mass, and query-dependent top-K weights break the conditions required by NexCP. The paper relies on ACI for its stated long-run empirical coverage control, while the learned weighted residual calibration itself has no established finite-sample coverage guarantee. Since ACI can equally wrap native TSFM quantiles, the comparison that supports the paper’s motivation is RareCP+ACI versus native quantiles conformalized with ACI or CQR. This baseline is missing, even though some papers are cited for exactly this approach. The claim of “closing the gap” also fails on the matched Toto2 backbone, where RareCP scores 0.70 against the native 0.64.
- The aggregation is fragile. Unweighted means over datasets whose nWink ranges from 0.006 to 4.4 are dominated by a few hard datasets. Several test sets contain only 84–160 targets;
- The calibration-pool scope differs across backbones and is not sufficiently justified or ablated. Sec 4.1 says all methods receive the same calibration examples, but Tabs 6–7 show that RareCP retrieves from an all-dataset pool for three of the four backbones. Although Appendix C.2 describes dataset-level target normalization, it remains unclear whether this scaling adequately aligns the residual distributions used in cross-dataset retrieval. A per-dataset versus cross-dataset pooling ablation is needed. It is unclear whether competitors can use the 10% fitting split, and ACI is withheld from the weighted baselines even though it contributes 10% of RareCP’s gain in Fig. 4(a);
- The substantial undercoverage of KOWCPI and ResCP raises concerns about the calibration and tuning protocol. It would be useful to verify their implementations and report coverage-matched comparisons. KOWCPI (average coverage 0.68) and ResCP (0.74) under-cover substantially. HopCPT’s configurations are identical across three backbones despite supposedly independent searches. MoE-CP should be interpreted only as an additional end-to-end reference rather than a matched-backbone comparison;
- Key mechanistic claims are not supported. Section 4.3 reports a 0.99 cosine similarity between generated maps and their dataset mean, so the “sample-dependent maps” that distinguish RareCP effectively reduce to one affine map per dataset. The use of dataset identifiers makes zero-shot transfer to unseen datasets unclear, and no leave-one-dataset-out evaluation is provided. The ablations use only TiRex, and their starting point (0.7257) already beats every CP competitor, which suggests that much of the gain comes from the shared pipeline rather than top-K retrieval, the DCA or the MoE.
- The experiments do not test the stated motivation. There is no evidence that RareCP retrieves distant recurring regimes, such as a lag distribution of retrieved neighbors or a comparison with a recency-restricted variant. Regime adaptivity should be evaluated through regime-conditional coverage and interval efficiency, rather than aggregate coverage alone, but only marginal coverage is reported, and the interpretability claim has no supporting case study;
- Some assumptions are questionable and some constants are vacuous. A1 is used only through a covariance inequality, so α-mixing would suffice. More importantly, it is implausible under the source-then-time ordering for cross-sectionally correlated panels such as solar, LOOP_SEATTLE and electricity. Although A2 holds formally, κ = e^{2/τ_e} reaches about 10⁸–10¹³ under the reported β values, which makes bound (31) vacuous. Although A2 ensures asymptotically growing effective sample size for fixed retrieval temperatures, the extremely large worst-case weight-ratio constants may render the finite-sample bound uninformative. Reporting the empirical effective sample size would help assess the practical relevance of the theory;
- The theorem does not describe the deployed algorithm, and its interpretation is overstated. Deployment uses a fixed K, a fixed-capacity FIFO pool, data-dependent ACI levels α_t and a validation-selected K_n. The theorem establishes consistency for fixed quantile levels, but does not establish uniform quantile consistency over the range of adaptively selected miscoverage levels used by ACI. A version uniform in p would at least cover ACI. ε_q(h) also absorbs calibration-to-test distribution shift, not just “learning and approximation error”.
Questions for authors
- How does RareCP compare with native TSFM quantiles conformalized via ACI, CQR or normalized scores on the matched backbones, and what is the case for RareCP if they perform comparably?
- How are residuals standardized before cross-dataset retrieval, and how sensitive are the results to the calibration-pool scope? Could the authors report per-dataset versus cross-dataset pooling results on each backbone?
- What fraction of temperature_rain test targets are imputed, and how do the per-dataset and average results change when these targets are excluded?
- Can you show that retrieved neighbors are temporally distant recurrences and report regime-conditional coverage? Can they also provide ablations for joint versus separate training, a per-dataset affine map in place of the hypernetwork, leave-one-dataset-out transfer, and a causal buffer in the leave-one-out training episodes?
- How are multiple series ordered within one ACI stream, how are α_t ≤ 0 and α_t ≥ 1 handled, and how does RareCP perform at 50%, 90% and 95% nominal levels?
Author response on calibration baselines, theoretical assumptions, and coverage
We thank the reviewers for their careful reading and constructive feedback. We will revise and resubmit the paper.
GRjE and oNsS asked how RareCP compares with native uncertainty estimates after recalibration. GRjE also asked for ACI on the weighted conformal baselines. We will evaluate native quantiles with ACI and CQR on matched backbones and apply ACI to the strongest conformal baselines.
fKBQ, oNsS, and GRjE asked how the growing-pool theorem relates to the benchmark protocol. The theorem characterizes when sampling error and localization bias vanish for RareCP’s fitted top-K retrieval rule. Figure 3 studies that behavior by varying calibration size while holding the fitted model fixed. On GIFT-Eval, we use a fixed-capacity rolling pool to follow the competing methods’ online protocols more closely. Within each backbone, methods share point forecasts and chronological test targets. All three reviewers also raised questions about calibration access, therefore, we will provide RareCP’s performance in the comparable setting only building the calibration pool from samples within the same series.
As GRjE and oNsS noted, the dependence assumptions could better reflect the structure of panel datasets such as solar, traffic, and electricity. Grouping observations by timestamp in the analysis would allow us to model cross-series dependence within each timestamp while keeping the retrieval rule as is. As GRjE points out, the proof uses mixing through a covariance bound, so we will state the weaker dependence condition it needs. We will also report empirical effective sample sizes to show how concentrated the retrieval weights are in practice.
GRjE and oNsS asked what coverage RareCP’s retrieval rule provides. Our theorem shows that, as the calibration pool grows, the maximum endpoint error at a fixed history h converges in probability to εq(h), the discrepancy between the fitted mixture quantiles and the history-conditional target quantiles. Under the assumptions of Appendix B.4, if εq(h)=0 and the target CDF is continuous at both endpoints, an independent test residual at that history has limiting coverage 1−α.
Related methods handle this issue differently. HopCPT’s conditional coverage argument assumes that the correct error regime is identified for the test input. ResCP establishes coverage conditional on its reservoir state, which can group together histories with different residual quantiles. RareCP retains the resulting endpoint discrepancy explicitly, allowing the analysis to cover imperfect retrieval. Finite-sample validity for RareCP’s retrieval rule remains open. ACI separately controls long-run empirical miscoverage under its update and boundary conventions, which we will document as requested by fKBQ and GRjE.