- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of Bounded-Abstention Multi-Horizon Forecasting, which aims to minimize prediction risk for accepted time steps while satisfying a minimum coverage constraint. Given the sequential temporal structure inherent in multi-horizon forecasting, the authors propose three structured abstention strategies: Full Abstention (FAbFor), Partial Abstention (PAbFor), and Interval Abstention (IntAbFor). The paper establishes a unified optimization framework for these strategies and derives optimal selection functions based on conditional prediction risk. In terms of implementation, the authors employ joint training of a forecaster and a risk estimator, utilizing a calibration set to determine the abstention parameters. Experiments conducted across 24 datasets demonstrate that the proposed methods outperform several baselines in terms of selective prediction risk.
Critical strengths
S1. Existing selective prediction methods primarily focus on single-step or independent predictions, making them ill-suited for multi-horizon forecasting, which requires continuous prediction intervals. To address this, the paper proposes a structured abstention mechanism—an approach with sound practical motivation and significant research value.
S2. The authors unify full, partial, and interval-based abstention into a bounded-abstention optimization framework and analyze optimal selection strategies under various structural constraints. These three strategies follow a clear, progressive relationship, which sheds light on how the flexibility of the selection process influences prediction risk.
Critical weaknesses
W1. The mathematical definition of the randomized calibration mechanism lacks sufficient rigor. In Eq. (9), if the empirical coverage function exhibits discontinuities (jumps), the two defined thresholds may coincide, rendering it impossible to achieve the target coverage via the randomization strategy described in Eq. (10). While a binary search algorithm might circumvent this issue in practice, the mathematical definition in the paper requires further clarification. The relationship between conditional risk and conditional variance requires further elaboration. Section 3 equates the conditional prediction risk (under MSE) to the conditional variance; however, this equality generally holds only if the predictor is equal to the true conditional mean. Although Gaussian β-NLL can be used to estimate the conditional squared prediction error, the authors should explicitly clarify the relationship between their estimation objective and the theoretical conditional risk.
W2. The fairness of comparisons with external baselines warrants further validation. While the authors compare various abstention strategies and risk estimation methods, the external baselines employ different training objectives, which may introduce discrepancies in base predictive performance. Furthermore, risk comparisons based on a fixed target coverage may be confounded by deviations in the actual test-time coverage. Conducting experiments that utilize a fixed predictor while matching the actual achieved coverage would significantly strengthen the credibility of the conclusions.
Questions for authors
Q1. When the empirical coverage function exhibits a jump—causing the two thresholds in Eq. (9) to coincide—how does the mechanism in Eq. (10) ensure that the target coverage is met?
Q2. Could the authors clarify whether the Gaussian β-NLL estimates the conditional variance of the data or the conditional expected squared error relative to the current predictor? How do these relate to the theoretical conditional risk?
Q3. Could the authors clarify whether the external baselines utilize the same point predictions? Additionally, could they provide risk comparisons based on the actual test-time coverage to mitigate the influence of differences in predictors and deviations in coverage on the experimental conclusions?
5 comments
Experiments with fixed test-time coverage. We calibrate the thresholds on a held-out calibration set as is standard in selective prediction [1,2]. To fully remove the effect of coverage deviations, we repeat Q1, setting each method’s rejection threshold directly on the test set. Thus, all methods have the same test coverage. While this is not a deployable procedure, it isolates how well each method ranks the series by risk. Across the six target coverages, FAbFor outperforms the competitors in 21/24 datasets and reduces the selective risk by 14% vs AdaptiveCF and MQ-RNN, and 18% vs DP-RNN. The gains are essentially unchanged from the Q1 results (i.e., 14%, 14%, and 19% in Q1, respectively). We report below each dataset’s selective risk for .
| Dataset | FAbFor | AdaptiveCF | DP-RNN | MQ-RNN |
|---|---|---|---|---|
| chlorine | .0048 ± .0003 | .0057 ± .0005 | .0066 ± .0006 | .0056 ± .0006 |
| ERA5 | .0045 ± .0008 | .0065 ± .0009 | .0135 ± .0016 | .0066 ± .0013 |
| handoutlines | .0045 ± .0023 | .0054 ± .0022 | .0070 ± .0028 | .0124 ± .0114 |
| inlineskate | .0031 ± .0006 | .0036 ± .0020 | .0035 ± .0004 | .0049 ± .0019 |
| freezer | .0026 ± .0027 | .0117 ± .0164 | .0294 ± .0136 | .0038 ± .0038 |
| wafer | .0002 ± .0001 | .0027 ± .0047 | .0099 ± .0037 | .0009 ± .0005 |
| ecg | .0591 ± .0046 | .0833 ± .0126 | .1307 ± .0165 | .0794 ± .0102 |
| ITpower | .0715 ± .0154 | .0942 ± .0217 | .0880 ± .0147 | .0918 ± .0189 |
| melbourne | .0333 ± .0078 | .0857 ± .0829 | .0752 ± .0225 | .0640 ± .0270 |
| mixedshapes | .0288 ± .0035 | .0400 ± .0062 | .0534 ± .0058 | .0423 ± .0101 |
| symbols | .0259 ± .0135 | .0363 ± .0109 | .0467 ± .0148 | .0452 ± .0119 |
| yoga | .0192 ± .0040 | .0221 ± .0021 | .0324 ± .0057 | .0244 ± .0049 |
| covid | .2782 ± .1865 | .2965 ± .0945 | .2520 ± .0564 | .3934 ± .1361 |
| eeg | .1530 ± .0045 | .1702 ± .0188 | .2200 ± .0255 | .1851 ± .0215 |
| fiftywords | .2765 ± .1283 | .4074 ± .0705 | .3332 ± .0917 | .4241 ± .1259 |
| wordrecognition | .1426 ± .0239 | .1552 ± .0238 | .2017 ± .0179 | .1656 ± .0273 |
| middlephalanx | .1167 ± .0247 | .1409 ± .0321 | .1677 ± .0237 | .1613 ± .0321 |
| swedishleaf | .1636 ± .0469 | .1552 ± .0303 | .2529 ± .0342 | .1514 ± .0212 |
| faceall | .3987 ± .0436 | .4782 ± .0669 | .4985 ± .0248 | .4732 ± .0521 |
| fetalecg | .3933 ± .0906 | .4743 ± .1128 | .4918 ± .0941 | .4945 ± .1245 |
| ford | .4843 ± .0162 | .4895 ± .0234 | .5757 ± .0159 | .4738 ± .0335 |
| refrigeration | .5492 ± .0609 | .7506 ± .1131 | .8311 ± .0901 | .7299 ± .1092 |
| sonyaiborobot | .4136 ± .0351 | .4568 ± .0352 | .4860 ± .0621 | .4648 ± .0343 |
| temperature | .6326 ± .1106 | .7573 ± .2946 | .9299 ± .4246 | .8066 ± .3589 |
[1] Hendrickx, Kilian, et al. "Machine learning with a reject option: A survey." Machine Learning 113.5 (2024): 3073-3110.
[2] Ruggieri, Salvatore, and Andrea Pugnana. "Things machine learning models know that they don’t know." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 27. 2025.
Dear reviewer u8be,
Thanks again for the feedback. As the author-discussion phase comes to an end, please let us know whether we have addressed your concerns or if there are any remaining points we can clarify.
We are glad that the Reviewer highlighted the sound practical motivation and significant research value of our paper. We address the reviewer’s comments in the following.
Jumps in empirical coverage. We thank the reviewer for spotting this imprecision. In practice, the thresholds are obtained via binary search (Algorithms 1-2, Appendix D), which keeps ϕ^(g^γ^ℓ)≤cH≤ϕ^(g^γ^r), so the randomization always reaches the target coverage. We have revised Eq. (9) to define γ^ℓ,γ^r as the bracketing thresholds returned by the binary search. The method and results are unchanged.
Conditional variance vs conditional expected squared error. We thank the reviewer for raising this point. The second head estimates the conditional expected squared error of the current predictor, not the conditional variance of the data. This is exactly the conditional risk ρt required by Theorems 1-3:
=\mathbb E\bigl[(Y_t-\hat f_t(y_{1:T}))^2\mid Y_{1:T}=y_{1:T}\bigr] =\operatorname{Var}(Y_t\mid Y_{1:T}=y_{1:T}) +\bigl(\mathbb E[Y_t\mid Y_{1:T}=y_{1:T}]-\hat f_t(y_{1:T})\bigr)^2,$$ which reduces to the conditional variance only when $\hat{f}_t$ equals the true conditional mean. For a fixed $\hat{f}$, the expected gradient of Eq. (8) with respect to $\hat{\sigma}^2_t$ vanishes exactly at $\hat{\sigma}^2_t = \rho_t$; the stop-gradient weight only rescales it, and no Gaussian assumption is needed. We have revised Section 3 to better describe the second head as a conditional-risk estimator. **Experiments with the same point-wise predictors.** We thank the reviewer for the question. In section 4.2, the baselines and our methods do not use the same point predictions. Following the reviewer's suggestion, we ran an additional ablation in which all baselines use the same point predictor as FAbFor, and the methods differ only in the scores used for abstention. We report the results in the new Appendix F.11. Even in this setting, FAbFor outperforms the baselines, reducing the selective risk on average by 10% vs AdaptiveCF, by 13% vs MQ-RNN, and more than 20% vs DP-RNN. This confirms the effectiveness of our abstention strategy when all methods use the same forecaster.