acceptodds
Under review as a conference paper at ICLR 2027

Selective Prediction Intervals: Jointly Learning to Predict, Bound, and Abstain

Abstract

Many high-stakes regression decisions ask three things of a model at once: a point estimate to act on, an interval that bounds the risk of acting, and the option to abstain when neither can be trusted. Deep learning provides these in isolation: prediction-interval methods return an interval, and at best a value estimate inside it, for every input, while selective-prediction methods learn when to abstain, but only for point predictions and with no statement of how far the truth may lie from the estimate. We introduce SelPI, which trains interval, value and abstention heads jointly under a single loss with an explicit acceptance-rate budget. A natural ob jection is that interval width already tells us when to abstain. We show that it does not: across 18 tabular benchmarks, width predicts large point errors well (AUROC 0.72) but is anti-correlated with interval misses (AUROC 0.38), so abstaining on wide intervals discards precisely the inputs the interval got right. SelPI therefore selects with two gates, one trained on the interval loss and one on the value loss, coupled by a Fr´echet penalty that ties their separate budgets to the joint acceptance rate they deliver. With the base predictor held fixed, the learned gate beats width thresholding on a proper interval score on all 18 tabular datasets, closing a median 87% of the gap to an oracle selector, and it also beats selection by deep-ensemble disagreement on every dataset. The advantage grows with the heteroscedasticity of the task, and we report the settings in which it disappears.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
prediction intervals, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cSa1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors of this paper propose SelPI. This method jointly learns to predict both prediction intervals and a point estimate for the given regression task, and also an abstention mechanism.This is done via jointly optimising a dual-gate mechanism, one targeting interval miscoverage and the other point-prediction errors. A main argument in the paper is that they argue that using a method such as IPIV and simply applying a post-hoc abstention gate based upon the width of prediction intervals alone may fail to capture epistemic uncertainty. The authors establish an evaluation protocol across 18 tabular and 2 image datasets. The authors show that SelPI outperforms width-based abstention on selective interval scores, while comparisons against SelectiveNet as a selective regressor are more mixed.

Critical strengths
  1. I think that the paper's evaluation is well-designed, thorough and justifiable, especially given the absence of a directly comparable method. The resulting conclusions drawn are then generally appropriate and reasonably convincing evidence for the paper's central claims.

  2. The paper provides useful insights into the limitations of interval width-based abstention, showing how this approach can be particularly poor at identifying interval miscoverage.

Critical weaknesses
  1. IPIV and SelectiveNet already employ somewhat similar optimisation strategies by combining prediction losses with constraints. Their integration into SelPI is therefore relatively straightforward and natural. However, this means that the resulting methodology is somewhat incremental in design, and the novelty of the approach is low. In saying that, this doesn't mean that the paper doesn't offer secondary insight into how methods such as a post-hoc add-on to IPIV are limiting. This secondary storyline helps to bolster the impact of the paper. Does this then leave us with a contribution that is sufficiently substantial for ICLR?

  2. The paper states in the Introduction "Width is an aleatoric signal, largely blind to epistemic failure". Given that I think this exploration into width-based abstention is a strength of this paper, this specific claim is one that I think is not sufficiently justified within the paper's main findings. The authors provide convincing evidence that interval width alone is not sufficient for reliably identifying interval miscoverage, but this does not necessarily demonstrate that width is largely blind to epistemic uncertainty. Prediction interval widths can reflect epistemic uncertainty depending on how they are constructed. Although the authors do compare against deep-ensemble disagreement, this does not directly demonstrate that their learned gates capture epistemic uncertainty either. I think a more direct investigation under controlled distribution shift or varying levels of training data would help to support this interpretation. Otherwise, I think the claim should be made more cautiously.

Questions for authors

Q1. (Relating to W2): How do the authors distinguish between improved identification of prediction failures and improved identification of epistemic uncertainty?

Q2. You provide many ablations for different methodological aspects of the method, but I could not find any relating to the necessity of the Frechet penalty beyond the individual date constraints. Could the authors provide an ablation demonstrating the necessity of the Frechet penalty, particularly its effect on the joint acceptance rate and predictive performance?

Q3. (Relating to W1): The authors provide several ablations for different methodological aspects of SelPI, but I could not find one examining the necessity of the Frechet penalty beyond the individual gate constraints. Could the authors provide such an ablation that singles out the effect of this part, particularly examining its effect on the joint acceptance rate and predictive performance?

Q4. The authors acknowledge the SelPI and SelectiveNet evaluation setups (such as a different backbone for SelectiveNet). However could the authors comment further on how much these differences in experimental configuration might influence the results, and whether a more consistent setup might change the overall findings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cSa1

We thank the reviewer for a careful and useful review. Q2 and Q3 ask for the same Fréchet ablation, so the questions come down to three requests. Each led to new experiments. Based on these experiments, we revised some of the claims made in the paper. New results are in the revised PDF; code is at https://anonymous.4open.science/r/selpi-code-EBCF/.

W1. Is the contribution sufficient? We agree that the architecture is a natural combination, and we now say so in §1. The combination was not mechanical, however. It required design choices that neither parent method supplies: gate-weighted coverage and width statistics with an effective-sample-size scaling, an auxiliary head triple that keeps interval-relevant features alive for rejected inputs, and a two-gate mechanism in which each gate has its own loss and its own budget. The contributions we claim (revised after the requested experiments) are:

  1. The finding the reviewer lists as a strength. Width-based abstention fails on interval networks under a proper score. It now holds against four more selector families (kNN variance, two-head variance, deep ensemble, conformal width), under three more proper scores, and under a held-out-region shift.
  2. A single-model selective interval predictor. Its learned gate matches the best post-hoc estimator of the conditional interval loss (requested by Reviewer AUid), with no second model.
  3. The evaluation protocol: a same-predictor comparison with an oracle-normalized selection efficiency.
  4. The two failure modes created by the interval–point coupling, and gates that specialize to them.

The revised paper also reports what did not hold (the Fréchet penalty, the advantage over SelectiveNet, the epistemic reading). We think a precisely scoped account is more useful than the broader claims of the submission.

W2 / Q1. Width and epistemic uncertainty. The reviewer is right, and we withdrew the sentence. The submission did not separate failure detection from epistemic detection, so we ran the controlled experiments the reviewer suggested (App. U).

(a) Synthetic task with an unseen region. One quadrant is removed from training and compared with a seen region of the same noise level. AUROC for separating unseen from seen points:

Ensemble disagreementEnsemble total varianceSelPI single gateTwo-head varianceWidthSelPI dual gatekNN variance
0.870.610.530.460.400.380.37

Only ensemble disagreement singles out the unseen region. The gates do not.

(b) Training-size sweep (six datasets, φ = 0.7). Share of the oracle gap that the gate closes over width, on the interval score:

Training data used10%25%50%100%
Median over datasets0.630.720.720.82

The gate's advantage over width grows with data. If it came from detecting epistemic errors, which shrink as data grows, it would shrink instead. The kNN and two-head selectors also improve with data (all six datasets, 10% to 100%), and on three of the six datasets (concrete, superconduct, power-plant) the gate's curve is not monotone.

(c) Held-out region on six real datasets. One k-means cluster is removed from training and validation, and every selector keeps its threshold. Median change in the rejection rate on the held-out cluster: gate +0.34, kNN variance +0.18, width −0.27. The gate reacts strongly on three datasets (+0.59 to +0.66) and hardly at all on the other three. Width moves the wrong way on five of six: it becomes more confident in the unfamiliar region.

Answer to Q1. The gates are trained to rank prediction failures, and that is what the new selector table (Table 3) and the gate diagnostics (App. R) measure. They are not estimators of epistemic uncertainty: (a) shows that they miss a purely unfamiliar region, (b) shows that their advantage does not come from data scarcity, and (c) shows a partial, dataset-dependent reaction to shift. The revised claim (§1, §7) is the one the data support: width does not bound coverage failure, identifies misses no better than chance on these tasks, and becomes more confident under shift on five of six datasets. We make no claim that the gates capture epistemic uncertainty.

AuthorsOpenReview ↗

Response to Reviewer cSa1(part 2)

Q2 / Q3. Ablation of the Fréchet penalty. We ran it on all 18 datasets (dual gate, tuned weight vs. weight 0, same seeds, splits 0–2), with a weight sweep on six datasets (App. S).

Change when the penalty is removedMedian over datasets (range)p
Selective RMSE+0.2% (−10% to +12%)1.00
Selective RMSE at matched acceptance+0.4% (−10% to +9%)0.83
Selective interval score+1.5% (−15% to +23%)0.67
Joint acceptance rate−0.23 points (−3.6 to +3.9)0.39

The penalty has no measurable effect, and the reason is structural. Since (1−gPI)(1−gval)≥0(1-g_{\mathrm{PI}})(1-g_{\mathrm{val}}) \ge 0, the soft joint acceptance of any two gates is at least ϕ^(gPI)+ϕ^(gval)−1\hat\phi(g_{\mathrm{PI}}) + \hat\phi(g_{\mathrm{val}}) - 1. Whenever each gate meets its own budget, which its own penalty enforces, the Fréchet term is exactly zero. It was inactive at the end of training in all 104 models (52 with the tuned weight; for the 52 with weight 0 we evaluate the penalty it would have added). In a stress test with both budgets at 0.95, where the gates can no longer meet their budgets, it became active on 3 of 6 datasets, and its effects on joint acceptance and error were small and inconsistent in sign (six datasets, one run per setting).

Consequence. The sentence "without it ... the intersection collapses well below either budget" was not supported and is removed. §4.3 now presents the Fréchet term as an optional safeguard that engages only when the per-gate budgets cannot be met. The dual-gate results do not depend on it, so its weight need not be tuned.

Q4. SelectiveNet setup. The reviewer's concern was justified. We retrained SelectiveNet on SelPI's backbone (same width, epochs, learning rate, decay and batch size; its own selection head) on all 18 datasets and five splits, and ran a small tuning grid for it on six datasets (App. T).

Selective RMSE at φ = 0.7SelPI better / level / worseMedian difference (+: SelPI better)p
Submission: SelectiveNet with its own backbone14 / 1 / 3+4.5% (SelPI better)0.031
Same backbone, at SelPI's realized acceptance rate6 / 1 / 11−2.9%0.23
Same backbone, SelectiveNet trained for φ = 0.75 / 2 / 11−1.9%0.20

With the backbone matched, most of the advantage reported in the submission disappears; the Yacht margin falls from 78% to 29%. The two methods are not significantly different at φ = 0.7. Over the whole acceptance range, SelectiveNet is ahead of the single-gate SelPI on 14 of 18 datasets (p = 0.016). Tuning SelectiveNet changes its error by at most about 6% and does not change the winner on any of the six datasets.

A second asymmetry is now stated in §5 and App. D. The search that set the dual gate's budgets and loss weights scored its trials by selective RMSE on the test portions of splits 0-2, which the evaluation also uses. We report the median trial rather than the best to limit this, but it remains an advantage that SelectiveNet did not have, so the comparison above is, if anything, tilted toward SelPI. The single-gate model behind all same-predictor results (the paper's Q3 and the new selector table) is trained with a fixed per-dataset configuration and no search over budgets or loss weights, so this asymmetry does not apply to it.

Consequence. The paper's Q2 (§6) is rewritten on the same-backbone comparison. We no longer claim that SelPI beats SelectiveNet in selective point error. The supported statement is that SelPI adds a prediction interval and interval-aware abstention at a small cost in point error relative to a dedicated selective regressor of the same capacity (median 2–3% at φ = 0.7, not significant). The main result of the paper (its Q3: learned gate vs. width on a fixed predictor) does not involve SelectiveNet and is unchanged.

We thank the reviewer again. These questions made the paper more accurate. We will be happy any additional questions to the best of our abilities.

I thank the authors for their comments and the additional experiments. After considering these responses alongside the concerns raised by reviewers AUid and ic51, I feel that several of my original concerns regarding the novelty and overall contribution in the paper still remain relevant. While I appreciate that the authors have addressed the technical questions, some of the new findings also very much narrow the original claims of the paper and is therefore difficult to assess in the same way as in my original review.

Due to the fact that I was already borderline in my assessment, I now lean towards a weak rejection. I think that the work offers useful insights, but would benefit from a more substantial revision that better establishes and positions its main contributions before resubmission.

Reviewer_ic51OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses a combination of learning a point estimate, an upper and a lower bound (Simhayev 2022) and two predictor gates by using the risk prediction of (Geifman&El-Yaniv 2019) for these values instead of the target.

Width of the interval is considered aleatoric while the point estimation error is considered epistemic. I am not fully certain, I understand that separating aleatoric and epistemic error is a very difficult task beyond the scope of the paper. Still in my understanding aleatoric and epistemic error is mixed in both width and point error, although as analogy the discussion can be considered OK.

I have some questions regarding tabular foundation models as relevant methods, and proper scoring rules, as potential evaluation metrics to consider, see Question.

Critical strengths

In high stakes decision making, point prediction alone is insufficient. The paper addresses and solves in part important questions regarding learning the uncertainty of a prediction.

Critical weaknesses

W1 Tabular foundation models, in particular TabPFN, which is considered state of the art, is not mentioned, should be included as comparison or backbone

W2 limited novelty, simple combination of learning U, L, y (Simhayev 2022) and two predictor gates that output a risk by using the risk prediction of (Geifman&El-Yaniv 2019). Compared to (Geifman&El-Yaniv 2019), risk prediction is applied also for the interval and accepted if both the value and the interval is accepted, joint training, with acceptance budget.

Questions for authors

The experiments use only RMSE for point prediction evaluation. I would like to see proper scoring rules, e.g. log score, CRPS, Brier score. Also I understand that the paper addresses prediction intervals only, but bi/multimodal distributions should be mentioned as limitation or possible consideration, eg by using proper scoring rules.

How do tabular foundation models perform? See weaknesses.

I had a bit of discomfort understanding ϕ, lots of subscripted and with-hat variants. ϕ is the key parameter in the experiments, I would like to see a very clear definition.

Also, while I see no major issues in reproducing the method on a high level, direct comparison will not be possible in the future without source code. Please provide already for the review eg as an anonymous repo if you can no longer attach to the submission.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ic51

We thank the reviewer for the constructive review. All points are addressed below. New results are in the revised PDF; code is at https://anonymous.4open.science/r/selpi-code-EBCF/.

W1. Tabular foundation models (TabPFN). We ran TabPFN-3.5 (package tabpfn 9.2.0, checkpoint tabpfn-v3.5-20260909; defaults, no tuning, no fine-tuning) on all 18 tabular datasets, with the same splits and standardization as every other model. The context is the full training split, or a random 50,000 rows on the five datasets that exceed that size. The interval is the 2.5%–97.5% quantile range and the point prediction is the median (App. V).

As a predictor, TabPFN is clearly stronger than our base model (IPIV). With every point accepted, its interval score is lower on 18/18 datasets (median 44% lower) and its RMSE on 16/18 (median 30% lower). Its coverage is closer to the 95% target (median 0.956 vs. 0.941). With a simple width threshold it also has a lower selective interval score than SelPI across acceptance rates (median 55% lower than the single gate and 46% lower than the dual gate, 18/18 each). We state this plainly in the revised paper (§6, §8).

What this means for selection.

  • On TabPFN, width is a reasonable selector, and post-hoc loss regressors fitted on top of TabPFN do not improve on it (they are worse on 17–18 of 18 datasets).
  • On our interval network the opposite holds: width identifies misses no better than chance (AUROC 0.45), and a learned selector closes a median 88% of the gap to the oracle. Relative to its own predictor's loss, the gate removes about twice as much interval-score loss as TabPFN's width rule (0.62 vs. 0.31; 18/18 datasets). The two predictors start from different loss levels, so this compares the relative effect of the selectors, not the two pipelines. The comparison also favours SelPI: its reference is the full-acceptance score of a model trained to reject part of the data, and plain IPIV, trained without rejection, has a median 11% lower full-acceptance interval score than that model (lower on 16 of 18 datasets).
  • We therefore state the scope of our finding explicitly (§1, §8): the failure of width-based abstention is shown for interval networks trained with a width–coverage objective. We do not claim it for TabPFN's quantile intervals.

TabPFN as a backbone. TabPFN predicts in context with fixed weights, so a jointly trained gate requires fine-tuning it per dataset. That is a new method and beyond the rebuttal period; we list it as the natural next step (§8). SelPI's selection heads are architecture-agnostic. They remain relevant for image regression (two of our 20 tasks), which TabPFN does not address, and where inference cost matters (TabPFN: a median of 21 s per 1,000 test rows on our GPU; SelPI: one forward pass of a small network).

W2. Limited novelty. We agree that the architecture is a direct combination of IPIV's heads and SelectiveNet-style gates. We believe it was clear in the original submission, and we now say so explicitly in §1. The contribution we claim is what the combination reveals and how it is evaluated:

  1. Interval width, the obvious abstention signal for an interval model, identifies misses no better than chance and loses to a learned selector on 18/18 datasets under four proper scores.
  2. The same holds against kNN-variance, two-head, deep-ensemble and conformal selectors (new table in §6, requested by Reviewer AUid).
  3. The coupling between the interval and the point estimate produces two different failure modes, and the two gates specialize to them (Table 11).
  4. A same-predictor protocol with an oracle-normalized selection efficiency, which separates the quality of the selector from the quality of the base predictor.

Several claims of the submission were narrowed after the new experiments; the general response lists them.

AuthorsOpenReview ↗

Response to Reviewer ic51 (part 2)

Proper scoring rules. The headline metric of the paper, the interval score (eq. 12), is a strictly proper scoring rule for interval forecasts (Gneiting & Raftery, 2007). RMSE is used only for the point estimate. We now say this more prominently (§5), and we added further scores (App. W). Gate vs. width on the same predictor, 18 datasets, splits 0-4:

Score on the accepted setShare of oracle gap, median [IQR]Gate better / level / worsep
Pinball loss, lower bound (τ = 0.025)0.86 [0.71, 0.93]18 / 0 / 0<0.001
Pinball loss, upper bound (τ = 0.975)0.88 [0.81, 0.97]18 / 0 / 0<0.001
CRPS (two-piece normal fitted to L, ŷ, U)0.73 [0.57, 0.90]18 / 0 / 0<0.001
Interval score (reference)0.88 [0.81, 0.97]18 / 0 / 0<0.001

The conclusion is the same under every score, and the per-dataset winner agrees with the interval score on every dataset. Three notes:

  • The sum of the two pinball losses equals α/2\alpha/2 times the interval score, so it carries the same information.
  • SelPI outputs an interval and a point, not a density. Its CRPS therefore needs a stated convention, and a log score is not defined for it.
  • The Brier score applies to categorical outcomes, not to a regression target.

For the models that do output a distribution (the deep ensemble and TabPFN) we report native CRPS and log scores per dataset (App. W). TabPFN is the best of these on all 18 datasets.

Bi- and multimodal targets. We agree. An interval with one interior point estimate cannot represent a multimodal conditional distribution: the interval becomes wide and the point estimate may fall between modes. This is now stated in the limitations (§8), with mixture or quantile-grid heads scored by CRPS as the extension.

Aleatoric vs. epistemic. We agree that both are mixed in the width and in the point error, and we removed the sentence that framed width as aleatoric. We tested the question directly with a synthetic task, a training-size sweep and a held-out-region shift (App. U). In short, the gates are trained to rank failures and react only partly to unfamiliar inputs. The details are in our response to Reviewer cSa1 (part 1/2).

Definition of φ. We added a notation box (§3) and now use four symbols consistently:

  • ϕt\phi_t: the target acceptance rate given to a gate during training (ϕtPI\phi_t^{\mathrm{PI}} and ϕtval\phi_t^{\mathrm{val}} for the two gates of the dual model).
  • ϕ^(g)=1n∑ig(xi)\hat\phi(g) = \frac{1}{n}\sum_i g(x_i): the soft acceptance rate of a gate on a training batch (eq. 5).
  • ϕ^joint\hat\phi_{\mathrm{joint}}: the soft joint acceptance rate of the two gates (eq. 9).
  • ϕ\phi (no hat, no subscript): the fraction of test inputs actually accepted, after the thresholds are set on validation data. This is the x-axis of every curve and the acceptance level in every table.

Source code. The anonymous repository is at https://anonymous.4open.science/r/selpi-code-EBCF/. It contains the training code for SelPI and all baselines, the evaluation scripts, and the per-dataset configurations.

We hope these additions answer the reviewer's questions, and we are glad to provide anything further during the discussion.

Reviewer_ic51OpenReview ↗

The paper needs a major revision in a new submission

I would like to thank the authors' effort to answer all the reviewer questions. Overall, I think there is plenty of room to improve the paper beyond the limits of the rebuttal period and I recommend submission to a new conference after major revision, restructuring the goals, maybe further extending the experiments, and also making the text feel less ai-written.

I appreciate the answer to TabPFN, it is a very strong model and very difficult to position similar results against. A very careful analysis might be needed to position the paper - I understand the values and the theoretical contribution that is worth going through more evaluation.

Overall, I maintain my score.

Reviewer_AUidOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes SelPI, a selective prediction approach based on the width of uncertainty intervals and whether that interval contains the actual prediction. The proposed method improves over SelectiveNet across a quite extensive evaluation.

Critical strengths

The main strengths I see is that the method is reasonable, and the intuition behind the width of the intervals not working makes sense.

Critical weaknesses

I have several concerns:

  • Novelty. I am not sure the method is actually well positioned in the literature. For instance, Feng et al. [1] jointly learn prediction sets and a selection function, including intervals. Johansson et al. [2] study conformal regression with a reject option. I think both methods are worth discussing, and I would highlight how SelPI differs from them.

  • Method justification. I have a few concerns regarding the justification of the approach. Franc et al. [3] show that the optimal rejection strategy thresholds the conditional risk, or a score that preserves its ordering. Moreover, Zaoui et al. (2020) establish the corresponding conditional-variance rule for squared-error regression with the optimal predictor. Thus, for a given predictor, the quantity to estimate is its conditional expected loss. I do not see why the proposed method should yield a better ranking than directly estimating the conditional loss under the evaluation metric.

  • Experimental Evaluation. I have several concerns regarding the evaluation of the approach. I list here some missing baselines that I think are mandatory to assess the effectiveness of the approach.

    • simple plug-in strategy. (e.g., Zaoui et al., 2020). The simplest strategy that thresholds a conditional variance is missing. Zaoui et al., 2020 propose to use a K-NN for that purpose.
    • two-head neural network. One can estimate a conditional variance by using a two-head neural network with a frozen backbone, where one head estimates the prediction and one head estimates the conditional variance. Such a model can be trained using Gaussian NLL, as in the original deep-ensembles paper, or β−NLL\beta-NLL, as in Stradiotti et al. [4] (adapt it to a single time step).
    • proper deep ensemble. From my understanding, the way uncertainty is estimated for deep ensembles is lacking some parts. Indeed, one can use the members to learn better estimates of conditional variances by training each member using the β−NLL\beta-NLL loss, and then average their prediction. This is also suggested in the original deep ensembles paper.
    • some conformal prediction proxy. I also suggest adding an abstention method with an interval width, which simply thresholds the width of the conformal set (in the regression case, an interval).
  • Presentation. While the first part is simple to read, the empirical evaluation is really confusing. For example, Table 1 is really difficult to read: the caption is hard to interpret, and the placement of text across columns makes the rows difficult to parse.

  • Reproducibility. No code is provided.

[1] - Feng, J., Sondhi, A., Perry, J., & Simon, N. (2023). Selective prediction‐set models with coverage rate guarantees. Biometrics, 79(2), 811-825.

[2] - Johansson, U., Sönströd, C., & Boström, H. (2024). Conformal Regression with Reject Option. COPA, 230, 1-18.

[3] - Franc, V., Prusa, D., & Voracek, V. (2023). Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11), 1-49.

[4] - Stradiotti, L., Devos, L., Monreale, A., Davis, J., & Pugnana, A. (2026). Bounded-Abstention Multi-horizon Time-series Forecasting. arXiv preprint arXiv:2602.04714.

Questions for authors

Please address the weaknesses above

4 comments
AuthorsOpenReview ↗

Response to Reviewer AUid

We thank the reviewer for a detailed review and for the specific references and baselines. They changed the paper. We answer every point below. New results are in the revised PDF; code is at https://anonymous.4open.science/r/selpi-code-EBCF/.

On the summary. The description is close to our training loss, and we may not have been clear about the selection rule. Width and capture do enter SelPI's objective: the selective interval loss minimizes the width of the captured intervals under a coverage penalty, and the selective value loss weights the squared error (§4.2). What SelPI thresholds at test time is the learned gate output, not the width. In the submission the gates' rank correlation with width is moderate (per-dataset means from -0.77 to +0.21, §7). Thresholding the width is the baseline the paper argues against.

Novelty and positioning. Thank you for both references. They are now discussed in §2 and App. A.

  • Feng et al. (2023) train a selective prediction-set model by penalized loss minimization with an abstention cost, and give an inference procedure for the coverage of the accepted set. This is the closest prior work. SelPI differs in three ways. (i) It uses an explicit acceptance budget and is evaluated along the whole risk–coverage curve. (ii) It outputs a point estimate inside the interval; we show that this creates two distinct failure modes and study a gate for each. (iii) Its main empirical message is about what not to use as the selector, namely interval width. Their coverage inference is complementary to our work and would address the limitation we state in §8 (no finite-sample guarantee on the accepted set).
  • Johansson et al. (2024) add a reject option to conformal regression: Mondrian categories are created dynamically from a difficulty estimate of each instance and the requested rejection level, and the method is evaluated with random forests as the underlying model. The selector is post hoc and the non-rejected intervals are valid. SelPI learns the selector with the predictor and carries no such guarantee. Our new conformal baseline (next comment) belongs to this family.
  • We also cite Stradiotti et al. (2026) on optimal abstention strategies in forecasting.

Method justification (Franc et al.; Zaoui et al.). We agree with the reviewer's argument and now state it in §4.2 (new Remark 1).

  1. The gate is an instance of conditional-risk thresholding. For a fixed predictor and a per-sample loss, the minimizer of the gate-weighted risk under an acceptance budget (eq. 2) is the indicator of a sublevel set of the conditional expected loss. This is Chow's rule written as an optimization problem (new Remark 1; proof in App. R). The gate is therefore a parametric estimator of the ordering of the conditional loss, trained together with the predictor. It is not an alternative to that rule. One caveat: our training objective is the IPIV interval loss (eqs. 6-7: width of the captured intervals plus a coverage penalty), not the interval score, so Remark 1 describes an idealized version. That the gate matches a regressor fitted directly to the interval score (next comment) is the empirical evidence that the two orderings are close.
  2. Which loss matters. Conditional variance is the right score for squared error with the optimal predictor (Zaoui et al.). Our evaluation metric is the interval score. Its conditional expectation is the width plus 2/α2/\alpha times the expected distance by which yy falls outside the interval, so it depends on where the interval is likely to miss. Width identifies misses no better than chance (AUROC 0.45). A variance or width proxy therefore targets a different quantity from the one being scored.
  3. The reviewer's question, answered directly. "I do not see why the proposed method should yield a better ranking than directly estimating the conditional loss under the evaluation metric." The experiment this question calls for settles it. We ran that estimator (next comment) and it matches the gate, so we adjusted our claims accordingly: the gate reaches the quality of a direct conditional-loss estimator within a single model, and we no longer claim that it ranks better. The experiments support the reviewer's own framework: selectors that target the conditional loss under the evaluation metric (the gate, a loss regressor, a miss classifier) perform alike, and all of them clearly beat width and the two-head, deep-ensemble and conformal proxies; the kNN conditional variance of Zaoui et al. comes close (0.86 vs. 0.88).
AuthorsOpenReview ↗

Response to Reviewer AUid (part 2)

Protocol. Every selector below ranks the test points of the same trained SelPI model (single gate), with its threshold set on validation data. That model's predictor was trained jointly with its gate. The two-head variance model and the MLP regressors use its frozen backbone features; the kNN rule and the tree-based loss regressor and miss classifier use the inputs and the model's realized training losses; the ensemble and CQR scores come from separately trained models. "Share of oracle gap" is the selection efficiency of the paper (0 = width threshold, 1 = oracle), computed on the selective interval score. AUROC measures how well a score identifies intervals that miss yy. 18 datasets, splits 0–2.

SelectorShare of oracle gap, median [IQR]AUROC (misses)Gate better / level / worsep
SelPI learned gate0.88 [0.81, 0.97]0.79––
Loss regressor, fitted on training losses0.89 [0.81, 0.97]0.837 / 6 / 51.00
Miss classifier, fitted on training losses0.88 [0.82, 0.98]0.865 / 4 / 90.11
kNN conditional variance (Zaoui et al.)0.86 [0.69, 0.95]0.8113 / 2 / 3<0.001
Loss regressor, fitted on held-out data0.79 [0.59, 0.96]0.7315 / 2 / 1<0.001
Deep ensemble, total variance0.70 [0.54, 0.88]0.7516 / 2 / 0<0.001
Two-head variance, frozen backbone0.65 [0.55, 0.81]0.7218 / 0 / 0<0.001
Deep ensemble, member disagreement0.62 [0.42, 0.80]0.7118 / 0 / 0<0.001
Conformal (CQR) width0.58 [0.41, 0.79]0.6918 / 0 / 0<0.001
Interval width0 (reference)0.4518 / 0 / 0<0.001

Wilcoxon signed-rank test over datasets; "level" means a difference below 0.5%. Where a selector family has several variants (NLL or beta-NLL, gradient boosting or MLP, kNN target and feature space), the table shows the variant with the highest median share of the oracle gap on the interval score, which favours the baselines. Per-dataset values are in App. R.

The four requested baselines

  • kNN plug-in (Zaoui et al.). The variance of the predictor's errors among the kk nearest training neighbors. It is the strongest of the four; the gate is still better on 13 of 18 datasets.
  • Two-head network. Mean and variance heads on the frozen backbone, trained with the Gaussian NLL (reported) and, as a variant, the β-NLL with β = 0.5 (median share of the oracle gap 0.62, against 0.65 for the NLL).
  • Proper deep ensemble. Five members, each with a mean and a variance head trained by NLL, averaged as a mixture, as in the original deep-ensembles paper (a β-NLL ensemble is slightly lower: 0.67 for total variance). We rank by its total predictive variance and, separately, by the disagreement among members.
  • Conformal width. Conformalized quantile regression (CQR) with the same architecture and training settings; we threshold the width of the conformal interval. Note that split-conformal calibration of the model's own interval adds the same constant to every interval, so it leaves the width ranking, and hence the last row, unchanged.

The direct conditional-loss estimator. Beyond the four baselines, we added the estimator that the reviewer's argument points to: a second model fitted to the predictor's realized interval score, and a classifier for the miss event. Fitted on the predictor's training-set losses, both match the gate (differences not significant); the miss classifier has the higher AUROC for misses (0.86 vs. 0.79). Fitted on held-out losses (half of the validation split), the regressor is clearly worse, which we attribute to the smaller sample.

Our revised claims

  1. Estimating the conditional loss under the evaluation metric is what matters. Width and the two-head, deep-ensemble and conformal proxies fall well short of it under the interval score; kNN variance comes close (0.86 vs. 0.88).
  2. A jointly trained gate reaches the quality of the best post-hoc estimator of that loss with one model, one training run and one forward pass.
  3. We withdrew the statement that joint training is essential and revised the ablation text accordingly (§7). The post-hoc estimators here rank a predictor that was itself trained jointly with a gate, so these results do not say how they would fare on a predictor trained without one.
AuthorsOpenReview ↗

Response to Reviewer AUid (part 3)

Presentation. We revised the empirical part to make it easier to follow. In the revised version:

  • Table 1 is replaced by three short paragraphs, one per evaluation question, each naming its comparator and its metric.
  • Table 2 is split into two tables with one-sentence captions (now Tables 1 and 2): point error (Q2) and interval score (Q3).
  • The selector table of the previous comment replaces Table 3, with one row per selector and the same columns throughout; the earlier gate-diagnostics table moves to App. R.
  • A notation box in §3 defines the acceptance rate φ and its variants once.

Reproducibility. The code is at https://anonymous.4open.science/r/selpi-code-EBCF/. It contains the training code for SelPI and every baseline, including those added in this rebuttal, the evaluation and aggregation scripts, and the per-dataset configurations.

Summary of what changed in response to this review

ConcernWhat we didWhere
Positioning vs. Feng et al., Johansson et al.Both discussed; differences stated§2, App. A
Method justificationGate shown to be an instance of conditional-risk thresholding; direct conditional-loss estimator added; claims narrowed to "matches"§4.2, §6, App. R
kNN plug-inAdded; gate better on 13 / level on 2 / worse on 3§6, App. R
Two-head networkAdded; gate better on 18 / 18§6, App. R
Proper deep ensembleAdded; gate better on 16–18 of 18§6, App. R
Conformal widthAdded (CQR); gate better on 18 / 18§6, App. R
PresentationTable 1 replaced, Table 2 split, new selector table, notation box§3, §5, §6
CodeAnonymous repositoryReproducibility statement

We hope that the new baselines and the revised justification address the soundness concerns. The reviewer's framework is now stated in §4.2 and organizes the selector comparison of §6, and the paper is better for it. We are glad to run further checks during the discussion period.

I thank the authors for their reply.

I have a few comments.

First, I have appreciated that now the authors clearly identified and formulated better the goal of SelPI. I think their theoretical analysis is valuable somehow.

Second, I still do not like too much how things are being presented in the paper. The writing style is too AI based for my personal taste (see e.g., Tables captions and contents or the large number of acronyms introduced in the paper). I encourage the authors to have a read at the work and try to make it more human friendly.

Third, it is a courtesy to the reader to highlight in a different colour which parts of text are changed with respect to the previous version. I kindly ask the authors to do this.

Finally, I am not sure I get why one should use SelPI in the end if the direct risk estimator works better (as one could have expected). If I get the authors point, SelPI provides directly the abstain/predict decision, while estimating the risk would require an additional model. Although I can see this can be an advantage, I also think such an advantage is not huge, and, more importantly, in high-stakes settings, I would value better performance more than the cost of having two models. Could the author discuss on this point please?

Overall, I appreciated the efforts and have increased the score to a borderline reject.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment from the Authors

We thank the three reviewers for careful and constructive reviews. We ran every experiment that was requested (about 2,800 training and analysis runs on the 18 tabular datasets) and uploaded a revised PDF. Code: https://anonymous.4open.science/r/selpi-code-EBCF/.

Some of the new results strengthen the paper and some narrow its claims. We report both, and the revised paper claims only what the new evidence supports.

What the new results confirm

  1. On our interval predictor, width is a poor abstention signal under every score we tried. With the predictor held fixed, the learned gate beats the width threshold on 18/18 datasets under the interval score, the pinball loss at each bound, and CRPS (all p < 0.001). Width identifies interval misses no better than chance (AUROC 0.45 in the re-trained models, 0.38 in the submission), against 0.79 for the gate.
  2. The gate beats all four baselines requested by Reviewer AUid on the same predictor: kNN conditional variance (gate better on 13 datasets, level on 2, worse on 3), two-head variance (18/0/0), a proper deep ensemble (16/2/0 by total variance, 18/0/0 by member disagreement) and conformal (CQR) width (18/0/0); p < 0.001 for each.
  3. The advantage is not a small-data artifact. The share of the oracle gap that the gate closes grows from 0.63 to 0.82 as the training set grows from 10% to 100% (six datasets).
  4. Under a held-out-region shift, width moves the wrong way. On 5 of 6 real datasets the width rule rejects fewer points in a region removed from training. The gate rejects more on the median dataset (+0.34 vs. −0.27), clearly so on three of the six.

What the new results narrow

  1. The gate matches, but does not beat, a direct estimate of the conditional interval loss (Reviewer AUid). A second model fitted to the predictor's realized interval score selects as well as the gate (share of oracle gap 0.89 vs. 0.88; gate better / level / worse on 7 / 6 / 5 datasets, p = 1.0), and a miss classifier is not significantly different (5 / 4 / 9, p = 0.11). We now claim that the gate reaches the quality of the best post-hoc conditional-loss estimator within one model and one forward pass. We no longer claim that joint training is essential.
  2. The Fréchet penalty is a safeguard, not a driver (Reviewer cSa1). It is inactive at the end of training in all 104 dual-gate models (for the 52 trained without it, we evaluate the penalty it would have added), and removing it changes selective RMSE by +0.2% (median, p = 1.0).
  3. On a matched backbone, SelPI and SelectiveNet are comparable in selective point error (Reviewer cSa1): SelPI is better on 6 datasets, level on 1 and worse on 11 (p = 0.23). We no longer claim an advantage in point error.
  4. The gates rank failures; they are not epistemic-uncertainty detectors (Reviewers cSa1 and ic51). In a synthetic test, only ensemble disagreement singles out an unseen region. We withdrew the sentence "width is an aleatoric signal, largely blind to epistemic failure".
  5. TabPFN is a stronger tabular interval predictor than our base model (Reviewer ic51). We report this in full and state the scope of our finding accordingly.

Changes in the revised PDF

  • Main text. Abstract and §1: contributions rewritten around the findings above. §2: Feng et al., Johansson et al., Franc et al., Stradiotti et al. and TabPFN added. §3: a notation box for φ. §4.2: new Remark 1 (the gate as conditional-risk thresholding). §4.3: the Fréchet penalty as an optional safeguard. §5: Table 1 replaced by three short paragraphs. §6: the per-dataset table split in two (Tables 1 and 2), a new selector table (Table 3; the earlier one moves to App. R), Q2 rewritten on the same-backbone comparison, a paragraph on TabPFN. §7: controlled shift experiments and updated ablations. §8: limitations.
  • Appendices. New: R (selectors on the same predictor; proof of Remark 1), S (Fréchet ablation), T (SelectiveNet on SelPI's backbone), U (failure detection vs. epistemic uncertainty), V (TabPFN), W (proper scoring rules). App. D now states that the dual-gate search scored its trials on the evaluation splits.

Details and tables are in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.