- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors of this paper propose SelPI. This method jointly learns to predict both prediction intervals and a point estimate for the given regression task, and also an abstention mechanism.This is done via jointly optimising a dual-gate mechanism, one targeting interval miscoverage and the other point-prediction errors. A main argument in the paper is that they argue that using a method such as IPIV and simply applying a post-hoc abstention gate based upon the width of prediction intervals alone may fail to capture epistemic uncertainty. The authors establish an evaluation protocol across 18 tabular and 2 image datasets. The authors show that SelPI outperforms width-based abstention on selective interval scores, while comparisons against SelectiveNet as a selective regressor are more mixed.
Critical strengths
-
I think that the paper's evaluation is well-designed, thorough and justifiable, especially given the absence of a directly comparable method. The resulting conclusions drawn are then generally appropriate and reasonably convincing evidence for the paper's central claims.
-
The paper provides useful insights into the limitations of interval width-based abstention, showing how this approach can be particularly poor at identifying interval miscoverage.
Critical weaknesses
-
IPIV and SelectiveNet already employ somewhat similar optimisation strategies by combining prediction losses with constraints. Their integration into SelPI is therefore relatively straightforward and natural. However, this means that the resulting methodology is somewhat incremental in design, and the novelty of the approach is low. In saying that, this doesn't mean that the paper doesn't offer secondary insight into how methods such as a post-hoc add-on to IPIV are limiting. This secondary storyline helps to bolster the impact of the paper. Does this then leave us with a contribution that is sufficiently substantial for ICLR?
-
The paper states in the Introduction "Width is an aleatoric signal, largely blind to epistemic failure". Given that I think this exploration into width-based abstention is a strength of this paper, this specific claim is one that I think is not sufficiently justified within the paper's main findings. The authors provide convincing evidence that interval width alone is not sufficient for reliably identifying interval miscoverage, but this does not necessarily demonstrate that width is largely blind to epistemic uncertainty. Prediction interval widths can reflect epistemic uncertainty depending on how they are constructed. Although the authors do compare against deep-ensemble disagreement, this does not directly demonstrate that their learned gates capture epistemic uncertainty either. I think a more direct investigation under controlled distribution shift or varying levels of training data would help to support this interpretation. Otherwise, I think the claim should be made more cautiously.
Questions for authors
Q1. (Relating to W2): How do the authors distinguish between improved identification of prediction failures and improved identification of epistemic uncertainty?
Q2. You provide many ablations for different methodological aspects of the method, but I could not find any relating to the necessity of the Frechet penalty beyond the individual date constraints. Could the authors provide an ablation demonstrating the necessity of the Frechet penalty, particularly its effect on the joint acceptance rate and predictive performance?
Q3. (Relating to W1): The authors provide several ablations for different methodological aspects of SelPI, but I could not find one examining the necessity of the Frechet penalty beyond the individual gate constraints. Could the authors provide such an ablation that singles out the effect of this part, particularly examining its effect on the joint acceptance rate and predictive performance?
Q4. The authors acknowledge the SelPI and SelectiveNet evaluation setups (such as a different backbone for SelectiveNet). However could the authors comment further on how much these differences in experimental configuration might influence the results, and whether a more consistent setup might change the overall findings?
3 comments
Response to Reviewer cSa1(part 2)
Q2 / Q3. Ablation of the Fréchet penalty. We ran it on all 18 datasets (dual gate, tuned weight vs. weight 0, same seeds, splits 0–2), with a weight sweep on six datasets (App. S).
| Change when the penalty is removed | Median over datasets (range) | p |
|---|---|---|
| Selective RMSE | +0.2% (−10% to +12%) | 1.00 |
| Selective RMSE at matched acceptance | +0.4% (−10% to +9%) | 0.83 |
| Selective interval score | +1.5% (−15% to +23%) | 0.67 |
| Joint acceptance rate | −0.23 points (−3.6 to +3.9) | 0.39 |
The penalty has no measurable effect, and the reason is structural. Since , the soft joint acceptance of any two gates is at least . Whenever each gate meets its own budget, which its own penalty enforces, the Fréchet term is exactly zero. It was inactive at the end of training in all 104 models (52 with the tuned weight; for the 52 with weight 0 we evaluate the penalty it would have added). In a stress test with both budgets at 0.95, where the gates can no longer meet their budgets, it became active on 3 of 6 datasets, and its effects on joint acceptance and error were small and inconsistent in sign (six datasets, one run per setting).
Consequence. The sentence "without it ... the intersection collapses well below either budget" was not supported and is removed. §4.3 now presents the Fréchet term as an optional safeguard that engages only when the per-gate budgets cannot be met. The dual-gate results do not depend on it, so its weight need not be tuned.
Q4. SelectiveNet setup. The reviewer's concern was justified. We retrained SelectiveNet on SelPI's backbone (same width, epochs, learning rate, decay and batch size; its own selection head) on all 18 datasets and five splits, and ran a small tuning grid for it on six datasets (App. T).
| Selective RMSE at φ = 0.7 | SelPI better / level / worse | Median difference (+: SelPI better) | p |
|---|---|---|---|
| Submission: SelectiveNet with its own backbone | 14 / 1 / 3 | +4.5% (SelPI better) | 0.031 |
| Same backbone, at SelPI's realized acceptance rate | 6 / 1 / 11 | −2.9% | 0.23 |
| Same backbone, SelectiveNet trained for φ = 0.7 | 5 / 2 / 11 | −1.9% | 0.20 |
With the backbone matched, most of the advantage reported in the submission disappears; the Yacht margin falls from 78% to 29%. The two methods are not significantly different at φ = 0.7. Over the whole acceptance range, SelectiveNet is ahead of the single-gate SelPI on 14 of 18 datasets (p = 0.016). Tuning SelectiveNet changes its error by at most about 6% and does not change the winner on any of the six datasets.
A second asymmetry is now stated in §5 and App. D. The search that set the dual gate's budgets and loss weights scored its trials by selective RMSE on the test portions of splits 0-2, which the evaluation also uses. We report the median trial rather than the best to limit this, but it remains an advantage that SelectiveNet did not have, so the comparison above is, if anything, tilted toward SelPI. The single-gate model behind all same-predictor results (the paper's Q3 and the new selector table) is trained with a fixed per-dataset configuration and no search over budgets or loss weights, so this asymmetry does not apply to it.
Consequence. The paper's Q2 (§6) is rewritten on the same-backbone comparison. We no longer claim that SelPI beats SelectiveNet in selective point error. The supported statement is that SelPI adds a prediction interval and interval-aware abstention at a small cost in point error relative to a dedicated selective regressor of the same capacity (median 2–3% at φ = 0.7, not significant). The main result of the paper (its Q3: learned gate vs. width on a fixed predictor) does not involve SelectiveNet and is unchanged.
We thank the reviewer again. These questions made the paper more accurate. We will be happy any additional questions to the best of our abilities.
I thank the authors for their comments and the additional experiments. After considering these responses alongside the concerns raised by reviewers AUid and ic51, I feel that several of my original concerns regarding the novelty and overall contribution in the paper still remain relevant. While I appreciate that the authors have addressed the technical questions, some of the new findings also very much narrow the original claims of the paper and is therefore difficult to assess in the same way as in my original review.
Due to the fact that I was already borderline in my assessment, I now lean towards a weak rejection. I think that the work offers useful insights, but would benefit from a more substantial revision that better establishes and positions its main contributions before resubmission.
Response to Reviewer cSa1
We thank the reviewer for a careful and useful review. Q2 and Q3 ask for the same Fréchet ablation, so the questions come down to three requests. Each led to new experiments. Based on these experiments, we revised some of the claims made in the paper. New results are in the revised PDF; code is at https://anonymous.4open.science/r/selpi-code-EBCF/.
W1. Is the contribution sufficient? We agree that the architecture is a natural combination, and we now say so in §1. The combination was not mechanical, however. It required design choices that neither parent method supplies: gate-weighted coverage and width statistics with an effective-sample-size scaling, an auxiliary head triple that keeps interval-relevant features alive for rejected inputs, and a two-gate mechanism in which each gate has its own loss and its own budget. The contributions we claim (revised after the requested experiments) are:
The revised paper also reports what did not hold (the Fréchet penalty, the advantage over SelectiveNet, the epistemic reading). We think a precisely scoped account is more useful than the broader claims of the submission.
W2 / Q1. Width and epistemic uncertainty. The reviewer is right, and we withdrew the sentence. The submission did not separate failure detection from epistemic detection, so we ran the controlled experiments the reviewer suggested (App. U).
(a) Synthetic task with an unseen region. One quadrant is removed from training and compared with a seen region of the same noise level. AUROC for separating unseen from seen points:
Only ensemble disagreement singles out the unseen region. The gates do not.
(b) Training-size sweep (six datasets, φ = 0.7). Share of the oracle gap that the gate closes over width, on the interval score:
The gate's advantage over width grows with data. If it came from detecting epistemic errors, which shrink as data grows, it would shrink instead. The kNN and two-head selectors also improve with data (all six datasets, 10% to 100%), and on three of the six datasets (concrete, superconduct, power-plant) the gate's curve is not monotone.
(c) Held-out region on six real datasets. One k-means cluster is removed from training and validation, and every selector keeps its threshold. Median change in the rejection rate on the held-out cluster: gate +0.34, kNN variance +0.18, width −0.27. The gate reacts strongly on three datasets (+0.59 to +0.66) and hardly at all on the other three. Width moves the wrong way on five of six: it becomes more confident in the unfamiliar region.
Answer to Q1. The gates are trained to rank prediction failures, and that is what the new selector table (Table 3) and the gate diagnostics (App. R) measure. They are not estimators of epistemic uncertainty: (a) shows that they miss a purely unfamiliar region, (b) shows that their advantage does not come from data scarcity, and (c) shows a partial, dataset-dependent reaction to shift. The revised claim (§1, §7) is the one the data support: width does not bound coverage failure, identifies misses no better than chance on these tasks, and becomes more confident under shift on five of six datasets. We make no claim that the gates capture epistemic uncertainty.