acceptodds
Under review as a conference paper at ICLR 2027

Router-Uniform Selective Prediction under Candidate-Set Shift

Abstract

Vision-language systems often choose candidate labels after observing an image. Even when each candidate mechanism is individually reliable, input-dependent routing can concentrate its errors and invalidate selective-risk guarantees. We characterize this failure using labeled replay: one image is evaluated under every registered mechanism. Two joint events, a selectable accepted error and an unavoidable correct acceptance, determine a router-uniform risk envelope that is exact for deterministic labels. We prove that synchronizing acceptance gates preserves worst-router coverage and cannot increase worst-router risk. The resulting objective is to predict and certify whether any mechanism errs. For set-independent scores, two nested endpoint sets additionally certify all intermediate sets. These results yield \method, which combines a learned shared gate with exact image-level binomial or remaining-pool hypergeometric bounds. On Flowers-102, a label-free router raises the risk of per-mechanism MSP gates from to . Joint replay instead certifies coverage for a DeepSets gate at risk target , while a marginal union-bound certificate for the same score rejects all inputs. A logistic gate attains mean coverage across five training seeds and improves on ViLU with the same replay certificate by – percentage points across six dataset–model systems.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Selective prediction, Risk control, Uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P4mGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses selective prediction in vision-language systems where candidate-label mechanisms are dynamically selected by input-dependent routing. It demonstrates that even individually calibrated mechanisms can violate risk guarantees when a router concentrates their errors. While prior work has largely focused on assessing the reliability of individual predictions, this paper shifts the focus toward guaranteeing the reliability of an entire system that dynamically switches between prediction paths. To address this challenge, this paper establish a router-uniform risk bound using joint replay, prove the minimax advantage of shared acceptance gates, and extend guarantees to nested candidate sets. Building on these theoretical results, they propose SetGuard-R, which learns to predict whether any mechanism makes an error and provides finite-sample risk guarantees through exact statistical certification. Experiments across three datasets demonstrate the failure of conventional calibration under adaptive routing and show improved certified coverage over existing baselines.

Critical strengths
  1. The paper provides a clear theoretical formulation of selective-risk failures under input-dependent candidate-set routing. The router-uniform risk envelope and shared-gate principle offer meaningful theoretical contributions, while joint replay enables less conservative risk certification than separate per-mechanism bounds.

  2. The experiments support these findings by demonstrating substantial routing-induced risk violations and improved certified coverage.

Critical weaknesses
  1. While the paper motivates its problem through candidate-set selection in deployed vision-language systems, the empirical validation remains limited to tightly controlled settings. The proposed guarantees assume a fixed portfolio, catalog, retrievers, and preprocessing pipeline, with an unchanged image-label distribution. Also, the experiments are restricted to CLIP-based models and predefined candidate mechanisms. This leaves a gap between the deployment-oriented motivation and the empirical evidence supporting the framework's applicability.

  2. Despite the theoretical value of joint replay and shared-gate certification, the incremental benefit of the proposed union-error learning approach is not sufficiently demonstrated. For example, the synchronized local MLP baseline achieves 47.8% certified coverage, nearly matching the 47.9% of the proposed logistic union-error gate on the primary split, although their threshold configurations differ. This raises questions about whether union-error learning provides a meaningful empirical advantage beyond shared-gate synchronization and joint certification.

Questions for authors
  1. Additional empirical evidence would be valuable for assessing the applicability of the proposed framework beyond the current CLIP-based, fixed-portfolio setting. In particular, evaluations involving other multimodal model families or the out-of-contract experiments mentioned in the appendix would help clarify its practical scope.

  2. It would be helpful to more clearly distinguish the benefit of union-error learning from those of shared-gate synchronization and joint replay certification. In particular, matched comparisons with synchronized local gates across multiple datasets and training seeds would provide further insight into the individual contributions of these components.

1 comment

Thank you for the detailed response. The additional experiments address my main concerns, particularly regarding the contribution of union supervision. I agree with the reviewer Wcko's concern about the unexplained reversal at the stricter risk level, but I find the overall evidence sufficiently strengthened. I have raised my rating to 3.

Reviewer_zm2aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method for principled output risk estimation for image classification pipelines using image-dependent candate set selection. The main challenge being addressed is that the choice of the label set and how confidently a classifier evaluates labels are strongly connected, which makes it challenging to robustly select which classifications should be discarded as too risky. The proposed method, SetGuard-R, replays the candidate label selection using different mechanisms, and a shared vulnerability scorer learns to estimate whether any of the candiate routers would misclassify the image. The main goal of the method to provide a certifiable error bound that survives a potentially adversarial router.

Critical strengths

The method demonstrates clear improvement over techniques that do not take routing into account, especially in the case of adversarial routing, allowing for higher coverage.

Critical weaknesses

The overall narrative and flow of the paper is difficult to follow. The language and wording are imprecise in many places, and many terms are used without an explanation or definition, which makes the paper difficult to comprehend on first reading. The abstract and intro sections include technical details and references to results that cannot be contextualized without having already read the later sections of the paper, while providing very little guiding explanation or motivation for the problem or method.

The method itself has a rather narrow scope when it comes to real world deployment. All the candidate mechanisms are assumed deterministic, and must be replayed for every image, increasing inference cost. It is also conservative, and adding more routers can only reduce coverage.

Questions for authors

Did you evaluate the simple baselines of Table 1 on the tasks of Table 2?

Reviewer_WckoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies selective prediction for vision-language systems where candidate sets are chosen by input-dependent routers. The authors show that risk guarantees obtained for individual candidate mechanisms may fail under adaptive routing, even when the underlying image-label distribution remains unchanged. They derive a router-uniform risk envelope and establish a shared-gate principle under a minimax coverage objective. Building on these results, SetGuard-R learns an image-level union-error predictor and provides finite-sample risk certificates using binomial or hypergeometric bounds.

Critical strengths
  1. The paper addresses an interesting and well-defined problem. The observation that input-dependent routing can invalidate individual selective-risk guarantees is important for systems with dynamically selected candidate sets.

  2. The theoretical contributions are meaningful.

  3. The experiments make an effort to separate the effects of uncertainty scoring and risk certification.

Critical weaknesses
  1. The applicability of the theoretical guarantee is relatively restricted. The guarantees assume a fixed portfolio of candidate mechanisms and an unchanged image-label distribution. This is a reasonable setting for studying the effect of routing alone, but it excludes several common deployment changes, such as adding new candidate generators, updating the retriever, or modifying the label catalog.

  2. The empirical advantage of learned union-error scoring over synchronized local gates is less clear. Although the shared-gate principle is theoretically well motivated, the experiments show that carefully constructed local gates can also provide useful risk control. In Table 1, synchronized local MLP gates achieve 47.8% certified coverage, compared with 47.9% for the logistic union-error gate. This suggests that much of the benefit may come from joint replay certification and synchronized acceptance rather than learning the union-error target itself.

  3. The effectiveness of union-error supervision is not consistent at stricter risk levels. At the stricter risk target of 0.05, Table 11 shows that the mean-target DeepSets model achieves 20.0% certified coverage, while the union-target DeepSets model fails to certify any nonzero coverage. A similar pattern appears across random seeds in Appendix Table 8. This is interesting because the theoretical analysis motivates learning the conditional union-error probability. The discrepancy does not invalidate the theoretical result, but it raises questions about the statistical or optimization difficulty of learning the union-error target.

Questions for authors
  1. Can the authors provide an empirical analysis of how certified coverage changes when the registered portfolio grows or its mechanisms are replaced?

  2. Under what conditions does union-error supervision outperform synchronized local gates? Is the advantage larger when mechanism errors become more heterogeneous or less correlated?

  3. Why does mean-target DeepSets outperform union-target DeepSets at the stricter risk level? Is this mainly due to estimation variance, optimization difficulty, or the distribution of union-error labels?

  4. Can the router-uniform replay certificate be applied to option-conditioned generative VLMs without relying on the nested endpoint closure property?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.