- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses selective prediction in vision-language systems where candidate-label mechanisms are dynamically selected by input-dependent routing. It demonstrates that even individually calibrated mechanisms can violate risk guarantees when a router concentrates their errors. While prior work has largely focused on assessing the reliability of individual predictions, this paper shifts the focus toward guaranteeing the reliability of an entire system that dynamically switches between prediction paths. To address this challenge, this paper establish a router-uniform risk bound using joint replay, prove the minimax advantage of shared acceptance gates, and extend guarantees to nested candidate sets. Building on these theoretical results, they propose SetGuard-R, which learns to predict whether any mechanism makes an error and provides finite-sample risk guarantees through exact statistical certification. Experiments across three datasets demonstrate the failure of conventional calibration under adaptive routing and show improved certified coverage over existing baselines.
Critical strengths
-
The paper provides a clear theoretical formulation of selective-risk failures under input-dependent candidate-set routing. The router-uniform risk envelope and shared-gate principle offer meaningful theoretical contributions, while joint replay enables less conservative risk certification than separate per-mechanism bounds.
-
The experiments support these findings by demonstrating substantial routing-induced risk violations and improved certified coverage.
Critical weaknesses
-
While the paper motivates its problem through candidate-set selection in deployed vision-language systems, the empirical validation remains limited to tightly controlled settings. The proposed guarantees assume a fixed portfolio, catalog, retrievers, and preprocessing pipeline, with an unchanged image-label distribution. Also, the experiments are restricted to CLIP-based models and predefined candidate mechanisms. This leaves a gap between the deployment-oriented motivation and the empirical evidence supporting the framework's applicability.
-
Despite the theoretical value of joint replay and shared-gate certification, the incremental benefit of the proposed union-error learning approach is not sufficiently demonstrated. For example, the synchronized local MLP baseline achieves 47.8% certified coverage, nearly matching the 47.9% of the proposed logistic union-error gate on the primary split, although their threshold configurations differ. This raises questions about whether union-error learning provides a meaningful empirical advantage beyond shared-gate synchronization and joint certification.
Questions for authors
-
Additional empirical evidence would be valuable for assessing the applicability of the proposed framework beyond the current CLIP-based, fixed-portfolio setting. In particular, evaluations involving other multimodal model families or the out-of-contract experiments mentioned in the appendix would help clarify its practical scope.
-
It would be helpful to more clearly distinguish the benefit of union-error learning from those of shared-gate synchronization and joint replay certification. In particular, matched comparisons with synchronized local gates across multiple datasets and training seeds would provide further insight into the individual contributions of these components.
Thank you for the detailed response. The additional experiments address my main concerns, particularly regarding the contribution of union supervision. I agree with the reviewer
Wcko's concern about the unexplained reversal at the stricter risk level, but I find the overall evidence sufficiently strengthened. I have raised my rating to 3.