- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how semantic conditioning affects the evaluation of reliability scores when comparing predictions from an original and an updated model. The main observation is that, for a given semantic query, some harmful and corrective changes can already be distinguished from predicted-label membership alone. The authors build on this to introduce the Semantic-Membership Baseline, interior evidence, and support-order loss, aiming to separate this structural effect from actual discrimination between harmful and corrective changes. They also analyze conflicting ranking requirements across queries and their implications for fallback decisions.
The framework is evaluated across several tasks, including image classification, medical imaging, audio, intent classification, and domain shift, with additional foundation-model experiments. The results show that raw conditional AUROC and interior evidence can lead to different scorer rankings. The paper also reports improvements in downstream fallback utility in some settings, although these gains are not consistent across configurations.
Critical strengths
The main strength of the paper is that it identifies an important issue with conditional AUROC: part of the ranking between harmful and corrective changes can already be determined by semantic membership. The proposed decomposition and Semantic-Membership Baseline provide a simple and useful way to distinguish this structural effect from actual discrimination. I think this is a valuable insight for evaluating reliability scores.
Another strength is the breadth of the experimental evaluation. The authors consider several datasets, tasks, and model architectures, and show that raw AUROC and interior evidence can lead to substantially different scorer rankings. This makes a convincing case that the issue matters in practice and is not limited to a particular setting, even if the downstream benefits remain less convincing.
Critical weaknesses
My main concern is that the practical benefits of selecting reliability scorers using interior evidence are not consistently supported by the experiments. While the main text highlights positive results, the complete results in Appendix P are much more mixed, including cases where interior-based selection actually reduces downstream utility. The validation-resampling experiments also show that some of the reported improvements are sensitive to the validation sample. This makes it difficult to assess how reliable the proposed selection criterion is in practice.
A second concern is the gap between interior AUROC and the actual downstream objective. Interior AUROC only considers discrimination between harmful and corrective changes, whereas fallback decisions must also account for both-wrong predictions. Therefore, better interior discrimination does not necessarily translate into better decisions. Moreover, many semantic queries have very limited interior samples, making the proposed metric difficult to estimate reliably or even undefined. Although the paper acknowledges these limitations, it does not sufficiently establish when interior-based evaluation should be preferred for scorer selection. I find this particularly important given that practical scorer selection is one of the paper's main contributions.
Questions for authors
(1) The main text highlights improvements in downstream utility, but the complete results in Table 36 and the validation-resampling results in Table 35 are considerably more mixed. How do the authors interpret these findings? Is there evidence that interior-based scorer selection provides an overall benefit, or is the main claim simply that it can improve utility in certain cases?
(2) Since interior AUROC does not account for both-wrong predictions, it is not clear why it should be a better criterion for selecting scorers when the actual objective is downstream fallback utility. Could the authors clarify what makes interior AUROC preferable to selecting scorers directly based on validation utility? This is particularly important given the limited interior support available for many semantic queries.