acceptodds
Under review as a conference paper at ICLR 2027

Semantic Support Shapes Reliability Evaluation and Downstream Decisions

Abstract

Semantic conditional reliability asks whether one predictor harms or corrects another within a specified set of true classes. Existing AUC decompositions and model-update methods do not isolate how semantic support affects reliability-scorer selection. Our support decomposition shows that predicted-label membership resolves every harmed–corrected comparison involving a query boundary, leaving only interior–interior comparisons unresolved. The resulting Semantic-Membership Baseline (SMB), normalized interior evidence, and support-order loss separate support-induced ranking from discrimination between unresolved harmed (H) and corrected (C) cases. We further characterize when semantic queries require incompatible rankings and connect this evaluation distinction to scorer selection, fallback decisions, and downstream utility. Across image classification, medical imaging, audio, intent classification, domain shift, and collaborative occupancy prediction, support-induced ranking is substantial but heterogeneous. Positive interior evidence can coexist with raw conditional AUROC either above or below SMB, and practical reliability scores can be ranked differently by raw AUROC and interior evidence. Support-aware evaluation also has an operational consequence: selecting scorers by interior evidence rather than raw conditional AUROC can change downstream fallback decisions and improve held-out utility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reliability evaluation, semantic support, conditional AUROC, model updates, selective prediction, fallback decisions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jeiSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how semantic conditioning affects the evaluation of reliability scores when comparing predictions from an original and an updated model. The main observation is that, for a given semantic query, some harmful and corrective changes can already be distinguished from predicted-label membership alone. The authors build on this to introduce the Semantic-Membership Baseline, interior evidence, and support-order loss, aiming to separate this structural effect from actual discrimination between harmful and corrective changes. They also analyze conflicting ranking requirements across queries and their implications for fallback decisions.

The framework is evaluated across several tasks, including image classification, medical imaging, audio, intent classification, and domain shift, with additional foundation-model experiments. The results show that raw conditional AUROC and interior evidence can lead to different scorer rankings. The paper also reports improvements in downstream fallback utility in some settings, although these gains are not consistent across configurations.

Critical strengths

The main strength of the paper is that it identifies an important issue with conditional AUROC: part of the ranking between harmful and corrective changes can already be determined by semantic membership. The proposed decomposition and Semantic-Membership Baseline provide a simple and useful way to distinguish this structural effect from actual discrimination. I think this is a valuable insight for evaluating reliability scores.

Another strength is the breadth of the experimental evaluation. The authors consider several datasets, tasks, and model architectures, and show that raw AUROC and interior evidence can lead to substantially different scorer rankings. This makes a convincing case that the issue matters in practice and is not limited to a particular setting, even if the downstream benefits remain less convincing.

Critical weaknesses

My main concern is that the practical benefits of selecting reliability scorers using interior evidence are not consistently supported by the experiments. While the main text highlights positive results, the complete results in Appendix P are much more mixed, including cases where interior-based selection actually reduces downstream utility. The validation-resampling experiments also show that some of the reported improvements are sensitive to the validation sample. This makes it difficult to assess how reliable the proposed selection criterion is in practice.

A second concern is the gap between interior AUROC and the actual downstream objective. Interior AUROC only considers discrimination between harmful and corrective changes, whereas fallback decisions must also account for both-wrong predictions. Therefore, better interior discrimination does not necessarily translate into better decisions. Moreover, many semantic queries have very limited interior samples, making the proposed metric difficult to estimate reliably or even undefined. Although the paper acknowledges these limitations, it does not sufficiently establish when interior-based evaluation should be preferred for scorer selection. I find this particularly important given that practical scorer selection is one of the paper's main contributions.

Questions for authors

(1) The main text highlights improvements in downstream utility, but the complete results in Table 36 and the validation-resampling results in Table 35 are considerably more mixed. How do the authors interpret these findings? Is there evidence that interior-based scorer selection provides an overall benefit, or is the main claim simply that it can improve utility in certain cases?

(2) Since interior AUROC does not account for both-wrong predictions, it is not clear why it should be a better criterion for selecting scorers when the actual objective is downstream fallback utility. Could the authors clarify what makes interior AUROC preferable to selecting scorers directly based on validation utility? This is particularly important given the limited interior support available for many semantic queries.

Reviewer_Ls8vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies reliability evaluation when comparing an original and a revised predictor within semantic class subsets. It shows that predicted-label membership can determine some harmed-versus-corrected comparisons, potentially inflating conditional AUROC. The authors derive a support-based AUROC decomposition and introduce the Semantic-Membership Baseline (SMB), interior evidence, and support-order loss. Experiments across several classification tasks show that these measures can change reliability-scorer rankings and affect downstream fallback decisions.

Critical strengths

The paper shows that semantic conditioning can make parts of the harmed/corrected ranking deterministic. SMB and interior evidence provide useful diagnostics for distinguishing structural ranking advantages from discrimination on unresolved cases.

Critical weaknesses
  1. The paper is difficult to follow, particularly in the introduction. The problem setting and practical motivation for semantic conditional AUROC are not clearly established. The paper also introduces many related concepts without sufficient intuition or a concrete running example explaining their roles. Moreover, the distinction between evaluation metrics and downstream decision objectives is not made clear early enough, and the connections among the theoretical results, scorer selection, and fallback decisions are not sufficiently developed. As a result, the main contributions and their practical significance require considerable effort to understand.

  2. The theoretical novelty is limited. The main AUROC decomposition largely follows from partitioning harmed–corrected pairs into boundary and interior cases, where the ordering of boundary cases is already determined by semantic membership. The paper does not sufficiently establish why interior evidence should be preferred for selecting reliability scorers. Interior discrimination does not necessarily translate into better downstream decisions, since fallback utility also depends on boundary cases, both-wrong predictions, and the magnitude of expected loss reduction. Although the paper reports positive results in several settings, the benefits of interior-aware scorer selection are configuration-dependent, and support projection appears more consistently effective. A comparison with directly selecting scorers based on validation fallback utility would help clarify the practical advantage of the proposed criterion.

Questions for authors
  1. In what practical scenarios is semantic conditional AUROC used to evaluate model updates, and why is this evaluation setting important?

  2. Why should interior evidence be preferred over directly optimizing validation fallback utility when selecting reliability scorers?

  3. Under what conditions does better interior discrimination lead to better downstream utility, given that the reported improvements are configuration-dependent?

Reviewer_dnMBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how a reliability score should be evaluated when it is used to decide whether to keep a revised model's prediction or fall back to the earlier model, and when evaluation is restricted to a semantic query, that is, a set of true classes. The authors observe that part of the comparison is settled before any score is consulted: if the earlier prediction lies inside the query and the revised one lies outside, the revision must be harmful, and the reverse case must be a correction. Only changes whose two predictions both lie inside the query remain genuinely open. They turn this into a decomposition of conditional ranking quality into a baseline available from query membership alone plus a residual measure of discrimination on the open cases, add a loss term capturing how often a given score violates the ordering implied by membership, and show that different queries can demand incompatible orders from a single scalar score while a shared posterior read out per query can satisfy them. They then recommend selecting reliability scores by the residual measure rather than by raw conditional ranking, and evaluate this on nine tasks with fixed predictor pairs, reporting that the two criteria often disagree and that in selected cases the resulting fallback decisions differ.

Critical strengths
  1. The central observation is simple, correct, and easy to underestimate in practice: for paired-prediction evaluation under a semantic query, a nontrivial share of conditional ranking quality is fixed by query geometry rather than by the score. The paper does not merely assert this; it derives the resulting floor explicitly, gives the residual measure that removes it, and demonstrates across nine tasks spanning natural images, medical images, audio, intent classification and domain shift that interior evidence and aggregate conditional ranking can point in opposite directions.

  2. The paper is disciplined about what its own decomposition does and does not license. It separates the exact identity for support-respecting readouts from the version that applies to arbitrary scores and warns against using them interchangeably; it states that positive interior evidence can coexist with lower aggregate ranking; it gives a worked counterexample in which the conditional posterior and the expected decision gain rank two candidates differently; and it retains undefined cases rather than dropping them silently.

Critical weaknesses
  1. The operational claim in the abstract is not supported by the paper's own replication, and the main text presents only the favourable draws. Section 5.2 reports three retrospective cases with gains of 0.726, 1.875 and 0.203 per 100 changed candidates, each interval excluding zero, but those intervals resample TEST data with the selected scorer held fixed, so they measure test noise rather than selection stability. The appendix repeats the selection over 50 resampled validation sets in 51 configurations with TEST fixed, and the picture inverts: 21 negative, 23 positive, 7 exact ties, and not one of the 51 intervals excludes zero. All six ImageNet configurations are negative, all three intent-classification configurations are exact ties, and the large positives come from configurations where one or two eligible queries exist in only 22 to 41 of 50 draws with an interval lower endpoint of exactly zero. The dataset and seed that supplies a headline number makes the point: PathMNIST seed 17 gives +1.875 [+1.084, +2.390] under fixed-validation selection and +0.2669 [−2.2748, +1.9605] when the validation sample is resampled. Why this is consequential: the stated contribution is that support-aware evaluation improves downstream decisions, and if selection by interior evidence is a coin flip across validation draws, the contribution reduces to a reporting recommendation — a qualification the appendix states and the abstract does not.

  2. The recommended criterion is never validated against the quantity it serves, and the margin it turns on is within noise. In the PathMNIST example interior evidence prefers one scorer by 0.846 against 0.850, a difference of 0.004 on 24 harmed and 17 corrected validation interiors; the ImageNet clothing case turns on 3 harmed and 16 corrected. No interval is attached to the interior-evidence estimate, and nothing shows selection by this criterion beats chance. The obvious alternative is missing: validation labels are available, so one can select the scorer directly on validation fallback utility, the quantity optimized at deployment, rather than on either ranking surrogate. The same gap appears at the framework level: the query-conditional readout derived from the shared posterior is the constructive answer to the ranking-incompatibility result, yet the selection experiments use only query-agnostic scorers. Why this is consequential: "select on interior evidence" is the actionable output, and it rests on a surrogate neither compared to the direct target nor shown to be stable.

  3. The paper does not establish its problem, and the concepts it introduces are never connected into an argument. The abstract opens on "semantic conditional reliability" and on one predictor harming or correcting another within a set of true classes, none of it defined or motivated there, and the reader meets five coined objects — support decomposition, the membership baseline, interior evidence, support-order loss, query-macro utility — before any application appears. The introduction then opens on a different frame, model updates, information fusion and alternative inference pipelines, none of which the abstract mentions. The chain is asserted rather than built: AUC decompositions and model-update methods are placed side by side as literatures that "do not isolate" the effect without explaining their relation, and the joining premise, that scorers are in practice selected by raw conditional ranking, is never documented with a citation, benchmark or example. Why this is consequential: an evaluation paper's contribution only makes sense as a correction to an existing practice, so the reader must be shown that the practice exists and is wrong for a specific reason; as written, a reader outside this sub-question cannot reconstruct the significance from the abstract and introduction.

Questions for authors
  1. Which result should a reader take as the paper's claim about utility: the three fixed-validation cases, or the 50-draw replication? The two disagree in sign for the same dataset and seed. If you regard the replication as the more informative experiment, would you be willing to make it the main-text result and present the fixed-split cases as illustrations of mechanism?

  2. Have you compared selection by interior evidence against selecting the scorer directly on validation fallback utility? The validation labels that make interior evidence computable also make utility computable, so this baseline seems both available and the natural competitor. If it performs as well or better, what does interior evidence add beyond being a smoother surrogate?

  3. The derived query-conditional readout is presented as the constructive resolution of the incompatibility result, but the experiments select among query-agnostic scores. Would evaluating that readout, with the shared posterior estimated on validation, change the conclusions about which criterion selects better scorers?

  4. Who chooses the query family in practice, and how much do the headline disagreements depend on that choice? The number of validation-eligible ImageNet queries ranges from 43 to 72 out of 395 across validation draws, and several reported effects rest on queries with only a handful of interior pairs. Does the 37.3% disagreement figure survive restricting to queries with substantial interior support on both sides?

  5. Will code be released? The protocol contains many bespoke switches, including eligibility thresholds, tie-breaking rules and the support tiers used in the projection analysis, and the reported numbers are hard to reconstruct without them.

  6. Which published or deployed pipeline selects a reliability scorer by raw conditional ranking over a semantic query? The paper's premise is that this is the practice being corrected, and naming a concrete case, a benchmark protocol or a system that does it would let a reader judge whether the correction targets something real rather than a straw man constructed from the metric alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.