acceptodds
Under review as a conference paper at ICLR 2027

ADOQ: Predefined Analytical Regions for Selective Classification

Abstract

Selective classification usually treats confidence scores as ranking devices and relies on held-out data to determine operating thresholds for specific coverage or risk targets. ADOQ introduces a different construction in which a single learned score organizes the latent decision space into successive predefined analytical shells around fixed class centers, combining a prescribed numerical scale with an ordering driven by discriminative difficulty. Once training and model selection are complete, the encoder, class geometry, score scales, and the entire continuous family of shell boundaries are fixed analytically, without empirical threshold estimation or per-operating-point recalibration, and certification is performed only afterward on these frozen regions. A single unlabeled certification sample provides distribution-free occupancy bounds uniformly over the predefined shell family, labeled observations provide exact reliability bounds for each fixed shell, and a new prediction is thus assigned to a region carrying an independently certified reliability bound rather than only the classifier's global accuracy or an arbitrary confidence score. Experiments on CIFAR-10, SVHN, Imagenette, and CIFAR-100 show that successive analytical shells strongly stratify predictive reliability, with inner and intermediate regions remaining substantially more reliable than the outer tail even when global performance or score-law fidelity is markedly weaker. Thus, a globally imperfect classifier, which would typically be discarded or retrained outright, may still contain substantial predefined regions with independently certified population-level predictive reliability, while less reliable outer regions are explicitly isolated for deferral.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
selective classification, uncertainty quantification, distribution-free coverage guarantees, post-hoc statistical certification, difficulty-ordered representation learning, chi-squared reference law, abstention, risk-coverage

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_29eHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ADOQ, which combines fixed class centers, normalized distances, and class-conditional margin ranks to learn a representation with prescribed chi-square radial targets for selective classification. Once the model and analytical regions are frozen, independent data certify regional occupancy and accuracy using DKW and binomial confidence bounds. Experiments on four image-classification datasets evaluate selective performance and regional reliability. While the training construction is interesting and the certification appears sound under the stated assumptions, the missing matched certification comparison, single-run evaluation, and differing training protocols leave ADOQ’s incremental benefit and its robustness insufficiently established.

Critical strengths

An interesting and clearly formulated training construction. The combination of fixed-center geometry, margin-based ordering, and prescribed radial targets provides a concrete approach to organizing classification difficulty on a common numerical coordinate.

Well-specified certification with evidence of reliable subsets. The regional guarantees appear technically valid under the stated independence and multiple-comparison conditions, without requiring an exact chi-square fit. The experiments demonstrate reliability stratification, and the paper transparently discusses distributional mismatch and operational limitations.

Critical weaknesses

The central incremental value is not established by the comparisons. The certification argument does not depend on ADOQ’s geometry or training objective. Strong frozen scores such as MSP or Mahalanobis distance can also define regions before certification and receive the same type of statistical guarantees. However, the paper lacks a comparison with these alternatives under matched certification sample budgets, confidence levels, region-family sizes, and construction/selection protocols. The SGR experiment uses a different confidence level and selection mechanism and is explicitly not a performance comparison. Thus, the reported reliable subsets do not establish the added value of the specialized representation learning.

The operational benefit of the prescribed scale remains unclear under substantial mismatch. Proposition 3 characterizes an ideal population target, rather than guaranteeing the distribution or reliability ordering of the learned operational score. On CIFAR-100, the reference KS statistic is 0.3900, while the outer region with nominal mass 0.1% contains 30.67% of test examples. This does not invalidate certification, but substantially weakens the nominal occupancy interpretation. The paper has not established what practical benefit this prescribed coordinate provides over ordinary fixed thresholds or quantiles constructed independently of certification.

The experimental design limits conclusions about robustness and attribution. Each main configuration uses a single training run, so the stability of the learned structure and benefits across retraining is unknown. ADOQ also uses 400 epochs, compared with 200 or 300 for several baselines, and Imagenette augmentation differs. These differences particularly complicate attribution on Imagenette, the only dataset where ADOQ outperforms MSP in both accuracy and AURC. On the other three datasets, its AURC is worse. Different training recipes and non-SOTA performance are not independently grounds for rejection, but without controlled comparisons and independent repetitions, the evidence does not establish a robust additional benefit that compensates for these performance trade-off.

Questions for authors
  1. Under matched certification sample budgets, confidence levels, region-family sizes, and construction/selection protocols, what advantage does ADOQ provide over strong frozen scores? In particular, does it improve certified retention or reduce certification sample requirements?

  2. What operational benefit remains when empirical occupancy differs substantially from the prescribed scale, and is that benefit supported by independent training repetitions and controls for training budget and augmentation?

Reviewer_ADGMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the ADOQ method, which maps the ranking of classification difficulty onto a chi-square distribution scale. It constructs analytical regions around fixed class centers and utilizes independent data to verify the coverage and prediction accuracy. Experiments demonstrate that, even when the overall model performance is limited, these regions can effectively distinguish between reliable predictions and those should be rejected.

Critical strengths
  1. This method combines difficulty ranking with geometric structure, providing a new framework for constructing provable prediction regions.

  2. The method is simple, utilizing only a single loss function.

Critical weaknesses
  1. The structures and writing of the paper are terrible. In Section 3 (METHOD), there are no diagrams, pipeline overviews, or pseudocode, difficult to understand the workflow.

  2. Formatting issue. For example, line 354 is missing references to the CIFAR-10, SVHN, CIFAR-100, and Imagenette datasets. On line 418 "CIFAR-100 provides the strongest stress test. Despite ...." lacks citations to the experiments.

Questions for authors
  1. It unclear whether these gains from the specific training strategy. It is recommended to compare with a classifier that uses cross-entropy, while employing the same center points.

  2. This method relies on pre-defined, non-updatable class centers; however, its sensitivity to this geometric configuration remains unclear. It is suggested that supplementary experiments be conducted by varying the spacing between class centers and introducing positional perturbations.

Reviewer_gViBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses selective classification (SC), in which a classifier may abstain from making uncertain predictions to reduce classification risk among the retained predictions.

The paper proposes ADOQ, which learns a confidence score based on normalized distances to fixed class centers and uses a prescribed reference distribution to define successive regions, termed analytical shells. After training and model selection, the model and region boundaries are frozen. An independent held-out dataset is then used to obtain statistical bounds on the occupancy and predictive reliability of these regions.

Critical strengths

The paper presents a selective-classification method supported by statistical arguments for bounding the population occupancy and predictive reliability of fixed regions.

Critical weaknesses

1. The presentation is difficult to follow and would benefit from substantial clarification.

I have several concerns about the clarity and organization of the manuscript.

  1. Key terminology is introduced before sufficient explanation is provided. The abstract and introduction rely heavily on terms such as “analytical shells,” “shell boundaries,” and “analytical coordinate,” whose precise meanings become clear only later in the manuscript. Although definitions are subsequently provided, explaining the underlying ideas in plain language when these terms first appear would help readers understand the overall approach and its motivation.

  2. Some motivating statements require further justification. For example, the first paragraph of the introduction states, “The distinction matters whenever operating rules must remain fixed while new evidence is accumulated.” It is not immediately clear why this requirement motivates the proposed construction, or what practical settings make it important. A concrete example would help explain the significance of this distinction.

  3. The contribution statement does not sufficiently distinguish methodological description from demonstrated novelty and value. Of the four listed contributions, the fourth most directly establishes the relevance of the proposed method to selective classification through empirical results. The first three primarily describe components or design choices of the method. Although methodological innovations can constitute contributions, the manuscript should explain more clearly what is novel about these components and why they provide additional value beyond existing approaches. As currently presented, I find the fourth contribution more convincing than the first three.

2. The discussion and evaluation omit an important related work [1].

The manuscript does not discuss [1], which is relevant to both the motivation for ADOQ and the choice of confidence-score baselines. Although [1] focuses on selective classification under distribution shifts, its analysis of score ordering and margin-based confidence scores is also pertinent to the present paper.

  1. The relationship between ranking quality and the prescribed numerical scale needs clarification. For a fixed classifier, [1] emphasizes that the risk–coverage curve is determined by the ordering induced by the confidence score. ADOQ, meanwhile, argues that ranking alone is insufficient to provide predefined regions with a stable numerical meaning, and therefore incorporates a prescribed score scale into representation learning. The manuscript should discuss this difference in emphasis and clarify what is gained by learning that scale during training, rather than obtaining a useful ranking and subsequently applying a monotone transformation and independent certification. Such a discussion would help establish whether the two perspectives address different objectives or whether ADOQ offers an additional benefit for the same selective-classification objective.

  2. Relevant alternatives to softmax-response scores are missing. Reference [1] identifies a scale-sensitivity issue in several widely used softmax-response confidence scores and studies confidence-margin and geometric-margin alternatives based on raw logits. Although the present manuscript includes non-softmax distance-based baselines, it does not discuss or evaluate these margin-based alternatives. Their inclusion would make the comparison more complete and help justify the selection of confidence-score baselines.

3. The practical significance of the distinction between threshold calibration and certification is not convincingly established.

The authors emphasize that ADOQ differs from existing score-based approaches because its region boundaries are fixed after training and model selection, while held-out data are used only for certification. I understand the procedural distinction, but I am not yet convinced that the manuscript establishes its practical advantage for selective classification.

Appendix C.8 distinguishes the following two procedures:

ProcedureWhat is fixed beforehand?What held-out data determine
Coverage-targeted threshold calibrationA desired coverage level, such as 80%A threshold that achieves approximately that coverage
ADOQ occupancy certificationA particular threshold or shellIts actual coverage and a confidence bound

Both procedures require held-out evidence to quantify their operating behavior. The distinction is whether that evidence is used to select an acceptance boundary or to assess one fixed beforehand. For coverage, these procedures estimate the threshold–coverage relationship in opposite directions. While this distinction is statistically meaningful, its practical benefit in the present setting requires further justification.

My concerns are as follows:

  1. A fixed ADOQ boundary does not necessarily deliver a specified actual coverage. ADOQ avoids empirical threshold fitting while allowing nominal and actual coverage to differ substantially. For example, its nominal .99 cumulative region on CIFAR-100 contains only 64.73% of the test observations (Table 14). The authors acknowledge this discrepancy, but it raises the question of what advantage predefined thresholds offer when deployment requires a particular actual coverage.

  2. The same fixed-boundary assessment is available for ordinary confidence scores. For example, one could specify a softmax-confidence threshold before examining held-out data and then estimate its occupancy and accuracy using the same certification arguments, under the stated sampling assumptions. ADOQ's particular geometry is different, but fixing and subsequently certifying a region is not unique to that construction. The manuscript should therefore clarify what additional benefit follows from its analytical geometry.

  3. Selecting an operating region after certification reintroduces data-dependent selection. If one examines the certified occupancy results and chooses the value of ρ\rho that achieves approximately 80% actual coverage, one is effectively selecting a threshold using held-out data, expressed through a different coordinate. Appropriate simultaneous bounds may preserve the validity of such selection, but the resulting operating rule still depends on the held-out observations. The manuscript should distinguish fixing a family of candidate regions from fixing the final acceptance rule and explain how this distinction affects its claimed advantage over threshold calibration.

[1] Liang, Hengyue, Le Peng, and Ju Sun. "Selective classification under distribution shifts." Transactions on machine learning research 2024 (2024): dmxMGW6J7N.

Questions for authors

I would appreciate clarification of whether the concerns above accurately reflect the intended scope and claims of the paper. If any point rests on a misunderstanding, please identify the relevant distinction. Otherwise, I would encourage the authors to address the presentation issues, the relationship to [1], and the practical benefit of predefined regions relative to conventional confidence-score calibration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.