acceptodds
Under review as a conference paper at ICLR 2027

Conformal Non-Coverage Risk Control (CNCRC): Pairwise Cost-Weighted Nonconformity Scores for Risk-Aware Prediction Sets

Abstract

Split-conformal prediction gives marginal coverage for any fixed nonconformity score, but it does not determine which alternatives a prediction set retains. We study CNCRC as an orientation-aware score-design family for fixed-label multiclass prediction with a supplied, bounded pairwise ambiguity-cost matrix. Its row and column forms are dual design choices: the row form supports conditional retained-set accounting, whereas the column form weights candidate admission by incoming cost. MAX and SUM provide peak-sensitive and accumulation-sensitive reductions while leaving the calibration template unchanged. We separate generic coverage and bounded-cost omission from the retained-set accounting properties induced by each score. In a validation-only, matched-coverage clinical benchmark, both orientations yield comparable composition metrics, with the column MAX form attaining AmbCost 0.293 and Hazard 10.1%. The poison stress test and the large-label clinical benchmark are interpreted as operating-point evidence about set composition. A poison-class control with the same recall and average set size shows lower poison-label admission on benign examples than random set inflation. The risk-budget sweep and a synthetic cost-matrix perturbation analysis characterize the operating range and sensitivity of the score family. These results characterize oriented score design for the stated benchmark without claiming a universal safety or downstream-utility guarantee.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Cost-Sensitive Learning, Risk Control, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SiVmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Cost-Normalized Conformal Risk Composition (CNCRC), a family of pairwise cost-weighted nonconformity scores for multiclass split-conformal prediction. It studies MAX and SUM reductions and row/column orientations of a supplied cost matrix, while retaining the standard split-conformal calibration procedure. The theoretical discussion distinguishes generic coverage and bounded-cost omission guarantees from deterministic weighted-ambiguity inequalities. The experiments examine prediction-set composition using a poison-class stress test and a MIMIC-IV/DrugBank benchmark.

Critical strengths

The paper addresses a relevant issue: marginal coverage and average set size do not fully describe the consequences of the alternatives retained in a prediction set.

Critical weaknesses
  1. The Introduction introduces the "proposed score layer" at line 038 and defines the scores in Eq. (1), but the practical rationale for these particular constructions remains underdeveloped. Considerable space is devoted to qualifying the guarantees before a concrete example explains how the cost matrix changes candidate scores and the resulting set. A small worked example or workflow figure contrasting CNCRC with standard CP would substantially improve the exposition. Algorithm 1 summarizes calibration and thresholding, but does not illustrate the consequences of choosing MAX versus SUM or row versus column orientation. These choices are central to the claimed contribution and need a clearer explanation of when and why each is useful.

  2. Proposition 1 and Corollary 2 follow from standard split-conformal coverage and bounded costs. The weighted-ambiguity inequalities in Section 3.6 follow directly from the score definitions and set inclusion, as the authors acknowledge. The conversion to raw AmbCost additionally requires a retained-score lower bound and may be uninformative. This leaves much of the contribution resting on the empirical value of the proposed scores. The manuscript provides limited guidance on selecting a score and orientation for a given objective. Furthermore, the Cost-Aware CP baseline is a particular additive penalty specified in Appendix E; the relationship between this baseline and the utility-/decision-aware methods discussed in Section 2 needs a more substantive justification. The current evidence does not convincingly establish the significance of the proposed score family.

  3. Appendix E defines empirical R_NC as the average of c(Y_i, yhat_i) times the indicator that Y_i is absent from the prediction set (lines 723-728). Under the disclosed normalization, 0 <= c <= 1, so this empirical quantity must satisfy R_NC <= 1 - empirical coverage on the same examples. However, Table 1(a) reports coverage 0.900 and R_NC 0.413 for Standard CP, and coverage 0.901 and R_NC 0.412 for Cost-Aware CP. These values cannot simultaneously satisfy the stated definition on the normalized scale. The inequality also holds after averaging over the same runs. If the risk column uses a different cost scale, denominator, or evaluation sample, that distinction must be specified and reconciled with the risk-alignment claims. This is a material obstacle to interpreting the main empirical comparison.

  4. For the disclosed row-oriented MAX score, the normalized poison row has c(p,b_i) = 10/150. If the predictor outputs probabilities, then s_max(x,p) = (10/150) max_i p_hat_{b_i}(x) <= 10/150 for every input. At the approximately 0.142 threshold reported in Appendix E, the poison label would therefore be retained on every example, implying 100% poison-label admission on benign examples. Table 2 instead reports an FPR of 14.2% for MAX. Section 3.1 permits arbitrary nonnegative model scores, so this is a conditional concern: the authors need to specify the actual score scale and explain how the reported threshold, matrix, and FPR are compatible. The random +1 baseline also needs an explicit label-sampling rule to make its reported 100% poison recall and matched-size comparison interpretable.

  5. Although the appendices provide split sizes, broad predictor architectures, and some baseline formulas, they do not adequately specify the clinical prediction target and label construction, the complete cost-mapping rules actually used, the nested set family calibrated by CRC, or the number of runs and construction of the reported 95% intervals. These details are particularly important because the conclusions depend on the supplied cost matrix and operating-point selection. The clinical matrix is also approximately symmetric, so the similar row/column results provide limited evidence about the proposed orientation trade-off for substantially asymmetric costs. Resolving the reporting inconsistencies and providing a fully specified protocol are necessary to assess the claimed benefits.

Questions for authors

See weaknesses.

Reviewer_vuedOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper purposes Cost-Normalized Conformal Risk Composition (CNCRC), a method that incorporates pairwise ambiguity costs into the construction of nonconformity scores for multi-class conformal prediction. In specific, it introduces MAX and SUM scores that aggregate prediction scores weighted by pairwise distraction costs. These scores are then calibrated using standard split conformal prediction, retaining the usual coverage guarantees under exchangeability. The paper also discusses bounds on the weighted ambiguity costs, which largely follow from standard conformal guarantees. Experiments on a synthetic poison-class task and a clinical benchmark are presented to demonstrate improvements in ambiguity-related metrics over the baselines.

Critical strengths
  • The problem motivation is interesting. Accounting for the different ambiguity-related consequences of retaining incorrect labels is an important aspect that standard conformal methods do not directly address.
  • The proposed method shows empirical improvements in ambiguity-related metrics over the considered baselines.
Critical weaknesses
  • The proposed method mainly introduces two cost-weighted nonconformity scores, followed by standard split conformal prediction. The coverage and risk guarantees follow directly from existing conformal results, while the score-specific bounds are relatively straightforward consequences of the score definitions. The methodological and theoretical contributions seem rather limited.
  • While the motivation is interesting, the connection between the purposed scores and the ambiguity-related targets is not entirely clear.
  • There appear to be some inconsistencies in the experimental results. In Table 1(a), CRC reports nearly 100% coverage but still has a nonzero non-coverage risk (RNC). Moreover, standard CP reports coverage of 0.9 and RNC of 0.413, which seems inconsistent with the stated cost normalization to [0,1], under which RNC should not exceed the miscoverage rate of 0.1.
Questions for authors

Could the authors clarify how the empirical RNC is computed and normalized in Table 1(a)?

Reviewer_azquOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Given a bounded pairwise ambiguity-cost matrix c(y,z), the paper defines two nonconformity scores for split-conformal multiclass prediction, MAX and SUM. Each comes in a row orientation c(y,z) and a column orientation c(z,y). Calibration is the standard split-CP procedure. The paper separates generic properties from score-specific retained-set accounting identities. Experiments include a MIMIC-IV × DrugBank benchmark with 3,421 candidate drugs and patient-disjoint splits.

Critical strengths
  • the authors separate clearly what split CP guarantees and state their scope explicitly
  • on the clinical benchmark, high-hazard retained distractors are reduced substantially at matched coverage
Critical weaknesses
  • restricted scope and unclear significance: "position of this work" mainly defines what the contribution is not, downstream utility are out of scope
  • the central conceptual element (orientation) is exercised only in the synthetic test, no real task is shown where the row/column distinction matters
  • missing related work: ConfTr (Stutz et al., ICLR 2022) Cost-sensitive conformal training (Jia et al. 2025) Conformal Decision Theory (Lekeufack et al. 2024)
  • clinical task is not well defined, the paper never states what the single “reference label” is for a MIMIC-IV patient context. patients typically receive several drugs at once, so “which drug, given this context” is not a fixed single-label problem check for example GAMENet (Shang et al., AAAI 2019) and SafeDrug (Yang et al., IJCAI 2021)
Questions for authors
  • What is the label in the MIMIC-IV × DrugBank task: how is a single reference drug chosen per patient context, and how are concurrent prescriptions handled?
  • How does CNCRC compare with CRC or LTT calibrated directly on AmbCost or Hazard, with ConfTr-style training, and with a DDI-aware recommender such as SafeDrug?
  • Is there a real-world cost matrix with strong asymmetry in which row vs column orientation matters ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.