- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Cost-Normalized Conformal Risk Composition (CNCRC), a family of pairwise cost-weighted nonconformity scores for multiclass split-conformal prediction. It studies MAX and SUM reductions and row/column orientations of a supplied cost matrix, while retaining the standard split-conformal calibration procedure. The theoretical discussion distinguishes generic coverage and bounded-cost omission guarantees from deterministic weighted-ambiguity inequalities. The experiments examine prediction-set composition using a poison-class stress test and a MIMIC-IV/DrugBank benchmark.
Critical strengths
The paper addresses a relevant issue: marginal coverage and average set size do not fully describe the consequences of the alternatives retained in a prediction set.
Critical weaknesses
-
The Introduction introduces the "proposed score layer" at line 038 and defines the scores in Eq. (1), but the practical rationale for these particular constructions remains underdeveloped. Considerable space is devoted to qualifying the guarantees before a concrete example explains how the cost matrix changes candidate scores and the resulting set. A small worked example or workflow figure contrasting CNCRC with standard CP would substantially improve the exposition. Algorithm 1 summarizes calibration and thresholding, but does not illustrate the consequences of choosing MAX versus SUM or row versus column orientation. These choices are central to the claimed contribution and need a clearer explanation of when and why each is useful.
-
Proposition 1 and Corollary 2 follow from standard split-conformal coverage and bounded costs. The weighted-ambiguity inequalities in Section 3.6 follow directly from the score definitions and set inclusion, as the authors acknowledge. The conversion to raw AmbCost additionally requires a retained-score lower bound and may be uninformative. This leaves much of the contribution resting on the empirical value of the proposed scores. The manuscript provides limited guidance on selecting a score and orientation for a given objective. Furthermore, the Cost-Aware CP baseline is a particular additive penalty specified in Appendix E; the relationship between this baseline and the utility-/decision-aware methods discussed in Section 2 needs a more substantive justification. The current evidence does not convincingly establish the significance of the proposed score family.
-
Appendix E defines empirical R_NC as the average of c(Y_i, yhat_i) times the indicator that Y_i is absent from the prediction set (lines 723-728). Under the disclosed normalization, 0 <= c <= 1, so this empirical quantity must satisfy R_NC <= 1 - empirical coverage on the same examples. However, Table 1(a) reports coverage 0.900 and R_NC 0.413 for Standard CP, and coverage 0.901 and R_NC 0.412 for Cost-Aware CP. These values cannot simultaneously satisfy the stated definition on the normalized scale. The inequality also holds after averaging over the same runs. If the risk column uses a different cost scale, denominator, or evaluation sample, that distinction must be specified and reconciled with the risk-alignment claims. This is a material obstacle to interpreting the main empirical comparison.
-
For the disclosed row-oriented MAX score, the normalized poison row has c(p,b_i) = 10/150. If the predictor outputs probabilities, then s_max(x,p) = (10/150) max_i p_hat_{b_i}(x) <= 10/150 for every input. At the approximately 0.142 threshold reported in Appendix E, the poison label would therefore be retained on every example, implying 100% poison-label admission on benign examples. Table 2 instead reports an FPR of 14.2% for MAX. Section 3.1 permits arbitrary nonnegative model scores, so this is a conditional concern: the authors need to specify the actual score scale and explain how the reported threshold, matrix, and FPR are compatible. The random +1 baseline also needs an explicit label-sampling rule to make its reported 100% poison recall and matched-size comparison interpretable.
-
Although the appendices provide split sizes, broad predictor architectures, and some baseline formulas, they do not adequately specify the clinical prediction target and label construction, the complete cost-mapping rules actually used, the nested set family calibrated by CRC, or the number of runs and construction of the reported 95% intervals. These details are particularly important because the conclusions depend on the supplied cost matrix and operating-point selection. The clinical matrix is also approximately symmetric, so the similar row/column results provide limited evidence about the proposed orientation trade-off for substantially asymmetric costs. Resolving the reporting inconsistencies and providing a fully specified protocol are necessary to assess the claimed benefits.
Questions for authors
See weaknesses.