acceptodds
Under review as a conference paper at ICLR 2027

The Probabilistic Concordance Correlation Coefficient: A Chance-Corrected Performance Metric for Classification Models

Abstract

Evaluation of probabilistic classifiers commonly considers discrimination and calibration. The area under the receiver operating characteristic curve summarizes ranking performance but is insensitive to calibration, whereas calibration measures such as expected calibration error do not quantify case-specific predictive information. The Brier score provides a strictly proper assessment of overall probabilistic accuracy, but its magnitude depends on class prevalence, case mix, and irreducible uncertainty, limiting its interpretation across prediction problems. We propose the Probabilistic Concordance Correlation Coefficient (P-CCC), , a metric for evaluating probabilistic classifiers through chance-corrected agreement between predicted probability vectors and realized class labels. P-CCC extends the repeated-measures concordance correlation coefficient, originally developed for repeated continuous measurements, to prediction-label agreement on the probability simplex. Equivalently, P-CCC is one minus the Brier score divided by the error the same forecaster would incur against independent labels. This representation yields a bounded scale and a prior-independent zero point, with P-CCC interpreted as the proportion of model-specific chance-level error removed through prediction-label dependence. We derive an exact U-statistic estimator, establish asymptotic normality, and provide confidence intervals. Theoretical analysis shows that P-CCC penalizes uninformative predictive variation but may favor overconfident forecasts and is therefore not a proper scoring rule. Simulations and benchmark experiments demonstrate that P-CCC provides information not captured by AUC, expected calibration error, or the Brier score. We recommend reporting P-CCC alongside the Brier score as complementary measures of chance-corrected prediction-label agreement and overall probabilistic accuracy, respectively.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
model evaluation, calibration, probabilistic classification, proper scoring rules, Brier score, concordance correlation coefficient, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uemJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Probabilistic Concordance Correlation Coefficient (P-CCC) for evaluating probabilistic classifiers. For a predicted probability vector pp and one-hot label qq, the coefficient is a trace-based concordance statistic:

{\operatorname{tr}(\operatorname{Cov}(p)) + \operatorname{tr}(\operatorname{Cov}(q)) + \lVert\mathbb{E}[p]-\mathbb{E}[q]\rVert^2}.$$ Equivalently, it is 1 minus the (multiclass) Brier score divided by the squared error obtained by pairing predictions with independent labels having the same marginal distribution. The authors derive bounds, relationships to the conventional Brier skill score and calibrated forecasters, an $O(nK)$ representation of a leave-one-out U-statistic-based estimator, asymptotic inference, and a prevalence-weighted extension. They also show analytically and empirically that P-CCC is improper: it may reward overconfident probabilities. Synthetic studies and experiments on five tabular/standard datasets compare its rankings and recalibration responses with Brier score, AUC, ECE, and log loss. The proposed use is as an additional descriptive diagnostic alongside a proper scoring rule, not as a standalone training or model-selection objective.
Critical strengths
  1. Clean and interpretable mathematical characterization. The equivalence between P-CCC and a Brier score normalized by an independence baseline is a concise bridge between agreement statistics and probabilistic prediction. The calibrated-forecast identity, ρP=2BSS1+BSS,\rho_P = \frac{2 \mathrm{BSS}}{1+\mathrm{BSS}}, makes the behavior of the coefficient easy to interpret and sharply delineates where it may add information beyond Brier score.

  2. Careful analysis of an important failure mode. The authors do not market this coefficient as a proper score. Proposition 2 and the temperature-scaling experiments show concretely why optimizing concordance can favor overconfident forecasts. This self-critical treatment is valuable and helps prevent misuse.

Critical weaknesses
  1. Limited novelty and unresolved practical added value. The central statistic is an application of the existing trace-based concordance correlation coefficient to one-hot outcomes, and its main skill-score identity is a short second-moment expansion. In the binary case it is exactly Lin’s scalar CCC; moreover, for hard one-hot predictions it reduces at the population level to Cohen’s κ\kappa: if PoP_o is classification agreement and Pe=∑kP(y^=k)P(y=k)P_e = \sum_k P(\hat y=k)P(y=k), then ρP=Po−Pe1−Pe\rho_P = \frac{P_o-P_e}{1-P_e}. The paper cites κ\kappa as an analogy but does not establish this exact special case or distinguish its contribution fully from this existing family. More importantly, ranking differences are not themselves evidence that the new ranking is useful. Equation (4) shows that calibrated models have the same ranking as Brier score; differences are driven by miscalibration, where the coefficient can favor worse predictions. In Table 1, model D has a worse Brier score (0.523) than the uninformative model B (0.500) and the oracle A (0.413), yet P-CCC ranks D (0.384) above A (0.295). The paper needs a compelling use case in which reporting P-CCC changes a sound diagnostic or scientific conclusion beyond existing metrics.

  2. Experimental evidence does not sufficiently isolate or validate the proposed diagnostic. The real-data evaluation uses only five small/standard datasets, and the key observations are ranking correlations or movements under recalibration rather than a demonstrated downstream benefit. As acknowledged in §8, Platt and isotonic “recalibration” refit the base estimator, changing AUC and confounding the interpretation of changes in P-CCC. No paired uncertainty analysis is shown for the highlighted model comparisons, even though some held-out sets contain only 36 or 114 examples (Table A4). The run totals also do not reconcile: §8 reports 116 total model/treatment combinations and 87 recalibrations, whereas Table A4 lists 30 raw models; 30+87=11730+87=117. In addition, Algorithm 1 claims O(nK)O(nK) complexity, which applies to the unweighted or diagonal-weight case, not an arbitrary dense positive-definite WW, which generally requires O(nK2)O(nK^2) arithmetic. These issues appear fixable, but weaken confidence in the claimed empirical added value.

Questions for authors
  1. Novelty and actionable use. Can the authors make explicit the exact Cohen’s-κ\kappa reduction for hard one-hot predictions, and explain what is substantively new relative to classical CCC/κ\kappa and normalized Brier scores? Is there a concrete setting where adding P-CCC to (Brier, AUC, ECE) enables a better-grounded diagnostic decision, rather than simply preferring a miscalibrated, sharpened model? A positive example would materially affect my assessment.

  2. Controlled empirical evidence. Can the authors report calibration experiments in which all recalibrators operate on fixed, precomputed base-model predictions (with no base-model refitting), together with paired bootstrap or jackknife uncertainty for the specific ranking reversals claimed to be informative? How often is a P-CCC disagreement both statistically distinguishable and practically meaningful?

  3. Experimental bookkeeping and baselines. Please reconcile the 116 total combinations, 87 recalibrations, and 30 raw fits in Table A4, specifying any excluded model/treatment. Also, is the “lazy” forecaster B in §7.1 based on prevalence estimated from the evaluation labels? If so, please replace it with a training-only or known-population prior to avoid evaluation-label leakage in that example.

Reviewer_HzdUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the probabilistic concordance correlation coefficient (P-CCC). The P-CCC is one minus the Brier score (the expected value with respect to the prediction and label of the squared Euclidian distance) divided by the expected value with respect to the prediction and an independent label of the squared Euclidean distance. The P-CCC has a chance-corrected zero point (like an AUROC of 0.5). The authors include experiments on a synthetic dataset and five public datasets. They ultimately recommend reporting both the Brier score and P-CCC.

Critical strengths
  • Overall, I like this paper quite a lot. The paper is well written. The proposed metric is principled, and the authors include theoretical properties (Property 1, Theorem 2, Theorem 3, and Proposition 2).
  • The authors demonstrate what P-CCC rewards. Proposition 2 shows that the P-CCC penalizes uninformative predictive variation but may favor overconfident predictions. This is visualized in Figure 2 where P-CCC prefers a temperature smaller than T=1T=1.
  • The authors extend their work to weighted P-CCC for class imbalance. This extension is valuable for communities like machine learning for healthcare where class imbalance is a common problem.
Critical weaknesses
  • The proposed method is not all that novel. The P-CCC is a special case of the Brier skill score (Brier, 1950) that can be expressed using Murphy’s decomposition (Murphy, 1973) as one minus the ratio of resolution over the sum of reliability and uncertainty.
  • The authors do not recommend replacing the Brier score with P-CCC. Brier score seems to be a better calibration metric and area under the receiver operator characteristic (AUROC) curve seems to be a better discrimination metric. Why should I report the Brier score and the P-CCC instead of the Brier score and AUROC?
Questions for authors
  • I found the expectation notation E∥p–q∥2\mathbb{E}\|p – q\|^2 confusing. An expectation should be taken over a probability distribution, such as Ep,q[∥p–q∥2]\mathbb{E}_{p, q}\left[ \|p – q\|^2 \right] or EX,Y[∥f(X)–Y∥2]\mathbb{E}_{X, Y}\left[ \|f(X) – Y\|^2 \right]. Since that probability distribution is not specified in the subscript, I found myself rereading the setup to remind myself what the random variable are.
  • Why are some results better than the Bayes estimator in Table 1?
Reviewer_61zXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Probabilistic Concordance Correlation Coefficient (P-CCC, ρP\rho_P), which adapts the trace-ratio concordance correlation coefficient (King et al., 2007) to evaluate agreement between predicted probability vectors p∈ΔK−1p \in \Delta^{K-1} and one-hot labels q∈{e1,…,eK}q \in \{e_1, \dots, e_K\}: ρP=2tr⁡(Σpq)tr⁡(Σp)+tr⁡(Σq)+∥μp−μq∥2=1−BS(p,q)BSindep(p,q)\rho_P = \frac{2\operatorname{tr}(\Sigma_{pq})}{\operatorname{tr}(\Sigma_p) + \operatorname{tr}(\Sigma_q) + \|\mu_p - \mu_q\|^2} = 1 - \frac{\text{BS}(p, q)}{\text{BS}_{\text{indep}}(p, q)} The authors derive an exact O(nK)O(nK) U-statistic estimator, establish asymptotic normality with jackknife Fisher-zz confidence intervals, and prove that ρP\rho_P is an improper metric that rewards overconfident sharpening of the true conditional probabilities (Proposition 2c). Because ρP\rho_P is improper, the authors recommend reporting (BS,ρ^P)(\text{BS}, \hat{\rho}_P) jointly as a diagnostic pair.

Critical strengths

Rigorous and Transparent Analysis of Incentives

Critical weaknesses
  1. Questionable Practical Value of an Improper Metric That Penalizes Calibration: The central issue is not mathematical correctness, but utility: why should practitioners adopt a probability evaluation metric that ranks an overconfident, miscalibrated distortion (T∗=0.28T^*=0.28, ECE=0.182\text{ECE}=0.182, ρP=0.387\rho_P=0.387) strictly higher than the true Bayes-optimal posterior (T=1T=1, ECE=0\text{ECE}=0, ρP=0.295\rho_P=0.295)?
    • By Eq. (4), among calibrated models, ρP=2 BSS1+BSS\rho_P = \frac{2\,\text{BSS}}{1+\text{BSS}} is a strictly monotone transformation of BSS (and BS), providing zero new ranking information.
    • Among miscalibrated models, ρP\rho_P deviates from BS/BSS precisely by rewarding overdispersion/overconfidence (increasing tr⁡(Σp)\operatorname{tr}(\Sigma_p) in the denominator of BS/BSindep\text{BS}/\text{BS}_{\text{indep}}). On the credit-g MLP benchmark (Sec. 8), temperature scaling improves Brier score from 0.476→0.3730.476 \to 0.373, yet ρP\rho_P drops from 0.339→0.2720.339 \to 0.272 because calibration softened the overconfident predictions. A metric that moves in the wrong direction when calibration improves is a liability in model evaluation tables.
  2. Redundancy with Classical Murphy / Brier Decompositions for Dispersion Diagnostics: The authors argue that reporting (BS,ρ^P)(\text{BS}, \hat{\rho}_P) together serves as a diagnostic check on whether a model is over- or under-dispersed relative to a calibrated curve ρP=2 BSS/(1+BSS)\rho_P = 2\,\text{BSS}/(1+\text{BSS}). However, if a practitioner already has BS\text{BS} and wants to diagnose calibration vs. dispersion/sharpness, the classical Murphy decomposition (BS=Reliability−Resolution+Uncertainty\text{BS} = \text{Reliability} - \text{Resolution} + \text{Uncertainty}) or simply reporting (BS,ECE,tr⁡(Σp))(\text{BS}, \text{ECE}, \operatorname{tr}(\Sigma_p)) directly isolates miscalibration and predictive variance without introducing an improper ratio where covariance and variance are entangled.
  3. Trivial Cancellation Failure Mode in Multiclass (tr⁡(Σpq)=0\operatorname{tr}(\Sigma_{pq}) = 0): As noted in Remark 1, because ρP\rho_P only looks at the diagonal sum tr⁡(Σpq)=∑k=1KCov⁡(pk,qk)\operatorname{tr}(\Sigma_{pq}) = \sum_{k=1}^K \operatorname{Cov}(p_k, q_k), a multiclass forecaster that permutes class labels or has negative covariance on some classes and positive on others can yield ρP=0\rho_P = 0 despite carrying strong predictive information.
Questions for authors
  1. Diagnostic Advantage over (BS,ECE,tr⁡(Sp))(\text{BS}, \text{ECE}, \operatorname{tr}(S_p)): Since ρP\rho_P is an algebraic function of (BS,tr⁡(Σp),tr⁡(Σq),∥μp−μq∥2)(\text{BS}, \operatorname{tr}(\Sigma_p), \operatorname{tr}(\Sigma_q), \|\mu_p - \mu_q\|^2), what concrete diagnostic insight does the pair (BS,ρ^P)(\text{BS}, \hat{\rho}_P) provide that is not more transparently shown by reporting (BS,BSS)(\text{BS}, \text{BSS}) alongside marginal bias ∥pˉ−qˉ∥2\|\bar{p} - \bar{q}\|^2 and sharpness tr⁡(Sp)\operatorname{tr}(S_p)—which do not suffer from rewarding overconfident forecasts?
  2. Deep Learning Benchmarks: Modern neural networks are famously overconfident out-of-the-box (Guo et al., 2017). Since ρP\rho_P rewards overconfidence (T<1T < 1), wouldn't reporting ρP\rho_P on modern deep classifiers systematically favor uncalibrated networks over temperature-scaled ones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.