The Probabilistic Concordance Correlation Coefficient: A Chance-Corrected Performance Metric for Classification Models

Evaluation of probabilistic classifiers commonly considers discrimination and calibration. The area under the receiver operating characteristic curve summarizes ranking performance but is insensitive to calibration, whereas calibration measures such as expected calibration error do not quantify case-specific predictive information. The Brier score provides a strictly proper assessment of overall probabilistic accuracy, but its magnitude depends on class prevalence, case mix, and irreducible uncertainty, limiting its interpretation across prediction problems. We propose the Probabilistic Concordance Correlation Coefficient (P-CCC), ${\rho }_P$, a metric for evaluating probabilistic classifiers through chance-corrected agreement between predicted probability vectors and realized class labels. P-CCC extends the repeated-measures concordance correlation coefficient, originally developed for repeated continuous measurements, to prediction-label agreement on the probability simplex. Equivalently, P-CCC is one minus the Brier score divided by the error the same forecaster would incur against independent labels. This representation yields a bounded scale and a prior-independent zero point, with P-CCC interpreted as the proportion of model-specific chance-level error removed through prediction-label dependence. We derive an exact $O(nK)$ U-statistic estimator, establish asymptotic normality, and provide confidence intervals. Theoretical analysis shows that P-CCC penalizes uninformative predictive variation but may favor overconfident forecasts and is therefore not a proper scoring rule. Simulations and benchmark experiments demonstrate that P-CCC provides information not captured by AUC, expected calibration error, or the Brier score. We recommend reporting P-CCC alongside the Brier score as complementary measures of chance-corrected prediction-label agreement and overall probabilistic accuracy, respectively.