- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the Probabilistic Concordance Correlation Coefficient (P-CCC) for evaluating probabilistic classifiers. For a predicted probability vector and one-hot label , the coefficient is a trace-based concordance statistic:
{\operatorname{tr}(\operatorname{Cov}(p)) + \operatorname{tr}(\operatorname{Cov}(q)) + \lVert\mathbb{E}[p]-\mathbb{E}[q]\rVert^2}.$$ Equivalently, it is 1 minus the (multiclass) Brier score divided by the squared error obtained by pairing predictions with independent labels having the same marginal distribution. The authors derive bounds, relationships to the conventional Brier skill score and calibrated forecasters, an $O(nK)$ representation of a leave-one-out U-statistic-based estimator, asymptotic inference, and a prevalence-weighted extension. They also show analytically and empirically that P-CCC is improper: it may reward overconfident probabilities. Synthetic studies and experiments on five tabular/standard datasets compare its rankings and recalibration responses with Brier score, AUC, ECE, and log loss. The proposed use is as an additional descriptive diagnostic alongside a proper scoring rule, not as a standalone training or model-selection objective.Critical strengths
-
Clean and interpretable mathematical characterization. The equivalence between P-CCC and a Brier score normalized by an independence baseline is a concise bridge between agreement statistics and probabilistic prediction. The calibrated-forecast identity, makes the behavior of the coefficient easy to interpret and sharply delineates where it may add information beyond Brier score.
-
Careful analysis of an important failure mode. The authors do not market this coefficient as a proper score. Proposition 2 and the temperature-scaling experiments show concretely why optimizing concordance can favor overconfident forecasts. This self-critical treatment is valuable and helps prevent misuse.
Critical weaknesses
-
Limited novelty and unresolved practical added value. The central statistic is an application of the existing trace-based concordance correlation coefficient to one-hot outcomes, and its main skill-score identity is a short second-moment expansion. In the binary case it is exactly Lin’s scalar CCC; moreover, for hard one-hot predictions it reduces at the population level to Cohen’s : if is classification agreement and , then . The paper cites as an analogy but does not establish this exact special case or distinguish its contribution fully from this existing family. More importantly, ranking differences are not themselves evidence that the new ranking is useful. Equation (4) shows that calibrated models have the same ranking as Brier score; differences are driven by miscalibration, where the coefficient can favor worse predictions. In Table 1, model D has a worse Brier score (0.523) than the uninformative model B (0.500) and the oracle A (0.413), yet P-CCC ranks D (0.384) above A (0.295). The paper needs a compelling use case in which reporting P-CCC changes a sound diagnostic or scientific conclusion beyond existing metrics.
-
Experimental evidence does not sufficiently isolate or validate the proposed diagnostic. The real-data evaluation uses only five small/standard datasets, and the key observations are ranking correlations or movements under recalibration rather than a demonstrated downstream benefit. As acknowledged in §8, Platt and isotonic “recalibration” refit the base estimator, changing AUC and confounding the interpretation of changes in P-CCC. No paired uncertainty analysis is shown for the highlighted model comparisons, even though some held-out sets contain only 36 or 114 examples (Table A4). The run totals also do not reconcile: §8 reports 116 total model/treatment combinations and 87 recalibrations, whereas Table A4 lists 30 raw models; . In addition, Algorithm 1 claims complexity, which applies to the unweighted or diagonal-weight case, not an arbitrary dense positive-definite , which generally requires arithmetic. These issues appear fixable, but weaken confidence in the claimed empirical added value.
Questions for authors
-
Novelty and actionable use. Can the authors make explicit the exact Cohen’s- reduction for hard one-hot predictions, and explain what is substantively new relative to classical CCC/ and normalized Brier scores? Is there a concrete setting where adding P-CCC to (Brier, AUC, ECE) enables a better-grounded diagnostic decision, rather than simply preferring a miscalibrated, sharpened model? A positive example would materially affect my assessment.
-
Controlled empirical evidence. Can the authors report calibration experiments in which all recalibrators operate on fixed, precomputed base-model predictions (with no base-model refitting), together with paired bootstrap or jackknife uncertainty for the specific ranking reversals claimed to be informative? How often is a P-CCC disagreement both statistically distinguishable and practically meaningful?
-
Experimental bookkeeping and baselines. Please reconcile the 116 total combinations, 87 recalibrations, and 30 raw fits in Table A4, specifying any excluded model/treatment. Also, is the “lazy” forecaster B in §7.1 based on prevalence estimated from the evaluation labels? If so, please replace it with a training-only or known-population prior to avoid evaluation-label leakage in that example.