- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates multiclass calibration through conditional moments and proposes D-GCM, a training regularizer that learns a bounded critic on predicted probability vectors. Its central contribution is replacing squared minibatch moments with a U-statistic, removing diagonal self-interactions that reward residual energy even under perfect calibration. The theory establishes unbiasedness for fixed witnesses and characterizes calibration when the witness family is sufficiently rich. Experiments on three image benchmarks and 20 Newsgroups report the lowest ECE and SCE among the compared calibration regularizers, with an accuracy cost relative to focal loss. Ablations examine debiasing and critic adaptivity, while additional evaluations assess computational overhead, binning sensitivity, and compatibility with temperature scaling.
Critical strengths
-
I particularly value the demonstration that the naive squared batch moment rewards residual energy even under perfect calibration. Removing its diagonal contribution addresses a specific, mathematically identifiable bias. This matters because the estimator is itself optimized: its bias can change what the critic learns to detect.
-
The comparisons between learned and fixed witnesses, and between U- and V-statistics, directly examine the method’s two main design choices. Together with calibration improvements across four benchmarks and low measured training overhead, these experiments provide a credible practical case for the regularizer.
Critical weaknesses
-
Unbiasedness is established for a fixed, sample-wise witness. When the critic is updated and then used for the model update on the same minibatch, it can adapt to sampling noise; removing diagonal terms alone does not eliminate this effect. Appendix A acknowledges this limitation, but no independent-batch control quantifies its practical importance. The unspecified handling of BatchNorm in the ResNet experiments further obscures whether the assumptions hold. This weakens the connection between the theoretical justification and the implemented algorithm.
-
ECE evaluates confidence calibration, while SCE evaluates individual probability coordinates; neither directly measures calibration conditional on the entire probability vector. Moreover, the comparisons omit a full-simplex KCE/SKCE baseline, despite its close theoretical relationship. A held-out multivariate calibration assessment and a comparable expressive fixed-kernel baseline would help establish whether adaptive witnesses provide the claimed additional benefit. Without these comparisons, that central empirical claim remains insufficiently isolated.
Questions for authors
- In the ResNet experiments, were BatchNorm statistics frozen when computing the regularizer? Were the critic and classifier updated using the same minibatch? Please specify the forward passes and batch reuse, since these determine which unbiasedness assumptions apply to the reported results.
- How large is the gap between critic objectives evaluated on the critic’s update batch and on an independent batch? Do the calibration gains persist when critic and classifier updates use independent batches under comparable computation? This would clarify the practical effect of adaptation to batch noise.
- Do the gains persist under a held-out calibration measure that evaluates the full probability vector, and against an expressive fixed full-simplex KCE/SKCE baseline? These comparisons would materially affect my assessment of the benefit attributable to adaptive witnesses.
Response to reviewer 7DNq
Thank you for the encouraging and precise review. The answers below follow your three questions and rely on a new controlled study on CIFAR-100/ResNet-56.
In the ResNet experiments, BatchNorm statistics were not frozen, and the critic and the classifier were updated on the same mini-batch. Each iteration runs one training-mode forward pass that is shared by the classification loss and the regularizer. The critic takes one ascent step on the detached probabilities, and the model step then backpropagates through both the residual and the witness with the critic frozen.
The diagonal terms are removed exactly, as CU in Eq. (7) contains no i=j term for any batch or critic. The unbiasedness of Proposition 3 applies to a fixed witness and a sample-wise predictor, and the controls below test these two conditions. We will state this beside Eq. (10).
We agree that batch reuse needs a direct check. The objective gap you ask about is small next to the diagonal term that the U-statistic removes. With each trained D-GCM-Critic classifier frozen, we draw 200 pairs of disjoint validation batches (A, B), restore the saved critic and its optimizer state, and take one ascent step on A. After the step, CU(A) exceeds CU(B) by 0.032±0.035, of which the step itself contributes 0.019±0.018 (in units of 10−3, SD over three classifiers). Proposition 1 gives a diagonal reward of up to E∥R∥2/B≈3 in the same units at B=128 (E∥R∥2 is the Brier score, about 0.4), about 100 times larger.
The training controls are in the table. All rows share Focal Loss (γ=3) and the schedule of Table 9, and all D-GCM-Critic rows (D-GCM below) share one λ and the default critic settings. "Frozen BN" computes the regularizer from a second forward pass with the running statistics saved before the current batch. "Lagged critic" applies the critic from before its update on the current batch, at the same cost, and "independent batch" fits the critic on a disjoint auxiliary batch. Values are mean ± SD over 3 seeds.
The protocol of the paper keeps the accuracy of Focal Loss and lowers ECE and SCE at a cost of 1.6s per epoch, and freezing BN keeps most of the mean gain. The gain does depend on same-batch training: with a lagged or independent critic, ECE stays at or slightly below the Focal Loss level, accuracy is 0.6 to 1.3 points lower than with same-batch training, and SCE is higher. Same-batch training of the critic is therefore the most effective variant, and we will present it as part of the algorithm with Proposition 3 as the fixed-witness statement.
Thank you for asking about the full probability vector. On a held-out study of the full probability vector, D-GCM with the paper's protocol is at the same level with Focal Loss and with a fixed kernel trained with an SKCE regularizer. It improves ECE and SCE over Focal Loss, and accuracy, ECE and SCE over the fixed kernel.
Test SKCE uses one Gaussian mixture kernel for all methods, with bandwidths from the median heuristic on Focal Loss validation predictions. The fixed kernel baseline uses a Gaussian mixture kernel (bandwidths 0.2, 0.5, 1.0) under the same base loss, schedule and λ search as D-GCM. D-GCM reaches an SKCE of 0.034, against 0.036 for Focal Loss and 0.034 for the fixed kernel, with the smallest variation across seeds, and the frozen BN variants have higher SKCE. Against the fixed kernel, D-GCM is more accurate in every seed (by 1.7 to 1.9 points on average) and has lower mean ECE and SCE, while the fixed kernel leaves SCE at the Focal Loss level.
The adaptive witness therefore improves the accuracy/calibration trade-off, at comparable SKCE under the paper's protocol. We will state the empirical claim in these terms, in line with the restricted-family scope of Proposition 2, and add these controls to the revision. Because every row uses default settings, absolute values differ from the tuned results in Table 2.