acceptodds
Under review as a conference paper at ICLR 2027

Debiased Generalized Calibration Moments

Abstract

Modern neural networks can be accurate but poorly calibrated. We revisit train-time multiclass calibration through a generalized moment view that unifies binning, mean-matching, and kernel objectives as fixed witness function choices on the probability simplex, and we ask whether the witness itself can be learned. We propose Debiased Generalized Calibration Moments (D-GCM), a lightweight min-max regularizer that trains a small critic on predicted probabilities to expose miscalibration patterns. The key challenge is empirical estimation: the naive squared batch moment is a biased V-statistic whose diagonal self-interaction term biases the objective for any fixed witness and becomes an optimization shortcut for a learned critic, rewarding sample-wise residual energy even when the target calibration moment is zero. D-GCM removes this shortcut with the corresponding U-statistic over distinct sample pairs. We theoretically show that rich witness families characterize strong multiclass calibration, while restricted families control only the miscalibration components they can approximate. Empirically, the debiased learned-critic objective achieves the best calibration among train-time calibration regularizers on four image and text benchmarks while maintaining competitive prediction accuracy, adds modest computational overhead, and remains complementary to post-hoc temperature scaling.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multiclass calibration, witness function

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7DNqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates multiclass calibration through conditional moments and proposes D-GCM, a training regularizer that learns a bounded critic on predicted probability vectors. Its central contribution is replacing squared minibatch moments with a U-statistic, removing diagonal self-interactions that reward residual energy even under perfect calibration. The theory establishes unbiasedness for fixed witnesses and characterizes calibration when the witness family is sufficiently rich. Experiments on three image benchmarks and 20 Newsgroups report the lowest ECE and SCE among the compared calibration regularizers, with an accuracy cost relative to focal loss. Ablations examine debiasing and critic adaptivity, while additional evaluations assess computational overhead, binning sensitivity, and compatibility with temperature scaling.

Critical strengths
  1. I particularly value the demonstration that the naive squared batch moment rewards residual energy even under perfect calibration. Removing its diagonal contribution addresses a specific, mathematically identifiable bias. This matters because the estimator is itself optimized: its bias can change what the critic learns to detect.

  2. The comparisons between learned and fixed witnesses, and between U- and V-statistics, directly examine the method’s two main design choices. Together with calibration improvements across four benchmarks and low measured training overhead, these experiments provide a credible practical case for the regularizer.

Critical weaknesses
  1. Unbiasedness is established for a fixed, sample-wise witness. When the critic is updated and then used for the model update on the same minibatch, it can adapt to sampling noise; removing diagonal terms alone does not eliminate this effect. Appendix A acknowledges this limitation, but no independent-batch control quantifies its practical importance. The unspecified handling of BatchNorm in the ResNet experiments further obscures whether the assumptions hold. This weakens the connection between the theoretical justification and the implemented algorithm.

  2. ECE evaluates confidence calibration, while SCE evaluates individual probability coordinates; neither directly measures calibration conditional on the entire probability vector. Moreover, the comparisons omit a full-simplex KCE/SKCE baseline, despite its close theoretical relationship. A held-out multivariate calibration assessment and a comparable expressive fixed-kernel baseline would help establish whether adaptive witnesses provide the claimed additional benefit. Without these comparisons, that central empirical claim remains insufficiently isolated.

Questions for authors
  1. In the ResNet experiments, were BatchNorm statistics frozen when computing the regularizer? Were the critic and classifier updated using the same minibatch? Please specify the forward passes and batch reuse, since these determine which unbiasedness assumptions apply to the reported results.
  2. How large is the gap between critic objectives evaluated on the critic’s update batch and on an independent batch? Do the calibration gains persist when critic and classifier updates use independent batches under comparable computation? This would clarify the practical effect of adaptation to batch noise.
  3. Do the gains persist under a held-out calibration measure that evaluates the full probability vector, and against an expressive fixed full-simplex KCE/SKCE baseline? These comparisons would materially affect my assessment of the benefit attributable to adaptive witnesses.
1 comment
AuthorsOpenReview ↗

Response to reviewer 7DNq

Thank you for the encouraging and precise review. The answers below follow your three questions and rely on a new controlled study on CIFAR-100/ResNet-56.

1. Forward passes and batch reuse.

In the ResNet experiments, BatchNorm statistics were not frozen, and the critic and the classifier were updated on the same mini-batch. Each iteration runs one training-mode forward pass that is shared by the classification loss and the regularizer. The critic takes one ascent step on the detached probabilities, and the model step then backpropagates through both the residual and the witness with the critic frozen.

The diagonal terms are removed exactly, as C^U\widehat{\mathcal{C}}_{U} in Eq. (7) contains no i=j term for any batch or critic. The unbiasedness of Proposition 3 applies to a fixed witness and a sample-wise predictor, and the controls below test these two conditions. We will state this beside Eq. (10).

2. Same batch versus independent batches.

We agree that batch reuse needs a direct check. The objective gap you ask about is small next to the diagonal term that the U-statistic removes. With each trained D-GCM-Critic classifier frozen, we draw 200 pairs of disjoint validation batches (A, B), restore the saved critic and its optimizer state, and take one ascent step on A. After the step, C^U(A)\widehat{\mathcal{C}}_{U}(A) exceeds C^U(B)\widehat{\mathcal{C}}_{U}(B) by 0.032±0.0350.032\pm0.035, of which the step itself contributes 0.019±0.0180.019\pm0.018 (in units of 10−310^{-3}, SD over three classifiers). Proposition 1 gives a diagonal reward of up to E∥R∥2/B≈3\mathbb{E}\lVert R\rVert^{2}/B \approx 3 in the same units at B=128 (E∥R∥2\mathbb{E}\lVert R\rVert^{2} is the Brier score, about 0.4), about 100 times larger.

The training controls are in the table. All rows share Focal Loss (γ=3\gamma=3) and the schedule of Table 9, and all D-GCM-Critic rows (D-GCM below) share one λ\lambda and the default critic settings. "Frozen BN" computes the regularizer from a second forward pass with the running statistics saved before the current batch. "Lagged critic" applies the critic from before its update on the current batch, at the same cost, and "independent batch" fits the critic on a disjoint auxiliary batch. Values are mean ±\pm SD over 3 seeds.

SettingAcc (%)ECE (%)SCE (10−310^{-3})SKCE (10−310^{-3})s/epoch
Focal Loss71.00±0.6071.00\pm0.602.53±0.332.53\pm0.332.19±0.042.19\pm0.040.036±0.0100.036\pm0.01020.8
D-GCM, training BN (paper)71.00±0.3371.00\pm0.332.23±0.182.23\pm0.181.98±0.011.98\pm0.010.034±0.0030.034\pm0.00322.5
D-GCM, frozen BN70.83±0.7270.83\pm0.722.35±0.442.35\pm0.442.03±0.162.03\pm0.160.062±0.0180.062\pm0.01840.3
D-GCM, frozen BN, lagged critic70.21±0.6170.21\pm0.612.50±0.482.50\pm0.482.28±0.032.28\pm0.030.068±0.0060.068\pm0.00640.9
D-GCM, frozen BN, independent batch69.56±0.6069.56\pm0.602.41±0.592.41\pm0.592.25±0.042.25\pm0.040.084±0.0220.084\pm0.02245.7
Fixed kernel (SKCE), frozen BN69.14±0.1969.14\pm0.192.44±0.542.44\pm0.542.19±0.032.19\pm0.030.034±0.0070.034\pm0.00738.1

The protocol of the paper keeps the accuracy of Focal Loss and lowers ECE and SCE at a cost of 1.6s per epoch, and freezing BN keeps most of the mean gain. The gain does depend on same-batch training: with a lagged or independent critic, ECE stays at or slightly below the Focal Loss level, accuracy is 0.6 to 1.3 points lower than with same-batch training, and SCE is higher. Same-batch training of the critic is therefore the most effective variant, and we will present it as part of the algorithm with Proposition 3 as the fixed-witness statement.

3. Full probability vector and fixed kernel baseline.

Thank you for asking about the full probability vector. On a held-out study of the full probability vector, D-GCM with the paper's protocol is at the same level with Focal Loss and with a fixed kernel trained with an SKCE regularizer. It improves ECE and SCE over Focal Loss, and accuracy, ECE and SCE over the fixed kernel.

Test SKCE uses one Gaussian mixture kernel for all methods, with bandwidths from the median heuristic on Focal Loss validation predictions. The fixed kernel baseline uses a Gaussian mixture kernel (bandwidths 0.2, 0.5, 1.0) under the same base loss, schedule and λ\lambda search as D-GCM. D-GCM reaches an SKCE of 0.034, against 0.036 for Focal Loss and 0.034 for the fixed kernel, with the smallest variation across seeds, and the frozen BN variants have higher SKCE. Against the fixed kernel, D-GCM is more accurate in every seed (by 1.7 to 1.9 points on average) and has lower mean ECE and SCE, while the fixed kernel leaves SCE at the Focal Loss level.

The adaptive witness therefore improves the accuracy/calibration trade-off, at comparable SKCE under the paper's protocol. We will state the empirical claim in these terms, in line with the restricted-family scope of Proposition 2, and add these controls to the revision. Because every row uses default settings, absolute values differ from the tuned results in Table 2.

Reviewer_EGWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes train-time multiclass calibration using a learned witness on predicted probability vectors. It removes the diagonal residual-energy term from a squared minibatch moment and alternates critic and classifier updates using the resulting U-statistic. This extends the fixed-kernel training perspective of Kumar et al., Trainable Calibration Measures for Neural Networks from Kernel Mean Embeddings (2018). The paper gives population characterizations of calibration, an efficient implementation, and results on four classification benchmarks.

Critical strengths
  1. The analysis makes the diagonal contribution explicit, including its positive expectation under perfect calibration. Widmann et al., Calibration tests in multi-class classification: A unifying framework (2019), already provide unbiased kernel calibration estimators. The useful contribution here is the diagnosis of the diagonal reward when learning a witness, together with a simple implementation that avoids a batch-by-batch interaction matrix.

  2. Table 3 includes a useful learned/fixed-witness by V/U-statistic ablation. The appendix also reports seed variation, binning sensitivity, temperature scaling, and critic hyperparameter checks. These support the method's empirical promise, particularly on Tiny-ImageNet and 20 Newsgroups.

Critical weaknesses
  1. The quality of the trained critic remains insufficiently validated. For a fixed critic, Equation (7) is a kernel calibration U-statistic with k_psi(p,q)=phi_psi(p)^T phi_psi(q). Learning this kernel transfers the choice of geometry to the network class and its optimization. Dependence on kernel choice is critical, discussed in Huang et al., H-Calibration: Rethinking Classifier Recalibration With Probabilistic Error-Bounded Objective (2025). Theorem 1 requires a rich population witness family, while Proposition 2 assumes an approximation error that is not controlled for the implemented critic. The default single ascent step does not establish that the critic tracks a strong witness. Moreover, Proposition 3 is for a fixed witness, so it does not establish unbiasedness after fitting the critic on the same batch. Appendix acknowledges these limitations. The ECE curves and dimension/learning-rate sweeps are useful, but do not directly test what residual structure the trained critic detects on fresh data.

  2. The evaluation does not directly test the full-vector calibration target or establish the advantage over the closest full-multiclass alternatives. ECE and SCE inspect coarser conditional relationships. Marx et al., Calibration by Distribution Matching: Trainable Kernel Calibration Metrics (2023), is already discussed, but its differentiable MMD regularizer is missing from the experiments, as is a full-simplex KCE comparator.

  3. Matching the base loss and tuning budget would help isolate the benefit of learning the witness. This matters alongside the accuracy cost: relative to Focal Loss, D-GCM loses 2.43 points on CIFAR-100 and 2.99 on 20 Newsgroups.

Questions for authors
  1. With the classifier frozen, how does the online critic compare with independently fitted audit critics using more ascent steps and multiple initializations, evaluated on disjoint data? On synthetic calibrated and deliberately miscalibrated distributions, can it detect known full-vector residual patterns rather than fitting batch noise? Such evidence would be more informative than a general neural-network convergence proof.

  2. Which choices of critic capacity and regularization control the empirical-to-population gap? Please position these choices against Dikkala et al., Minimax Estimation of Conditional Moment Models (2020), and distinguish the calibration-specific contribution from the broader adversarial conditional-moment framework.

  3. Were BatchNorm statistics frozen when evaluating the ResNet regularizer, as required by the sample-wise assumptions discussed in Appendix B.9? A held-out full-vector audit and a matched kernel comparison would also help interpret the gains. Please distinguish improved ECE/SCE from evidence that the learned witness covers the full calibration residual.

2 comments
AuthorsOpenReview ↗

Response to reviewer EGWd (part 1/2)

Thank you for the detailed review. We agree that the diagnosis of the diagonal term for a learned witness and its efficient removal are the core of the paper. The controls you asked for are now complete on CIFAR-100/ResNet-56.

1. Matched kernel baseline and SKCE.

We agree that the comparison needs matched settings. Under a matched base loss, schedule and tuning budget, D-GCM-Critic (D-GCM below) is more accurate than a fixed kernel on the full probability vector and has lower ECE and SCE. All rows use Focal Loss (γ=3\gamma=3), the schedule of Table 9 and default critic settings, and D-GCM and the kernel baseline receive the same three-value λ\lambda search on validation data. Absolute values therefore differ from the tuned results in Table 2.

The baseline is the SKCE regularizer k(pi,pj)⋅ri⊤rjk(p_i,p_j)\cdot r_i^{\top}r_j with a Gaussian mixture kernel (bandwidths 0.2, 0.5, 1.0). With the label kernel 1{y=y′}1\{y=y^{\prime}\} and the forecast-sampled label integrated out, the MMD objective of Marx et al. (2023) for canonical calibration reduces to the same expression, so this row covers both baselines. Test SKCE uses one Gaussian mixture kernel for all methods, with bandwidths from the median heuristic on Focal Loss validation predictions.

Method (mean ±\pm SD, 3 seeds)Acc (%)ECE (%)SCE (10−310^{-3})SKCE (10−310^{-3})
Focal Loss71.00±0.6071.00\pm0.602.53±0.332.53\pm0.332.19±0.042.19\pm0.040.036±0.0100.036\pm0.010
D-GCM, training BN (paper)71.00±0.3371.00\pm0.332.23±0.182.23\pm0.181.98±0.011.98\pm0.010.034±0.0030.034\pm0.003
D-GCM, frozen BN70.83±0.7270.83\pm0.722.35±0.442.35\pm0.442.03±0.162.03\pm0.160.062±0.0180.062\pm0.018
Fixed kernel (SKCE), frozen BN69.14±0.1969.14\pm0.192.44±0.542.44\pm0.542.19±0.032.19\pm0.030.034±0.0070.034\pm0.007

"Training BN" is the protocol of the paper: BatchNorm statistics were not frozen, and the loss and the regularizer share one training-mode forward pass. "Frozen BN" computes the regularizer with the running statistics saved before the current batch (Appendix B.9). With the paper's protocol, D-GCM keeps the accuracy of Focal Loss, lowers ECE and SCE, and reaches the same SKCE as the kernel baseline, which loses 1.9 accuracy points and leaves SCE at the Focal Loss level.

With frozen BN, D-GCM remains 1.7 points more accurate than the fixed kernel with lower ECE and SCE, and its SKCE is higher. The benefit of the learned witness is therefore in accuracy, ECE and SCE, at an SKCE that matches the fixed kernel under the paper's protocol, and we will state the empirical claim in these terms. On the accuracy cost, the CIFAR-100 difference in Table 2 is 0.63 points (71.52 vs 70.89), next to 2.99 on 20 Newsgroups.

AuthorsOpenReview ↗

Response to reviewer EGWd (part 2/2)

2. Critic quality on fresh data.

The online critic matches independently fitted audit critics. With the classifier frozen, we fit audit critics of the same architecture for 2,000 ascent steps from five initializations, with disjoint validation halves for fitting and selection, and evaluate C^U\widehat{\mathcal{C}}_{U} (Eq. 7) on test data. For D-GCM with frozen BN, the online critic reaches 0.059±0.0260.059\pm0.026 and the audit critics 0.051±0.0310.051\pm0.031 (in units of 10−310^{-3}), so one ascent step per iteration tracks a witness that holds up on fresh data. Audits of Focal Loss and the fixed kernel give 0.040±0.0080.040\pm0.008 and 0.036±0.0080.036\pm0.008, the same ordering as SKCE.

On synthetic data with known residuals, the critic's objective is consistent with zero for a calibrated distribution, so batch noise alone produces no signal. For a residual that ECE cannot see (mass moved between the second and third-ranked classes), the critic recovers 94% of the oracle moment after 20,000 ascent steps, up from 2% at the initial 2,000 (the online critic takes about 70,000 steps). A parity residual invisible to both ECE and SCE is not recovered at either budget, and this limit of the critic will be reported with Proposition 2.

3. Capacity and positioning.

D-GCM applies the min–max conditional-moment template of Dikkala et al. (2020) to calibration. Their bounds depend on the critical radius of the test-function class, its approximation error, and a norm penalty. In D-GCM the witness family is controlled by the critic size (12.7K parameters), the unit-ball constraint of Eq. (9) in place of the norm penalty, and one Adam ascent step per iteration (learning rate 10−310^{-3}). With these choices, one ascent step raises the objective on its own batch over an independent batch by 0.019±0.0180.019\pm0.018 in the same units.

Two points are specific to calibration. The conditioning variable is the predictor's own output PθP_{\theta}, and the squared vector moment estimated on mini-batches rewards residual energy, up to E∥R∥2/B≈3\mathbb{E}\lVert R\rVert^{2}/B \approx 3 in the same units at B=128, even under perfect calibration (Proposition 1). The U-statistic removes this term exactly at O(BKd)O(BKd) cost, and the term is two orders of magnitude above the same-batch effect. The learned kernel kψ=ϕψ⊤ϕψk_{\psi}=\phi_{\psi}^{\top}\phi_{\psi} moves the choice of geometry from a bandwidth to the critic and its optimizer, which the audit and synthetic tests examine. We will add this positioning, the kernel dependence discussed by Huang et al. (2025), and all results above to the revision.

Reviewer_zsQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Debiased Generalized Calibration Moments (D-GCM), a train-time regularizer for multiclass calibration. Instead of using fixed calibration statistics, the method learns a critic on predicted probability vectors to identify calibration errors. The authors show that the naive squared batch moment includes diagonal terms that introduce bias and can provide a shortcut for the learned critic. They replace it with a U-statistic that removes these self-interactions. Experiments on four image and text classification datasets show improved calibration compared with several existing methods, with relatively small computational overhead.

Critical strengths
  1. The analysis of the diagonal terms in the V-statistic is interesting. It explains why a naive squared batch moment may be problematic when the witness is learned. The U-statistic provides a simple way to remove these terms, and the ablation in Table 3 supports this choice.

  2. D-GCM-Critic obtains the lowest reported ECE and SCE among the compared calibration methods on all four datasets. Table 4 also shows improvements when the regularizer is combined with different classification losses. The additional training cost appears to be small.

Critical weaknesses
  1. Proposition 3 shows that the U-statistic is unbiased for a fixed witness. However, the critic is trained on the same mini-batch used to evaluate the objective. Maximizing an unbiased estimator over a learned critic does not necessarily give an unbiased estimate of the population maximum. The authors mention this limitation in Appendix A, but I would like to understand how much it matters in practice. Removing the diagonal terms addresses one source of bias, but it does not completely resolve the issue introduced by learning the critic from finite batches.

  2. Proposition 3 shows that the U-statistic is unbiased for a fixed witness. However, the critic is trained on the same mini-batch used to evaluate the objective. Maximizing an unbiased estimator over a learned critic does not necessarily give an unbiased estimate of the population maximum. The authors mention this limitation in Appendix A, but I would like to understand how much it matters in practice. Removing the diagonal terms addresses one source of bias, but it does not completely resolve the issue introduced by learning the critic from finite batches.

Questions for authors
  1. How much does optimizing the critic on the same mini-batch affect the empirical U-statistic objective? Would the results change if the critic were evaluated on an independent batch? Also, Appendix B.9 notes that BatchNorm can violate the sample-wise assumption. How is this handled in the ResNet experiments?

  2. How sensitive is D-GCM-Critic to batch size? In particular, does the method still work well with smaller batches, where informative cross-sample pairs may be limited?

1 comment
AuthorsOpenReview ↗

Response to reviewer zsQe

Thank you for the careful review and for recognizing the diagonal-term analysis. Your questions led us to a new controlled study on CIFAR-100/ResNet-56, and we answer them in turn.

1. How much same-batch critic training matters.

Thank you for raising the same-batch effect concern. The U-statistic removes the diagonal terms exactly for any batch and any critic, including a critic fitted on that same batch, because C^U\widehat{\mathcal{C}}_{U} in Eq. (7) contains no i=j term. This is the meaning of "debiased" in the paper (Appendix B3), and Proposition 3 adds unbiasedness for a fixed witness. Fitting the critic on the same batch is a separate effect, noted in Appendix A, and we have now measured it.

With each trained D-GCM-Critic classifier frozen, we draw 200 pairs of disjoint validation batches (A, B), restore the saved critic and its optimizer state, and take one ascent step on A. After the step, C^U(A)\widehat{\mathcal{C}}_{U}(A) exceeds C^U(B)\widehat{\mathcal{C}}_{U}(B) by 0.032±0.0350.032\pm0.035, of which the step itself contributes 0.019±0.0180.019\pm0.018 (in units of 10−310^{-3}, SD over three classifiers).

For comparison, Proposition 1 gives a diagonal reward of up to E∥R∥2/B≈3\mathbb{E}\lVert R\rVert^{2}/B \approx 3 in the same units at B=128, where E∥R∥2\mathbb{E}\lVert R\rVert^{2} is the Brier score, about 0.4 here. The diagonal reward that the U-statistic removes can thus be about 100 times larger than the remaining same-batch effect.

2. BatchNorm and independent batches.

In the ResNet experiment, BatchNorm statistics were not frozen. The classification loss and the regularizer share one training-mode forward pass, the critic takes one ascent step on the detached probabilities of that batch, and the model step then uses the same batch with the critic frozen.

The table below tests both choices. All rows share Focal Loss (γ=3\gamma=3) and the schedule of Table 9, and all D-GCM-Critic rows (D-GCM below) share one λ\lambda and the default critic settings. "Frozen BN" computes the regularizer from a second forward pass that uses the running statistics saved before the current batch, which makes the predictor sample-wise (Appendix B.9). "Lagged critic" applies the critic from before its update on the current batch, and "independent batch" fit the critic on a disjoint auxiliary batch. Values are mean ±\pm SD over 3 seeds.

BSettingAcc (%)ECE (%)SCE (10−310^{-3})s/epoch
128Focal Loss71.00±0.6071.00\pm0.602.53±0.332.53\pm0.332.19±0.042.19\pm0.0420.8
128D-GCM, training BN (paper)71.00±0.3371.00\pm0.332.23±0.182.23\pm0.181.98±0.011.98\pm0.0122.5
128D-GCM, frozen BN70.83±0.7270.83\pm0.722.35±0.442.35\pm0.442.03±0.162.03\pm0.1640.3
128D-GCM, frozen BN, lagged critic70.21±0.6170.21\pm0.612.50±0.482.50\pm0.482.28±0.032.28\pm0.0340.9
128D-GCM, frozen BN, independent batch69.56±0.6069.56\pm0.602.41±0.592.41\pm0.592.25±0.042.25\pm0.0445.7
32Focal Loss67.12±0.2067.12\pm0.203.68±0.523.68\pm0.522.11±0.082.11\pm0.0882.7
32D-GCM, frozen BN66.89±0.6166.89\pm0.613.49±0.633.49\pm0.632.06±0.162.06\pm0.16100.6

The protocol of the paper keeps the accuracy of Focal Loss and lowers ECE from 2.53 to 2.23 and SCE from 2.19 to 1.98, with every D-GCM seed below every Focal Loss seed in SCE, at a cost of 1.6 s per epoch. Freezing BN keeps most of the mean gain (ECE 2.35, SCE 2.03), so the gain holds when the predictor is sample-wise.

Separating the critic batch from the model batch does change the results: ECE stays at or slightly below the Focal Loss level (2.50 and 2.41), accuracy is 0.6 to 1.3 points lower than with same-batch training, and SCE rises to 2.25–2.28. Same-batch training of the critic is therefore the most effective variant, at no extra cost. We will describe it as part of the algorithm beside Eq. (10) and keep Proposition 3 as the fixed-witness statement.

3. Batch size.

The mean gains persist at B=32. With the learning rate and the number of epochs unchanged (1,406 updates per epoch), both methods are less accurate than at B=128, and D-GCM keeps lower mean ECE and SCE than Focal Loss (3.49 vs 3.68 and 2.06 vs 2.11) with a 0.23-point accuracy difference. Small batches are also where the U-statistic matters most, because the diagonal term grows as 1/B (Eq.6) and reaches about 13×10−313\times10^{-3} at B=32.

This study uses default settings in every row so that only the protocol varies, which is why absolute values differ from the tuned results in Table 2. We will add the protocol description, the same-batch measurement and this table to the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

We thank the Area Chair and reviewers for their time and constructive feedback, and appreciate the reviewers’ recognition of the diagonal-term analysis, the supporting ablations, and the calibration improvements across four benchmarks.

Paper summary: We connect several existing calibration objectives through a common witness formulation and study a critic that learns to identify miscalibration from the full predicted probability vector. Our analysis shows that diagonal terms in squared minibatch moments can reward this critic even when the predictor is perfectly calibrated. D-GCM removes this diagonal shortcut exactly with an efficient U-statistic that avoids constructing a batch-by-batch interaction matrix.

Our additional experiments address the shared questions below, with detailed settings and results in the individual responses.

1. What does “debiased” guarantee when the critic is learned?

The diagonal terms are removed exactly for every batch and critic, including a critic fitted on that batch (Eq 8). The unbiasedness result applies to a fixed, sample-wise predictor and witness (Proposition 3), and our new controls separately measure the effect of fitting the critic on the evaluated batch.

2. How are BatchNorm and batch reuse handled?

The ResNet experiments use one shared forward pass with BatchNorm in training mode and update the critic and classifier on the same batch. In the new CIFAR-100/ResNet-56 controls, freezing BatchNorm preserves lower mean ECE and SCE than Focal Loss, and the same-batch update outperforms the lagged and independent-batch variants on mean accuracy, ECE and SCE.

3. What does the comparison with a full-vector calibration baseline show?

In the new study, the original D-GCM-Critic protocol lowers mean ECE and SCE at essentially the same accuracy as Focal Loss, and achieves similar mean held-out SKCE to the fixed-kernel regularizer with 1.86 percentage points higher accuracy. With BatchNorm frozen for both methods, D-GCM-Critic retains a 1.70-point accuracy advantage and lower mean ECE and SCE, while the fixed kernel has lower SKCE.

4. What do the independent critic audits establish?

At the frozen-BN checkpoints, independently fitted audit critics of the same architecture do not obtain a larger mean test objective than the saved online critic. The audit and synthetic checks evaluate the implemented critic within the restricted witness family discussed in Proposition 2, while Theorem 1 characterizes calibration with a sufficiently rich witness family.


We welcome remaining follow-up questions during discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.