- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This proposed COG, which reframes post-hoc calibration as a bilevel (Stackelberg) optimization problem: the backbone is the follower and the calibrator is the leader. Instead of fitting the calibrator on held-out data after training, COG updates both sets of parameters jointly. Experiments report competitive or improved accuracy, NLL and Brier score relative to temperature scaling, matrix scaling, label smoothing.
Critical strengths
- COG-FC amounts to a few lines of code, requires no held-out calibration set, and adds one hyperparameter.
- Interpreting post-hoc calibration as a decoupled approximation of a bilevel problem is a clean conceptual framing. It places TS, MS and TST in a common view.
Critical weaknesses
-
The mechanism behind is not explained. It is unclear why slowing the last layer improves calibration.
-
The experimental setup is not fair across methods.
- Baselines are trained differently from COG. Per App. B.2, baselines use a learning rate of 1e-4, whereas COG uses = 1e-3 (or 5e-4/1e-2) together with a step scheduler (÷10 every 10 epochs). The reported gains may therefore stem from the learning-rate magnitude or schedule rather than from the proposed formulation.
- COG receives more tuning than the baselines. COG learning rates are tuned per dataset and per variant (Tab. 8), while baselines use a single fixed learning rate and TST uses default settings.
- Post-hoc calibrators are disadvantaged. The CIFAR-100 validation split is only 5% (about 2,500 samples), which is small for fitting matrix scaling with 10,100 parameters. The reported "failure to converge" may reflect overfitting or an unregularized implementation rather than an inherent limitation of post-hoc calibration.
-
Ablation and sensitivity analyses are missing. The paper introduces a single new hyperparameter, yet never studies its effect. In particular:
- No matched baseline. There is no run with under the identical optimizer, scheduler and Brier-based checkpoint selection. This is the most direct test of whether the leader–follower separation matters.
- No sensitivity to the learning-rate ratio. A sweep over would show whether calibration improves monotonically as the ratio decreases, and how robust the method is to this choice.
- No ablation of other training choices. The roles of the step scheduler, Brier-based early stopping and the absolute value of are not isolated.
- No test of the bilevel interpretation. For example, the leader could be updated on a held-out batch ().
-
ECE, a central calibration metric, is excluded from the main evaluation. Although ECE has known limitations, it remains the most widely used calibration metric. This matters because the appendix results show essential calibration gap between COG and baseline methods.
Questions for authors
- Why is validation split different for cifar10/cifar100? The most common ratio should be 10%.
- What changes if the leader uses a held-out batch? This would test whether the bilevel interpretation matters.
- Is calibration monotone in ? How sensitive are results to this ratio?
5 comments
W3.1 and W3.2, partial W3.3 and Q3: Ablations of the two learning rates
We have added ablation studies for both COG-FC and COG-TS in Appendix E.1 of the revised manuscript, varying the leader-to-follower learning rate ratio, both with and without learning rate schedulers.
When the learning rates are equal (), COG reduces to the baseline model trained with learning rate , providing a matched baseline under identical optimization settings. As the ratio increases, performance generally deteriorates, particularly when the ratio exceeds 1, regardless of scheduling.
These results confirm the importance of a slower-updating leader, although the relationship between the learning rate ratio and performance is not strictly monotonic. The results with scheduling are shown below, with the full ablations reported in Appendix E.1.
| Ratio | SVHN | CIFAR-10 | CIFAR-100 | |||
|---|---|---|---|---|---|---|
| ACC | Brier | ACC | Brier | ACC | Brier | |
| 0.01 | ||||||
| 0.1 | ||||||
| 1.0 | ||||||
| 10.0 |
W3.3: Ablation of other training choices
We would like to clarify that all methods in our experiments (including all COG variants, the baseline, label smoothing, and (V)TST) use the Brier score for checkpoint selection, as it can favor better-calibrated models. As noted in [5]:
“Brier score, another proper loss, is less sensitive than logloss to sharp variations in calibration error and can be a good alternative for early stopping for practitioners who want to avoid fitting TS every epoch.”
Furthermore, according to Fiez et. al. (2020), the scheduler can help with the convergence of our method.
In App. E.1 we have added an ablation study examining the effect of the absolute value of in COG-FC. The results show only minor variations in accuracy across the tested values, suggesting that COG is robust to this choice.
W3.4 and Q2: COG using held-out data
Updating the leader on a separate held-out batch requires additional forward passes, increasing computational overhead and undermining COG's key advantage: improved calibration with minimal changes to standard training.
Nevertheless, we have added an ablation using held-out data for the leader update in Appendix E.2 of the revised PDF. The results show comparable performance to the original COG formulation, confirming that separate leader and follower datasets provide no clear empirical advantage while increasing computational cost.
W4: Choice of calibration metrics
ECE has well-documented limitations as a calibration metric. This is why we already report ECE, RMSCE, and MCE using multiple binning configurations. We already provided a brief discussion of ECE's limitations in Appendix C.1 and have now summarized this discussion to the main manuscript in the updated version (Paragraph “Metrics.” lines 275-285). As highlighted by several recent works since 2020, there is no consensus on how calibration should be properly evaluated [1]: “Second, there is no consensus on how to properly evaluate calibration: commonly used metrics such as Expected Calibration Error (ECE) are known to be sensitive to design choices, making comparisons unreliable”.
[1] Berta, Eugène, David Holzmüller, Francis Bach, and Michael I. Jordan. "CalArena: A Large-Scale Post-Hoc Calibration Benchmark." In Proceedings of the Fortieth Annual Conference on Neural Information Processing Systems, 2026.
[2] Berta, Eugène, David Holzmüller, Michael I. Jordan, and Francis Bach. “Structured Matrix Scaling for Multi-Class Calibration.” In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, 2026
[3] Murphy, Kevin P. Probabilistic machine learning: Advanced topics. MIT press, 2023.
[4] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
[5] Berta, Eugène, David Holzmüller, Michael I. Jordan, and Francis Bach. "Rethinking early stopping: Refine, then calibrate." arXiv preprint arXiv:2501.19195 (2025).
Thank you to the author for the explanation and the supplementary experiments; most of my concerns have been addressed. However, I have one additional question that I hadn’t mentioned before: The reported accuracies are well below standard results e.g., ~90% vs ~95% for CIFAR10 and ~66% vs ~78% for CIFAR100?
We thank the reviewer for their detailed feedback and for recognizing the conceptual simplicity and practical efficiency of COG. We have substantially strengthened the revised manuscript with additional experiments, ablation studies, and theoretical clarifications that directly address the concerns regarding soundness, experimental fairness, and the role of the leader–follower learning rate hierarchy. All changes are highlighted in different colour. These additions reinforce our central contribution: COG provides a principled, efficient, and broadly applicable framework for calibration, overcoming limitations of conventional post-hoc approaches with minimal changes to standard training. We address each concern below.
W1: Why slowing training improves calibration
As explained in the paragraph “From Post-hoc Calibration to COG,” intermediate representations of neural networks can already have strong discriminative power before the final classification layer. Building on this, COG treats the learned representation as the classifier and the final classification layer as the calibrator.
The intuition behind using a smaller learning rate for the final layer follows the two-timescale Stackelberg optimization framework of Fiez et al. (2020): when the feature extractor (follower) updates faster than the final layer (leader), the leader can approximately treat the follower as being at equilibrium. This allows the final layer to adapt to a relatively stable representation, mimicking the behavior of post-hoc calibration while retaining joint optimization.
Specifically, COG-FC is closely related to TST: restricting TST's second-stage to a single linear layer yields an architecture equivalent to COG-FC, albeit trained in two stages rather than jointly.
W2.1 and W2.2 and Q1: Experimental Fairness and Setup
Our experimental setup follows that of TST (Jordahn, M., & Olmos, P. M. (2024)), which accounts for the different validation split used for CIFAR-100. By adopting TST's experimental settings, we also use the hyperparameters tuned by its authors, which we refer to as the default. For the other baselines, we use the default hyperparameter settings commonly adopted in prior work on these datasets.
Nevertheless, to address the concern about unequal tuning budgets, we conducted additional experiments across multiple learning rates (or learning-rate ratios for COG), both with and without learning rate scheduling. While COG generally benefits from scheduling (see Section 3), the results confirm that COG outperforms temperature scaling on average across all evaluated configurations. These additional results are included in Figures 10 and 11 of Appendix F in the revised manuscript.
W2.3: Data size for post-hoc calibrators
We disagree that post-hoc calibrators are disadvantaged. Unlike COG, which performs calibration during training using only the training data, post-hoc methods have access to an additional held-out calibration set. Nevertheless, calibrators whose parameter count grows with the number of classes face inherent scalability challenges. For example, Matrix Scaling requires over one million parameters on ImageNet (1,000 classes), while one-vs-rest approaches require 1,000 separate calibrators [1]. Moreover, the risk of overfitting in post-hoc calibration is well documented [1–4], as already discussed in our manuscript.
We acknowledge that the limited calibration-set size may contribute to the poor performance of Matrix Scaling on CIFAR-100. However, this illustrates a practical limitation of parameter-intensive post-hoc methods rather than an unfair experimental comparison. COG addresses this limitation by integrating calibration into training without requiring additional held-out calibration data.