acceptodds
Under review as a conference paper at ICLR 2027

Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Abstract

Neural networks (NNs), while highly flexible and versatile, are often poorly calibrated. Post-hoc calibration methods adjust the outputs of an already trained model using a separate calibration dataset and additional calibration parameters. This process adds overhead that requires separate optimization of the calibrator and subsequent model selection. In this paper, we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation. This formulation eliminates the need for a separate calibration set or a dedicated calibration loss by jointly optimizing accuracy and calibration using cross-entropy, introducing only one additional hyperparameter: the calibrator learning rate. Although our framework supports calibrators ranging from temperature and matrix scaling to NN layers, our results show that using the final classification layer itself as the calibrator yields accurate, well-calibrated models with no additional parameters. Our approach outperforms SOTA calibrators with minimal added complexity, while also avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Calibration, Classification, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F9naOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This proposed COG, which reframes post-hoc calibration as a bilevel (Stackelberg) optimization problem: the backbone is the follower and the calibrator is the leader. Instead of fitting the calibrator on held-out data after training, COG updates both sets of parameters jointly. Experiments report competitive or improved accuracy, NLL and Brier score relative to temperature scaling, matrix scaling, label smoothing.

Critical strengths
  1. COG-FC amounts to a few lines of code, requires no held-out calibration set, and adds one hyperparameter.
  2. Interpreting post-hoc calibration as a decoupled approximation of a bilevel problem is a clean conceptual framing. It places TS, MS and TST in a common view.
Critical weaknesses
  1. The mechanism behind is not explained. It is unclear why slowing the last layer improves calibration.

  2. The experimental setup is not fair across methods.

    • Baselines are trained differently from COG. Per App. B.2, baselines use a learning rate of 1e-4, whereas COG uses λf\lambda_f = 1e-3 (or 5e-4/1e-2) together with a step scheduler (÷10 every 10 epochs). The reported gains may therefore stem from the learning-rate magnitude or schedule rather than from the proposed formulation.
    • COG receives more tuning than the baselines. COG learning rates are tuned per dataset and per variant (Tab. 8), while baselines use a single fixed learning rate and TST uses default settings.
    • Post-hoc calibrators are disadvantaged. The CIFAR-100 validation split is only 5% (about 2,500 samples), which is small for fitting matrix scaling with 10,100 parameters. The reported "failure to converge" may reflect overfitting or an unregularized implementation rather than an inherent limitation of post-hoc calibration.
  3. Ablation and sensitivity analyses are missing. The paper introduces a single new hyperparameter, yet never studies its effect. In particular:

    • No matched baseline. There is no run with λc=λf\lambda_c = \lambda_f under the identical optimizer, scheduler and Brier-based checkpoint selection. This is the most direct test of whether the leader–follower separation matters.
    • No sensitivity to the learning-rate ratio. A sweep over λc/λf∈{1,0.5,0.1,0.05,0.01}\lambda_c/\lambda_f \in \{1, 0.5, 0.1, 0.05, 0.01\} would show whether calibration improves monotonically as the ratio decreases, and how robust the method is to this choice.
    • No ablation of other training choices. The roles of the step scheduler, Brier-based early stopping and the absolute value of λf\lambda_f are not isolated.
    • No test of the bilevel interpretation. For example, the leader could be updated on a held-out batch (Dcal≠Dtrain\mathcal{D}_{cal} \neq \mathcal{D}_{train}).
  4. ECE, a central calibration metric, is excluded from the main evaluation. Although ECE has known limitations, it remains the most widely used calibration metric. This matters because the appendix results show essential calibration gap between COG and baseline methods.

Questions for authors
  1. Why is validation split different for cifar10/cifar100? The most common ratio should be 10%.
  2. What changes if the leader uses a held-out batch? This would test whether the bilevel interpretation matters.
  3. Is calibration monotone in λc/λf\lambda_c/\lambda_f? How sensitive are results to this ratio?
5 comments

We thank the reviewer for their detailed feedback and for recognizing the conceptual simplicity and practical efficiency of COG. We have substantially strengthened the revised manuscript with additional experiments, ablation studies, and theoretical clarifications that directly address the concerns regarding soundness, experimental fairness, and the role of the leader–follower learning rate hierarchy. All changes are highlighted in different colour. These additions reinforce our central contribution: COG provides a principled, efficient, and broadly applicable framework for calibration, overcoming limitations of conventional post-hoc approaches with minimal changes to standard training. We address each concern below.

W1: Why slowing training improves calibration

As explained in the paragraph “From Post-hoc Calibration to COG,” intermediate representations of neural networks can already have strong discriminative power before the final classification layer. Building on this, COG treats the learned representation as the classifier and the final classification layer as the calibrator.

The intuition behind using a smaller learning rate for the final layer follows the two-timescale Stackelberg optimization framework of Fiez et al. (2020): when the feature extractor (follower) updates faster than the final layer (leader), the leader can approximately treat the follower as being at equilibrium. This allows the final layer to adapt to a relatively stable representation, mimicking the behavior of post-hoc calibration while retaining joint optimization.

Specifically, COG-FC is closely related to TST: restricting TST's second-stage to a single linear layer yields an architecture equivalent to COG-FC, albeit trained in two stages rather than jointly.

W2.1 and W2.2 and Q1: Experimental Fairness and Setup

Our experimental setup follows that of TST (Jordahn, M., & Olmos, P. M. (2024)), which accounts for the different validation split used for CIFAR-100. By adopting TST's experimental settings, we also use the hyperparameters tuned by its authors, which we refer to as the default. For the other baselines, we use the default hyperparameter settings commonly adopted in prior work on these datasets.

Nevertheless, to address the concern about unequal tuning budgets, we conducted additional experiments across multiple learning rates (or learning-rate ratios for COG), both with and without learning rate scheduling. While COG generally benefits from scheduling (see Section 3), the results confirm that COG outperforms temperature scaling on average across all evaluated configurations. These additional results are included in Figures 10 and 11 of Appendix F in the revised manuscript.

W2.3: Data size for post-hoc calibrators

We disagree that post-hoc calibrators are disadvantaged. Unlike COG, which performs calibration during training using only the training data, post-hoc methods have access to an additional held-out calibration set. Nevertheless, calibrators whose parameter count grows with the number of classes face inherent scalability challenges. For example, Matrix Scaling requires over one million parameters on ImageNet (1,000 classes), while one-vs-rest approaches require 1,000 separate calibrators [1]. Moreover, the risk of overfitting in post-hoc calibration is well documented [1–4], as already discussed in our manuscript.

We acknowledge that the limited calibration-set size may contribute to the poor performance of Matrix Scaling on CIFAR-100. However, this illustrates a practical limitation of parameter-intensive post-hoc methods rather than an unfair experimental comparison. COG addresses this limitation by integrating calibration into training without requiring additional held-out calibration data.

W3.1 and W3.2, partial W3.3 and Q3: Ablations of the two learning rates

We have added ablation studies for both COG-FC and COG-TS in Appendix E.1 of the revised manuscript, varying the leader-to-follower learning rate ratio, both with and without learning rate schedulers.

When the learning rates are equal (λc=λf\lambda_c = \lambda_f), COG reduces to the baseline model trained with learning rate λc\lambda_c, providing a matched baseline under identical optimization settings. As the ratio λc/λf\lambda_c / \lambda_f increases, performance generally deteriorates, particularly when the ratio exceeds 1, regardless of scheduling.

These results confirm the importance of a slower-updating leader, although the relationship between the learning rate ratio and performance is not strictly monotonic. The results with scheduling are shown below, with the full ablations reported in Appendix E.1.

RatioSVHNCIFAR-10CIFAR-100
ACCBrierACCBrierACCBrier
0.010.9600.0010.960_{0.001}0.0630.0010.063_{0.001}0.8990.0010.899_{0.001}0.1560.0020.156_{0.002}0.7060.0030.706_{0.003}0.4080.0040.408_{0.004}
0.10.9590.0010.959_{0.001}0.0630.0010.063_{0.001}0.9010.0030.901_{0.003}0.1560.0040.156_{0.004}0.7050.0050.705_{0.005}0.4160.0060.416_{0.006}
1.00.9580.0020.958_{0.002}0.0650.0030.065_{0.003}0.8960.0030.896_{0.003}0.1650.0030.165_{0.003}0.6660.0030.666_{0.003}0.4630.0060.463_{0.006}
10.00.9550.0020.955_{0.002}0.0700.0030.070_{0.003}0.8670.0130.867_{0.013}0.2030.0150.203_{0.015}0.6500.0110.650_{0.011}0.4940.0070.494_{0.007}

W3.3: Ablation of other training choices

We would like to clarify that all methods in our experiments (including all COG variants, the baseline, label smoothing, and (V)TST) use the Brier score for checkpoint selection, as it can favor better-calibrated models. As noted in [5]:

“Brier score, another proper loss, is less sensitive than logloss to sharp variations in calibration error and can be a good alternative for early stopping for practitioners who want to avoid fitting TS every epoch.”

Furthermore, according to Fiez et. al. (2020), the scheduler can help with the convergence of our method.

In App. E.1 we have added an ablation study examining the effect of the absolute value of λf\lambda_f in COG-FC. The results show only minor variations in accuracy across the tested values, suggesting that COG is robust to this choice.

W3.4 and Q2: COG using held-out data

Updating the leader on a separate held-out batch requires additional forward passes, increasing computational overhead and undermining COG's key advantage: improved calibration with minimal changes to standard training.

Nevertheless, we have added an ablation using held-out data for the leader update in Appendix E.2 of the revised PDF. The results show comparable performance to the original COG formulation, confirming that separate leader and follower datasets provide no clear empirical advantage while increasing computational cost.

W4: Choice of calibration metrics

ECE has well-documented limitations as a calibration metric. This is why we already report ECE, RMSCE, and MCE using multiple binning configurations. We already provided a brief discussion of ECE's limitations in Appendix C.1 and have now summarized this discussion to the main manuscript in the updated version (Paragraph “Metrics.” lines 275-285). As highlighted by several recent works since 2020, there is no consensus on how calibration should be properly evaluated [1]: “Second, there is no consensus on how to properly evaluate calibration: commonly used metrics such as Expected Calibration Error (ECE) are known to be sensitive to design choices, making comparisons unreliable”.

[1] Berta, Eugène, David Holzmüller, Francis Bach, and Michael I. Jordan. "CalArena: A Large-Scale Post-Hoc Calibration Benchmark." In Proceedings of the Fortieth Annual Conference on Neural Information Processing Systems, 2026.
[2] Berta, Eugène, David Holzmüller, Michael I. Jordan, and Francis Bach. “Structured Matrix Scaling for Multi-Class Calibration.” In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, 2026
[3] Murphy, Kevin P. Probabilistic machine learning: Advanced topics. MIT press, 2023.
[4] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
[5] Berta, Eugène, David Holzmüller, Michael I. Jordan, and Francis Bach. "Rethinking early stopping: Refine, then calibrate." arXiv preprint arXiv:2501.19195 (2025).

Thank you to the author for the explanation and the supplementary experiments; most of my concerns have been addressed. However, I have one additional question that I hadn’t mentioned before: The reported accuracies are well below standard results e.g., ~90% vs ~95% for CIFAR10 and ~66% vs ~78% for CIFAR100?

Reviewer_7MofOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training procedure names Calibration in One Go (COG), which jointly optimizes a classifier and a calibration component using same cross-entropy objective and minibatches and has a smaller learning rate for it. The method is motivated through a bilevel/Stackelberg formulation. COG-FC is the simplest variant of it and use the final classification layer as the calibrator, so no more model parameters are used in it. Image classification, OOD detection, distribution shifts and CelebA multi-attribute prediction constitute the evaluation part. The result suggests that this method leads to a great improvement in some settings of predictive and calibration-related metrics. And the benefits may vary owing to different datasets or distribution shifts.

Critical strengths
  1. Easy to practice. No more model parameters and separate post-hoc calibration fitting stage are used in COG-FC, and its parameter-group learning rates setting leads to easy use in existing training pipelines.

  2. Comprehensive empirical evaluation. The paper evaluates different calibrator variants, datasets, backbones, OOD detection and different multi-attribute classifications. We can also see many results with multiple random seeds and additional calibration metrics, and the accuracy result in CIFAR-10 improves from 65.04% to 70.46% and reduces NLL from 2.98 to 1.08, which is really promising.

Critical weaknesses
  1. The results on CIFAR-10-C need more discussion. On CIFAR-10-C (Table 4), COG-FC reduces NLL relative to the baseline but decreases accuracy from 74.28% to 68.80% and worsens the Brier score from 0.45 to 0.50. These results make it difficult to draw a clear conclusion about the benefit under distribution shift.

  2. Couldn’t reconcile the follower objective in Eq. (1) with the update in Eq. (2). As written, the follower objective in Eq. (1) doesn’t include the calibrator and is independent of its parameters, but Eq. (2) updates the follower through the composed calibrated model. Please state the intended follower objective explicitly and explain how it leads to the update.

Questions for authors
  1. Does the improvement remain when uses the same optimizer, scheduler, checkpoint-selection criterion, and a comparable search budget?

  2. What is the intended follower objective, given the difference between Eqs. (1) and (2)?

  3. Which assumptions and convergence result apply to the actual shared-objective updates?

2 comments

We thank the reviewer for their constructive feedback. We address their concerns below and provide a revised PDF, showing changes in a different colour, with additional experiments and clarifications that further strengthen our main contribution: COG offers a principled, efficient, and broadly applicable approach to calibration, addressing fundamental limitations of post-hoc methods with minimal changes to standard training.

W1: CIFAR-10-C results

We thank the reviewer for highlighting this point. COG targets improved calibration, rather than explicitly optimizing robustness under distribution shift. On CIFAR-10-C, COG-FC substantially improves NLL (2.03 → 1.57), despite lower accuracy and a higher Brier score. These metrics capture different aspects of predictive performance: NLL is particularly sensitive to overconfident errors, whereas accuracy measures classification correctness and Brier score captures squared probability errors. Thus, improvements in NLL need not translate into uniform gains across metrics under distribution shift.

Importantly, on SVHN-C, COG-FC improves both NLL and Brier score, while Tab. 3 further demonstrates improved OOD detection on SVHN and CIFAR-10. These results support the benefits of COG for uncertainty estimation beyond the training distribution, without claiming uniform robustness improvements. We have clarified this distinction in the revised manuscript.

W2 and Q2: Follower's objective in COG

In Eq. (1), we describe standard post-hoc calibration, where the calibrator is optimized independently of the model parameters, thereby decoupling the leader and follower optimization problems.

In contrast, COG explicitly couples both optimization problems by incorporating the calibrator into the follower's training objective. The resulting bilevel formulation is now explicitly stated in Eq. (2) of the revised PDF, with the corresponding parameter updates given in Eq. (3). Crucially, the follower optimizes the loss of the calibrated model, rather than the uncalibrated model, capturing the interaction between the calibrator and the backbone.

This addition clarifies the theoretical formulation underlying COG and its distinction from standard post-hoc calibration.

Q1: Comparison under similar budget

We clarify that all methods (COG variants, baseline, label smoothing, and (V)TST) use the Adam optimizer and select checkpoints based on the Brier score, as already specified in the Appendix B.

To ensure a fair comparison, we conducted additional experiments across multiple learning rates (or learning-rate ratios for COG), both with and without learning rate scheduling. While COG generally benefits from scheduling (see our response to Q3), the results confirm that COG outperforms temperature scaling on average across all evaluated configurations.

These additional results are included in Figures 10 and 11 of Appendix F in the revised manuscript.

Q3: Assumptions and convergence of COG

Our convergence analysis follows Fiez et al. (2020), whose stochastic convergence guarantee requires the leader-to-follower learning rate ratio to vanish asymptotically, allowing the follower to approach equilibrium faster than the leader. We have clarified this condition in the revised manuscript.

To empirically assess its importance, we added ablations of the learning rate ratio and scheduler for COG-FC and COG-TS in Appendix E.1 (Figures 8 and 9). The results show that, e.g., in COG-FC, violating the learning rate hierarchy (λc>λf\lambda_c > \lambda_f) degrades performance, consistent with the theoretical analysis. A subset of the COG-FC results with scheduling is summarized below.

RatioSVHNCIFAR-10CIFAR-100
ACCBrierACCBrierACCBrier
0.010.9600.0010.960_{0.001}0.0630.0010.063_{0.001}0.8990.0010.899_{0.001}0.1560.0020.156_{0.002}0.7060.0030.706_{0.003}0.4080.0040.408_{0.004}
0.10.9590.0010.959_{0.001}0.0630.0010.063_{0.001}0.9010.0030.901_{0.003}0.1560.0040.156_{0.004}0.7050.0050.705_{0.005}0.4160.0060.416_{0.006}
1.00.9580.0020.958_{0.002}0.0650.0030.065_{0.003}0.8960.0030.896_{0.003}0.1650.0030.165_{0.003}0.6660.0030.666_{0.003}0.4630.0060.463_{0.006}
10.00.9550.0020.955_{0.002}0.0700.0030.070_{0.003}0.8670.0130.867_{0.013}0.2030.0150.203_{0.015}0.6500.0110.650_{0.011}0.4940.0070.494_{0.007}

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback. In our rebuttal, we have addressed all your comments by providing additional clarifications and new experimental results. We have also incorporated the corresponding revisions into the manuscript, with all changes highlighted in blue.

As we are approaching the end of the discussion period, we would appreciate it if you could let us know whether any questions or concerns remain that we could further address. If all concerns have been resolved, we would be grateful if you could reconsider your assessment in light of our responses and the submitted summary of changes to the manuscript.

Thank you for your time and consideration.

Reviewer_Fvy4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COG (Calibration in One Go), which reinterprets post-hoc calibration as a bilevel (Stackelberg) optimization problem in which the backbone is the follower and the calibrator is the leader. The authors argue that standard post-hoc calibration is a decoupled approximation of this bilevel problem and that this decoupling is only justified when the training problem has a unique solution, which is not the case for neural networks. COG instead updates backbone and calibrator simultaneously using the same cross-entropy loss, the same minibatch, and a single forward/backward pass, with the only modification to standard training being a separate, smaller learning rate for the calibrator (λ_c < λ_f). Any post-hoc calibrator can be dropped into this scheme (COG-TS, COG-MS, COG-TST).

Critical strengths
  • Proposed method is simple to implement and requires no held-out calibration set, no calibration-specific loss, and no second training stage.

  • The bilevel framing is a clean conceptual lens that unifies post-hoc calibration, TST, and progressive freezing (PLP) as special cases of one problem, and the "decoupling is only optimal under uniqueness" argument is a sensible motivation.

Critical weaknesses
  • Scale and scope are limited to 32×32 datasets trained from scratch; no pretrained/fine-tuning, larger-resolution, or non-vision setting is tested, although the conclusion claims modality independence.

  • The most comparable baselines — single-stage trainable calibration losses (ESD, MMCE, focal) — are absent. Can at least one be added, since label smoothing is not calibration-specific?

Questions for authors
  • Missing related work: ESD (Yoon et al., ICLR 2023), MMCE (Kumar et al., ICML 2018), and other trainable calibration objectives are neither cited nor compared.
  • "PyTorch version 2.11" — please confirm the version string.
  • What happens when λ_c = λ_f or λ_c > λ_f? A sweep of the ratio would show whether the leader/follower hierarchy itself matters.
  • On ViT, COG-FC has the best NLL/Brier but ECE 7.70 vs 1.65 for TS. How do the authors interpret this?
2 comments

We thank the reviewer for their constructive feedback. We address their concerns below and provide a revised PDF, showing changes in a different colour, with additional experiments and clarifications that further strengthen our main contribution: COG offers a principled, efficient, and broadly applicable approach to calibration, addressing fundamental limitations of post-hoc methods with minimal changes to standard training.

W1: Scope of experiments and different modality

We initially focused on standard datasets and settings in classifier calibration research. To address the reviewer's concern, we have extended our evaluation to tabular data using SDSS17, a dataset used in [1,2], with ~78K samples, 11 features, and 3 classes. The results, partially reported below and in the new paragraph “Calibrating tabular data.” of Section 4.2, show that COG-FC achieves the best overall ranking and that COG variants outperform their counterparts, demonstrating the effectiveness of COG beyond vision tasks.

MethodACCNLLBrier
COG-FC96.83070.026896.8307_{0.0268}0.10980.0010.1098_{0.001}0.05210.00020.0521_{0.0002}
Temp. Scaling96.81610.049096.8161_{0.0490}0.11240.00110.1124_{0.0011}0.05370.00040.0537_{0.0004}

W2 and Q1: Additional Train time calibration baselines

We have added ESD and MMCE to the related work section. We have also incorporated ESD into our experimental evaluation, since its the newest work.

In Tab.1 we first provide a qualitative comparison between ESD and our approach, highlighting several practical differences:

  1. Additional data requirements: ESD uses held-out data at each training epoch, requiring additional data and increasing the computational cost per epoch.
  2. ESD introduces a regularization weight that needs to be tuned.
  3. ESD has an O(B2)O(B^2) computational complexity, where (B) denotes the batch size.
  4. Memory requirements: Since ESD jointly updates the model parameters using two different data batches, both computational graphs need to be kept during backpropagation, leading to increased memory consumption.

To provide a quantitative comparison, we have added experiments with both ESD and its COG counterpart (COG+ESD). The results are reported below and included in Tab. 13. Across the three datasets, ESD generally performs worse than both COG+ESD and COG-FC, with COG-FC outperforming the other two.

MethodSVHNCIFAR-10CIFAR-100
ACCBrierACCBrierACCBrier
COG-FC95.840.2095.84_{0.20}0.060.000.06_{0.00}89.940.2589.94_{0.25}0.160.000.16_{0.00}70.460.370.46_{0.3}0.410.000.41_{0.00}
COG+ESD94.620.8094.62_{0.80}0.080.010.08_{0.01}87.670.8187.67_{0.81}0.180.010.18_{0.01}69.190.6769.19_{0.67}0.420.010.42_{0.01}
ESD94.770.4194.77_{0.41}0.080.010.08_{0.01}86.701.3086.70_{1.30}0.200.020.20_{0.02}59.421.7059.42_{1.70}0.580.010.58_{0.01}

Q2: PyTorch version

We clarified the PyTorch version by providing the full version string: PyTorch 2.11.0 with CUDA 12.6.

Q3: On the two learning rates of COG

When the two learning rates are equal (i.e., λc=λf\lambda_c = \lambda_f), COG reduces to the baseline model, with a learning rate of λc\lambda_c. However, according to Fiez et al. (2020), the theoretical convergence guarantee requires (λc<λf\lambda_c < \lambda_f).

To address the reviewer's question, we have added an ablation study sweeping the learning rate ratio (λc/λf\lambda_c / \lambda_f). The results, partially summarized below and presented in full in Appendix E.1, confirm that violating the required learning rate condition leads to worse performance, providing empirical evidence that the leader-follower learning rate hierarchy is important in COG.

RatioSVHNCIFAR-10CIFAR-100
ACCBrierACCBrierACCBrier
0.010.9600.0010.960_{0.001}0.0630.0010.063_{0.001}0.8990.0010.899_{0.001}0.1560.0020.156_{0.002}0.7060.0030.706_{0.003}0.4080.0040.408_{0.004}
0.10.9590.0010.959_{0.001}0.0630.0010.063_{0.001}0.9010.0030.901_{0.003}0.1560.0040.156_{0.004}0.7050.0050.705_{0.005}0.4160.0060.416_{0.006}
1.00.9580.0020.958_{0.002}0.0650.0030.065_{0.003}0.8960.0030.896_{0.003}0.1650.0030.165_{0.003}0.6660.0030.666_{0.003}0.4630.0060.463_{0.006}
10.00.9550.0020.955_{0.002}0.0700.0030.070_{0.003}0.8670.0130.867_{0.013}0.2030.0150.203_{0.015}0.6500.0110.650_{0.011}0.4940.0070.494_{0.007}

Q4: ECE and ViT

ECE has well-documented limitations as a calibration metric. We already provided a brief discussion of ECE's limitations in Appendix C.1 and have now summarized this discussion to the main manuscript in the updated version (Paragraph “Metrics.” lines 275-285). As discussed by several recent works, there is no consensus on how calibration should be properly evaluated [1]: “Second, there is no consensus on how to properly evaluate calibration: commonly used metrics such as Expected Calibration Error (ECE) are known to be sensitive to design choices, making comparisons unreliable”.

[1] Berta, Eugène, David Holzmüller, Francis Bach, and Michael I. Jordan. "CalArena: A Large-Scale Post-Hoc Calibration Benchmark." In Proceedings of the Fortieth Annual Conference on Neural Information Processing Systems, 2026.
[2] Erickson, Nick, et al. "Tabarena: A living benchmark for machine learning on tabular data." Advances in Neural Information Processing Systems 38 (2026).

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback. In our rebuttal, we have addressed all your comments by providing additional clarifications and new experimental results. We have also incorporated the corresponding revisions into the manuscript, with all changes highlighted in blue.

As we are approaching the end of the discussion period, we would appreciate it if you could let us know whether any questions or concerns remain that we could further address. If all concerns have been resolved, we would be grateful if you could reconsider your assessment in light of our responses and the submitted summary of changes to the manuscript.

Thank you for your time and consideration.

Comments on the paper

1 comment

We thank all reviewers for their careful reading, constructive feedback, and the time invested in evaluating our work. We have addressed each comment in detail in our individual responses.

The added improvements further clarify and strengthen our central contribution: Calibration in One Go (COG), provides a principled, efficient, and broadly applicable framework for calibration, with minimal changes to standard training. COG adopts a bilevel Stackelberg formulation to jointly optimize the classifier (follower) and calibrator (leader). It requires no additional calibration data or calibration-specific loss, and only one additional hyperparameter: a smaller calibrator learning rate (λc<λf\lambda_c < \lambda_f). COG supports existing post-hoc calibrators (COG-TS, COG-MS, COG-TST), while COG-FC uses the final classification layer as the calibrator, adding no trainable parameters.

Changes in the manuscript are highlighted in blue.. We summarize below the corresponding changes made to the manuscript:

Related Works

  • Added ESD (Yoon et al., ICLR 2023), MMCE (Kumar et al., ICML 2018) - (Reviewer Fvy4)

Methodology

  • Added qualitative comparison of COG and ESD in Tab. 1 (Reviewer Fvy4)
  • Clarified and provided intuition on the importance of the ratio of the two learning rates in COG (Reviewers Fvy4, 7Mof), and how it relates to the convergence of COG (Reviewer 7Mof)
  • Explicitly stated the objective of COG in Eq. 2 (Reviewer 7Mof)
  • Rephrased to clarify why COG-FC helps with calibration (Reviewer F9na)

Experiments

  • Added ESD and its COG counterpart to the experiments of Section 4.2 (Reviewer Fvy4)
  • Added a summary of Appendix C to Section 4 to clarify the selection of metrics (Reviewer F9na)
  • Clarified the interpretation of the results in CIFAR-10-C (Reviewer 7Mof)
  • Added an experiment on calibrating for a different modality (tabular data) in Section 4.2 under the paragraph "Calibrating tabular data." (Reviewer Fvy4)

Appendix

  • Added an ablation on the learning rate ratio of COG in Appendix E.1 under paragraph "Learning Rate Ratio λc/λf\lambda_c/\lambda_f." (Reviewers Fvy4, F9na)
  • Added an ablation where no learning rate scheduler is used in Appendix E.1 under paragraph "Effect of the scheduler." (Reviewers 7Mof, F9na)
  • Added an ablation on the follower's learning rate, in Appendix E.1 under paragraph "Effect of λf\lambda_f". (Reviewer F9na)
  • Added an ablation where the leader in COG is trained on held-out data, in Appendix E.2 (Reviewer F9na)
  • Added a comparison of COG and baselines under similar search budget in Appendix F (Reviewers 7Mof, F9na)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.