acceptodds
Withdrawn from ICLR 2027

Reliability-Aware Checkpoint Selection for Domain Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable probabilities on unseen target domains. Source–target distribution shifts can alter accuracy rankings, while accuracy alone does not measure predictive probability quality. We identify an empirical selection opportunity within fixed training trajectories: reselecting among checkpoints with near-optimal source accuracy can improve mean target probability quality with small observed changes in mean target accuracy. We study accuracy-constrained reliability selection (AC), which retains checkpoints within a tolerance of the best source-validation accuracy and ranks them by source reliability. Our reference rule aggregates within-set normalized negative log-likelihood (NLL) and class-wise calibration error (CwECE) using \(D_\infty\). AC uses no target data and requires neither additional training nor weight averaging. We evaluate five DG training algorithms on three benchmarks, using PACS to develop the objectives and a 0.5-percentage-point tolerance. In exploratory aggregation comparisons on 360 OfficeHome and TerraIncognita runs, the reference rule reduces mean target soft-bin squared-gap ECE and CwECE by 0.240 and 0.182 on the 100 scale, respectively, and NLL by 0.030 relative to Source-Acc. Mean target accuracy changes by +0.213 percentage points. These results identify opportunities for reliability-aware reselection, while the additional benefit of joint over single-objective ranking remains unresolved.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
domain generalization, checkpoint selection, predictive reliability, confidence calibration, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3hxrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies reliability-aware checkpoint selection for domain generalization. The proposed method, AC, first restricts checkpoints to those within a tolerance of the best source-validation accuracy and then selects a checkpoint using source reliability metrics. The reference method combines normalized NLL and CwECE through a maximum-distance criterion. Experiments on three DG benchmarks suggest improvements in target probability quality compared with accuracy-based checkpoint selection, without additional training.

Critical strengths
  1. The paper addresses a practical problem in domain generalization. Checkpoint selection is often overlooked, despite its impact on deployment.

  2. The method is simple and inexpensive to implement, requiring neither target-domain data nor additional training.

  3. The experimental evaluation is fairly extensive, and the authors are careful in distinguishing exploratory findings from confirmed results.

Critical weaknesses
  1. The methodological contribution is limited. Accuracy-filtered calibration-based model selection has already been considered in prior work, particularly Wald et al. (2021). The proposed method mainly adapts this idea to checkpoint selection within a fixed training trajectory.

  2. The improvement over Source-Acc is relatively modest. On the 360 post-development runs, the mean reductions in ECE and CwECE are approximately 0.24 and 0.18 percentage points, respectively. The practical significance of these improvements is not entirely clear.

  3. The motivation for the specific reference rule is not sufficiently strong. The experiments do not establish a clear advantage of jointly ranking NLL and CwECE over the corresponding single-objective alternatives. Although the authors acknowledge this limitation, it leaves the added value of the proposed aggregation rule uncertain.

Wald, Y., Feder, A., Greenfeld, D., & Shalit, U. (2021). On calibration and out-of-domain generalization. Advances in Neural Information Processing Systems (NeurIPS), 34, 2215–2227.

Questions for authors

1. In Section 3.2, NLL and CwECE are normalized using the minimum and maximum values within the feasible checkpoint set. When only two checkpoints are eligible, small differences in the raw metrics can lead to very different normalized scores. How stable is the selection under small perturbations of the validation metrics? Would normalization using statistics from all saved checkpoints produce different selections?

2. The tolerance δ=0.5\delta=0.5 was selected using PACS. However, Table 4 shows that increasing δ\delta to 1.0 further reduces the average reliability errors. What is the justification for retaining 0.5 as the reference choice? Is there a criterion for choosing δ\delta automatically using source-validation data?

3. Table 2 shows that AC-NC performs similarly to AC-NLL and AC-CwECE. How often does the joint rule select a checkpoint that neither single-objective rule selects? In these cases, does the joint rule provide any consistent advantage on the target domain?

4. In the temperature-scaling experiment (Table 5), the differences between Source-Acc and AC-NC become very small after calibration, and AC-NC has slightly worse NLL and CwECE. Does this suggest that much of the improvement from checkpoint reselection could also be achieved through ordinary post-hoc calibration? It would be helpful to clarify when checkpoint reselection offers a distinct practical advantage.

5. The reported confidence intervals are based on run-level bootstrap resampling, although runs share datasets, held-out domains, and hyperparameter configurations. How sensitive are the conclusions to this dependence? For example, would resampling at the held-out-domain or configuration level lead to similar intervals?

Reviewer_FjLmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies checkpoint selection for DG. Instead of picking the checkpoint with the best source-validation accuracy, all checkpoints within δ = 0.5 points of the best source accuracy are kept, and then the one with the best source reliability is chosen.

Critical strengths
  1. The development (PACS) and evaluation datasets is separated, which is a good practise for DG.
  2. The paper reports multiple ablations and large set of evaluation (540 runs) is reported.
Critical weaknesses
  1. As acknowledged, the accuracy-filtered calibration selection principle is already done by Wald et al. (2021). The paper simply add more parts such as joint NLL and CwECE score, normalization, D∞ aggregation. However the ablation itself show that they do not help. AC-NC is not significantly better than single-objective rules (Table 2), and D1 and D∞ differ in only 19 of 540 runs (Table E.2).
  2. The accuracy change of 0.21 pp is not considered that significant. And AC-NC reduces ECE by only 0.24 and NLL by 0.03- such gains are too small to matter in practice.
  3. Only older DG algorithms and three small benchmarks are used. DomainNet, VLCS, and modern methods (SWAD, MIRO, CLIP-based) are missing.
  4. The theoretical results are simple and do not explain why AC should help on the target.

Minor

  1. In table. 4, the AC-NC, delta 0.5 is better than 1.0 in terms of Acc but is bolded incorrectly.
  2. Presentation and writing is not proper.
Questions for authors
  1. What is the benefit of the joint NC rule over simply using AC-NLL, given that all differences are within the confidence intervals?
  2. Why should a practitioner use AC instead of checkpoint-SWAD or temperature scaling, which give larger improvements? And is it possible to combine AC with checkpoint-SWAD, for example by using AC to choose the averaging window.
  3. How does AC perform on larger benchmarks such as DomainNet, and with modern DG methods?
Reviewer_fG2AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies checkpoint selection for domain generalization. It retains checkpoints with near-best source validation accuracy, then ranks them using NLL and class-wise calibration error on the same validation data. Experiments show that this strategy can improve calibration on target domains.

Critical strengths

1 The method selects an existing checkpoint using only source validation data, without target data or additional training.

2 Experiments on OfficeHome and TerraIncognita show lower mean target calibration error after checkpoint reselection.

Critical weaknesses

1 The technical novelty and contribution require clarification. Wald et al. (NeurIPS 2021, Section 5.1) already proposed selecting the model with the lowest average source ECE subject to a validation-accuracy threshold. This paper applies this idea to checkpoints within a training trajectory, but Table 2 does not show a clear advantage of joint NLL–CwECE ranking over single-objective ranking.

2 The additional benefit of reliability-based ranking over simple checkpoint reselection is not sufficiently supported. Table 3 reports AC-early and AC-mean only on PACS, which was used for method development. These controls are not reported for OfficeHome or TerraIncognita, making it unclear whether the gains come from reliability-based ranking or simply from reselecting a near-optimal checkpoint.

Questions for authors

Could the authors report AC-mean and AC-early results on OfficeHome and TerraIncognita using the existing trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.