- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies reliability-aware checkpoint selection for domain generalization. The proposed method, AC, first restricts checkpoints to those within a tolerance of the best source-validation accuracy and then selects a checkpoint using source reliability metrics. The reference method combines normalized NLL and CwECE through a maximum-distance criterion. Experiments on three DG benchmarks suggest improvements in target probability quality compared with accuracy-based checkpoint selection, without additional training.
Critical strengths
-
The paper addresses a practical problem in domain generalization. Checkpoint selection is often overlooked, despite its impact on deployment.
-
The method is simple and inexpensive to implement, requiring neither target-domain data nor additional training.
-
The experimental evaluation is fairly extensive, and the authors are careful in distinguishing exploratory findings from confirmed results.
Critical weaknesses
-
The methodological contribution is limited. Accuracy-filtered calibration-based model selection has already been considered in prior work, particularly Wald et al. (2021). The proposed method mainly adapts this idea to checkpoint selection within a fixed training trajectory.
-
The improvement over Source-Acc is relatively modest. On the 360 post-development runs, the mean reductions in ECE and CwECE are approximately 0.24 and 0.18 percentage points, respectively. The practical significance of these improvements is not entirely clear.
-
The motivation for the specific reference rule is not sufficiently strong. The experiments do not establish a clear advantage of jointly ranking NLL and CwECE over the corresponding single-objective alternatives. Although the authors acknowledge this limitation, it leaves the added value of the proposed aggregation rule uncertain.
Wald, Y., Feder, A., Greenfeld, D., & Shalit, U. (2021). On calibration and out-of-domain generalization. Advances in Neural Information Processing Systems (NeurIPS), 34, 2215–2227.
Questions for authors
1. In Section 3.2, NLL and CwECE are normalized using the minimum and maximum values within the feasible checkpoint set. When only two checkpoints are eligible, small differences in the raw metrics can lead to very different normalized scores. How stable is the selection under small perturbations of the validation metrics? Would normalization using statistics from all saved checkpoints produce different selections?
2. The tolerance was selected using PACS. However, Table 4 shows that increasing to 1.0 further reduces the average reliability errors. What is the justification for retaining 0.5 as the reference choice? Is there a criterion for choosing automatically using source-validation data?
3. Table 2 shows that AC-NC performs similarly to AC-NLL and AC-CwECE. How often does the joint rule select a checkpoint that neither single-objective rule selects? In these cases, does the joint rule provide any consistent advantage on the target domain?
4. In the temperature-scaling experiment (Table 5), the differences between Source-Acc and AC-NC become very small after calibration, and AC-NC has slightly worse NLL and CwECE. Does this suggest that much of the improvement from checkpoint reselection could also be achieved through ordinary post-hoc calibration? It would be helpful to clarify when checkpoint reselection offers a distinct practical advantage.
5. The reported confidence intervals are based on run-level bootstrap resampling, although runs share datasets, held-out domains, and hyperparameter configurations. How sensitive are the conclusions to this dependence? For example, would resampling at the held-out-domain or configuration level lead to similar intervals?