- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when adversarial training (AT) should be replaced by empirical risk minimization (ERM) for out-of-distribution generalization. The authors first show that the benefit of AT is stage-dependent: switching from AT to ERM at an appropriate checkpoint can outperform either method alone. They then introduce class-conditional cross-domain variance (CDV), which measures how much within-class feature variance differs across source domains, and use the evolution of CDV on held-out source data to trigger an AT-to-ERM switch. The theory shows, under a simplified invariant-spurious feature model, that CDV is proportional to the fourth power of the normalized spurious-feature coefficient and that the CDV decay rate must fall below any fixed positive threshold in finite time. Experiments on five DomainBed benchmarks with ResNet and Transformer backbones show consistent gains over ERM/AT and several DG baselines, and the method also improves ERM++.
Critical strengths
- The training-dynamics perspective is useful and practically motivated. The paper clearly demonstrates that the contribution of adversarial training to OOD generalization is not monotonic in training time, and the proposed rollback-and-switch procedure is simple to integrate into standard DomainBed-style training without using target-domain data.
- The empirical evaluation is reasonably broad. The method is tested on five DG benchmarks and four backbones, improves both ERM and AT in most reported settings, and also transfers to the stronger ERM++ framework. The CDV analysis further addresses the concern that a low discrepancy score could simply reflect representation collapse.
Critical weaknesses
-
The methodological novelty is relatively incremental. The overall recipe of adversarial pretraining followed by standard/ERM fine-tuning has clear precedents, and using cross-domain feature statistics to monitor invariance is also well established. The main new ingredient is therefore CDV as a switching signal. However, Table 6 weakens the case that CDV itself is a distinctive contribution: CDV-guided switching reaches 70.88 average OOD accuracy versus 70.70 for CORAL, a difference of only 0.18 points in the reported ResNet-50 comparison. This suggests that the broader two-stage AT-to-ERM schedule may be doing most of the work, while the specific value of CDV is not yet sufficiently isolated.
-
The theoretical justification does not fully match the practical claim. The result CDV ∝ |w_sp|^4 is informative but holds in a highly simplified binary invariant-spurious model. The finite-switching-time theorem only guarantees that the CDV decay rate cannot remain above a fixed positive threshold forever (essentially because CDV is nonnegative); it does not show that the CDV minimum or deterioration point coincides with the OOD-optimal switching epoch. In practice, the algorithm also relies on several heuristic monitoring hyperparameters (EMA coefficient, accuracy ratio, tolerance, deterioration threshold, patience, and minimum epoch). Stronger evidence that the selected checkpoint robustly tracks the OOD-optimal switch across datasets/backbones/seeds is needed to support the 'principled' switching claim.
Questions for authors
-
Can the authors more clearly isolate the contribution of CDV from the contribution of simply using a two-stage AT-to-ERM schedule? In particular, please compare CDV against strong and simple switching signals (e.g., CORAL/MMD and source-validation-based early stopping) under the same tuning budget across all major backbones and datasets, and report the distribution of the gap between the selected switching epoch and the oracle OOD-optimal switching epoch. If CDV consistently selects a substantially better epoch, this would materially strengthen the paper.
-
Can the authors clarify the theoretical link between Theorem 5.3 and the actual rollback/deterioration rule in Algorithm 1? The theorem establishes existence of a finite time at which the CDV decay rate becomes small, whereas the implementation switches after a smoothed CDV value deteriorates relative to a retained minimum under an accuracy constraint. Is there a formal or empirical argument showing that these two criteria identify the same transition region, and how sensitive are the results to the six monitoring hyperparameters?
3 comments
The additional experiments partially address my empirical concerns. However, my main concerns remain.
First, the methodological novelty and independent contribution of CDV are still insufficiently established. Second, the theoretical results do not adequately justify the practical switching rule or its relationship to the optimal switching point.
Overall, while I acknowledge the additional evidence, I do not find the rebuttal sufficient to resolve these concerns. Therefore, I will maintain my original rating.
Author's Response
Thank you for your follow-up and for highlighting concerns about methodological novelty, CDV’s independent contribution, and the link between our theoretical results and practical switching. We would like to address these concerns below.
Methodological novelty.
From an adversarial training dynamics perspective, we find that the benefits of AT for OOD generalization are stage-dependent: extending AT does not consistently improve OOD performance. This observation suggests that the limited OOD benefits of AT cannot be attributed to perturbation strength alone. Motivated by this finding, we use early AT to promote robust representation learning, followed by ERM for continued empirical fitting in the learned representation space. The key is to determine when to switch. To this end, we propose a practical metric, CDV, which monitors cross-domain differences in the within-class variance of normalized features on source-domain validation, forming an adaptive AT-to-ERM strategy without target-domain data.
Independent contribution of CDV: comparison of switching rules under the same two-stage training.
To further assess CDV’s independent contribution, we compare five switching rules with ResNet-50 over three seeds under the same AT-to-ERM protocol and effective update budget. We additionally compare CDV, CORAL, and MMD with ViT-B/16 using one fixed seed.
(a) ResNet-50: OOD accuracy (%, mean ± standard deviation over three seeds).
| Switching rule | PACS | OfficeHome | VLCS | TerraIncognita | Avg. |
|---|---|---|---|---|---|
| Source-val. fixed epoch | 87.8 ± 0.3 | 71.3 ± 0.2 | 76.9 ± 0.8 | 44.6 ± 0.5 | 70.14 |
| Source-val. accuracy | 87.5 ± 0.7 | 71.7 ± 0.2 | 78.1 ± 0.7 | 45.5 ± 1.0 | 70.69 |
| CORAL | 86.1 ± 0.5 | 71.7 ± 0.2 | 77.6 ± 0.1 | 46.9 ± 0.0 | 70.58 |
| MMD | 86.1 ± 0.5 | 71.6 ± 0.1 | 78.1 ± 0.2 | 43.8 ± 0.0 | 69.90 |
| CDV (ours) | 87.8 ± 0.1 | 72.2 ± 0.2 | 78.6 ± 0.2 | 49.2 ± 0.4 | 71.95 |
(b) ViT-B/16: OOD accuracy (%) with one fixed seed.
| Switching rule | PACS | OfficeHome | VLCS | Avg. |
|---|---|---|---|---|
| CORAL | 95.8 | 83.0 | 82.3 | 87.03 |
| MMD | 94.5 | 82.9 | 81.6 | 86.33 |
| CDV (ours) | 95.6 | 83.2 | 82.8 | 87.20 |
These results show that CDV achieves the highest average OOD accuracy with both ResNet-50 (71.95%) and ViT-B/16 (87.20%), supporting the additional OOD gains from CDV as a switching signal within the same two-stage training procedure.
Theory and practical switching.
We further clarify that:
(1) In our simplified model, the theory establishes a quantitative link between CDV and the spurious feature’s relative contribution to the representation. It also shows that AT reduces this contribution while CDV tracks the change.
(2) In practice, we observe that CDV trajectories first decrease and then rise (Figures 2–3). This observation motivates our switching rule: we restore the retained low-CDV checkpoint and begin ERM when sustained CDV deterioration is confirmed.
(3) And the candidate-oracle comparison reported in our initial rebuttal provides empirical support for CDV-based checkpoint selection.
We will clarify the scope of our theory and the empirical basis for our switching rule in the revised manuscript.
Author's Response
Thank you for your constructive comments. They help us isolate the contribution of CDV as a switching signal, clarify the theoretical contribution, and assess the sensitivity of checkpoint selection to the monitoring hyperparameters.
(W1,Q1): The contribution of CDV as a switching signal.
We agree that prior work has used adversarial pretraining followed by ERM fine-tuning. However, our contribution is a CDV-guided adaptive two-stage strategy. Within a single training process, the CDV trajectory on source validation data guides two decisions: when to end AT and which AT checkpoint to restore for ERM. We clarify that Table 6 evaluates the practical value of CDV as a proxy for invariant feature learning by comparing it with two established cross-domain discrepancy measures (CORAL and MMD) as switching signals under the same AT-to-ERM protocol.
To more clearly isolate the contribution of CDV, we compare five AT-to-ERM switching rules using ResNet-50 across 16 held-out domains in PACS, OfficeHome, VLCS, and TerraIncognita. Within each domain, five rules use the same evaluation seed and the budget of 5,001 parameter updates, excluding updates discarded by rollback. CDV, CORAL, and MMD share the same trigger settings. For the fixed-schedule baseline, we test three switching epochs (5, 10, and 20). We select the schedule whose final model has the highest average validation accuracy across the source domains. The accuracy-based rule uses source-validation accuracy to decide when to switch from AT to ERM. For each held-out domain, the candidate oracle is the evaluated switching schedule with the highest OOD accuracy. The epoch gap is the selected switching epoch minus the oracle epoch. OOD accuracy loss is the oracle's OOD accuracy minus that of the selected schedule.
The result shows that CDV matches the candidate oracle in 7 of 16 domains, and has the lowest mean OOD accuracy loss. Under the same trigger settings, CDV triggers in all 16 domains, while CORAL fails to trigger in 10 and MMD in one. Compared with the strongest simple baseline(the source-validation accuracy rule), CDV selects an epoch closer to the oracle in 10 of 16 domains.
(W2,Q2) 1.Link between Theory and Algorithm 1.
This is a deeply insightful question. We agree that the theory studies a different event from the one used to switch in practice, but we would like to clarify that Algorithm 1 is not derived from Theorem 5.3.
(1) Prior work suggests that AT can reduce reliance on spurious features. To this end, we study the relative contribution of a single spurious feature to the representation during AT in a binary model and show that CDV decreases strictly and that its decay rate reaches or falls below any prescribed positive threshold in finite time.
(2) However, we empirically observe that CDV decreases rapidly early in AT and subsequently rises (Figures 2–3). Based on this observation, we propose Algorithm 1, which ends AT when the smoothed CDV shows a sustained increase relative to its best recorded value and then restores the corresponding checkpoint to begin ERM.
2. The CDV-selected epoch coincides with an OOD-optimal candidate epoch in 7/16 held-out domains, the highest count among the five rules. CDV also has the lowest mean OOD accuracy loss relative to the oracle (1.04 percentage points). This supports starting ERM from the retained low-CDV checkpoint (see W1, Q1).
3.Sensitivity of checkpoint selection to monitoring hyperparameters. We assess the sensitivity of the ERM starting checkpoint S to the six hyperparameters. We replay the switching rule on eight AT trajectories, one per held-out domain in PACS (ResNet-50) and OfficeHome (ResNet-18). Starting from the settings in Table 8, we vary one parameter at a time and test two alternative values for each.
(n/8) indicates that n held-out domains retain their baseline checkpoint S. The results show that 96 replays trigger a switch, and 80 (83.3%) retain the same checkpoint S as the baseline, suggesting that checkpoint selection is generally stable under the single-parameter changes.
Thank you again for your constructive comments. We will upload the revised manuscript with all changes marked in blue before the discussion period ends.