acceptodds
Under review as a conference paper at ICLR 2027

When to Switch: From Adversarial Training to ERM for Out-of-Distribution Generalization

Abstract

A key challenge in out-of-distribution (OOD) learning is that models trained with empirical risk minimization (ERM) may learn spurious features. It is well known that adversarial training (AT) can suppress such learning, but OOD generalization may deteriorate in later training stages. In the present paper, we propose an adaptive two-stage AT-to-ERM strategy which uses early AT to encourage invariant feature learning, followed by ERM for empirical fitting based on the learned representation. The key is to determine when to switch. We introduce class conditional cross domain variance (CDV) to quantify cross domain discrepancies in the within-class variances of normalized features. We switch from AT to ERM when CDV shifts from decreasing to increasing. In the invariant spurious feature model, we prove that CDV is proportional to the fourth power of the normalized spurious feature coefficient, and gradient flow analysis shows that, beyond a finite time, the decay rate of CDV is at most a prescribed threshold. Experiments on five benchmarks with convolutional and Transformer backbones show that our strategy improves mean OOD accuracy by percentage points over AT and by percentage points over ERM, while outperforming MAT, Fish, and IRM. The strategy is also competitive when integrated into the strong ERM++ baseline, and CDV guided switching yields better average OOD performance than CORAL or MMD guided switching.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
out-of-distribution generalization, domain generalization, adversarial training, training dynamics, invariant representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aj8rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when adversarial training (AT) should be replaced by empirical risk minimization (ERM) for out-of-distribution generalization. The authors first show that the benefit of AT is stage-dependent: switching from AT to ERM at an appropriate checkpoint can outperform either method alone. They then introduce class-conditional cross-domain variance (CDV), which measures how much within-class feature variance differs across source domains, and use the evolution of CDV on held-out source data to trigger an AT-to-ERM switch. The theory shows, under a simplified invariant-spurious feature model, that CDV is proportional to the fourth power of the normalized spurious-feature coefficient and that the CDV decay rate must fall below any fixed positive threshold in finite time. Experiments on five DomainBed benchmarks with ResNet and Transformer backbones show consistent gains over ERM/AT and several DG baselines, and the method also improves ERM++.

Critical strengths
  1. The training-dynamics perspective is useful and practically motivated. The paper clearly demonstrates that the contribution of adversarial training to OOD generalization is not monotonic in training time, and the proposed rollback-and-switch procedure is simple to integrate into standard DomainBed-style training without using target-domain data.
  2. The empirical evaluation is reasonably broad. The method is tested on five DG benchmarks and four backbones, improves both ERM and AT in most reported settings, and also transfers to the stronger ERM++ framework. The CDV analysis further addresses the concern that a low discrepancy score could simply reflect representation collapse.
Critical weaknesses
  1. The methodological novelty is relatively incremental. The overall recipe of adversarial pretraining followed by standard/ERM fine-tuning has clear precedents, and using cross-domain feature statistics to monitor invariance is also well established. The main new ingredient is therefore CDV as a switching signal. However, Table 6 weakens the case that CDV itself is a distinctive contribution: CDV-guided switching reaches 70.88 average OOD accuracy versus 70.70 for CORAL, a difference of only 0.18 points in the reported ResNet-50 comparison. This suggests that the broader two-stage AT-to-ERM schedule may be doing most of the work, while the specific value of CDV is not yet sufficiently isolated.

  2. The theoretical justification does not fully match the practical claim. The result CDV ∝ |w_sp|^4 is informative but holds in a highly simplified binary invariant-spurious model. The finite-switching-time theorem only guarantees that the CDV decay rate cannot remain above a fixed positive threshold forever (essentially because CDV is nonnegative); it does not show that the CDV minimum or deterioration point coincides with the OOD-optimal switching epoch. In practice, the algorithm also relies on several heuristic monitoring hyperparameters (EMA coefficient, accuracy ratio, tolerance, deterioration threshold, patience, and minimum epoch). Stronger evidence that the selected checkpoint robustly tracks the OOD-optimal switch across datasets/backbones/seeds is needed to support the 'principled' switching claim.

Questions for authors
  1. Can the authors more clearly isolate the contribution of CDV from the contribution of simply using a two-stage AT-to-ERM schedule? In particular, please compare CDV against strong and simple switching signals (e.g., CORAL/MMD and source-validation-based early stopping) under the same tuning budget across all major backbones and datasets, and report the distribution of the gap between the selected switching epoch and the oracle OOD-optimal switching epoch. If CDV consistently selects a substantially better epoch, this would materially strengthen the paper.

  2. Can the authors clarify the theoretical link between Theorem 5.3 and the actual rollback/deterioration rule in Algorithm 1? The theorem establishes existence of a finite time at which the CDV decay rate becomes small, whereas the implementation switches after a smoothed CDV value deteriorates relative to a retained minimum under an accuracy constraint. Is there a formal or empirical argument showing that these two criteria identify the same transition region, and how sensitive are the results to the six monitoring hyperparameters?

3 comments
AuthorsOpenReview ↗

Author's Response

Thank you for your constructive comments. They help us isolate the contribution of CDV as a switching signal, clarify the theoretical contribution, and assess the sensitivity of checkpoint selection to the monitoring hyperparameters.

(W1,Q1): The contribution of CDV as a switching signal.
We agree that prior work has used adversarial pretraining followed by ERM fine-tuning. However, our contribution is a CDV-guided adaptive two-stage strategy. Within a single training process, the CDV trajectory on source validation data guides two decisions: when to end AT and which AT checkpoint to restore for ERM. We clarify that Table 6 evaluates the practical value of CDV as a proxy for invariant feature learning by comparing it with two established cross-domain discrepancy measures (CORAL and MMD) as switching signals under the same AT-to-ERM protocol.

To more clearly isolate the contribution of CDV, we compare five AT-to-ERM switching rules using ResNet-50 across 16 held-out domains in PACS, OfficeHome, VLCS, and TerraIncognita. Within each domain, five rules use the same evaluation seed and the budget of 5,001 parameter updates, excluding updates discarded by rollback. CDV, CORAL, and MMD share the same trigger settings. For the fixed-schedule baseline, we test three switching epochs (5, 10, and 20). We select the schedule whose final model has the highest average validation accuracy across the source domains. The accuracy-based rule uses source-validation accuracy to decide when to switch from AT to ERM. For each held-out domain, the candidate oracle is the evaluated switching schedule with the highest OOD accuracy. The epoch gap is the selected switching epoch minus the oracle epoch. OOD accuracy loss is the oracle's OOD accuracy minus that of the selected schedule.

Switching ruleEarly / exact / late relative to oracle (domains)Mean OOD accuracy loss (percentage points)
CDV4 / 7 / 51.04
Source-validation accuracy6 / 2 / 81.47
Fixed epoch chosen by source validation4 / 1 / 112.13
MMD4 / 0 / 122.41
CORAL4 / 3 / 93.08

The result shows that CDV matches the candidate oracle in 7 of 16 domains, and has the lowest mean OOD accuracy loss. Under the same trigger settings, CDV triggers in all 16 domains, while CORAL fails to trigger in 10 and MMD in one. Compared with the strongest simple baseline(the source-validation accuracy rule), CDV selects an epoch closer to the oracle in 10 of 16 domains.

(W2,Q2) 1.Link between Theory and Algorithm 1.
This is a deeply insightful question. We agree that the theory studies a different event from the one used to switch in practice, but we would like to clarify that Algorithm 1 is not derived from Theorem 5.3.
(1) Prior work suggests that AT can reduce reliance on spurious features. To this end, we study the relative contribution of a single spurious feature to the representation during AT in a binary model and show that CDV decreases strictly and that its decay rate reaches or falls below any prescribed positive threshold in finite time.
(2) However, we empirically observe that CDV decreases rapidly early in AT and subsequently rises (Figures 2–3). Based on this observation, we propose Algorithm 1, which ends AT when the smoothed CDV shows a sustained increase relative to its best recorded value and then restores the corresponding checkpoint to begin ERM.

2. The CDV-selected epoch coincides with an OOD-optimal candidate epoch in 7/16 held-out domains, the highest count among the five rules. CDV also has the lowest mean OOD accuracy loss relative to the oracle (1.04 percentage points). This supports starting ERM from the retained low-CDV checkpoint (see W1, Q1).

3.Sensitivity of checkpoint selection to monitoring hyperparameters. We assess the sensitivity of the ERM starting checkpoint SS to the six hyperparameters. We replay the switching rule on eight AT trajectories, one per held-out domain in PACS (ResNet-50) and OfficeHome (ResNet-18). Starting from the settings in Table 8, we vary one parameter at a time and test two alternative values for each.

Monitoring parameterBaseline (Table 8)Lower settingHigher setting
EMA coefficient0.850.80 (5/8)0.90 (4/8)
Accuracy ratio0.950.90 (8/8)0.99 (5/8)
Tolerance1%0.5% (8/8)2% (7/8)
Deterioration threshold1%0.5% (8/8)2% (6/8)
Patience21 (6/8)3 (7/8)
Minimum epoch53 (8/8)10 (8/8)

(n/8)(n/8) indicates that nn held-out domains retain their baseline checkpoint SS. The results show that 96 replays trigger a switch, and 80 (83.3%) retain the same checkpoint SS as the baseline, suggesting that checkpoint selection is generally stable under the single-parameter changes.

Thank you again for your constructive comments. We will upload the revised manuscript with all changes marked in blue before the discussion period ends.

The additional experiments partially address my empirical concerns. However, my main concerns remain.

First, the methodological novelty and independent contribution of CDV are still insufficiently established. Second, the theoretical results do not adequately justify the practical switching rule or its relationship to the optimal switching point.

Overall, while I acknowledge the additional evidence, I do not find the rebuttal sufficient to resolve these concerns. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Author's Response

Thank you for your follow-up and for highlighting concerns about methodological novelty, CDV’s independent contribution, and the link between our theoretical results and practical switching. We would like to address these concerns below.

Methodological novelty.

From an adversarial training dynamics perspective, we find that the benefits of AT for OOD generalization are stage-dependent: extending AT does not consistently improve OOD performance. This observation suggests that the limited OOD benefits of AT cannot be attributed to perturbation strength alone. Motivated by this finding, we use early AT to promote robust representation learning, followed by ERM for continued empirical fitting in the learned representation space. The key is to determine when to switch. To this end, we propose a practical metric, CDV, which monitors cross-domain differences in the within-class variance of normalized features on source-domain validation, forming an adaptive AT-to-ERM strategy without target-domain data.

Independent contribution of CDV: comparison of switching rules under the same two-stage training.

To further assess CDV’s independent contribution, we compare five switching rules with ResNet-50 over three seeds under the same AT-to-ERM protocol and effective update budget. We additionally compare CDV, CORAL, and MMD with ViT-B/16 using one fixed seed.

(a) ResNet-50: OOD accuracy (%, mean ± standard deviation over three seeds).

Switching rulePACSOfficeHomeVLCSTerraIncognitaAvg.
Source-val. fixed epoch87.8 ± 0.371.3 ± 0.276.9 ± 0.844.6 ± 0.570.14
Source-val. accuracy87.5 ± 0.771.7 ± 0.278.1 ± 0.745.5 ± 1.070.69
CORAL86.1 ± 0.571.7 ± 0.277.6 ± 0.146.9 ± 0.070.58
MMD86.1 ± 0.571.6 ± 0.178.1 ± 0.243.8 ± 0.069.90
CDV (ours)87.8 ± 0.172.2 ± 0.278.6 ± 0.249.2 ± 0.471.95

(b) ViT-B/16: OOD accuracy (%) with one fixed seed.

Switching rulePACSOfficeHomeVLCSAvg.
CORAL95.883.082.387.03
MMD94.582.981.686.33
CDV (ours)95.683.282.887.20

These results show that CDV achieves the highest average OOD accuracy with both ResNet-50 (71.95%) and ViT-B/16 (87.20%), supporting the additional OOD gains from CDV as a switching signal within the same two-stage training procedure.

Theory and practical switching.

We further clarify that:
(1) In our simplified model, the theory establishes a quantitative link between CDV and the spurious feature’s relative contribution to the representation. It also shows that AT reduces this contribution while CDV tracks the change.
(2) In practice, we observe that CDV trajectories first decrease and then rise (Figures 2–3). This observation motivates our switching rule: we restore the retained low-CDV checkpoint and begin ERM when sustained CDV deterioration is confirmed.
(3) And the candidate-oracle comparison reported in our initial rebuttal provides empirical support for CDV-based checkpoint selection.
We will clarify the scope of our theory and the empirical basis for our switching rule in the revised manuscript.

Reviewer_6HmzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adaptive training strategy that switches from adversarial training (AT) to empirical risk minimization (ERM) to improve out-of-distribution (OOD) generalization. The authors introduce class-conditional cross-domain variance (CDV) to guide the switching process. They provide theoretical analysis relating CDV to spurious feature reliance under a simplified model and evaluate the proposed method on multiple domain generalization benchmarks.

Critical strengths

The paper studies a practically relevant problem and proposes a simple adaptive training strategy for improving OOD generalization.

The experimental evaluation is relatively extensive, covering multiple domain generalization benchmarks and model architectures.

Critical weaknesses

The proposed CDV metric lacks sufficient justification and validation. CDV measures the cross-domain variance of within-class feature variances, but it is unclear why this quantity can reliably characterize domain-invariant representations or the model's reliance on spurious features. In particular, similar within-class variances do not necessarily imply similar feature distributions across domains. The theoretical analysis establishes a connection between CDV and spurious feature reliance only under a highly simplified setting where the invariant feature has zero class-conditional variance. Moreover, the empirical analysis does not provide sufficient direct evidence supporting CDV as a reliable indicator of invariant feature learning or as an effective criterion for selecting the switching time.

The empirical advantages over existing methods are not sufficiently convincing. Although the proposed method achieves competitive OOD generalization performance, its improvements over strong baselines, particularly CORAL, are relatively modest and not consistent across all settings. More importantly, the paper does not sufficiently clarify the fundamental advantages of switching between AT and ERM over existing approaches such as CORAL, which encourage domain-invariant representations through an additional regularization term in the training objective. Beyond improvements in benchmark accuracy, it remains unclear what distinctive benefits the proposed training strategy offers over these established alternatives.

Questions for authors

Could the authors provide further justification and empirical evidence that CDV reliably reflects invariant feature learning and spurious feature reliance?

Could the authors clarify the key advantages of the proposed AT-to-ERM switching strategy over regularization-based methods such as CORAL?

5 comments
AuthorsOpenReview ↗

Author's Response

Thank you for your constructive comments. They help us clarify the link between CDV and reliance on spurious features, as well as the advantages of AT-to-ERM switching over CORAL regularization.

(W1,Q1): Evidence linking CDV to reliance on spurious features.
CDV is an effective signal for monitoring the model's reliance on spurious features during AT and for deciding when to switch to ERM. We provide evidence through theory and switching ablations in this manuscript, and further add a controlled experiment that directly tests sensitivity to color (spurious feature).

1.Theoretical evidence. Lemma 5.1 gives a concrete interpretation of declining CDV in a simplified binary model. CDV is proportional to the fourth power of the normalized spurious feature coefficient. A lower CDV therefore corresponds to a smaller relative contribution of the spurious feature to the representation. This relationship also extends to invariant features with positive within-class variance. In the block representation, an invariant feature with the same class-conditional distribution across domains contributes the same term to each domain's total within-class variance. This common term cancels when we compute the variance across domains. Therefore, the relationship still holds as long as the spurious feature's within-class variance differs across source domains. We will clarify this extension in the revised manuscript.

2.Experimental evidence.
(1) Tables 1-2 evaluate the OOD performance of CDV-guided AT-to-ERM. On the four backbones, AT-to-ERM achieves higher mean OOD accuracy than both AT and ERM. It also outperforms CORAL on three backbones, while CORAL achieves higher mean accuracy on ResNet-18;
(2) Tables 4 and 5 examine when to switch and from which checkpoint to start ERM, respectively. Table 4 shows that switching at the CDV-selected epoch yields higher mean OOD accuracy than switching one epoch earlier or later. Table 5 shows that rolling back to the retained checkpoint improves mean OOD accuracy compared with starting ERM at the deterioration trigger ;
(3) Table 3 tests the compatibility of our strategy with ERM++. Table 6 directly compares the switching signal itself, CDV guided switching yields better average OOD performance than CORAL or MMD guided switching, indicating that CDV is a competitive switching signal.
3.Direct evidence linking CDV to spurious-feature reliance. On Colored MNIST, we render each test digit in red and green while keeping its handwriting and label unchanged. We measure sensitivity to color using the squared distance between the normalized hidden representations of the two versions, and report classification accuracy when color conflicts with the label. We evaluate the 30 independent runs at AT steps 100 and 3,750.

MetricAT step 100AT step 3,750
Mean source-validation CDV0.05990.0112
Mean squared distance between red and green representations3.3531.960
Mean accuracy when color conflicts with the label0.21%85.54%
Spearman's ρ\rho between CDV and representation distance across runs0.9720.978

At each checkpoint, models with lower CDV show smaller changes in their hidden representations. From step 100 to step 3,750, mean CDV decreases while mean accuracy increases when color conflicts with the label. These results provide evidence from both hidden representations and classification behavior that CDV helps track reliance on the spurious color cue.

(W2,Q2): Advantages of AT-to-ERM switching over CORAL regularization
We would like to clarify that:
(1) Motivation from training dynamics. Figure 1 shows that the benefit of AT-to-ERM for OOD generalization depends on the switching epoch. Extending AT does not consistently improve OOD accuracy. This observation motivates our adaptive switching strategy. The empirical results show that AT-to-ERM outperforms AT and ERM. It also outperforms CORAL on three backbones, while CORAL achieves higher mean accuracy on ResNet-18.
(2) Use of cross-domain statistics. CORAL adds a cross-domain alignment penalty to the training loss, so the alignment criterion directly affects parameter updates. Our method uses CDV on the source validation sets to decide when to end AT and which checkpoint to use for ERM. Table 7 shows the value of this strategy: The CDV-selected AT checkpoint retains feature variation across multiple directions (effective rank of 18.435), and its inter-class separation is close to that of CORAL (0.8672 vs. 0.8647). When training continues from this checkpoint with ERM, inter-class separation increases to 1.3231.
(3) Compatibility with stronger training pipelines. Integrating the same switching strategy into ERM++ improves OOD accuracy on all four datasets (Table 3).

Thank you again for your constructive comments. We will upload the revised manuscript with all changes marked in blue before the discussion period ends.

Reviewer_6HmzOpenReview ↗

Thanks for Replying

Thanks for replying, my concerns on the experimental results are still remain. The used model is weak i.e., ResNet or ViT. The benchmark datasets are old. The reported results are weaker than some existing literature on PACS and VLCS e.g., "Out-of-distribution Generalization with Causal Invariant Transformations". Owing to these, I will remain my score.

AuthorsOpenReview ↗

Author's Response

Thank you for your follow-up and for highlighting concerns about experimental coverage and comparison settings. We would like to address your concerns about benchmark selection and the comparison with RICE as follows.

1.Model and dataset selection.

(1) We would like to clarify that our use of ResNet/ViT backbones and DomainBed benchmarks [1], including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet, is consistent with recent studies. For example, ERM++ (WACV 2025) evaluates ResNet-50 and ViT-B/16 [2], while CDA (ICCV 2025) evaluates ResNet-50 and ViT-S/16 on these datasets [3]. We adopt this setup to facilitate comparisons with existing domain generalization methods under a common data-splitting and evaluation protocol.

(2) To further address your concerns, we have added experiments on NICO++ using the same source/target domain partition as RiskyDiff (AAAI 2026) [4]. Using ConvNeXt-Base [5] pretrained with DINOv3 [6], we train on Autumn, Rock, Dim, and Grass and evaluate on the unseen Outdoor and Water domains.

MethodOutdoorWaterAverage
ERM82.2976.3179.30
AT83.8278.7681.29
AT-to-ERM87.6083.9185.75

The results show that AT-to-ERM (ours) improves accuracy on both target domains, with average gains of 6.45 and 4.46 percentage points over ERM and AT, respectively. This provides further evidence of the effectiveness of our method on an additional benchmark with a newer pretrained model.

2.Comparison with RICE.

(1) Thank you for recommending Out-of-distribution Generalization with Causal Invariant Transformations. We examined the experimental settings of RICE and our DomainBed setup. Specifically, the original RICE setup differs from ours in dropout (0.1 vs. 0), training budget (20 epochs vs. 5,001 steps), and learning-rate schedule (multiplication by 0.2 at epochs 6, 12, and 20 vs. a fixed learning rate of 5×10−55\times10^{-5}). These differences limit direct comparison.

(2) Since we could not find an official RICE implementation, we independently implemented it in DomainBed using the publicly available CycleMix PACS generator and the original regularization coefficient λ0=0.5\lambda_0=0.5. We compare only on PACS, as we could not find a usable pretrained generator for VLCS and it would be difficult to train one from scratch within the remaining rebuttal period. RICE and our method use the same classifier backbone, pretrained weights, data splits, and source-validation model-selection rule.

MethodArtCartoonPhotoSketchAvg.
RICE (CycleMix pretrained generators)87.25 ± 2.7877.20 ± 1.6897.28 ± 0.6884.50 ± 1.6986.56 ± 0.98
AT-to-ERM (ours)90.28 ± 0.7181.97 ± 1.1898.48 ± 0.1682.26 ± 0.9688.25 ± 0.06

The results show that AT-to-ERM improves average accuracy by 1.69 percentage points.

[1] Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. In International Conference on Learning Representations (ICLR), 2021.
[2] Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Kate Saenko, and Bryan Plummer. ERM++: An Improved Baseline for Domain Generalization. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pp. 8514–8524, 2025.
[3] Yuyang Ji, Zeyi Huang, Haohan Wang, and Yong Jae Lee. Customizing Domain Adapters for Domain Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 934–944, 2025.
[4] Han Yu, Hao Zou, Xingxuan Zhang, Zhengyi Wang, Yue He, Kehan Li, and Peng Cui. Generating Risky Samples with Conformity Constraints via Diffusion Models. In Proceedings of the AAAI Conference on Artificial Intelligence, 40(33):27934–27942, 2026.
[5] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976–11986, 2022.
[6] Oriane Siméoni et al. DINOv3. Transactions on Machine Learning Research, 2026.

Reviewer_DP4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when to switch from adversarial training to standard empirical risk minimization for out-of-distribution generalization. The idea is to use adversarial training early to reduce reliance on spurious features, then switch to ERM to improve predictive performance. The proposed switching signal, Cross-Domain Variance (CDV), measures differences in within-class feature variance across source domains. The method monitors a smoothed CDV on source-validation data, saves suitable checkpoints, and switches to ERM after CDV starts getting worse, restoring the saved checkpoint first. The paper evaluates the approach on DomainBed datasets with several backbones and also integrates it with ERM++. It includes comparisons with alternative switching signals, a rollback ablation, representation-collapse diagnostics, and an analysis of CDV dynamics in a simplified model.

Critical strengths

The method addresses a practical decision using only source-domain validation data. Choosing when to stop adversarial training is a meaningful problem, and the proposed procedure does not require access to target-domain data for switching. The combination of a feature-based signal, an accuracy safeguard, and checkpoint rollback is straightforward to implement. Table 2 reports a ResNet-50 average of 66.34, compared with 64.74 for ERM and 63.62 for continued adversarial training, which makes the approach worth investigating.

The paper includes several useful checks beyond the main benchmark results. I appreciate the comparisons with MMD and CORAL as switching criteria, the rollback ablation, and the representation-collapse diagnostics. The ERM++ experiment is also valuable because it tests whether the idea still helps on top of a stronger training recipe. These experiments make the empirical case more informative than a simple comparison against ERM alone, although some of the improvements are small.

Critical weaknesses

The theory studies a different event from the one used to switch in practice. Theorem 5.3 establishes that the decrease in CDV eventually becomes sufficiently slow. Algorithm 1 instead waits for a sustained increase relative to the best recorded CDV. These are different conditions. In fact, the appendix derives a strictly decreasing CDV under the stated theoretical setup, so that analysis does not establish a finite switching time for the implemented deterioration-based rule. The paper should narrow its theoretical claim or explain what additional mechanism connects the analysis to the observed turnaround. There is also a limitation in interpreting CDV as evidence of invariant features. CDV compares total within-class variance across domains. Two domains can have the same total within-class variance while their class-conditional feature locations or covariance directions differ. The collapse diagnostics address one possible failure mode, but they do not establish that a low CDV means spurious domain information has been removed.

The experiments do not yet isolate how much CDV adds beyond a well-chosen training schedule. Table 4 compares earlier and later switches, but the offsets and selection procedure are not clear enough to assess how competitive those baselines are. A fixed switch selected using source validation, and a switching rule based directly on source-validation accuracy, would be important controls. These comparisons would help distinguish the benefit of combining adversarial training with ERM from the benefit of the proposed CDV criterion. This matters because some of the most relevant margins are modest: CDV exceeds CORAL-based switching by 0.18 average points in Table 6, and the ERM++ integration improves the average by about 0.52 points in Table 3. Neither table reports variability. The method also does not outperform every baseline in every setting; for example, CORAL has a higher ResNet-18 average in Table 1. I would like stronger evidence that CDV is a reliably better switching signal.

Questions for authors

Could you clarify how Theorem 5.3 supports Algorithm 1? The theorem concerns a slowdown in CDV decrease, while the algorithm requires deterioration. How often does the practical rule fail to trigger within the training budget, and how does a plateau-based rule compare with the current one?

How does CDV compare with a fixed switching epoch tuned on source-validation data and with a source-validation-accuracy switching rule, under the same tuning budget? Please also define the earlier and later switches in Table 4 and report variability for the criterion comparison and ERM++ results.

Could you provide the full adversarial-training configuration, especially the FGSM perturbation radius and how perturbations are applied relative to input normalization? Please also clarify the total training budgets and whether rollback restores optimizer state. What is the measured wall-clock cost, including CDV evaluation and the adversarial-training steps discarded by rollback?

What evidence links low CDV specifically to reduced reliance on spurious features, beyond ruling out representation collapse? A controlled case where domains have matching within-class total variance but different class-conditional feature distributions would help clarify the scope of the signal.

2 comments
AuthorsOpenReview ↗

Author's Response(1)

Thank you for your constructive comments. They help us clarify the theoretical contribution, compare CDV-guided switching with source-validation baselines, and assess the link between lower CDV and reduced reliance on spurious features.

(W1,Q1):Link between Theorem 5.3 and Algorithm 1.
This is a deeply insightful question. We agree that the theory studies a different event from the one used to switch in practice, but we would like to clarify that Algorithm 1 is not derived from Theorem 5.3.
(1) Prior work suggests that AT can reduce reliance on spurious features. To this end, we study the relative contribution of a single spurious feature to the representation during AT in a binary model, and show that CDV decreases strictly and that its decay rate reaches or falls below any prescribed positive threshold in finite time.
(2) However, we empirically observe that CDV decreases rapidly early in AT and subsequently rises (Figures 2–3). Based on this observation, we propose Algorithm 1, which ends AT when the smoothed CDV shows a sustained increase relative to its best recorded value and then restores the corresponding checkpoint to begin ERM. In every complete run we examined, the rule triggered before step 2,000, well within the budget of 5,001 updates.
We will clarify the scope of Theorem 5.3 and the empirical motivation for the switching rule in Algorithm 1 in the revised manuscript.

(W1,Q4): Link between low CDV and reduced reliance on spurious features.
1.We agree that CDV measures an aggregate property of the representation and may not capture differences in class-conditional means or covariance directions. However, we would like to clarify that using CDV on source-domain validation data during training is practical, since small differences in variance along individual feature directions may reflect sampling variability when a domain has few validation examples.

2.Lemma 5.1 shows that CDV is proportional to the fourth power of the normalized spurious-feature coefficient.
We also conduct a color intervention on Colored MNIST to illustrate the usefulness and limitations of CDV. Color agrees with the label in 90% of examples in one training domain and 60% in the other. We compute CDV on the source validation sets and independently assess color reliance on 10,000 test digits.
(1) Evidence of CDV as a proxy for invariant feature learning during AT: Following the intervention approach of Plumb et al.[1], we render each test digit in red and green while keeping its handwriting and label unchanged. The color-flip rate is the percentage of red–green pairs with different predicted labels. We also use Conditional Spurious Variation (CSV), following Yi et al., to measure the sensitivity of the prediction loss to color [2].

Metric (mean over 30 independent runs)AT step 100AT step 3,750
Source-validation CDV0.05990.0112
Color-flip rate99.79%12.83%
CSV (cross-entropy)0.8770.278

All 30 runs show the same pattern. The checkpoint with lower source-validation CDV (step 3,750) also has a lower color-flip rate and lower CSV than the earlier checkpoint (step 100). This provides direct behavioral evidence that declining CDV accompanies reduced reliance on color.
(2) A low CDV at a single checkpoint does not establish domain invariance. We freeze one model at step 100 and construct two domains in which the normalized representations have the same total within-class variance for each class but different class-conditional means. CDV is therefore zero for this pair of domains, yet changing only the color flips the model’s prediction for 99.62% of the test digits.

[1] Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. “Finding and Fixing Spurious Patterns with Explanations.” Transactions on Machine Learning Research (TMLR), 2022.
[2] Mingyang Yi, Ruoyu Wang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. “Breaking Correlation Shift via Conditional Invariant Regularizer.” International Conference on Learning Representations (ICLR), 2023.

AuthorsOpenReview ↗

Author's Response(2)

(W2,Q2): CDV vs. source-validation switching rules (including variability across seeds).
To assess the reliability of CDV as a switching signal, we compare our method with two source-validation baselines under the same AT-to-ERM procedure and training budget. Table R1 reports OOD accuracy over three evaluation seeds. Table R2 and Table R3 extend the switching-signal comparison in Table 6 and the ERM++ integration in Table 3 to three random seeds, respectively.

(1)Fixed switching epoch selected by source validation. For each held-out target domain, we use the same tuning seed to compare three candidate schedules that switch from AT to ERM after epoch 5, 10, or 20. We then select the epoch that gives the highest final-model validation accuracy, and use it for all three evaluation seeds.
(2) Switching based on source-validation accuracy. This baseline uses the same checkpoint-and-rollback procedure as the CDV method, with mean source-validation accuracy as the switching signal.

Table R1. OOD accuracy (%) for three AT-to-ERM switching strategies.

Switching strategyPACSOfficeHomeVLCSTerraIncognitaAverage
Fixed epoch selected by source validation87.871.376.944.670.15
Source-validation accuracy87.571.778.145.570.70
CDV (ours)87.872.278.649.271.95

Table R2. OOD accuracy (%, mean ± standard deviation over three seeds) for three switching signals under the same AT-to-ERM procedure.

Switching signalPACSOfficeHomeVLCSTerraIncognitaAvg.
CORAL86.1 ± 0.571.7 ± 0.277.6 ± 0.146.9 ± 0.070.58
MMD86.1 ± 0.571.6 ± 0.178.1 ± 0.243.8 ± 0.069.90
CDV (ours)87.8 ± 0.172.2 ± 0.278.6 ± 0.249.2 ± 0.471.95

Table R3. OOD accuracy (%, mean ± standard deviation over three seeds).

MethodPACSOfficeHomeVLCSTerraIncognitaAvg.
ERM++89.90 ± 0.0874.52 ± 0.1077.65 ± 0.7851.10 ± 1.8473.29
AT-to-ERM++ (ours)90.43 ± 0.0775.26 ± 0.7078.59 ± 0.6052.79 ± 2.5474.27

Under the same AT-to-ERM procedure, CDV improves average OOD accuracy by 1.80 percentage points over the source-tuned fixed schedule and by 1.25 points over accuracy-based switching (Table R1). In the three seed comparison, CDV improves mean OOD accuracy by 1.37 percentage points over CORAL and 2.05 over MMD (Table R2). Both gains are larger than those in Table 6. With ERM++, the CDV-guided strategy raises average OOD accuracy from 73.29% to 74.27% (Table R3). These comparisons support CDV as a reliable switching signal.

Definition of the earlier and later switches in Table 4: For each run, let SS denote the epoch of the AT checkpoint retained by CDV. The Earlier, CDV-selected, and Later schedules switch from AT to ERM after epochs S−1S-1, SS, and S+1S+1, respectively.

(Q3):Training configuration and wall-clock cost.
Our adversarial training procedure follows the AT.FGSM routine in the https://github.com/NOVAglow646/NIPS22-MAT-and-LDAT-for-OOD. The attack configuration, training budget, and measured wall-clock cost are as follows.

AT configuration. We apply perturbations after channel-wise ImageNet normalization and generate adversarial examples using 10-step ℓ2\ell_2 projected gradient ascent with a random start. The perturbation radius ϵ=0.1\epsilon=0.1 and step size α=0.1\alpha=0.1 are defined in this normalized input space. For both gradient normalization and perturbation projection, we compute the ℓ2\ell_2 norm over the entire minibatch for each source domain.

Training budget and optimizer rollback. ERM and AT each perform 5,001 parameter updates. Let SS denote the AT step at which the selected checkpoint is saved. When the switch is triggered at AT step GG, we restore the model and full Adam state (both moment estimates and step counters) from that checkpoint and perform 5001−S5001-S ERM updates. The run retains 5001 updates and executes 5001+(G−S)5001+(G-S) in total, including the G−SG-S discarded AT updates.

Wall-clock cost. The total wall-clock time for AT-to-ERM includes both CDV evaluation on the source validation sets and AT updates later discarded at rollback.

DatasetAT (min)AT-to-ERM total (min)Discarded AT time (min)Total AT-to-ERM updates
PACS129.8189.625.245,085
OfficeHome205.8763.594.015,123
VLCS121.39131.664.405,073
TerraIncognita112.80148.6813.805,203
DomainNet194.62372.71104.096,409

Thank you again for your constructive comments. We will upload the revised manuscript with all changes marked in blue before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.