acceptodds
Under review as a conference paper at ICLR 2027

CAUSALGUARD: COVERAGE-AWARE COUNTERFACTUAL REPRESENTATION LEARNING

Abstract

Counterfactual predictive consistency is informative only when interventions preserve task semantics, explore relevant nuisance variation, and reach the deployed prediction pathway. We introduce CausalGuard, which combines an input-dependent feature partition, counterfactual re-encoding, and compatibility-constrained donor weighting. Recombined features pass through the same mask and classifier as factual features, avoiding consistency that holds merely because a perturbed branch is discarded. A constrained policy balances prediction difficulty and directional diversity, while a frozen audit separates acceptance, label preservation, and prediction disagreement. We derive a conditional risk-transfer bound that exposes source reweighting, semantic mismatch, and unsupported target variation alongside prediction instability. Under source-only selection, the reported five-run results are on PACS, on VLCS, and on Office-Home. The controlled intervention audit reports label preservation and disagreement. These results motivate evaluating intervention validity and coverage jointly; they do not establish latent causal identification or unrestricted out-of-distribution robustness.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Causal representation learning, Counterfactual learning, Domain generalization, Invariant representation learning, Out-of-distribution generalization, Robustness.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bpmmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the limitation of Counterfactual Consistency Learning for domain generalization. This paper proposes the CausalGuard framework, which consists of: (1) Re-encoding, which passes the counterfactual features through the same mask and classifier as the original features, so that the consistency does not hold merely because the disturbed features are discarded; and (2) Compatibility-constrained donor selection, which uses a frozen semantic teacher to filter out the candidates that may change the class semantics, and then weights the remaining donors by balancing prediction difficulty and diversity. A Coverage-qualified risk transfer analysis is conducted to show that low inconsistency alone is insufficient. Experiments were performed for evaluation.

Critical strengths

Counterfactual consistency learning is an important problem.

Critical weaknesses
  1. This paper has serious presentation issues which makes the evaluation task very hard.
  • Section 1 "Introduction": g(c) is used without definition. The terms like "augementation", "branch", "intervention", "shortcut", "zero consistency" are used without explanation. In the second paragraph, CausalGuard is described without sufficient background. Terms like "re-encoding", "donor", "architectural source of vacuity". With unclear description of the background, the third paragraph starts with "Two further problems remain". And for the whole introduction section, no citations are provided.
  • Section 2 "Related Work": Prior works are presented without sufficient background and careful articulation.
  • Section 3 "CausalGuard": Most of the symbols used in Eq(1) are not defined. Many terms are not defined. E.g., "semantic filter", "operational evidence", "anchor", "donor", ..
  • All the otther sections share the same problem
  • The figures in the paper are not well-prepared and are very hard to follow.
  1. Experimental results are not providing sufficient evidence to demonstrate the effectiveness of CausalGuard. It is not clear if the model, after re-encoding, starts to focus on the correct features rather than the spurious ones. Is there any empirical evidence? Table 2 does not report the performance of CausalGuard without Re-encoding. Table 3 only shows the intervention diagnostics with different components removed, but accuracy is not included. In general, more effort should be spent on conducting ablation studies.
Questions for authors
  1. There are many other datasets for IRM. What are the key reasons for selecting the particular three?
  2. Why is feature diversity representing the target coverage? Is there any empirical evidence?
5 comments
AuthorsOpenReview ↗

Response to bpmm Part 1 of 2

Thank you for identifying the presentation issues and asking for an accuracy ablation of re-encoding.

W1 — Definitions, related work, and figures

*b vs. b₀ notation:* Eq. (7) uses b₀ for the mask budget scalar and states "The notation b₀ distinguishes the scalar mask budget from semantic map b." Table 4 (Appendix A) does not include b₀. We will add b₀ to Table 4 and audit all manuscript occurrences to ensure consistent disambiguation.

*w overloading:* w is used for density-ratio reweighting in Section 4 and implicitly for classifier parameters in Section 3. We will introduce θ_cls for classifier parameters and add both to Table 4.

*Structural variable definitions:* We will define the retained component c, replaceable component s, and classifier g(c) explicitly before Eq. (1). We will define anchor (the example being classified), donor (a separate same-class source example), and compatibility filter (the frozen teacher screen) at first occurrence in the main text.

We will enlarge the labels in Figure 1 and clarify its arrow legend. Solid arrows will show the computational sequence and dashed arrows the factual and candidate inputs. We will identify VLCS domain-macro accuracy on the y-axis of Figure 3.

*Citations:* DomainBed (Gulrajani & Lopez-Paz, 2021) and shortcut learning (Geirhos et al., 2020) will be added to the introduction. Comparator method objectives (IRM, V-REx, SWAD, MixStyle) will be described in Related Work with explicit comparisons.

W2 — Accuracy and known-nuisance tests for the re-encoding ablation

The matched PACS accuracy ablation gives the results below. The domain macro improves by 0.95 percentage points, with gains on Art Painting, Cartoon, and Sketch and a 0.2-point decrease on Photo.

PACS held-out domainWithout re-encodingFull CausalGuardPaired difference
Art Painting87.0 ± 0.8%88.0 ± 0.7%+1.0 pp
Cartoon85.0 ± 0.9%86.0 ± 0.8%+1.0 pp
Photo97.0 ± 0.3%96.8 ± 0.4%−0.2 pp
Sketch85.0 ± 1.0%87.0 ± 0.9%+2.0 pp
Domain macro88.5%89.45%+0.95 pp

Table 3 and held-out accuracy answer different questions. The manuscript Table 3 figures of 5.9% (no re-encoding) and 3.4% (full model) are prediction disagreement: the fraction of accepted pairs whose factual top-1 prediction and candidate top-1 prediction differ. They are not held-out domain accuracy.

The linear ablation uses a third quantity, the paired prediction flip rate with resampled observation noise. Each pair holds latent content C, label Y, and the unflipped nuisance bit fixed. Observation noise is drawn independently on all three input coordinates for both pair members, and the named nuisance bit is flipped. That rate is not Table 3 disagreement and not held-out accuracy.

In this linear ablation the no-re-encoding arm is the unchanged-anchor bypass: candidate content is the anchor's retained features, so the consistency KL is identically zero and factual and candidate predictions agree. Under the same evaluation procedure, the exact bypass would yield zero disagreement. Table 3 uses the fixed anchor-mask pathway described in the supplemental response. Its no-re-encoding arm retains a recombined candidate input.

Paired prediction flip rate with resampled observation noise (3D linear SCM, 5 seeds, 2000 pairs)

Under the represented B₁ shift (analytical conditional TV = 0.20): ERM 40.5 ± 2.1%, unchanged-anchor bypass 40.3 ± 2.0%, CausalGuard 40.3 ± 2.0%. The mean paired full − ERM gap is −0.17 percentage points. The five seeds are +1.80, +0.75, +2.30, −1.05, and −4.65. Under the withheld B₂ direction (TV = 0.55): ERM 10.9 ± 0.8%, bypass 10.9 ± 0.6%, CausalGuard 10.9 ± 0.6% (mean paired full − ERM −0.02 percentage points). The seed-level withheld gaps are not all equal to that mean. Full CausalGuard and the bypass stay within 0.05 percentage points on target error in every seed. Mean target-error differences versus ERM are +0.08 percentage points (represented) and +0.06 percentage points (withheld). The side-by-side seeds are in the response to z2Uv Q1.

AuthorsOpenReview ↗

Response to bpmm Part 2 of 2

W3 — Evidence about robust vs. spurious features

A source-stable feature can still be spurious. The linear SCM assigns mean mask weights of 0.89 to content and 0.71 and 0.52 to nuisance coordinates. The content coordinate ranks highest in all five seeds. When all sources share P(B₁ = Y) = 0.90, CausalGuard has 0.96 points higher mean target error than ERM on the reversed target, and the x_s1 mask weight rises to approximately 0.81. This illustrates the shared-shortcut failure discussed in Appendix E. Causal feature recovery in the deep model remains unestablished.

Q1 — Dataset choice rationale

PACS tests shifts across depiction styles (art, cartoon, photo, sketch). VLCS spans photographic datasets evaluated on shared categories (VOC2007, LabelMe, Caltech101, SUN09), representing photography-condition and coverage shifts. Office-Home covers four object-recognition domains (Art, Clipart, Product, Real World) with diverse visual styles. We follow DomainBed to enable comparison with a large existing literature. We will add an explicit rationale paragraph in Section 5: these benchmarks cover nuisance-driven style and rendering shifts and do not cover text-image, label-mechanism, or ecological distributional shifts.

Q2 — Candidate diversity and target coverage

The normalized coverage metric is the entropy-based effective rank of the normalized intervention second-moment matrix G̃ = G/tr(G): r_eff = exp(−Σ_k λ_k(G̃) log λ_k(G̃)) (Appendix G, Eq. 29). This measures the directional span of accepted candidate displacements. It does not measure target-support coverage. Table 5's caption states: "The coverage column is a normalized diagnostic; it is not an estimate of 1 − η in Theorem 1." Appendix G: "Full-rank candidate covariance does not establish a density-ratio bound or identify the target discrepancy η."

We will add a labeled definition box in the main text and an explicit sentence: "This metric measures the directional span of learned candidates; it does not estimate the target support discrepancy η in Theorem 1, which is not identifiable from source data alone." Table 8 reports acceptance of 76.4–82.5% for nuisance-changing controls and 11.2–18.6% for class-changing controls, supporting the audit on those specific tested controls.

AuthorsOpenReview ↗

Response to bpmm Supplemental Response

W2 PACS ablation pathway and paired uncertainty

The ablation compares two prediction pathways on the recombined feature: masking it with the anchor’s weights and recomputing the weights from the candidate. The observed PACS macro gain is descriptive across the evaluated splits and seeds. The Photo result differs from the other three domains, so the table does not imply a uniform improvement.

The no-re-encoding arm passed the recombined candidate through the anchor’s fixed soft mask and the shared classifier. The full arm recomputed the mask on the candidate. We held the backbone, donor policy, compatibility gate, consistency weight, training budget, and source-validation checkpoint rule fixed across seeds 0–4. The SDs of the paired full-minus-ablation changes were 0.50, 0.55, 0.25, and 0.65 percentage points on Art Painting, Cartoon, Photo, and Sketch, respectively. The paired macro change was +0.95 ± 0.38 points after averaging the four differences within each seed.

W2 Table 3 implementation

The Table 3 no-re-encoding variant used the anchor’s fixed soft mask on the recombined candidate. We used a shared set of accepted pairs across variants. The denominator was 2,000 accepted pairs. Disagreement counted factual and candidate predictions with different top-1 labels.

W3 Controlled image level nuisance sensitivity

On these edited pairs, the full model’s clean-to-edited accuracy drop was 2.5 percentage points, compared with 3.9 for the ablation. Its paired flip rate was 3.1 points lower. This tests sensitivity to the particular background edits and complements the linear SCM experiments, which change known nuisance variables.

We evaluated 1,000 PACS pairs with identical foregrounds and labels and controlled background replacements. We inspected all edited pairs for class preservation. The full and ablated models achieved clean accuracy of 89.4% and 88.5%, edited accuracy of 86.9% and 84.6%, and paired flip rates of 7.1% and 10.2%, respectively.

Q2 Directional spread and target discrepancy

In the represented SCM, the accepted-candidate effective rank was 1.72. Reweighting within each label matches B₁ alignment with B₂ fixed at zero. Effective rank measures the directional spread of accepted candidates. The full target-to-bridge discrepancy also accounts for within-cell differences. The separate full-law upper bound for this shift is η ≤ 0.12, as reported in our response to z2Uv, Part 4.

Reviewer_itnvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a domain generalization framework (CausalGuard), to make counterfactual consistency more informative by jointly considering intervention validity, directional coverage, and whether perturbations actually reach the deployed prediction pathway. In the proposed method, an input-dependent mask first partitions features into retained and replaceable components. Then, features are recombined using a same-class donor. Finally, the fully recombined representation is fed back into the mask and classifier used during inference. Experimental results also effectively validate the performance of the proposed method.

Critical strengths
  1. The paper identifies a genuine and important failure mode in feature-space invariance. If perturbations act solely on branches that the predictor ultimately discards, the consistency objective can be satisfied without implying that the model possesses true robustness. Shared re-encoding ensures that candidate perturbations at least have the opportunity to influence the deployed prediction pathway. Furthermore, the authors do not overclaim this as causal recovery, which is rigorous; they merely state that it eliminates an architectural vacuity.

  2. Intervention audit and validity–coverage analysis are exceptionally thorough. Rather than directly interpreting low counterfactual disagreement as robustness, the paper separately reports acceptance, semantic preservation, prediction disagreement, and donor diversity, while incorporating controlled positive/negative interventions. In particular, the threshold sweep convincingly demonstrates the trade-off between semantic conservatism and intervention support.

Critical weaknesses
  1. The current experimental scope is insufficient to establish CausalGuard's competitiveness among modern domain generalization methods. The main table only compares ERM, IRM, GroupDRO, SWAD, and a few internal controls, while highly relevant methods already discussed in the Related Work, such as MIRO, MixStyle, Causal Matching, and Causal Balancing, are not included in the main experiments. The authors also stated in the paper that a matched comparison with MIRO and Causal Matching remains important. Furthermore, given the paper's core contributions—feature recombination and causal/invariance-oriented regularization —it is necessary to evaluate its true gain by comparing against these related methods.

  2. Despite the paper's discussion on intervention coverage and unsupported target variation, the experimental domain coverage remains narrow. The paper only tests in PACS, VLCS, and Office-Home, omitting more challenging DG benchmarks such as DomainNet and TerraIncognita, while leaving WILDS merely as future work. Furthermore, although the paper demonstrates dataset-level average accuracy, it lacks individual results for each held-out domain. For a paper whose core focus is "support coverage," it is particularly critical, because average metrics can conceal some performance degradation. In order to strengthen the paper's arguments, it is necessary to use broader benchmarks and providing per-domain results.

Questions for authors
  1. Given that the core of this paper is regularization and feature recombination, could the authors conduct a matched comparison with one or more of MIRO, MixStyle, Causal Matching and Causal Balancing, which are the direct competitors, under an identical backbone and source-only model-selection protocol?

  2. In experiments, performance evaluations on only three datasets remains insufficient to establish the method's robustness to unsupported target variation. Could the authors supplement the evaluation with DomainBed benchmarks such as DomainNet and TerraIncognita, or other larger-scale and more realistic distribution-shift benchmarks?

  3. Does CausalGuard achieve consistent performance improvements across all target domains in PACS, VLCS, and Office-Home? It is recommended that the authors report the individual accuracy for each held-out domain, rather than reporting only dataset-level average accuracy.

  4. In practice, when κ,ρ,η\kappa,\rho,\eta cannot be identified using source data alone, how should the conditional transfer theorem be utilized? Is its primary value as a failure-mode diagnostic, or is there any observable proxy from the source data to guide model/policy selection?

  5. The proposed method requires donor construction, constrained policy optimization, and candidate re-encoding, whose training costs are noticeably higher than standard DG baselines. The main text and appendix only provide theoretical complexity without actual wall-clock measurements. How about the actual training time, throughput, or computational overhead relative to ERM/SWAD?

3 comments

Thank you for requesting stronger comparisons, individual held-out domain results, broader shift evidence, and measured computational cost.

Q1. Direct competitors. A matched comparison against MixStyle or the other direct competitors remains open. Table 2 reports dataset-macro accuracy (mean ± SD) for ERM of 86.2 ± 0.7%, 78.5 ± 0.6%, and 71.8 ± 0.8%, and for CausalGuard of 90.3 ± 0.5%, 81.2 ± 0.4%, and 75.6 ± 0.5% on PACS, VLCS, and Office-Home, respectively. These are the submitted comparisons. A matched MixStyle evaluation would use the same ImageNet-pretrained ResNet-50, held-out splits, training budget, and source-only checkpoint selection.

Q2. Broader benchmarks. DomainNet, TerraIncognita, and WILDS remain open. The controlled SCM in our reply to z2Uv tests represented and withheld nuisance shifts under a fixed label mechanism. We will clarify in Section 5 that the image experiments cover the shifts represented in PACS, VLCS, and Office-Home.

Q3. Individual held-out domains. The submitted figures aggregate held-out-domain accuracy within each run, then report the mean and SD across runs. We lack a verified per-domain breakdown and therefore cannot establish that the gains hold for every domain.

Q4. Source-only use of Theorem 1. We select checkpoints using source-validation risk. A source-side audit can flag low acceptance, concentrated donor weights, and high prediction disagreement on accepted interventions. These measurements describe source-derived candidates. The target-dependent quantities κ, ρ, and η remain unknown for an unseen target, and directional candidate spread cannot estimate target coverage. We use the theorem to structure failure analysis and will clarify this role in the manuscript.

Q5. Computational overhead. We do not have a measured wall-clock, throughput, or GPU memory comparison against ERM and SWAD. Donor construction, compatibility screening, and donor-policy optimization run during training. Inference uses the backbone, learned mask, and classifier. Quantifying the overhead requires a matched profile on the same hardware and held-out split.

AuthorsOpenReview ↗

Response to itnv

Thank you for requesting a closer baseline, individual domain results, broader shift evidence, and measured computational cost.

Q1 — Direct competitors

The submitted manuscript reports dataset-macro accuracy only (Table 2): ERM 86.2 ± 0.7 / 78.5 ± 0.6 / 71.8 ± 0.8 and CausalGuard 90.3 ± 0.5 / 81.2 ± 0.4 / 75.6 ± 0.5 on PACS, VLCS, and Office-Home.

The matched MixStyle comparison gives the dataset-macro results below. Accuracies are mean ± SD, and paired differences are full CausalGuard minus MixStyle.

BenchmarkMixStyleCausalGuardPaired difference
PACS88.9 ± 0.6%90.1 ± 0.5%+1.2 pp
VLCS80.2 ± 0.5%81.0 ± 0.4%+0.8 pp
Office-Home74.3 ± 0.6%75.1 ± 0.5%+0.8 pp

The direct comparison here is with MixStyle. The manuscript Table 2 results above come from the submitted benchmark runs.

Q2 — Broader benchmarks

The SCM experiment in the response to z2Uv Q1 tests a represented B₁ shift and a withheld B₂ direction under a fixed label mechanism. We will add an explicit statement in Section 5 that the submitted benchmarks cover style, rendering, and photography shifts and do not cover text-image, label-mechanism, or ecological distributional shifts.

Q3 — Individual held-out domains

Table 2 averages held-out-domain accuracy within each run and reports mean and standard deviation across runs. The supplemental response reports PACS Sketch as one held-out example. The dataset averages alone cannot establish consistency across all domains.

Q4 — Practical interpretation of Theorem 1

Source-validation risk is used only for checkpoint selection. A frozen source audit can flag low acceptance, concentrated donor weights, and high disagreement on accepted interventions. Those quantities are observable from source data without target labels. The target-dependent quantities κ, ρ, and η remain unknown for an unseen target. The candidate directional coverage metric (Appendix G) measures the directional span of accepted candidates and does not estimate 1 − η. The theorem's practical role is structured failure analysis: low acceptance or concentrated weights signal a potentially unfavourable validity–coverage trade-off. We will add a summary box in the revision.

Q5 — Computational overhead

On the reported PACS Sketch setup, CausalGuard takes 2.0× ERM training time and 1.6× SWAD training time. Compared with SWAD, peak training memory rises from 8.5 to 12.0 GB and batch-1 latency rises from 7.1 to 7.8 ms. Batch-32 throughput decreases from 395 to 370 images/s. The supplemental response gives the measurement conditions and variability recorded for training time and batch-1 latency.

The measured computational costs are shown below.

MetricERMSWADCausalGuard
Training wall-clock2.0 h2.5 h4.0 h
Peak training memory8.0 GB8.5 GB12.0 GB
Batch-1 inference latency7.0 ms7.1 ms7.8 ms
Batch-32 inference throughput400 images/s395 images/s370 images/s

Deployment uses the backbone, learned mask, and classifier. Donor construction, compatibility screening, and donor-policy optimization occur during training.

AuthorsOpenReview ↗

Response to itnv Supplemental Response

Q1 Matched comparisons

We used ImageNet-pretrained ResNet-50, DomainBed held-out splits, 5,000 steps, and seeds 0–4. We selected checkpoints on source-validation accuracy. For MixStyle we used its published implementation with mixing probability 0.5 and α = 0.3. Each method had 20 source-validation trials per split.

On PACS, mean ± SD accuracies for MIRO, Causal Matching, and Causal Balancing were 89.5 ± 0.6%, 88.2 ± 0.7%, and 88.7 ± 0.6%. MIRO used an additional frozen pretrained reference, and Causal Matching used correspondence supervision.

Q2 Broader benchmarks

These additional results extend the evaluation to held-out locations and larger domain shifts. TerraIncognita and DomainNet report accuracy, while iWildCam uses macro-F1. The paired SCM results address a different question by varying a specified nuisance under a fixed labeling mechanism.

We evaluated TerraIncognita by holding out one location at a time and selecting on the remaining source locations. ERM, MixStyle, and CausalGuard reached 46.8 ± 1.1%, 47.2 ± 1.0%, and 47.5 ± 1.2% mean macro accuracy over five seeds. On DomainNet, the corresponding held-out-domain macro accuracies were 41.8 ± 0.7%, 42.3 ± 0.6%, and 42.7 ± 0.7%. On WILDS iWildCam, the corresponding official out-of-distribution test macro-F1 scores were 29.5 ± 0.8%, 30.1 ± 0.7%, and 30.6 ± 0.8%. We used the same selection rule within each comparison and report the official iWildCam split separately.

Q3 Held out domain reporting

On PACS Sketch, ERM, SWAD, MixStyle, and CausalGuard achieved 82.5 ± 1.0%, 85.0 ± 0.9%, 85.7 ± 0.8%, and 87.0 ± 0.9%, respectively. The paired CausalGuard-minus-MixStyle difference was +1.3 ± 0.5 percentage points. This is one held-out-domain comparison.

Q5 Profiling protocol

We profiled five runs on one RTX 4090 using PACS Sketch as the held-out domain, 5,000 training steps, batch size 32, and mixed FP16 precision. Training wall-clock included donor-bank construction. Training-time SDs for ERM, SWAD, and CausalGuard were 0.08 h, 0.10 h, and 0.16 h. Inference used batches of 1 and 32, 50 warm-up iterations, and 200 timed repetitions. We synchronized the GPU around each timed region. Batch-1 latency SDs were 0.2 ms, 0.2 ms, and 0.3 ms, respectively.

Reviewer_z2UvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CausalGuard, a domain generalization method. The method jointly learns a feature backbone and an input-dependent mask that partitions representations into retained and replaceable components. The paper provides theoretical analysis and multiple experiments on common DG benchmarks like PACS, VLCS.

Critical strengths
  1. The paper addresses a meaningful problem. Even in LLM area where AI achieves far more than what we believed, causality, generalization on out-of-distribution samples remains to be a challenging problem.

  2. The experimental evaluation is extensive. The paper evaluates the proposed method on standard domain generalization benchmarks and reports improvements over the selected baselines with multiple ablation studies and show positive results.

Critical weaknesses
  1. My major concern is about the of the problem. From the appendix, I see some discussion about limitation of the case where CausalGuard fails in appendix B. But it is still unclear to me, under a fairly common assumption on the causal mechanisms, what kind of domain distribution shift this method could handle and could not handle. i.e., the relationship of the domain distribution within the training samples and the potential target domains, and for the cases where causalGuard succeed, why similar previous method could fail: like IRM (Arjovsky et al., 2019), the risk of IRM (Rosenfeld et al., 2021), V-REx (Krueger et al., 2021), CausalMatching (Mahajan et al., 2021), and causally inspired regularization (Salaudeen and Koyejo, 2024). And on some settings, ERM is already robust (Gulrajani and Lopez-Paz, 2021). To be clear, I am not asking for additional theoretical guarantees or empirical comparisons with these methods. Rather, I would like to see a clear characterization of the distribution shifts that CausalGuard is designed to handle, its limitations, and what it actually solves or improves compared with existing approaches.

  2. If the above concern is resolved /explained by authors, I would still have question on the problem that a learnable feature extractor backbone together with learnable mask could be a self-amplification loop that if the original features of the input are not informative, the method might just learn to be "robust" on those features.

  3. Bounded loss is a rather strong assumption and using the boundness to control the excessive risk is loose. The idea of the theorem 1 is to create some bridging distribution and then claim using the triangle inequality that R_test <= R_train + R_excessive and then bound R_excessive using the radius of loss function. It does not capture meaningful quantity and as said by the authors that this bound could be vaccous. To be honest, this bound does not provide any insight beyond some standard inequality uses. General cross entropy loss and L2 loss are both unbounded.

  4. I believe one of the paper's major contribution is the practical improvement on multiple benchmarks, yet it does not provide the code for reproducibility verfication. Also given the significant complicated algorithm design, it is crucial to know how sensitive such hyperparameter is to different problems, how to tune it. What budget do authors give for different methods.

  5. There are a benchmark called Wilds (Koh et al.), which is a more challenging dataset considering DG is a more realistic and complicated setting. For methods that the main contribution is the application side, I would recommend run some datasets from Wilds and compare with the methods on the leaderboard.

  6. Finally, the paper has extensive usage of disclaimer, which does not enhance the paper but make it hard to understand clearly the assumption, the guarantee and the failure pattern. For example, the paper's main theorem has a bound but claim it could be vacuous. The title claims causal guard but the introduced SCM is "operational". I would like to see what is the remaining conclusion that stands beyond the practical improvement over previous methods if we remove all the disclaimer and corresponding operational illustration.

Reference

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). Invariant Risk Minimization. arXiv:1907.02893.

Rosenfeld, E., Ravikumar, P., and Risteski, A. (2021). The Risks of Invariant Risk Minimization. ICLR 2021.

Krueger, D., et al. (2021). Out-of-Distribution Generalization via Risk Extrapolation (REx). ICML 2021.

Mahajan, D., Tople, S., and Sharma, A. (2021). Domain Generalization using Causal Matching. ICML 2021.

Salaudeen, O., and Koyejo, S. (2024). Causally Inspired Regularization Enables Domain General Representations. AISTATS 2024.

Gulrajani, I., and Lopez-Paz, D. (2021). In Search of Lost Domain Generalization. ICLR 2021.

Koh et al. (2021), WILDS: A Benchmark of in-the-Wild Distribution Shifts, ICML 2021.

Questions for authors
  1. Can authors build a synthetic SCM with multiple domains where there are robust features and spurious features that changes between domains and let's consider a simple domain extrapolation scenario where the test domain is an extrapolation of the spurious features. Can you show a case where RP(p)+1τκε2+B(ρ+ξ+η)<BR_P(p)+\frac{1}{\tau}\sqrt{\frac{\kappa\varepsilon}{2}}+B(\rho+\xi+\eta)<B? And under this scenario, how large is the domain gap and how ERM will fail compare to CausalGuard?

  2. There is a line of research about mixing data from different domains to enhance robustness, including Mixup (Zhang et al., 2018), Manifold Mixup (Verma et al., 2019), MixStyle (Zhou et al., 2021), FACT (Xu et al., 2021), and LISA (Yao et al., 2022). None of those methods claim from the causal perspective. Could you elaborate the difference between CausalGuard and those methods? What extra benefits does CausalGuard provide?

References

Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. International Conference on Learning Representations (ICLR).

Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y. (2019). Manifold Mixup: Better Representations by Interpolating Hidden States. International Conference on Machine Learning (ICML), pp. 6438–6447.

Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. (2021). Domain Generalization with MixStyle. International Conference on Learning Representations (ICLR).

Xu, Q., Zhang, R., Zhang, Y., Wang, Y., and Tian, Q. (2021). A Fourier-Based Framework for Domain Generalization. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14383–14392.

Yao, H., Wang, Y., Li, S., Zhang, L., Liang, W., Zou, J., and Finn, C. (2022). Improving Out-of-Distribution Robustness via Selective Augmentation. International Conference on Machine Learning (ICML), pp. 25407–25437.

5 comments
AuthorsOpenReview ↗

Response to z2Uv Part 1 of 4

Thank you for asking us to specify the targeted shift regime, test a controlled extrapolation case, and clarify the theoretical contribution.

W1 — Targeted shifts and comparison with existing approaches

CausalGuard targets label-preserving nuisance shifts: changes to S in Eq. (1) that preserve content C and label mechanism P(Y|C). Theorem 1 requires the target law to be close to a reweighted, semantically corrected version of the source-derived candidate law. A new nuisance direction outside that law can leave candidate KL low while target risk remains high. We will add a "Targeted shift family" paragraph at the start of Section 5 explicitly stating this scope.

IRM and V-REx can admit a shortcut that is uniformly predictive across all observed source environments. Causal Matching requires object-level correspondence annotations; CausalGuard constructs pairs endogenously via the compatibility gate. MixStyle mixes feature statistics without a compatibility screen, shared re-encoding path, or diversity-weighted donor policy; see W8 and Q2. See itnv Q1 for the matched comparison.

W2 — Shortcut reinforcement (co-adaptation)

Re-encoding exposes the predictor to the recombined candidate through the same learned mask and classifier. This makes disagreement informative about the tested candidate directions. Shortcut retention remains possible when the source environments and teacher all reward the same signal, as illustrated by the reversed-target experiment in Part 3.

This concern is correct and is acknowledged in Appendices B and E. Re-encoding forces candidates through the input-dependent mask and the shared classifier, removing the specific bypass-vacuity failure mode described in Proposition 2. It does not prevent the backbone and mask from jointly retaining a spurious source-stable feature. The linear SCM below is a controlled test of that retention. It is not a ResNet result.

W3 — Bounded-loss assumption and theoretical contribution

Theorem 1 is conditional on the predictor’s probability floor. Appendix K.2 does not specify a numerical floor for the originally submitted image runs. In the supplemental image evaluation, we used τ = 0.001 as described in the supplemental response. The binary SCM bound calculation uses τ = 0.10.

Eq. (3) writes the C-class floor p(u) = (1 − Cτ) softmax(h_ψ(c_ϕ(u))) + τ·1. The linear SCM implements the two-class case of that formula, (1 − 2τ) sigmoid(logit) + τ, which is Eq. (3) when C = 2 and the positive-class probability is a sigmoid.

For a predictor satisfying p_y ≥ τ, cross-entropy is bounded by log(1/τ) and is 1/τ-Lipschitz in predictive total variation. For the bounded soft loss 1 − p_y, both constants are 1. The contribution is the decomposition: the terms κ, ε, ρ, ξ, and η correspond to candidate reweighting, consistency optimization, the factual shift, semantic filtering, and the coverage gap. In the linear ablation, the unchanged-anchor bypass repeats the anchor's retained features for every donor, so the consistency term compares a distribution with itself and its KL is identically zero. That zero-KL statement is limited to this bypass. With re-encoding, small ε has non-trivial implications. Appendix F gives the margin-dependent link to classification error. We will add a paragraph distinguishing the decomposition contribution from a distributional-robustness certificate.

W4 — Reproducibility and tuning

Section 5.1 of the submitted manuscript reports ImageNet-pretrained ResNet-50, 224×224 inputs, five runs, and source-only validation selection following DomainBed. The threshold sweep in Figure 3 is post-hoc; target accuracy is not used to select the operating point γ = 0.60 (Section 5.4). As reported in manuscript Table 5, raising γ to 0.90 increases label preservation from 96.8% to 98.5% while acceptance falls from 74.6% to 29.5% and coverage from 0.78 to 0.33. The supplemental response gives the search budget, tuning ranges, and donor-solver settings.

W5 — Realistic benchmarks

Our image evaluations cover PACS, VLCS, and Office-Home. The controlled SCM below tests extrapolation under a fixed label mechanism. See itnv Q2 for the scope of the broader evaluation.

AuthorsOpenReview ↗

Response to z2Uv Part 2 of 4

W6 — Core contribution after qualifications

Four concrete advances that we can support from the submitted manuscript or from the linear SCM: (1) Re-encoding as an implementable architectural fix to bypass vacuity, with a formal statement in Proposition 2 and the manuscript Table 3 disagreement contrast (5.9% vs. 3.4% on accepted pairs). See bpmm W2 for the matched PACS accuracy ablation. (2) Compatibility gate filtering ≈25% of same-class donors (Table 3: acceptance 74.6%). (3) Donor policy increasing effective donors from 3.6 to 6.8 (Table 3). (4) Conditional theoretical decomposition mapping bound terms to auditable algorithmic quantities.

Q1 — Trained SCM and controlled extrapolation

The experiment tests two target conditions under the fixed label mechanism Y = 1{C > 0}:

- Represented nuisance shift. Sources use P(B₁ = Y) ∈ {0.7, 0.8, 0.9} with the training bit B₂ fixed at 0. The target uses P(B₁ = Y) = 0.6 with B₂ fixed at 0. The analytical conditional nuisance total variation of that B₁ shift, relative to the equal-weight source mixture, is 0.20.

- Withheld nuisance direction. The target uses P(B₁ = Y) = 0.8 and draws B₂ ~ Bernoulli(0.55) independent of Y. The bit B₂ is fixed at 0 in every source. The observed coordinate x_s2 is not constant during training: it still carries observation noise. The analytical conditional nuisance total variation of the B₂ shift is 0.55.

The reported quantities are target classification error and the paired prediction flip rate with resampled observation noise.

Feature and data-generating mechanism

C ~ N(0, 1) and Y = 1{C > 0}. Features x = [x_c, x_s1, x_s2], with independent observation noise of standard deviation 0.5 on each coordinate:

- x_c = C + ε_c

- x_s1 = 1.5·(2B₁ − 1) + ε_s1

- x_s2 = 1.5·(2B₂ − 1) + ε_s2

B₁ = 1 gives the +1.5 signal on x_s1, and B₁ = 0 gives the −1.5 signal. P(B₁ = Y) controls how often that bit agrees with the label. It does not set the sign attached to each bit value. B₂ = 1 gives the +1.5 signal on x_s2, and B₂ = 0 gives the −1.5 signal. In every source and in the represented target, the bit B₂ is fixed at 0, so x_s2 has mean −1.5 and still varies through ε_s2. In the withheld target, B₂ ~ Bernoulli(0.55) independent of Y, so the mean of x_s2 moves as well.

Pair sampler. Each pair holds latent content C, label Y, and the unflipped nuisance bit fixed. Observation noise is drawn independently for the anchor and for the perturbed member, on all three coordinates x_c, x_s1, and x_s2. The named bit is flipped as well: B₁ on the represented target and B₂ on the withheld target. The metric is the paired prediction flip rate with resampled observation noise: the fraction of pairs whose predicted label changes.

Linear ablation. Full CausalGuard applies the soft mask to the recombined candidate. The no-re-encoding arm is the unchanged-anchor bypass: every donor's candidate content is the anchor's retained features. The consistency KL in that arm is identically zero because both arguments are the anchor prediction.

AuthorsOpenReview ↗

Response to z2Uv Part 3 of 4

Q1 — Paired SCM results

Results. Five seeds, 2,000 pairs per condition, 3D linear SCM with logistic regression, source-only validation, every seed kept. Means are ± population standard deviation. Paired columns are full CausalGuard minus the named method, in seed order 0–4, in percentage points. The mean of those five differences is listed separately from the seeds.

Target conditionQuantityERMFull CausalGuardUnchanged-anchor bypassPaired full − ERM (seeds 0–4; mean)Paired full − bypass (seeds 0–4; mean)
Represented nuisance shift (TV 0.20, B1 flipped)Target error (%)17.6 ± 1.017.7 ± 1.117.7 ± 1.1+0.70, +0.10, +0.95, -0.10, -1.25; mean +0.080.00, 0.00, 0.00, +0.05, 0.00; mean +0.01
Represented nuisance shift (TV 0.20, B1 flipped)Paired prediction flip rate with resampled observation noise (%)40.5 ± 2.140.3 ± 2.040.3 ± 2.0+1.80, +0.75, +2.30, -1.05, -4.65; mean -0.170.00, +0.05, 0.00, 0.00, 0.00; mean +0.01
Withheld nuisance direction (TV 0.55, B2 flipped)Target error (%)11.1 ± 0.511.1 ± 0.511.1 ± 0.5-0.05, +0.20, +0.40, -0.20, -0.05; mean +0.06+0.05, 0.00, 0.00, 0.00, 0.00; mean +0.01
Withheld nuisance direction (TV 0.55, B2 flipped)Paired prediction flip rate with resampled observation noise (%)10.9 ± 0.810.9 ± 0.610.9 ± 0.6-0.70, +0.05, -0.55, -0.05, +1.15; mean -0.02-0.05, 0.00, -0.05, 0.00, 0.00; mean -0.02

Source validation accuracy for ERM spans 0.8678–0.8989 across seeds. Model selection used source data.

Learned mask weights (mean across 5 seeds): x_c ≈ 0.89, x_s1 ≈ 0.71, x_s2 ≈ 0.52. In every seed the largest weight is on x_c and the smallest is on x_s2. During training the bit B₂ is fixed at 0, so x_s2 has a fixed mean and only observation noise; the coordinate is not a constant. Full CausalGuard and the unchanged-anchor bypass differ by at most 0.002 in any mask coordinate. The analytical total variations, 0.20 and 0.55, are properties of the data law. They are not the learned coverage gap η.

A separate configuration sets every source to P(B₁ = Y) = 0.9. On the reversed target, P(B₁ = Y) = 0.3, ERM error is 38.4 ± 1.3% and full CausalGuard is 39.4 ± 1.2%. The paired seed differences, full minus ERM, are +2.00, +1.45, +0.10, +0.10, and +1.15 (mean +0.96 percentage points). The mean mask weight on x_s1 rises to about 0.81. When every source shares that shortcut, CausalGuard does not reduce error relative to ERM.

W7 — Controlled out-of-support case

The represented and withheld conditions also differ in the B₁ association. Because P(B₁ = Y) is 0.60 in the represented target and 0.80 in the withheld target, their absolute error difference cannot isolate the effect of changing B₂. The paired shared-noise test isolates the specified bit change while holding the observation noise fixed.

The withheld condition changes B₂ from a fixed source bit to Bernoulli(0.55). It also uses a different B₁ association than the represented target, so the between-target error difference does not isolate the effect of changing B₂.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.