- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper focuses on the limitation of Counterfactual Consistency Learning for domain generalization. This paper proposes the CausalGuard framework, which consists of: (1) Re-encoding, which passes the counterfactual features through the same mask and classifier as the original features, so that the consistency does not hold merely because the disturbed features are discarded; and (2) Compatibility-constrained donor selection, which uses a frozen semantic teacher to filter out the candidates that may change the class semantics, and then weights the remaining donors by balancing prediction difficulty and diversity. A Coverage-qualified risk transfer analysis is conducted to show that low inconsistency alone is insufficient. Experiments were performed for evaluation.
Critical strengths
Counterfactual consistency learning is an important problem.
Critical weaknesses
- This paper has serious presentation issues which makes the evaluation task very hard.
- Section 1 "Introduction": g(c) is used without definition. The terms like "augementation", "branch", "intervention", "shortcut", "zero consistency" are used without explanation. In the second paragraph, CausalGuard is described without sufficient background. Terms like "re-encoding", "donor", "architectural source of vacuity". With unclear description of the background, the third paragraph starts with "Two further problems remain". And for the whole introduction section, no citations are provided.
- Section 2 "Related Work": Prior works are presented without sufficient background and careful articulation.
- Section 3 "CausalGuard": Most of the symbols used in Eq(1) are not defined. Many terms are not defined. E.g., "semantic filter", "operational evidence", "anchor", "donor", ..
- All the otther sections share the same problem
- The figures in the paper are not well-prepared and are very hard to follow.
- Experimental results are not providing sufficient evidence to demonstrate the effectiveness of CausalGuard. It is not clear if the model, after re-encoding, starts to focus on the correct features rather than the spurious ones. Is there any empirical evidence? Table 2 does not report the performance of CausalGuard without Re-encoding. Table 3 only shows the intervention diagnostics with different components removed, but accuracy is not included. In general, more effort should be spent on conducting ablation studies.
Questions for authors
- There are many other datasets for IRM. What are the key reasons for selecting the particular three?
- Why is feature diversity representing the target coverage? Is there any empirical evidence?
5 comments
Response to bpmm Part 2 of 2
W3 — Evidence about robust vs. spurious features
A source-stable feature can still be spurious. The linear SCM assigns mean mask weights of 0.89 to content and 0.71 and 0.52 to nuisance coordinates. The content coordinate ranks highest in all five seeds. When all sources share P(B₁ = Y) = 0.90, CausalGuard has 0.96 points higher mean target error than ERM on the reversed target, and the x_s1 mask weight rises to approximately 0.81. This illustrates the shared-shortcut failure discussed in Appendix E. Causal feature recovery in the deep model remains unestablished.
Q1 — Dataset choice rationale
PACS tests shifts across depiction styles (art, cartoon, photo, sketch). VLCS spans photographic datasets evaluated on shared categories (VOC2007, LabelMe, Caltech101, SUN09), representing photography-condition and coverage shifts. Office-Home covers four object-recognition domains (Art, Clipart, Product, Real World) with diverse visual styles. We follow DomainBed to enable comparison with a large existing literature. We will add an explicit rationale paragraph in Section 5: these benchmarks cover nuisance-driven style and rendering shifts and do not cover text-image, label-mechanism, or ecological distributional shifts.
Q2 — Candidate diversity and target coverage
The normalized coverage metric is the entropy-based effective rank of the normalized intervention second-moment matrix G̃ = G/tr(G): r_eff = exp(−Σ_k λ_k(G̃) log λ_k(G̃)) (Appendix G, Eq. 29). This measures the directional span of accepted candidate displacements. It does not measure target-support coverage. Table 5's caption states: "The coverage column is a normalized diagnostic; it is not an estimate of 1 − η in Theorem 1." Appendix G: "Full-rank candidate covariance does not establish a density-ratio bound or identify the target discrepancy η."
We will add a labeled definition box in the main text and an explicit sentence: "This metric measures the directional span of learned candidates; it does not estimate the target support discrepancy η in Theorem 1, which is not identifiable from source data alone." Table 8 reports acceptance of 76.4–82.5% for nuisance-changing controls and 11.2–18.6% for class-changing controls, supporting the audit on those specific tested controls.
Response to bpmm Supplemental Response
W2 PACS ablation pathway and paired uncertainty
The ablation compares two prediction pathways on the recombined feature: masking it with the anchor’s weights and recomputing the weights from the candidate. The observed PACS macro gain is descriptive across the evaluated splits and seeds. The Photo result differs from the other three domains, so the table does not imply a uniform improvement.
The no-re-encoding arm passed the recombined candidate through the anchor’s fixed soft mask and the shared classifier. The full arm recomputed the mask on the candidate. We held the backbone, donor policy, compatibility gate, consistency weight, training budget, and source-validation checkpoint rule fixed across seeds 0–4. The SDs of the paired full-minus-ablation changes were 0.50, 0.55, 0.25, and 0.65 percentage points on Art Painting, Cartoon, Photo, and Sketch, respectively. The paired macro change was +0.95 ± 0.38 points after averaging the four differences within each seed.
W2 Table 3 implementation
The Table 3 no-re-encoding variant used the anchor’s fixed soft mask on the recombined candidate. We used a shared set of accepted pairs across variants. The denominator was 2,000 accepted pairs. Disagreement counted factual and candidate predictions with different top-1 labels.
W3 Controlled image level nuisance sensitivity
On these edited pairs, the full model’s clean-to-edited accuracy drop was 2.5 percentage points, compared with 3.9 for the ablation. Its paired flip rate was 3.1 points lower. This tests sensitivity to the particular background edits and complements the linear SCM experiments, which change known nuisance variables.
We evaluated 1,000 PACS pairs with identical foregrounds and labels and controlled background replacements. We inspected all edited pairs for class preservation. The full and ablated models achieved clean accuracy of 89.4% and 88.5%, edited accuracy of 86.9% and 84.6%, and paired flip rates of 7.1% and 10.2%, respectively.
Q2 Directional spread and target discrepancy
In the represented SCM, the accepted-candidate effective rank was 1.72. Reweighting within each label matches B₁ alignment with B₂ fixed at zero. Effective rank measures the directional spread of accepted candidates. The full target-to-bridge discrepancy also accounts for within-cell differences. The separate full-law upper bound for this shift is η ≤ 0.12, as reported in our response to z2Uv, Part 4.
Response to bpmm Part 1 of 2
Thank you for identifying the presentation issues and asking for an accuracy ablation of re-encoding.
W1 — Definitions, related work, and figures
*b vs. b₀ notation:* Eq. (7) uses b₀ for the mask budget scalar and states "The notation b₀ distinguishes the scalar mask budget from semantic map b." Table 4 (Appendix A) does not include b₀. We will add b₀ to Table 4 and audit all manuscript occurrences to ensure consistent disambiguation.
*w overloading:* w is used for density-ratio reweighting in Section 4 and implicitly for classifier parameters in Section 3. We will introduce θ_cls for classifier parameters and add both to Table 4.
*Structural variable definitions:* We will define the retained component c, replaceable component s, and classifier g(c) explicitly before Eq. (1). We will define anchor (the example being classified), donor (a separate same-class source example), and compatibility filter (the frozen teacher screen) at first occurrence in the main text.
We will enlarge the labels in Figure 1 and clarify its arrow legend. Solid arrows will show the computational sequence and dashed arrows the factual and candidate inputs. We will identify VLCS domain-macro accuracy on the y-axis of Figure 3.
*Citations:* DomainBed (Gulrajani & Lopez-Paz, 2021) and shortcut learning (Geirhos et al., 2020) will be added to the introduction. Comparator method objectives (IRM, V-REx, SWAD, MixStyle) will be described in Related Work with explicit comparisons.
W2 — Accuracy and known-nuisance tests for the re-encoding ablation
The matched PACS accuracy ablation gives the results below. The domain macro improves by 0.95 percentage points, with gains on Art Painting, Cartoon, and Sketch and a 0.2-point decrease on Photo.
Table 3 and held-out accuracy answer different questions. The manuscript Table 3 figures of 5.9% (no re-encoding) and 3.4% (full model) are prediction disagreement: the fraction of accepted pairs whose factual top-1 prediction and candidate top-1 prediction differ. They are not held-out domain accuracy.
The linear ablation uses a third quantity, the paired prediction flip rate with resampled observation noise. Each pair holds latent content C, label Y, and the unflipped nuisance bit fixed. Observation noise is drawn independently on all three input coordinates for both pair members, and the named nuisance bit is flipped. That rate is not Table 3 disagreement and not held-out accuracy.
In this linear ablation the no-re-encoding arm is the unchanged-anchor bypass: candidate content is the anchor's retained features, so the consistency KL is identically zero and factual and candidate predictions agree. Under the same evaluation procedure, the exact bypass would yield zero disagreement. Table 3 uses the fixed anchor-mask pathway described in the supplemental response. Its no-re-encoding arm retains a recombined candidate input.
Paired prediction flip rate with resampled observation noise (3D linear SCM, 5 seeds, 2000 pairs)
Under the represented B₁ shift (analytical conditional TV = 0.20): ERM 40.5 ± 2.1%, unchanged-anchor bypass 40.3 ± 2.0%, CausalGuard 40.3 ± 2.0%. The mean paired full − ERM gap is −0.17 percentage points. The five seeds are +1.80, +0.75, +2.30, −1.05, and −4.65. Under the withheld B₂ direction (TV = 0.55): ERM 10.9 ± 0.8%, bypass 10.9 ± 0.6%, CausalGuard 10.9 ± 0.6% (mean paired full − ERM −0.02 percentage points). The seed-level withheld gaps are not all equal to that mean. Full CausalGuard and the bypass stay within 0.05 percentage points on target error in every seed. Mean target-error differences versus ERM are +0.08 percentage points (represented) and +0.06 percentage points (withheld). The side-by-side seeds are in the response to z2Uv Q1.