- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies out-of-distribution generalization for image classification. The authors argue that cross-domain stability of learned representations does not imply causal effectiveness: a feature may be stable across source domains yet fail to drive classification under distribution shift. They propose to re-evaluate frozen backbone representations from the perspective of causal driving force. Layer-wise representations of a frozen vision transformer are treated as segments of a representation-level structural causal model. Through complementary treatment and control interventions, which replace selected segments with reference features drawn from a class- and domain-balanced distribution, the probability of necessity and sufficiency of each segment for cross-domain prediction is characterized by a two-sided continuous-PNS bound and converted into causal effectiveness scores. High-scoring segments are selected to form a representation subspace, on which the classifier is retrained. Experiments on three datasets and three pretrained backbones show consistent improvements on unseen domains, and theoretical derivations are provided for the proposed scoring mechanism.
Critical strengths
-
Novel and refreshing idea.The core reframing is crisp: cross-domain stability is observational, and does not imply causal effectiveness. This directly targets a real weakness of invariance-based objectives. The operationalization is also new, to my knowledge. It pairs a sufficiency-style probe with a necessity-style probe as complementary interventions on representation segments, and aggregates them conservatively via min{S⁺, N⁺}. Prior interventional feature evaluation is largely one-sided. Prior uses of PNS in OOD learning serve as training losses, not post-hoc selection criteria.
-
Detailed and rigorous theory. The method is fully formalized: the representation-level SCM, the reference distribution, the continuous-PNS extension, the two-sided bound, and its reduction to the classical binary case. I checked the derivations and found them correct. The assumptions are also stated honestly. For example, the anchor orderings are treated as empirically verifiable conditions, and they are verified rather than assumed.
-
Significant empirical improvements. All methods run on the same frozen features, across three datasets and three backbones. The method is best in 8 of 9 settings, with decisive margins in most (e.g., +2.55 on PACS with DINOv2). It needs only source-domain data, one forward pass, and linear heads.
Critical weaknesses
-
Incomplete analyses and missing details. The ablation covers PACS only, and efficiency is given only asymptotically — no wall-clock time or compute comparison. No sensitivity analysis covers K, R, or the reference design; the value of K and the selected layers are never stated at all.
-
The attribution analysis is disproportionate to the method, leaving the true gain undeterminable. The method carries a full causal apparatus, yet the evidence attributing the gains is thin. All baselines probe the 12-layer concatenation, which Table 4 shows is 1.8–2.7 points below a single well-chosen layer.
-
The paper's structure buries the method. The core pipeline is simple: score layers by swap interventions, keep the top K, retrain a head. But it appears only at the end of Section 3.4. Before that come an edge-free SCM, a two-sided bound, and an information-theoretic derivation. None of this theory is instantiated empirically. The reader thus struggles to identify the method's key point. Related work is also thin: one short paragraph, omitting layer probing, removal-based attribution, and a real comparison with CaSN. Moving the derivations to an appendix, stating the algorithm up front, and reporting the selected layers would sharpen the paper.
Questions for authors
-
Layer selection is a core component of the method, yet is never specified. The value of K, the rule for choosing it, and the selected layers themselves are all unreported. The method's key step can neither be reproduced nor verified.
-
The contribution of the causal mechanism is not demonstrated. On Office-Home and DomainNet, published linear probes of the same frozen DINOv2 ViT-B/14 already exceed the submission's best results (84.2 vs. 81.41; 53.5–55.9 vs. 53.11). The causal mechanism is the paper's core contribution, so its specific effect must be isolated. The current experiments do not do this: no control separates the causal scoring from simple layer selection. We cannot determine the true source of the gains from the paper as written, and hope the authors can clarify this in the rebuttal.
Using causal inference for OOD generalization is a genuinely inspiring direction. I want to see this line of work succeed. The current manuscript, however, has notable structural problems and a high reading cost. The attribution of its gains also remains unresolved. If the authors can answer this through the requested controls or a convincing explanation, the core contribution would be solidified. I would then consider raising my rating to weak accept.
12 comments
Q2. Independent Contribution of PNS Scoring.
To further evaluate the independent contribution of PNS scoring, we added comparisons with ALF, Head2Toe, ILC, and a Probe baseline at both layer and attention-head granularity.
In the original layer-level experiments, PNS exceeds Probe by 0.64, 0.20, and 0.00 percentage points on PACS, Office-Home, and DomainNet, respectively. We hypothesized that this is due to the coarse granularity of the original selection space, which contains only 12 complete layers. Different scoring criteria can identify the same or similar high-performing layers, leading to substantial overlap between selected subsets and limiting the observable advantage of PNS scoring.
To test this hypothesis, we refined the selection granularity from 12 complete layers to 144 individual attention heads, expanding the candidate space, and repeated the comparisons with other representation selection methods.
The following table reports the average target-domain accuracies across three pretrained backbones.
| Method | PACS | Office-Home | DomainNet |
|---|---|---|---|
| Layer-level Selection | |||
| ALF | 76.41 | 68.72 | 40.28 |
| Head2Toe | 76.36 | 69.08 | 40.63 |
| ILC | 75.60 | 68.91 | 40.62 |
| Probe-K | 75.77 | 68.88 | 40.63 |
| PNS Layer | 76.41 | 69.08 | 40.63 |
| Head-level Selection | |||
| ALF | 75.93 | 67.03 | 39.97 |
| Head2Toe | 76.52 | 69.08 | 41.05 |
| ILC | 76.59 | 68.73 | 40.86 |
| Probe-K | 76.72 | 68.86 | 40.97 |
| PNS Head | 79.70 | 69.62 | 41.61 |
| PNS vs. Best Head Baseline | +2.98 | +0.54 | +0.56 |
The results support our hypothesis. At the original layer level, PNS achieves accuracy comparable to other selection methods, suggesting that the coarse selection space limits the differences between scoring criteria.
At the finer head level, PNS achieves the highest average accuracy on all three datasets, outperforming the strongest competing methods by 2.98, 0.54, and 0.56 percentage points on PACS, Office-Home, and DomainNet, respectively. Compared with the Probe baseline, PNS improves accuracy by 2.98, 0.76, and 0.64 percentage points, respectively.
Head-level PNS also achieves higher average accuracy than layer-level PNS on all three datasets. These results show that the joint necessity-and-sufficiency criterion better distinguishes informative representation segments at finer granularity and provide further evidence of the independent contribution of PNS scoring.
Comparison with Previously Reported Linear Probe Results. The reviewer cites linear-probe accuracies reported in PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization (2026): 84.2% on Office-Home and 53.5% on DomainNet. Adapting to Distribution Shift by Visual Domain Prompt Generation (ICLR 2024) reports 55.9% on DomainNet.
Linear-probe results may differ under different experimental configurations. We examined the public repositories of both studies, but could not find the specific probe implementations or complete training configurations corresponding to these reported results. We therefore do not know their exact probe settings.
Under our own experimental configuration with a frozen DINOv2 ViT-B/14 backbone, PNS Layer achieves 81.41% on Office-Home and 53.11% on DomainNet. We also ran the Probe baseline locally under our experimental configuration, with the results reported in Q2. At the head level, PNS outperforms Probe by 2.98, 0.76, and 0.64 percentage points on PACS, Office-Home, and DomainNet, respectively.
W1. Computational Cost and Wall-Clock Runtime.
We measured four stages of the proposed method: frozen feature extraction, scoring-classifier fitting, PNS scoring, and final classifier training. We conducted the timing experiments on PACS and Office-Home with three pretrained backbones, for both layer- and head-level selection. All times are reported in seconds. The pipeline timings are aggregated over four leave-one-domain-out splits and averaged across three runs, while feature extraction is a one-time cost. The Total column records separately measured end-to-end wall-clock time; the other columns report the four main stages.
| Dataset | Backbone | Level | Extraction | Fitting | PNS | Final Head | Total |
|---|---|---|---|---|---|---|---|
| PACS | Sup. ViT | Layer | 10.60 | 57.38 | 1.72 | 7.72 | 79.89 |
| PACS | Sup. ViT | Head | 12.92 | 25.81 | 7.90 | 6.54 | 55.59 |
| PACS | DINOv2 | Layer | 14.40 | 23.88 | 1.68 | 6.69 | 48.49 |
| PACS | DINOv2 | Head | 17.10 | 15.37 | 8.18 | 6.66 | 49.39 |
| PACS | MAE | Layer | 11.49 | 89.09 | 1.71 | 6.66 | 110.86 |
| PACS | MAE | Head | 14.96 | 37.33 | 7.89 | 6.79 | 69.44 |
| Office-Home | Sup. ViT | Layer | 19.61 | 126.73 | 4.07 | 10.13 | 163.87 |
| Office-Home | Sup. ViT | Head | 21.70 | 37.17 | 28.64 | 10.14 | 101.13 |
| Office-Home | DINOv2 | Layer | 25.87 | 107.28 | 4.04 | 9.68 | 149.71 |
| Office-Home | DINOv2 | Head | 28.55 | 24.14 | 28.89 | 10.14 | 95.33 |
| Office-Home | MAE | Layer | 22.33 | 159.51 | 4.07 | 11.65 | 201.82 |
| Office-Home | MAE | Head | 25.68 | 54.09 | 28.48 | 10.39 | 122.28 |
Layer-level PNS scoring takes approximately 1.7 seconds on PACS and 4.1 seconds on Office-Home. With the candidate space expanded from 12 layers to 144 attention heads, scoring takes approximately 8 and 29 seconds, respectively.
PNS scoring is performed only during offline representation selection. Once the selected representations and final classifier are obtained, no additional PNS scoring is needed at inference time. The detailed stage-wise timing results and measurement protocol have been added to the Appendix.
Q1. Layer Selection and Reproducibility
We thank the reviewer for the careful reading and detailed comments. We have revised the manuscript and conducted additional experiments to address the points below.
In the original layer-level experiments, we selected K from {1,2,3,4} using the validation set. For PACS, Office-Home, and DomainNet, respectively, the selected values are K=(2,1,1) for DINOv2, K=(2,4,4) for MAE, and K=(4,2,4) for Supervised ViT. The new head-level experiments follow the same validation-based procedure, with K selected from {8,12,16,24,36}.
We have added four detailed tables to the Appendix: Table 7 reports the layer-level target-domain accuracies and K values; Table 8 lists the selected layer indices; Table 9 reports the head-level accuracies and K values; and Table 10 lists the selected attention-head indices. These tables cover all 42 dataset–backbone–target configurations across PACS, Office-Home, and DomainNet, using three pretrained backbones.