acceptodds
Under review as a conference paper at ICLR 2027

Causal-Driven Feature Evaluation for Cross-Domain Image Classification

Abstract

Out-of-distribution (OOD) generalization remains a core challenge in real-world classification. Most existing methods for OOD and domain generalization focus on cross-domain stability. However, cross-domain stability alone does not imply causal effectiveness. A feature may appear stable across source domains, yet fail to truly drive classification decisions under distribution shift. We re-evaluate learned representations from the perspective of causal driving force. By intervening on feature representations, we explicitly measure the necessity and sufficiency of each representation segment for cross-domain prediction and convert these measurements into causal effectiveness scores. Based on these scores, we select high-scoring segments to form a representation subspace and retrain the classifier on this subspace, encouraging the model to rely more on representation components that drive classification and thereby improving robustness on unseen domains. We validate the proposed method on multiple datasets and pretrained backbones, where it consistently improves classification performance on unseen domains. We further provide theoretical derivations for the proposed method.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
out-of-distribution generalization, probability of necessity and sufficiency (PNS), representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qBMFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies out-of-distribution generalization for image classification. The authors argue that cross-domain stability of learned representations does not imply causal effectiveness: a feature may be stable across source domains yet fail to drive classification under distribution shift. They propose to re-evaluate frozen backbone representations from the perspective of causal driving force. Layer-wise representations of a frozen vision transformer are treated as segments of a representation-level structural causal model. Through complementary treatment and control interventions, which replace selected segments with reference features drawn from a class- and domain-balanced distribution, the probability of necessity and sufficiency of each segment for cross-domain prediction is characterized by a two-sided continuous-PNS bound and converted into causal effectiveness scores. High-scoring segments are selected to form a representation subspace, on which the classifier is retrained. Experiments on three datasets and three pretrained backbones show consistent improvements on unseen domains, and theoretical derivations are provided for the proposed scoring mechanism.

Critical strengths
  1. Novel and refreshing idea.The core reframing is crisp: cross-domain stability is observational, and does not imply causal effectiveness. This directly targets a real weakness of invariance-based objectives. The operationalization is also new, to my knowledge. It pairs a sufficiency-style probe with a necessity-style probe as complementary interventions on representation segments, and aggregates them conservatively via min{S⁺, N⁺}. Prior interventional feature evaluation is largely one-sided. Prior uses of PNS in OOD learning serve as training losses, not post-hoc selection criteria.

  2. Detailed and rigorous theory. The method is fully formalized: the representation-level SCM, the reference distribution, the continuous-PNS extension, the two-sided bound, and its reduction to the classical binary case. I checked the derivations and found them correct. The assumptions are also stated honestly. For example, the anchor orderings are treated as empirically verifiable conditions, and they are verified rather than assumed.

  3. Significant empirical improvements. All methods run on the same frozen features, across three datasets and three backbones. The method is best in 8 of 9 settings, with decisive margins in most (e.g., +2.55 on PACS with DINOv2). It needs only source-domain data, one forward pass, and linear heads.

Critical weaknesses
  1. Incomplete analyses and missing details. The ablation covers PACS only, and efficiency is given only asymptotically — no wall-clock time or compute comparison. No sensitivity analysis covers K, R, or the reference design; the value of K and the selected layers are never stated at all.

  2. The attribution analysis is disproportionate to the method, leaving the true gain undeterminable. The method carries a full causal apparatus, yet the evidence attributing the gains is thin. All baselines probe the 12-layer concatenation, which Table 4 shows is 1.8–2.7 points below a single well-chosen layer.

  3. The paper's structure buries the method. The core pipeline is simple: score layers by swap interventions, keep the top K, retrain a head. But it appears only at the end of Section 3.4. Before that come an edge-free SCM, a two-sided bound, and an information-theoretic derivation. None of this theory is instantiated empirically. The reader thus struggles to identify the method's key point. Related work is also thin: one short paragraph, omitting layer probing, removal-based attribution, and a real comparison with CaSN. Moving the derivations to an appendix, stating the algorithm up front, and reporting the selected layers would sharpen the paper.

Questions for authors
  1. Layer selection is a core component of the method, yet is never specified. The value of K, the rule for choosing it, and the selected layers themselves are all unreported. The method's key step can neither be reproduced nor verified.

  2. The contribution of the causal mechanism is not demonstrated. On Office-Home and DomainNet, published linear probes of the same frozen DINOv2 ViT-B/14 already exceed the submission's best results (84.2 vs. 81.41; 53.5–55.9 vs. 53.11). The causal mechanism is the paper's core contribution, so its specific effect must be isolated. The current experiments do not do this: no control separates the causal scoring from simple layer selection. We cannot determine the true source of the gains from the paper as written, and hope the authors can clarify this in the rebuttal.

Using causal inference for OOD generalization is a genuinely inspiring direction. I want to see this line of work succeed. The current manuscript, however, has notable structural problems and a high reading cost. The attribution of its gains also remains unresolved. If the authors can answer this through the requested controls or a convincing explanation, the core contribution would be solidified. I would then consider raising my rating to weak accept.

12 comments
AuthorsOpenReview ↗

Q1. Layer Selection and Reproducibility

We thank the reviewer for the careful reading and detailed comments. We have revised the manuscript and conducted additional experiments to address the points below.

In the original layer-level experiments, we selected KK from {1,2,3,4}\{1,2,3,4\} using the validation set. For PACS, Office-Home, and DomainNet, respectively, the selected values are K=(2,1,1)K=(2,1,1) for DINOv2, K=(2,4,4)K=(2,4,4) for MAE, and K=(4,2,4)K=(4,2,4) for Supervised ViT. The new head-level experiments follow the same validation-based procedure, with KK selected from {8,12,16,24,36}\{8,12,16,24,36\}.

We have added four detailed tables to the Appendix: Table 7 reports the layer-level target-domain accuracies and KK values; Table 8 lists the selected layer indices; Table 9 reports the head-level accuracies and KK values; and Table 10 lists the selected attention-head indices. These tables cover all 42 dataset–backbone–target configurations across PACS, Office-Home, and DomainNet, using three pretrained backbones.

AuthorsOpenReview ↗

Q2. Independent Contribution of PNS Scoring.

To further evaluate the independent contribution of PNS scoring, we added comparisons with ALF, Head2Toe, ILC, and a Probe baseline at both layer and attention-head granularity.

In the original layer-level experiments, PNS exceeds Probe by 0.64, 0.20, and 0.00 percentage points on PACS, Office-Home, and DomainNet, respectively. We hypothesized that this is due to the coarse granularity of the original selection space, which contains only 12 complete layers. Different scoring criteria can identify the same or similar high-performing layers, leading to substantial overlap between selected subsets and limiting the observable advantage of PNS scoring.

To test this hypothesis, we refined the selection granularity from 12 complete layers to 144 individual attention heads, expanding the candidate space, and repeated the comparisons with other representation selection methods.

The following table reports the average target-domain accuracies across three pretrained backbones.

MethodPACSOffice-HomeDomainNet
Layer-level Selection
ALF76.4168.7240.28
Head2Toe76.3669.0840.63
ILC75.6068.9140.62
Probe-K75.7768.8840.63
PNS Layer76.4169.0840.63
Head-level Selection
ALF75.9367.0339.97
Head2Toe76.5269.0841.05
ILC76.5968.7340.86
Probe-K76.7268.8640.97
PNS Head79.7069.6241.61
PNS vs. Best Head Baseline+2.98+0.54+0.56

The results support our hypothesis. At the original layer level, PNS achieves accuracy comparable to other selection methods, suggesting that the coarse selection space limits the differences between scoring criteria.

At the finer head level, PNS achieves the highest average accuracy on all three datasets, outperforming the strongest competing methods by 2.98, 0.54, and 0.56 percentage points on PACS, Office-Home, and DomainNet, respectively. Compared with the Probe baseline, PNS improves accuracy by 2.98, 0.76, and 0.64 percentage points, respectively.

Head-level PNS also achieves higher average accuracy than layer-level PNS on all three datasets. These results show that the joint necessity-and-sufficiency criterion better distinguishes informative representation segments at finer granularity and provide further evidence of the independent contribution of PNS scoring.

Comparison with Previously Reported Linear Probe Results. The reviewer cites linear-probe accuracies reported in PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization (2026): 84.2% on Office-Home and 53.5% on DomainNet. Adapting to Distribution Shift by Visual Domain Prompt Generation (ICLR 2024) reports 55.9% on DomainNet.

Linear-probe results may differ under different experimental configurations. We examined the public repositories of both studies, but could not find the specific probe implementations or complete training configurations corresponding to these reported results. We therefore do not know their exact probe settings.

Under our own experimental configuration with a frozen DINOv2 ViT-B/14 backbone, PNS Layer achieves 81.41% on Office-Home and 53.11% on DomainNet. We also ran the Probe baseline locally under our experimental configuration, with the results reported in Q2. At the head level, PNS outperforms Probe by 2.98, 0.76, and 0.64 percentage points on PACS, Office-Home, and DomainNet, respectively.

AuthorsOpenReview ↗

W1. Computational Cost and Wall-Clock Runtime.

We measured four stages of the proposed method: frozen feature extraction, scoring-classifier fitting, PNS scoring, and final classifier training. We conducted the timing experiments on PACS and Office-Home with three pretrained backbones, for both layer- and head-level selection. All times are reported in seconds. The pipeline timings are aggregated over four leave-one-domain-out splits and averaged across three runs, while feature extraction is a one-time cost. The Total column records separately measured end-to-end wall-clock time; the other columns report the four main stages.

DatasetBackboneLevelExtractionFittingPNSFinal HeadTotal
PACSSup. ViTLayer10.6057.381.727.7279.89
PACSSup. ViTHead12.9225.817.906.5455.59
PACSDINOv2Layer14.4023.881.686.6948.49
PACSDINOv2Head17.1015.378.186.6649.39
PACSMAELayer11.4989.091.716.66110.86
PACSMAEHead14.9637.337.896.7969.44
Office-HomeSup. ViTLayer19.61126.734.0710.13163.87
Office-HomeSup. ViTHead21.7037.1728.6410.14101.13
Office-HomeDINOv2Layer25.87107.284.049.68149.71
Office-HomeDINOv2Head28.5524.1428.8910.1495.33
Office-HomeMAELayer22.33159.514.0711.65201.82
Office-HomeMAEHead25.6854.0928.4810.39122.28

Layer-level PNS scoring takes approximately 1.7 seconds on PACS and 4.1 seconds on Office-Home. With the candidate space expanded from 12 layers to 144 attention heads, scoring takes approximately 8 and 29 seconds, respectively.

PNS scoring is performed only during offline representation selection. Once the selected representations and final classifier are obtained, no additional PNS scoring is needed at inference time. The detailed stage-wise timing results and measurement protocol have been added to the Appendix.

Reviewer_1RcnOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a causal-driven feature evaluation framework for multi-source domain generalization. The method constructs a representation-level structural causal model and evaluates the necessity and sufficiency of different feature segments through complementary interventions. Based on the resulting causal efficacy scores, high-scoring Transformer layers are selected to construct a representation subspace for classifier retraining. Experiments on PACS, Office-Home, and DomainNet with three pretrained backbones demonstrate improvements in cross-domain classification accuracy.

Critical strengths
  1. The paper addresses an important problem in domain generalization by investigating whether representations that are effective in source domains can maintain their discriminative ability under distribution shifts. The proposed necessity-and-sufficiency-based evaluation provides an interesting perspective on representation selection.

  2. The experimental results are promising across three benchmark datasets and three pretrained backbones. The proposed method achieves the best performance in eight out of nine dataset–backbone settings, demonstrating the potential effectiveness of the proposed feature selection framework.

Critical weaknesses

1. Insufficient comparisons with competitive baselines.  Although several domain generalization methods are included, the comparisons lack strong baselines specifically designed for pretrained feature selection. Moreover, optimization-based methods such as GGA and ARITH may be disadvantaged by the frozen-backbone setting. In Table 4, Random-K is relatively weak, while Source-best selects only one layer rather than the same number of layers used by the proposed method. Stronger Top-K feature selection baselines under matched settings are needed. 

2. Inconsistent and marginal improvements in several settings.  Although the method achieves a substantial improvement on DINOv2 with Office-Home (+5.71%), its advantages are considerably smaller in other settings. For example, on MAE with PACS, it outperforms Source-best layer by only 0.25 percentage points, while on supervised ViT with DomainNet, the improvement over the strongest DG baseline is only 0.08 points. Furthermore, the method underperforms GGA on MAE with DomainNet. These results raise concerns about the consistency and robustness of the reported improvements. 

3. Insufficient justification for the causal formulation.  The proposed SCM treats features from different Transformer layers as separate causal nodes without directed dependencies. However, successive Transformer layers are computationally dependent. Furthermore, the proposed interventions manipulate representation segments rather than underlying semantic factors. The authors should clarify whether the resulting scores genuinely measure causal effectiveness or simply reflect conventional feature importance. 

4. Limited theoretical support for domain generalization.  The theoretical analysis characterizes representation necessity and sufficiency under specific assumptions, but does not directly establish that higher-scoring features generalize better to unseen domains. Although the appendix verifies the required anchor conditions empirically, the connection between causal efficacy and cross-domain transferability remains insufficiently demonstrated. 

5. Insufficient validation of intervention reliability.  The proposed interventions combine feature representations from different samples, potentially introducing semantic inconsistencies and unrealistic feature combinations. The appendix reports that the off-support rate increases from 5.11% to 13.86% after random reference replacement. However, the influence of such distribution shifts on feature rankings and final classification performance remains unclear. Comparisons with class-consistent or nearest-neighbor reference strategies would be valuable. 

6. Incomplete ablation and sensitivity analysis.  The current ablations mainly compare the proposed approach with Random-K and Source-best layer selection, without sufficiently isolating the contribution of the PNS-based scoring mechanism. In addition, the sensitivity to the number of selected layers K and the reference sampling strategy is not adequately investigated. More comprehensive ablations and per-domain results are needed to establish the robustness of the proposed framework.

Questions for authors

Please see the Weaknesses section above.

10 comments
AuthorsOpenReview ↗

W1. Competitive Pretrained Feature-Selection Baselines.

We thank the reviewer for the detailed comments. We added layer- and head-level feature-selection baselines, finer-grained representation-selection experiments, an OOD intervention-effect stability theorem, and sensitivity analyses of reference sampling and selection hyperparameters. We address each point below.

Following the reviewer's suggestion, we first added ALF, Head2Toe, ILC, and Probe-KK. The following results are average target-domain accuracies (%) across three pretrained backbones.

MethodPACSOffice-HomeDomainNet
Layer-level Selection
ALF76.4168.7240.28
Head2Toe76.3669.0840.63
ILC75.6068.9140.62
Probe-KK75.7768.8840.63
PNS Layer76.4169.0840.63

Across all three datasets, PNS matches the best layer-level average accuracy. The results of different selection methods are close. We hypothesized that the candidate space of only 12 complete Transformer layers is too coarse: different scoring criteria can select the same or similar high-performing layers, limiting differences between the selected subsets.

To test this hypothesis, we refined the selection granularity from 12 complete layers to 144 individual attention heads and repeated the comparisons in the expanded candidate space.

MethodPACSOffice-HomeDomainNet
Head-level Selection
ALF75.9367.0339.97
Head2Toe76.5269.0841.05
ILC76.5968.7340.86
Probe-K76.7268.8640.97
PNS Head79.7069.6241.61
PNS vs. Best Head Baseline+2.98+0.54+0.56

The finer-grained results support our hypothesis. PNS achieves the highest average head-level accuracy on all three datasets, exceeding the strongest competing head-level method by 2.98, 0.54, and 0.56 percentage points on PACS, Office-Home, and DomainNet, respectively. Against Probe, the gains are 2.98, 0.76, and 0.64 points. These experiments demonstrate the benefit of the joint necessity-and-sufficiency scoring criterion for pretrained representation selection at finer granularity.

AuthorsOpenReview ↗

W2. Consistency and Robustness of Improvements.

Detailed per-target results, including the selected KK values and exact layer/head indices, are provided in Appendix Tables 7–10.

We agree with the reviewer that some improvements in the original layer-level experiments were marginal, particularly on PACS with MAE and DomainNet with Supervised ViT, and that our method underperformed GGA on DomainNet with MAE.

To investigate whether these limitations could be alleviated, we extended our method from 12 complete Transformer layers to 144 individual attention heads and repeated the experiments across all three datasets and pretrained backbones.

The new head-level results directly address the three settings highlighted by the reviewer:

  • PACS + MAE: PNS Head achieves 67.01%, exceeding Source-best Layer (63.38%) by 3.63 percentage points, compared with the original 0.25-point advantage.
  • DomainNet + Supervised ViT: PNS Head achieves 37.84%, exceeding the strongest DG baseline (37.61%) by 0.23 points, compared with the original 0.08-point advantage.
  • DomainNet + MAE: PNS Head achieves 33.33%, exceeding GGA (31.67%) by 1.66 points, whereas the original layer-level method underperformed GGA.

More broadly, head-level PNS improves upon layer-level PNS in eight of the nine dataset–backbone combinations. The average accuracies across three backbones increase from 76.41% to 79.70% on PACS, from 69.08% to 69.62% on Office-Home, and from 40.63% to 41.61% on DomainNet.

We also compared head-level PNS against ALF, Head2Toe, ILC, and Probe under matched experimental settings. PNS achieves the highest average accuracy on all three datasets, outperforming the strongest competing head-level selection method by 2.98, 0.54, and 0.56 percentage points, respectively.

These results indicate that finer-grained representation selection improves the consistency of our method across datasets and backbones, including settings where the original improvements were marginal or negative. While some individual margins remain modest, the expanded experiments provide stronger evidence for the effectiveness of PNS-based selection.

AuthorsOpenReview ↗

W3. Representation-Level SCM and Transformer Dependencies.

Our SCM models the classification process from frozen representations to predictions, rather than the internal forward computation of the pretrained Transformer.

Let EE be the frozen encoder. For an input image XX, its extracted representation is

Z=E(X)=(R1,…,RL).Z=E(X)=(R_1,\ldots,R_L).

The computational dependencies between successive Transformer layers are contained in the encoder EE. Once feature extraction is complete, each representation segment has a fixed observed value and is passed to the downstream classifier.

We model this classification stage as

Rk=fk(U),k=1,…,L,R_k=f_k(U),\qquad k=1,\ldots,L,

Y^=g(R1,…,RL),\widehat{Y}=g(R_1,\ldots,R_L),

where UU denotes shared upstream factors generating the natural representations and gg is the fixed scoring classifier.

The initial values of different representation nodes may be statistically dependent because they originate from the same Transformer forward pass. At the downstream classification stage, however, these nodes no longer execute inter-layer Transformer computations; they jointly serve as inputs to gg.

Our interventions operate directly at this classifier-input level. For example, we perform

do⁡(Rk=R~k)\operatorname{do}(R_k=\widetilde{R}_k)

while keeping the other representation segments and gg unchanged. The intervention neither reruns the Transformer nor recomputes subsequent-layer features. The representation-level SCM therefore matches the intervention actually performed and characterizes the causal effect of an input representation segment on the downstream classification decision.

We evaluate this effect in complementary directions: the performance decrease when segment kk is removed, and the performance gain when only segment kk is retained. These correspond to necessity and sufficiency and are combined for representation ranking. Unlike Probe, which ranks segments by predictive performance, PNS evaluates their effects under explicit interventions. The additional Probe comparisons in W1 demonstrate the benefit of this scoring mechanism.

Reviewer_zcPrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work evaluates the causal effectiveness of frozen layer-wise representations for cross-domain image classification. The authors perform complementary feature interventions on representation segments and derive a continuous bidirectional PNS metric to quantify each segment’s causal predictive power under distribution shift. High-scoring segments are chosen to build a subspace for training the final classifier. Experiments on PACS, Office-Home and DomainNet with DINOv2, MAE and supervised ViT backbones show consistent OOD accuracy improvements and notable gains across multiple settings.

Critical strengths
  1. This work nicely frames layer and segment selection as a causal evaluation problem by adopting a representation-level SCM paired with explicit complementary feature-space interventions for treatment and control.
  2. It further makes a valuable extension of binary PNS to bounded continuous outcomes using the threshold-integral and positive-part expectation identity, deriving tractable lower and upper characterizations equipped with interpretable anchor orderings.
Critical weaknesses
  1. The SCM builds upon an abstraction assuming representation nodes are manipulable and causally independent given a fixed encoder. Feature-space interventions may fail to reflect true generative do-operations.
  2. The anchor orderings used to justify the bounds are based on Bayes-optimal monotonicity, but evaluation uses a fixed linear classifier. While empirically validated in the paper, this assumption may not hold universally over diverse architectures and training regimes.
  3. All baselines train merely a linear head on frozen features. Since IRM/DRO are developed for end-to-end optimization, they may be disadvantaged in this setting. Despite the controlled setup, the fairness of this comparison is open to discussion.
  4. Beyond predictive accuracy, it would be valuable to see evaluations of calibration and corruption robustness (e.g., ImageNet-C benchmarks). This could help verify whether selecting high-PNS segments also brings improvements in uncertainty estimation and robustness.
Questions for authors

Please see the weakness part.

4 comments
AuthorsOpenReview ↗

W1. Representation-Level SCM and Generative Interventions.

We thank the reviewer for the comments on causal modeling, theoretical assumptions, experimental comparisons, and prediction reliability. We clarify the representation-level causal model and its intervention mechanism under a fixed classifier, add competitive baselines specifically designed for pretrained representation selection, and provide new evaluations of target-domain calibration and corruption robustness. We address each concern below.

Our structural causal model concerns the downstream classification process from frozen representations to predictions, rather than the internal computation of the pretrained Transformer or the image-generation mechanism.

Let EE denote the frozen encoder. For an input image XX, the complete representation is

Z=E(X)=(R1,…,RL).Z=E(X)=(R_1,\ldots,R_L).

The computational dependencies between Transformer layers are already incorporated into the encoder EE. Once feature extraction is complete, each representation segment has an assigned value and serves as an input to the downstream classifier.

We describe this classification process as

Rk=fk(U),k=1,…,L,R_k=f_k(U),\qquad k=1,\ldots,L,

Y^=g(R1,…,RL),\widehat{Y}=g(R_1,\ldots,R_L),

where UU represents the shared upstream factors generating the natural representations and gg is the fixed scoring classifier.

The initial values of different representation nodes may be statistically correlated, since they are extracted by the same Transformer. In the downstream classification process modeled here, however, the nodes no longer perform inter-layer computations; instead, their extracted values jointly determine the prediction.

We therefore intervene directly at the classifier input, for example by applying

do(Rk=R~k),do(R_k=\widetilde{R}_k),

while leaving the other representation segments and the scoring classifier gg unchanged.

This formulation matches our implementation: we first extract the complete natural representation, then construct an intervened representation by retaining or replacing the specified segment, and finally measure the change in predictions using the same fixed scoring classifier.

Thus, the causal effect studied in our paper has a precise operational meaning: the effect of a representation segment on the downstream classification decision. Complementary retention and removal interventions evaluate its necessity and sufficiency, which are then used for representation selection.

AuthorsOpenReview ↗

W2. Bayes-Optimal Anchor Conditions and the Fixed Classifier.

Bayes-optimal information monotonicity provides the theoretical motivation for the anchor orderings, but we do not assume that these orderings automatically hold for an arbitrary fixed scoring classifier.

Instead, we explicitly check the two draw-wise conditions in Eq. (17) under the actual scoring classifier and intervention procedure:

\qquad Y_{x_1}^{(r)} \leq Y_{\mathrm{full}}.$$ As reported in Appendix C, we conducted 54,000 draw-level domain-accuracy evaluations across our experimental settings. Both conditions were satisfied in 100% of the evaluated cases. We therefore apply the anchor-dependent theoretical bound to these empirically verified settings, rather than assuming its validity for arbitrary classifiers or distributions. The anchor conditions are required to establish the upper bound constructed from Eq. (19), whereas the lower bound in Eq. (20) does not depend on these conditions. The resulting two-sided bound is given in Eq. (21). This approach connects the Bayes-optimal motivation to our actual fixed-classifier implementation through direct empirical verification of the required conditions.
AuthorsOpenReview ↗

W3. Fairness of Comparisons Under Frozen Backbones.

To address the reviewer's concern that optimization-based methods such as IRM and GroupDRO may be disadvantaged by frozen-backbone evaluation, we added competitive baselines specifically targeting pretrained representation selection. These include ALF, Head2Toe, ILC, and Probe-K.

We first evaluated these methods in the original space of 12 complete Transformer layers. The table reports average target-domain accuracy (%) over three pretrained backbones. We also include the corresponding results for the refined attention-head selection space.

MethodPACSOffice-HomeDomainNet
Layer-level Selection
ALF76.4168.7240.28
Head2Toe76.3669.0840.63
ILC75.6068.9140.62
Probe-K75.7768.8840.63
PNS Layer76.4169.0840.63
Head-level Selection
ALF75.9367.0339.97
Head2Toe76.5269.0841.05
ILC76.5968.7340.86
Probe-K76.7268.8640.97
PNS Head79.7069.6241.61

At the original layer level, PNS achieves the best or tied-best average accuracy on PACS, Office-Home, and DomainNet. However, the results of different methods are relatively close at this granularity.

We therefore hypothesized that selecting among only 12 complete Transformer layers provides a coarse candidate space, in which different methods often identify the same or similar high-performing layers. This can limit the observable difference between scoring criteria.

To test this hypothesis, we expanded the candidate representation space to 144 attention heads and repeated the comparisons.

The new results support our hypothesis. At the head level, PNS achieves cross-backbone average accuracies of 79.70%, 69.62%, and 41.61% on PACS, Office-Home, and DomainNet, respectively, exceeding ALF, Head2Toe, ILC, and Probe-K on all three datasets.

Compared with the strongest competing head-level method, the gains are 2.98, 0.54, and 0.56 percentage points, respectively. Relative to Probe-K, the improvements are 2.98, 0.76, and 0.64 percentage points. Moreover, head-level PNS improves over the original layer-level PNS in eight of the nine dataset–backbone combinations.

These additional experiments directly compare different scoring mechanisms under the same frozen-backbone setting, representation-selection granularity, and selection budget. They demonstrate the contribution of joint necessity-and-sufficiency scoring to pretrained representation selection and the OOD improvements obtained with finer-grained PNS selection.

AuthorsOpenReview ↗

W4. Prediction Calibration and Corruption Robustness.

Following the reviewer's suggestion, we further evaluated whether selecting high-PNS representation segments improves prediction reliability, using target-domain calibration and common image-corruption benchmarks.

  1. Target-Domain Calibration.

On the unseen target domains of PACS, Office-Home, and DomainNet, we measured classification accuracy, expected calibration error (ECE), and negative log-likelihood (NLL). ECE uses 15 equal-width confidence bins, without temperature scaling. The table reports averages across the three pretrained backbones.

DatasetMethodAcc ↑ECE ↓NLL ↓
PACSPNS Layer76.4113.840.941
ERM Layer73.0219.131.420
PNS Head79.7010.000.731
ERM Head75.3615.711.193
Office-HomePNS Layer69.0815.091.678
ERM Layer61.5928.934.079
PNS Head69.6213.301.513
ERM Head62.2726.873.477
DomainNetPNS Layer40.6321.294.046
ERM Layer37.0638.736.630
PNS Head41.6120.373.882
ERM Head37.6040.136.971

Across all nine dataset–backbone combinations, PNS achieves higher accuracy and lower ECE and NLL than the corresponding ERM baseline at both layer and head levels. The calibration improvements are particularly evident on Office-Home and DomainNet. For example, head-level ECE decreases from 26.87% to 13.30% on Office-Home and from 40.13% to 20.37% on DomainNet.

NLL also improves consistently, indicating that PNS representation selection improves not only target-domain accuracy but also the agreement between predicted probabilities and ground-truth labels.

  1. Image-Corruption Robustness.

We further evaluated classification and calibration under common image corruptions on CIFAR-10-C and ImageNet-100-C, using the DINOv2 backbone.

The experiments use three random seeds.

We report clean accuracy, clean ECE, mean corruption accuracy (mCA), accuracy drop, corruption ECE, and corruption NLL.

DatasetMethodClean AccClean ECEmCA ↑Drop ↓Corrupt ECE ↓Corrupt NLL ↓
CIFAR-10-CPNS Layer (K=3K=3)98.560.9987.9010.668.180.643
ERM Layer98.211.5785.2412.9713.172.212
PNS Head (K=48K=48)98.590.9088.1910.417.960.618
ERM Head98.391.1086.9011.489.090.749
ImageNet-100-CPNS Layer (K=2K=2)94.184.0982.6511.5312.101.259
ERM Layer93.186.3474.8318.3523.217.634
PNS Head (K=24K=24)94.074.1482.5011.5711.861.195
ERM Head92.886.5573.5719.3124.297.358

On both datasets and at both selection granularities, PNS achieves higher mean corruption accuracy, smaller accuracy drops, and lower corruption ECE and NLL.

On CIFAR-10-C, PNS Layer and PNS Head reach mCA values of 87.90% and 88.19%, exceeding the corresponding ERM results of 85.24% and 86.90%.

On ImageNet-100-C, PNS Layer attains an mCA of 82.65%, 7.82 percentage points above ERM Layer, while PNS Head attains 82.50%, 8.93 points above ERM Head. Meanwhile, head-level corruption ECE decreases from 24.29% to 11.86%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.