acceptodds
Under review as a conference paper at ICLR 2027

Model-Agnostic Invariant Causal Prediction Testing in Learned Representations

Abstract

Identifying invariant causal features in learned representations by modern machine learning models is central to robust generalization across environments and to the development of trustworthy AI systems. Based on invariant prediction, we develop a model-agnostic test of whether the conditional distribution of a response given a candidate feature subset remains invariant across environments. Under conditional invariance, the adjusted predictiveness scores coincide, even when the working prediction class is misspecified. We establish asymptotic normality of the component statistics and a chi-square null limit for their squared cyclic aggregate, providing asymptotic Type I error control. Power depends on whether the chosen working class and score distinguish changes in the conditional mechanism. Our approach can be readily integrated with pretrained foundation models. Simulation studies show that the proposed test reliably identifies causal features in complex settings and remains robust to severe model misspecification. Experiments using the Contrastive Language-Image Pretraining (CLIP) model further illustrate that our approach uncovers meaningful invariant features and improves generalization in downstream tasks.

Primary area:
causal reasoning
Keywords:
Causality, Invariance learning, Hypothesis testing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DacmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAICP, a model-agnostic framework for identifying invariant features across environments. It uses density adjustment and held-out prediction scores to reliably test conditional invariance across environments, even when the prediction model is misspecified. Experiments show that MAICP maintains reliable test calibration, identifies stable features in learned representations, and improves prediction accuracy under constructed distribution shifts.

Critical strengths
  1. The paper provides a density-adjusted invariance test with asymptotic Type I error control, even under model misspecification. It addresses a limitation of existing invariant prediction methods and strengthens the reliability of invariant feature selection.

  2. The paper demonstrates the practical applicability of MAICP to learned representations. Experiments with CLIP features show that MAICP can identify invariant semantic features and improve prediction accuracy under distribution shifts.

Critical weaknesses
  1. The paper mainly evaluates linear and logistic regression models under relatively simple settings. Experiments with more complex prediction models and diverse data-generating mechanisms would provide stronger evidence for the general applicability of MAICP.

  2. The paper does not sufficiently demonstrate the advantages of MAICP over existing methods. The baseline comparisons are limited and mainly presented in the appendix. Although the authors identify model misspecification as a key challenge, the experiments primarily compare MAICP with linear ICP and IRM. More direct comparisons with related methods, such as nonlinear ICP and conditional distribution testing methods, would help establish whether MAICP offers meaningful improvements over existing approaches.

Questions for authors
  1. How does MAICP compare with existing nonlinear invariant prediction methods?

  2. How well does MAICP perform with more complex models, such as DNN and transformers?

Reviewer_dHeXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MAICP, a model-agnostic statistical test for identifying feature subsets whose conditional relationship with a response remains invariant across environments. The method uses density-ratio weighting, working-class predictors, and held-out predictiveness contrasts to test conditional invariance, allowing the working prediction model to be misspecified. The authors establish asymptotic Type I error control and extend the test to multiple environments using squared cyclic aggregation, with power and exact recovery depending on additional separation conditions. Experiments on synthetic data and CLIP-based semantic features from CelebA demonstrate promising results under constructed distribution shifts.

Critical strengths
  • MAICP extends invariant prediction testing to flexible, potentially misspecified working models. Density adjustment addresses changes in covariate distributions that could otherwise invalidate comparisons of predictive performance across environments.
  • The paper establishes asymptotic calibration for its test statistics and carefully distinguishes false-inclusion control from exact recovery. The multi-environment extension and the discussion of the conditions needed for power provide a useful foundation for further research.
Critical weaknesses
  • The CLIP/CelebA experiments rely on constructed environments and a small candidate feature dictionary. For example, the downstream prediction experiment uses 180 training images from one environment and 20 from the other. Evaluation on held-out samples from these constructed environments does not establish improved generalization to independently collected, genuinely unseen environments. More realistic distribution-shift benchmarks and stronger comparisons with existing invariant prediction and domain generalization methods would strengthen the claims.
  • Although MAICP is model-agnostic in the sense of not requiring a correctly specified conditional response model, its power depends on whether the selected score and working class can detect changes in the conditional mechanism. The method also requires prespecified candidate features and potentially enumerates exponentially many subsets. The practical consequences of these restrictions, especially for high-dimensional foundation-model representations, are not sufficiently demonstrated.
Questions for authors
  • Can the authors evaluate MAICP on independently collected or standard out-of-distribution benchmarks, comparing it with relevant ICP, IRM, and domain generalization baselines under comparable settings?
  • How sensitive are the test's Type I error and statistical power to imperfect density-ratio estimation, limited overlap, and the choice of working class and predictiveness score? Additional systematic sensitivity analyses would help establish the practical reliability of the method.
  • What are the computational cost and feature-selection performance as the candidate dictionary grows? Can screening or alternative search strategies substantially reduce the subset-enumeration burden while preserving the stated statistical guarantees?
Reviewer_6CAUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper attempts to find invariance causal predictors for a label Y given covariates X without making structural assumptions about the true causal function Y=f(XS⋆)+EY = f(X_{S^\star}) + E, where S⋆S^\star indicates the true parent set, and EE is independent exogenous noise. The idea is to re-weight the samples in one environment to match the distribution of covariates in another reference environment, and then test whether the regression function for the re-weighted data matches with the regression function of the reference environment. This allows the authors to claim sufficiency of the test irrespective of true function class.

Critical strengths
  1. The main idea about adjusting the distribution of the covariates to mitigate realizability is interesting.

  2. The theoretical claims are clearly stated and I am approve of the soundness.

  3. The limitations are stated honestly, e.g., that the test suggests invariance but is not a proof of it.

Critical weaknesses
  1. The central technique used in the paper is the re-weighting of samples to match the distribution of covariates with that of a reference environment, is known as importance-weighted ERM for covariate shift (Sugiyama et al. 2007). The paper does not acknowledge the substantial existing work.

  2. Given the theoretical results around the covariate shift and unsupervised domain adaptation, I am not convinced that the paper introduces enough novelty in the theoretical aspect to merit as a mainly theoretical contribution. I'll digress on specific references, since the whole covariate shift literature is missing from the discussion despite its close relevance.

  3. There's work on non-linear ICP referenced in the paper but is missing from the experiments.

  4. The paper skips motivating the results altogether. The introduction motivates the broad topic, but never discusses why model-agnostic-ness is what the linear/non-linear ICP misses. Moreover, the objective of ICP is causal discovery, which is completely digressed in the current narrative. Additional discussion and motivation would be useful for the readers.

Questions for authors
  1. I understand the main message of the paper being an extension of ICP to relax the realizability assumption. Have you identified a failure case about that with ICP? The paper does not provide proper context for the reader about this particular limitation of ICP.

  2. What is the purpose of presenting the results around CLIP representation? Isn't the purpose to improve upon ICP, as a causal discovery method? If so, why did you choose CLIP representation instead of different examples of heterogenous data used to motivate ICP?

  3. As I understand, the features are hand-picked in the problem formulation, and then causal discovery happens on top of the features, independent of the CLIP embedding. Is this a fair reading of the results? Notice that the causal structure is often considered over the true causal variables, but the features scored by CLIP seem to be secondary representations/coarsenings, and causal discovery over them for a true label seems an odd objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.