acceptodds
Withdrawn from ICLR 2027

Lossy Representation Compression for Out-of-Distribution Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Out-of-distribution (OOD) generalization aims to maintain predictive performance under distribution shifts. Despite extensive research, invariant learning methods often fail to deliver reliable improvements across diverse settings. A key limitation is that prediction invariance across training environments encourages the learning of causal features but does not remove non-causal features from learned representations. We first show that causal and non-causal features can occupy different directions in the representation space, such that encoding both produces more complex representations than encoding causal features alone. Based on this observation, we introduce Lossy Causal Feature Learning (LoCaL), a simple and effective framework that regularizes learned representations using the well-established lossy coding rate from information theory. Geometrically, the lossy coding rate measures the volume occupied by a representation at a given precision. Constraining this quantity encourages a lower-dimensional, more compact representation, thereby suppressing non-causal features. Extensive experiments across diverse datasets demonstrate that LoCaL outperforms a broad range of strong baselines, with most improvements being statistically significant.

Primary area:
learning theory
Keywords:
Out-of-distribution generalization, invariant learning, causal representation learning, representation complexity, lossy coding rate

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_MYoaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that invariant learning methods only enforce prediction invariance but cannot remove non-causal features from representations, since causal and non-causal features occupy different directions and encoding both inflates representation complexity. The authors introduce a regularization method based on the lossy coding rate from information theory, yielding the LoCaL framework that compresses representation volume to suppress non-causal content. Across four OOD benchmarks, LoCaL improves results when paired with IRMv1 or REx, and remains effective under over-parameterization and color perturbations.

Critical strengths
  1. Reframing OOD as a compression problem. Most prior work focuses solely on invariance. This paper convincingly argues that invariance alone cannot eliminate non-causal features, and treating this as a representation complexity issue is a valuable conceptual shift.
  2. Practical, plug-and-play design. The lossy coding rate regularizer is theoretically sound and requires no architectural changes. Its consistent performance gains when attached to baselines like IRMv1 demonstrate clear practical utility.
  3. Robustness to over-parameterization. LoCaL stays stable as model capacity grows, unlike standard invariant learning methods that degrade in over-parameterized regimes.
Critical weaknesses
  1. Information-theoretic compression has already been explored in IB-IRM. The paper doesn't clearly explain how the lossy coding rate objective fundamentally differs from these prior approaches, making the contribution feel more like a reparameterization than a new direction.
  2. The reported improvement on CIFAR-MNIST is statistically non-significant (p=0.0778) despite being labeled as such. Furthermore, the absence of key recent baselines like Fishr and MixStyle makes it hard to gauge LoCaL's actual competitiveness.
  3. Proposition 3.1 only shows that variance is a first-order approximation of the coding-rate penalty; it does not establish that minimizing this penalty preferentially compresses non-causal directions without harming causal features in nonlinear representations. A large λ\lambda may also reduce the effective rank of causal features, yet no condition or failure analysis is provided for when this over-compression occurs.
  4. The empirical evidence is still concentrated on synthetic or semi-synthetic benchmarks, and some main results are incompletely reported. The paper does not evaluate LoCaL on more realistic domain-generalization benchmarks such as WILDS or DomainBed, limiting the external validity of its OOD claims. Moreover, Table 4 in the main text reports only the ERM row, so the full LoCaL and baseline results on CIFAR-MNIST, Colored Object, and CS-MNIST are not sufficiently visible for verification.
  5. Although the coding rate is not scale-invariant, all experiments fix ϵ2=10\epsilon^2=10, and λ\lambda is selected from a limited grid, without systematically disentangling structural compression from a generic scale penalty. The paper also does not compare the class-wise penalty with the pooled upper bound in Equation (6), nor report the training time and memory overhead of the per-batch log-determinant/Cholesky computation.
Questions for authors
  1. Given the p-value of 0.0778 on CIFAR-MNIST, can you justify claiming a significant improvement?
  2. Why are recent methods like Fishr and MixStyle excluded from the comparisons?
  3. Can you provide a formal condition under which minimizing the lossy coding rate provably suppresses non-causal directions without reducing the effective rank of causal features, especially for nonlinear encoders? Proposition 3.1 only gives a first-order variance approximation, so the causal-selectivity claim currently lacks a theoretical guarantee.
  4. Since all experiments fix ϵ2=10\epsilon^2=10 and tune λ\lambda only on a limited grid, can you report broader ablations on ϵ2\epsilon^2, λ\lambda, and feature scale to show the gains are not mainly a generic scale penalty? It would also help to compare the class-wise penalty with the pooled bound in Equation (6).
  5. Given the controlled benchmarks, can you provide results on more realistic domain-generalization benchmarks such as WILDS or DomainBed? Also, were multiple-comparison corrections applied to the reported pp-values, and were all baselines tuned with comparable hyperparameter search budgets?
Reviewer_LNAQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LoCaL, a method for promoting OOD generalization. The main idea is to combine an invariant learning objective with explicit control of representation complexity, implemented by minimizing a lossy coding rate.

Critical strengths
  • Minimizing coding rate in OOD generalization is a less explored idea.
Critical weaknesses
  • The narrative of "causal v.s. non-causal features" and "considering the structure of the learned representation" in OOD generalization has already received plenty of study in the past few years. Explicitly learning invariant representations (instead of invariant predictors such as in IRM and REx) has been well-explored, as opposed to the argument of the authors. The community has also gone further by exploring the richness and non-linear entanglement of features in explaining OOD failure, instead of solely using the lens of causality [1,2].
  • The proposed method is only evaluated on toy datasets, with limited performance gains compared to baselines.

[1] Rich feature construction for the optimization-generalization dilemma. ICML 2022.

[2] Feature contamination: Neural networks learn uncorrelated features and fail to generalize. ICML 2024.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.