- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that invariant learning methods only enforce prediction invariance but cannot remove non-causal features from representations, since causal and non-causal features occupy different directions and encoding both inflates representation complexity. The authors introduce a regularization method based on the lossy coding rate from information theory, yielding the LoCaL framework that compresses representation volume to suppress non-causal content. Across four OOD benchmarks, LoCaL improves results when paired with IRMv1 or REx, and remains effective under over-parameterization and color perturbations.
Critical strengths
- Reframing OOD as a compression problem. Most prior work focuses solely on invariance. This paper convincingly argues that invariance alone cannot eliminate non-causal features, and treating this as a representation complexity issue is a valuable conceptual shift.
- Practical, plug-and-play design. The lossy coding rate regularizer is theoretically sound and requires no architectural changes. Its consistent performance gains when attached to baselines like IRMv1 demonstrate clear practical utility.
- Robustness to over-parameterization. LoCaL stays stable as model capacity grows, unlike standard invariant learning methods that degrade in over-parameterized regimes.
Critical weaknesses
- Information-theoretic compression has already been explored in IB-IRM. The paper doesn't clearly explain how the lossy coding rate objective fundamentally differs from these prior approaches, making the contribution feel more like a reparameterization than a new direction.
- The reported improvement on CIFAR-MNIST is statistically non-significant (p=0.0778) despite being labeled as such. Furthermore, the absence of key recent baselines like Fishr and MixStyle makes it hard to gauge LoCaL's actual competitiveness.
- Proposition 3.1 only shows that variance is a first-order approximation of the coding-rate penalty; it does not establish that minimizing this penalty preferentially compresses non-causal directions without harming causal features in nonlinear representations. A large may also reduce the effective rank of causal features, yet no condition or failure analysis is provided for when this over-compression occurs.
- The empirical evidence is still concentrated on synthetic or semi-synthetic benchmarks, and some main results are incompletely reported. The paper does not evaluate LoCaL on more realistic domain-generalization benchmarks such as WILDS or DomainBed, limiting the external validity of its OOD claims. Moreover, Table 4 in the main text reports only the ERM row, so the full LoCaL and baseline results on CIFAR-MNIST, Colored Object, and CS-MNIST are not sufficiently visible for verification.
- Although the coding rate is not scale-invariant, all experiments fix , and is selected from a limited grid, without systematically disentangling structural compression from a generic scale penalty. The paper also does not compare the class-wise penalty with the pooled upper bound in Equation (6), nor report the training time and memory overhead of the per-batch log-determinant/Cholesky computation.
Questions for authors
- Given the p-value of 0.0778 on CIFAR-MNIST, can you justify claiming a significant improvement?
- Why are recent methods like Fishr and MixStyle excluded from the comparisons?
- Can you provide a formal condition under which minimizing the lossy coding rate provably suppresses non-causal directions without reducing the effective rank of causal features, especially for nonlinear encoders? Proposition 3.1 only gives a first-order variance approximation, so the causal-selectivity claim currently lacks a theoretical guarantee.
- Since all experiments fix and tune only on a limited grid, can you report broader ablations on , , and feature scale to show the gains are not mainly a generic scale penalty? It would also help to compare the class-wise penalty with the pooled bound in Equation (6).
- Given the controlled benchmarks, can you provide results on more realistic domain-generalization benchmarks such as WILDS or DomainBed? Also, were multiple-comparison corrections applied to the reported -values, and were all baselines tuned with comparable hyperparameter search budgets?