acceptodds
Under review as a conference paper at ICLR 2027

On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation

Abstract

Substantial efforts have been devoted to making deep learning objectives, representations, and architectures interpretable, with the goal of improving the safety, robustness, and generalisation of learning systems in diverse real-world applications. The recently proposed maximal coding rate reduction () offers a promising information-theoretic framework for learning structured, discriminative representations of class-wise submanifolds and has inspired interpretable white-box architectures. However, we observe that can completely fail under distribution shift, motivating our study of its out-of-distribution (OOD) generalisation limits. We establish two limitations of for OOD generalisation. First, the objective alone can admit complete prediction failure: a representation based entirely on unstable environmental features can achieve the global coding optimum yet fail completely after correlation reversal, despite an available perfectly stable feature. This exact-optimum example includes test inputs that cannot occur during training. Even when every possible test input can also occur during training, coding quality can be arbitrarily close to optimal while prediction error is arbitrarily close to 100%. Second, directly incorporating the invariance principle underlying widely successful invariant risk minimisation (IRM) and risk extrapolation (REx) does not eliminate this failure. The failing representation admits the same optimal coding operator across training environments, showing that shared coding optimality does not ensure stable prediction. Reliable OOD guarantees for therefore require additional new assumptions or learning principles that establish stable predictive relationships across environments.

Primary area:
learning theory
Keywords:
maximal coding rate reduction, out-of-distribution generalisation, representation learning, invariant learning, spurious correlations, learning theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nqLVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the OOD limitations of MCR² in a binary model with stable and environmental features. It derives the global coding optimum and constructs near-optimal environmental representations with almost maximal target error under correlation reversal. These representations also satisfy a shared-optimal-coding constraint. Exact-optimum failure requires changing input support; with common support, the failure concerns near-optimal representations. Waterbirds motivates the analysis, and synthetic experiments validate the formulas.

Critical strengths
  1. The theoretical characterization is precise. The global coding optimum, the environmental encoder's objective gap, and its optimal coding operators are explicitly derived. Correlation reversal preserving the coding objective while reversing the appropriate label assignments is a useful observation.

  2. The limitations are transparently stated. The authors acknowledge the potentially large distribution shift, the success of exact optimizers in the full-support model, and the absence of an analysis of which representations training selects.

Critical weaknesses
  1. The motivation and incremental contribution are insufficiently established. The central near-optimality/failure phenomenon is not specific to MCR². In the same model, an environmental predictor has source classification error δ\delta and target error 1−δ1-\delta; its optimal source cross-entropy is Hb(δ)→0H_b(\delta)\to0, approaching the global infimum. Related work already studies spurious-feature reliance, including Sagawa et al. (2020) and Nagarajan et al. (2021). Although Section 5 and Appendix A.2 acknowledge this distinction, the substantive insight added by the MCR²-specific calculations remains unclear. A same-model comparison should separate generic shift-related difficulties from properties particular to the coding objective.

  2. The experiments do not sufficiently connect the counterexample to practical learning. Waterbirds lacks a matched cross-entropy baseline, all four group results, and variability across seeds. Thus, the coding objective's role cannot be separated from initialization, data imbalance, and training choices. Synthetic experiments use prescribed encoders, so they do not establish that optimization selects the failing representations—especially when the stable encoder has a strictly better objective. Controlled encoder-training experiments would strengthen this connection. Encoder-gradient saliency alone also does not establish background reliance by the classifier.

Questions for authors
  1. Contribution clarification: Which conclusions distinguish MCR² from ordinary supervised risks under the same data model and shift assumptions? How do they advance the existing understanding of spurious correlations?

  2. Matched empirical comparison: Can the authors compare MCR² with cross-entropy using the same ResNet-18, initialization, training budget, and model-selection protocol, reporting source performance and all four test groups across seeds?

  3. Evidence that could change my assessment: Can actual MCR² training produce near-optimal but target-failing representations in a controlled experiment? Alternatively, can the authors sharpen the theoretical insight by characterizing guarantees under a bounded shift?

6 comments
AuthorsOpenReview ↗

Response to Reviewer nqLV, Part 1

We sincerely thank you for your detailed evaluation and constructive feedback. We appreciate your recognition of the precision of our theoretical characterisation and our transparent discussion of the paper’s limitations. Below, we address your concerns point by point. For clarity and ease of reference, we have slightly rephrased and reordered some comments while preserving their original meaning, so that our responses follow a logical flow.

Weakness 1: The motivation and incremental contribution are insufficiently established, as the failure mechanism also applies to cross-entropy.

We would like to clarify that our contribution is to characterise this limitation within the MCR² framework, rather than to identify a failure mechanism exclusive to MCR².

Regarding motivation, MCR² and related coding-based frameworks have attracted substantial interest as principles for learning structured representations and designing interpretable architectures. Their growing influence makes it important to understand both their capabilities and their fundamental limitations. Reliable generalisation across environments is crucial for practical machine learning applications, motivating us to examine whether the representation structure encouraged by MCR² is sufficient to ensure it.

Regarding novelty, since our work addresses OOD generalisation, it is natural to adopt a theoretical setting similar to those used in previous OOD studies, as it captures the practical challenge of predictive relationships changing across environments. While the theoretical setting is similar, the learning objective and the specific theoretical question differ from those in analyses of cross-entropy or IRM: we investigate whether the coding quality and representation structure encouraged by MCR² are sufficient to ensure reliable OOD prediction.

Specifically, analyses of cross-entropy examine predictive risk, while IRM examines whether the same classifier is optimal across environments. MCR² instead evaluates representations through coding-rate reduction. Our analysis therefore asks what constitutes globally optimal coding, how close an unstable representation can come to that optimum, and whether a shared optimal coding operator ensures predictive stability.

We conclude that our contribution is to establish the gap between these coding-based optimality conditions and OOD prediction. We derive the global MCR² optimum and the environmental representation’s exact objective gap, and show that correlation reversal can preserve both its coding value and its optimal coding operators while changing the label assignments required for accurate prediction. Thus, even exact agreement on an optimal coding operator across environments does not exclude a near-optimal representation with severe target prediction failure. These conclusions require analysing the MCR² objective and its optimisation structure; they cannot be obtained simply by transferring a failure result for cross-entropy or a shared-classifier condition from IRM.

Question 1: Which conclusions distinguish MCR² from ordinary supervised risks, and how do they advance the understanding of spurious correlations?

We would like to clarify the intended scope of our contribution: our aim is to establish a fundamental limitation of MCR² as an influential representation-learning framework, rather than to discover a new mechanism of spurious correlations. The fact that ordinary supervised losses can also fail does not establish whether the geometric structure and coding optimality encouraged by MCR² prevent such failure.

Our distinguishing conclusion is that correlation reversal can preserve both the MCR² coding value and its optimal coding operators, while changing the label assignments required for accurate prediction. Consequently, a representation can satisfy an exact shared-optimal-coding constraint across environments and remain arbitrarily close to the global coding optimum, yet incur target error approaching 100%. These coding-specific conclusions do not follow directly from the failure of cross-entropy.

The additional insight is that structured, discriminative representations and agreement on optimal coding geometry are not sufficient to rule out unstable predictive relationships. This identifies a limitation of what coding-based learning can guarantee and clarifies why predictive stability requires additional assumptions or learning principles.

AuthorsOpenReview ↗

Response to Reviewer nqLV, Part 2

Weakness 2 & Question 2: The experiments do not sufficiently connect the counterexample to practical learning. Can the authors compare MCR² with cross-entropy using the same ResNet-18, initialization, training budget, and model-selection protocol, reporting source performance and all four test groups across seeds?

Thank you for these suggestions. We have expanded the Waterbirds experiments to include a matched cross-entropy baseline, all four test groups, and five random seeds under both random and ImageNet-pretrained initialisation, giving 20 training runs in total.

To make the comparison controlled, both objectives use the same ResNet-18 encoder architecture, paired encoder initialisations, training data, image augmentations, optimiser settings, and training budget. Each run uses 50 epochs, AdamW with a constant learning rate of 3×10−43\times10^{-4}, and a batch size of 128. We report the final epoch for every run, without selecting checkpoints based on test performance.

The original Waterbirds training distribution is imbalanced, and we preserve exactly the same distribution for both objectives. The four training groups—landbird–land, landbird–water, waterbird–land, and waterbird–water—contain 3,498, 184, 56, and 1,057 images, respectively. Neither objective uses group reweighting or balanced sampling. This controls the data distribution in the comparison, while the two initialisation settings allow us to examine whether the observed behaviour persists with and without pretraining.

The table reports mean accuracy ± sample standard deviation (%) across five seeds (43–47). Training accuracy is measured on the complete training split using deterministic preprocessing. MCR² uses a source-trained linear probe; CE uses its jointly trained classification head.

InitialisationObjectiveTrainLandbird–LandLandbird–WaterWaterbird–LandWaterbird–Water
RandomMCR²92.74 ± 0.3997.74 ± 0.7054.20 ± 8.3312.87 ± 3.3570.59 ± 2.31
RandomCE95.14 ± 0.7896.17 ± 0.8451.62 ± 7.6117.01 ± 5.5977.91 ± 6.43
ImageNet-pretrainedMCR²97.84 ± 0.4199.26 ± 0.2081.75 ± 2.4239.94 ± 6.8087.35 ± 1.78
ImageNet-pretrainedCE99.12 ± 0.2798.35 ± 0.6973.26 ± 4.8255.05 ± 3.7393.43 ± 1.76

These experiments show that actual MCR² training can produce substantial performance disparities between background-matched and background-mismatched groups across multiple seeds and both initialisation settings. Pretraining improves performance but does not eliminate these disparities. CE also exhibits this difficulty, consistent with our position that the underlying failure mechanism is not exclusive to MCR².

To account for differences in classifier fitting, we additionally evaluate the CE representations using the same linear-probe procedure as MCR², with the results reported in Appendix F, Tables 3 and 4. The corresponding worst-group test accuracies are 17.10±3.98%17.10\pm3.98\% with random initialisation and 52.02±4.76%52.02\pm4.76\% with pretraining, showing that the group disparity also persists under this common evaluation protocol.

The complete settings, results, and analysis are included in Appendix F (pp. 35–39), with additions highlighted in light red. These experiments strengthen the connection to practical training; the precise near-optimality statements remain established by our theoretical analysis.

AuthorsOpenReview ↗

Response to Reviewer nqLV, Part 3

Question 3.1: Can actual MCR² training produce near-optimal but target-failing representations in a controlled experiment?

We have now conducted controlled experiments using actual MCR² training and evaluated target prediction performance. The results strongly support our theoretical analysis.

Specifically, we first clarify how MCR² optimality can be assessed in practice. We highlight that for a given dataset and encoder architecture, the global maximum of the MCR² objective is generally unknown. The objective depends on the sample size, representation dimension, and coding precision ϵ2\epsilon^2, while the achievable maximum also depends on the representations that the encoder can realise. So we cannot directly calculate the theoretical optimality gap.

However, previous MCR² theory establishes that optimal representations have mutually orthogonal class subspaces under its stated assumptions [1]. This provides a geometric property that can be examined empirically. Motivated by this result, we use training accuracy, the observed MCR² objective trajectory, and cross-class absolute cosine similarity as complementary indirect measures of training fit and representation geometry. Intuitively, when training accuracy approaches 100%, and representations from different classes become nearly orthogonal, these observations provide strong empirical evidence that MCR² training is approaching an optimal solution.

Based on this motivation, we conducted controlled experiments using actual MCR² training to examine whether well-fitted, approximately orthogonal class representations can nevertheless lead to target prediction failure. The experimental setting and results are reported in Appendix G (pp. 40–42).

We use a noise-free variant of Colored MNIST [2], a widely used benchmark for OOD generalisation. We group digits 0–4 into class 0 and digits 5–9 into class 1, without label flips. In the source environment, class 0 is always blue and class 1 is always red, while these colours are reversed in the target environment. Thus, both digit identity and colour perfectly predict the source labels, but only digit identity remains predictive under the distribution shift.

We train a single CNN using the actual MCR² objective (ϵ2=1\epsilon^2=1) for 500 full-batch updates across five random seeds. A linear classifier is then fitted exclusively on source representations and evaluated on the target environment. Full implementation details are provided in Appendix G.1.

The following table reports all five runs. The cosine columns report the mean absolute cosine similarity over all cross-class sample pairs. The final row reports the mean and sample standard deviation across the five seeds.

SeedTrain accuracy (%)Target accuracy (%)MCR² objectiveTrain mean abs. cosineTarget mean abs. cosine
43100.000.008.19730.0012540.001859
44100.000.008.09480.0025560.003640
45100.000.008.34550.0023730.002528
46100.000.008.13310.0019740.002680
47100.000.007.90110.0016150.001810
Mean ± SD100.00 ± 0.000.00 ± 0.008.1343 ± 0.16160.001954 ± 0.0005350.002503 ± 0.000745

Surprisingly, every run achieves exactly 100% source classification accuracy and 0% target accuracy, while maintaining very small cross-class similarities in both domains. Both source subsets separately achieve 100% accuracy. Moreover, the saved predictions consistently follow the source colour-to-label assignment on every evaluated source and target image.

These striking results demonstrate that actual MCR² training can produce highly discriminative, approximately orthogonal class representations while exhibiting complete OOD prediction failure. This provides strong empirical support for our theoretical analysis, showing that the identified failure mechanism also emerges in learned representations, rather than only in analytically prescribed encoders. The complete protocol, individual-seed results, objective trajectories, and class-sorted similarity heatmaps are provided in Appendix G.1–G.3, Table 9, and Figures 9–10.

Reviewer_6xcNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies whether maximal coding rate reduction (MCR2MCR^2) is a reliable framework to handle distribution shifts.

First, the authors empirically show that MCR2MCR^2 can fail on the Waterbirds classification dataset when the background-label association changes (e.g., a land bird has a water background at test time).

They then analyse a simple model with two classes, two-dimensional unit-norm representations, and two features: a stable feature (whichs correlated perfectyly with the ground truth class) and an environmental feature (whose association with the label can be imperfect at training time and can change at evaluation).

The paper shows two limitations of MCR2MCR^2 under this simple model:

  • The MCR2MCR^2 objective alone does not prevent inaccurate predictions under distribution shifts. More specifically, a representation built on the so-called environmental feature can reach the global coding optimum and still fail when the test environment is different, even though the stable feature is still predictive of the ground-truth class. Even when every test input "can occur during training", the coding value can come arbitrarily close to optimal while the prediction error approaches 100%.
  • Applying the invariance principle behind IRM and REx, by requiring the same coding operator to be optimal in every training environment, is not sufficient to fix the limitation of MCR2MCR^2, as the representation can fail while satisfying invariance.

The authors conclude that reliable OOD guarantees for MCR2MCR^2 need additional assumptions or learning principles.

Critical strengths
  • The paper provides theoretical evidence of why MCR2MCR^2 is not sufficient for OOD detection/distribution shifts. To the reviewers' knowledge, the math seems correct.
Critical weaknesses
  • insufficient empirical evidence: the only (partial) empirical evidence is on the Waterbirds dataset. This was run with a single seed and is not compared with other approaches, such as cross-entropy loss. Furthermore, the example uses a single feature extractor (ResNet-18) while the theory relies on two models (i.e., one for the stable feature and one for the environmental feature).
  • very idealized setting: the theoretical model makes many very strong assumptions: a noise-free stable feature, a nearly-redundant environmental feature wrt the labels, a very strong inversion of the correlation between the environmental feature and the label at test time, two feature extractors which each have access to their features, ... This last assumption implies that, for instance, in a foreground-background case, the foreground and the background are disentangled and that the classes are object-centric.
Questions for authors
  1. Could the authors formally define the notions of stable and environmental features in the paper?
  2. Could the authors provide more empirical evidence for the theoretical findings?
  3. Do the results still apply when using a single encoder, as for instance in [A]?

[A] Rosenfeld, E., Ravikumar, P., & Risteski, A. "The risks of invariant risk minimization". ICLR 2021.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6xcN, Part 1

We sincerely appreciate your careful assessment and constructive feedback. We address each of your concerns in detail below. For clarity and ease of reference, we have slightly rephrased some of your comments while preserving their original meaning.

Weakness 1: The empirical evidence is limited to partial results on the Waterbirds dataset and a single experimental run.

We would like to clarify that our focus on the two background-mismatched groups in Waterbirds is deliberate, as they directly reflect the OOD failure scenario investigated in our theoretical analysis. In the training data, landbirds predominantly appear on land backgrounds, while waterbirds predominantly appear on water backgrounds. We therefore specifically evaluate the two groups where these correlations are reversed: landbirds on water backgrounds and waterbirds on land backgrounds. By reporting both group-specific accuracies and their combined accuracy, we assess whether the MCR²-trained model maintains reliable predictions when the background–label correlations observed during training no longer hold.

To further address your concern, we have rerun the experiments using five different random seeds (43–47) and now report the results for all four Waterbirds groups, under both random and ImageNet-pretrained initialisation. The table below reports mean accuracy ± sample standard deviation (%) across five seeds, evaluated at the final training epoch without test-based checkpoint selection.

MCR² initialisationLandbird–LandLandbird–WaterWaterbird–LandWaterbird–Water
Random97.74 ± 0.7054.20 ± 8.3312.87 ± 3.3570.59 ± 2.31
ImageNet-pretrained99.26 ± 0.2081.75 ± 2.4239.94 ± 6.8087.35 ± 1.78

The performance gap between background-matched and background-mismatched groups persists across seeds and both initialisation settings. For example, with random initialisation, waterbird accuracy is 70.59±2.31%70.59\pm2.31\% on water backgrounds but only 12.87±3.35%12.87\pm3.35\% on land backgrounds. ImageNet pretraining improves performance, but the corresponding gap remains: 87.35±1.78%87.35\pm1.78\% versus 39.94±6.80%39.94\pm6.80\%. These results show that the observed difficulty is not confined to a single experimental run.

We have included the complete experimental settings, aggregate results, individual-seed results, and analysis in Appendix F (pp. 35–39), with the additions highlighted in light red. The four-group results appear in Table 3.

Weakness 2. The experiments lack comparisons with other approaches, such as cross-entropy.

We would like to clarify that our primary goal in this work is to establish a fundamental limitation of MCR² in OOD generalisation, rather than to propose a new method or demonstrate superiority over existing approaches.

Nevertheless, to address your concern further, we have added a cross-entropy baseline to provide a direct comparison with the MCR² results reported above. CE uses the same ResNet-18 encoder architecture, paired encoder initialisations, data splits, augmentations, and training budget: 50 epochs with AdamW, a learning rate of 3×10−43\times10^{-4}, and a batch size of 128. Its classification head is trained jointly with the encoder.

The table reports mean accuracy ± sample standard deviation (%) across five seeds (43–47). All results are evaluated at the final epoch, without test-based checkpoint selection.

CE initialisationLandbird–LandLandbird–WaterWaterbird–LandWaterbird–Water
Random96.17 ± 0.8451.62 ± 7.6117.01 ± 5.5977.91 ± 6.43
ImageNet-pretrained98.35 ± 0.6973.26 ± 4.8255.05 ± 3.7393.43 ± 1.76

CE also exhibits substantially lower accuracy on background-mismatched groups. Compared with the MCR² results above, neither objective consistently outperforms the other across all groups. This supports our position that the underlying OOD difficulty is not exclusive to MCR², while confirming that MCR² does not avoid it in these experiments.

The complete results and experimental details are included in Appendix F (pp. 35–39), with additions highlighted in light red. We additionally report CE representations evaluated using the same linear-probe procedure as MCR² to provide a comparison under a common classifier-fitting protocol.

AuthorsOpenReview ↗

Response to Reviewer 6xcN, Part 2

Weakness 3 & Question 3: The experiment uses a single ResNet-18, whereas the theory appears to require two encoders? Do the results still apply when using a single encoder, as for instance in [A]?

As these two comments are closely related, we respond to them together here.

We would like to clarify that our theory considers only a single encoder, consistent with our experiments. Therefore, our theoretical results directly apply to the single-encoder setting.

Specifically, fCf_C and fSf_S are two alternative candidate encoders within the same hypothesis class, not two networks operating together. Both receive the full input X=(C,S)X=(C,S): fCf_C uses the stable feature, while fSf_S uses the environmental feature. Each prediction pipeline contains one encoder followed by a classifier. The comparison shows that a perfectly predictive stable representation is available, yet an environmental representation can achieve optimal or arbitrarily near-optimal MCR² coding, depending on the support conditions, while failing under distribution shift.

Similarly, the single ResNet-18 in our Waterbirds experiment receives the complete image and can use information from the bird, its background, or both. The theory does not require separate networks or pre-segmented inputs. We will make this distinction clearer in our paper.

Weakness 4: The theoretical setting is highly idealised.

We would like to clarify that this simple setting is deliberate, because our aim is to demonstrate a fundamental limitation of MCR² in OOD generalisation. To establish this limitation, it is sufficient to show that MCR² can fail even in a argubly simplest and favourable setting.

Specifically, our construction provides a perfectly predictive stable feature and explicitly separates the stable and environmental features. In the foreground–background interpretation, this means that the two sources of information are already disentangled. Even with this separation and perfectly reliable stable information, MCR² still admits environmental representations that achieve optimal or arbitrarily near-optimal coding, depending on the support conditions, yet fail under distribution shift.

In practice, these factors can be much more complicated and entangled. If MCR² alone cannot guarantee stable prediction even under these favourable conditions, it cannot provide a general OOD guarantee in more complex settings that include this case. This is precisely why we use an idealised construction: it makes the limitation clear without introducing additional sources of difficulty.

Question 1: Could the authors formally define stable and environmental features?

We would like to clarify that the stable and environmental features are already formally specified in the original submission, in Section 2, under “A stable factor and an environmental factor,” Equation (6). We provide a more intuitive explanation below.

Intuitively, stable features provide predictive information that remains reliable across environments, whereas environmental features have predictive relationships with the label that can change across environments. For example, in cow recognition, features of the animal can remain informative whether the cow appears on grass or sand. Background information, however, may be predictive in the training data without remaining reliable in a different environment.

In our model, the input is X=(C,S)X=(C,S). The stable feature satisfies C=YC=Y in every environment, so the same prediction rule based on CC is always correct. The environmental feature satisfies

Thus, Pe(S≠Y)=δeP_e(S\neq Y)=\delta_e, and its predictive relationship with the label changes as δe\delta_e varies across environments.

The distinction concerns the stability of the feature–label relationship. These definitions describe predictive relationships and do not require causal assumptions. We will make their intuitive interpretation clearer in the paper.

AuthorsOpenReview ↗

Response to Reviewer 6xcN, Part 3

Question 2: Could the authors provide more empirical evidence for the theoretical findings?

We have expanded the Waterbirds experiments to include five random seeds, all four test groups, and a cross-entropy baseline under both random and ImageNet-pretrained initialisation, giving 20 training runs in total. These results show that poor performance on background-mismatched groups persists across seeds and both initialisation settings. Please see our responses to Weaknesses 1 and 2 and Appendix F (pp. 35–39) for the experimental settings, results, and analysis.

To connect the theoretical findings more directly to actual training, we have also added a controlled Colored MNIST experiment. We use clean digit-group labels without label noise, make colour perfectly predictive during training, and reverse the colour–label association at test time. We train a CNN with MCR² using ϵ2=1\epsilon^2=1 and five random seeds (43–47). Every run achieves 100% training accuracy and 0% target accuracy, while the learned representations exhibit very small cross-class absolute cosine similarities in both domains.

This experiment shows that actual MCR² training can produce highly discriminative, approximately orthogonal class representations together with complete OOD prediction failure, complementing the analytically constructed representations in our theoretical analysis.

Please see our response to Reviewer nqLV’s Question 3 for further discussion, including how we assess training fit and coding geometry indirectly when the global optimal objective value is unknown. The complete controlled experiment is presented in Appendix G (pp. 40–42), Table 9, and Figures 9–10. The additions in both appendices are highlighted in light red.

Reviewer_GhfpOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on MCR, a learning approach that explicitly specifies the geometry of the learned representation space. The paper makes the point that while MCR has attracted attention, it lacks robustness to distribution shifts, a phenomenon that has been understudied. The contributions of the paper are theoretical and formalize why MCR does not imply robustness under distribution shift: an encoder can minimize the MCR objective without being robust to spurious correlations. Finally, the paper looks at IRM and REx, approaches used to make ERM more robust. They state that these methods, which impose invariance of the prediction score to change of environments, cannot guarantee robustness then combined with MCR.

Critical strengths

Presentation: the paper is fell formulated and easy to follow. Enough care was put into presenting, motivating and explaining results.

Scientific rigor: The paper formalize the weaknesses of MCR using a model simple enough to be analyzed completely. The findings and examples attached formalize the reason why MCR can fundamentally not distinguish between stable and spurious correlations leading to representations that be show a small loss but high prediction error.

Critical weaknesses

Impact and scope. The analysis formalises a failure mode for MCR² but is limited to d=K=2d=K=2 with balanced classes, offers no solution to the problem, and offers limited exploration (waterbird examples) on whether, in practice, training selects the unstable features in common training pipeline.

Novelty. The core mechanism (a representation relying on an environmental feature is near-optimal on source data and fails after the feature-label correlation reverses) applies equally to cross-entropy. The results therefore do not isolate a behavior specific to MCR².

Minor. Lemma 3.1, Prop 3.2, Thm 3.3 and Thm 4.1 don't reference their proofs in Appendix B, and Appendix D is not referenced from the main text.

Questions for authors

The analysis assumes a balanced binary label distribution. How would the findings change if the classes were imbalanced?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ghfp: Part 1

We sincerely thank you for carefully evaluating our work and recognising its clear presentation and scientific rigour. We are encouraged by your appreciation of our theoretical analysis. Below, we address your concerns point by point. To improve the clarity and readability of our responses, we have slightly rephrased your comments without changing their original meaning.

Weakness 1: The theoretical analysis is restricted to a relatively simple setting (d=K=2d = K = 2 with balanced classes).

We would like to clarify that our primary goal in this work is to establish a fundamental limitation of MCR² in OOD generalisation. To this end, we deliberately consider arguably the simplest possible setting (d=K=2d = K = 2 with balanced classes) and demonstrate that MCR² can fail to achieve OOD generalisation even under these favourable conditions. In principle, the existence of such a failure in such minimal setting is sufficient to establish that MCR² alone cannot guarantee OOD generalisation in general. More complex settings, such as those involving higher-dimensional representations or larger numbers of classes, do not eliminate this limitation, as they can include our setting as a special case. Thus, the simplicity of our theoretical setting is a deliberate choice rather than a limitation of our central conclusion. In other words, if MCR² cannot guarantee the learning of stable predictive relationships even in this favourable simple setting, it cannot provide a universal guarantee in more complex settings.

Weakness 2: No remedy is proposed for the OOD limitations of MCR².

We agree that developing a remedy is an important direction for future research. However, we would like to clarify that the primary aim of this paper is to provide a constructive critical examination of the capabilities and limitations of MCR².

As discussed in our paper, MCR², formulated as a generalised information gain objective, has driven substantial progress across a broad range of settings, including supervised, unsupervised, and self-supervised learning, generative and denoising models, and interpretable architectures such as "white-box" transformers and convolutional networks. Its growing influence has also inspired a more ambitious research agenda: establishing mathematical principles underlying representation learning and intelligence, as reflected in the recent book Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory. These advances naturally raise a fundamental question: How far can MCR²-based representation learning take us? Our paper examines this question through OOD generalisation, demonstrating that, despite its ability to promote structured and discriminative representations, MCR² alone does not guarantee stable predictions across environments, which however is esepcially valuable for practical machine learning applications.

Specifically, our results show that optimal or near-optimal MCR² coding can coexist with severe OOD prediction failure, and that the shared-optimal-coding constraint studied here does not resolve this limitation. This exposes a gap between learning discriminative representations of observed data and learning predictive relationships that remain stable across environments. Rigorously identifying this gap is a meaningful contribution, as it clarifies what coding-based criteria leave unconstrained and what future methods must establish. A satisfactory solution must explain how coding quality can be connected to predictive stability, rather than treating coding optimality as sufficient evidence of OOD generalisation.

Looking ahead, our findings motivate exploring how coding-based learning can adapt to environmental changes. One promising approach is to develop closed-loop learning systems that evaluate and refine their representations through feedback, aligning with the principles of parsimony and self-consistency discussed by Ma et al. [1]. A concrete example is i-CTRL, proposed by Tong et al. [2], which uses a rate-reduction-based closed-loop framework to learn new classes while preserving previously learned knowledge. Extending this idea to OOD generalisation, a system could evaluate its predictions across environments and, when predictions become unreliable, use feedback to adjust its parameters and refine its representations. Through repeated evaluation and adaptation, the goal would be to learn predictive relationships that remain stable across environments, rather than relying solely on source-domain coding quality.

We have added this discussion of closed-loop learning as a potential direction for addressing the OOD limitations of MCR² to Appendix H (p. 43).

[1] Ma et al., On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence, FITEE, 2022.

[2] Tong et al., Incremental Learning of Structured Memory via Closed-Loop Transcription, ICLR, 2023.

AuthorsOpenReview ↗

Response to Reviewer Ghfp: Part 2

Weakness 3: Do practical machine learning methods indeed favour unstable features in common training pipelines?

Yes, and we would like to provide two complementary lines of evidence below.

1. Practical relevance and existing evidence

In common machine learning practice, when spurious correlations are stronger or more predictive than stable correlations in the training data, models can preferentially learn these spurious features to minimise their training objectives [3]. This is a well-documented phenomenon and a central motivation for extensive research on out-of-distribution (OOD) generalisation.

Specifically, models trained using empirical risk minimisation (ERM) can exploit spurious correlations that are highly predictive in the training distribution but become unreliable when the environment changes. Even models achieving high training accuracy (low training loss) may suffer substantial performance degradation under distribution shifts, which commonly arise across locations, time periods, and data-collection conditions [3,4]. A classic example is the cows-and-camels example in IRM [3], where a model may associate cows with green pastures and camels with deserts rather than learning to recognise the animals themselves. This tendency to rely on spurious features has been extensively studied [5,6].

To investigate whether MCR² exhibits this failure in practice, we conduct experiments on Waterbirds, a widely used benchmark for evaluating robustness to spurious correlations between bird categories and image backgrounds [5]. Our results demonstrate that, under a standard training setup, MCR² can learn representations that lead to poor OOD predictive performance. This empirical evidence complements our theoretical analysis, showing that the inability of coding quality alone to guarantee OOD generalisation is not merely an artefact of our theoretical construction but can also manifest in practical training.

2. Additional evidence from actual MCR² training

We have further expanded the Waterbirds experiments to include five random seeds, all four test groups, and a matched cross-entropy baseline under both random and ImageNet-pretrained initialisation. The performance disparities between background-matched and background-mismatched groups persist across seeds and both initialisation settings. For example, with random initialisation, waterbird accuracy is 70.59 ± 2.31% on water backgrounds but only 12.87 ± 3.35% on land backgrounds. With pretraining, the corresponding accuracies are 87.35 ± 1.78% and 39.94 ± 6.80%. These results confirm that the observed difficulty is not confined to a single run or initialisation setting. Complete experimental details and results are provided in Appendix F (pp. 35–39).

We have also added a controlled Colored MNIST experiment to connect the theoretical failure mechanism more directly to actual training. We use clean digit-group labels without label noise, make colour perfectly predictive during training, and reverse the colour–label association at test time. We train a CNN with MCR² using ϵ2=1\epsilon^2=1 across five random seeds (43–47). Every run achieves 100% training accuracy and 0% target accuracy, while the learned representations exhibit very small cross-class absolute cosine similarities in both domains. The predictions consistently follow the source colour-to-label assignment on every evaluated source and target image.

These additional experiments demonstrate that actual MCR² training can produce highly discriminative, approximately orthogonal class representations together with complete OOD prediction failure. Thus, the failure mechanism studied in our theory can emerge in learned representations, rather than occurring only in analytically prescribed encoders. The controlled experiment, objective trajectories, and class-sorted similarity heatmaps are presented in Appendix G (pp. 40–42), Table 9, and Figures 9–10. Please also see our response to Reviewer nqLV’s Question 3 for further discussion. The additions in both appendices are highlighted in light red.

[3] Arjovsky et al., Invariant Risk Minimization, arXiv, 2019.

[4] Koh et al., WILDS: A Benchmark of in-the-Wild Distribution Shifts, ICML, 2021.

[5] Sagawa et al., Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, ICLR, 2020.

[6] Sagawa et al., An Investigation of Why Overparameterization Exacerbates Spurious Correlations, ICML, 2020.

AuthorsOpenReview ↗

Response to Reviewer Ghfp: Part 3

Weakness 4: The failure mechanism in OOD setting also applies to cross-entropy and is therefore not specific to MCR².

We would like to clarify that reliance on environmental features whose predictive relationships vary across environments is actually a general source of OOD failure in common machine learning training pipelines. In principle, any empirical risk minimisation (ERM) learning objectives, such as mean squared error (MSE) for regression and cross-entropy for classification, can favour predictors that exploit correlations in the training distribution without guaranteeing that these correlations remain valid in unseen environments. We therefore do not claim that this general failure mechanism is specific to MCR².

Rather, our central question is whether MCR², as a principled framework for structured representation learning and architectural design, can overcome this fundamental limitation. Unlike conventional prediction losses, MCR² provides a coding-based learning principle with explicit geometric interpretations. It is therefore important to investigate whether the representation structures encouraged by this principle are sufficient to ensure reliable generalisation across environments. Our paper demonstrates that they are not, even when a perfectly predictive stable feature is available.

We summarise that our novelty therefore lies in establishing the limitations of MCR²-based learning criteria, rather than identifying a failure mechanism exclusive to MCR². While the vulnerability of cross-entropy to spurious correlations is well established, this does not imply that the geometric structure or shared coding optimality promoted by MCR² necessarily suffers from the same limitation. Establishing this requires the objective-specific theoretical analysis developed in our paper.

Weakness 5 (Minor): The theoretical results in the main text do not explicitly reference their proofs in the appendix.

Thank you very much for your detailed review. We have added explicit references to the corresponding proofs in the appendix and highlighted these additions in light red.

Question 1: How do the findings change when the classes are imbalanced?

Our main conclusions also hold for imbalanced classes. Specifically, the results derived from the balanced-class assumption can be extended to any fixed class prior P(Y=1)=p∈(0,1)P(Y=1)=p\in(0,1), provided that the prior remains the same across source and target environments.

Intuitively, class imbalance changes how much each class contributes to the coding objective, but it does not resolve the underlying problem: a feature can be highly predictive during training without maintaining that predictive relationship at test time. For example, having more landbirds than waterbirds does not make the association between bird category and background stable across environments. A representation based on background can still distinguish the classes well in the source environment and support incorrect predictions when that association reverses.

Our extended analysis shows that, for any fixed class imbalance with 0<p<10<p<1, an environmental representation can still achieve arbitrarily near-optimal MCR² coding while its target prediction error approaches 100%, even with identical source and target input support and an available perfectly predictive stable feature. The shared-optimal-coding result also extends to these imbalanced environments: the same coding operator can remain optimal across training environments and after correlation reversal. In other words, changing the relative frequencies of the two classes does not eliminate the mismatch between optimal coding geometry and stable label prediction.

Class imbalance changes the optimal coding value and adds one condition: the source mismatch probability must be smaller than the minority-class probability, i.e., δe<min⁡{p,1−p}\delta_e<\min \{p,1-p\}. Intuitively, the environmental feature must be sufficiently reliable during training that the source-optimal classifier follows it rather than simply predicting the majority class. This condition can be satisfied for any fixed p∈(0,1)p\in(0,1) by choosing sufficiently small source mismatch probabilities.

The failure is therefore not an artefact of balanced classes. Class imbalance changes the quantitative expressions and the required strength of the source correlation, but does not eliminate the gap between coding quality and predictive stability. We have included the extended analysis and complete proofs in Appendix C.6, “Extension to imbalanced classes” (pp. 26–29), with the additions highlighted in light red.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We sincerely thank all reviewers for their detailed and constructive feedback, which has helped us clarify our contributions and strengthen the paper. We appreciate their positive comments on:

  • Clear presentation and motivation
  • Scientific rigour
  • Precise theoretical characterisation
  • Useful insights into the OOD limitations of MCR²
  • Transparent discussion of the scope and limitations

Below, we summarise our main clarifications and revisions. The added analyses and experiments are highlighted in light red in the revised paper.

For ease of reference during the rebuttal process, we have placed most of the new material in the appendices, preserving the structure of the main text. After the rebuttal period, we will integrate the relevant additions into the main text.

  • Concern 1: Empirical evidence is not sufficient, for example, only a single run. (Reviewers Ghfp, 6xcN, and nqLV)

    We expanded the Waterbirds experiments to include five random seeds, all four test groups, and matched cross-entropy comparisons under both random and ImageNet-pretrained initialisation (Appendix F).

    To further support our theoretical analysis, we also added controlled experiments on the arguably most widely used Colored MNIST dataset (in ODD generalisation research) using actual MCR² training (Appendix G).

  • Concern 2: The theoretical setting is simple. (Reviewers Ghfp and 6xcN)

    We clarified that this choice is deliberate: our aim is to establish that MCR² alone cannot guarantee OOD generalisation. A counterexample in a simple setting—even with a perfectly predictive stable feature—is sufficient to establish this limitation. A universal guarantee over more complex settings that include this case is therefore also ruled out.

  • Concern 3: The failure setting of MCR² also applies to cross-entropy. (Reviewers Ghfp and nqLV)

    We clarified that OOD failure due to spurious correlations is already well established for cross-entropy, so it is unsurprising that the underlying failure setting is shared. Our aim is to determine whether MCR²’s coding-based learning principle can overcome this limitation. Our contribution therefore lies in characterising this limitation within the MCR² framework, which requires analysing its coding objective and optimisation structure rather than directly transferring cross-entropy failure results.

  • Concern 4: Lack of comparison with cross-entropy. (Reviewers 6xcN and nqLV)

    We clarified that our aim is to identify a fundamental limitation of MCR² in OOD generalisation, rather than propose a new method or demonstrate superiority over existing approaches. Nevertheless, considering that this comparison might provide readers with a more comprehensive understanding of our work, we therefore added matched cross-entropy comparisons (Appendix F).

  • Concern 5: The theory appears to require two encoders, whereas the experiments use only one? (Reviewer 6xcN)

    We clarified that our theory only requires one encoder. fCf_C and fSf_S are two alternative candidate encoders within the same hypothesis class, not two networks operating together.

  • Concern 6: Can the analysis be extended to imbalanced classes? (Reviewer Ghfp)

    We extended the analysis to any fixed, nondegenerate class prior shared across source and target environments, showing that our main conclusions continue to hold under class imbalance (Appendix C.6).

  • Concern 7: No remedy is proposed. (Reviewer Ghfp)

    We clarified that the primary aim of this work is to identify the fundamental limitations of MCR², while developing a remedy is an important next step. We have added a discussion of closed-loop learning systems that refine representations through environmental feedback as a promising research direction in Appendix H (p. 43).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.