- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the OOD limitations of MCR² in a binary model with stable and environmental features. It derives the global coding optimum and constructs near-optimal environmental representations with almost maximal target error under correlation reversal. These representations also satisfy a shared-optimal-coding constraint. Exact-optimum failure requires changing input support; with common support, the failure concerns near-optimal representations. Waterbirds motivates the analysis, and synthetic experiments validate the formulas.
Critical strengths
-
The theoretical characterization is precise. The global coding optimum, the environmental encoder's objective gap, and its optimal coding operators are explicitly derived. Correlation reversal preserving the coding objective while reversing the appropriate label assignments is a useful observation.
-
The limitations are transparently stated. The authors acknowledge the potentially large distribution shift, the success of exact optimizers in the full-support model, and the absence of an analysis of which representations training selects.
Critical weaknesses
-
The motivation and incremental contribution are insufficiently established. The central near-optimality/failure phenomenon is not specific to MCR². In the same model, an environmental predictor has source classification error and target error ; its optimal source cross-entropy is , approaching the global infimum. Related work already studies spurious-feature reliance, including Sagawa et al. (2020) and Nagarajan et al. (2021). Although Section 5 and Appendix A.2 acknowledge this distinction, the substantive insight added by the MCR²-specific calculations remains unclear. A same-model comparison should separate generic shift-related difficulties from properties particular to the coding objective.
-
The experiments do not sufficiently connect the counterexample to practical learning. Waterbirds lacks a matched cross-entropy baseline, all four group results, and variability across seeds. Thus, the coding objective's role cannot be separated from initialization, data imbalance, and training choices. Synthetic experiments use prescribed encoders, so they do not establish that optimization selects the failing representations—especially when the stable encoder has a strictly better objective. Controlled encoder-training experiments would strengthen this connection. Encoder-gradient saliency alone also does not establish background reliance by the classifier.
Questions for authors
-
Contribution clarification: Which conclusions distinguish MCR² from ordinary supervised risks under the same data model and shift assumptions? How do they advance the existing understanding of spurious correlations?
-
Matched empirical comparison: Can the authors compare MCR² with cross-entropy using the same ResNet-18, initialization, training budget, and model-selection protocol, reporting source performance and all four test groups across seeds?
-
Evidence that could change my assessment: Can actual MCR² training produce near-optimal but target-failing representations in a controlled experiment? Alternatively, can the authors sharpen the theoretical insight by characterizing guarantees under a bounded shift?
6 comments
Response to Reviewer nqLV, Part 2
Weakness 2 & Question 2: The experiments do not sufficiently connect the counterexample to practical learning. Can the authors compare MCR² with cross-entropy using the same ResNet-18, initialization, training budget, and model-selection protocol, reporting source performance and all four test groups across seeds?
Thank you for these suggestions. We have expanded the Waterbirds experiments to include a matched cross-entropy baseline, all four test groups, and five random seeds under both random and ImageNet-pretrained initialisation, giving 20 training runs in total.
To make the comparison controlled, both objectives use the same ResNet-18 encoder architecture, paired encoder initialisations, training data, image augmentations, optimiser settings, and training budget. Each run uses 50 epochs, AdamW with a constant learning rate of , and a batch size of 128. We report the final epoch for every run, without selecting checkpoints based on test performance.
The original Waterbirds training distribution is imbalanced, and we preserve exactly the same distribution for both objectives. The four training groups—landbird–land, landbird–water, waterbird–land, and waterbird–water—contain 3,498, 184, 56, and 1,057 images, respectively. Neither objective uses group reweighting or balanced sampling. This controls the data distribution in the comparison, while the two initialisation settings allow us to examine whether the observed behaviour persists with and without pretraining.
The table reports mean accuracy ± sample standard deviation (%) across five seeds (43–47). Training accuracy is measured on the complete training split using deterministic preprocessing. MCR² uses a source-trained linear probe; CE uses its jointly trained classification head.
| Initialisation | Objective | Train | Landbird–Land | Landbird–Water | Waterbird–Land | Waterbird–Water |
|---|---|---|---|---|---|---|
| Random | MCR² | 92.74 ± 0.39 | 97.74 ± 0.70 | 54.20 ± 8.33 | 12.87 ± 3.35 | 70.59 ± 2.31 |
| Random | CE | 95.14 ± 0.78 | 96.17 ± 0.84 | 51.62 ± 7.61 | 17.01 ± 5.59 | 77.91 ± 6.43 |
| ImageNet-pretrained | MCR² | 97.84 ± 0.41 | 99.26 ± 0.20 | 81.75 ± 2.42 | 39.94 ± 6.80 | 87.35 ± 1.78 |
| ImageNet-pretrained | CE | 99.12 ± 0.27 | 98.35 ± 0.69 | 73.26 ± 4.82 | 55.05 ± 3.73 | 93.43 ± 1.76 |
These experiments show that actual MCR² training can produce substantial performance disparities between background-matched and background-mismatched groups across multiple seeds and both initialisation settings. Pretraining improves performance but does not eliminate these disparities. CE also exhibits this difficulty, consistent with our position that the underlying failure mechanism is not exclusive to MCR².
To account for differences in classifier fitting, we additionally evaluate the CE representations using the same linear-probe procedure as MCR², with the results reported in Appendix F, Tables 3 and 4. The corresponding worst-group test accuracies are with random initialisation and with pretraining, showing that the group disparity also persists under this common evaluation protocol.
The complete settings, results, and analysis are included in Appendix F (pp. 35–39), with additions highlighted in light red. These experiments strengthen the connection to practical training; the precise near-optimality statements remain established by our theoretical analysis.
Response to Reviewer nqLV, Part 3
Question 3.1: Can actual MCR² training produce near-optimal but target-failing representations in a controlled experiment?
We have now conducted controlled experiments using actual MCR² training and evaluated target prediction performance. The results strongly support our theoretical analysis.
Specifically, we first clarify how MCR² optimality can be assessed in practice. We highlight that for a given dataset and encoder architecture, the global maximum of the MCR² objective is generally unknown. The objective depends on the sample size, representation dimension, and coding precision , while the achievable maximum also depends on the representations that the encoder can realise. So we cannot directly calculate the theoretical optimality gap.
However, previous MCR² theory establishes that optimal representations have mutually orthogonal class subspaces under its stated assumptions [1]. This provides a geometric property that can be examined empirically. Motivated by this result, we use training accuracy, the observed MCR² objective trajectory, and cross-class absolute cosine similarity as complementary indirect measures of training fit and representation geometry. Intuitively, when training accuracy approaches 100%, and representations from different classes become nearly orthogonal, these observations provide strong empirical evidence that MCR² training is approaching an optimal solution.
Based on this motivation, we conducted controlled experiments using actual MCR² training to examine whether well-fitted, approximately orthogonal class representations can nevertheless lead to target prediction failure. The experimental setting and results are reported in Appendix G (pp. 40–42).
We use a noise-free variant of Colored MNIST [2], a widely used benchmark for OOD generalisation. We group digits 0–4 into class 0 and digits 5–9 into class 1, without label flips. In the source environment, class 0 is always blue and class 1 is always red, while these colours are reversed in the target environment. Thus, both digit identity and colour perfectly predict the source labels, but only digit identity remains predictive under the distribution shift.
We train a single CNN using the actual MCR² objective () for 500 full-batch updates across five random seeds. A linear classifier is then fitted exclusively on source representations and evaluated on the target environment. Full implementation details are provided in Appendix G.1.
The following table reports all five runs. The cosine columns report the mean absolute cosine similarity over all cross-class sample pairs. The final row reports the mean and sample standard deviation across the five seeds.
| Seed | Train accuracy (%) | Target accuracy (%) | MCR² objective | Train mean abs. cosine | Target mean abs. cosine |
|---|---|---|---|---|---|
| 43 | 100.00 | 0.00 | 8.1973 | 0.001254 | 0.001859 |
| 44 | 100.00 | 0.00 | 8.0948 | 0.002556 | 0.003640 |
| 45 | 100.00 | 0.00 | 8.3455 | 0.002373 | 0.002528 |
| 46 | 100.00 | 0.00 | 8.1331 | 0.001974 | 0.002680 |
| 47 | 100.00 | 0.00 | 7.9011 | 0.001615 | 0.001810 |
| Mean ± SD | 100.00 ± 0.00 | 0.00 ± 0.00 | 8.1343 ± 0.1616 | 0.001954 ± 0.000535 | 0.002503 ± 0.000745 |
Surprisingly, every run achieves exactly 100% source classification accuracy and 0% target accuracy, while maintaining very small cross-class similarities in both domains. Both source subsets separately achieve 100% accuracy. Moreover, the saved predictions consistently follow the source colour-to-label assignment on every evaluated source and target image.
These striking results demonstrate that actual MCR² training can produce highly discriminative, approximately orthogonal class representations while exhibiting complete OOD prediction failure. This provides strong empirical support for our theoretical analysis, showing that the identified failure mechanism also emerges in learned representations, rather than only in analytically prescribed encoders. The complete protocol, individual-seed results, objective trajectories, and class-sorted similarity heatmaps are provided in Appendix G.1–G.3, Table 9, and Figures 9–10.
Response to Reviewer nqLV, Part 1
We sincerely thank you for your detailed evaluation and constructive feedback. We appreciate your recognition of the precision of our theoretical characterisation and our transparent discussion of the paper’s limitations. Below, we address your concerns point by point. For clarity and ease of reference, we have slightly rephrased and reordered some comments while preserving their original meaning, so that our responses follow a logical flow.
Weakness 1: The motivation and incremental contribution are insufficiently established, as the failure mechanism also applies to cross-entropy.
Question 1: Which conclusions distinguish MCR² from ordinary supervised risks, and how do they advance the understanding of spurious correlations?