acceptodds
Under review as a conference paper at ICLR 2027

The Optimisation Landscape of Maximal Coding Rate Reduction

Abstract

Understanding how neural networks organise learned representations is important for improving their generalisation, robustness, interpretability, and ultimately safety. The recent proposed framework Maximal Coding Rate Reduction (MCR) offers an information-theoretic perspective: it aims to capture the underlying structure of each class while arranging different classes in orthogonal subspaces. Despite promising applications, the optimisation landscape of MCR remains poorly understood. Due to the requirement of every feature should have the same scale of information, the landscape of MCR is difficult to analyse. Previous work has therefore studied a relaxed Frobenius-regularised variant. We show that this variant can perform poorly in practice, limiting the practical guidance offered by its landscape analysis. In this work, we characterise key features of local and global optima of MCR itself and analyse other critical points. In particular, every local optimum whose occupied class ranks sum to at most the representation dimension separates the classes orthogonally. We also describe global solutions in several important settings, show how classes may overlap when feature space is limited, and identify configurations where local optimisation can stall or second-order tests are inconclusive. These results explain when MCR favours discriminative representations under first-order optimisation and where its guarantees stop. Extensive experiments with synthetic data and real image datasets support the theoretical analysis.

Primary area:
learning theory
Keywords:
Maximal Coding Rate Reduction (MCR²), Optimization Landscape, Representation Learning, Class Orthogonality, Strict Saddles, Tight Frames

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_12r1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Maximum coding rate reduction(MCR2) aims to capture the underlying structure of each class by arranging classes in different subspaces. However, there is a lack of understanding as to the optimization dynamics that lead to sufficiently good representations. Other work has introduced relaxed assumptions that lead to poor performance. The authors present an interesting analysis of these dynamics and find a variety of dimension based considerations. These include settings such as global vs. local optima, settings with limited feature spaces, and when MCR2 doesn't lead to more discriminative representations.

Critical strengths
  1. These are very interesting settings of analysis within the context of representation learning literature. For example, finding local optima where the same class separation occurs is especially improtant in settings like self supervised learning where obtaining global optima is less likely due to the lack of structured task information. This idea of representation size of the space is also interesting because it under scores a relevant consideration in constrainted domains with limited diversity or training data that have a difficult time obtaining high ranks. Identifying these settings I think is especially interesting for future work on this topic.

  2. The orthogonal class separation result is especially impactful as it suggests that the MCR2 setup can induce additional class separation with limited intervention on the represetnations which is abstracted to a rotation in this work.

  3. The conclusions are notably more specific than previous work that has attempted to analyze MCR2 in a theoretical manner: https://arxiv.org/pdf/2406.01909

  4. The authors show toy experiments of their theoretical perspectives.

Critical weaknesses
  1. A lot of the discussion is somewhat similar to work that has appeared within the self supervisd representation learning literature and it is difficult to appreciate the novelty of your analysis without some discussion as to these other works.

For exmaple, in your theorem 1, there was a discussion about a class rotating part of his energy towards an unused direction in order to increase the total rate.

In Theorem 3, there was a discussion of the total coding rate largest when the d eigenvalues are effectively equal. Statements like this are used throughout the paper and it raises questions as to whether any of this analysis is especially important in the context of larger represetnation learning literature. What I mean by this is that this paper is advertising itself as understanding how the optimization landscape of MCR2 works. It is, by definition, constraining its comparisons with MCR2 literature. However, I feel that this is very limiting. Ultimately, MCR2 is a representation learning approach. All representation learning approaches attempt to do something similar to what is described in this paper. Therefore, it is possible to view the results in this paper as off shoots of other representation learning literature. In other words, this paper could potentially be showing that MCR2 functions as other representation learning approaches.

For example, a variety of works have shown that greater rank corresponds to better downstream performance and better rank also corresponds to equality between eigenvalues:

Effective rank: The effective rank: a measure of effective dimensionality

Alpha-REQ: alpha-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay

LiDAR: LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures

Other works have their own perspective on how samples distribute in a representation space:

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization: Analyzes the behavior of equalizing eigenvalues in the context of covariance regularization

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere: Analyzes the spread of samples in the case of a hypersphere geometry

My point is that I feel that many of the conclusions within the context of MCR2 are different ways of saying the same results from representation learning literature in general.

  1. It's difficult to understand how the experiments support the theoretical conclusions.

Class separation with sufficient dimension: The interpretation of these figures is lacking. I think its necessary to refer back to which theories or assumptions are proven here to make the paper wholistic. Right now you state what's in here, but I'm really unsure where to look back in the theories to show that some interesting connection back to the theorey exist.

Many samples and real-data class geometry: Is the point here to show class separability? It is unclear what geometry you are trying to support here. My point is that are you trying to show that MCR2 separates classes? The problem with that point is that the original paper already shows this, what is new by this separability in the context of your theory.

Rotation formula check: I understand your theory discusses how simple rotations can fill gaps in the represetnation space, but that doesn't come out here.

A similar problem happens in the other sections.

  1. In the theoretical discussion, you don't make it clear what the the practical implication of your result suggests about the MCR2 framework. Look at what I said in your strengths section. This is me trying to interpret from your theorems as to what it suggests about what previously wasn't understood before. This problem transfers to your experiments where there are many statements, but no integration with how the results suggests something impactful about the theory and its implication for MCR2. Ideally, what I would prefer is that each theoretical statement goes back to an original not understood problem about MCR2 optimality. Right now, what I see is a discussion of what the behavior should look like such as using pockets of space with unused feature directions, but the implication of this theory doesn't feel impactful at the moment.

I am happy to be told that I am wrong and explained in the rebuttal, but for the moment I have reservations. I can imagine where these points could be impactful, but without that confirmation in writing, its difficult to know.

Questions for authors
  1. I think you need additional clarification of different terms for me to understand this fully. A class rank is the rank of the features of samples only belonging to a single class?

See some points in the weaknesses. I want to understand why each of the theorems is impactful in the context of MCR2 and how it rethinks the optimization landscape.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 12r1, Part 1

We sincerely thank the reviewer for such thoughtful, detailed, and insightful comments. We also thanks for recognising the value of our class-separation results and dimension-dependent analysis. Below, we address your concerns point by point. To make our responses clearer and easier to follow, we have slightly rephrased your comments while preserving their original meaning.

Weakness 1: The novelty relative to the broader representation-learning literature needs clarification.

We would like to clarify that the primary goal of our paper is to understand the intrinsic optimisation properties of the MCR² objective itself, rather than to study representation geometry in general. In particular, we aim to understand what kinds of representations MCR² encourages, under what conditions they emerge, and what limitations exist in its optimisation landscape. Below, we explain why studying this objective is important and how our work differs from the broader representation-learning literature.

1. Why understanding the MCR² objective matters

As discussed in our response to Reviewer 3Wfx, Weakness 1, MCR² has inspired principled approaches to neural network design. For example, CRATE [1] derives transformer blocks from iterative optimisation of a sparse rate-reduction objective and achieves competitive performance on ImageNet, approaching that of highly engineered Vision Transformers.

These developments highlight the practical importance of understanding the learning objectives underlying modern neural architectures. In particular, what makes MCR² encourage useful representations, and under what conditions might its favourable properties fail? Answering these questions can provide deeper theoretical insights into MCR² and potentially guide the development of related learning objectives and architectures.

Our paper focuses precisely on these fundamental questions by directly investigating the optimisation landscape of the MCR² objective. We characterise its local and global optimality properties under different conditions and identify important limitations of its landscape. We believe that understanding these properties is valuable in its own right and provides a theoretical foundation for further developments.

2. How our work differs from existing representation-learning studies

We thank the reviewer for highlighting these relevant works. Indeed, previous studies have provided valuable insights into representation geometry and learning objectives. For example, effective rank [2], α-ReQ [3], and LiDAR [4] investigate spectral properties of learned representations and their relationships with representation quality or generalisation. Meanwhile, the information-theoretic analysis of VICReg [5] and alignment–uniformity theory [6] investigate the theoretical properties of other representation-learning objectives.

However, these studies address different research questions from ours. The former primarily characterise or evaluate the properties of learned representations, while the latter investigate different learning objectives. None of these cited works directly analyses the optimisation landscape of the MCR² objective, which is the central focus of our paper.

We do not claim that geometric concepts such as orthogonality, low rank, or isotropy are themselves new. Rather, our contribution is to establish when such structures are guaranteed to emerge from MCR² optimisation, how they relate to local and global optimality, and when these guarantees break down. This distinction is important because identifying desirable geometric properties is fundamentally different from establishing whether a particular learning objective necessarily produces them.

We have incorporated this discussion and the relevant references into the dedicated Related Work section in Appendix A, with detailed mathematical examples provided separately in Appendix F.4. These additions clarify how our landscape analysis relates to existing studies of representation geometry and identify the additional guarantees and limitations established by our results.

[1] Yu et al. (2023). White-Box Transformers via Sparse Rate Reduction. NeurIPS.

[2] Roy and Vetterli (2007). The Effective Rank: A Measure of Effective Dimensionality. EUSIPCO.

[3] Agrawal et al. (2022). α-ReQ: Assessing Representation Quality in Self-Supervised Learning by Measuring Eigenspectrum Decay. NeurIPS.

[4] Thilak et al. (2024). LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures. ICLR.

[5] Shwartz-Ziv et al. (2023). An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization. NeurIPS.

[6] Wang and Isola (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML.

AuthorsOpenReview ↗

Response to Reviewer 12r1, Part 2

Weakness 2: The connection between the experiments and the theoretical results is not clear enough.

We have revised Section 5 and the captions of Figures 3–7 to clarify the connections between our experiments and theoretical results. All changes are highlighted in red in the revised manuscript. Below, we summarise these revisions and explain how each experiment relates to our theoretical findings.

1. Class separation with sufficient dimension — Figure 3

This experiment examines Theorem 1 and Corollary 1 on local class separation, together with Theorem 2 on global optimality. We set d=md=m, ensuring that ∑krank⁡(Zk)≤d\sum_k \operatorname{rank}(Z_k)\leq d throughout optimisation. Across all 900 runs, the final representations exhibit nearly orthogonal class subspaces. However, 215 endpoints have objective-value gaps greater than 10−710^{-7} relative to the exact global optimum. This highlights an important distinction: class orthogonality can emerge without achieving the globally optimal within-class rank structure. Thus, the experiment illustrates both the predicted separation and its limitations.

2. Many samples and real-data class geometry — Figure 4

This experiment examines an important implication of our theory: the dimension requirement depends on the sum of class ranks, rather than the total number of samples. Specifically, we use the digits dataset with 1,797 samples and project each class into a four-dimensional subspace, giving a total class rank of 40. We then test ambient dimensions d∈{40,64,80}d\in\{40,64,80\}. Notably, even at d=40d=40, the rank condition is satisfied despite having far more samples than representation dimensions.

Importantly, we keep all within-class Gram matrices fixed throughout optimisation. Therefore, class separation arises entirely from changes in the relative orientations of class subspaces, rather than changes in their internal geometry. This directly reflects the class-rotation mechanism underlying our theoretical analysis and demonstrates its applicability to data-derived class structures.

3. Verification of the class-rotation mechanism — Figure 6(a–b)

These experiments numerically verify the class-rotation mechanism established in Theorem 1. Starting from constructed nonorthogonal critical configurations, we rotate an entire class towards an unused direction while preserving its within-class geometry. The measured objective improvements agree with our exact theoretical formula, with a maximum absolute error of 3.19×10−133.19\times10^{-13}. Independent automatic differentiation further confirms the predicted positive curvature. These results provide quantitative verification of the theoretical escape direction, beyond simply observing class orthogonality.

4. Dimension-limited structure and optimisation limitations — Figures 5–7

Figure 5 examines three additional theoretical predictions: Theorem 3, which characterises rank-one class representations and overall isotropy in the full problem under the condition max⁡kpk≤1/d\max_k p_k\leq 1/d; Theorem 4, which characterises the optimal spectrum for imbalanced classes within the rank-one family; and Proposition 1, which establishes a sharp overlap bound under fixed higher-rank class geometry. These experiments illustrate how the optimal representation structure depends on the representation dimension, class balance, and feasible set.

Figures 6(c) and 7 further examine the limitations established by Proposition 2 and Theorem 5. Specifically, they illustrate that local stability does not necessarily guarantee global optimality, and that a second-order Hessian test can miss a higher-order improving direction. These findings complement our positive theoretical guarantees by revealing important limitations of the MCR² optimisation landscape.

AuthorsOpenReview ↗

Response to Reviewer 12r1, Part 3

Weakness 3: The practical implications of the theoretical results for MCR² are insufficiently clear.

We thank the reviewer for encouraging us to clarify the practical significance of our findings. We acknowledge that our primary contribution is to advance the theoretical understanding of the MCR² objective, rather than to propose a new training strategy or neural architecture. Nevertheless, we believe that understanding when MCR² encourages desirable representation structures, and when these properties may fail, can provide valuable insights for future practical developments. Below, we explain these implications and their limitations.

1. What new understanding does our analysis provide?

Our theoretical results reveal several important properties of the MCR² objective:

  • Class separation does not require global optimality (Theorem 1 and Corollary 1).
  • Global optimality does not always guarantee complete class separation, particularly when representation dimensions are limited (Theorem 3).
  • Favourable representation geometry and second-order stationarity do not necessarily guarantee global optimality (Proposition 2 and Theorem 5).

Together, these results clarify what MCR² can guarantee, what it cannot guarantee, and why seemingly desirable representation structures do not always imply successful optimisation.

2. How can these findings guide practical research?

Although our results are primarily theoretical, they suggest several directions for improving the design, training, and evaluation of MCR²-based learning methods.

First, our analysis highlights the importance of representation dimension and class ranks, which may inform the design of representation spaces. Second, our counterexamples suggest that class separation alone is insufficient to assess optimisation success, motivating more comprehensive evaluation of learned representations. Finally, the existence of degenerate saddles suggests potential directions for developing more effective optimisation methods.

We emphasise that these are theoretically motivated directions rather than experimentally validated improvements. Translating these insights into practical methods requires further investigation.

3. What are the implications for objective and architecture design?

As discussed in our response to Weakness 1, rate-reduction principles have inspired neural architectures such as ReduNet and CRATE. Their development highlights the practical value of understanding the learning objectives underlying neural-network design.

In this context, our analysis provides a theoretical foundation for investigating and improving related learning objectives and architectures. For example, understanding which properties of MCR² are preserved or lost under different modifications may help guide the design of future methods.

However, extending these theoretical guarantees to modified objectives and parameterised neural networks, and demonstrating corresponding practical improvements, remains future work.

To clarify these implications, we have provided a detailed discussion in Appendix L and revised Section 6 (Conclusion) to distinguish our established theoretical contributions from their potential practical impact. The discussion is placed in the appendix to keep the main text concise. These revisions are highlighted in red.

Question 1: Does “class rank” refer to the rank of the features belonging to a single class?

Yes, that is correct. Let Zk∈Rd×mkZ_k\in\mathbb{R}^{d\times m_k} denote the feature matrix of class kk, whose columns are the feature vectors of its mkm_k samples. The class rank is defined as rk=rank⁡(Zk),r_k=\operatorname{rank}(Z_k), which measures the dimension of the linear subspace spanned by the features of class kk.

For example, a class may contain many samples, but its features may span only a four-dimensional subspace, giving rk=4r_k=4. Therefore, the condition ∑krk≤d\sum_k r_k\leq d depends on the dimensions of the class subspaces, rather than the number of samples. This condition ensures that the representation space has sufficient dimensions to accommodate mutually orthogonal class subspaces.

Reviewer_3WfxOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the optimization landscape of the Maximal Coding Rate Reduction objective under sample-wise unit-norm constraints. The authors argue that previous landscape analyses of a Frobenius-regularized variant do not directly characterize the original objective, and they develop a geometric analysis based on class-wise rotations, rank constraints, and spectral properties. The main results include characterizations of local and global optima under different dimensionality and class-balance conditions, an analysis of rank-one and higher-rank class structures, and counterexamples demonstrating suboptimal local maxima and the limitations of second-order optimality tests. The paper conducts synthetic experiments, numerical checks of predicted geometric properties, an experiment on class-subspace overlap using the digits dataset, and a comparison on CIFAR-10.

Critical strengths
  1. The distinction between the sample-wise unit-norm MCR² objective and its Frobenius-regularized variant is meaningful. Analyzing the original objective avoids assuming that conclusions established for a different feasible set automatically transfer to the formulation used in practice.

  2. The paper identifies important limitations of local optimization. The examples show that a local maximum may not be globally optimal and that a second-order Hessian test may miss an improving direction.

  3. The theoretical analysis covers multiple regimes. The paper studies both sufficient and limited representation dimensions. Explicit conditions and equality cases make the results informative.

  4. The numerical checks support the theoretical analysis. Experiments examine isotropy, class-subspace overlap, and stationary-point behavior. High-precision calculations are especially useful for analyzing the degenerate example, where standard second-order tests may be inconclusive.

Critical weaknesses
  1. The connection to practical deep-learning optimization remains limited. The principal results concern the geometric objective over feature matrices with unit-norm columns. In an actual neural network, these features are generated by a parameterized encoder, and the optimization dynamics are affected by the architecture, parameterization, data distribution, and optimizer. The paper does not establish that the characterized landscape directly predicts the behavior of end-to-end network training.

  2. The experimental evidence is primarily geometric rather than task-oriented. The synthetic experiments do not establish that the theoretical guarantees improve classification accuracy, robustness, or training reliability in realistic representation-learning settings. The digits experiment study feature geometry with fixed within-class Gram matrices rather than training and evaluating a complete encoder.

  3. The CIFAR-10 comparison needs careful interpretation. The reported accuracy difference between MCR² and the Frobenius-regularized variant is substantial, but this comparison alone does not isolate the effect of the optimization landscape. The paper should make the training protocols, hyperparameter selection, normalization choices, and computational budgets sufficiently explicit to support a fair comparison. More importantly, poor performance of one regularized formulation does not by itself establish that prior landscape results for that formulation have limited value in every setting. Besides, experiments on larger and more diverse datasets, as well as modern deep representation learning models, would help establish whether the theoretical findings translate to modern AI applications.

Questions for authors
  1. Can the authors provide additional evidence that the theoretical landscape properties predict the behavior of end-to-end neural-network training, rather than only optimization directly over the feature matrix?

  2. For the CIFAR-10 comparison, can the authors clarify the full experimental protocol? Are the results robust to reasonable changes in these choices?

  3. Can the authors discuss whether the suboptimal local maximum and the degenerate stationary point are likely to arise in higher-dimensional or realistic representation-learning problems, or whether they are primarily illustrative counterexamples?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Wfx, Part 1

We sincerely thank the reviewer for the detailed comments and positive assessment of our theoretical contributions and numerical validation. Below, we address your concerns point by point. For clarity and readability, we have slightly rephrased your comments while preserving their original meaning.

Weakness 1: The analysis focuses primarily on representation geometry, leaving its connection to practical deep-learning optimisation unclear.

We would like to clarify that our work does not study representation geometry for its own sake. Instead, our goal is to understand the optimisation landscape of the MCR² learning objective used to train neural networks. In other words, the geometric structure of learned representations is a consequence of optimising this objective, rather than a prescribed constraint.

Specifically, MCR² measures the difference between the coding rate of the overall representation and the weighted coding rates within individual classes. It encourages within-class compression while promoting the overall expansion of representations across classes. In practice, these representations are produced by an encoder, Z=fθ(X)Z=f_\theta(X), and training updates the network parameters θ\theta to maximise this objective. Under the conditions established in our analysis, orthogonal class subspaces emerge naturally from optimising MCR², rather than being explicitly imposed.

Therefore, our characterisation of the MCR² optimisation landscape provides a theoretical explanation for why end-to-end training with this objective can favour orthogonal class representations.

Below, we want to briefly discuss the (possible) broader significance of our MCR² analysis for practical deep-learning model design.

As discussed in our paper, modern deep-learning architectures have largely been developed through empirical design, often lacking clear mathematical explanations for why particular architectural choices work or principled guidance on how individual components should be designed. MCR² offers a promising alternative by providing a mathematical perspective for neural network design. Early work explored this idea through white-box convolutional networks, while more recent advances have demonstrated its practical potential in transformer architectures. For example, CRATE [1] derives interpretable transformer blocks from the principle of sparse rate reduction and achieves competitive performance with Vision Transformers (ViTs) on large-scale image classification tasks. CRATE-α [2] further improves the scalability of this approach. More recently, extensions of coding-rate reduction have inspired new attention mechanisms, including linear-complexity attention [4], which can substantially reduce the computational and memory costs of transformers.

These promising practical advances naturally raise a fundamental question: Why can such diverse variants of MCR² achieve strong practical performance, despite their substantially different formulations and approximations? The underlying MCR² objective arguably must possess certain favourable properties that can be preserved even under different and even relatively relaxed approximations. This is a central motivation for our theoretical investigation of the MCR² optimisation landscape. We believe that a deeper understanding of these properties can guide the development of better learning objectives and neural architectures. In the long term, our ambition is to (possibly) help develop white-box transformers that can match or even surpass highly engineered architectures in both performance and computational efficiency.

[1] Yu et al. White-Box Transformers via Sparse Rate Reduction. NeurIPS, 2023.

[2] Yang et al. Scaling White-Box Transformers for Vision. NeurIPS, 2024.

[3] Wang et al. Attention-Only Transformers via Unrolled Subspace Denoising. ICML, 2025.

[4] Wu et al. Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction. ICLR, 2025.

Weakness 2: The experiments primarily examine representation geometry, leaving the practical implications for downstream tasks unclear.

We would like to clarify that MCR² itself is motivated by practical classification tasks, aiming to compress representations of samples from the same class while separating representations of samples from different classes. The MCR² objective encourages this by reducing within-class coding rates while increasing the overall coding rate. Therefore, the resulting representation geometry is directly relevant to classification, rather than being merely a geometric property of theoretical interest.

AuthorsOpenReview ↗

Response to Reviewer 3Wfx, Part 2

Weakness 3: The CIFAR-10 comparison requires clearer experimental settings and stronger evidence of a fair comparison.

To address this concern, we have now independently trained MCR² and regularised MCR² under two matched experimental protocols, each with five random seeds. Both use ResNet-18 with 128-dimensional representations and ε2=0.5\varepsilon^2=0.5. These new experiments resolve the dimensional mismatch and single-run limitation of our original comparison.

1. Two matched training protocols

We consider two different training settings to make the comparison more comprehensive:

  • SGD for 500 epochs: We use a batch size of 1000, an initial learning rate of 0.01, momentum of 0.9, and weight decay of 5×10−45\times10^{-4}. The learning rate decreases tenfold after epochs 200 and 400. Training uses all 50,000 CIFAR-10 training images, without a validation split.
  • Adam for 100 epochs: To reflect a more practical training setting, we also use Adam with a shorter training budget and a separate validation set. Specifically, we use balanced batches of 400, an initial learning rate of 10−310^{-3}, weight decay of 5×10−45\times10^{-4}, and a cosine learning rate schedule decreasing to 10−510^{-5}. We use 45,000 images for training and 5,000 for validation.

To ensure a fair comparison, both formulations share identical initial parameters, minibatch orders, augmentation seeds, and training budgets within each protocol and seed. Both are evaluated on the official 10,000-image test set. Importantly, we report all tested Frobenius coefficients using the final checkpoint, without selecting the best coefficient or training epoch.

2. Results across both protocols

The following table reports the nearest-subspace test accuracy across five random seeds.

FormulationSGD, 500 epochsAdam, 100 epochs
MCR²92.66 ± 0.1486.43 ± 0.41
Regularised MCR², λ=10−2\lambda=10^{-2}N/A: 5/5 NaN17.69 ± 2.81
λ=10−3\lambda=10^{-3}13.98 ± 1.3218.22 ± 2.23
λ=10−4\lambda=10^{-4}13.24 ± 1.3819.92 ± 2.54
λ=10−5\lambda=10^{-5}13.48 ± 0.8616.01 ± 1.18
λ=10−6\lambda=10^{-6}13.28 ± 2.19†18.07 ± 1.92

Results are reported as mean ± sample standard deviation (%). †For SGD with λ=10−6\lambda=10^{-6}, the statistics are calculated over four successful seeds, as one run encountered a NaN loss. All other numerical results are based on five seeds. Failed runs are reported separately rather than assigned zero accuracy.

For nearest-subspace classification, we use unit-normalised features for MCR² and raw features for regularised MCR². To further check whether the results depend on the choice of classifier, we also evaluate a linear Softmax classifier with fixed C=1C=1 under SGD and a Ridge classifier under Adam, both showing the same performance gap. Only the downstream Ridge regularisation coefficient is selected using validation data; this does not affect encoder training or the selection of the Frobenius coefficient. Full experimental details and supplementary results are provided in Appendices J and K.

3. Implications for our analysis

The results consistently show a substantial performance gap between MCR² and regularised MCR² across both training protocols and all successfully completed runs, including those with the smaller coefficients λ=10−5\lambda=10^{-5} and 10−610^{-6}. This suggests that the observed gap is not simply due to a particular random initialisation, training protocol, or choice of classifier.

More importantly, these experiments show that replacing sample-wise normalisation with a Frobenius penalty can lead to substantially different practical performance, even under otherwise identical training settings. This further motivates our decision to analyse the sample-wise normalised MCR² objective directly.

Finally, we clarify that our theoretical conclusions are established independently through the landscape analysis, including counterexamples demonstrating suboptimal local maxima and degenerate saddles.

AuthorsOpenReview ↗

Response to Reviewer 3Wfx, Part 3

Question 1: Can you provide additional evidence connecting the theoretical landscape properties to end-to-end neural-network training?

To address this concern, we have conducted additional end-to-end CIFAR-10 experiments using ResNet-18 under two matched training protocols: SGD for 500 epochs and Adam for 100 epochs, each with five random seeds. Unlike our theoretical analysis over feature representations, these experiments directly optimise neural-network parameters and evaluate the learned representations on unseen test images.

As detailed in our responses to Weaknesses 2 and 3, the substantial performance gap between MCR² and regularised MCR² consistently persists across both training protocols and different downstream classifiers. These results provide additional empirical evidence that the distinction between the two objectives has important practical implications for end-to-end neural-network training.

More importantly, as explained in our response to Weakness 1, our theoretical analysis characterises the representation structures favoured by the MCR² objective and the conditions under which they emerge. The new experiments complement these theoretical findings by demonstrating that the choice of objective can substantially affect the quality and generalisation of learned representations in practice.

Question 2: Can you clarify the full CIFAR-10 protocol and whether the results are robust to reasonable changes in experimental choices?

We have now provided complete experimental protocols and additional results in Appendices J and K, including the network architecture, representation dimension, dataset splits, optimisation settings, learning-rate schedules, training budgets, feature normalisation, and downstream classifiers. These details are also summarised in our response to Weakness 3.

To ensure a fair comparison, both MCR² and regularised MCR² are independently trained under identical settings within each protocol and random seed. We also evaluate all five Frobenius regularisation coefficients, ranging from 10−210^{-2} to 10−610^{-6}, and report results from the final checkpoints without selecting the best coefficient or training epoch. Any numerical failures are explicitly reported rather than excluded without explanation.

The results consistently show a substantial performance gap across both SGD and Adam, five random seeds, different regularisation coefficients, and multiple downstream classifiers. This demonstrates that the observed difference is not limited to a particular initialisation, optimisation protocol, or classifier. The same experiments are also discussed in our response to Reviewer JRM6, Weakness 1.

Question 3: Are the suboptimal local maximum and degenerate stationary point likely to arise in higher-dimensional or realistic representation-learning problems, or are they primarily illustrative counterexamples?

We would like to clarify that these constructions are rigorous analytical counterexamples rather than empirical claims about how frequently such configurations occur during neural-network training. Their purpose is to identify fundamental limitations of general optimisation guarantees for the sample-wise normalised MCR² objective.

Specifically, the suboptimal local maximum demonstrates that class orthogonality alone does not guarantee global optimality, while the degenerate saddle shows that second-order conditions may fail to detect an improving direction. As further discussed in our response to Reviewer JRM6, Weakness 2, these counterexamples establish that the MCR² landscape is not globally benign, even in a simple, explicitly constructed setting.

Of course, the existence of these counterexamples does not necessarily imply that they frequently arise in higher-dimensional representations or practical neural-network training. Whether similar configurations retain their local properties in higher dimensions, and how often optimisation algorithms approach them, requires further investigation. We will clarify this distinction between theoretical existence and practical prevalence in the revised manuscript.

Reviewer_JRM6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article studies maximal coding-rate reduction under per-sample unit-norm constraints, rather than a Frobenius-regularized relaxation. It characterizes when this objective produces orthogonal class subspaces, tight-frame representations, or overlapping class directions. The authors first show empirically on CIFAR-10 that the Frobenius relaxation fails to preserve orthogonal class blocks and drops test accuracy from 93.70% to 45.57%. Using Riemannian geometry and whole-class orthogonal rotations, they prove that whenever the sum of occupied class ranks is at most the ambient dimension (∑krk≤d\sum_k r_k \le d), any non-orthogonal critical point is a strict saddle, meaning all local maxima achieve orthogonal class separation. When d < K (insufficient dimension), they show that balanced classes collapse to rank-one isotropic tight frames, characterize the full rank-one landscape for imbalanced classes, and provide sharp overlap bounds for fixed higher-rank class geometries. Finally, using a 3-sample analytical counterexample, they prove that the landscape is not globally benign, and it contains suboptimal orthogonal local maxima and degenerate non-strict saddles with fourth-order ascent paths. These findings are verified through extensive numerical optimization on synthetic setups and fixed-Gram orientations of the scikit-learn digits dataset.

Critical strengths
  • The theory addresses the sample-wise normalization used in the implemented MCR² objective and establishes a local separation result. Unlike prior work that relaxed the unit-sphere constraint into a Frobenius penalty, this paper analyzes the exact manifold used in practice (Theorem 1).

  • The paper distinguishes global structure from optimization guarantees, with both exact characterizations and informative counterexamples. By constructing exact 3-sample counterexamples (Proposition 2 and Theorem 5), the authors prove that class orthogonality does not prevent suboptimal within-class rank collapse or degenerate non-strict saddles. Theorems 3 and 4 provide a complete picture of the feature-level landscape.

Critical weaknesses
  • The CIFAR-10 contrast (93.70% vs. 45.57%) relies on single-seed regularized runs and no independently retrained normalized baseline. As the authors acknowledge, it does not establish an accuracy advantage between formulations.

  • The landscape guarantees rely on rank constraints or fixed within-class geometry, leaving unrestricted imbalanced optimization and higher-rank local landscapes unresolved. It seems that broader guarantees cannot be assumed.

  • While Theorem 3 solves the full sample-wise problem for balanced classes (pk≤1/dp_k \le 1/d), Theorem 4 restricts the feasible set to rank-one classes (Zk=uksk⊤Z_k = u_k s_k^\top) to handle unequal class masses. Whether unrestricted sample-wise optimization automatically collapses heavy classes to rank one when pk>1/dp_k > 1/d is not discussed.

Questions for authors
  • In the 900 sufficient-dimensional runs, is the condition ∑krank(Zk)≤d guaranteed by the experimental setup, or does it require rank reduction during optimization? What evidence supports its applicability in the intended feature-learning regime?
  • Your proof states that rotational stationarity implies CkS=SCk, while the numerical validation describes nonorthogonal critical configurations with noncommuting class matrices. Does noncommuting refer to pairs Ck,Cℓ, rather than Ck,S?
  • Is the 93.70% normalized-MCR² result obtained under a protocol directly comparable to the reported regularized runs? If not, will the headline comparison be explicitly presented as non-comparative context rather than evidence of an accuracy advantage?
  • In appendix I, have you tested smaller values (e.g., λ∈{10−5,10−6}\lambda \in \{10^{-5}, 10^{-6}\}) where the penalty scale better matches the coding rate gradient, and does the accuracy gap in Table 1 persist when properly tuned?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JRM6, Part 1

We sincerely thank the reviewer for the constructive feedback and positive assessment of the paper’s clarity and theoretical contributions. Below, we address your concerns point by point. For clarity and readability, we have slightly rephrased your comments while preserving their original meaning.

Weakness 1: The CIFAR-10 comparison lacks an independently retrained MCR² baseline and relies on a single run.

First, we would like to clarify that in our initial experiments, regularised MCR² almost completely failed under the original MCR² training settings reported in [2], using 128-dimensional representations, achieving only around 13% test accuracy. This is actually not surprising, as we will explain below. We later found that increasing the representation dimension to 512 improved its performance to approximately 45%, although it still fell substantially behind MCR². However, we reported this result in the original manuscript without clearly explaining the difference in representation dimensions, which may have caused confusion. We apologise for this lack of clarity and have now addressed the issue through additional experiments under strictly matched settings, as detailed below.

We have now independently trained MCR² and regularised MCR² under two matched experimental protocols, each with five random seeds. Both comparisons use 128-dimensional representations. The complete experimental details and results are provided in Appendices J and K.

1. Five-seed comparison using SGD for 500 epochs

We first compare both formulations using ResNet-18, 128-dimensional representations, ε2=0.5\varepsilon^2=0.5, and SGD for 500 epochs. Both use a batch size of 1000, an initial learning rate of 0.01, momentum of 0.9, and weight decay of 5×10−45\times10^{-4}. The learning rate is reduced by a factor of ten after epochs 200 and 400. We train on all 50,000 CIFAR-10 training images and evaluate on the official 10,000-image test set, without a validation split. For each seed, both formulations use identical initial parameters, minibatch orders, and augmentation seeds.

We evaluate the final epoch-500 checkpoint using nearest-subspace classification. For a broader assessment, we also report results using a linear Softmax classifier with a fixed inverse regularisation parameter C=1C=1. Both classifiers use unit-normalised features for MCR² and raw features for regularised MCR². We report all tested Frobenius regularisation coefficients without selecting the best-performing one.

FormulationNearest-subspace accuracy (%)Softmax accuracy (%)NaN runs
MCR²92.66 ± 0.1493.30 ± 0.130/5
Regularised MCR², λ=10−2\lambda=10^{-2}N/AN/A5/5
λ=10−3\lambda=10^{-3}13.98 ± 1.3216.20 ± 0.950/5
λ=10−4\lambda=10^{-4}13.24 ± 1.3816.03 ± 0.480/5
λ=10−5\lambda=10^{-5}13.48 ± 0.8615.53 ± 0.770/5
λ=10−6\lambda=10^{-6}13.28 ± 2.1914.29 ± 2.571/5

The results are reported as mean ± sample standard deviation over successful seeds. For λ=10−6\lambda=10^{-6}, the statistics are calculated over four seeds. All five runs with λ=10−2\lambda=10^{-2} encountered NaN losses at epoch 1, while one run with λ=10−6\lambda=10^{-6} encountered a NaN loss at epoch 26. We report these failures separately rather than assigning them zero accuracy.

2. Additional five-seed comparison using Adam

Considering that in practice, Adam is often preferred for its faster convergence, which can reduce the number of training epochs required. Moreover, real-world practical deep learning experiments typically use a validation set for model selection. To better reflect these practical settings, we conducted an additional five-seed experiment using Adam with a shorter training budget of 100 epochs and a separate validation set. Both formulations again use 128-dimensional representations and ε2=0.5\varepsilon^2=0.5, with identical training settings to ensure a fair comparison.

We use 45,000 training images, 5,000 validation images, and the official 10,000-image test set. All tested Frobenius coefficients are reported using the final epoch-100 checkpoint. In addition to nearest-subspace classification, we evaluate a Ridge classifier, whose regularisation coefficient is selected on the validation set using the same candidate grid for both formulations. Importantly, this selection applies only to the downstream Ridge classifier; it does not affect encoder training or involve selecting the Frobenius coefficient.

FormulationNearest-subspace accuracy (%)Ridge accuracy (%)
MCR²86.43 ± 0.4186.79 ± 0.37
Regularised MCR², λ=10−2\lambda=10^{-2}17.69 ± 2.8120.84 ± 1.92
λ=10−3\lambda=10^{-3}18.22 ± 2.2319.49 ± 1.99
λ=10−4\lambda=10^{-4}19.92 ± 2.5421.33 ± 2.15
λ=10−5\lambda=10^{-5}16.01 ± 1.1819.33 ± 2.26
λ=10−6\lambda=10^{-6}18.07 ± 1.9221.49 ± 1.79
AuthorsOpenReview ↗

Response to Reviewer JRM6, Part 2

The results are reported as mean ± sample standard deviation across all five seeds. Ridge classification uses normalised features for both formulations, while nearest-subspace classification uses unit-normalised features for MCR² and raw features for regularised MCR². Using raw regularised features for Ridge produces similar mean accuracies, ranging from 19.32% to 21.54%.

The results are consistent across both protocols: regularised MCR² performs substantially worse than MCR², even with the smaller regularisation coefficients 10−510^{-5} and 10−610^{-6} suggested by the reviewer.

3. Why these findings are consistent with previous landscape results

We emphasise that a favourable optimisation landscape over feature matrices does not necessarily guarantee strong generalisation when training a neural network. Wang et al. [1] consider a relatively simple experimental setting involving balanced tiny CIFAR-10 subsets of only 1,000–2,500 images, a two-layer MLP, and full-batch Adam training for 4000 epochs. Their experiments demonstrate near-optimal training objective values and approximately orthogonal training representations, but do not report test-set classification accuracy or reflect practical large-scale training scenarios. In contrast, our experiments directly evaluate the generalisation of learned representations to unseen images under larger-scale minibatch training.

Furthermore, normalising feature representations and penalising their magnitude are fundamentally different operations. The original MCR² paper [2] observes that coding-rate reduction is monotonic with respect to the overall feature scale and therefore introduces class-wise energy constraints or sample-wise unit normalisation to control representation magnitude.

In contrast, a Frobenius penalty discourages large feature magnitudes without explicitly enforcing these constraints. Instead, it introduces a trade-off between coding-rate reduction and representation energy, controlled by λ\lambda. Although this remains a meaningful optimisation objective, theoretical guarantees for the regularised formulation do not automatically extend to sample-wise normalised MCR².

Overall, our new five-seed experiments establish a substantial and consistent performance gap under two matched protocols, while eliminating the dimensional mismatch and single-run limitations of the original comparison. These findings further motivate our direct theoretical analysis of the sample-wise normalised MCR² objective.

[1] Wang et al. (2024). A Global Geometric Analysis of Maximal Coding Rate Reduction. ICML.

[2] Yu et al. (2020). Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. NeurIPS.

Weakness 2: The guarantees depend on rank conditions or fixed within-class geometry, leaving unrestricted imbalanced optimisation and higher-rank local landscapes unresolved.

Thanks for the insightful comment. We agree that our theoretical results do not fully characterise these more general settings. BFor imbalanced data, class-balanced resampling provides a practical way to obtain a balanced training setting in which our theoretical results apply. However, we acknowledge that this approach changes the effective class weights rather than theoretically resolving the original imbalanced objective. Similarly, a complete characterisation of higher-rank local landscapes remains a challenging open problem.

Weakness 3: Does unrestricted optimisation also produce rank-one heavy classes when pk>1/dp_k>1/d?

Thank you for raising this important distinction. Rank one is a conclusion of Theorem 3, but an assumption of Theorem 4. When pk>1/dp_k>1/d, a rank-one class contributes an eigenvalue mk>m/dm_k>m/d, making total isotropy impossible. Consequently, the two bounds used in Theorem 3 cannot be achieved simultaneously, so its argument does not directly extend to this setting. Theorem 4 therefore characterises the optimisation landscape within the rank-one family, rather than establishing that unrestricted optimisation necessarily produces rank-one heavy classes. Whether these classes remain rank one or benefit from higher-rank representations requires further investigation. We will clarify this limitation in the revised manuscript.

Question 1.1: Is the rank condition guaranteed in all 900 runs?

Yes, the rank condition is guaranteed by construction in all 900 runs. Specifically, we set d=m=∑kmkd=m=\sum_k m_k, which ensures that

This holds throughout optimisation. Each sample is freely optimised on the unit sphere, without imposing any additional rank constraints or rank reduction.

AuthorsOpenReview ↗

Response to Reviewer JRM6, Part 3

Question 1.2: Is this rank condition practically relevant to deep learning?

Yes, the rank condition can naturally be accommodated by choosing a sufficiently large representation space relative to the occupied class ranks. This is consistent with the motivation of MCR²: mapping high-dimensional data into low-dimensional class subspaces while encouraging separation between classes. Both the original MCR² paper [1] and the ReduNet paper [2] illustrate this structure on CIFAR-10, where learned class representations concentrate around roughly a dozen principal directions within a 128-dimensional feature space. These observations motivate providing enough ambient dimensions to accommodate the class subspaces simultaneously, making the rank condition practically relevant. The observed spectral concentration supports this motivation, although it does not by itself establish the exact rank condition.

[1] Yu et al. Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. NeurIPS, 2020.

[2] Chan et al. ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction. JMLR, 2022.

Question 2: Does “noncommuting” refer to CkC_k and CℓC_\ell, rather than CkC_k and SS?

Yes, “noncommuting” refers to pairs of class matrices CkC_k and CℓC_\ell, rather than CkC_k and SS. In our constructed critical configurations, SS is a scalar multiple of the identity on the occupied subspace. Therefore, every CkC_k commutes with SS, although different class matrices need not commute with each other. This is fully consistent with the rotational-stationarity condition. We will clarify the wording as “class matrices that do not commute pairwise” in the revised manuscript to avoid ambiguity.

Question 3: Was the 93.70% MCR² result obtained under a protocol directly comparable to the regularised runs?

The original comparison was not fully matched: MCR² used 128-dimensional representations, whereas the reported regularised MCR² result used 512 dimensions. We apologise for not making this distinction clear.

To address this, we have now independently trained both formulations under two strictly matched protocols—SGD for 500 epochs and Adam for 100 epochs—each across five random seeds, using 128-dimensional representations and ε2=0.5\varepsilon^2=0.5. All shared training and evaluation settings are matched within each protocol, and all tested regularisation coefficients and numerical failures are reported. The substantial performance gap persists under both protocols. Full details appear in Appendices J and K and our response to Weakness 1.

Question 4: Have smaller coefficients λ=10−5\lambda=10^{-5} and 10−610^{-6} been tested, and does the performance gap persist?

Yes, we have tested both coefficients in our additional matched comparison across five random seeds. The nearest-subspace test accuracies (mean ± sample standard deviation) are 16.01 ± 1.18% for λ=10−5\lambda=10^{-5} and 18.07 ± 1.92% for λ=10−6\lambda=10^{-6}, compared with 86.43 ± 0.41% for MCR².

These results confirm that the substantial performance gap persists under the matched 100-epoch Adam protocol, even with the smaller coefficients suggested by the reviewer. We report all tested values from λ=10−2\lambda=10^{-2} to 10−610^{-6} in Appendix J. While these results provide stronger empirical evidence of the performance gap, they do not establish the best achievable performance of the regularised formulation under all possible training configurations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We sincerely thank all reviewers for their detailed and constructive feedback, which has helped us clarify our contributions and strengthen the paper. We particularly appreciate their recognition of:

  • The importance of directly analysing the MCR² objective itself.
  • Our theoretical guarantees on class separation and dimension-dependent optimal structures.
  • The counterexamples revealing fundamental limitations of the optimisation landscape.
  • The numerical experiments supporting our theoretical findings.

Below, we summarise the main clarifications and revisions. All revised passages and updated table and figure captions are highlighted in red in the revised manuscript.

  • Concern 1: The CIFAR-10 comparison requires stronger evidence and matched experimental settings. (Reviewers JRM6 and 3Wfx)

    We independently retrained MCR² and regularised MCR² under two matched protocols—SGD for 500 epochs and Adam for 100 epochs—each with five random seeds (Appendices J and K). We provided the fully reproducible code in the supplementary material.

  • Concern 2: The scope and limitations of the theoretical guarantees need clarification. (Reviewers JRM6 and 3Wfx)

    We have clarified the roles of the class-rank condition, class balance, and fixed within-class geometry in our theoretical results. We also distinguish rank one as a conclusion of Theorem 3 from its role as an assumption in Theorem 4. We further clarify that our counterexamples establish fundamental limitations of general optimisation guarantees.

  • Concern 3: The practical significance of the theoretical results needs clarification. (Reviewers 3Wfx and 12r1)

    We have clarified that our primary contribution is to advance the theoretical understanding of MCR², while our findings also suggest potential directions for representation dimension selection, optimisation diagnostics, and objective or architecture design (Appendix L).

  • Concern 4: The novelty relative to the broader representation-learning literature needs clarification. (Reviewer 12r1)

    We have added a dedicated Related Work section (Appendix A), providing concise comparisons with duality, VICReg, and alignment–uniformity analyses, alongside studies of spectral measures of representation quality. Simple mathematical examples are also provided (Appendix F.4).

  • Concern 5: The connection between experiments and theoretical results is unclear. (Reviewer 12r1)

    We have revised Section 5 and the captions of Figures 3–7 to explicitly connect each experiment to its corresponding theoretical results, assumptions, and implications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.