- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Maximum coding rate reduction(MCR2) aims to capture the underlying structure of each class by arranging classes in different subspaces. However, there is a lack of understanding as to the optimization dynamics that lead to sufficiently good representations. Other work has introduced relaxed assumptions that lead to poor performance. The authors present an interesting analysis of these dynamics and find a variety of dimension based considerations. These include settings such as global vs. local optima, settings with limited feature spaces, and when MCR2 doesn't lead to more discriminative representations.
Critical strengths
-
These are very interesting settings of analysis within the context of representation learning literature. For example, finding local optima where the same class separation occurs is especially improtant in settings like self supervised learning where obtaining global optima is less likely due to the lack of structured task information. This idea of representation size of the space is also interesting because it under scores a relevant consideration in constrainted domains with limited diversity or training data that have a difficult time obtaining high ranks. Identifying these settings I think is especially interesting for future work on this topic.
-
The orthogonal class separation result is especially impactful as it suggests that the MCR2 setup can induce additional class separation with limited intervention on the represetnations which is abstracted to a rotation in this work.
-
The conclusions are notably more specific than previous work that has attempted to analyze MCR2 in a theoretical manner: https://arxiv.org/pdf/2406.01909
-
The authors show toy experiments of their theoretical perspectives.
Critical weaknesses
- A lot of the discussion is somewhat similar to work that has appeared within the self supervisd representation learning literature and it is difficult to appreciate the novelty of your analysis without some discussion as to these other works.
For exmaple, in your theorem 1, there was a discussion about a class rotating part of his energy towards an unused direction in order to increase the total rate.
In Theorem 3, there was a discussion of the total coding rate largest when the d eigenvalues are effectively equal. Statements like this are used throughout the paper and it raises questions as to whether any of this analysis is especially important in the context of larger represetnation learning literature. What I mean by this is that this paper is advertising itself as understanding how the optimization landscape of MCR2 works. It is, by definition, constraining its comparisons with MCR2 literature. However, I feel that this is very limiting. Ultimately, MCR2 is a representation learning approach. All representation learning approaches attempt to do something similar to what is described in this paper. Therefore, it is possible to view the results in this paper as off shoots of other representation learning literature. In other words, this paper could potentially be showing that MCR2 functions as other representation learning approaches.
For example, a variety of works have shown that greater rank corresponds to better downstream performance and better rank also corresponds to equality between eigenvalues:
Effective rank: The effective rank: a measure of effective dimensionality
Alpha-REQ: alpha-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay
LiDAR: LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Other works have their own perspective on how samples distribute in a representation space:
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization: Analyzes the behavior of equalizing eigenvalues in the context of covariance regularization
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere: Analyzes the spread of samples in the case of a hypersphere geometry
My point is that I feel that many of the conclusions within the context of MCR2 are different ways of saying the same results from representation learning literature in general.
- It's difficult to understand how the experiments support the theoretical conclusions.
Class separation with sufficient dimension: The interpretation of these figures is lacking. I think its necessary to refer back to which theories or assumptions are proven here to make the paper wholistic. Right now you state what's in here, but I'm really unsure where to look back in the theories to show that some interesting connection back to the theorey exist.
Many samples and real-data class geometry: Is the point here to show class separability? It is unclear what geometry you are trying to support here. My point is that are you trying to show that MCR2 separates classes? The problem with that point is that the original paper already shows this, what is new by this separability in the context of your theory.
Rotation formula check: I understand your theory discusses how simple rotations can fill gaps in the represetnation space, but that doesn't come out here.
A similar problem happens in the other sections.
- In the theoretical discussion, you don't make it clear what the the practical implication of your result suggests about the MCR2 framework. Look at what I said in your strengths section. This is me trying to interpret from your theorems as to what it suggests about what previously wasn't understood before. This problem transfers to your experiments where there are many statements, but no integration with how the results suggests something impactful about the theory and its implication for MCR2. Ideally, what I would prefer is that each theoretical statement goes back to an original not understood problem about MCR2 optimality. Right now, what I see is a discussion of what the behavior should look like such as using pockets of space with unused feature directions, but the implication of this theory doesn't feel impactful at the moment.
I am happy to be told that I am wrong and explained in the rebuttal, but for the moment I have reservations. I can imagine where these points could be impactful, but without that confirmation in writing, its difficult to know.
Questions for authors
- I think you need additional clarification of different terms for me to understand this fully. A class rank is the rank of the features of samples only belonging to a single class?
See some points in the weaknesses. I want to understand why each of the theorems is impactful in the context of MCR2 and how it rethinks the optimization landscape.
5 comments
Response to Reviewer 12r1, Part 2
Weakness 2: The connection between the experiments and the theoretical results is not clear enough.
We have revised Section 5 and the captions of Figures 3–7 to clarify the connections between our experiments and theoretical results. All changes are highlighted in red in the revised manuscript. Below, we summarise these revisions and explain how each experiment relates to our theoretical findings.
1. Class separation with sufficient dimension — Figure 3
This experiment examines Theorem 1 and Corollary 1 on local class separation, together with Theorem 2 on global optimality. We set , ensuring that throughout optimisation. Across all 900 runs, the final representations exhibit nearly orthogonal class subspaces. However, 215 endpoints have objective-value gaps greater than relative to the exact global optimum. This highlights an important distinction: class orthogonality can emerge without achieving the globally optimal within-class rank structure. Thus, the experiment illustrates both the predicted separation and its limitations.
2. Many samples and real-data class geometry — Figure 4
This experiment examines an important implication of our theory: the dimension requirement depends on the sum of class ranks, rather than the total number of samples. Specifically, we use the digits dataset with 1,797 samples and project each class into a four-dimensional subspace, giving a total class rank of 40. We then test ambient dimensions . Notably, even at , the rank condition is satisfied despite having far more samples than representation dimensions.
Importantly, we keep all within-class Gram matrices fixed throughout optimisation. Therefore, class separation arises entirely from changes in the relative orientations of class subspaces, rather than changes in their internal geometry. This directly reflects the class-rotation mechanism underlying our theoretical analysis and demonstrates its applicability to data-derived class structures.
3. Verification of the class-rotation mechanism — Figure 6(a–b)
These experiments numerically verify the class-rotation mechanism established in Theorem 1. Starting from constructed nonorthogonal critical configurations, we rotate an entire class towards an unused direction while preserving its within-class geometry. The measured objective improvements agree with our exact theoretical formula, with a maximum absolute error of . Independent automatic differentiation further confirms the predicted positive curvature. These results provide quantitative verification of the theoretical escape direction, beyond simply observing class orthogonality.
4. Dimension-limited structure and optimisation limitations — Figures 5–7
Figure 5 examines three additional theoretical predictions: Theorem 3, which characterises rank-one class representations and overall isotropy in the full problem under the condition ; Theorem 4, which characterises the optimal spectrum for imbalanced classes within the rank-one family; and Proposition 1, which establishes a sharp overlap bound under fixed higher-rank class geometry. These experiments illustrate how the optimal representation structure depends on the representation dimension, class balance, and feasible set.
Figures 6(c) and 7 further examine the limitations established by Proposition 2 and Theorem 5. Specifically, they illustrate that local stability does not necessarily guarantee global optimality, and that a second-order Hessian test can miss a higher-order improving direction. These findings complement our positive theoretical guarantees by revealing important limitations of the MCR² optimisation landscape.
Response to Reviewer 12r1, Part 3
Weakness 3: The practical implications of the theoretical results for MCR² are insufficiently clear.
We thank the reviewer for encouraging us to clarify the practical significance of our findings. We acknowledge that our primary contribution is to advance the theoretical understanding of the MCR² objective, rather than to propose a new training strategy or neural architecture. Nevertheless, we believe that understanding when MCR² encourages desirable representation structures, and when these properties may fail, can provide valuable insights for future practical developments. Below, we explain these implications and their limitations.
1. What new understanding does our analysis provide?
Our theoretical results reveal several important properties of the MCR² objective:
- Class separation does not require global optimality (Theorem 1 and Corollary 1).
- Global optimality does not always guarantee complete class separation, particularly when representation dimensions are limited (Theorem 3).
- Favourable representation geometry and second-order stationarity do not necessarily guarantee global optimality (Proposition 2 and Theorem 5).
Together, these results clarify what MCR² can guarantee, what it cannot guarantee, and why seemingly desirable representation structures do not always imply successful optimisation.
2. How can these findings guide practical research?
Although our results are primarily theoretical, they suggest several directions for improving the design, training, and evaluation of MCR²-based learning methods.
First, our analysis highlights the importance of representation dimension and class ranks, which may inform the design of representation spaces. Second, our counterexamples suggest that class separation alone is insufficient to assess optimisation success, motivating more comprehensive evaluation of learned representations. Finally, the existence of degenerate saddles suggests potential directions for developing more effective optimisation methods.
We emphasise that these are theoretically motivated directions rather than experimentally validated improvements. Translating these insights into practical methods requires further investigation.
3. What are the implications for objective and architecture design?
As discussed in our response to Weakness 1, rate-reduction principles have inspired neural architectures such as ReduNet and CRATE. Their development highlights the practical value of understanding the learning objectives underlying neural-network design.
In this context, our analysis provides a theoretical foundation for investigating and improving related learning objectives and architectures. For example, understanding which properties of MCR² are preserved or lost under different modifications may help guide the design of future methods.
However, extending these theoretical guarantees to modified objectives and parameterised neural networks, and demonstrating corresponding practical improvements, remains future work.
To clarify these implications, we have provided a detailed discussion in Appendix L and revised Section 6 (Conclusion) to distinguish our established theoretical contributions from their potential practical impact. The discussion is placed in the appendix to keep the main text concise. These revisions are highlighted in red.
Question 1: Does “class rank” refer to the rank of the features belonging to a single class?
Yes, that is correct. Let denote the feature matrix of class , whose columns are the feature vectors of its samples. The class rank is defined as which measures the dimension of the linear subspace spanned by the features of class .
For example, a class may contain many samples, but its features may span only a four-dimensional subspace, giving . Therefore, the condition depends on the dimensions of the class subspaces, rather than the number of samples. This condition ensures that the representation space has sufficient dimensions to accommodate mutually orthogonal class subspaces.
Response to Reviewer 12r1, Part 1
We sincerely thank the reviewer for such thoughtful, detailed, and insightful comments. We also thanks for recognising the value of our class-separation results and dimension-dependent analysis. Below, we address your concerns point by point. To make our responses clearer and easier to follow, we have slightly rephrased your comments while preserving their original meaning.
Weakness 1: The novelty relative to the broader representation-learning literature needs clarification.
1. Why understanding the MCR² objective matters
2. How our work differs from existing representation-learning studies
[1] Yu et al. (2023). White-Box Transformers via Sparse Rate Reduction. NeurIPS.
[2] Roy and Vetterli (2007). The Effective Rank: A Measure of Effective Dimensionality. EUSIPCO.
[3] Agrawal et al. (2022). α-ReQ: Assessing Representation Quality in Self-Supervised Learning by Measuring Eigenspectrum Decay. NeurIPS.
[4] Thilak et al. (2024). LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures. ICLR.
[5] Shwartz-Ziv et al. (2023). An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization. NeurIPS.
[6] Wang and Isola (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML.