The Optimisation Landscape of Maximal Coding Rate Reduction

Understanding how neural networks organise learned representations is important for improving their generalisation, robustness, interpretability, and ultimately safety. The recent proposed framework Maximal Coding Rate Reduction (MCR${}^2$) offers an information-theoretic perspective: it aims to capture the underlying structure of each class while arranging different classes in orthogonal subspaces. Despite promising applications, the optimisation landscape of MCR${}^2$ remains poorly understood. Due to the requirement of every feature should have the same scale of information, the landscape of MCR${}^2$ is difficult to analyse. Previous work has therefore studied a relaxed Frobenius-regularised variant. We show that this variant can perform poorly in practice, limiting the practical guidance offered by its landscape analysis. In this work, we characterise key features of local and global optima of MCR${}^2$ itself and analyse other critical points. In particular, every local optimum whose occupied class ranks sum to at most the representation dimension separates the classes orthogonally. We also describe global solutions in several important settings, show how classes may overlap when feature space is limited, and identify configurations where local optimisation can stall or second-order tests are inconclusive. These results explain when MCR${}^2$ favours discriminative representations under first-order optimisation and where its guarantees stop. Extensive experiments with synthetic data and real image datasets support the theoretical analysis.