Wrapped Gaussian Mixture on the Low-Rank Matrix Manifold

Low-rank matrices are widely used in parameter-efficient machine learning models, such as LoRA-based LLM fine-tuning and latent-factor recommender systems. To characterize their generalizability, we commonly use PAC-Bayes bounds, and choose Gaussian priors and posteriors to make the KL divergence practically computable. However, we identify a fundamental issue: if assigning Gaussian distributions over rank-$k$ matrices, then no Gaussian can have its support fully cover $\mathcal{M}\_k$ – the set of all $m\times n$ matrices of rank-$k$, also a smooth embedded manifold of dimension $(m+n-k)k$. This partial covering can make the posterior not absolutely continuous with respect to the prior, resulting in infinite KL divergence. To address this issue, we first propose a new statistical tool for characterizing low-rank matrices – a wrapped Gaussian mixture that covers $\mathcal{M}\_k$, constructed based on differential-geometric analysis. We then show that this mixture admits a practically computable KL divergence bound in closed-form. Finally, we apply the mixture to establish a PAC-Bayes bound for stochastic LoRA neural networks. Empirical results show that the bound is nonvacuous and captures LoRA generalization behavior, suggesting the utility of the mixture for analyzing low-rank model performance.