acceptodds
Under review as a conference paper at ICLR 2027

Wrapped Gaussian Mixture on the Low-Rank Matrix Manifold

Abstract

Low-rank matrices are widely used in parameter-efficient machine learning models, such as LoRA-based LLM fine-tuning and latent-factor recommender systems. To characterize their generalizability, we commonly use PAC-Bayes bounds, and choose Gaussian priors and posteriors to make the KL divergence practically computable. However, we identify a fundamental issue: if assigning Gaussian distributions over rank- matrices, then no Gaussian can have its support fully cover – the set of all matrices of rank-, also a smooth embedded manifold of dimension . This partial covering can make the posterior not absolutely continuous with respect to the prior, resulting in infinite KL divergence. To address this issue, we first propose a new statistical tool for characterizing low-rank matrices – a wrapped Gaussian mixture that covers , constructed based on differential-geometric analysis. We then show that this mixture admits a practically computable KL divergence bound in closed-form. Finally, we apply the mixture to establish a PAC-Bayes bound for stochastic LoRA neural networks. Empirical results show that the bound is nonvacuous and captures LoRA generalization behavior, suggesting the utility of the mixture for analyzing low-rank model performance.

Primary area:
learning theory
Keywords:
low-rank matrix, smooth manifold, wrapped Gaussian distribution, differential geometry, KL divergence, PAC-Bayes bound

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TH2MOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The concerned paper is interested in pinpointing and solving challenges in computing KL divergences on measures supported on low rank matrices. To this end the paper first argues that there exists gaussian measures supported on the space of rank k matrices whose actual support does not extend to the entire set of k rank matrices but merely a subset. Consequently there can be two measures whose actual supports differ by a set of non-zero measure (i.e. in jargon one is not absolutely continuous with respect to another) and have infinite KL divergence as a consequence.

I would note that this observation is heuristically pretty much obvious. For k=1k=1 for instance you can begin by considering a mm by nn matrix which is rank 1 in a (fixed) "direction" uu i.e. the matrix is given by ux⊤ux^\top. You can then put a gaussian noise on this "centre" which respects the direction uu in the sense that it has no noise component (i.e. no variation) in the the direction orthogonal to uu. This distribution must be entirely supported on rank 1 matrices which are rank 1 in the "direction" uu. Voila, this is pretty much the counterexample or construction in the paper, but written in more formal terms and for general kk in 2.2

To solve this issue the paper suggests that we consider a different family. To this end first consider a gaussian measure on R(m+n−k)k\mathbb{R}^{(m+n-k)k}. Then use a pushformward of a diffeomorphic map to define a measure on the space of rank kk matrices. Unfortunately a single such diffeomorphism might not suffice. So the paper instead uses a collection of diffeomorphisms and takes mixtures between each family of generated measures to get a family of measures which is supported on the entirety of the space of rank k m×nm \times n matrices. This idea in isolation could be of interest elsewhere, but firstly as the paper itself notes that it is not novel. Apart from the references in the paper there is for instance the paper "Density estimation on smooth manifolds with normalizing flows" which also has a similar idea of charts of diffeomorphisms for a similar purpose.

The rest of the paper is concerned with proving that their construction indeed work as advertised and deriving some KL divergence bounds in this new class. Finally some examples of using their construction for specific initializations are discussed.

More importantly it is unclear to me why such a complicated construction is needed to begin with. Running along the counterexample for the case k=1k=1 we could represent every rank 11 matrix as xy⊤xy^{\top} for some xx and some yy. The problem in the counterexample is essentially that one side, say xx, doesn't vary enough. We could solve this by simply taking xx to be distributed from a measure in a family of measures that is expressive enough in Rm\mathbb{R}^{m} and for yy a similar measure which is expressive enough in Rn\mathbb{R}^n (We can take gaussians whose covariance matrix have rank mm and nn respectively for instance). As such the product of these two random variables should indeed be supported on the entirety of the space of rank 11 matrices. There even exist inequalities which would give you the KL divergence between two measures defined in this way with respect to the base component measures (e.g. dKL(UV∣∣XY)<dKL(U∣∣X)+dKL(V∣∣Y)d_{KL}(UV || XY) < d_{KL}(U || X) + d_{KL} (V || Y), where the KL divergence between two random variables should be read as the same between their underlying measures). While product of gaussians is not a gaussian, its unclear what exact benefit a diffeomorphically mapped gaussian chart gives. Gaussianity is simply an assumption that is used in proofs for simplicity of proof; its often not a divine mandate.

Critical strengths

The central construction is somewhat interesting as a purely mathematical curiosity.

Critical weaknesses

Some of the observations seem to be trivial e.g. see summary : "I would note that this observation is heuristically pretty much obvious. For k=1k=1 for instance you can begin by considering a mm by nn matrix which is rank 1 in a (fixed) "direction" uu i.e. the matrix is given by ux⊤ux^\top. You can then put a gaussian noise on this "centre" which respects the direction uu in the sense that it has no noise component (i.e. no variation) in the the direction orthogonal to uu. This distribution must be entirely supported on rank 1 matrices which are rank 1 in the "direction" uu. Voila, this is pretty much the counterexample or construction in the paper, but written in more formal terms and for general kk in 2.2"

Questions for authors

Please clarify why we should use this specific complicated class of measures for initialization when simpler methodologies exist e.g. see the last paragraph of the summary

4 comments

Thank you very much for the insightful review. We hope our response addresses your concerns. Please let us know if you have further questions.

I would note that this observation is heuristically pretty much obvious. For k=1k=1 for instance you can begin by considering a mm by nn matrix which is rank 1 in a (fixed) "direction" uu, i.e. the matrix is given by uv⊤uv^\top. ... this is pretty much the counterexample or construction in the paper.

Thank you for raising this concern. We believe this is a misunderstanding. Our original statement in Lines 151-152 is that: It is impossible to find one Gaussian distribution whose support is exactly Mk\mathcal{M}_k. We think your counterexample does not justify the impossibility of this statement.

You can take a rank-11 matrix uxTux^T, fix xx and assign a Gaussian distribution on uu, such that the support of uxTux^T is a linear subspace within M1\mathcal{M}_1, but does not equal to M1\mathcal{M}_1. We have discussed this in Example 2.3. (b)(c). This counterexample shows ''It is possible to find one rank-1 Gaussian matrix whose support is not M1\mathcal{M}_1'', but it does not show ''It is impossible to find one rank-1 Gaussian matrix whose support is M1\mathcal{M}_1''.

The justification of impossibility is more complex. Our justification in Fact 2.4 is based on the geometric mismatching between Gaussian supports and Mk\mathcal{M}_k, and its formal proof in Appendix B relies heavily on differential geometric analysis.

We are not aware of other approaches that can also prove this impossibility. Our approach based on differential geometry is the simplest one we know so far.

This idea in isolation could be of interest elsewhere, but firstly as the paper itself notes that it is not novel. Apart from the references in the paper there is for instance the paper "Density estimation on smooth manifolds with normalizing flows" which also has a similar idea of charts of diffeomorphisms for a similar purpose.

Thank you for sharing this work. We have added this reference in our revised manuscript.

We think this work provides only a general framework. It shares the same fundamental idea as ours: using multiple charts to estimate the density on a manifold. However, their work neither provides an explicit choice of charts, nor provides explicit diffeomorphisms for each chart. If these math objects are not made explicit, we can hardly conduct a more detailed mathematical analysis.

The novelty of our construction is to make these math objects explicit for Mk\mathcal{M}_k:

  • We explicitly choose Mk∩UIJI∈Ik,J∈Jk\\{\mathcal{M}_k \cap \mathcal{U}_{IJ}\\}_{I \in \mathcal{I}_k, J \in \mathcal{J}_k} as a valid chart covering for Mk\mathcal{M}_k.

  • We explicitly construct the diffeomorphism $\begin{bmatrix} A & B \\ C & D \end{bmatrix} \mapsto \begin{bmatrix} A & B \\ C & D - CA^{-1}B \end{bmatrix}$ for each chart. The benefit of this diffeomorphism is that its Jacobian has determinant 11. In this case, the change of variables $f_{\mathsf{X}}(X) = f_{\mathsf{Y}}(E(X))\left|\det\left(\left.\frac{d E(z)}{dz}\right|_{z = X}\right)\right| $ can be simplified to fX(X)=fY(E(X))f_{\mathsf{X}}(X) = f_{\mathsf{Y}}(E(X)). This greatly facilitates the analysis of density transformation.

Based on these explicit choices of charts and diffeomorphisms, our construction makes the density analysis on Mk\mathcal{M}_k much easier. We hope this explanation clarifies its novelty.

There even exist inequalities which would give you the KL divergence between two measures defined in this way with respect to the base component measures (e.g. dKL(UV∣∣XY)<dKL(U∣∣X)+dKL(V∣∣Y)d_{KL}(UV||XY)< d_{KL}(U||X) + d_{KL}(V||Y) ... ). While product of gaussians is not a gaussian, its unclear what exact benefit a diffeomorphically mapped gaussian chart gives. Gaussianity is simply an assumption that is used in proofs for simplicity of proof.

Thank you for bringing up this simpler solution based on Gaussian product. We think this inequality can be generalized to the rank-kk case. Let U1,U2U_1, U_2 be m×km \times k Gaussian matrices, V1,V2V_1, V_2 be n×kn \times k Gaussian matrices, and suppose U1U_1 is independent of V1V_1, U2U_2 is independent of V2V_2. Let f(U,V)=UVTf(U, V) = UV^T be a function. Then by data processing inequality, \begin{equation*} KL(U_1V_1^T||U_2V_2^T) = KL(f(U_1, V_1)||f(U_2, V_2)) \le KL(U_1, V_1||U_2, V_2) = KL(U_1||U_2) + KL(V_1||V_2) \end{equation*}

We have added this Gaussian Product (GP) as a baseline in our experiment (Section 5 and Appendix C). The results are as follows (pasted from Table 1), where model (l,k)(l, k) means ll hidden layers with rank-kk LoRA.

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
r(u1)r(u_1)0.0670.0620.0660.0640.0620.0650.0630.0650.067
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.0670.0630.0660.0650.0640.0650.0630.650.067
KL29.9941.4348.0235.2544.9360.5735.2142.8663.82
Bound0.2070.2080.2150.2080.2100.2200.2060.2120.224

For comparison, the results of our Wrapped Gaussian Mixture (WGM) is as follows:

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
r(u1)r(u_1)0.1190.1130.1200.1220.1230.1230.1040.2130.148
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.1740.1730.1440.2180.1920.1840.3050.2160.180
KL161.7156.4146.0577.6972.2883.98241.0775.9653.80
Bound0.3630.3600.3280.3800.3510.3470.5130.3770.332

The results show that, although GP yields a tighter bound, for a fixed ll, the bound increases with kk, indicating that a larger rank results in a worser performance of LoRA. This does not match the empirical observations of (Shuttleworth et al., 2025), where LoRA with larger rank tends to perform better, as a larger rank enables LoRA to have more parameters, making it more expressive. This indicates that GP may not be a suitable distribution to characterize the performance of LoRA.

In contrast, WGM results in a bound that decrease with kk, matching the empirical observations of (Shuttleworth et al., 2025). This indicates that WGM can be a better distribution to characterize the performance of LoRA than GP.

Besides, a minor benefit of WGM is that it allows us to calculate the probability assigned on the subsets of Mk\mathcal{M}_k much easier. This calculation is difficult for GP, because f(U,V)=UVTf(U, V) = UV^T is surjective, not bijective; its density transformation from (U,V)(U, V) to UVTUV^T is very complicated. In contrast, ff is bijective for WGM, making the density transformation easier to analyze. This is discussed in Lines 1152-1162.

Reference:

Reece Shuttleworth, et al. Lora vs Full Fine-tuning: An illusion of equivalence. NeurIPS, 2025.

To point 1: Yes you are right, my writing was probably unclear on this point. You are right that this does not show that it is impossible to build a gaussian rank 1 matrix with full support. The point however is that particular construction (being a very common one) informs a heuristic or intuition where we expect it to be not quite possible. As such I agree that rigorously showing impossibility is a technical result. However it is not a substantially interesting one. (e.g. the opposite result i.e. a gaussian with full support would be interesting since it is counter-intuitive)

To point 2: The experiments do suggest (not prove) that gaussian product KL doesnt decay with k. But then again your comparing experiments do not have a substantially smaller KL either (and in many cases is larger). More importantly isnt the regime where we want to apply low rank theory be one where k is small like say k < 100?

Thank you for your response.

For point 1, our thinking is that, if we only show readers a counterexample, they may still think that the geometric mismatching between Gaussian supports and Mk\mathcal{M}_k is evitable within the Gaussian framework. We use this impossibility argument to convince readers that this mismatching is inevitable, and acknowledging them that this issue is fundamental and should be addressed from a geometric perspective. Motivated by this goal, we propose our geometric construction of wrapped Gaussian.

For point 2, we consider this a trade-off. When replacing Gaussian product with wrapped Gaussian mixture, one gets a slightly looser bound caused by a larger KL divergence, but this bound can more accurately reflect the relationship between the LoRA performance and rank.

We have tested the ranks k=8,16,64k = 8, 16, 64, as shown in the model (l,k)(l, \boldsymbol{k}) in the first row of each table. This is within the low rank regime k<100k < 100 as you mentioned.

Reviewer_3uR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper constructs distributions on the manifold Mk\mathcal{M}_k of rank-kk matrices using charts indexed by invertible k×kk\times k minors. Each chart draws Gaussian free blocks A,B,CA,B,C and completes the matrix via CA−1BCA^{-1}B. A mixture over charts defines a wrapped Gaussian mixture on Mk\mathcal{M}_k, whose intractable marginal KL is upper-bounded by an augmented KL that includes the chart index. Using parameter sharing across charts, the paper claims an O(mn)O(mn) closed form and applies it to PAC-Bayes bounds for stochastic LoRA networks on binary FashionMNIST.

Critical strengths
  1. The chart map X↦(A,B,C,D−CA−1B)X\mapsto(A,B,C,D-CA^{-1}B) gives a clean pushforward construction on fixed-rank matrices, and mixing over charts overcomes the coverage limitation of a single chart.

  2. Upper-bounding the marginal KL by augmenting with the chart index is valid in principle. Combined with same-chart KL invariance and parameter sharing, it could yield an efficient bound once the derivation is corrected.

Critical weaknesses
  1. The main KL formula (Eqs. 10–12) is wrong. In each chart (I,J)(I,J), only block DD is zeroed out while A,B,CA,B,C are kept, so entry (r,c)(r,c) is retained whenever r∈Ir\in I or c∈Jc\in J. Under uniform chart weights, its inclusion probability is k(m+n−k)/(mn)k(m+n-k)/(mn), whereas Eq. (11) counts only AA (r∈Ir\in I and c∈Jc\in J) and Eq. (12) obtains k2/(mn)k^2/(mn) (for m=n=2,k=1m=n=2,k=1, the true factor is 3/43/4, not 1/41/4). Eq. (10) also misprints the Gaussian mean term, which should be (u1−u2)j2/(s2)j(u_1-u_2)_j^2/(s_2)_j. If implemented as written, Eq. (12) underestimates the KL by a factor of (m+n−k)/k(m+n-k)/k (>120>120 for every layer at k=8k=8), which makes the (1,8)(1,8) bound in Table 1 vacuous (≈1.03\approx 1.03 instead of 0.5860.586).

  2. The measure-theoretic and density arguments are flawed. Section 2.4 and Appendix E define ρ≪π\rho\ll\pi as supp(ρ)⊂supp(π)\mathrm{supp}(\rho)\subset\mathrm{supp}(\pi) and infer singularity from a single point; however, the minor-zero sets excluded by a chart are lower-dimensional and have zero probability under the other nondegenerate Gaussian charts, so the two single-chart distributions in Appendix E are in fact mutually absolutely continuous (any infinite KL must instead come from non-integrability). Moreover, Eq. (3) uses the unit determinant of the ambient mn×mnmn\times mn Jacobian rather than the induced manifold area factor (for F(a,b,c)=(a,b,c,bc/a)F(a,b,c)=(a,b,c,bc/a), JF=1+(b/a)2+(c/a)2+(bc/a2)2=2J_F=\sqrt{1+(b/a)^2+(c/a)^2+(bc/a^2)^2}=2 at a=b=c=1a=b=c=1).

Questions for authors
  1. Which KL expression and coefficient were implemented in the code? Please provide corrected Eqs. (10)–(12) and updated Tables 1–2, and clarify whether any bounds remain non-vacuous after re-optimizing with the corrected KL.

  2. With respect to which reference measure on Mk\mathcal{M}_k are the densities defined, and how do you revise the change-of-variables formula in Eq. (3) and the infinite-KL argument in Section 2.4 / Appendix E?

6 comments

Thank you for your insightful and constructive suggestions. We have carefully looked into the issues you pointed out and have fixed them cautiously. Please let us know if you have any concerns remain.

The main KL formula (Eqs. 10–12) is wrong. ... Under uniform chart weights, its inclusion probability is (m+n−k)kmn\frac{(m + n - k)k}{mn}. ... If implemented as written, Eq. (12) underestimates the KL by a factor of (m+n−k)k(m + n - k)k (>120>120 for every layer at k=8k = 8), which makes the bound in Table 1 vacuous.

Thank you very much for pointing out this critical miscalculation issue. We have corrected the scalar in Eq. (11), have fixed the code and have rerun all experiments, and have updated the results in Tables 1 and 2. Below are the details.

The scalar of Eq. (11) is corrected in this way: In an m×nm \times n matrix, when choosing kk rows from mm and kk columns from nn, any specific entry will not appear in (m−1k)\binom{m - 1}{k} combinations of rows and (n−1k)\binom{n - 1}{k} combinations of columns; consequently, it will not appear in (m−1k)(n−1k)\binom{m - 1}{k}\binom{n - 1}{k} charts. Hence, the correct scalar of Eq. (11) should be \begin{equation*} \frac{\binom{m}{k}\binom{n}{k} - \binom{m - 1}{k}\binom{n - 1}{k}}{\binom{m}{k}\binom{n}{k}} = \frac{(m + n - k)k}{mn} \end{equation*}

We have rerun the experiments on this corrected scalar, and the changes of Table 1 are shown below

Before

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.4470.3950.2040.4640.4060.3080.5050.4260.278
KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}})29.0952.8352.8236.0789.32554.3927.3869.4433.84
Bound0.5860.5460.3560.6070.5720.5720.6440.5840.421

After

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.1740.1730.1440.2180.1920.1840.3050.2160.180
KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}})161.7156.4146.0577.6972.2883.98241.0775.9653.80
Bound0.3630.3600.3280.3800.3510.3470.5130.3770.332

The results show that, the empirical risk estimation E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] becomes smaller, and the KL divergence KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) becomes larger. The bound remains non-vacuous, and even becomes tighter (i.e., it drops from (0.5,0.6)(0.5, 0.6) to (0.3,0.4)(0.3, 0.4)).

The reason for E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] becoming smaller is that, after the scalar is fixed, we are able to optimize the bound correctly through Eq. (17). Before correction, the obtained posterior fXf_{\mathsf{X}} is far away from the optimal, making E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] large; after correction, the obtained fXf_{\mathsf{X}} closer to the optimal, making E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] smaller. The reduction in E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] is the main reason that results in a tighter Bound.

The KL divergence increases after fixing the scalar, but it does not scale as much as (m+n−k)/k(m + n - k)/k as predicted. The reason is that, in the previous computation of Eq. (12), we took KL(fXC∥∥gXC)=k2mnKL(v1∣∣v2)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) = \frac{k^2}{mn}KL(\mathsf{v}_1||\mathsf{v}_2), where the prior v2\mathsf{v}_2 is fixed and the posterior v1\mathsf{v}_1 is not fixed. When we correct its scalar to (m+n−k)kmn\frac{(m + n - k)k}{mn} and recompute Eq. (17), we get a new posterior v1′{\textsf{v}_1}', which gives KL(v1′∣∣v2)<KL(v1∣∣v2)KL({\mathsf{v}_1}'||\mathsf{v}_2) < KL(\mathsf{v}_1||\mathsf{v}_2). This new posterior makes KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) scales much less than (m+n−k)/k(m + n - k)/k.

The results in Table 2 are also fixed. The main change is that the optimal parameter α\alpha is shifted from 10−310^{-3} to 10−510^{-5}. Please see our revised manuscript for more detail.

We have also uploaded our revised code in the supplemental materials.

Which KL expression and coefficient were implemented in the code?

We implement Eq (10)-(12) in the code. The implementation can be found in bound1.py, bound2.py and bound3.py. We use bound1.py for example:

Lines 24-25: KLDiv(A2, B2, A1, B1): kl = 0.5 * (torch.log(B1 / B2) + (B2 + (A2 - A1)**2) / B1 - 1)

This is the Gaussian KL divergence. A2, B2 represent the posterior.

Line 160: kl = ((1024 - rk)*rk)/(512 * 512) * KLDiv(A1_1, B1_1, self.A2_1, self.B2_1) + KLDiv(A1_2, B1_2, self.A2_2, self.B2_2)

This computes the KL divergence Eq. (12). It sums up the KL divergences of the hidden layer and the output layer, as we consider them independent (see Lines 463-465 in our paper). A1_1, B1_1, A1_2, B1_2 represent the posterior. rk is the rank of the hidden layer.

The measure-theoretic and density arguments are flawed. Section 2.4 and Appendix E define ρ≪π\rho \ll \pi as supp⁡(ρ)⊂supp⁡(π)\operatorname{supp}(\rho) \subset \operatorname{supp}(\pi) and infer singularity from a single point; however, the minor-zero sets excluded by a chart are lower-dimensional and have zero probability under the other nondegenerate Gaussian charts, so the two single-chart distributions in Appendix E are in fact mutually absolutely continuous (any infinite KL must instead come from non-integrability).

Thank you for pointing out this critical issue. We have fixed this flawed measure-theoretic argument in the paper. We have fixed the arguments in Section 2.4 and have added a new Section 3.2 to formally discuss this issue; the original argument in Appendix E is removed.

We note that if ρ\rho and π\pi come from two different charts, ρ≪π\rho \ll \pi is not automatically satisfied, as π\pi can be undefined on supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus \operatorname{supp}(\pi). This is because, they form two different probability spaces (Mk∩Ui,Σρ,ρ)(\mathcal{M}_k\cap \mathcal{U}_i, \Sigma_\rho, \rho) and (Mk∩Uj,Σπ,π)(\mathcal{M}_k\cap \mathcal{U}_j, \Sigma_\pi, \pi). The set supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus \operatorname{supp}(\pi) lies in Mk∩Ujc\mathcal{M}_k\cap \mathcal{U}_j^c, which is outside of the space of π\pi, so π\pi is undefined on it. In this case, KL(ρ∥∥π)KL(\rho\|\|\pi) is undefined.

To address this, we need to extend π\pi from Mk∩Uj\mathcal{M}_k\cap \mathcal{U}_j to the entire manifold Mk\mathcal{M}_k. To make this process rigor, we introduce Assumption 3.3 to define that any measurable subset in Mk∩Ujc\mathcal{M}_k\cap \mathcal{U}_j^c has zero measure under π\pi. Only under this assumption we can say supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus \operatorname{supp}(\pi) is a minor-zero set under π\pi.

If not under this assumption, supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus \operatorname{supp}(\pi) may not be a minor-zero set, as we can extend π\pi to cover Mk\mathcal{M}_k alternatively. For example, we can define a measure π~\tilde{\pi} on Mk\mathcal{M}_k such that π~(A)=0.8π(A)\tilde{\pi}(A) = 0.8\pi(A) for any measurable subset AA in Mk∩Uj\mathcal{M}_k \cap \mathcal{U}_j and π~(Mk∩Ujc)=0.2\tilde{\pi}(\mathcal{M}_k \cap \mathcal{U}_j^c) = 0.2. In this case, supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus \operatorname{supp}(\pi) has positive probability under π~\tilde{\pi}, so ρ≪̸π~\rho \not\ll \tilde{\pi}.

Moreover, Eq. (3) uses the unit determinant of the ambient mn×mnmn\times mn Jacobian rather than the induced manifold area factor (For F(a,b,c)=(a,b,c,bc/a)F(a, b, c) = (a, b, c, bc/a)).

We think the FF you write is a mapping from the Euclidean space to Mk\mathcal{M}_k, rather than from the reverse direction. We follow this notation below.

Actually, we pad a dimension dd to make this mapping become F(a,b,c,d)=(a,b,c,d+bc/a)F(a, b, c, d) = (a, b, c, d + bc/a). This yields a squared Jacobian matrix of size mn×mnmn \times mn, which makes the determinant in Eq. (3) defined.

If you use F(a,b,c)=(a,b,c,bc/a)F(a, b, c) = (a, b, c, bc/a), then you get an unpadded Jacobian of the size mn×(m+n−k)kmn \times (m + n - k)k. In this case, the determinant in Eq. (3) is undefined, since the Jacobian matrix is non-squared.

This unpadded Jacobian can be found in our proof, see Lines 806-809. We avoid using it in Eq. (3), since it will cause an undefined determinant.

With respect to which reference measure on Mk\mathcal{M}_k are the densities defined, and how do you revise the change-of-variables formula in Eq. (3) and the infinite-KL argument in Section 2.4 / Appendix E?

We think the change of variables of Eq. (3) is correct, so we keep it. We agree that our infinite-KL argument in Section 2.4 / Appendix E is flawed, and have fixed this with a rigorous measure-theoretical argument in Section 3.2.

Thank you for correcting the KL coefficient and rerunning the experiments. The counting error was a central concern in my initial review, and I appreciate the update.

I recognize the rationale for full-manifold coverage, explicit chart densities, and a computable KL upper bound. The single-chart comparison favors WGM, but full coverage alone does not establish that a mixture is needed for finite KL. Gaussian Product also reports tighter bounds in all nine settings. Matching an expected rank trend does not demonstrate better characterization of generalization, since the ordering of upper bounds need not reflect actual risk. What concrete advantage does WGM offer over this simpler baseline?

The density issue remains unresolved. Padding with a deterministic coordinate does not justify a full-dimensional change of variables. Please clarify the common reference measure and substantiate the claimed benefit of explicit chart densities.

Reviewer_iePCOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper studies PAC-Bayes bounds for low-rank models such as LoRA. The authors argue that no Gaussian distribution can cover the full set of rank-k matrices, so the posterior may not be absolutely continuous with respect to the prior and the KL divergence may become infinite. To address this issue, they build a mixture of wrapped Gaussians, one for each chart in a standard cover of the rank-k matrix manifold. They upper-bound its KL divergence by a closed-form expression, then apply it to a PAC-Bayes bound for stochastic LoRA networks, reporting non-vacuous bounds on an MLP fine-tuned on a binary version of FashionMNIST.

Critical strengths

The paper addresses a relevant question with a principled, geometry-aware approach, offering an alternative to norm-based generalization bounds, which cannot capture the effects of low rank. The construction is clearly presented and could be of independent interest beyond PAC-Bayes.

Critical weaknesses
  1. Possible miscount in eq. (11-12). In each chart, the Gaussian is placed on the blocks A, B and C, which is consistent with the stated dimension (m+n−k)k(m+n-k)k. However, eq. (11) counts each entry as appearing in (m−1k−1)(n−1k−1)\binom{m-1}{k-1}\binom{n-1}{k-1} charts, which is the number of charts where the entry lies in both a selected row and a selected column (block A). Considering that an entry is used whenever its row or its column is selected, the factor in eq. (12) should be (m+n−k)kmn\frac{(m+n-k)k}{mn} rather than k2mn\frac{k^2}{mn}. If this is the case, the KL values in Tables 1-2 would need to be recomputed, and the non-vacuousness claim re-examined, as the correction could push several bounds closer to 1.
  2. The necessity of the multi-chart mixture should be further motivated. The paper’s argument in Section 2.4 relies on the trained weights lying outside the prior’s chart. This happens only if one specific k×kk\times k block of the trained matrix is exactly singular, a non-generic (and typically zero-probability) event. If a single chart is sufficient, the multi-chart mixture, which is the paper’s main contribution, would not be needed. Could the authors expand the discussion on this point, which is central to the paper? See also Questions.
Questions for authors
  1. Could you motivate why defining prior and posterior on the same single chart is not sufficient? Section 4 suggests this would yield a finite, closed-form Gaussian KL, apparently without the need for a mixture. A comparison with such a single-chart baseline in the experiments would also help.
  2. Does the event motivating the multi-chart mixture have non-zero probability? If not, would absolute continuity, and hence finiteness of the KL, still hold for a single-chart construction?
4 comments

Thank you very much for your insightful and constructive comments. We have carefully revised our paper based on your suggestions. Please let us know if we have addressed you concerns.

Possible miscount in eq. (11-12). In each chart, the Gaussian is placed on the blocks A, B and C, which is consistent with the stated dimension (m+n−k)k(m + n - k)k. ... Considering that an entry is used whenever its row or its column is selected... If this is the case, the KL values in Tables 1-2 would need to be recomputed, and the non-vacuousness claim re-examined, as the correction could push several bounds closer to 1.

Thank you for pointing out this critical miscalculation issue. Yes, it should be "its row or its column selected''. For a single entry in a m×nm\times n matrix, it will not appear in (m−1k)\binom{m - 1}{k} combinations of rows and (n−1k)\binom{n - 1}{k} combinations of columns. Consequently, it will not appear in (m−1k)(n−1k)\binom{m - 1}{k}\binom{n - 1}{k} charts, and the correct scalar in Eq. (11) should be \begin{equation*} \frac{\binom{m}{k}\binom{n}{k} - \binom{m - 1}{k}\binom{n - 1}{k}}{\binom{m}{k}\binom{n}{k}} = \frac{(m + n - k)k}{mn} \end{equation*}

Based in this change, we have fixed our code and have rerun all experiments. Here is the change of Table 1:

Before

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.4470.3950.2040.4640.4060.3080.5050.4260.278
KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}})29.0952.8352.8236.0789.32554.3927.3869.4433.84
Bound0.5860.5460.3560.6070.5720.5720.6440.5840.421

After

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.1740.1730.1440.2180.1920.1840.3050.2160.180
KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}})161.7156.4146.0577.6972.2883.98241.0775.9653.80
Bound0.3630.3600.3280.3800.3510.3470.5130.3770.332

The results show that, the empirical risk estimation E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] becomes smaller, and the KL divergence KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) becomes larger. The bound remains non-vacuous, and even becomes tighter (i.e., it drops from (0.5,0.6)(0.5, 0.6) to (0.3,0.4)(0.3, 0.4)).

The reason for E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] becoming smaller is that, after the scalar is fixed, we are able to optimize the bound correctly through Eq. (17). Before correction, the obtained posterior fXf_{\mathsf{X}} is far away from the optimal, making E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] large; after correction, the obtained fXf_{\mathsf{X}} closer to the optimal, making E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] smaller. The reduction in E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] is the main reason that results in a tighter Bound.

The KL divergence increases after fixing the scalar, but it does not scale as much as (m+n−k)/k(m + n - k)/k as predicted. The reason is that, in the previous computation of Eq. (12), we took KL(fXC∥∥gXC)=k2mnKL(v1∣∣v2)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) = \frac{k^2}{mn}KL(\mathsf{v}_1||\mathsf{v}_2), where the prior v2\mathsf{v}_2 is fixed and the posterior v1\mathsf{v}_1 is not fixed. When we correct its scalar to (m+n−k)kmn\frac{(m + n - k)k}{mn} and recompute Eq. (17), we get a new posterior v1′{\textsf{v}_1}', which gives KL(v1′∣∣v2)<KL(v1∣∣v2)KL({\mathsf{v}_1}'||\mathsf{v}_2) < KL(\mathsf{v}_1||\mathsf{v}_2). This new posterior makes KL(fXC∥∥gXC)KL(f_{\mathsf{XC}}\|\|g_{\mathsf{XC}}) scales much less than (m+n−k)/k(m + n - k)/k.

The results in Table 2 are also fixed. The main change is that the optimal parameter α\alpha is shifted from 10−310^{-3} to 10−510^{-5}. Please see our revised manuscript for more detail.

We have also uploaded our revised code in the supplemental materials.

The paper’s argument in Section 2.4 relies on the trained weights lying outside the prior’s chart. This happens only if one specific block of the trained matrix is exactly singular, a non-generic (and typically zero-probability) event.

This is a great question. Rigorously speaking, the probability of "lying outside the chart'' is undefined, and cannot be treated as zero without extra assumptions.

Suppose a prior π\pi is from a single chart, then it forms a probability space (Mk∩Uj,Σπ,π)(\mathcal{M}_k \cap \mathcal{U}_j, \Sigma_{\pi}, \pi). We can determine whether a measurable set within Mk∩Uj\mathcal{M}_k \cap \mathcal{U}_j has zero probability under π\pi, but the out-of-chart set Mk∩Ujc\mathcal{M}_k \cap \mathcal{U}_j^c is outside of the space, and π\pi is undefined on it. We cannot say that it has zero probability under π\pi.

To address this, we need to extend π\pi from Mk∩Uj\mathcal{M}_k \cap \mathcal{U}_j to Mk\mathcal{M}_k, such that Mk∩Ujc\mathcal{M}_k \cap \mathcal{U}_j^c is assigned a measure under π\pi. This requires an extra assumption: we need to assume that every measurable set in Mk∩Ujc\mathcal{M}_k \cap \mathcal{U}_j^c has zero measure under π\pi. Based on this, we have provided a rigorous argument in Section 3.2.

This assumption is necessary. If not under this assumption, we can find an alternative extension for π\pi that assigns a positive probability to Mk∩Ujc\mathcal{M}_k \cap \mathcal{U}_j^c. For example, we can define a measure π~\tilde{\pi} on Mk\mathcal{M}_k such that π~(A)=0.8π(A)\tilde{\pi}(A) = 0.8\pi(A) for any measurable subset AA in Mk∩Uj\mathcal{M}_k \cap \mathcal{U}_j and π~(Mk∩Ujc)=0.2\tilde{\pi}(\mathcal{M}_k \cap \mathcal{U}_j^c) = 0.2.

Could you motivate why defining prior and posterior on the same single chart is not sufficient? Section 4 suggests this would yield a finite, closed-form Gaussian KL, apparently without the need for a mixture. A comparison with such a single-chart baseline in the experiments would also help.

Thank you for this insightful suggestion. We have added this single-chart baseline in our experiment, and get a new empirical finding: comparing with the wrapped Gaussian mixture, the single-chart wrapped Gaussian has smaller KL divergence but looser bound.

The results of single-chart baseline are shown in Table 1, which we paste here:

Model (l,k)(l, k)(1,8)(1, 8)(1,16)(1, 16)(1,64)(1, 64)(2,8)(2, 8)(2,16)(2, 16)(2,64)(2, 64)(3,8)(3, 8)(3,16)(3, 16)(3,64)(3, 64)
r(u1)r(u_1)0.0640.0590.0610.0600.0590.0610.0570.0590.063
E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})]0.5120.5110.5050.5020.5020.4940.5000.5010.507
KL0.3710.7643.9530.3130.6363.1350.0190.0181.854
Bound0.6290.6290.6260.6190.6200.6150.6170.6180.626

The looseness of the bound is caused by a large E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] (the v=200v = 200-samples estimation of the stochastic empirical risk r(X)r(\mathsf{X})). This is because, when we take $\mathsf{X} = \begin{bmatrix} \mathsf{A} & \mathsf{B} \\ \mathsf{C} & \mathsf{C}\mathsf{A}^{-1}\mathsf{B} \end{bmatrix}$ and let $\mathsf{A}, \mathsf{B}, \mathsf{C}$ be Gaussian, the distribution of CA−1B\mathsf{C}\mathsf{A}^{-1}\mathsf{B} is heavy tailed, and this heavy tail can cause a large variance in r(X)r(\mathsf{X}), making its empirical estimation E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] large.

In contrast, if using a wrapped Gaussian mixture, the heavy tail of CA−1B\mathsf{C}\mathsf{A}^{-1}\mathsf{B} from all charts are averaged. The averaged tail results in less variance in r(X)r(\mathsf{X}), making E^fXv[r(X)]\hat{\mathbb{E}}^v_{f_{\mathsf{X}}}[r(\mathsf{X})] smaller and resulting in a tighter bound.

Does the event motivating the multi-chart mixture have non-zero probability?

The probability on the "out of chart" event is undefined, but we can assume that it has zero probability. See our Assumption 3.3.

If we use the multi-chart mixture, then this assumption is no longer required, making the theoretical framework more elegant.

I thank the authors for their thorough reply and the additional experiments. The counting issue appears to be resolved. However, I am still not sure that the reply and revision fully address the necessity of a mixture over multiple charts. If I understand correctly, the revised paper itself shows, in Corollary 3.2, that a single chart already suffices to keep the KL divergence finite, as the posterior puts zero probability outside the prior’s chart. Moreover, Assumption 3.3 does not add a real requirement, as the only way to extend the prior to the whole manifold is to assign zero probability outside its chart, which is exactly what Assumption 3.3 states.

Independently of this point, the revision presents the benefit of the mixture as a matter of elegance, while the original submission presented it as a necessity, and the opening of Section 2.4, as well as the abstract, introduction and conclusion, still present the mixture as necessary to avoid an infinite KL divergence. I believe these parts should be revised to be consistent with the updated claims.

Finally, as a minor remark, Assumption 3.3 as currently written seems to apply to every set not entirely contained in the chart, including the whole manifold, and would therefore assign it zero probability.

Thank you very much for your thoughtful and constructive feedback. We have uploaded a new revision of the paper. We hope it addresses your remained concerns.

The opening of Section 2.4, as well as the abstract, introduction and conclusion, still present the mixture as necessary to avoid an infinite KL divergence. I believe these parts should be revised to be consistent with the updated claims.

We appreciate your kind reminding. We have fixed all statements that would indicate "mixture is necessary to cover Mk\mathcal{M}_k" or "single-chart cannot cover Mk\mathcal{M}_k" in these parts. The revised statements suggest the mixture as a feasible solution rather than a necessary solution.

We have swapped Sections 2.3 and 2.4. Section 2.3 directly follows from Section 2.2, taking Gaussian as an example to illustrate the infinite KL divergence issue. Section 2.4 suggests the wrapped Gaussian as an solution. We think this sequence is more reasonable to establish our argument.

We have also removed the Figure 1 (c) that incorrectly illustrates the supports of wrapped Gaussians to avoid misleading.

In Corollary 3.2, that a single chart already suffices to keep the KL divergence finite, as the posterior puts zero probability outside the prior’s chart.

We apologize for this confusion. We have revised Corollary 3.2 as

Let ρ\rho and π\pi be two wrapped Gaussian distributions in two different charts, then ρ(supp⁡(ρ)∖supp⁡(π))=0\rho(\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi)) = 0. Without extra assumptions, π(supp⁡(ρ)∖supp⁡(π))\pi(\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi)) is undefined.

Simply having ρ(supp⁡(ρ)∖supp⁡(π))=0\rho(\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi)) = 0 is insufficient to satisfy ρ≪π\rho \ll \pi. We also need π(supp⁡(ρ)∖supp⁡(π))=0\pi(\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi)) = 0, but this probability is undefined.

This is because, the set supp⁡(ρ)∖supp⁡(π)\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi) is outside of the chart of π\pi. Within the chart, π\pi is pushed from an Euclidean space through a diffeomorphism. For any set outside of the chart, we do not have a diffeomorphism that connects it to the Euclidean space, so π\pi cannot be assigned on it. This is why the probability of outside-of-chart sets is undefined under π\pi.

To make KL(ρ∣∣π)KL(\rho||\pi) well-defined, we need π(supp⁡(ρ)∖supp⁡(π))\pi(\operatorname{supp}(\rho)\setminus\operatorname{supp}(\pi)) being defined and equals to zero. This is why we need Assumption 3.3.

We have also revised Section 3.2 to make it more readable.

Assumption 3.3 does not add a real requirement, as the only way to extend the prior to the whole manifold is to assign zero probability outside its chart, which is exactly what Assumption 3.3 states.

We agree with this. The goal of Assumption 3.3 is to describe how to rigorously assign zero probabilities to those sets outside of the chart. It is mainly about logical construction and may not directly related to real applications.

Assumption 3.3 as currently written seems to apply to every set not entirely contained in the chart, including the whole manifold, and would therefore assign it zero probability.

Yes, you are right. Our previous extension "For A∈ΣM∖ΣπA \in \Sigma_{\mathcal{M}} \setminus \Sigma_{\pi}, π(A)=0\pi(A) = 0'' is incorrect, the set difference operation can cause issues in manipulating the σ\sigma-algebras. Thus, we have fixed this to be "For A∈ΣMA \in \Sigma_{\mathcal{M}}, π(A)=π(A∩(Mk∩UIJ))\pi(A) = \pi(A \cap (\mathcal{M}_k \cap \mathcal{U}_{IJ}))''. In this case, if A∈Mk∩UIJcA \in \mathcal{M}_k \cap \mathcal{U}_{IJ}^c, then π(A)=0\pi(A) = 0. This approach manipulates σ\sigma-algebras using set intersection, which is safer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.