- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The concerned paper is interested in pinpointing and solving challenges in computing KL divergences on measures supported on low rank matrices. To this end the paper first argues that there exists gaussian measures supported on the space of rank k matrices whose actual support does not extend to the entire set of k rank matrices but merely a subset. Consequently there can be two measures whose actual supports differ by a set of non-zero measure (i.e. in jargon one is not absolutely continuous with respect to another) and have infinite KL divergence as a consequence.
I would note that this observation is heuristically pretty much obvious. For for instance you can begin by considering a by matrix which is rank 1 in a (fixed) "direction" i.e. the matrix is given by . You can then put a gaussian noise on this "centre" which respects the direction in the sense that it has no noise component (i.e. no variation) in the the direction orthogonal to . This distribution must be entirely supported on rank 1 matrices which are rank 1 in the "direction" . Voila, this is pretty much the counterexample or construction in the paper, but written in more formal terms and for general in 2.2
To solve this issue the paper suggests that we consider a different family. To this end first consider a gaussian measure on . Then use a pushformward of a diffeomorphic map to define a measure on the space of rank matrices. Unfortunately a single such diffeomorphism might not suffice. So the paper instead uses a collection of diffeomorphisms and takes mixtures between each family of generated measures to get a family of measures which is supported on the entirety of the space of rank k matrices. This idea in isolation could be of interest elsewhere, but firstly as the paper itself notes that it is not novel. Apart from the references in the paper there is for instance the paper "Density estimation on smooth manifolds with normalizing flows" which also has a similar idea of charts of diffeomorphisms for a similar purpose.
The rest of the paper is concerned with proving that their construction indeed work as advertised and deriving some KL divergence bounds in this new class. Finally some examples of using their construction for specific initializations are discussed.
More importantly it is unclear to me why such a complicated construction is needed to begin with. Running along the counterexample for the case we could represent every rank matrix as for some and some . The problem in the counterexample is essentially that one side, say , doesn't vary enough. We could solve this by simply taking to be distributed from a measure in a family of measures that is expressive enough in and for a similar measure which is expressive enough in (We can take gaussians whose covariance matrix have rank and respectively for instance). As such the product of these two random variables should indeed be supported on the entirety of the space of rank matrices. There even exist inequalities which would give you the KL divergence between two measures defined in this way with respect to the base component measures (e.g. , where the KL divergence between two random variables should be read as the same between their underlying measures). While product of gaussians is not a gaussian, its unclear what exact benefit a diffeomorphically mapped gaussian chart gives. Gaussianity is simply an assumption that is used in proofs for simplicity of proof; its often not a divine mandate.
Critical strengths
The central construction is somewhat interesting as a purely mathematical curiosity.
Critical weaknesses
Some of the observations seem to be trivial e.g. see summary : "I would note that this observation is heuristically pretty much obvious. For for instance you can begin by considering a by matrix which is rank 1 in a (fixed) "direction" i.e. the matrix is given by . You can then put a gaussian noise on this "centre" which respects the direction in the sense that it has no noise component (i.e. no variation) in the the direction orthogonal to . This distribution must be entirely supported on rank 1 matrices which are rank 1 in the "direction" . Voila, this is pretty much the counterexample or construction in the paper, but written in more formal terms and for general in 2.2"
Questions for authors
Please clarify why we should use this specific complicated class of measures for initialization when simpler methodologies exist e.g. see the last paragraph of the summary
4 comments
There even exist inequalities which would give you the KL divergence between two measures defined in this way with respect to the base component measures (e.g. ... ). While product of gaussians is not a gaussian, its unclear what exact benefit a diffeomorphically mapped gaussian chart gives. Gaussianity is simply an assumption that is used in proofs for simplicity of proof.
Thank you for bringing up this simpler solution based on Gaussian product. We think this inequality can be generalized to the rank- case. Let be Gaussian matrices, be Gaussian matrices, and suppose is independent of , is independent of . Let be a function. Then by data processing inequality, \begin{equation*} KL(U_1V_1^T||U_2V_2^T) = KL(f(U_1, V_1)||f(U_2, V_2)) \le KL(U_1, V_1||U_2, V_2) = KL(U_1||U_2) + KL(V_1||V_2) \end{equation*}
We have added this Gaussian Product (GP) as a baseline in our experiment (Section 5 and Appendix C). The results are as follows (pasted from Table 1), where model means hidden layers with rank- LoRA.
| Model | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| 0.067 | 0.062 | 0.066 | 0.064 | 0.062 | 0.065 | 0.063 | 0.065 | 0.067 | |
| 0.067 | 0.063 | 0.066 | 0.065 | 0.064 | 0.065 | 0.063 | 0.65 | 0.067 | |
| KL | 29.99 | 41.43 | 48.02 | 35.25 | 44.93 | 60.57 | 35.21 | 42.86 | 63.82 |
| Bound | 0.207 | 0.208 | 0.215 | 0.208 | 0.210 | 0.220 | 0.206 | 0.212 | 0.224 |
For comparison, the results of our Wrapped Gaussian Mixture (WGM) is as follows:
| Model | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| 0.119 | 0.113 | 0.120 | 0.122 | 0.123 | 0.123 | 0.104 | 0.213 | 0.148 | |
| 0.174 | 0.173 | 0.144 | 0.218 | 0.192 | 0.184 | 0.305 | 0.216 | 0.180 | |
| KL | 161.7 | 156.4 | 146.05 | 77.69 | 72.28 | 83.98 | 241.07 | 75.96 | 53.80 |
| Bound | 0.363 | 0.360 | 0.328 | 0.380 | 0.351 | 0.347 | 0.513 | 0.377 | 0.332 |
The results show that, although GP yields a tighter bound, for a fixed , the bound increases with , indicating that a larger rank results in a worser performance of LoRA. This does not match the empirical observations of (Shuttleworth et al., 2025), where LoRA with larger rank tends to perform better, as a larger rank enables LoRA to have more parameters, making it more expressive. This indicates that GP may not be a suitable distribution to characterize the performance of LoRA.
In contrast, WGM results in a bound that decrease with , matching the empirical observations of (Shuttleworth et al., 2025). This indicates that WGM can be a better distribution to characterize the performance of LoRA than GP.
Besides, a minor benefit of WGM is that it allows us to calculate the probability assigned on the subsets of much easier. This calculation is difficult for GP, because is surjective, not bijective; its density transformation from to is very complicated. In contrast, is bijective for WGM, making the density transformation easier to analyze. This is discussed in Lines 1152-1162.
Reference:
Reece Shuttleworth, et al. Lora vs Full Fine-tuning: An illusion of equivalence. NeurIPS, 2025.
To point 1: Yes you are right, my writing was probably unclear on this point. You are right that this does not show that it is impossible to build a gaussian rank 1 matrix with full support. The point however is that particular construction (being a very common one) informs a heuristic or intuition where we expect it to be not quite possible. As such I agree that rigorously showing impossibility is a technical result. However it is not a substantially interesting one. (e.g. the opposite result i.e. a gaussian with full support would be interesting since it is counter-intuitive)
To point 2: The experiments do suggest (not prove) that gaussian product KL doesnt decay with k. But then again your comparing experiments do not have a substantially smaller KL either (and in many cases is larger). More importantly isnt the regime where we want to apply low rank theory be one where k is small like say k < 100?
Thank you for your response.
For point 1, our thinking is that, if we only show readers a counterexample, they may still think that the geometric mismatching between Gaussian supports and is evitable within the Gaussian framework. We use this impossibility argument to convince readers that this mismatching is inevitable, and acknowledging them that this issue is fundamental and should be addressed from a geometric perspective. Motivated by this goal, we propose our geometric construction of wrapped Gaussian.
For point 2, we consider this a trade-off. When replacing Gaussian product with wrapped Gaussian mixture, one gets a slightly looser bound caused by a larger KL divergence, but this bound can more accurately reflect the relationship between the LoRA performance and rank.
We have tested the ranks , as shown in the model in the first row of each table. This is within the low rank regime as you mentioned.
Thank you very much for the insightful review. We hope our response addresses your concerns. Please let us know if you have further questions.
Thank you for raising this concern. We believe this is a misunderstanding. Our original statement in Lines 151-152 is that: It is impossible to find one Gaussian distribution whose support is exactly Mk. We think your counterexample does not justify the impossibility of this statement.
You can take a rank-1 matrix uxT, fix x and assign a Gaussian distribution on u, such that the support of uxT is a linear subspace within M1, but does not equal to M1. We have discussed this in Example 2.3. (b)(c). This counterexample shows ''It is possible to find one rank-1 Gaussian matrix whose support is not M1'', but it does not show ''It is impossible to find one rank-1 Gaussian matrix whose support is M1''.
The justification of impossibility is more complex. Our justification in Fact 2.4 is based on the geometric mismatching between Gaussian supports and Mk, and its formal proof in Appendix B relies heavily on differential geometric analysis.
We are not aware of other approaches that can also prove this impossibility. Our approach based on differential geometry is the simplest one we know so far.
Thank you for sharing this work. We have added this reference in our revised manuscript.
We think this work provides only a general framework. It shares the same fundamental idea as ours: using multiple charts to estimate the density on a manifold. However, their work neither provides an explicit choice of charts, nor provides explicit diffeomorphisms for each chart. If these math objects are not made explicit, we can hardly conduct a more detailed mathematical analysis.
The novelty of our construction is to make these math objects explicit for Mk:
We explicitly choose Mk∩UIJI∈Ik,J∈Jk as a valid chart covering for Mk.
We explicitly construct the diffeomorphism $\begin{bmatrix} A & B \\ C & D \end{bmatrix} \mapsto \begin{bmatrix} A & B \\ C & D - CA^{-1}B \end{bmatrix}$ for each chart. The benefit of this diffeomorphism is that its Jacobian has determinant 1. In this case, the change of variables $f_{\mathsf{X}}(X) = f_{\mathsf{Y}}(E(X))\left|\det\left(\left.\frac{d E(z)}{dz}\right|_{z = X}\right)\right| $ can be simplified to fX(X)=fY(E(X)). This greatly facilitates the analysis of density transformation.
Based on these explicit choices of charts and diffeomorphisms, our construction makes the density analysis on Mk much easier. We hope this explanation clarifies its novelty.