- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tensorizes weight updates, attention matrices, and output Gram matrices into matrix-product-state chains and measures singular-value entropy across their cuts. LoRA updates exhibit concentrated spectra and a central entropy valley, while attention-related entropies change less across adaptation methods. The authors interpret these observations as a no-hair-like explanation of LoRA’s effectiveness, supported by random-matrix calculations and an MPS adapter extension.
Critical strengths
-
Meaningful controls separate spectral concentration from nominal rank. Appendix J matches trainable Q/V projections and uses three paired seeds. Appendix K truncates all transplant arms to rank 256 and exchanges spectra and singular vectors. Transplanting the LoRA spectrum closes 97.71% of the off-center profile gap and 97.98% of the valley-depth gap. These results go beyond the elementary rank constraint. [Appendices J–K, pp. 51–54, Tables 3–4]
-
The descriptive observations receive substantial robustness checks. Appendix L tests 21 block-preserving factor orders and eight basis transforms; Appendix M includes all 32 heads at three layers and 100 unpadded nontraining inputs. Appendix I.3 also reports breakdown under extreme scaling. These strengthen the descriptive contribution within the tested regimes. [pp. 41–42, 54–55]
Critical weaknesses
-
Entropy similarity does not establish functional invariance. At a rank- update’s row–column cut, the bound follows from rank alone; the matched-rank controls add evidence about spectral concentration, but neither fact explains comparable adaptation performance. Normalized singular-value entropy discards scale and orientation. Even identical output Gram matrices can accompany different downstream predictions under fixed downstream weights. Appendix K evaluates transplanted entropy profiles, and Appendix M compares entropy at loss-matched checkpoints; neither directly demonstrates the proposed suppression of functional differences. [Abstract; Figure 1; pp. 8–9; Appendices K–M]
-
The theory contains a false bound and does not directly match the empirical observable. In Lemma E.2/F.2, gives and an entropy upper bound of zero, although . The attention asymptotic also concerns full row–column entropy as sequence length grows, whereas the experiments primarily scan tensorized cuts at fixed length. For , there are artificial sites, so entropy proportional to is extensive in that site count. Moreover, the output theorem concerns , whereas Figure 6 tensorizes . These distinctions undermine the current area-law interpretation without further justification. [pp. 22–28; Theorems F.1/F.3; Figure 6]
Questions for authors
-
In the existing rank-matched transplantation experiment, how do attention/output distances, predictive KL, and held-out loss change alongside entropy? Evidence connecting the spectral interventions to functional behavior would address the proposed explanation of LoRA's effectiveness.
-
What subsystem and size variable define the claimed area law, and how do the theorems connect to the trained causal-attention and output-Gram observables? Please identify which measured quantities the asymptotic statements actually cover.
-
Can the entropy bound be corrected and the dependent conclusions rechecked? Correct mathematics, together with functional evidence or a consistently narrower descriptive claim, would support reconsideration of the rating.
2 comments
Dear Reviewer 5XeT,
Thank you again for your time and careful comments on our paper. We have uploaded the revised PDF, in which all changes are marked in blue. Following your comments, the revision corrects Lemma E.2/F.2, adds Corollary F.4 for , withdraws the area-law interpretation and states the scope of Theorem E.1, and reports the functional measurements you asked for (Appendix N.2).
We would be grateful to know whether our response has addressed your concerns. Please feel free to reach out to us if you have any further questions or suggestions. We would be happy to discuss them, and we can still include further changes in the PDF before the revision deadline.
Thank you very much!
Best regards,
Authors
Thank you very much for your careful and constructive comments. We provide a point-by-point response as follows.
We agree. In the revision, we no longer claim functional invariance. The attention-level result is stated descriptively: attention entanglement entropy is insensitive to the internal differences between adaptation methods. The sentence that this "explains why LoRA performs comparably to FFT" is removed from the Abstract, Section 1 and Section 6 (blue text). We also agree that the normalized entanglement entropy (EE) discards scale and orientation. In our new measurements, the profile of ΔW is set by its singular values, whereas the function depends on its singular vectors (see below).
Yes, this is correct, and thank you for pointing it out. The revised Section 5 now states this limitation, and we no longer read the EE of the Gram matrix as evidence about predictions. Table 7 gives a direct example: changing only ΔWV changes the layer-0 output by 7.8% and the predictions by more than the seed-to-seed KL, while the EE of XX⊤ stays at 0.559 and 0.558 bits.
We rebuilt the transplanted updates of Appendix K for all 16 layers and measured these quantities together with the EE (Appendix N.2, Table 6). The EE of ΔW does not predict function. The update with the LoRA singular values and the dense singular vectors has the LoRA profile but the worst held-out NLL (1.92 instead of 1.57), whereas the update with the dense singular values and the LoRA singular vectors has the dense profile and an NLL of 1.60. Attention and output distances order these updates in the same way as the predictive KL. Updates whose attention EE is within 0.008 bits of LoRA still differ in their predictions by about 1.8 times the seed-to-seed KL, so the attention EE does not capture such small differences in function.
Thank you very much for raising this point, and we agree. The corrected lemma uses the exact tail mass ε=δ1/(1+δ1), where δ1=∑i≥2λi/λ1, and gives S(M)≤h2(ε)+εlog(T−1). For M=I2, this gives ε=1/2 and the bound log2, so it is tight. The lemma is used in Theorem E.3/F.3 and in the new Corollary F.4. The conclusion S(X)=O(logT/T) of the theorem still holds, because δ1≤(T−1)η with η=O(T−3) gives ε≤δ1=O(T−1). The revised PDF includes this correction (blue text).
We agree, and we withdraw the area-law interpretation. The subsystem is the row index of A (n=log2T of the 2n sites), and the size variable is T. As you note, a logT growth is then extensive in the site count. Theorem E.1/F.1 covers only the central row-column cut, the limit T→∞, and the idealized non-causal isotropic model of Assumption 1. It does not cover the off-center cuts at fixed T in Figures 5, 19 and 20, or trained causal attention. The revised Abstract, Sections 1, 5 and 6, and Appendix E state this scope (blue text).
For trained causal attention, we now test the theorem directly (Appendix N.4). Assumption 1 does not hold there: at layer 8, s1(A) grows from 6.5 to 14.2 as T goes from 128 to 1024, instead of tending to 1. The EE at the central cut still grows roughly linearly in logT, by 0.22 to 0.50 bits per doubling of T, which is 2 to 4 times smaller than the coefficient of the theorem. So the logarithmic growth carries over to trained attention, but its coefficient does not.
We agree. The revised PDF adds Corollary F.4 for the measured object O=XX⊤. At the central cut, its probabilities are qi∝λi2, while those of X are ∝λi. With η=rstable(O)−1 and εG=η/(1+η), it gives S(O)≤h2(εG)+εGlog(T−1), so η=O(T−1) implies S(O)=O(logT/T). This bounds the central cut only and does not cover the off-center cuts of Figure 6.
In summary, following your Q3, we corrected the bound and rechecked the dependent results, added the functional measurements, and narrowed the claims to descriptive ones. We hope we have addressed your concerns. Thank you very much.