acceptodds
Under review as a conference paper at ICLR 2027

How Much Rank Does LoRA Need? Task-Dependent Rank-Error Bounds for Transformer Attention

Abstract

We study how accurately a rank- low-rank adaptation (LoRA) update can approximate a target attention function on downstream inputs. For a fixed pretrained attention head, target attention function, and downstream input distribution, we bound the minimum expected Kullback-Leibler (KL) error achievable by any rank- query update. Our main results relate this error to the target update's spectral tail, weighted by its effect on downstream attention scores. We also derive alternative bounds under different assumptions on the target attention probabilities and candidate scores. To show that finite-logit rank can overestimate the rank needed to approximate attention, we construct target families that require rank for exact realization at every finite logit scale, but whose attention probabilities can be approximated arbitrarily well with rank . Finally, we extend the error analysis to fused multi-head LoRA and joint query/key updates, accounting for shared rank budgets and query/key factorization constraints.

Primary area:
learning theory
Keywords:
Parameter-Efficient Fine-Tuning (PEFT), Low-Rank Adaptation (LoRA), Transformers, Attention Mechanism, Deep Learning Theory, Rank-Error Bounds, Task-Dependent Bounds, Softmax Saturation, Spectral Bounds, Approximation Theory, Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Rg6zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how well a rank-rr query LoRA can approximate a target full-rank attention function, with the approximation error measured by the expected KL divergence between the target and the adapted attention distributions.

The authors first establish upper and lower bounds on the softmax KL divergence in terms of the centered logit error (Theorem 3.1). They then use these bounds to characterize the minimum achievable approximation error through the tail singular values of a weighted target update (Theorem 4.1), under assumptions on the independence of the attention geometry and input features, the target probability floor, and certain moment and geometry conditions. The upper bound is achieved by a closed-form rank-rr approximation.

The appendices further provide extensions that relax the independence assumption, derive a local Fisher-based bound, and establish a high-mass lower bound. The authors also construct a Walsh-character example showing that the rank required to reproduce attention probabilities can be strictly smaller than the rank needed to match the logits. Finally, the analysis is extended to multi-head attention with a shared rank budget and joint query/key LoRA, where an additional realizability term appears.

The paper is entirely theoretical and does not include experimental evaluations.

Critical strengths

1. The theoretical analysis is well motivated and technically sound. The paper provides a useful characterization of the approximation error of low-rank attention updates, showing that the relevant spectrum is the downstream-weighted spectrum rather than the singular values of Δ∗\Delta_{\ast} alone. The use of the robust functions ψ\psi and ψup\psi_{\rm up} accounts for softmax saturation, and Proposition C.1 justifies the necessity of the linear branch. I checked the proofs and did not find any major errors; however, some minor technical issues raised in the Questions section appear straightforward to address and do not affect the stated results.


2. The closure-rank result in Theorem 6.1 is an interesting and potentially novel contribution. The authors establish an exact separation between the rank required to represent finite logits and the rank required to approximate the corresponding attention probabilities in the closure, reducing the required rank from kk to k−⌊k/3⌋k-\lfloor k/3\rfloor. I could not identify prior work establishing this exact separation. The lower-bound argument is particularly interesting, as the restricted Fourier coefficients constrain each limiting block to the form B(α,β,γ)B(\alpha,\beta,\gamma), for which a rank-one representation would require the impossible condition −α2=1-\alpha^2=1.

Critical weaknesses
  1. The novelty of several theoretical results is limited, and the paper does not sufficiently discuss the closest related work. While the connection between low-rank approximation and global KL bounds for softmax attention is interesting, several components of the analysis are already established in the literature.

    • The upper bound in Theorem 3.1 follows from Hoeffding's lemma [1] and softmax shift invariance, while the lower bound is closely related to the local strong convexity of log-sum-exp and generalized self-concordance [2, 3].
    • The closed-form solution in Eq. (12) and Lemma D.3 is a special case of generalized rank-constrained matrix approximation studied by Friedland and Torokhti [4].
    • Two-sided weighting using input statistics and output curvature has already been investigated in low-rank approximation and LLM compression, including GFWSVD [5], IO-SVD [6], and KQ-SVD [7]. Related one-sided approaches include FWSVD [8] and ASVD [9]. Curvature-Guided LoRA [10] also establishes a related characterization of the optimal residual in terms of curvature-weighted singular values under an NTK and K-FAC approximation.
    • The target-Fisher weighting in Appendix F is closely related to the output-side KL curvature used in IO-SVD [6], while the moment argument in Lemma D.1 is a standard L4L^4–L2L^2 equivalence argument associated with the small-ball method [11].

    I think the authors should more clearly distinguish the established analytical tools from their actual contributions and discuss how their results differ from those of these studies. In particular, the rank-indexed KL bounds, the multi-head extension, the Q/K realizability term, and the exact closure-rank separation appear to be the main contributions and novel.


2. The practical implications of the theoretical results are insufficiently established. While the paper provides a theoretical characterization of the approximation capacity of low-rank attention updates, it is unclear whether the proposed bounds are sufficiently tight or computationally practical to guide LoRA rank selection.

  • The bounds require access to the dense target update Δ∗\Delta_{\ast}, which is generally unavailable without full-rank training. It is unclear whether estimating this update would be more efficient than directly evaluating different LoRA ranks.
  • The bounds depend on constants that may make them extremely loose in practice. In particular, the lower bound depends on the minimum target attention probability aa, which can be extremely small in realistic attention distributions. Consequently, the bounds may not be sufficiently informative to distinguish between different rank choices.
  • The sufficient rank obtained from the upper bound applies to the closed-form approximation MrM_r, rather than to LoRA trained on the downstream objective. It is unclear how well the theoretically predicted ranks correspond to those required in actual LoRA training.
  • The proposed weighting scheme is not directly optimized for attention KL divergence. Alternative choices, such as the target-Fisher weighting discussed in Appendix F, may provide better approximations, but the paper does not establish how these approaches compare in practice.
  • The paper does not provide any experimental evaluation or representative numerical example demonstrating that the theoretical bounds lead to meaningful rank-selection decisions.

I understand that the main contribution is theoretical, and an extensive experimental evaluation may not be necessary. However, given that practical rank selection is an important motivation of the paper, I think the authors should provide at least a small empirical evaluation demonstrating the usefulness of the bounds, or clarify that the contribution is primarily a theoretical characterization rather than a practical rank-selection procedure.


[1] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30, 1963.

[2] F. Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, 4:384–414, 2010.

[3] T. Sun and Q. Tran-Dinh. Generalized self-concordant functions: A recipe for Newton-type methods. Mathematical Programming, 178:145–213, 2019. arXiv:1703.04599.

[4] S. Friedland and A. Torokhti. Generalized rank-constrained matrix approximations. SIAM Journal on Matrix Analysis and Applications, 29(2):656–659, 2007. arXiv:math/0603674.

[5] V. Chekalina, D. Moskovskiy, D. Cherniuk, M. Kurkin, A. Kuznetsov, E. Frolov. Generalized Fisher-weighted SVD: Scalable Kronecker-factored Fisher approximation for compressing large language models. arXiv:2505.17974v1, 2025.

[6] A. Abbasi, C. Thrash, H. Qin, H. Pirsiavash, S. Kolouri. IO-SVD: Input-output whitened SVD for adaptive-rank LLM compression. arXiv:2605.15626, 2026.

[7] D. Lesens, B. T. Rakhshan, G. Rabusseau. KQ-SVD: Compressing the KV cache with provable guarantees on attention fidelity. AISTATS, PMLR 300, 2026. arXiv:2512.05916.

[8] Y.-C. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, H. Jin. Language model compression with weighted low-rank factorization. ICLR, 2022. arXiv:2207.00112.

[9] Z. Yuan et al. ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv:2312.05821, 2023.

[10] F. Zheng and A. Proutière. Curvature-guided LoRA: Matching full fine-tuning in function space. arXiv:2603.29824, 2026.

[11] S. Mendelson. Learning without concentration. COLT, 2014; Journal of the ACM, 62(3), 2015. arXiv:1401.0304.

Questions for authors

Questions That Could Change My Assessment

  1. How tight are the proposed bounds in practice? Can the authors evaluate Theorem 4.1 on attention updates from at least one pretrained language model, for example Qwen2.5-0.5B or Llama-3.2-1B, using a query-only LoRA target? It would be useful to compute TrT_r and the corresponding upper and lower KL bounds across different ranks, and compare them with the actual attention KL achieved by MrM_r. What values do aa, Λ\Lambda, κh\kappa_h, and crc_r take in this setting?

In particular, does the dependence on the minimum attention probability make the lower bound vacuous, and how conservative is the upper bound? Are there ways to obtain more informative guarantees using typical rather than worst-case constants? This is probably the most important question for my assessment of the paper's practical significance.


  1. Can the proposed bounds actually guide LoRA rank selection? The analysis assumes access to the dense target update Δ∗\Delta_{\ast}, which may require full-rank training. Could the authors evaluate whether a small number of training steps provides a sufficiently accurate estimate of the weighted spectral tail TrT_r?

For example, it would be useful to compare the ranks predicted using an early estimate of Δ∗\Delta_{\ast} with those obtained from the final target update. More importantly, how do the sufficient ranks predicted by the bounds compare with the ranks selected through a conventional LoRA rank sweep on the downstream task?

Even a small experiment with ranks 4, 8, 16, 32, and 64 would help clarify whether the theoretical characterization leads to meaningful rank-selection decisions.


  1. How does the proposed weighting compare with alternative approximations? The main result uses the weighted spectrum of G1/2Δ∗Σ1/2G^{1/2}\Delta_{\ast}\Sigma^{1/2}, while Appendix F considers a target-Fisher weighting that is more directly related to the local curvature of attention KL.

Can the authors compare MrM_r with plain SVD, input-only weighted SVD, and target-Fisher weighted SVD across different ranks?

It would be particularly useful to report both the achieved attention KL and how accurately the corresponding spectral tails predict approximation difficulty across heads. Can the authors also compare MrM_r with a rank-rr update directly optimized to minimize attention KL? This would clarify how close the proposed closed-form approximation is to the best achievable error at a given rank.


  1. Does attention KL provide a reliable measure of downstream approximation quality? The paper focuses on matching attention probabilities, but it is unclear how improvements in attention KL translate into changes in the model's output distribution or downstream performance.

Can the authors evaluate whether approximations with lower attention KL also preserve the final model outputs more accurately?


Questions on Technical Details

  1. Uniformity of the constant in Theorem 8.1. The theorem states that for every feasible pair (A,B)(A,B), there exists a constant κ≥1\kappa\geq1. However, κ\kappa also appears in the final bound (32), and the proof in Eq. (120) appears to use a single constant for all feasible pairs. Should the quantifiers be exchanged, as in the definition of κr\kappa_r in Theorem E.1?

  1. Scope of the head-output guarantees. Equations (112)–(113) derive bounds on the attention-head outputs using Pinsker's inequality, while the Introduction and Figure 1 suggest that the analysis is restricted to attention probabilities. Do the authors intend to claim guarantees on the attention-head outputs as well? If so, I think this result should be stated more explicitly, with Dh(u)D_h(u) and diamV,h(u)\mathrm{diam}_{V,h}(u) properly defined.

  1. Normalization in Eqs. (85)–(86). These equations appear to omit the factor n−1/2n^{-1/2}. Does this normalization cancel in the subsequent argument, or should the equations and dependent expressions be corrected?

  1. Construction in Proposition C.1. The condition G=IG=I appears to require β=1\beta=1. Is this an implicit assumption, or does the construction also apply to general β\beta?

There are also two minor presentation issues:

  • Line 215 should refer to the squared residual Frobenius error.
  • Σ\Sigma should be described as a second-moment matrix unless the features are assumed to be centered.
Reviewer_GUrVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors prove bounds on the KL divergence of the softmax distribution of an attention layer between a low-rank and a full-rank query update. Their bounds are based on the rank of the low-rank update. They extend these results to multi-head, joint Q/K, and saturation contexts.

The motivation for why the reader may care about such bounds is weak. The paper does not include experiments, but experiments would be beneficial to alleviate concerns of vacuousness.

Critical strengths

None.

Critical weaknesses
  • Weak motivation for their work. It is unclear why a reader should care about these bounds or what these bounds tell us about selecting a LoRA adapter. Their claim that their approach somehow aids in rank selection seems to not be true in practice.
  • Theorem 3.1 is an application of routine techniques and should be called a Lemma. With the exception of Theorem 6.1 the other Theorems seem only slightly less trivial than Theorem 3.1.
  • In practice most bounds are vacuous. With a=min⁡p∗a=\min p_* the lower bounds quickly decay to zero. In practical transformer training I expect that aa will be quite small.
  • Assumptions do not hold for self-attention (see below).
Questions for authors
  1. Why should I care? In practice these bounds require already having Δ∗\Delta_*, and if you already have Δ∗\Delta_* you can easily attain an empirical rank-error curve which has the added benefit of directly measuring what you actually care about.
  2. As stated your results do not apply to standard attention. How would you extend your work to cover architectures that are used in practice? In particular, how do you extend your results to the case when G(u)G(u) and h(u)h(u) are not independent?
  3. Can you run some experiments to find how tight these bounds are in practice? If the bounds are mostly vacuous I think that you would need a method to establish bounds which are tighter in real-world applications.
  4. Why does KL error of a softmax distribution of an interior layer lead to worse outcomes in model risk (loss)? It's not obvious to me that the central assumption, that minimizing the KL divergence between the dense and low-rank adaptations, is even beneficial at a macroscopic scale.
Reviewer_BdsAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper establishes a theoretical explanation for why low-rank LoRA may suffice even when the target update has high matrix rank.

The paper does, however, not provide any numerical experiments to illustrate or validate the findings.

Critical strengths
  • The authors isolate the representational limitations imposed by LoRA rank, independently of optimization and training dynamics. The formulation in terms of minimum achievable expected attention KL divergence is precise and provides a meaningful way to quantify approximation capacity.

  • Task-dependent characterization of approximation error. The central result (Theorem 4.1) connects achievable attention KL divergence to the spectral tail of a downstream-weighted target update. This is a meaningful refinement of standard low-rank approximation theory because it accounts for the geometry of the actual input distribution.

Critical weaknesses

Lack of empirical illustration and numerical experiments.

Questions for authors
  • The main spectral characterization assumes independence between query activations h(u)h(u) and key geometry G(u)G(u), which generally does not hold in self-attention. Theorem E.1 permits dependence but introduces additional uniform comparison assumptions. Can the authors demonstrate that these conditions hold approximately in realistic models, or provide verifiable sufficient conditions?

  • The paper contains no empirical experiments. A comparison of the predicted spectral rank-error curves against measured attention KL for real Transformer heads would substantially strengthen the work. How well does the proposed weighted-SVD construction perform relative to standard truncated SVD and LoRA adapters optimized directly for attention KL?

  • The authors distinguish their theoretical characterization from adaptive rank allocation methods such as AdaLoRA (Zhang et al., 2023) in Section 9. However, it remains unclear whether the proposed bounds provide actionable insights beyond existing adaptive-rank methods, including AdaLoRA and dynamical low-rank training approaches such as GeoLoRA. Could the authors clarify whether their task-weighted spectral characterization can inform online rank adaptation without assuming a known dense target update? If this is not possible, what is the intended practical use of the characterization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.