How Much Rank Does LoRA Need? Task-Dependent Rank-Error Bounds for Transformer Attention

We study how accurately a rank-$r$ low-rank adaptation (LoRA) update can approximate a target attention function on downstream inputs. For a fixed pretrained attention head, target attention function, and downstream input distribution, we bound the minimum expected Kullback-Leibler (KL) error achievable by any rank-$r$ query update. Our main results relate this error to the target update's spectral tail, weighted by its effect on downstream attention scores. We also derive alternative bounds under different assumptions on the target attention probabilities and candidate scores. To show that finite-logit rank can overestimate the rank needed to approximate attention, we construct target families that require rank $k$ for exact realization at every finite logit scale, but whose attention probabilities can be approximated arbitrarily well with rank $k-\lfloor k/3\rfloor$. Finally, we extend the error analysis to fused multi-head LoRA and joint query/key updates, accounting for shared rank budgets and query/key factorization constraints.