A Depth-Indexed Sample-Complexity Exchange Rate Between In-Context Learning and Low-Rank Fine-Tuning

In-context learning (ICL) and low-rank (LoRA) fine-tuning are rarely compared on equal footing. We place both on one statistical model, a rank-$r$ linear map estimated from $K$ demonstrations or $m$ fine-tuning examples, and ask how many demonstrations $K^{\ast }(m,L)$ match an $m$-example fine-tune at depth $L$. The answer depends on what attention can store. Any in-context estimator that is linear in the labels, including gradient-descent linear attention and any head bottleneck, needs asymptotically at least $d^2/[r(2d-r)]$ demonstrations per fine-tuning example, at every depth. With $r$ register tokens, the role played by system prompts, soft prompts and memory tokens, a fixed weight-tied block implements factored gradient descent exactly, so ICL and LoRA become the same algorithm. The exchange rate then becomes depth-indexed: it diverges below a critical depth $L^{\ast }=\log ({\sigma }_r/{\lambda }_0)/[2\log (1+2\eta {\sigma }_r)]$ and reaches $1:1$ above it, and we prove a two-sided escape-time bound whose constants are independent of the initialization scale ${\lambda }_0$, which the registers set. Task families with $k<rd$ effective parameters lower the achievable rate to $k/[r(2d-r)]$, and pretrained registers shift $L^{\ast }$ by roughly $4\times$ without removing the transition. Trained linear transformers confirm the separation.