- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the sample-complexity relationship between in-context learning (ICL) and low-rank fine-tuning (LoRA). Within a unified reduced-rank regression framework, the authors first establish a statistical efficiency lower bound for label-linear ICL estimators and construct a linear attention mechanism with register tokens that exactly implements LoRA-style gradient descent. Building on this construction, they analyze finite-depth optimization dynamics and derive two-sided bounds on the escape time around a critical depth, revealing the relationship between initialization, model depth, and sample efficiency. The paper further examines these theoretical results through synthetic experiments and trained linear Transformers.
Critical strengths
-
The paper places ICL and LoRA within a unified statistical learning framework and introduces a depth-indexed sample-complexity exchange rate, providing a clear and insightful perspective on the relationship between these two adaptation mechanisms.
-
The theoretical analysis is rigorous and substantial. The authors establish a statistical lower bound for label-linear estimators, derive an exact discrete-time recursion for factored gradient descent, and prove two-sided escape-time bounds with explicit constants.
-
The register attention construction is elegant. By using dedicated token states and fixed attention weights, the authors exactly implement factored gradient descent for LoRA. This construction establishes a rigorous connection between the two learning mechanisms and sheds light on how Transformers can perform implicit optimization during forward propagation.
-
The paper presents a thorough set of experiments that complement and support the theoretical analysis.
Critical weaknesses
-
The authors find that trained full-head linear Transformers can reach the reduced-rank regression error floor with only a few layers, without requiring register tokens. This is an interesting observation, suggesting that these models may learn a more efficient in-context learning mechanism than explicit factored gradient descent. Although this does not directly contradict the theoretical results, it raises questions about how broadly the proposed critical-depth law characterizes ICL. A deeper analysis of how these models achieve task-adaptive low-rank learning, and how their learning dynamics differ from the theoretical construction, would be valuable.
-
Although the theoretical analysis is strong, the empirical evidence is largely based on synthetic linear regression tasks. In particular, the critical-depth transition, the alignment between initialization and the task distribution, and their relationship to sample efficiency are promising findings that may have practical implications. Testing some of these predictions on more realistic Transformer architectures or learning tasks would further strengthen the paper.
Questions for authors
My questions are covered by the weaknesses above. They mainly concern the interpretation and generalizability of the theoretical results and do not undermine the paper's established contributions. I would be particularly interested in seeing whether these theoretical insights can predict phenomena in more realistic settings, even if some predictions do not hold. Such results would still be valuable for understanding the scope and limitations of the theory.
2 comments
Thank you for the detailed response, and in particular for the additional experiments in more realistic settings. These results address the concern I raised about empirical validation and strengthen the practical relevance of the theory.
I acknowledge that the other reviewers have raised different concerns, and I respect their perspectives. In my view, however, the paper is theoretically rigorous and empirically well supported, and these concerns do not outweigh its substantial contributions. I therefore maintain my rating.
Thank you for the strong assessment and for two questions that led to our most informative new results. The predictions for the main experiments were fixed before the runs.
W1. How trained full-head models reach the floor. On the same demonstrations, we compared each trained model's estimate with four candidate estimators: reduced-rank regression (RRR), shrunken RRR (the same output-space directions, each weighted by an optimal universal shrinkage), the best spectral filter of OLS, and OLS. The models compute shrunken RRR:
Relative per-task distance from the model's estimate; two seeds at depth 4, one at depth 6.
Read off direction by direction, the shrinkage the model applies converges to the optimal curve with depth: on noise directions 0.36, −0.07 and 0.005 at depths 2, 4 and 6 (optimum 0.003), and on signal directions 0.909, 0.960 and 0.964 (optimum 0.966). This answers both parts of your question.
They also do not route through registers. Controlling register scale and alignment leaves the depth to the floor unchanged, even when registers carry as much of the first-layer update as the demonstrations (slope 0.0 layers per decade), and the same holds when the registers are trained from each scale. Models with 8 to 16 layers, which diverged in the submission, now train stably and settle at the same shrunken-RRR level (risk 0.578 to 0.580). The critical-depth law thus characterizes factored descent, which LoRA and the register construction share. Section 8 now states this scope, with this mechanism as the account of trained models; App. J.8 reports the distances and the implied and optimal shrinkage curves, and App. J.7 the register arms.
W2. Realistic tests, including predictions that fail. We ran three, with predictions written down beforehand. Outcomes:
As you anticipated, the outcomes are informative about scope. The depth transition and the effect of alignment carry over qualitatively, while the theory's constants and the register mechanism are specific to linear attention. All checks and their outcomes are in App. J.9 and J.10.