acceptodds
Under review as a conference paper at ICLR 2027

A Depth-Indexed Sample-Complexity Exchange Rate Between In-Context Learning and Low-Rank Fine-Tuning

Abstract

In-context learning (ICL) and low-rank (LoRA) fine-tuning are rarely compared on equal footing. We place both on one statistical model, a rank- linear map estimated from demonstrations or fine-tuning examples, and ask how many demonstrations match an -example fine-tune at depth . The answer depends on what attention can store. Any in-context estimator that is linear in the labels, including gradient-descent linear attention and any head bottleneck, needs asymptotically at least demonstrations per fine-tuning example, at every depth. With register tokens, the role played by system prompts, soft prompts and memory tokens, a fixed weight-tied block implements factored gradient descent exactly, so ICL and LoRA become the same algorithm. The exchange rate then becomes depth-indexed: it diverges below a critical depth and reaches above it, and we prove a two-sided escape-time bound whose constants are independent of the initialization scale , which the registers set. Task families with effective parameters lower the achievable rate to , and pretrained registers shift by roughly without removing the transition. Trained linear transformers confirm the separation.

Primary area:
learning theory
Keywords:
Sample Complexity, Exchange Rate, LoRA, ICL

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_GQqUOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the sample-complexity relationship between in-context learning (ICL) and low-rank fine-tuning (LoRA). Within a unified reduced-rank regression framework, the authors first establish a statistical efficiency lower bound for label-linear ICL estimators and construct a linear attention mechanism with register tokens that exactly implements LoRA-style gradient descent. Building on this construction, they analyze finite-depth optimization dynamics and derive two-sided bounds on the escape time around a critical depth, revealing the relationship between initialization, model depth, and sample efficiency. The paper further examines these theoretical results through synthetic experiments and trained linear Transformers.

Critical strengths
  1. The paper places ICL and LoRA within a unified statistical learning framework and introduces a depth-indexed sample-complexity exchange rate, providing a clear and insightful perspective on the relationship between these two adaptation mechanisms.

  2. The theoretical analysis is rigorous and substantial. The authors establish a statistical lower bound for label-linear estimators, derive an exact discrete-time recursion for factored gradient descent, and prove two-sided escape-time bounds with explicit constants.

  3. The register attention construction is elegant. By using dedicated token states and fixed attention weights, the authors exactly implement factored gradient descent for LoRA. This construction establishes a rigorous connection between the two learning mechanisms and sheds light on how Transformers can perform implicit optimization during forward propagation.

  4. The paper presents a thorough set of experiments that complement and support the theoretical analysis.

Critical weaknesses
  1. The authors find that trained full-head linear Transformers can reach the reduced-rank regression error floor with only a few layers, without requiring register tokens. This is an interesting observation, suggesting that these models may learn a more efficient in-context learning mechanism than explicit factored gradient descent. Although this does not directly contradict the theoretical results, it raises questions about how broadly the proposed critical-depth law characterizes ICL. A deeper analysis of how these models achieve task-adaptive low-rank learning, and how their learning dynamics differ from the theoretical construction, would be valuable.

  2. Although the theoretical analysis is strong, the empirical evidence is largely based on synthetic linear regression tasks. In particular, the critical-depth transition, the alignment between initialization and the task distribution, and their relationship to sample efficiency are promising findings that may have practical implications. Testing some of these predictions on more realistic Transformer architectures or learning tasks would further strengthen the paper.

Questions for authors

My questions are covered by the weaknesses above. They mainly concern the interpretation and generalizability of the theoretical results and do not undermine the paper's established contributions. I would be particularly interested in seeing whether these theoretical insights can predict phenomena in more realistic settings, even if some predictions do not hold. Such results would still be valuable for understanding the scope and limitations of the theory.

2 comments

Thank you for the strong assessment and for two questions that led to our most informative new results. The predictions for the main experiments were fixed before the runs.

W1. How trained full-head models reach the floor. On the same demonstrations, we compared each trained model's estimate with four candidate estimators: reduced-rank regression (RRR), shrunken RRR (the same output-space directions, each weighted by an optimal universal shrinkage), the best spectral filter of OLS, and OLS. The models compute shrunken RRR:

DepthShrunken RRRRRRBest OLS filterOLS
40.0320.0480.0640.250
60.0140.0360.0560.251

Relative per-task distance from the model's estimate; two seeds at depth 4, one at depth 6.

Read off direction by direction, the shrinkage the model applies converges to the optimal curve with depth: on noise directions 0.36, −0.07 and 0.005 at depths 2, 4 and 6 (optimum 0.003), and on signal directions 0.909, 0.960 and 0.964 (optimum 0.966). This answers both parts of your question.

  • Why they beat the RRR floor. Shrinkage removes variance that hard truncation keeps: shrunken RRR has risk 0.575 against 0.592 for RRR, and the depth-6 models reach 0.576.
  • Why so shallow, and how this differs from the construction. Factored descent must escape a small-initialization plateau whose length Theorem 1 gives; that slope predicts random-initialization critical depths to within 4% in all 10 settings we tested. The trained models instead compute output-space selection and shrinkage directly, which a few label-nonlinear layers can do with no plateau to escape.

They also do not route through registers. Controlling register scale and alignment leaves the depth to the floor unchanged, even when registers carry as much of the first-layer update as the demonstrations (slope 0.0 layers per decade), and the same holds when the registers are trained from each scale. Models with 8 to 16 layers, which diverged in the submission, now train stably and settle at the same shrunken-RRR level (risk 0.578 to 0.580). The critical-depth law thus characterizes factored descent, which LoRA and the register construction share. Section 8 now states this scope, with this mechanism as the account of trained models; App. J.8 reports the distances and the implied and optimal shrinkage curves, and App. J.7 the register arms.

W2. Realistic tests, including predictions that fail. We ran three, with predictions written down beforehand. Outcomes:

PredictionSettingOutcome
Alignment with the task family lowers the depth needed, with sharp transitionsSoftmax transformers, 4 heads, width 128Passed. Floor at 2 layers when matched (then the family-oracle risk, 0.339 against 0.340) and 4 for a diverse mixture; an unrelated family does not transfer.
In-context predictions become decodable abruptly with depthQwen2.5 0.5B, 1.5B, 3B; six tasksPassed in 13 of 15 model–task pairs; the logit lens agrees.
Below a critical layer, intermediate-layer ICL cannot match LoRA; above it, it canSamePassed on at least three tasks per model.
At matched budget, the rate rises with LoRA's step budgetSameHolds against the best LoRA checkpoint at each budget (post hoc), for example 0.00 to 2.61 on ag_news; not as registered, because LoRA with a fixed learning rate overfits after 4 to 32 steps.
Soft prompts aligned with the task family lower the transition layerQwen2.5 with 8 prompt-tuned tokensHolds with 4 demonstrations on both sentiment targets for 1.5B and 3B (for example layer 17 against 21); not on ag_news or with 16 demonstrations.

As you anticipated, the outcomes are informative about scope. The depth transition and the effect of alignment carry over qualitatively, while the theory's constants and the register mechanism are specific to linear attention. All checks and their outcomes are in App. J.9 and J.10.

Thank you for the detailed response, and in particular for the additional experiments in more realistic settings. These results address the concern I raised about empirical validation and strengthen the practical relevance of the theory.

I acknowledge that the other reviewers have raised different concerns, and I respect their perspectives. In my view, however, the paper is theoretically rigorous and empirically well supported, and these concerns do not outweigh its substantial contributions. I therefore maintain my rating.

Reviewer_JWeBOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this work the authors investigate the relationship between LoRA fine-tuning and in-context learning sample efficiency for linear attention as a model case. The authors show that vanilla linear attention is inferior to LoRA fine-tuning in sample efficiency due to non-linearity of the LoRA updates with respect to labels. In order to combat that, the authors propose to use additional register tokens that explicitly store intermediate A and B states. Their carefully crafted linear transformer with the registers allow exact LoRA fine tuning with only forward updates. This allows to establish a critical depth where sample efficiency of LoRA and ICL match exactly. This theoretical result derive exact conditions for efficient low rank ICL learning, while the application to more realistic scenarios remains open.

Critical strengths

The analogy between registers and known examples in Transformer literature like memory tokens or soft prompts is intriguing.

Critical weaknesses
  1. The paper lacks more realistic experimental settings that could help mapping theoretical results to real optimisation dynamics in transformers.
  2. The paper is difficult to read and lacks intuitive examples. Analogies of theorem conditions to real ICL settings could strengthen the contribution.
Questions for authors
  1. Based on your theoretical results, could you predict conditions in real ICL setting where ICL sample efficiency match one of LoRA?
  2. Could you propose experiments that would support or refute your critical depth idea outside carefully crafted linear transformer examples?
2 comments

Thank you for the review, and for finding the register analogy intriguing: it is the link we now test directly in real models. We address realism and presentation, then your two questions. The predictions for the main experiments were fixed before the runs.

Realistic settings. We added three experiments outside the constructed linear transformers.

  1. Softmax transformers trained to learn regression tasks in context (App. J.9). Alignment between the pretraining and downstream task families lowers the depth needed: 2 layers when matched, 4 for a diverse mixture, and no transfer from an unrelated family. The transitions are sharp.
  2. Qwen2.5-0.5B, 1.5B and 3B on six text-classification tasks (sst2, rotten_tomatoes, ag_news, trec, subj, dbpedia_14), comparing in-context learning with rank-8 LoRA (App. J.10). Read at every layer with a tuned lens, in-context answers become decodable over a narrow band late in the network (for sentiment, near layer 15 of 24, 17 of 28 and 25 of 36) in 13 of 15 model–task pairs. The plain logit lens agrees.
  3. Soft prompts as registers (App. J.10). Following the analogy you highlighted, we prompt-tuned 8 soft tokens on one task family and tested them on unseen targets. With 4 demonstrations, prompts tuned on the matching family make predictions decodable 1 to 4 layers earlier than prompts from another family, on both sentiment targets for the 1.5B and 3B models (for example layer 17 against 21 on sst2 with 1.5B). They also lower the final loss in 12 of 18 settings. The effect is clearest with few demonstrations, as the revision states.

Presentation. Section 3 now includes a worked example that walks through the paper's three results on one concrete configuration. A new table (Table 4) maps each theorem condition to its practical counterpart and states the prediction: a label-nonlinear readout to predictions that depend on earlier ones; registers to system, soft or prefix prompts; the initialization scale λ0 to how relevant the prompt is to the task family; depth above L* to refinement steps on the demonstrations; and a structured task family (k < rd) to a narrow domain seen in pretraining. App. K adds a notation table.

Q1. When does ICL match LoRA in sample efficiency? The theory gives three conditions.

  • The in-context learner must be label-nonlinear. Label-linear attention needs at least d²/[r(2d − r)] times more examples at any depth (Prop. 3), and at matched budget this bound is attained (3.37 against 3.34).
  • Its effective depth must exceed the critical depth, which falls logarithmically as the starting point aligns with the task family.
  • Parity is the baseline only when the task family has no structure beyond its rank. When pretraining has absorbed more structure (k < rd effective parameters), ICL can need fewer examples than LoRA, at a rate of k/[r(2d − r)].

For pretrained LLMs this predicts that on tasks close to pretraining, in-context learning should match or beat LoRA once past the transition layer. Qwen2.5 is consistent with this: last-layer ICL needs 0.01 to 0.4 demonstrations per LoRA example on sst2, rotten_tomatoes, dbpedia_14 and trec, while ag_news is the exception (2.6 to 3.2). We present this qualitatively, since LoRA was run with one learning rate.

Q2. Experiments that could support or refute the critical-depth idea. We designed three, each with a stated way to fail.

  • Abrupt transition. In-context predictions should become decodable abruptly with depth, not gradually. Passed in 13 of 15 model–task pairs.
  • Alignment moves the transition. Better-aligned starting points should lower the depth needed. Passed in softmax transformers (2 against 4 layers), and visible for soft prompts in Qwen2.5 with 4 demonstrations on sentiment.
  • Budget dependence. At matched budget, the ICL-to-LoRA rate should rise with LoRA's step budget. This holds in all five synthetic settings. In Qwen2.5 the same rise appears against the best LoRA checkpoint at each budget, for example 0.00, 0.11, 0.78 and 2.61 on ag_news with the 1.5B model and 16 examples; with a fixed learning rate and no early stopping, LoRA overfits after 4 to 32 steps, so this reading is post hoc.

All three tests, with every pre-registered check and its outcome, are reported in App. J.9 and J.10.

I thank the authors for their response, addressing my core concerns. The paper's readability and its connection to real setups has been improved, therefore I am willing to increase my score.

Reviewer_9UE8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares in-context learning and low-rank fine-tuning within a shared linear-regression setting, asking how many in-context demonstrations are needed to match fine-tuning on a given number of examples. It shows that estimators restricted to linear dependence on the demonstration labels cannot fully exploit the task’s low-rank structure. The authors then construct an attention mechanism with additional memory tokens that exactly implements gradient descent on low-rank factors. For this mechanism, they derive a critical-depth law describing how long learning takes to move beyond a weak initialization. With sufficient optimization, in-context learning and fine-tuning approach the same statistical performance. Synthetic experiments test the predictions across initialization, step size, noise, dimension, and task structure, while trained linear-transformer experiments investigate whether learned models follow the constructed mechanism.

Critical strengths

The paper takes on an interesting and timely question: putting ICL and fine-tuning on a common statistical footing. Its analysis appears technically careful. The escape-time bound seems to improve on continuous-time approximations, and the distinction between label-linear and label-nonlinear attention is conceptually useful. The claims are checked extensively against closed forms and simulations, and the authors are transparent about their assumptions and scope.

Critical weaknesses
  1. Under the register assumption, ICL and LoRA appear to become the same algorithm. The depth-indexed exchange rate may then mostly reflect the difference between a limited step budget and full convergence, rather than a difference between the two mechanisms. Clarifying what this quantity captures would help.
  2. The main theorem relies on idealized assumptions, such as aligned initialization and hand-constructed weights. The paper's own trained models seem to reach good performance without the analyzed mechanism. It is therefore unclear how far the critical-depth conclusions extend to trained or real transformers.
  3. The experiments are synthetic and small-scale, so the practical implications remain somewhat speculative.
Questions for authors

Please see weakness

2 comments

Thank you for the careful summary and for noting the paper's technical care and transparency. Each of your three points led to new experiments, summarized below. The predictions for the main experiments were fixed before the runs.

1. What the depth-indexed rate captures. You are right: under the register assumption ICL and LoRA run the same algorithm, so comparing a depth-capped run with a converged one measures budget. The revision now separates two quantities.

  • The budget rate compares register ICL at depth L with converged LoRA. It measures the step budget, and the abstract now says so.
  • The mechanism rate compares an in-context mechanism that differs from LoRA, with the same budget L on both sides. For label-linear (GD-form) attention against gradient-trained LoRA (d = 20, r = 3, m = 140), it is 0.65 at L = 10 and 3.37 at large depth, with 95% intervals that exclude 1 away from the crossing. It replicates in all five (d, r) settings we tried, its large-depth value attains Proposition 3's bound (3.34), and Theorem 1 locates where it crosses 1. Details are in our response to Reviewer bGzs and the new Fig. 4.

2. Idealized assumptions and trained models. Three new results show how far the conclusions extend.

  • Beyond aligned initialization. Theorem 1's slope predicts the critical depth of factored descent from random initialization at finite K to within 4% in all 10 settings tested (five configurations × two step sizes, 2,000 trials each; App. C). For the preconditioned update, Proposition 6 already gives the theorem exactly at every finite K.
  • What trained models do instead. Trained full-head linear transformers compute shrunken reduced-rank regression. Per task, their estimate lies within 1.4% of it at depth 6, closer than to plain RRR (3.6%) or to the best spectral filter of OLS (5.6%), and the shrinkage they apply matches the optimal curve (App. J.8). This is how they reach the floor in a few layers without the constructed mechanism.
  • Where the depth law applies. Controlling the scale and alignment of registers in trained models leaves the depth to the floor unchanged (slope 0.0 layers per decade; App. J.7). The critical-depth law therefore describes factored descent, which LoRA and the register construction share. Section 8 now states this scope explicitly, and models with 8 to 16 layers now train stably (App. J.6).

3. Beyond synthetic linear models. We added two tests outside linear attention.

  • Softmax transformers (4 heads, width 128, MLP blocks; App. J.9). Pretraining on the downstream task family reaches 1.1× the RRR floor at 2 layers and goes on to the family-oracle risk (0.339 against 0.340). A diverse mixture of families reaches it at 4 layers, and an unrelated family does not transfer at any depth. This is the ordering App. I predicts: alignment with the task family lowers the depth needed.
  • Qwen2.5 models (0.5B, 1.5B and 3B; six text-classification tasks; App. J.10). Read layer by layer, in-context answers become decodable over a narrow band of layers late in the network in 13 of 15 model–task pairs. This is consistent with a critical-depth transition; we treat it as qualitative, since the theory's constants are not expected to transfer.

The revision reports every pre-registered check with its outcome (App. J.7 to J.10).

I want to thank the authors for the careful clarifications which address my questions. I would like to keep my current score as weak acceptance.

Reviewer_bGzsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares in-context learning and LoRA on one linear model, a rank-rr map estimated from KK demonstrations or mm fine-tuning examples and asks how many demonstrations match an mm-example fine-tune at depth LL.

For aligned and balanced initialization, the author derived an exact scalar discrete recursion and two-sided hitting-time bounds with offsets independent of initialization scale. Synthetic simulations examine this transient, task-family oracles, warm starts and learned linear transformers. The principal qualification is that the theorem concerns one constructed optimizer, not arbitrary transformer depth: random finite-sample dynamics are largely empirical, registers are not necessary, and trained full-head models reach the reduced-rank floor without selecting the constructed mechanism.

Critical strengths

A clear decomposition of the ICL v.s fine-tuning gap into a statistical floor (Props. 1, 5), what attention can store (Props. 3, 9, 11), and optimization budget (Thm. 1), shown side by side with forward passes of five constructed mechanisms on the same tasks

Critical weaknesses

The headline depth-indexed rate measures an optimization budget, not an ICL v.s. fine-tuning difference. Register ICL and LoRA are the same map (Lemma 1), so K(m,L)K(m,L) compares L steps of factored GD with the same algorithm run to convergence. Stopping fine-tuning after LL steps gives K=mK^=m at every depth. What remains is the known small-initialization escape of factored GD, with new discrete-time, λ0\lambda_0-uniform constants proved only for task-aligned balanced initialization in the population limit (Thm. 1, Assumption 2). Prop. 3's rate is the parameter-count ratio d2/r(2d−r)d^2/r(2d-r) of an LMMSE bound under an isotropic prior. I feel the technical advance is modest for this framing.

Questions for authors

if fine-tuning is also stopped after LL factored steps from the same initialization law, do you agree K(m,L)=mK(m,L)=m at every LL (Lemma 1)? If so, the abstract should say that the divergence below L∗L^{\ast} measures ICL's depth-capped budget, as p. 9 already does. I would raise my score if a natural budget-matched comparison in which the rate still departs from 1 for reasons other than budget

Please run the test you name as the natural next step (p. 9): in the App. J.6 setting, control the initialization scale or alignment (e.g., of the learned registers) and check whether the depth at which trained models reach the RRR floor scales with log⁡(σr/λ0)\log(\sigma_r/\lambda_0). A positive result would be welcomed, a null result should be reported

3 comments

Thank you for a precise and constructive review, and for stating what would change your assessment. We ran both experiments you asked for, together with tests of Theorem 1 beyond its assumptions. The predictions for the main experiments were fixed before the runs. In brief:

  • A budget-matched rate that departs from 1. With the same L on both sides, the exchange rate is 0.65 at L = 10 and 3.37 at large depth, with 95% intervals that exclude 1 away from the crossing. It replicates in all five (d, r) settings we tried.
  • Theorem 1 locates where that rate crosses 1, and its slope predicts the critical depth from random initialization at finite K to within 4% in all 10 settings tested.
  • Your register test, run as specified. The depth trained models need is the same at every register scale and alignment. This pins down the theorem's scope, and we identify what those models compute instead.

Q1. Matched budget. We agree: if fine-tuning is also stopped after L factored steps from the same initialization, K* = m at every L (Lemma 1), as p. 9 notes. The abstract now says so directly. The informative comparison is with an in-context mechanism that differs from LoRA: label-linear (GD-form) attention, where one layer is exactly one gradient step. Both arms get the same L (d = 20, r = 3, m = 140; LoRA trained by gradient descent from its standard initialization, B = 0):

L (layers = steps)101520305080 and above
K*/m0.651.663.073.543.343.37
95% interval0.57 to 0.791.39 to 2.352.47 to 4.223.18 to 3.833.27 to 3.413.30 to 3.45

Because the budgets are equal, the departure from 1 comes from the two algorithms, in two regimes:

  • Large depth: label-linearity. The rate settles at 3.37, within the interval of Proposition 3's value for this setting (3.34). The bound is attained at matched budget, so it is the operative constant rather than a loose one.
  • Small depth: the factored plateau. Gradient-trained LoRA must first escape its small initialization; label-linear attention has no such plateau. Theorem 1 predicts where this ends: shifting LoRA's initialization scale by one decade moves the crossing by 9.5 steps, against 8.9 predicted (d = 10, r = 2).

The crossover appears in all five (d, r) settings we tried, with d from 10 to 64 and both initializations. The large-depth rate tracks d²/[r(2d − r)] across a fourfold range (1.85 to 7.66, at 93 to 94% of the asymptotic value, as expected at finite m), and the crossover persists when each arm uses its own tuned step size (new Fig. 4, App. J.11).

Q2. Register initialization. We ran the test you describe in the App. J.6 setting, with registers frozen at scales 10⁻³, 10⁻², 10⁻¹ and 1, and separately aligned with the task family at overlap 0, 0.5 and 0.9. The depth at which trained models reach the floor is the same at every scale and every alignment. Across scales the fitted slope is 0.0 layers per decade (95% interval −1.2 to 0.0), which excludes the roughly +20 layers per decade the register mechanism would need. Following your example, we also trained the registers themselves from each scale: both seeds reach the floor at depth 4 at every scale, frozen or trained (risk/RRR 0.996 to 1.009), so the slope there is exactly 0. The registers are not simply ignored: at scale 1 they contribute as much to the first-layer update as the demonstrations do.

This is the null result you allowed for, and it is informative. It confirms the scope stated in Section 8: Theorem 1 is an exact law for the factored-descent mechanism, and trained models reach the floor by another route. We identify that route. Trained full-head models compute shrunken reduced-rank regression: per task, their estimate lies within 3.2% of it at depth 4 and 1.4% at depth 6, closer than to plain RRR (4.8%, 3.6%) or to the best spectral filter of OLS (6.4%, 5.6%). The shrinkage they apply matches the optimal curve (0.964 against 0.966 on signal directions, 0.005 against 0.003 on noise), which also explains why they slightly beat the RRR floor. Section 8 now reports these results in place of the "natural next test" sentence (details in App. J.7 and J.8).

Scope of Theorem 1. Both parts of this remark are now addressed. Finite K: Proposition 6 (App. A.6) already shows that for the preconditioned update the theorem holds exactly at every finite K. Random initialization: the predicted slope matches the measured critical depth from random symmetric initialization at finite K to within 4% in all 10 settings tested (five (d, r, K) configurations × two step sizes, 2,000 trials each; ratio 0.96 to 1.02), and an additive offset fits 10 to 27 times more tightly than a multiplicative one, as the theorem's form predicts (App. C). The proof itself covers aligned balanced initialization, as Section 8 states; the new evidence is reported in Section 6 and App. C.

On the contribution. The reframing sharpens the paper's claims: a matched-budget exchange rate with two regimes, whose crossing Theorem 1 locates and whose large-depth value is set by Proposition 3, a bound the rate attains to within 1%; discrete two-sided escape-time constants that predict random-initialization critical depths to within 4%; an exact register construction; and an account of what trained models compute instead. We also now describe Proposition 3 as a short linear-MMSE argument whose content is its reach. The general response reports a softmax-transformer test and experiments on Qwen2.5 models.

Changes in the revision. Abstract (matched-budget reading); Section 7 and new Fig. 4 (matched-budget rates); Section 8 (register results and scope); App. C (random-initialization slopes); App. J.6 to J.8 and J.11 (deep models now train stably, every register arm, the mechanism analysis, the five-setting replication). All code are in the updated supplement.

Thank author for the thorough response. It answered my questions and the framing and scope are now much clearer.

I keep my rating at 2. My remaining concern is significance. I am not sure what does knowing the exchange rate tell the community about ICL or SFT in practice. The results are derived for a linear model and authors admit they do not expect to transfer. In the Qwen 2.5 experiments, the exchange rate varies widely across tasks. The theory explains those variation qualitatively. But in practice, the rate would be measured per task rather than predicted. Also the real choice between ICL and SFT usually depends on lots of other factors.

I see this work as a rigorous analysis of a linear model. But the significance is limited.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for careful and constructive reports. The requested experiments have been run, and they strengthen the paper. A revised PDF with all changes is uploaded, together with an updated supplement containing every script. The predictions for the main experiments were fixed before the runs.

1. Two exchange rates, now separated (bGzs, 9UE8). We agree that under Assumption 1 register ICL and LoRA run the same algorithm, so stopping fine-tuning after L steps gives K*/m = 1 at every L (Lemma 1). The revision names two quantities. The budget rate compares register ICL at depth L with converged LoRA; Theorem 1 locates its transition exactly. The mechanism rate compares estimators that differ in what they can represent, and it departs from 1 even at matched budget. As bGzs requested, both arms now receive the same L: label-linear attention with L layers against LoRA stopped after L steps (new Fig. 4; d = 20, r = 3, m = 140, 200 tasks, 95% bootstrap intervals).

L (layers = steps)1015203050≥ 80
K*/m, LoRA from B = 00.651.663.073.543.343.37 [3.30, 3.45]

At large depth the rate attains Proposition 3's bound (3.34). Below a crossover depth label-linear attention needs fewer demonstrations than equal-budget LoRA, which must first escape its initialization; from a symmetric initialization at scale 10⁻³ the rate starts at ≤ 0.04 and crosses 1 near L = 32. The crossover appears in all five (d, r) settings tested, d from 10 to 64, and moves with LoRA's initialization scale at the slope Theorem 1 predicts (App. J.11).

2. What trained models do, and where the depth law applies (bGzs, 9UE8, GQqU).

  • Mechanism (App. J.8). Trained full-head linear transformers compute shrunken reduced-rank regression. Per task, their estimate lies within 3.2% of it at depth 4 and 1.4% at depth 6, closer than to RRR (4.8%, 3.6%) or to the best spectral filter of OLS (6.4%, 5.6%), and the shrinkage they apply converges to the optimal curve. This explains why they reach the floor in few layers and slightly beat RRR.
  • Register control, the test named on p. 9 (App. J.7). Depth to the floor does not change with register scale or alignment, frozen or trained: slope 0.0 layers per decade, an interval that excludes the roughly +20 layers per decade the register mechanism would need. Theorem 1 is thus an exact law for factored descent, the mechanism LoRA and the construction share.
  • Theorem 1 beyond its assumptions (App. C). From random initialization at finite K, its slope predicts the critical depth to within 4% in all 10 settings tested (2,000 trials each).
  • Deep models (App. J.6). Models with 8 to 16 layers, which diverged in the submission, now train stably.

3. Beyond linear attention (JWeB, GQqU, 9UE8).

  • Softmax transformers (App. J.9). Pretraining on the downstream task family reaches the floor at 2 layers and then the family-oracle risk; a diverse mixture needs 4 layers; an unrelated family does not transfer. This is the ordering App. I predicts.
  • Qwen2.5-0.5B, 1.5B, 3B on six classification tasks (App. J.10). In-context predictions become decodable over a narrow band of layers in 13 of 15 model–task pairs, consistent with a sharp critical-depth transition, and soft prompts tuned on the matching family move that band 1 to 4 layers earlier with 4 demonstrations. App. J.10 reports every pre-registered check with its outcome.

4. Readability (JWeB). Section 3 now has a worked example that walks through the three main results, Table 4 maps each theorem condition to its practical counterpart, and App. K gives a notation table.

Summary of revisions. Abstract, introduction and contributions name the budget and mechanism rates; Sec. 7 and Fig. 4 (matched-budget rate); Sec. 6 and 8 (mechanism, register result, scope); App. C (random-initialization slopes); App. J.6–J.11 (new experiments); App. K (notation); App. G (disclosure); supplement with run_all.sh and all rebuttal code. The theory is unchanged.

AuthorsOpenReview ↗

Revised updated supplementary material

Dear Area Chair and Reviewers,

We thank you again for your time and for the constructive discussion. We have uploaded updated supplementary material that reflect the points raised in the reviews. We summarize the changes briefly below for your convenience.

Updated supplementary material

A top-level README maps each section, figure and appendix of the revised paper to its script and the exact command used to run it. code/: the originally submitted code, unchanged. rebuttal/: the experiments added in the revision. It includes the pre-registration documents, kept exactly as written before the runs. extra/: a small set of additional runs supporting Appendices J.6, J.7 and J.11. These were not pre-registered and are labelled as such. The folder includes their logs and a single command to rerun each one. All scripts use fixed seeds, and the supplementary material is anonymized.

We have also uploaded a pdf file in supplementary with the changes highlighted in blue for reference, which highlight the changes from original submission.

We would be glad to clarify any point or provide further details during the remaining discussion period.

Kind regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.