acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws For Attention-To-Recurrence Distillation

Abstract

An efficient way of reducing the computational cost of pretrained large language models is to generate hybrids by replacing some of the traditional attention layers, whose complexity scales quadratically with context length, with recurrent alternatives such as state-space models, which scale linearly with context length. Approaches such as MOHAWK allow distilling hybrids under limited token budgets compared to output-level Knowledge Distillation (KD), while retaining model capabilities. Yet, they lack the theoretical foundation needed to guide practitioners on how many full-attention layers can be replaced and at what cost. Here, we aim to close this gap. Using arguments from spectral theory, we argue for a conditional lower bound on the Kullback–Leibler (KL) divergence loss between hybrid and non-hybrid models in a restricted componentwise setting, connecting it to the attention matrix approximation error in MOHAWK’s matrix-orientation phase. We also study a scaling law for the context-dependent KL loss based on the ratio of replaced layers, their expressivity and the Phase 2 token budget. Finally, we verify experimentally that the KL divergence loss often exhibits power-law decay within a fixed conversion setup, while finding that its late-budget shape does not transfer cleanly across model sizes. Overall, our work sets the foundation for a theory that characterises the practical utility of distilling hybrids from pretrained models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling laws, Distillation, LLMs, SSMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1qi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how the teacher–student imitation gap scales with training data when converting pretrained Transformers into recurrent or hybrid models using MOHAWK. It combines a spectral analysis of attention matrices with a conditional KL lower bound for a controlled single-head setting, and evaluates empirical power-law forecasts across fifteen model-size and replacement-ratio configurations. Additional experiments examine context length and recurrent state size. The results show that early-budget fits provide approximate forecasts within fixed configurations, while late-budget trajectories differ across model sizes and larger recurrent states do not consistently reduce final KL. The paper advocates treating architectural conversion as a conditional scaling problem and documents limitations of a simple joint scaling description.

Critical strengths

The paper offers a useful diagnostic distinction between representation capacity, teacher imitation, and absolute language-modeling quality. In particular, the matched-context evaluation shows that student cross-entropy can decrease while teacher–student KL increases (§6.4), demonstrating why aggregate language-modeling loss alone can obscure a conversion gap. The authors also report results that challenge their initial hypothesis, including the non-monotonic effect of state size (§6.3), and explicitly acknowledge the restricted scope of their theorem. Together, these observations motivate a more careful examination of how local approximation constraints relate to complete-model behavior.

Critical weaknesses
  1. The central forecasting result has not demonstrated an advantage over elementary baselines. Section 6.2 reports a median absolute percentage error of 8.5% when forecasting D64 from D1–D32, arguing that this is “small enough to help decide whether another doubling is likely to be worthwhile” (p. 7, lines 355–359). However, the paper omits the basic baseline of predicting D64 using the observed D32 KL. oreover, accurately predicting the KL level does not necessarily mean accurately predicting the marginal benefit of additional training.
  2. The late-budget reversals are confounded by learning-rate selection, weakening their interpretation as evidence about conversion scaling. The paper explicitly states that “Lower learning-rate runs did not show the same late reversal” (§6.1, p. 6, line 322). Nevertheless, the primary learning rate was calibrated only at (r=0.5), using approximately 18.9M–47.1M tokens, and then applied to all replacement ratios and budgets up to approximately 1.21B tokens (§5.2 and Appendix B.3). The lower-rate runs are described without their complete configurations, curves, or quantitative results. A common recipe provides a valid descriptive comparison, but the current evidence does not distinguish conversion-related differences from a correctable mismatch between learning rate and training horizon. This concern is reinforced by Bjorck et al., Scaling Optimal LR Across Token Horizons (ICLR 2025, §3.1), which empirically demonstrates that longer horizons favor smaller learning rates.
  3. The theoretical bound remains quantitatively disconnected from the actual distillation experiments. Theorem 1 assumes “Residual excitation” and “Downstream observability,” freezes the value matrix and downstream mapping, and holds other contributions fixed or removes them (§4.2; Appendix A.1–A.2). These assumptions specify conditions under which a matrix discrepancy survives to observable logits, but the paper neither measures the corresponding coefficients nor establishes their relevance in the trained models. Furthermore, (\lambda_F=V\rho^2) depends on the minimum probability along the evaluated teacher–student logit interpolations; strict positivity alone does not ensure a numerically informative bound. The authors acknowledge that “We do not establish that condition or a power-law rate in training data” (p. 5, lines 238–240), and identify Equation (6) as an empirical candidate rather than a derived law. Consequently, the theorem does not yet explain the measured Phase-2 scaling behavior.
  4. The organization and presentation obscure the logical development and require more substantive editorial revision. The introduction moves quickly from naming MOHAWK to “Our first hypothesis was deliberately simple” and “Parts of this picture survive” (p. 1, lines 43–48), presenting the research narrative before sufficiently establishing the methodological context. Although MOHAWK receives a brief functional description, “Phase 0” is invoked in line 46 before its stages are explained in §3 (p. 3, lines 117–129). The introduction also does not adequately explain why this particular conversion framework is the appropriate starting point for the broader architectural questions posed. Subsequently, §3 combines established matrix-mixer background, a review of the inherited MOHAWK procedure, and the paper’s response-variable definition, making the boundary between prior knowledge and the present contribution unnecessarily difficult to follow. Clearly identified preliminaries, more focused positioning in §2, and a more explicit progression from the research question to the assumptions, analysis, and empirical tests would improve the argument substantially. The authors disclose that LLM assistants suggested changes to “wording, structure and length throughout the paper” and “drafted parts of the related-work and appendix text” (p. 9, lines 479–481). This transparency is appreciated. However, the unresolved organizational problems and the gap between broad introductory promises and narrowly validated conclusions raise a concern about possible overreliance on AI-assisted writing without sufficient substantive revision.
  5. There is also a concrete formatting issue: the abstract contains two paragraphs (p. 1, lines 11–18 and 19–28), whereas the official ICLR 2027 LaTeX template, in iclr2027_conference.tex, lines 59–60, explicitly states: “The abstract must be limited to one paragraph.” The Author Guidelines require use of the provided template.
Questions for authors
  1. Can the authors verify this comparison using unrounded results and evaluate last-endpoint and local-trend baselines, including their ability to predict the direction and magnitude of further training gains?
  2. Can the authors report the existing lower-rate controls and determine whether the cross-size forecasting failures persist under a learning-rate policy calibrated across training horizons?
  3. Can the authors quantify the bound in the controlled setting, or demonstrate that spectral residuals—or their value- and downstream-weighted versions—predict KL on held-out configurations?
4 comments

We thank the reviewer for the careful reading and detailed feedback, which has helped us improve the manuscript. We address the concerns in order.

1. Forecasting baselines. We refer to “Early-budget forecasts” in our shared response. We compared three methods: the latest checkpoint (Last), extrapolation of the most recent absolute KL change (Trend), and our power law (Power). At the earlier budgets, Last performs substantially worse than the other two, so both extrapolation methods improve on this simple baseline. Trend and Power give closer results, with Trend achieving a lower pooled RMSE. For forecasts two doublings ahead, Power performs better at 530M and 750M, although Trend remains better at 300M.

2. LR sensitivity. We refer to “LR calibration and late-budget KL reversals” in the shared response. For the OLMo-1 750M student with 25% attention replacement, KL rose by 17.81% from D32 to D64 under the original LR of 1e-4. We recovered a continuation from D32 using LR 3e-5 and cosine decay. Its D64 KL was 37.08% below the original D64 result, with no reversal. This shows that the rebound depends on the optimization recipe and is not unavoidable for this architecture. Since both LR and scheduler changed at D32, the result does not isolate LR as the cause or establish an optimal configuration. It is consistent with the possibility that a high LR becomes less suitable as the curve flattens.

3. Matrix-to-KL connection. As described in “Matrix-to-KL bound”, we tested two controlled approximations: A, a trained recurrent mixer, and B, the optimal rank-constrained approximation from truncated SVD (Eckart–Young). We evaluated four windows, three state sizes (16, 64, 256), and one early layer. We replaced one head's cross-cut block while keeping the teacher values and other pathways fixed. In all six cases, matrix error propagated to the head output and logits, producing nonzero KL. B had much lower errors than A, with the largest A/B KL ratio at state size 256. Incomplete optimization at this state size is one possible explanation, but we have not isolated the cause.

We also computed observed ratios related to residual excitation (head²/matrix², ν_eff), downstream observability (logit²/head², μ²), and the KL factor (nKL/logit², λ_F/2). In the table, matrix² and head² denote squared Frobenius errors; logit² is the squared centered-logit difference. Brackets give the reported ranges. The last two columns use the scale factors shown in their headers.

CasedMatrix²/tail²Head²/matrix²Logit²/head² (×10³)nKL/logit² (×10⁻⁶)
A1637.1 [33.2–41.3]5.39 [4.24–6.73]1.75 [1.33–1.97]2.95 [2.64–3.25]
A64257 [204–311]5.48 [4.30–6.83]1.73 [1.33–1.96]2.96 [2.62–3.27]
A2567950 [6970–9940]5.99 [4.71–7.04]1.58 [1.17–1.77]2.84 [2.56–3.24]
B161.004.43 [3.06–5.46]2.21 [1.75–2.68]2.68 [2.02–3.32]
B641.007.61 [5.31–11.18]5.56 [4.60–6.15]2.50 [2.04–2.92]
B2561.0012.13 [11.53–12.64]76.3 [66.7–90.6]2.48 [1.99–2.99]

These measurements show that the approximation error remains observable downstream. The ratios help illustrate the possible magnitudes of the theorem's quantities, but they are not estimates of its uniform coefficients. We have added details to Section 4.2 and Appendix A. We will experiment with more checkpoints to provide robust empirical support for the theorem.

4–5. Presentation and formatting. We have rewritten the abstract as one paragraph and changed the title to “Conditional Scaling Laws for Attention-to-Recurrence Distillation”. We also revised the introduction and contributions to clarify the paper's scope. Further presentation improvements are underway, taking the reviewer's suggestions into account.

Questions: We address the three questions in the same order.

  1. Question 1: Addressed in point 1.
  2. Question 2: If we understood the question correctly, it has two parts:
    • (a) Report the lower-LR results: Addressed in point 2.
    • (b) Determine whether cross-size forecasting failures persist under horizon-dependent LR calibration: This would require further experiments. Our current results do not establish whether these failures persist under such calibration. We will therefore remove any unsupported claims from the paper unless further evidence becomes available.
  3. Question 3: Addressed in point 3. We will also evaluate more checkpoints to provide stronger empirical support for the theorem's proposed mechanism.

I appreciate the authors for adding experiments and providing clarifications. The new prediction baselines support the general claim that observations under early budgets can help predict the later KL. However, Trend performs comparably or even better, leaving the value of power-law modeling itself unclear. Please further elaborate why power-law modeling is needed and what additional scientific insights or practical benefits it offers over simple local trend extrapolation.

In addition, I would like the authors to present a clearer and more specific revision plan for the paper, including which claims will be retained, narrowed down, or removed, and how the newly added evidence will change the main conclusions of the paper. In particular, the revised discussion should adequately reflect the comparisons against prediction baselines, the sensitivity of curve inversion to learning rates, and the scope of applicability of controlled theoretical validation. Without a sufficiently explicit revision plan, I am currently unable to judge whether the contributions and supporting evidence of the revised manuscript justify acceptance.

AuthorsOpenReview ↗

Trend vs Power

We agree that the short-range comparisons give limited grounds for preferring the power law. A local trend can closely approximate a smooth power law over a small budget interval. We retain the power law as a testable hypothesis about whether each doubling yields an almost constant proportional reduction in KL across a wider range of distillation budgets under our MOHAWK conditioned recipe.

Our current results show that Trend has lower pooled forecast error, with advantages for Power in some model-size and horizon combinations. A rough backtest on the seven plotted observations in Kaplan et al.’s dataset-size curve illustrates why the horizon matters. Trend and Power had mean absolute percentage errors of 1.62% and 0.95% one doubling ahead (four forecasts), and 4.23% and 2.67% three doublings ahead (two forecasts). Chinchilla offers a complementary example of testing an extrapolation at scale. Hoffmann et al. used fits from smaller experiments to choose a model size and token budget, then tested that allocation at Gopher’s compute budget.

Kaplan et al., Figure 1: https://arxiv.org/pdf/2001.08361#page=3

Hoffmann et al., §§3–4: https://arxiv.org/pdf/2203.15556#page=7

We think a more discriminating test for distillation would do three things.

  1. Calibrate peak learning rates at early and late budgets, with the decay schedule matched to each target budget.
  2. Repeat representative configurations across independent seeds to measure run-to-run variation.
  3. Compare Power and Trend at farther held-out budgets, using the same available checkpoints for each forecast.

This would test whether the fitted exponent is reproducible across seeds and fitting windows, and whether it improves longer-range forecasts beyond the latest observed trend.

This test is within our computational capabilities in the next few days and can be added to the final version if accepted. We compromise to this.

We are finishing the answer for the plan

AuthorsOpenReview ↗

Revision plan

We group the changes by topic. Section 1 is already in the revised manuscript. Sections 2–4 rely on results already reported in this discussion, with two planned additions: a longer-range comparison of Trend and Power (Section 2) and the single-head experiment on more checkpoints (Section 1). Beyond the points listed, we will revise the writing of the full paper for clarity.

1. Theory and controlled validation (done)

  • Added: Section 4.3 and Appendix A.4 report the single-head experiment: a trained mixer (A) against the best rank-d matrix from SVD (B), for d_state = 16, 64, 256. One table follows the error from the cross-cut block to the head output, the centred logits and the KL. A second table compares A and B (KL is 40–91× lower for B).
  • Narrowed: The three measured ratios are reported as observed magnitudes, not estimates of the theorem's constants. After Theorem 1 we state that λ_F is conservative and that the bound is structural, not a numerical forecast of KL.
  • Scope: The appendix states the limits: one head, four windows, one block replaced in the teacher, no trained-student KL and no uniform constants. The tail falls by 99.5% from d = 16 to 256 while the KL of the trained mixers does not fall, so we do not present the tail as a predictor of trained KL.
  • Removed: "close this gap" and "sets the foundation for a theory" from the abstract. The title is now "Conditional Scaling Laws…", and the abstract is one paragraph. The abstract, introduction and contributions state that the theory and the forecasts are separate, and that the forecasts use Phase-2 measurements only.
  • Planned: As stated in our first reply, we will repeat the single-head experiment on more checkpoints to make this result more robust.

2. Early-budget forecasts (Section 6.2)

  • Retained: Within a fixed setup, early Phase-2 endpoints predict later KL much better than the last checkpoint (pooled RMSE 0.034 for Last, 0.005 for Trend, 0.009 for Power, one doubling ahead). The D64 test and its errors structured by model size also stay.
  • Added: Both baseline tables, with the plain statement that Trend has the lower pooled RMSE and that Power is better at 530M and, two doublings ahead, at 750M.
  • Planned: We will add a longer-range comparison of Trend and Power, as outlined in our previous comment. We expect Power to predict better at long range, and we commit to reporting the results whichever method they favour.
  • Narrowed: The power law is presented as a description of each curve and a testable hypothesis (a near-constant proportional reduction per doubling), not as a better forecasting method. The section title "Early budgets forecast a useful range" will change accordingly.
  • Removed: "small enough to help decide whether another doubling is likely to be worthwhile" and "well enough to support run planning". We claim no compute saving.

3. Late-budget reversals and learning rate (Section 6.1, Appendix B.3)

  • Replaced: The sentence "Lower learning-rate runs did not show the same late reversal" becomes the matched control (750M, r = 0.25: +14.2% at 1e-4, −26.8% at 3e-5, difference 0.046 nats/token, 95% interval [0.039, 0.054]) and the cosine continuation, each with its table.
  • Retained: The fixed-rate curves remain the primary comparison, including the continued improvement at 300M.
  • Narrowed: The reversals are described as observations under one learning rate calibrated at short budgets. "The late-budget shape does not transfer cleanly across model sizes" gets the same qualifier, as do the negative exponents and positive curvature at 530M and 750M (Table 7).
  • Removed: Wording that attributes the reversals to model size or conversion quality. We state that the control covers one configuration, and that cross-size behaviour under a horizon-calibrated rate is open.

4. Other changes

  • OLMo-2 1B: described as learning curves in a second teacher family, not a forecasting test or a transferable exponent.
  • Discussion and limitations: rewritten to match points 1–3, including the outcome of the two planned additions.
  • Presentation: Section 3 will be split into background (matrix mixers, MOHAWK) and our own setup (response variable). The introduction now explains the three phases before the hypothesis.

Revised main conclusions

  1. Under a fixed recipe, early budgets forecast later KL. The power law describes these curves, but it is not shown to forecast better than a local trend.
  2. The late reversal is sensitive to the learning rate in the tested configuration. We do not present it as a property of the architecture.
  3. The spectral result is a representation limit whose effect on KL we traced in one head. It is not a full-model floor and does not explain the Phase-2 scaling.
Reviewer_rtTcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is focused on distilling self-attention-based pretrained model (or parts of it) into SSM models (or, SSM/self-attention hybrids) using a recently proposed MOHAWK (Bick et al., 2024), which allows for bypassing costly pretraining of SSM models. MOHAWK uses three phases, and the manuscript proposes techniques that could allow estimating final quality based on measurements after only the first phase.

Critical strengths

The experimental results concerning the prediction of the final KL show the predictions are relatively accurate.

Critical weaknesses

It is unclear how the theoretical results in Section 4 influence the actual predictive methodology in Section 5.

The practical savings are limited to two-fold or four-fold since the approach uses D16 or D32 (values from quarter or half of the full D64 run).

Questions for authors

In addition to weaknesses above, it would be very useful to go beyond the full grid analysis (as in Table 1) and show one use case of how the methodology actually significantly improves efficiency the distillation process. It would be especially useful if the use case would use a somewhat different model than the one on which extensive results have been quantified.

1 comment

Thank you for the review and for noting the accuracy of the KL forecasts. We clarify how the theory relates to forecasting and how the method can inform practical distillation decisions.

1. Relation between theory and forecasting.

Section 4 establishes a representation constraint: the spectral tail limits how closely a finite-state recurrent mixer can approximate an attention cross-cut, and Theorem 1 relates this error to KL in a controlled single-head setting. This motivates the study of teacher–student KL, but the bound is not used to fit the empirical scaling law. In Section 5, we fit power laws to measured Phase-2 KL at early checkpoints and predict later Phase-2 endpoints within a fixed conversion setup. Phase 0 and Phase 1 are held fixed; forecasts are not based on measurements from Phase 0. We clarified this distinction in the revised introduction, Section 3 and Section 4.2, and added a controlled matrix-to-KL diagnostic in Section 4.3 (see the shared response).

2. Earlier forecasts and training budgets.

Our new tests use D4 and D8 as forecast origins, rather than only D16 or D32 for the original D64 test. For OLMo-1 530M, across five replacement ratios and two-doubling forecasts (D4→D16 and D8→D32), our power law achieves KL RMSE 0.00926, compared with 0.01518 for the local-trend baseline and 0.03552 for the latest-checkpoint baseline. Thus, a forecast of the later endpoint can be obtained with one-quarter of the target Phase-2 budget. The shared response, “Early-budget forecasts,” reports the comparisons across all three model sizes.

3. Use case: choosing which conversions to continue.

Consider five candidate replacement ratios for a 530M teacher, with D16 as the target budget. After training each candidate to D4, we can fit its power law using D1, D2 and D4 and forecast its D16 KL. The forecasts let us compare expected distillation quality before committing the remaining three-quarters of the D16 Phase-2 tokens to each candidate, and prioritize those worth continuing. The reported RMSE quantifies the forecast accuracy available for this screening step.

We also studied a different teacher family, OLMo-2 1B, at 50%, 75% and 100% attention replacement. Between its D8 and D32 budgets (0.50B–2.01B Phase-2 tokens), measured KL fell by 36.0%, 34.1% and 31.9%, respectively. This provides a cross-family check of the budget-dependent improvements; the quantitative forecast comparison above is from OLMo-1.

Reviewer_C9XfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses MOHAWK to study how the replacement ratio and distillation budget affect teacher-to-student KL when some attention layers in a pretrained Transformer are replaced with recurrent modules. Using the rank constraint imposed by the recurrent state dimension, the authors derive a KL lower bound from the spectral tail of the teacher's attention matrix. This bound applies to a controlled single-head comparison under assumptions about error propagation. In OLMo-1, power laws fitted to early budget points within a fixed configuration roughly predict KL at larger budgets, but late-stage trends differ across the model sizes tested. Full-model KL did not decrease monotonically with larger state dimensions. In the context-length comparison, longer contexts reduced student cross-entropy and increased teacher-to-student KL.

Critical strengths
  1. A model for distillation error within a fixed conversion setup. The authors measure distillation error using teacher-to-student KL on a held-out set. For a given conversion setup and context length, they use a power law to model how KL changes with the Phase 2 token budget. They also compare context lengths separately on OLMo-1. 2. The paper derives a lower bound on matrix approximation error for recurrent layers and a conditional single-head KL lower bound.The authors use the rank constraint that the recurrent state dimension imposes on cross-cut matrices to derive a lower bound on approximation error from the spectral tail of the teacher's attention matrix. In a single-head comparison with the other pathways held fixed, they model how this error propagates to the head output and then to the logits. This gives a conditional lower bound on KL. 3. A state-dimension comparison guided by spectral analysis and a test of budget predictions. The spectral analysis shows that a larger recurrent state dimension relaxes the rank constraint on the cross-cut block. This motivates a comparison of three state dimensions in OLMo-2 1B. The authors also fit a power law to smaller budget points within each of 15 OLMo-1 conversion configurations. They compare the predicted and measured KL at �64 to quantify the prediction error within each fixed configuration.
Critical weaknesses
  1. Missing controlled single-head diagnostic. Theorem 1 (Section 4.2, Eq. (5)) links the spectral tail of the teacher's attention matrix to a KL lower bound in a single-head comparison with other pathways fixed, assuming that enough of the matrix error reaches the head output and then the logits. Figure 1 measures the teacher's spectral tail, and Figure 4 (right) reports full-model KL after training. A controlled comparison would quantify how much of the matrix error remains after multiplication by the value matrix and how much of the resulting change reaches the logits through the downstream network.
  2. Limited quantitative reporting for the lower-learning-rate runs. In Section 6.1 and Appendix B.3, the authors note that follow-up runs at lower learning rates did not show the same KL rebound. The paper describes this observation qualitatively, leaving the model sizes, replacement ratios, Phase 2 peak learning rates, training budgets, and corresponding held-out KL curves unspecified.
  3. The title suggests a broader scaling result than the experiments support. The phrase "Scaling Laws" may lead readers to expect a relationship that applies across model sizes and replacement ratios. In Figure 3, the authors fit a power law separately within each fixed configuration and predict that configuration's held-out D64 endpoint. Late-budget trends also differ across model sizes. The paper treats the power law as an approximation within a fixed configuration and over the budgets studied. The title could reflect that scope more closely (Sections 5.1 and 6.2).
Questions for authors
  1. Could the authors select a representative configuration from Figure 4 and replace only the cross-cut block for a single head, keeping the inputs, value matrix, and other pathways fixed? It would be helpful to report the matrix residual, the change in the head output, the logit difference after removing any common shift, and the resulting KL divergence.
  2. Section 6.1 and Appendix B.3 note that follow-up runs at lower learning rates did not show the same rebound in KL. Could the authors report the model sizes, replacement ratios, Phase 2 peak learning rates, and training budgets for these runs and provide the corresponding KL curves on the held-out set?
1 comment

Thank you for your careful review and for distinguishing the representation-level result from the conditional single-head KL bound. We address your three concerns below. The first two also answer your explicit questions.

1. Controlled single-head diagnostic

We performed the intervention you suggested, as described in “Matrix-to-KL bound: controlled validation” in our shared response. We used the OLMo-2 1B, 50%-replacement D32 students at state sizes 16, 64 and 256. For four matched 2,048-token WikiText-2 test windows, we changed only the 1,024 × 1,024 suffix-to-prefix block of layer 1, head 0. The inputs, teacher value vectors, other attention contributions and downstream weights were held fixed. We tested both the trained recurrent mixer (A) and a truncated-SVD rank-d oracle (B).

The shared response reports the matrix errors, head-output errors and suffix KL values for both interventions across all three state sizes. To complete the four-stage diagnostic you requested, we additionally report the centred-logit error: at d = 64, its mean squared norm across the four windows is 32,728.1 for the trained mixer and 566.618 for the rank-64 SVD oracle. Centering removes the common vocabulary shift, which leaves softmax probabilities unchanged.

dA matrix²B matrix²A head²B head²A centered logits²B centered logits²A KLB KLA/B KL
163.561790.09625118.81880.43203232569.9907.2889.32533e-52.34516e-639.8×
643.570560.01404119.16890.10393932728.1566.6189.38314e-51.36936e-668.5×
2563.918240.00050023.21670.00605136146.0456.0009.96130e-51.09732e-690.8×

The measured errors remain nonzero after multiplication by the teacher values and propagation through the downstream network. Across the three state sizes, the oracle produces 39.8–90.8 times lower controlled KL than the trained mixer. These measurements demonstrate error transmission in the controlled setting, but do not establish a full-model KL floor or uniform theorem coefficients. The probe covers one head and four sequential windows.

2. Quantitative lower-learning-rate results

We agree that the earlier qualitative description was insufficient. We report two complementary comparisons: a matched LR control and a recovered continuation that changes both LR and scheduler. In the matched control, we used the 750M DataDecide/OLMo-1 student with 25% replacement, state size 64 and seed 42. Both arms resumed the same full-state checkpoint, keeping the optimiser settings, data lineage and WSD schedule boundaries fixed. Only the subsequent peak LR differed. D32 and D64 correspond to approximately 603.46M and 1,206.91M cumulative Phase-2 tokens.

Peak LRD32 KLD64 KLRelative change
10⁻⁴0.112370.12833+14.2%
3 × 10⁻⁵0.111870.08191−26.8%

All four checkpoints were evaluated identically on 256 held-out Dolma documents at context length 2048. The target selection differs from that used for the original curves, so the absolute KL values are not directly comparable. The lower LR removes the endpoint rebound in this configuration, demonstrating LR sensitivity without establishing an optimal policy across model sizes or training horizons.

The recovered cosine continuation (peak LR 3e-5), described in the shared response, has the following held-out direct KL trajectory at context 2048. D32 is the reported starting reference; the original WSD column shows only its endpoints.

BudgetContinuation: 3e-5, cosineOriginal: 1e-4, WSD
D32 reference0.103885060.10388506
D360.09883957—
D400.09098134—
D440.08785468—
D480.08400646—
D520.08062241—
D560.07847276—
D600.07708659—
D640.076999190.12238444

The continuation improves at every measured checkpoint. Dashes mark original-recipe measurements not supplied here. Both LR and scheduler changed, so this test is recipe sensitivity. The matched WSD control above isolates LR but provides only two endpoints.

3. Title and scope

We agree that the title should make the conditional scope explicit. The revised manuscript title is Conditional Scaling Laws for Attention-to-Recurrence Distillation. The empirical fits are made separately within each conversion configuration and tested at later budgets; they do not establish a shared exponent or a universal law that transfers across model sizes and replacement ratios. The late-budget behaviour is empirically sensitive to the optimisation recipe, as the LR control demonstrates. The theoretical analysis characterises a local representation constraint, while the experiments characterize learning curves under fixed conversion recipes. We do not derive the empirical power law from Theorem 1.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To All Reviewers

Early-budget forecasts

We tested power-law forecasts at earlier budgets, rather than only D64, where some curves flatten or reverse. Each forecast uses only CKPTs available at its origin. We compare: Last: no KL change; Trend: repeat the latest doubling's absolute KL change (twice for two doublings); Power: OLS fit of log KL vs. log tokens using D1 through the origin, then extrapolate. All five replacement ratios are retained at each size. Tables report KL RMSE (lower is better).

One doubling: D4→D8, D8→D16, D16→D32 (15 forecasts/size). For D4→D8, Power fits D1, D2, D4.

SizeLastTrendPower
300M.05426.00460.01318
530M.01786.00546.00525
750M.01696.00406.00586
Pooled.03441.00474.00886

Power is slightly better at 530M; Trend wins at 300M, 750M and pooled.

Two doublings: D4→D16, D8→D32 (10 forecasts/size). For D4→D16, Power again fits D1, D2, D4 but predicts 4x the origin budget.

SizeLastTrendPower
300M.10708.01163.02192
530M.03552.01518.00926
750M.03361.01118.01071
Pooled.06797.01279.01506

Power wins at 530M and 750M (narrowly at 750M); Trend wins at 300M and pooled. Median percentage errors yield the same rankings in both tables. Weak D64 results do not mean Power fails at earlier budgets: it beats both baselines in some settings, especially 530M, but has no consistent overall advantage over Trend. These are repeated forecasts on the same configs, not independent new runs. Some pre-D64 intervals also reverse. This supplements the D64 test.

LR calibration and late-budget reversals

We recovered a lower-LR run to measure the reversal's dependence on optimization. The original Phase-2 LR, 10⁻⁴, was selected in short calibrations at r=0.5 for 300M, 530M and 750M (18.9M, 33.3M, 47.1M tokens).

For OLMo-1 750M, 25% replacement and Mamba-2 state 64, direct teacher→student KL at context 2048 rose 17.81% from D32 to D64 with the original 10⁻⁴ WSD. A recovered continuation from a recorded D32 CKPT trained another 603.5M tokens at LR=3×10⁻⁵ with cosine decay.

RecipeD32 KLD64 KLChange
WSD, 10⁻⁴.103885.122384+17.81%
Cosine, 3×10⁻⁵.103885*.076999−25.88%

D32 is the reference evaluation for the recorded starting CKPT.

KL fell across all eight (D36–D64). At D64 it was 37.08% lower than the original WSD result. The rebound therefore depends on the optimization recipe and is not inevitable for this architecture. Since both LR and scheduler changed, the gain cannot be attributed to LR alone. Our primary curves describe fixed recipes.

Matched LR control of the rebound

We also ran a matched Phase-2 LR control for the 750M DataDecide student (25% replacement, d_state=64, seed 42). Both arms resumed the same CKPT at step 33,149. Peak LR changed from 1e-4 to 3e-5; architecture, optimizer, data lineage and WSD boundaries were fixed. D32/D64 evaluations used the same teacher and 256 held-out Dolma documents (16,384 targets; context 2048).

LRD32 KLD64 KLD32→D64
1e-4.11237.12833+14.2%
3e-5.11187.08191−26.8%

Original LR rebounds; lower LR improves. The difference in D32→D64 changes is 0.04593 nats/token (paired-document bootstrap 95% CI: [0.03912, 0.05385]). At D64 the lower LR gives 36.2% lower KL. The rebound is LR-policy-sensitive, not unavoidable for this architecture. The fixed LR in our main grid ensures consistent comparisons; this control shows its importance at longer budgets.

Matrix-to-KL bound: controlled validation

We tested how attention approximation error reaches output KL using two interventions: A (trained): replace one teacher head's cross-cut block with that of the trained recurrent mixer and track matrix, head-output and downstream errors; B (optimal): use truncated SVD to obtain the minimum rank-d matrix error.

Setup: OLMo-2 1B, 50% replacement, D32 (2.01B Phase-2 tokens), d=16/64/256, layer 1 (zero-indexed), head 0, four 2,048-token WikiText-2 test windows. Only the 1,024×1,024 suffix-to-prefix block changes; teacher values, other attention contributions and the downstream network are fixed. Table (4-window means): spectral tail τd, squared Frobenius matrix/head-output errors (m²/h²), and forward KL over the affected suffix.

dτdA m²B m²A h²B h²A KLB KLA/B
16.09633.562.096318.82.4329.33e-52.35e-639.8×
64.01403.571.014019.17.1049.38e-51.37e-668.5×
256.00053.918.000523.22.0069.96e-51.10e-690.8×

We compute three observed ratios: head²/matrix² (related to ν_eff), centered-logit²/head² (related to μ²), and n·KL/centered-logit² (related to λ_F/2). Across states, A gives 5.39–5.99, 1.58–1.75×10³, and 2.84–2.96×10⁻⁶, respectively. These illustrate possible magnitudes, not direct estimates of theorem parameters, which require uniform guarantees beyond the tested cases. Errors reach outputs and produce measurable KL. Gap theoretical capacity and approx.: B gives nonzero KL but A has much larger errors.

AuthorsOpenReview ↗

Clarification/Comment

We thank all three reviewers for their careful evaluation and constructive feedback. Their comments raise important questions about the practical value of our forecasts, the influence of optimisation choices, and the connection between our theoretical analysis and experimental results.

In this common response, we addressed the main scientific concerns raised across the reviews, supported by additional analyses and experimental evidence. We focused there on the points most relevant to the paper's central contributions and addressed more specific questions and suggestions in our individual responses to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.