- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how the teacher–student imitation gap scales with training data when converting pretrained Transformers into recurrent or hybrid models using MOHAWK. It combines a spectral analysis of attention matrices with a conditional KL lower bound for a controlled single-head setting, and evaluates empirical power-law forecasts across fifteen model-size and replacement-ratio configurations. Additional experiments examine context length and recurrent state size. The results show that early-budget fits provide approximate forecasts within fixed configurations, while late-budget trajectories differ across model sizes and larger recurrent states do not consistently reduce final KL. The paper advocates treating architectural conversion as a conditional scaling problem and documents limitations of a simple joint scaling description.
Critical strengths
The paper offers a useful diagnostic distinction between representation capacity, teacher imitation, and absolute language-modeling quality. In particular, the matched-context evaluation shows that student cross-entropy can decrease while teacher–student KL increases (§6.4), demonstrating why aggregate language-modeling loss alone can obscure a conversion gap. The authors also report results that challenge their initial hypothesis, including the non-monotonic effect of state size (§6.3), and explicitly acknowledge the restricted scope of their theorem. Together, these observations motivate a more careful examination of how local approximation constraints relate to complete-model behavior.
Critical weaknesses
- The central forecasting result has not demonstrated an advantage over elementary baselines. Section 6.2 reports a median absolute percentage error of 8.5% when forecasting D64 from D1–D32, arguing that this is “small enough to help decide whether another doubling is likely to be worthwhile” (p. 7, lines 355–359). However, the paper omits the basic baseline of predicting D64 using the observed D32 KL. oreover, accurately predicting the KL level does not necessarily mean accurately predicting the marginal benefit of additional training.
- The late-budget reversals are confounded by learning-rate selection, weakening their interpretation as evidence about conversion scaling. The paper explicitly states that “Lower learning-rate runs did not show the same late reversal” (§6.1, p. 6, line 322). Nevertheless, the primary learning rate was calibrated only at (r=0.5), using approximately 18.9M–47.1M tokens, and then applied to all replacement ratios and budgets up to approximately 1.21B tokens (§5.2 and Appendix B.3). The lower-rate runs are described without their complete configurations, curves, or quantitative results. A common recipe provides a valid descriptive comparison, but the current evidence does not distinguish conversion-related differences from a correctable mismatch between learning rate and training horizon. This concern is reinforced by Bjorck et al., Scaling Optimal LR Across Token Horizons (ICLR 2025, §3.1), which empirically demonstrates that longer horizons favor smaller learning rates.
- The theoretical bound remains quantitatively disconnected from the actual distillation experiments. Theorem 1 assumes “Residual excitation” and “Downstream observability,” freezes the value matrix and downstream mapping, and holds other contributions fixed or removes them (§4.2; Appendix A.1–A.2). These assumptions specify conditions under which a matrix discrepancy survives to observable logits, but the paper neither measures the corresponding coefficients nor establishes their relevance in the trained models. Furthermore, (\lambda_F=V\rho^2) depends on the minimum probability along the evaluated teacher–student logit interpolations; strict positivity alone does not ensure a numerically informative bound. The authors acknowledge that “We do not establish that condition or a power-law rate in training data” (p. 5, lines 238–240), and identify Equation (6) as an empirical candidate rather than a derived law. Consequently, the theorem does not yet explain the measured Phase-2 scaling behavior.
- The organization and presentation obscure the logical development and require more substantive editorial revision. The introduction moves quickly from naming MOHAWK to “Our first hypothesis was deliberately simple” and “Parts of this picture survive” (p. 1, lines 43–48), presenting the research narrative before sufficiently establishing the methodological context. Although MOHAWK receives a brief functional description, “Phase 0” is invoked in line 46 before its stages are explained in §3 (p. 3, lines 117–129). The introduction also does not adequately explain why this particular conversion framework is the appropriate starting point for the broader architectural questions posed. Subsequently, §3 combines established matrix-mixer background, a review of the inherited MOHAWK procedure, and the paper’s response-variable definition, making the boundary between prior knowledge and the present contribution unnecessarily difficult to follow. Clearly identified preliminaries, more focused positioning in §2, and a more explicit progression from the research question to the assumptions, analysis, and empirical tests would improve the argument substantially. The authors disclose that LLM assistants suggested changes to “wording, structure and length throughout the paper” and “drafted parts of the related-work and appendix text” (p. 9, lines 479–481). This transparency is appreciated. However, the unresolved organizational problems and the gap between broad introductory promises and narrowly validated conclusions raise a concern about possible overreliance on AI-assisted writing without sufficient substantive revision.
- There is also a concrete formatting issue: the abstract contains two paragraphs (p. 1, lines 11–18 and 19–28), whereas the official ICLR 2027 LaTeX template, in iclr2027_conference.tex, lines 59–60, explicitly states: “The abstract must be limited to one paragraph.” The Author Guidelines require use of the provided template.
Questions for authors
- Can the authors verify this comparison using unrounded results and evaluate last-endpoint and local-trend baselines, including their ability to predict the direction and magnitude of further training gains?
- Can the authors report the existing lower-rate controls and determine whether the cross-size forecasting failures persist under a learning-rate policy calibrated across training horizons?
- Can the authors quantify the bound in the controlled setting, or demonstrate that spectral residuals—or their value- and downstream-weighted versions—predict KL on held-out configurations?
4 comments
I appreciate the authors for adding experiments and providing clarifications. The new prediction baselines support the general claim that observations under early budgets can help predict the later KL. However, Trend performs comparably or even better, leaving the value of power-law modeling itself unclear. Please further elaborate why power-law modeling is needed and what additional scientific insights or practical benefits it offers over simple local trend extrapolation.
In addition, I would like the authors to present a clearer and more specific revision plan for the paper, including which claims will be retained, narrowed down, or removed, and how the newly added evidence will change the main conclusions of the paper. In particular, the revised discussion should adequately reflect the comparisons against prediction baselines, the sensitivity of curve inversion to learning rates, and the scope of applicability of controlled theoretical validation. Without a sufficiently explicit revision plan, I am currently unable to judge whether the contributions and supporting evidence of the revised manuscript justify acceptance.
Trend vs Power
We agree that the short-range comparisons give limited grounds for preferring the power law. A local trend can closely approximate a smooth power law over a small budget interval. We retain the power law as a testable hypothesis about whether each doubling yields an almost constant proportional reduction in KL across a wider range of distillation budgets under our MOHAWK conditioned recipe.
Our current results show that Trend has lower pooled forecast error, with advantages for Power in some model-size and horizon combinations. A rough backtest on the seven plotted observations in Kaplan et al.’s dataset-size curve illustrates why the horizon matters. Trend and Power had mean absolute percentage errors of 1.62% and 0.95% one doubling ahead (four forecasts), and 4.23% and 2.67% three doublings ahead (two forecasts). Chinchilla offers a complementary example of testing an extrapolation at scale. Hoffmann et al. used fits from smaller experiments to choose a model size and token budget, then tested that allocation at Gopher’s compute budget.
Kaplan et al., Figure 1: https://arxiv.org/pdf/2001.08361#page=3
Hoffmann et al., §§3–4: https://arxiv.org/pdf/2203.15556#page=7
We think a more discriminating test for distillation would do three things.
- Calibrate peak learning rates at early and late budgets, with the decay schedule matched to each target budget.
- Repeat representative configurations across independent seeds to measure run-to-run variation.
- Compare Power and Trend at farther held-out budgets, using the same available checkpoints for each forecast.
This would test whether the fitted exponent is reproducible across seeds and fitting windows, and whether it improves longer-range forecasts beyond the latest observed trend.
This test is within our computational capabilities in the next few days and can be added to the final version if accepted. We compromise to this.
We are finishing the answer for the plan
Revision plan
We group the changes by topic. Section 1 is already in the revised manuscript. Sections 2–4 rely on results already reported in this discussion, with two planned additions: a longer-range comparison of Trend and Power (Section 2) and the single-head experiment on more checkpoints (Section 1). Beyond the points listed, we will revise the writing of the full paper for clarity.
1. Theory and controlled validation (done)
- Added: Section 4.3 and Appendix A.4 report the single-head experiment: a trained mixer (A) against the best rank-d matrix from SVD (B), for d_state = 16, 64, 256. One table follows the error from the cross-cut block to the head output, the centred logits and the KL. A second table compares A and B (KL is 40–91× lower for B).
- Narrowed: The three measured ratios are reported as observed magnitudes, not estimates of the theorem's constants. After Theorem 1 we state that λ_F is conservative and that the bound is structural, not a numerical forecast of KL.
- Scope: The appendix states the limits: one head, four windows, one block replaced in the teacher, no trained-student KL and no uniform constants. The tail falls by 99.5% from d = 16 to 256 while the KL of the trained mixers does not fall, so we do not present the tail as a predictor of trained KL.
- Removed: "close this gap" and "sets the foundation for a theory" from the abstract. The title is now "Conditional Scaling Laws…", and the abstract is one paragraph. The abstract, introduction and contributions state that the theory and the forecasts are separate, and that the forecasts use Phase-2 measurements only.
- Planned: As stated in our first reply, we will repeat the single-head experiment on more checkpoints to make this result more robust.
2. Early-budget forecasts (Section 6.2)
- Retained: Within a fixed setup, early Phase-2 endpoints predict later KL much better than the last checkpoint (pooled RMSE 0.034 for Last, 0.005 for Trend, 0.009 for Power, one doubling ahead). The D64 test and its errors structured by model size also stay.
- Added: Both baseline tables, with the plain statement that Trend has the lower pooled RMSE and that Power is better at 530M and, two doublings ahead, at 750M.
- Planned: We will add a longer-range comparison of Trend and Power, as outlined in our previous comment. We expect Power to predict better at long range, and we commit to reporting the results whichever method they favour.
- Narrowed: The power law is presented as a description of each curve and a testable hypothesis (a near-constant proportional reduction per doubling), not as a better forecasting method. The section title "Early budgets forecast a useful range" will change accordingly.
- Removed: "small enough to help decide whether another doubling is likely to be worthwhile" and "well enough to support run planning". We claim no compute saving.
3. Late-budget reversals and learning rate (Section 6.1, Appendix B.3)
- Replaced: The sentence "Lower learning-rate runs did not show the same late reversal" becomes the matched control (750M, r = 0.25: +14.2% at 1e-4, −26.8% at 3e-5, difference 0.046 nats/token, 95% interval [0.039, 0.054]) and the cosine continuation, each with its table.
- Retained: The fixed-rate curves remain the primary comparison, including the continued improvement at 300M.
- Narrowed: The reversals are described as observations under one learning rate calibrated at short budgets. "The late-budget shape does not transfer cleanly across model sizes" gets the same qualifier, as do the negative exponents and positive curvature at 530M and 750M (Table 7).
- Removed: Wording that attributes the reversals to model size or conversion quality. We state that the control covers one configuration, and that cross-size behaviour under a horizon-calibrated rate is open.
4. Other changes
- OLMo-2 1B: described as learning curves in a second teacher family, not a forecasting test or a transferable exponent.
- Discussion and limitations: rewritten to match points 1–3, including the outcome of the two planned additions.
- Presentation: Section 3 will be split into background (matrix mixers, MOHAWK) and our own setup (response variable). The introduction now explains the three phases before the hypothesis.
Revised main conclusions
- Under a fixed recipe, early budgets forecast later KL. The power law describes these curves, but it is not shown to forecast better than a local trend.
- The late reversal is sensitive to the learning rate in the tested configuration. We do not present it as a property of the architecture.
- The spectral result is a representation limit whose effect on KL we traced in one head. It is not a full-model floor and does not explain the Phase-2 scaling.
We thank the reviewer for the careful reading and detailed feedback, which has helped us improve the manuscript. We address the concerns in order.
1. Forecasting baselines. We refer to “Early-budget forecasts” in our shared response. We compared three methods: the latest checkpoint (Last), extrapolation of the most recent absolute KL change (Trend), and our power law (Power). At the earlier budgets, Last performs substantially worse than the other two, so both extrapolation methods improve on this simple baseline. Trend and Power give closer results, with Trend achieving a lower pooled RMSE. For forecasts two doublings ahead, Power performs better at 530M and 750M, although Trend remains better at 300M.
2. LR sensitivity. We refer to “LR calibration and late-budget KL reversals” in the shared response. For the OLMo-1 750M student with 25% attention replacement, KL rose by 17.81% from D32 to D64 under the original LR of 1e-4. We recovered a continuation from D32 using LR 3e-5 and cosine decay. Its D64 KL was 37.08% below the original D64 result, with no reversal. This shows that the rebound depends on the optimization recipe and is not unavoidable for this architecture. Since both LR and scheduler changed at D32, the result does not isolate LR as the cause or establish an optimal configuration. It is consistent with the possibility that a high LR becomes less suitable as the curve flattens.
3. Matrix-to-KL connection. As described in “Matrix-to-KL bound”, we tested two controlled approximations: A, a trained recurrent mixer, and B, the optimal rank-constrained approximation from truncated SVD (Eckart–Young). We evaluated four windows, three state sizes (16, 64, 256), and one early layer. We replaced one head's cross-cut block while keeping the teacher values and other pathways fixed. In all six cases, matrix error propagated to the head output and logits, producing nonzero KL. B had much lower errors than A, with the largest A/B KL ratio at state size 256. Incomplete optimization at this state size is one possible explanation, but we have not isolated the cause.
We also computed observed ratios related to residual excitation (head²/matrix², ν_eff), downstream observability (logit²/head², μ²), and the KL factor (nKL/logit², λ_F/2). In the table, matrix² and head² denote squared Frobenius errors; logit² is the squared centered-logit difference. Brackets give the reported ranges. The last two columns use the scale factors shown in their headers.
These measurements show that the approximation error remains observable downstream. The ratios help illustrate the possible magnitudes of the theorem's quantities, but they are not estimates of its uniform coefficients. We have added details to Section 4.2 and Appendix A. We will experiment with more checkpoints to provide robust empirical support for the theorem.
4–5. Presentation and formatting. We have rewritten the abstract as one paragraph and changed the title to “Conditional Scaling Laws for Attention-to-Recurrence Distillation”. We also revised the introduction and contributions to clarify the paper's scope. Further presentation improvements are underway, taking the reviewer's suggestions into account.
Questions: We address the three questions in the same order.