acceptodds
Withdrawn from ICLR 2027

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hybrid linear attention reduces the quadratic cost and KV-cache growth of softmax attention, but converting a pretrained Transformer remains brittle. Projection transfer copies weights yet leaves the student's new recurrent dynamics—memory decay, write strength, value scale, and output gating—unspecified. We propose Taylor-Calibrate, a lightweight teacher-conditioned initialization for these unmatched dynamics. An exact online form of softmax attention exposes a delta update; a Taylor and online-regression approximation then identifies the corresponding recurrent quantities. We estimate their operating point from teacher attention distance, concentration, and output scale, followed by brief layer-local alignment. Across four teachers and three retained-layer policies, Taylor-Calibrate produces stronger zero-shot students. On five single-run matched-PPL recovery comparisons spanning three teachers, conservative lower-bound Stage-2 token savings range from 4.9 to 9.2 relative to projection-copy initialization. Held-out representation and KL measurements confirm that the gain reflects closer teacher behavior, not perplexity alone. Matched three-seed experiments further transfer the framework to GLA and KDA. Separately, five-layer KDA conversion with a 64-GPU-sharded 975B Inkling teacher reaches PPL 6.498 versus 6.454 and 90.2% teacher top-1 agreement, validating scale integration. The complete 1.5B calibration costs 0.024 H100-hours.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Initialization, Knowledge Distillation, Linear Attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tdkmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper attempts to introduce a principled way to perform distillation of an attention layer into linear-attention variants, with particular focus on GLA / GDN / KDA linear recurrent cells. The procedure aims at initialising the newly-added student parameters specific to the recursion (gates, decay, write strength, and so on) with statistics coming from the teacher. The description of the exact mechanism is rather confused: the paper first establishes a connection between an online update view of softmax attention, approximated via Taylor expansion, and a delta-rule style RNN, but this does not seem to be directly leveraged for the initialisation, which instead relies on teacher statistics such as mean look-back and per-head entropy, although the underlying justification eludes me. This somehow forms phase 1 of the distillation procedure. Phase 2 then follows, which is standard per-layer distillation. The evaluation procedure covers ablations on Phase1 parameters initialisation, zero-shot scores for 4 teachers, recovery speed for distillation, and distillation a 975B MoE.

Critical strengths
  • The problem is relevant
Critical weaknesses
  • The paper in its current form, is extremely hard to parse, to the point that it hinders my ability to express a clear judgment on it
Questions for authors
  • Clarity: The paper is extremely hard to parse, for a number of reasons. Necessary terminology is not properly introduced when needed, and must be either reverse-engineered from other parts of the paper or plainly guessed. Similarly with a number of mathematical quantities or acronyms introduced, which oftentimes are not properly described. Sentences are often borderline nonsensical, or employing obscure terms, at great cost to clarity. The paper lacks a coherent structure, failing to properly introduce the key aspects of the method. Figures, rather than helping highlighting the particularities of the method, are quite frankly uninformative. The fact that the paper keeps on switching terminology to refer to given components further adds to the confusion. In short, the paper is in severe need of a thorough rewrite.
Reviewer_1EBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an initialization technique conditioned on the teacher for hybrid models upcycling techniques. The key idea is that the recurrent dynamics for the student in traditional approaches are not translated when initialized purely with the Q, K, V projections to initialize the chosen linear layers. With Taylor-calibrate the authors propose a two step approach to approximate the first order taylor priors and then calibrate per layer residual error before distillation. The results are shown across a range of dense and MoE models including a large Inking 975B parameter model. The paper also evaluates across GDN, GLA and KDA layers and showcases broad applicability.

Critical strengths

The paper motivates a key gap in nitialization for linearization from quadratic to linear attention hybrid models of missing recurrent dynamics in typical project only initialization approaches. The two stage approach is well motivated and ablation studies show improvements from use of the layer-wise residual errors. The experimental coverage is broad across scale of model with one example of 975B inkling model as well as coverage across dese and MoE models. The results showcase convergence with fewer tokens, which are key benefits for such upcycling techniques. Overall the paper is well motivated and results show broad applicability across scale of models and types of linear layers (GDN, KDA, GLA).

Critical weaknesses

The abstract emphasizes initialization with recurrent dynamics as a key motivator, but the results indicate that the stage-2 is critical to improve ppl scores. There are no comparisons to other upcycling techniques such as Mamba-In-Llama, LolCATs, Zebra-Llama etc. which are referenced in the paper, but comparisons are missing. An entire class of popular MLA models are missing, if the approach is not applicable it should be noted in the paper.

Questions for authors

For the layer selection, are there any key design choices to which layers provide better convergence, how are the 5 layers for inkling chosen? Is there a difference in chosing SWA v/s full attention layers? Notably there are no models that use MLA attention, does this approach also apply to open source models with MLA layers? Can authors add additional comparison of performance against other similar upcycling approaches like Mamba-in-Llama, Lolcats etc. to showcase benefits of initialization.

Reviewer_3TPLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel method to convert transformer layers to linear attention layers. The method does not only copy QKV-weight matrices, but also tries to initialize the gates with sensible values. After initialization (Phase 1), the outputs of the transformer teacher and linear attention student are aligned with MSE Loss (Phase 2).

Critical strengths
  • The main idea of the paper is sound and well explained. Although I am not entirely certain if it is really "principled".
  • The selection of datasets is sensible, with the exception of RULER, where only the shortest variant is reported.
Critical weaknesses
  • The training consists of Phase 1 and Phase 2. In the results the different Initialization schemes are compared to calibrated results. Without Phase 2, the proposed method seems to have little effect.
  • The main idea is well described, but some sections, for example 4.2 could be improved substantially.
Questions for authors

Long context retrieval is where linear attention is expected to be worse than full attention. How does your method affect this behavior? Can you report longer RULER variants? Concerning the mixed reporting of results: Can you provide a full comparison where Phase 1 and Phase 2 are not mixed? Does the initialization improve Phase 2 or is Phase 2 beneficial for all settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.