acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Attention Linearization

19 views

Abstract

Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pretraining such models, a growing body of work distills them from already trained full-attention transformers by aligning the attention layers' hidden states and matching the models' next-token distributions. We find that these distilled models often collapse on long-context retrieval and reasoning tasks, particularly when operating in thinking mode, where the efficiency gains of hybrid architectures matter most. Since linear attention layers must compress context into a fixed-size state, their errors compound over long sequences. As off-policy distillation never teaches the student model to recover from this drift, tasks that necessitate longer sequence lengths become especially challenging. We introduce On-Policy Attention Linearization (OPAL) in which the hybrid attention student samples its own long-context trajectories and receives dense supervision from the frozen full-attention teacher. Applying OPAL to Qwen3-4B and MiMo-7B-RL-0530, we recover of full attention performance on commonsense reasoning, on needle-in-a-haystack (NIAH) retrieval, and on mathematical reasoning with only 3B training tokens. We achieve these results without Supervised Fine-Tuning (SFT) or Reinforcement Learning with Verifiable Rewards (RLVR) training stages and outperform competitive methods that utilize such techniques by on long-context retrieval in terms of recovery to teacher.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Gated DeltaNet, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j1fLOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a 3 stage framework OPAL, to distill the full-attention teacher to linearization student. OPAL improve the OPD stage with data and LR strategies, including difficulty curriculum, length curriculum, and WSD lr schedule. OPAL reaches the SOTA performance on NIAH and mathematical reasoning compared with prior linearization methods.

Critical strengths

The comprehensive experiment on both NIAH and math reasoning, reaching high recovery rate from 83.4% to 100.7%, and outperforming prior works with a large margin.

Critical weaknesses
  1. Ablation study is not enough. The paper does not include ablation on length curriculum, difficulty curriculum and WSD schedule to isolate the effect of their contribution.
  2. Contribution need clarify. From my perspective, the length curriculum[1], difficulty curriculum[2,3] and WSD schedule have been proposed in privous works. The authors should clarify their contribution over previous works.

[1]Zhang Y, Chai J, Fu Y, et al. Are Full Rollouts Necessary for On-Policy Distillation?[J]. arXiv preprint arXiv:2605.31490, 2026.

[2]Jia Y, Zhang C, Diao X, et al. What makes a good curriculum? disentangling the effects of data ordering on llm mathematical reasoning[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 34472-34488.

[3] Gao Z, Kim J, Sun W, et al. Prompt curriculum learning for efficient llm post-training[C]//International Conference on Learning Representations. 2026, 2026: 58614-58652.

Questions for authors
  1. Are the baselines trained on the same dataset with OPAL? It seems that the authors use the publicly release model without further training, and if so, it will make it hard to compare methods on the math reasoning task. I suggest the authors compare methods with matched training data.
  2. The stage 3, Prolonged On-Policy Distillation, perform OPD over long sequences, however, the context length in NIAH is up to 32K, while HypeNet reports results on 256K context length. Could the author further provide the result on longer context rather than 32k?
  3. Regarding the question3, actually I think applying OPAL on long context task and large model is hard, since we always need a full-attention model as a teacher to compute the logits over long context, especially in OPD, you need to compute it for each rollout, making it inefficient. Could the author further discuss this question?
3 comments
AuthorsOpenReview ↗

Ablation Depth and Explanation of Novel Components

Thank you for your constructive comments. We aim to address your feedback.

W1. Ablation Depth

  • WSD Scaling Ladder - In early experiments, we tested the use of cosine decay vs. WSD for OPD. Both WSD schedules outperform the Cosine schedule in early training. However, an important distinction is that we do not make the claim that Cosine would continue to lag with extended training. Other than early performance signals, WSD was picked because it allowed us to scale our experiments from 200M OPD tokens to 2B OPD tokens without significant rollbacks.
MATH-500AIME'24AIME'25
Cosine79.118.317.5
WSD, separate LR for GDN/inherited81.521.221.7
WSD, uniform LR83.623.318.8
Repeated Steps @ 200MRepeated Steps @ 400MRepeated Steps @ 600M
Cosine0241366
WSD011122334
  • Horizon Doubling - Doubling the horizon at KL plateau is specifically useful early in training where the student deviates significantly from the teacher model even with short rollouts. Figure 9-12 help make this point clear as initial rungs of the scaling ladder show high reverse KL even in early positions. By introducing longer rollouts later in training, we avoid wasting compute by providing supervision on later token positions that provide less meaningful signal. We provide a comparison to directly using 32K sequences for the first 200M training tokens below.
TokensRunRKL, positions 0–511RKL, positions 512–4KRKL, positions 4K–8KTokens to reach doubling's 50M-token RKL
0–100MDoubling0.2560.2290.21050M
0–100MFixed 32K0.303 (+18%)0.240 (+5%)0.211 (+1%)not reached
100–200MDoubling0.2280.1990.179–
100–200MFixed 32K0.255 (+12%)0.211 (+6%)0.184 (+3%)171M (3.4×)
  • Difficulty Curriculum - Examples from DeepScaleR and DAPO-Math-17K were incorporated at around 1B OPD tokens. Notably, certain metrics such as pass@1/pass@8 performance on AIME'24 and pass@8 performance on AIME'25 had seen a relative plateau over the prior few checkpoints, as is visible in Figure 3. The following checkpoints after incorporating harder examples escaped from this flat region.

W2. Contribution Over Previous Works

Thank you for posing this question. We highlight how our method differs from prior work.

  • We are the first work that shows that off-policy KD underperforms for reasoning tasks when used for linearization. Our controls compare OPD with continued off-policy distillation from the same hybrid initialization. As OPD consistently achieves stronger reasoning performance, our results support the importance of on-policy methods and supervising the student on its own generated trajectories.
  • Our continual learning-style recipe for OPD is novel to the best of our knowledge. A WSD schedule allows us to extend from pre-decay checkpoints while producing annealed checkpoints for evaluation at successive token budgets. Our contribution is applying this kind of continual learning structure to OPD-based recovery.

We agree aspects of the recipe exist in other forms, but these elements are new and, based on our results, quite effective when applied to the problem of linearization.

Q1. Baseline Comparison

The baselines are not trained on the same dataset as OPAL. However, we have two responses.

  • Comparing to released checkpoints for prior methods is common practice for linearization methods. RADLADS, Llamba, Zebra-Llama, and LoLCATs all benchmark in exactly this way despite using different training data. We believe the overall training recipe is also important to evaluate.
  • Our controls provide the strongest evidence for our method's effectiveness with all else being held equal. Since all prior baselines use some variation of off-policy KD, our off-policy control provides proof that OPD improves recovery. We have extended the off-policy KD stage to 1B tokens.
MATH-500AIME'24AIME'25
OPD (1B Tokens)90.545.835.8
Off-Policy KD (1B Tokens)82.926.222.1

Q2. Longer Context Evaluation

Our experiments were initially structured around Qwen3-4B which has a context length of 32K. However, we've since added additional experiments that demonstrate that the method generalizes even in regimes where the model is forced to locate information outside of its native context and training generation length using YaRN when necessary.

LongBenchHELMET (64K)HELMET (128K)
Qwen3-4B53.867.264.8
OPAL-Qwen342.8 (80%)56.0 (83%)52.8 (81%)

Q3. Efficiency Analysis

First, we would like to clarify that using a teacher model to compute logits over long context is necessary for any distillation pipline, including off-policy KD. The additional cost of OPD is specifically rollout generation, which we provide a wall-clock control for (see our response to Reviewer jW6Q for additional details).

MATH-500AIME'24AIME'25
OPD (400M Tokens)87.033.331.7
OPD (600M Tokens)89.036.731.2
Off-Policy KD (1B Tokens)82.926.222.1

Thank you for the detailed response and the additional experiments. The new results address most of my concerns and demonstrate strong performance. I’m raising my rating from 2 to 3 and lowering my confidence score.

Thank you so much for your response! Please let us know if there are any other concerns you would like us to address.

Reviewer_jW6QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces On-Policy Attention Linearization (OPAL), a framework designed to convert pretrained, full-attention transformer LLMs into hybrid models (where ~75% of full-attention layers are replaced by linear attention layers, specifically Gated DeltaNet or Mamba-3) without sacrificing long-context retrieval or multi-step mathematical reasoning capabilities.

Critical strengths

State-of-the-Art Long-Context and Reasoning Recovery with High Token Efficiency: OPAL achieves impressive empirical recovery rates on challenging long-horizon tasks—recovering 100% of teacher NIAH retrieval at 32K context and 83–93% of teacher mathematical reasoning performance (achieving 67.6% and 72.2% average accuracy vs. 21.6% for the best prior baseline). Crucially, these results are achieved using only 3B training tokens without requiring reinforcement learning with verifiable rewards (RLVR) or supervised fine-tuning (SFT).

Critical weaknesses

High Computational/Inference Overhead during On-Policy Rollout Generation: Although OPAL requires only 3B tokens in terms of SGD update volume, Stage 3 requires sampling long trajectories (up to 32K tokens) from the student policy using active inference engines (vLLM) followed by forward passes through the frozen full-attention teacher. The paper does not provide a wall-clock or GPU-hour comparison against off-policy baselines or pretraining-from-scratch/continual-pretraining methods. Since on-policy sampling over 32K context horizons is computationally expensive, comparing pure token count understates the total compute cost of the pipeline.

Questions for authors

Wall-clock and GPU-hour Cost Comparison: Could you provide the actual wall-clock time or total GPU hours (e.g., H100 GPU hours) required for Stage 3 OPD compared to Stage 2 off-policy distillation? Since sampling 32K token rollouts and scoring them with a full-attention teacher incurs significant inference overhead, a compute-cost comparison (in GPU-hours or FLOPs) would clarify the practical budget required.

Mitigating Commonsense Reasoning Degradation: Did you experiment with adding standard pretraining/likelihood data (or off-policy loss terms on short contexts) during Stage 3 OPD to prevent the degradation on short-context commonsense benchmarks (PIQA, ARC, MMLU)?

Generalizability to Pure Linear Architectures: Does OPAL require at least a small fraction of full-attention layers (e.g., 1:3 ratio) to anchor the teacher-student alignment during on-policy rollouts, or can it be applied to convert a model into a 100% pure linear attention / SSM architecture?

1 comment
AuthorsOpenReview ↗

Wall-Clock Controlled Experiment and Fewer Full-Attention Layers

Thank you for your helpful review. We would like to respond to your individual comments.

W1/Q1. Wall-Clock Time of OPD Training

It is absolutely correct that OPD training incurs additional rollout generation cost compared with standard off-policy KD. While this is a valid concern, we would like to present two key responses that justify the overall cost of training.

  • Implementation Efficiency/Wall-Clock Controlled Experiments - In our implementation we use a single GPU for rollout generation and three for model training, overlapping the two processes. In our Qwen3-4B training loop, waiting for rollouts accounted for approximately 12% of logged training time. Our sustained training throughput for Stage 3 at 32K sequences is 4640 tok/s, whereas our training throughput for Stage 2B at equal sequence length is 9034 tok/s. We extended the training of our off-policy controls to 1B training tokens, and based on the numbers listed above, compared these checkpoints to OPD checkpoints that required roughly the same number of GPU hours. The following results suggest that even in this setting, off-policy baselines still underperform and flatline early in training.
MATH-500AIME'24AIME'25
OPD (400M Tokens)87.033.331.7
OPD (600M Tokens)89.036.731.2
Off-Policy KD (1B Tokens)82.926.222.1
  • Scalability - Linearization may prove especially useful as we scale model size and the infrastructure used for conversion. In these settings that creating efficient, deployment-friendly versions of models is particularly desirable. With greater infrastructure, coordination across multiple generator and trainer ranks provides greater throughput gains. Recent works (AsyncOPD, KDFlow) suggest that OPD benefits significantly from additional infrastructure and careful design of generation/training overlap. We believe that as we scale this method, we can better utilize the infrastructure to reduce generation slowdowns.

Q2. Mitigating Commonsense Reasoning Degradation

This is a great question, and points to a possible solution for the CSR degradation we noticed in our experiments. Recent works suggest that replaying pretraining-like data, changing the training objective, and more careful data schedule design can improve reasoning fine-tuned model's capabilities on pretraining benchmarks. However, we feel that post-training data schedule and objective design constitutes its own investigation, and is a direction we are currently pursuing based on the results of this paper. Our initial attempts provided marginal changes in performance but no concrete gains; although, we believe more careful co-design of the pre- and post-training objectives could yield more meaningful results based on the investigations of related literature.

Q3. Generalizability to Pure Linear Architectures

While our method is technically agnostic to the linear attention conversion ratio, we have three responses to the question about full linear architectures.

  • First, we justify our experimental choice of 1 full-attention layer:3 GDN layers. Our choice of one full-attention layer for every three GDN layers follows an established design in prominent open weight hybrid models. Qwen3-Next and Qwen3.5 adopt this same ratio, while Kimi K3 use a similar pattern with approximately three KDA layers per MLA layer. Across these architectures, roughly three quarters of the layers summarize the preceding context in a fixed-size recurrent state, while the remaining layers retain access to token-level representations. Our 1:3 full-attention-to-GDN configuration follows this design pattern.
  • Second, we tested our method with a ratio of 1 full attention:8 GDN layers. The following are math reasoning results from our experiments. These results show that OPAL can preserve mathematical reasoning capability under more aggressive architectural conversion, albeit with some amount of performance degradation. The gap relative to the 1:3 configuration highlights the trade-off between attention sparsity and recovered accuracy at a fixed training budget.
MATH-500AIME’24 (pass@8)AIME’25 (pass@8)
1:3 Ratio94.473.360.0
1:8 Ratio87.363.353.3
  • Finally, our linearization pipeline relies on preexisting linear attention layers. While linear attention layers have become significantly more expressive over time, they still have not been able to completely match the representational capabilities of exact KV retrieval. This seems to be why, in practice, they are used in conjunction with full attention for many open weight models. Thus, a linearization pipeline is ultimately still bound by the capabilities of the linear attention layers it uses. Extending this study to fully linear students, and disentangling architectural limitations from initialization and training-budget effects, remains an area of future work that we are excited to pursue as linear attention layers become increasingly powerful.
Reviewer_HCUrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper converts full-attention LLMs into hybrid models by replacing 75% of attention layers with Gated DeltaNet to improve inference efficiency while preserving long-context retrieval and reasoning. It proposes OPAL, which adds prolonged on-policy distillation with reverse KL, rollout-horizon doubling, a difficulty curriculum, and WSD scheduling. On Qwen3-4B and MiMo-7B-RL-0530, the hybrid models approach teacher performance on 32K retrieval, retain 83.4%/93.4% math reasoning, and report ~4× lower attention-state memory at 128K and 2× higher decoding throughput. Matched-budget comparisons support on-policy distillation.

Critical strengths
  1. Strong evidence of long-context and reasoning recovery after aggressive attention linearization. The paper convincingly shows that pretrained reasoning models can retain much of their capability after replacing 75% of full-attention layers. Table 1 reports near-perfect retrieval for OPAL-Qwen3 up to 32K and 97.7% average retrieval for OPAL-MiMo vs. 97.0% for its teacher. Table 2 shows MATH-500 scores of 92.0% vs. 96.4% and 93.8% vs. 97.0%, despite substantial reasoning-oriented post-training and without additional SFT or RLVR. Figure 4 further supports practical inference benefits.

  2. Convincing evidence that on-policy training improves recovery over off-policy distillation. The paper gives a well-motivated explanation: small prediction errors can shift recurrent states and expose the student to trajectories poorly covered by teacher-forced training. Figure 2 shows degenerate long-horizon generations after off-policy distillation and their reduction under OPD. Table 3 directly supports this: at 400M tokens, OPD achieves 33.3% on AIME’24 and 31.7% on AIME’25 vs. 17.1% and 15.8% for dataset-based off-policy distillation. This controlled comparison provides direct evidence that training on the student’s own trajectory distribution is important.

Critical weaknesses
  1. Insufficient component-level ablation and limited compute-matched comparison at the full training budget. The method combines OPD with adaptive rollout-horizon doubling, a difficulty curriculum, and a WSD-based scaling ladder, but the paper does not isolate their individual contributions. It remains unclear whether standard OPD with a fixed rollout horizon and sufficient compute could achieve comparable results. Moreover, Table 3 compares OPD and off-policy distillation only at 200M and 400M tokens, whereas final OPAL models receive 2B on-policy tokens. Figure 3 shows continued gains with more OPD compute, but the corresponding long-budget off-policy comparison is absent. Since OPD itself is established, clearer isolation of the additional mechanisms is needed to establish methodological novelty.

  2. Limited validation of general long-context capabilities and nontrivial residual degradation. Long-context retrieval evaluation mainly uses synthetic single-needle, multi-key, and multi-query tasks (Table 1), rather than multi-evidence reasoning, long-document QA, or information synthesis across long inputs. Math benchmarks primarily assess generation-time reasoning, not reasoning requiring long-input integration. Meaningful gaps also remain: OPAL-Qwen3 achieves 40.4% AIME’25 pass@1 vs. 63.7% for its teacher, and average commonsense performance drops from 68.6% to 59.6% (Table 9). Finally, quality is evaluated only up to 32K, while efficiency is reported at 128K (Figure 4), leaving longer-context preservation unverified.

Questions for authors
  1. What are the individual contributions of adaptive horizon doubling, the difficulty curriculum, and the WSD scaling ladder? Please provide component-level ablations against standard OPD with a fixed rollout horizon, without the curriculum, or without the proposed scaling schedule. Do off-policy baselines continue to underperform when trained substantially longer than the 400M tokens evaluated in Table 3?

  2. Do the recovered long-context abilities extend beyond synthetic NIAH-style tasks to multi-hop reasoning or information integration across long documents? Has quality been evaluated beyond 32K context length, particularly given the 128K efficiency results in Figure 4?

  3. How should the AIME differences between Tables 2 and 4 be interpreted—e.g., OPAL-Qwen3 40.4% vs. 35.0% on AIME’25 and OPAL-MiMo 65.4% vs. 59.6% on AIME’24? Are these due to different evaluation configurations, checkpoints, or sampling variability? Please provide evaluation details and, ideally, repeated-run variability.

1 comment
AuthorsOpenReview ↗

Additional Ablations and Long-Context Experiments

Thank you so much for your thoughtful review. We answer your questions below.

W1/Q1. Component-Level Ablations

We aim to highlight the importance of different components of our method below.

  • WSD Scaling Ladder - In early experiments, we tested the use of cosine decay vs. WSD for OPD training. Both WSD schedules outperform the Cosine schedule in early training. However, an important distinction is that we do not make the claim that Cosine would continue to lag with extended training. Crucially, other than early performance signals, WSD was picked because our overall training horizon was not known beforehand. Using WSD allowed us to scale our experiments from 200M OPD tokens to 2B OPD tokens without having to waste significant amounts of training time rolling back to checkpoints very early in training.
MATH-500AIME'24AIME'25
Cosine79.118.317.5
WSD, separate LR for GDN/inherited81.521.221.7
WSD, uniform LR83.623.318.8
Repeated Steps @ 200MRepeated Steps @ 400MRepeated Steps @ 600M
Cosine0241366
WSD011122334
  • Horizon Doubling - Doubling the horizon at KL plateau is specifically useful early in training where the student deviates significantly from the teacher model even with short rollouts. Figure 9-12 help make this point clear as initial rungs of the scaling ladder show high reverse KL even in early token positions. By introducing longer rollouts later in training, we avoid wasting compute by providing supervision on later token positions that provide less meaningful signal. We provide a comparison to directly using 32K sequences for the first 200M training tokens below.
TokensRunRKL, positions 0–511RKL, positions 512–4KRKL, positions 4K–8KTokens to reach doubling's 50M-token RKL
0–100MDoubling0.2560.2290.21050M
0–100MFixed 32K0.303 (+18%)0.240 (+5%)0.211 (+1%)not reached
100–200MDoubling0.2280.1990.179–
100–200MFixed 32K0.255 (+12%)0.211 (+6%)0.184 (+3%)171M (3.4×)
  • Difficulty Curriculum - The impact of the difficulty curriculum is made evident by the performance of the model checkpoints before and after utilizing more difficult math examples from DeepScaleR and DAPO-Math-17K. These examples were incorporated at around 1B OPD tokens. Notably, certain metrics such as pass@1/pass@8 performance on AIME'24 and pass@8 performance on AIME'25 had seen a relative plateau over the prior few checkpoints, as is visible in Figure 3. Incorporating harder examples helped the model escape from this flat region.
  • Off-Policy Baseline - We extend the off-policy baselines to 1B training tokens and report the results below. Even at 1B training tokens, the off-policy baseline lags significantly. This reinforces the idea that off-policy distillation does not prevent the failure mode of compunding errors in the converted model's recurrent state
MATH-500AIME'24AIME'25
OPD (1B Tokens)90.545.835.8
Off-Policy KD (1B Tokens)82.926.222.1

W2/Q2. Generalization of Long-Context Capabilities

Thank you for bringing up this concern; this is a valid point and we would like to justify our benchmarks and provide additional evaluations. Our choice of NIAH retrieval benchmarks and math evaluations come from benchmarks often used in prior linearization papers (HALO, KL-LS, Zebra-Llama). Thus, the choice of benchmarks is to help make our method more directly comparable to prior work. As for using 32K sequences, the native context of Qwen3-4B, which we had originally structured our experiments around, is 32K without YaRN.

We complement those experiments with results on HELMET and LongBench, more difficult retrieval tasks, using YaRN when necessary. These results show recovery-to-teacher in the same range as reasoning for Qwen3-4B despite no multi-hop, RAG, or ICL-specific training.

LongBenchHELMET (64K)HELMET (128K)
Qwen3-4B53.867.264.8
OPAL-Qwen342.8 (80%)56.0 (83%)52.8 (81%)

Finally, we recognize there are settings in which full recovery is still not achieved. We believe OPAL is a significant step towards efficient architecture conversion, as evidenced by the comparison to prior baselines, but there are still open questions about perfect recovery in more difficult evaluations. Our method does not use RLVR or MOPD, post-training techniques that we are actively exploring to improve performance.

Q3. AIME Differences Thank you for the question, this helped us catch a genuine mistake in our displayed results. Table 2 displays the results after the final decay phase, in other words when OPD training was officially complete. In contrast, Table 4 presents the results prior to the last decay phase, the last evaluations that we had completed when writing this part of the manuscript, and was not updated upon the completion of training. This is also why you see a small, but consistent, reduction in performance in Table 4. We will update the table accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.