acceptodds
Under review as a conference paper at ICLR 2027

Turnout: Adaptive Attention Switching for Efficient Reasoning

Abstract

Post-hoc transformer linearization replaces the key-value cache of full attention, which grows linearly with generation length, with a fixed size state. This change reduces memory consumption but often hurts accuracy. In reasoning tasks, which require long generations, this effect is especially pronounced. In this work we introduce Turnout, a method that mitigates accuracy loss as a result of linearization, while maintaining its memory-reduction benefits. We apply post-hoc linearization to a pretrained full-attention model, training a linear path through distillation while leaving the original full-attention path unchanged. During response generation, the resulting hybrid model retains both paths and uses a learned policy to control when each layer switches from full to linear attention. We train the policy with reinforcement learning using a reward that balances answer correctness and memory use. Empirical evaluation of our method shows that the learned adaptive policy achieves a better accuracy-efficiency Pareto frontier on reasoning tasks compared to fixed switching and static hybrid baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Reasoning, LLM, RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_9McXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Turnout, which preserves the original full-attention path and trains a hybrid linear branch (GDN plus sliding-window attention and sink tokens) via distillation. A router trained with reinforcement learning makes input-dependent, per-layer, per-token decisions about whether and when to irreversibly switch to the linear branch, aiming to reduce KV-cache memory while preserving accuracy on long reasoning generations.

Critical strengths
  1. Turnout enables input-dependent, layer-wise switching between full and linear attention, providing a flexible approach to balancing reasoning accuracy and memory efficiency.
  2. The method achieves a favorable accuracy–memory trade-off across five reasoning benchmarks.
Critical weaknesses

1.It remains unclear what fundamental challenges make simpler heuristic or search-based switching strategies insufficient, and why reinforcement learning is a particularly appropriate choice for this problem.

  1. The baselines mainly consider fixed layer subsets or uniform switching positions, without relevant prior work on hybrid layer selection. The current comparisons may therefore overstate the advantage of learned adaptive routing over simpler alternatives.

  2. Section 3 reveals substantial differences between Qwen3-8B and Llama 3.1-8B in their responses to linearization and sliding-window attention. However, the proposed RL-based routing is evaluated only on Qwen3-8B, leaving its effectiveness across different model architectures unverified.

  3. With 36 layers and a rank-128 LoRA adapter on each output projection, standard independent adapters alone would introduce approximately 37.75M parameters, exceeding the reported 9.09M additional parameters for the entire linear branch. The paper does not clearly explain whether parameter sharing, weight merging, or another implementation accounts for this discrepancy.

  4. No implementation code is provided, and critical details of the dynamic attention-switching mechanism, particularly how the GDN recurrent state is constructed when transitioning from full attention, remain insufficiently specified.

Questions for authors
  1. Does linear-branch distillation include trajectories that transition from full to linear attention? If not, how is the GDN recurrent state initialized at the switching point, and what evidence supports the frozen branch's robustness to mixed-attention hidden-state distributions?
  2. How frequently do all-incorrect rollout groups occur under the accuracy-scaled reward, and how are these zero-advantage groups handled during training? Are they skipped, resampled, or simply included without contributing gradients?
3 comments

We thank the reviewer for the feedback and for recognizing Turnout's input-dependent, layer-wise switching, its flexibility in balancing reasoning accuracy and memory efficiency, and its favorable accuracy–memory trade-off across five reasoning benchmarks. We also uploaded updated version of the paper with all the changes marked with blue color.

W1. Motivation for reinforcement learning and simpler switching strategies. In our opinion, these properties of the problem make heuristics and search hard:

  1. The reward is known only at the end. Whether a switch hurts the answer is known only after the full response is generated and graded. Answer accuracy is not differentiable with respect to the discrete switch decisions.
  2. The decisions interact. A switch in one layer changes the hidden states of all later layers and all later tokens, and it cannot be undone. The effect of one decision therefore depends on all other decisions.
  3. The best decision depends on the input. Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.) keeps different layers in full attention for math, code and science (Figure 6). A schedule found by search is fixed for all inputs.

Search is possible only over fixed schedules, because each candidate must be evaluated by generating and grading full responses on a dataset. Manual switching and per-layer hybridization are such fixed schedules. Turnout is more accurate than both (Table 8):

MethodAvg. accuracyCache saving (%)
Manual switching (2K tokens)59.527.9
Per-layer hybridization (50% full layers)74.634.7
Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.)82.425.4

We chose RL since it fits the required properties. GRPO needs only a reward for each sampled response, and it does not need gradients through the generation. It compares K=4K=4 routings of the same prompt, so the advantage does not depend on how hard the prompt is. Training is cheap: the router has 0.65M parameters and is trained for 100 steps on four GPUs. We would appreciate if the reviewer could elaborate the details of a simpler heuristic-based approach that can satisfy these three non-trivial criteria.

W2. Prior work on hybrid layer selection.

The per-layer hybridization baseline in the paper already selects layers based on data. It keeps the layers with the largest final activation-distillation loss (MSE between full and linear path) in full attention (see Section 5.2 and Appendix A.1). These are the layers that are the hardest to linearize.

We now add FlashMorph (Lan et al., 2026) as a baseline from prior work. It selects the full-attention layers with trained per-layer gates. We use two layer rankings for it:

  • FlashMorph (ours): no official code is available, so tried to reimplement the method with the full and linear paths of the paper and trained the gates on the router-training data.
  • FlashMorph (paper): the layer ranking that Lan et al. (2026) report for Qwen3-8B.

For both, we skip the fine-tuning step after layer selection, so all baselines differ only in which layers keep full attention. With 50% of the layers in full attention accuracies are:

MethodGSM8KGPQA-DMATH-500MBPP+HE+Avg.
Per-layer hybridization93.239.489.473.577.474.6
FlashMorph (ours)58.626.356.024.531.739.4
FlashMorph (paper)93.540.789.872.670.173.3
Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.)95.350.093.091.382.382.4

FlashMorph (paper) is close to our per-layer selection, while FlashMorph (ours) is much lower on GPQA-D, MATH-500 and code, because its gates keep the first layers in full attention (see Table 9 in the paper). Turnout is more accurate than per-layer hybridization and both FlashMorph variants.

We added both FlashMorph variants to Figure 5 and Section 5.2, selected layers are listed in Appendix B.4 (Table 9). We did not find other post-hoc layer-selection methods that work without retraining the model. If the reviewer has a specific method in mind, we are happy to add it.

W3. Llama result We trained Turnout routers for Llama 3.1-8B-Instruct and added the results to Table 1:

MethodGSM8KGPQA-DMATH-500MBPP+HE+Avg.Cache saving (%)
Full attention78.223.748.673.058.556.40.0
All linear51.28.128.031.225.628.8-68.4
Turnout (RscaledR_{\mathrm{scaled}}, Mem.)78.123.249.271.457.956.0-1.4
Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.)77.926.052.071.760.457.61.3

Both routers keep the accuracy of full attention, while the all-linear model loses 27.6 points. The cache saving is small because Llama 3.1-8B-Instruct does not generate long reasoning traces. One linear layer uses the memory of 584 KV-cache tokens (512 for the GDN state, 64 for the sliding window and 8 sink tokens). The prompt and response are shorter than 584 tokens for 97.5% of GSM8K, 99.5% of MBPP+ and 91.5% of HumanEval+ samples. For these samples, full attention uses less memory than the linear path, which is why the all-linear model has a negative saving. The router learns to keep full attention on such short sequences, which is the intended effect of the break-even point BB in the reward (Section 4.3).

We add this explanation to Section 5.2.

W4. Parameter count

Thank you for noticing this inconsistency. The parameter count in Table 2 did not include the LoRA adapters, because we assume that the adapter of a layer is merged into its output projection when the layer switches. This is safe because a layer never switches back to full attention (Section 4.2). After the switch, the layer uses only the linear path, so the original projection is no longer needed.

As a result, the LoRA parameters add no compute and no weight transfer during decoding. Before the switch, the full path does not use the adapter. After the switch, the layer reads one merged projection layer of the original size. The only extra cost is a one-time merge per layer.

We agree that this must be stated explicitly. Table 2 now has an extra row for Turnout with unmerged LoRA adapters:

MethodAdded parametersOverheadWeights (GiB)
Turnout (LoRA merged)9.74M0.12%15.28
Turnout (LoRA not merged)47.49M0.58%15.42

The 36 rank-128 adapters add 37.75M parameters, as the reviewer computed. Section 5.3 now explains the merge.

W5/Q1. Code and GDN state transition We have done our best to include all the implementation details and hyperparameters in the paper and appendix. We are working on releasing the code and plan to do it upone paper acceptance. We also expanded the description of the switching mechanism in Section 4.1 and added Appendix B.2, which gives the equations of the state computation.

The linearization stage uses the linear path throughout the training sequence and there is no switching mechanism at this stage. However, after linearization both attention paths share the key and value projections. While a layer runs in full attention, the KV cache stores its keys and values, and we also store the two scalar GDN gates of each token and head. At the switch, we compute the GDN state from the cached keys, values and gates with the chunked GDN kernel of Flash Linear Attention library. Given the same layer inputs, this is exactly the state that the linear path would have after the same prefix. We then keep only the 8 sink tokens and the last 64 tokens of the KV cache, and remove the rest.

After a switch, the layer inputs come from a mix of full and linear layers. The linear path was not trained on this mix. Our empirical results suggest that the frozen paths handles inconsistency. Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.) runs in this mixed mode on every response and reaches 82.4 average accuracy, 2.7 points below full attention and 36.5 points above the all-linear model (Table 8).

Q2. All-incorrect rollouts. We do not treat all incorrect group specifically. We did not observe that happens often, but if it does, the advantage becomes zero and so does the gradient. We have added the note about it in the Section 4.4.

Thank you for the detailed response, which answers my concerns. I would suggest that you present the motivation for RL more clearly in the paper and this strengthens the rationale of your method. I will raise my score to 3.

Reviewer_MZ9EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the accuracy degradation caused by post-hoc linearization of reasoning models. It then proposes TURNOUT, a hybrid framework that preserves the original full-attention path while training a complementary linear-attention path through distillation. After processing the prompt with full attention, a lightweight router determines when each layer switches to linear attention and releases its unused KV cache. The router is trained with GRPO to balance answer correctness and cache efficiency. Experiments across five reasoning benchmarks demonstrate improved trade-off between accuracy and efficiency over fixed switching and static hybrid baselines.

Critical strengths

The problem is timely and well motivated, and the proposed design is clear and practically appealing. Learning when individual layers should switch provides flexibility across both layers and generation steps. The evaluation covers mathematical, scientific, and coding tasks, with useful ablations supporting the linearization recipe and router design. The overall presentation is clear and well-structured.

Critical weaknesses
  1. Sec. 4 and Fig. 3: the measured throughput and peak GPU memory results compare the full-attention and fully linearized models, but do not include the final adaptive TURNOUT model. Sec. 5 instead evaluates efficiency through an estimated cache-memory proxy. While cache savings are useful, they do not necessarily translate into faster inference, particularly when routing and state-conversion overheads are considered.
  2. Sec. 5.2 and Table 2: to what extent do changes in response length contribute to the reported cache savings? Table 2 provides cache estimates at fixed lengths, but the benchmark comparisons do not report response-length distributions or clearly specify whether the full-attention reference used to calculate cache savings has a matched output length. Please clarify this calculation. Comparisons at matched prompt and output lengths, together with end-to-end measurements on naturally terminated responses, would help distinguish the effects of attention switching from changes in generation length.
  3. Sec. 5.2 and Fig. 6: the router analysis reports only domain-level medians of normalized per-layer switching positions after training. How does this behavior develop during training, particularly given the reported early collapse under the additive, memory-only reward? Appendix B.1 specifies the initialization, but switching-time distributions and representative trajectories at initialization, intermediate checkpoints, and the final checkpoint would help assess convergence and how input-dependent behavior emerges. Reporting absolute token positions alongside normalized positions would also help separate routing changes from changes in response length.
Questions for authors
  1. As mentioned in W1, could the authors report measured peak GPU memory, decoding throughput (tokens/s), and end-to-end latency per problem for TURNOUT and the baselines?
  2. As mentioned in W2, could the authors provide additional length-based analysis?
  3. (minor) Would it be helpful if a transition sentence is provided to connect the name "TURNOUT" with the paper?
3 comments

We thank the reviewer for the feedback. We are glad that the reviewer recognizes the timely motivation, practical design and method flexibility, broad evaluation, useful ablations, and clear presentation. Below we address all the questions/weaknesses raised in the review. We also uploaded updated version of the paper with all the changes marked with blue color.

W1/Q1: Measured latency and memory footprint for the turnout This is an important addition. Indeed, we only report theoretical memory footprint for Turnout in Section 5. To adress this limitation we profiled Full attention, Linear, and Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.), all based on Qwen3-8B, on all MATH500 problems on a single H100 GPU and a 16K-token limit. Latency is the time per problem and includes prefill, routing, and all switching work. Throughput 4K-8K is the decoding speed while the model generates the 4K-th to the 8K-th token of a response. Latency and peak memory 4K-8K are averages over the problems whose prompt and response together have 4K to 8K tokens. The 8K-16K columns are defined in the same way.

ModelTokens/s 4K-8KTokens/s 8K-16KLatency (s) 4K-8KLatency (s) 8K-16KPeak GPU memory (GiB) 4K-8KPeak GPU memory (GiB) 8K-16K
Full39.624.8109.2341.216.517.8
Linear48.048.0119.1301.715.615.6
Turnout43.632.9112.7311.016.617.3

Turnout decodes 10% faster than Full from 4K to 8K tokens and 33% faster from 8K to 16K tokens. From 8K to 16K tokens, it also has 9% lower latency and 0.5 GiB lower peak memory than Full.

We added these measurements to Section 5.3 (new Figure 7: a) throughput, b) latency, and c) peak memory, for all length ranges).

W2/Q2: Additional Length-based analysis The reported cache saving does not come from shorter responses. For each sample, the full-attention cache estimate uses the same prompt and response length as the routed sample, so the saving always compares equal lengths. We have updated the manuscript to meantion this explicitely. We have also added the length histograms for all five benchmarks (Appendix B.6, Figure 8). It shows that on all five benchmarks Turnout and Full generate responses of similar length. Here are the average generation lengths for the refence:

ModelGSM8KMATH-500GPQA-DHumanEval+MBPP+
Full1,4005,2836,4753,1003,073
Linear2,0367,8559,9315,3765,504
Turnout1,3785,3356,2632,8233,130

W3: Switching distribution at initialization and during training We have added the training curves and the switch-token distributions at different steps (Figure 9) as well as the per-layer absolute positions for all routers (Figure 10) to Appendix B.7 of the updated manuscript.

All runs have the same initialization but learn different switching behaviour. Below we report switching tokens index in the training rollouts of Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.) and Turnout (RaddR_{\mathrm{add}}, Mem.) at steps 0, 10, 50 and 99. We report the median switch token over layers that switch before the response ends, and the share of layers that never switch. The full curves over all steps are in the top row of Figure 9, and the distributions at these four steps are in its bottom row:

StepRscaledR_{\mathrm{scaled}}, BE-Cos.: median switch tokenRscaledR_{\mathrm{scaled}}, BE-Cos.: never-switched layersRaddR_{\mathrm{add}}, Mem.: median switch tokenRaddR_{\mathrm{add}}, Mem.: never-switched layers
01820%1800%
1043347%1180%
501,30847%30%
991,33958%00%

At initialization, the two routers behave the same. With Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.), about half of the layers stop switching within the first 10 steps, and the other layers switch later and later. With Turnout (RaddR_{\mathrm{add}}, Mem.), an incorrect answer still receives the efficiency reward, so the router learns to switch as early as possible. This is the collapse reported in Section 5.2.

Absolute switch positions of Turnout (RscaledR_{\mathrm{scaled}}, BE-Cos.) also differ across the evaluation benchmarks. We count a layer as staying in full attention if it switches in fewer than half of the responses. For the other layers, we report the median switch token in generated tokens. The per-layer values are in the left panel of Figure 10:

DomainLayers in full attention (of 36)Median switch token of switching layers
Math (GSM8K, MATH-500)29684
Code (HumanEval+, MBPP+)221,213
Science (GPQA-D)211,186

Q3: Context for the method name Thank you for the suggestion, we have added a fottnote with a short description to the updated vesrion of the manuscript.

Thank you again for your constructive feedback. We hope that we have addressed all your questions and weaknesses:

  • we measured Turnout throughput, latency, and peak memory;
  • we clarified the response length distributions;
  • we provided additional analysis of router training dynamics.

With the discussion period ending soon, we would appreciate your feedback on whether these additions resolve your concerns, and would be happy to clarify any remaining points.

I appreciate the reviewers' careful response. Based on the response, I will raise my score to weak accept (3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.