acceptodds
Under review as a conference paper at ICLR 2027

GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

Abstract

As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck. To address this, we propose GLIDE, a Guided Layerwise Hybrid Attention that strategically integrates sliding-window softmax attention with linear recurrent aggregation. GLIDE is motivated by layer-wise heterogeneity: early layers exhibit high sensitivity to softmax removal, while deeper layers demonstrate redundancy and tolerate aggressive replacement by linear alternatives. Leveraging this insight, GLIDE introduces a layer-wise adaptive mechanism wherein each layer is balanced with fully variable configurations between efficient linear recurrence and full-accuracy softmax. Employing front-loading softmax retention strategies, GLIDE non-uniformly compresses the softmax allocation in the model, reducing aggregate KV cache I/O while retaining model accuracy in sensitive layers. We demonstrate GLIDE’s framework adaptability and superior performance-efficiency tradeoffs with multiple linear and hybrid attention models, reducing end-to-end latency for long-context generation without compromising quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Hybrid Attention, KV Cache Compression, Sliding Window Attention, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Mn8vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GLIDE, a strategy that allocates local softmax attention and linear recurrent computation non-uniformly according to the sensitivity of different layers to linearization. The network is divided into early, middle, and late blocks: early blocks retain the full local softmax window, middle blocks use varying degrees of hybrid computation, and late blocks are fully linearized to explore the trade-off between accuracy and inference efficiency.

The authors evaluate task accuracy across six standard benchmarks, KV-cache I/O, and end-to-end latency on Llama-3-8B and Mistral-7B, comparing GLIDE with native softmax, uniform hybrid attention, and fully linear configurations. They report an improved Pareto trade-off, with reduced KV-cache bandwidth requirements while retaining 92–96% of baseline accuracy, and 1.4–2.0× end-to-end speedups on Llama-3-8B. They also report that GLIDE outperforms uniform hybrid attention at comparable accuracy. However, the speedup is not universal: the reported GLIDE configurations do not reduce latency relative to the native Mistral-7B model at context lengths of 4K or 8K. In addition, the NIAH experiments show that GLIDE variants retain recall on held-out tasks up to 3K tokens compared with pure linear kernels.

Critical strengths
  1. The allocation strategy is simple and practically relevant. Allocating attention computation according to the sensitivity of different layers to linearization offers a reasonable direction for reducing memory traffic during long-context decoding. The three-block design is relatively easy to implement and evaluate, and provides a potentially useful alternative to uniform hybrid attention.
  2. The Llama experiments demonstrate a useful trade-off. Table 2 shows that the (0,0,w) configuration achieves approximately 2× and 1.4× speedups over the hybrid-attention baseline at 4K and 32K, respectively. Meanwhile, Table 1 reports a relatively small decrease in average accuracy, from 69.12 to 67.84. These results support the feasibility of achieving a practical accuracy-efficiency trade-off through non-uniform allocation.
Critical weaknesses
  1. The efficiency accounting is unclear, raising concerns about the reliability of the conclusions. Table 1 reports 128 MB of KV-cache I/O per token for the hybrid baseline, whereas Fig. 7 reports approximately 16.3 GB of accumulated I/O at 128K tokens. If both use the same measurement convention and the total covers 128K generated tokens, the accumulated traffic should be approximately 16.4 TB. The paper should explain this discrepancy and distinguish accumulated KV-cache traffic from peak memory usage when discussing the claimed 2–3× increase in concurrent users. Furthermore, all reported non-vanilla Mistral configurations, including (0,15w/16,w), have higher end-to-end latency than the native model at 4K and 8K. At 32K, their latencies remain close to that of the hybrid baseline. Although these configurations can run at longer sequence lengths where the native model runs out of memory, the current results do not support a general claim that GLIDE accelerates inference across different models.
  2. The comparisons do not sufficiently establish the contribution of the layer-wise allocation strategy itself. The different configurations within GLIDE primarily constitute a sensitivity analysis, and the comparison with existing frameworks is insufficient. The experiments mainly vary parameters within GLIDE, without adequately comparing against existing architectures that apply a uniform hybridization strategy across all layers, as discussed in the introduction. Although the paper includes the uniform hybrid baseline (0,0,0), its total softmax budget differs from those of the non-uniform configurations. It is therefore difficult to determine whether the gains result from a more effective allocation across layers or simply from reducing softmax computation.
Questions for authors
  1. The text states that when (δ<w)(\delta<w), the effective softmax window decreases from $(w) to w−δ)w-\delta), but the summation range in Eq. (2) contains (w+\delta$ positions when the prefix is sufficiently long. Is this a contradiction?
  2. Can the accumulated I/O in Fig. 7 be derived from the per-token I/O in Table 1? Please clarify the relationship between these quantities.
  3. Can the authors add a comparison between uniform allocation and GLIDE under the same total softmax budget and training conditions? Alternatively, can they directly compare GLIDE with the existing architectures discussed in the introduction that use uniform hybrid attention across all layers, to establish the advantage of non-uniform allocation?
  4. The native softmax baseline is not fine-tuned with LoRA, whereas the GLIDE configurations undergo LoRA-based parameter-efficient fine-tuning. How much of the reported benefit is attributable to fine-tuning, and how much to the allocation strategy?
  5. The reported GLIDE configurations are slower than the native Mistral-7B model at 4K and 8K, and provide limited acceleration over the hybrid baseline at 32K. These results do not demonstrate a clear latency advantage on this model. How do the authors explain and address this limitation?
4 comments
AuthorsOpenReview ↗

In response to Question 1

Thank you for your review. The token partition depends on the partition our backbone kernel already uses. For Liger and the experimental results in Table 1, all tokens are fed to GLA, and tokens in the sliding window go through softmax. This is faithful to Liger's own implementation. For the LoLCATS results in Fig. 8, only tokens outside of the sliding window are fed to the linear attention branch. We try to stay faithful to the linear kernels we experimented with, for proper A/B testing.

We also acknowledge the typo in Equations 1–3, and have provided the following fixes:

O_i^{\text{Glide}} = O^{\text{linear}}_{i-(w-\delta)} \oplus O^{\text{SWA}}_{i-(w-\delta)+1}, \tag{1}

O^{\text{SWA}}_{i-(w-\delta)+1} = \frac{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T}) \cdot v_{j'}}{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T})}. \tag{2}

O^{\text{Linear}}_{i-(w-\delta)} = \frac{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T} \cdot v_{j'}}{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T}}. \tag{3}

AuthorsOpenReview ↗

In response to Question 2

Yes. Figure 7 is the running sum, or integral over generated tokens, of the per-token I/O in Table 1.

AuthorsOpenReview ↗

In response to Weakness 2 and Question 3

Based on your feedback, we have decided to do some additional investigations on GLIDE's capabilities versus baseline linear attention kernels under fairer softmax budgets.

We ran two long-context capability experiments, using Liger as the backbone and replacing Liger's original softmax window with GLIDE's layer-varying window. In Pair A, we use 32 layers of baseline Liger kernels and compare them against 32 GLIDE (0,w2,w)(0, \tfrac{w}{2}, w) layers, evaluated on RULER benchmarks up to 32K context length. In Pair B, to investigate hybrid attention setups, we replace half of these hybrid attention layers with full causal softmax layers, keep all other parameters identical, and evaluate both models on held-out NIAH benchmarks.

Within each pair, both models use Llama-3.1-8B (L=32L = 32), window w=64w = 64, and the same training recipe (Alpaca LoRA fine-tuning followed by the same needle-retrieval recall stage). Each softmax token costs 4 KiB of KV I/O per decoded token. The softmax budgets are calculated below.

Pair A

  • Uniform Liger
    • Span 6464 on all 3232 layers
    • 32⋅64=2,04832 \cdot 64 = 2{,}048 softmax tokens (8.0 MiB/token)
  • GLIDE (0,w2,w)(0, \tfrac{w}{2}, w)
    • Span 6464 on 1212 layers, 3232 on 1010 layers and 00 on 1010 layers
    • 1,0881{,}088 softmax tokens (4.25 MiB/token), 46.9% fewer

Pair B

  • Uniform Liger
    • Span 6464 on the 1616 hybrid layers
    • 16N+1,02416N + 1{,}024 softmax tokens (132,096132{,}096 at N=8N = 8K; 516.0 MiB/token)
  • GLIDE (0,w2,w)(0, \tfrac{w}{2}, w)
    • Span 6464 on 66 hybrid layers, 3232 on 55 and 00 on 55
    • 16N+54416N + 544 softmax tokens (131,616131{,}616 at N=8N = 8K; 514.1 MiB/token)

The results can be found here:

In both cases, the GLIDE version showed superior long-context and retrieval capabilities compared to the baseline Liger kernel, with a smaller (Pair A) and a near-identical (Pair B) softmax budget. The only difference is the allocation strategy: our non-uniform, front-loaded strategy versus Liger's uniform allocation. We maintain full softmax windows in earlier layers, while later layers can have their softmax replaced and still improve the model's performance, since sharp softmax signals on every layer, even when unnecessary, can dilute the needle signal.

We hope this provides a satisfactory answer to your question and addresses the weaknesses you raised!

AuthorsOpenReview ↗

In response to Question 4

In Table 1, alongside the baseline Llama/Mistral model comparisons, we also provided GLIDE (0,0,0)(0, 0, 0), which is the baseline Liger implementation under GLIDE. This baseline model natively incorporates LoRA: "Using lightweight fine-tuning with Low-Rank Adaptation (LoRA), Liger restores the performance…" (Lan et al., 2025, Abstract). GLIDE does not incorporate any LoRA fine-tuning on its own. We respect the baseline linear kernel implementations, and compare their baseline (with LoRA) to other GLIDE configurations. By incorporating non-uniform GLIDE (0,w/2,w)(0, w/2, w), we cut KV I/O by half, from 128 to 64 MB/token, while accuracy decreases by only 1.39 points on Llama-3-8B. Both configurations are based on Liger with LoRA, therefore we can attribute the benefits to the allocation strategy.

Reviewer_jrdUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLIDE, a framework that allocates sliding-window softmax attention and linear recurrent attention differently across model depth. Motivated by layer-sensitivity experiments, it preserves more softmax attention in early layers and applies stronger linearization in deeper layers, followed by LoRA fine-tuning to recover accuracy. Experiments cover Llama-3-8B and Mistral-7B on six reasoning and knowledge benchmarks, decoding latency, and synthetic retrieval with several hybrid attention backbones. The paper reports reduced KV-cache traffic and useful latency improvements on Llama, although speedups vary considerably across models and the retrieval experiments use additional full-attention layers.

Critical strengths
  1. Allocating attention budgets according to layer sensitivity is a practical idea. The simple block-wise policy is easy to implement and can build on existing hybrid attention methods.

  2. The Llama results show a useful trade-off. At 32k, the (0,0,w)(0,0,w) configuration is about 1.4× faster than the uniform hybrid baseline, with an average accuracy drop of approximately 1.3 points.

Critical weaknesses
  1. The experiments do not clearly isolate the benefit of nonuniform allocation. The main table lacks a fine-tuned uniform baseline with the same total softmax budget. The retrieval experiments also add six full-attention layers and separate recall training, making it unclear how much improvement comes from GLIDE itself.

  2. The efficiency claims are broader than the evidence supports. On Mistral at 32k, the best configuration is only about 2.7% faster than the hybrid baseline, while two others are slower. The cumulative I/O numbers also appear inconsistent with the reported per-token traffic, weakening the claims about memory savings and serving capacity.

Questions for authors
  1. Does GLIDE outperform uniform and reversed layer allocations under the same total softmax budget and fine-tuning protocol? For retrieval, can you compare these policies with identical full-attention anchors and recall training, while reporting the resulting cache costs?

  2. Do the reported I/O values represent measured traffic, estimated traffic, or resident cache memory? At 128 MB per token, generating 128k tokens would imply roughly 16 TB of cumulative traffic, rather than the 16.3 GB reported in Figure 7. Please also clarify whether recurrent-state reads and writes are included.

  3. Which configurations that preserve accuracy support the headline speedup claims on each model? Table 2 shows substantial gains on Llama but little or no improvement on Mistral. Reporting accuracy, latency, and peak memory for the same configurations would help establish the practical trade-off.

  4. What token partition is used in the implementation? Equation 2 starts the softmax window at i−w−δ+1i-w-\delta+1, giving a window of w+δw+\delta tokens. This conflicts with the stated effective window w−δw-\delta and the claim that δ=w\delta=w produces fully linear attention.

4 comments
AuthorsOpenReview ↗

In response to Weakness 1 and Question 1.

Based on your feedback, we have decided to do some additional investigations on GLIDE's capabilities versus baseline linear attention kernels under fairer softmax budgets.

We ran two long-context capability experiments, using Liger as the backbone and replacing Liger's original softmax window with GLIDE's layer-varying window. In Pair A, we use 32 layers of baseline Liger kernels and compare them against 32 GLIDE (0,w2,w)(0, \tfrac{w}{2}, w) layers, evaluated on RULER benchmarks up to 32K context length. In Pair B, to investigate hybrid attention setups, we replace half of these hybrid attention layers with full causal softmax layers, keep all other parameters identical, and evaluate both models on held-out NIAH benchmarks.

Within each pair, both models use Llama-3.1-8B (L=32L = 32), window w=64w = 64, and the same training recipe (Alpaca LoRA fine-tuning followed by the same needle-retrieval recall stage). Each softmax token costs 4 KiB of KV I/O per decoded token. The softmax budgets are calculated below.

Pair A

  • Uniform Liger
    • Span 6464 on all 3232 layers
    • 32⋅64=2,04832 \cdot 64 = 2{,}048 softmax tokens (8.0 MiB/token)
  • GLIDE (0,w2,w)(0, \tfrac{w}{2}, w)
    • Span 6464 on 1212 layers, 3232 on 1010 layers and 00 on 1010 layers
    • 1,0881{,}088 softmax tokens (4.25 MiB/token), 46.9% fewer

Pair B

  • Uniform Liger
    • Span 6464 on the 1616 hybrid layers
    • 16N+1,02416N + 1{,}024 softmax tokens (132,096132{,}096 at N=8N = 8K; 516.0 MiB/token)
  • GLIDE (0,w2,w)(0, \tfrac{w}{2}, w)
    • Span 6464 on 66 hybrid layers, 3232 on 55 and 00 on 55
    • 16N+54416N + 544 softmax tokens (131,616131{,}616 at N=8N = 8K; 514.1 MiB/token)

The results can be found here:

In both cases, the GLIDE version showed superior long-context and retrieval capabilities compared to the baseline Liger kernel, with a smaller (Pair A) and a near-identical (Pair B) softmax budget. The only difference is the allocation strategy: our non-uniform, front-loaded strategy versus Liger's uniform allocation. We maintain full softmax windows in earlier layers, while later layers can have their softmax replaced and still improve the model's performance, since sharp softmax signals on every layer, even when unnecessary, can dilute the needle signal.

We hope this provides a satisfactory answer to your question and addresses the weaknesses you raised!

AuthorsOpenReview ↗

In response to Question 2

The values in Table 1 are analytic estimates of KV-cache read traffic rather than hardware measurements. These are the same 4 KiB per cached token per layer for Llama-3-8B and Mistral-7B, since both have 8 KV heads of dimension 128 in bf16.

We acknowledge the arithmetic for Fig. 7 is inaccurate, and have redone the figure. At 128 MiB per token, generating 128K tokens gives about 16 TiB of cumulative traffic. The corrected values at 128K are:

  • (0,0,0)(0, 0, 0): 15.9 TiB
  • (0,15w/16,w)(0, 15w/16, w): 5.8 TiB
  • Vanilla softmax: about 1.0 PiB

The corrected figure can be viewed here: corrected-figure-7.tiiny.site

Recurrent-state reads and writes are not included. That cost is constant per token, independent of both sequence length and the δ\delta-allocation.

AuthorsOpenReview ↗

In response to Question 4

Thank you for the review! These were sign-error typos. We have added parentheses around the window-size term (w−δ)(w-\delta) in Equations 1–3, which gives their correct form:

O_i^{\text{Glide}} = O^{\text{linear}}_{i-(w-\delta)} \oplus O^{\text{SWA}}_{i-(w-\delta)+1}, \tag{1}

O^{\text{SWA}}_{i-(w-\delta)+1} = \frac{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T}) \cdot v_{j'}}{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T})}. \tag{2}

O^{\text{Linear}}_{i-(w-\delta)} = \frac{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T} \cdot v_{j'}}{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T}}. \tag{3}

AuthorsOpenReview ↗

Revised PDF, Mistral Results and Overall Summary - Gentle reminder

We have uploaded a revised version of the paper, with all changes highlighted in blue. Below we point to where each of your questions and concerns is addressed.

  • Q1 (uniform vs. GLIDE under the same softmax budget and training recipe). See Section 4.2 (Budget-Matched Comparison), Figure 9, and Appendix E for the full budget calculations and results. Within each pair, the backbone (Llama-3.1-8B), window size and training recipe (Alpaca LoRA fine-tuning followed by the same recall stage) are identical, with different allocation strategies. GLIDE consistently outperforms baseline Liger at retrieval tasks (RULER and held-out NIAH) at 46.9% less softmax tokens.
  • Q2 (nature of the I/O values; Figure 7). Section 4.1 now defines KV I/O as an analytic estimate of the per-token softmax-window KV-cache read traffic, excluding recurrent-state reads and writes. Figure 7 has been fixed. The KV I/O Efficient Reasoning Scalability paragraph in Section 4.2 reports the corrected values (15.9 TiB for (0,0,0)(0,0,0) and 5.8 TiB for (0,15w/16,w)(0,15w/16,w) at 128K).
  • Q3 (accuracy-preserving configurations and the Mistral speedup). We re-measured Tables 1 and 2 and Figure 6(b), and revised Section 4.2 (End-to-End Latency). On Mistral-7B, (0,w/2,w)(0, w/2, w) retains 93% of vanilla accuracy (67.17 vs. 72.37) while reducing 32K latency by 15% relative to the hybrid baseline and 27% relative to vanilla softmax. On Llama-3-8B, the same configuration retains 94% of vanilla accuracy with a 1.4–2.0$\times$ speedup over the hybrid baseline.
  • Q4 (token partition in Equations 1–3). Equations 1–3 in Section 3.2 now use i−(w−δ)i-(w-\delta), so the softmax window contains w−δw-\delta tokens and δ=w\delta = w yields fully linear attention.

We would be happy to clarify any remaining concerns or questions that you may have. We believe these fixes in the revised PDF elevate the contribution and impact of GLIDE, and we hope they lead to a more positive assessment of our work.

Thank you for your time and engagement!

Reviewer_qiyoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduced GLIDE, a layer-wise hybrid attention methods that perform layer-wise adaptation for efficient memory footprint. It uses less than 128 MB of KV memory in comparison to 4 GB memory by full softmax for Llama-3-8B and Mistral-7B while maintaining decent performance.

Critical strengths

GLIDE provides a simple way to reduce reported KV-cache I/O by exploiting differences in sensitivity to linearization across layers. The Llama-3-8B experiments demonstrate meaningful latency improvements over the uniform-hybrid baseline while retaining much of its task performance.

Critical weaknesses

The baseline in main tables are only the full softmax despite there are a lot of KV-cache optimization methods existing, e.g. PyramidKV already saves 88 percent. The efficiency gains are not consistent across models. While GLIDE achieves clear speedups on Llama-3-8B, its best Mistral-7B configuration improves latency by only 2.6% at 32K, while average accuracy drops from 71.31 to 68.23 (Tables 1–2). The other two configurations are slower than the uniform-hybrid baseline. This variation across models makes the benefit of additional compression less convincing, especially given the performance degradation.

Questions for authors

Could you provider comparison with other memory-efficient methods? Why do the Mistral configurations show little or no latency improvement over the uniform-hybrid baseline despite substantially lower reported KV I/O?

1 comment
AuthorsOpenReview ↗

Revised PDF, Mistral Results and Overall Summary - Gentle reminder

We have uploaded a revised version of the paper, with all changes highlighted in blue. Below we point to where each of your questions and concerns is addressed.

  • Comparison with other memory-efficient methods (Q1, "PyramidKV already saves 88 percent"). Section 2 (Policies of Sliding Window Attention) now explains that KV-cache compression methods such as H2O, SnapKV, StreamingLLM and PyramidKV are training-free eviction policies for an unmodified, full softmax model, while GLIDE is a framework for layer-dynamic softmax allocation in hybrid attention models. We therefore focus our comparisons on other hybrid attention methods (Liger, LoLCATS, Lizard).
  • Baselines beyond full softmax. Table 1 includes the uniform hybrid baseline (GLIDE (0,0,0)(0,0,0), i.e. Liger with LoRA) and the fully linear baseline (w,w,w)(w,w,w). We have also added a budget-matched comparison against uniform Liger (Section 4.2, Budget-Matched Comparison, Figure 9, with full details in Appendix E). GLIDE consistently outperforms baseline Liger at retrieval tasks (RULER and held-out NIAH) at 46.9% less softmax tokens.
  • Mistral latency and consistency across models. We have reran the Mistral latency experiments. All Mistral results in Tables 1 and 2 and Figure 6(b) have been re-measured, and Section 4.2 (End-to-End Latency) is rewritten accordingly. At 32K, GLIDE (0,w/2,w)(0, w/2, w) is 15% faster than the hybrid baseline (1198.55s →\rightarrow 1013.71s) and 27% faster than vanilla softmax (1394.15s →\rightarrow 1013.71s), with 67.17 average accuracy versus 67.48 for the hybrid baseline.

We would be happy to clarify any remaining concerns or questions that you may have. We believe these fixes in the revised PDF elevate the contribution and impact of GLIDE, and we hope they lead to a more positive assessment of our work.

Thank you for your time and engagement!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.