- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GLIDE, a strategy that allocates local softmax attention and linear recurrent computation non-uniformly according to the sensitivity of different layers to linearization. The network is divided into early, middle, and late blocks: early blocks retain the full local softmax window, middle blocks use varying degrees of hybrid computation, and late blocks are fully linearized to explore the trade-off between accuracy and inference efficiency.
The authors evaluate task accuracy across six standard benchmarks, KV-cache I/O, and end-to-end latency on Llama-3-8B and Mistral-7B, comparing GLIDE with native softmax, uniform hybrid attention, and fully linear configurations. They report an improved Pareto trade-off, with reduced KV-cache bandwidth requirements while retaining 92–96% of baseline accuracy, and 1.4–2.0× end-to-end speedups on Llama-3-8B. They also report that GLIDE outperforms uniform hybrid attention at comparable accuracy. However, the speedup is not universal: the reported GLIDE configurations do not reduce latency relative to the native Mistral-7B model at context lengths of 4K or 8K. In addition, the NIAH experiments show that GLIDE variants retain recall on held-out tasks up to 3K tokens compared with pure linear kernels.
Critical strengths
- The allocation strategy is simple and practically relevant. Allocating attention computation according to the sensitivity of different layers to linearization offers a reasonable direction for reducing memory traffic during long-context decoding. The three-block design is relatively easy to implement and evaluate, and provides a potentially useful alternative to uniform hybrid attention.
- The Llama experiments demonstrate a useful trade-off. Table 2 shows that the (0,0,w) configuration achieves approximately 2× and 1.4× speedups over the hybrid-attention baseline at 4K and 32K, respectively. Meanwhile, Table 1 reports a relatively small decrease in average accuracy, from 69.12 to 67.84. These results support the feasibility of achieving a practical accuracy-efficiency trade-off through non-uniform allocation.
Critical weaknesses
- The efficiency accounting is unclear, raising concerns about the reliability of the conclusions. Table 1 reports 128 MB of KV-cache I/O per token for the hybrid baseline, whereas Fig. 7 reports approximately 16.3 GB of accumulated I/O at 128K tokens. If both use the same measurement convention and the total covers 128K generated tokens, the accumulated traffic should be approximately 16.4 TB. The paper should explain this discrepancy and distinguish accumulated KV-cache traffic from peak memory usage when discussing the claimed 2–3× increase in concurrent users. Furthermore, all reported non-vanilla Mistral configurations, including (0,15w/16,w), have higher end-to-end latency than the native model at 4K and 8K. At 32K, their latencies remain close to that of the hybrid baseline. Although these configurations can run at longer sequence lengths where the native model runs out of memory, the current results do not support a general claim that GLIDE accelerates inference across different models.
- The comparisons do not sufficiently establish the contribution of the layer-wise allocation strategy itself. The different configurations within GLIDE primarily constitute a sensitivity analysis, and the comparison with existing frameworks is insufficient. The experiments mainly vary parameters within GLIDE, without adequately comparing against existing architectures that apply a uniform hybridization strategy across all layers, as discussed in the introduction. Although the paper includes the uniform hybrid baseline (0,0,0), its total softmax budget differs from those of the non-uniform configurations. It is therefore difficult to determine whether the gains result from a more effective allocation across layers or simply from reducing softmax computation.
Questions for authors
- The text states that when , the effective softmax window decreases from $(w) to , but the summation range in Eq. (2) contains (w+\delta$ positions when the prefix is sufficiently long. Is this a contradiction?
- Can the accumulated I/O in Fig. 7 be derived from the per-token I/O in Table 1? Please clarify the relationship between these quantities.
- Can the authors add a comparison between uniform allocation and GLIDE under the same total softmax budget and training conditions? Alternatively, can they directly compare GLIDE with the existing architectures discussed in the introduction that use uniform hybrid attention across all layers, to establish the advantage of non-uniform allocation?
- The native softmax baseline is not fine-tuned with LoRA, whereas the GLIDE configurations undergo LoRA-based parameter-efficient fine-tuning. How much of the reported benefit is attributable to fine-tuning, and how much to the allocation strategy?
- The reported GLIDE configurations are slower than the native Mistral-7B model at 4K and 8K, and provide limited acceleration over the hybrid baseline at 32K. These results do not demonstrate a clear latency advantage on this model. How do the authors explain and address this limitation?
4 comments
In response to Question 2
Yes. Figure 7 is the running sum, or integral over generated tokens, of the per-token I/O in Table 1.
In response to Weakness 2 and Question 3
Based on your feedback, we have decided to do some additional investigations on GLIDE's capabilities versus baseline linear attention kernels under fairer softmax budgets.
We ran two long-context capability experiments, using Liger as the backbone and replacing Liger's original softmax window with GLIDE's layer-varying window. In Pair A, we use 32 layers of baseline Liger kernels and compare them against 32 GLIDE layers, evaluated on RULER benchmarks up to 32K context length. In Pair B, to investigate hybrid attention setups, we replace half of these hybrid attention layers with full causal softmax layers, keep all other parameters identical, and evaluate both models on held-out NIAH benchmarks.
Within each pair, both models use Llama-3.1-8B (), window , and the same training recipe (Alpaca LoRA fine-tuning followed by the same needle-retrieval recall stage). Each softmax token costs 4 KiB of KV I/O per decoded token. The softmax budgets are calculated below.
Pair A
- Uniform Liger
- Span on all layers
- softmax tokens (8.0 MiB/token)
- GLIDE
- Span on layers, on layers and on layers
- softmax tokens (4.25 MiB/token), 46.9% fewer
Pair B
- Uniform Liger
- Span on the hybrid layers
- softmax tokens ( at K; 516.0 MiB/token)
- GLIDE
- Span on hybrid layers, on and on
- softmax tokens ( at K; 514.1 MiB/token)
The results can be found here:
- Pair A: glide-pair-a.tiiny.site
- Pair B: glide-pair-b.tiiny.site
In both cases, the GLIDE version showed superior long-context and retrieval capabilities compared to the baseline Liger kernel, with a smaller (Pair A) and a near-identical (Pair B) softmax budget. The only difference is the allocation strategy: our non-uniform, front-loaded strategy versus Liger's uniform allocation. We maintain full softmax windows in earlier layers, while later layers can have their softmax replaced and still improve the model's performance, since sharp softmax signals on every layer, even when unnecessary, can dilute the needle signal.
We hope this provides a satisfactory answer to your question and addresses the weaknesses you raised!
In response to Question 4
In Table 1, alongside the baseline Llama/Mistral model comparisons, we also provided GLIDE , which is the baseline Liger implementation under GLIDE. This baseline model natively incorporates LoRA: "Using lightweight fine-tuning with Low-Rank Adaptation (LoRA), Liger restores the performance…" (Lan et al., 2025, Abstract). GLIDE does not incorporate any LoRA fine-tuning on its own. We respect the baseline linear kernel implementations, and compare their baseline (with LoRA) to other GLIDE configurations. By incorporating non-uniform GLIDE , we cut KV I/O by half, from 128 to 64 MB/token, while accuracy decreases by only 1.39 points on Llama-3-8B. Both configurations are based on Liger with LoRA, therefore we can attribute the benefits to the allocation strategy.
In response to Question 1
Thank you for your review. The token partition depends on the partition our backbone kernel already uses. For Liger and the experimental results in Table 1, all tokens are fed to GLA, and tokens in the sliding window go through softmax. This is faithful to Liger's own implementation. For the LoLCATS results in Fig. 8, only tokens outside of the sliding window are fed to the linear attention branch. We try to stay faithful to the linear kernels we experimented with, for proper A/B testing.
We also acknowledge the typo in Equations 1–3, and have provided the following fixes:
O_i^{\text{Glide}} = O^{\text{linear}}_{i-(w-\delta)} \oplus O^{\text{SWA}}_{i-(w-\delta)+1}, \tag{1}
O^{\text{SWA}}_{i-(w-\delta)+1} = \frac{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T}) \cdot v_{j'}}{\sum_{j'=i-(w-\delta)+1}^{i} \exp(q_i k_{j'}^{T})}. \tag{2}
O^{\text{Linear}}_{i-(w-\delta)} = \frac{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T} \cdot v_{j'}}{\sum_{j'=1}^{i-(w-\delta)} \phi(q_i) \cdot \phi(k_{j'})^{T}}. \tag{3}