acceptodds
Under review as a conference paper at ICLR 2027

IGAL: Importance-Guided Attention Linearization of Large Language Models

Abstract

The quadratic cost of self-attention makes long-context processing expensive for pretrained Transformers. Converting self-attention to more efficient alternatives, such as sliding-window attention (SWA) and linear attention (LA), reduces this cost but can degrade accuracy, particularly on tasks requiring long-range retrieval. Attention heads exhibit substantial functional heterogeneity, with different heads specializing in local interactions and long-range retrieval. Existing conversion methods typically approximate each pretrained attention head with a combination of LA and SWA, relying on head-specific mixing weights and/or branch parameters to adapt this hybrid form to different functional roles. In contrast, we leverage pretrained head specialization to directly determine the attention operator assigned to each head. Specifically, we propose Importance-Guided Attention Linearization (IGAL), which repurposes existing head-importance scores—previously used to allocate larger KV-cache budgets—to guide heterogeneous attention conversion. IGAL converts a small subset of high-scoring heads to LA for long-range modeling, while converting the remaining heads to SWA for local interactions. We further find that the optimal proportion of LA heads differs between short- and long-context tasks. To accommodate both regimes within a single checkpoint, we introduce IGAL-R, which sets the LA budget using the prompt length and learns which heads receive it with a lightweight prompt-level router, fine-tuned on a mixture of general-language and retrieval data. Under task-specific fine-tuning on Llama 3.1 and Qwen3, IGAL improves general-language and long-context retrieval performance over the existing state-of-the-art linearization methods. On Llama-3.1-8B, assigning only of GQA groups to LA achieves needle-in-a-haystack retrieval accuracy at 32K after 1K-length data fine-tuning, with a smaller decoding cache than full attention. Under matched mixed-data fine-tuning, IGAL-R raises 16–32K retrieval accuracy from to over static IGAL, with comparable general-language performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, efficient inference, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pUuqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IGAL, a framework for converting pretrained Transformers into models with linear attention complexity and bounded decoding-state memory. It reuses HeadKV importance scores to assign a small subset of GQA groups to recurrent linear attention and the remaining groups to sliding-window softmax attention. Adaptation uses a single stage of task-based fine-tuning with LoRA and trainable Q/K convolutions. IGAL-R additionally adjusts the recurrent budget by prompt length, learns prompt-conditioned group selection, and optionally rescales forget gates during inference. Evaluation covers Llama-3.1-8B-Instruct and Qwen3-8B, short-context language benchmarks, and single-needle retrieval up to 32K. The paper reports strong retrieval performance, substantial improvements over random allocation at a matched recurrent budget, and a constant 14 MiB decoding cache for the reported Llama configuration. A mixed-data IGAL-R checkpoint also substantially improves retrieval while retaining competitive short-context performance.

Critical strengths
  1. Strong evidence for importance-guided recurrent allocation. Table 4 holds the GLA operator and the 6.25% recurrent-group budget fixed while changing the allocation criterion. HeadKV-R improves 16–32K retrieval accuracy from 28.33% under random allocation to 83.33%. This directly supports the paper’s central design choice and demonstrates a consequential benefit beyond simply adding recurrent memory.

  2. A compelling bounded-memory retrieval result. The retrieval-specific configuration reaches 89.33% accuracy at 32K on Llama after the reported 1K-length fine-tuning. Table 3 also measures a constant 14 MiB persistent cache, compared with 4096 MiB for full attention at 32K, with an explicit accounting protocol. Together, these results demonstrate a useful retrieval–memory trade-off.

Critical weaknesses
  1. The incremental value of learned prompt-conditioned selection remains unclear. IGAL-R changes the length-dependent recurrent budget, learns which groups receive that budget, and applies inference-time gate scaling. Table 5 shows that IGAL-R already improves long retrieval from 21.67% to 51.67% without scaling, and that scaling further increases it to 78.33%. However, there is no deterministic length-conditioned allocation baseline using the same budgets and gate scaling, nor a static baseline with matched scaling. The benefit of the complete configuration is supported, but the necessity of learning prompt-dependent group selection is not established.

  2. Long-context validation is restricted to single-needle retrieval. The long-context results cover S-NIAH variants, while the language benchmarks primarily test short-context capabilities. These evaluations do not establish whether the conversion preserves long-context abilities requiring multiple pieces of evidence or broader document understanding. This limits the demonstrated significance of the framework beyond retrieval.

Questions for authors
  1. Controlled evidence: Under identical mixed-data training, GLA operators, and inference-time gate scaling, how does IGAL-R compare with a deterministic policy that uses the same short/long recurrent budgets but always selects groups by the fixed importance ranking? What happens when the static IGAL-GLA baseline also receives gate scaling?

  2. Training clarification: What are the prompt-length distributions used to train IGAL-R, and what fraction of examples satisfy (T_p \geq 1024)? Appendix A.3 states that only long-regime response losses train the selector. Please clarify how this protocol relates to the reported 1K-length retrieval fine-tuning and whether the learned routes depart from their initialization.

  3. Calibration clarification: Were importance scores computed separately for each backbone or transferred from existing scores? What calibration data and context lengths were used, and how were these examples separated from the training and evaluation data? This matters for reproducibility and the interpretation of the short-to-long extrapolation result.

5 comments

W1 / Q1. Controlled evidence for learned routing

We thank the reviewer for suggesting these controlled comparisons. We have now trained a deterministic length-conditioned baseline (B) under the same mixed-data training setup and GLA operators as IGAL-R (C), with identical short-/long-prompt LA budgets (3.125% / 6.25%). Unlike IGAL-R, B selects groups solely according to the fixed HeadKV ranking, without learned routing. We also evaluate the static IGAL-GLA baseline (A) with the same inference-time gate scaling.

RowLA budget (short / long)Group selectionGate scaling βLM-EvalNIAH 2K to 8KNIAH 16K to 32K
A. Static IGAL-GLA6.25 / 6.25HeadKV rankingoff72.0670.0021.67
A. Static IGAL-GLA (new)6.25 / 6.25HeadKV rankingon72.0687.7838.33
B. Length rule (new)3.125 / 6.25HeadKV ranking, no routeroff71.7992.2241.67
B. Length rule (new)3.125 / 6.25HeadKV ranking, no routeron71.7995.5670.00
C. IGAL-R3.125 / 6.25Learned routeroff72.4790.0051.67
C. IGAL-R3.125 / 6.25Learned routeron72.4794.4478.33

Learned routing under matched budgets and scaling (B vs. C). At 16K–32K, IGAL-R improves retrieval accuracy from 70.00% to 78.33% with gate scaling (+8.33 points), and from 41.67% to 51.67% without scaling (+10.00 points). Since B and C share the same training setup, LA budgets, and gate-scaling settings, these comparisons isolate the incremental benefit of learned group selection beyond length-dependent allocation. At 2K–8K, the deterministic baseline performs slightly better with scaling (95.56% vs. 94.44%), indicating that learned routing's advantage is concentrated at longer contexts.

Gate scaling for static IGAL-GLA (A). Applying the same scaling improves 16K–32K retrieval accuracy from 21.67% to 38.33%, confirming that scaling also benefits the static baseline. Nevertheless, this remains below both the deterministic length rule (70.00%) and IGAL-R (78.33%) under matched scaling.

Overall, these controlled comparisons show that learned group selection provides additional long-context retrieval gains beyond length-dependent budgeting and gate scaling, improving 16K–32K accuracy by 8.33 points over the deterministic baseline without increasing the LA budget.

Q2. Training regimes, prompt lengths, and learned route changes

Clarifying training vs. inference regimes. We apologize for the ambiguity in the submission. During training, regimes are assigned by data source rather than prompt length: all S-NIAH examples use the long regime (16 LA groups, router active), while Alpaca examples use the short regime (8 groups), regardless of TpT_p. The threshold Tp≥1024T_p \geq 1024 applies only at inference. Thus, "only long-regime response losses train the selector" means that the router receives training signals from the response tokens of all S-NIAH examples, but none from Alpaca.

This source-based assignment is intentional: applying the inference-time length rule during training would exclude 80.7% of S-NIAH prompts from router training, leaving only 2,014 long prompts, almost all from S-NIAH-1/2. Source-based assignment exposes the router to the full retrieval training distribution.

Prompt-length distribution. The mixed-data setup interleaves Alpaca-clean and S-NIAH examples one to one. Here, TpT_p is the position of the first supervised token.

SourcenT_p mean ± stdT_p rangeAnswer lengthT_p ≥ 1024
Alpaca-clean51,76048.0 ± 20.129–633139.3 ± 127.00.00%
S-NIAH-3 (UUID; "1K" retrieval)8,456770.3 ± 50.9661–1,21217–310.17% (14)
S-NIAH-1 (numeric)1,0001,226.5 ± 1.71,223–1,2334100%
S-NIAH-2 (numeric)1,0001,188.1 ± 75.71,027–1,4744100%
All S-NIAH10,456853.9 ± 179.6661–1,47420.0 ± 8.019.26% (2,014)

Only 19.26% of S-NIAH examples satisfy Tp≥1024T_p \geq 1024 (approximately 9.63% of all training examples under one-to-one interleaving). Nevertheless, all 10,456 S-NIAH examples contribute to router training under the source-based protocol.

Relation to 1K retrieval fine-tuning. The "1K" retrieval dataset refers to S-NIAH-3 (mean Tp=770T_p=770), used in the task-specific training setup. The mixed-data setup used by IGAL-R and static IGAL-GLA "Mix" additionally includes 2,000 S-NIAH-1/2 examples with approximately 1.2K-token prompts. Thus, the router is trained on retrieval prompts spanning 661–1,474 tokens, but activated at inference only when Tp≥1024T_p \geq 1024. Evaluation at 2K–32K therefore tests extrapolation beyond the training prompt lengths.

Do learned routes depart from initialization? Yes. The router is initialized to reproduce the fixed HeadKV ranking, but its learned selections differ from this initialization on every examined prompt.

  • Training prompts: Across 10,456 S-NIAH prompts, the router consistently retains 5 of the 8 initially selected switchable groups, replacing the other 3. We observe 8 distinct routes, with the three most common accounting for 95.7% of prompts (58.3%, 27.9%, and 9.6%). They share two substitutions, (16,6)(16,6) and (20,6)(20,6), and differ in the third, where (ℓ,g)(\ell,g) denotes layer and KV group.
  • Evaluation prompts: Across various RULER prompts (1K–32K), the router never predicts its initial allocation. We observe two distinct routes, differing in one selected group.
  • Router parameters: The prompt-conditioned score term reaches ∥Weˉ(x)∥∞=3.80\|W\bar e(x)\|_\infty=3.80, compared with a maximum absolute bias of 0.999. The learned group ranking has a Spearman correlation of only 0.26 with the initial importance ranking.

Overall, the router learns to refine the initial importance-based allocation rather than simply reproducing the HeadKV ranking. Its selections also vary across prompts, although most groups remain shared.

Q3. Calibration of the importance scores

Computed per backbone. The HeadKV scores are computed separately for each backbone: we ran HeadKV's released head-detection script on the frozen Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct and Qwen3-8B/14B, rather than reusing scores released for other models.

Calibration data. Calibration uses HeadKV's own needle-in-a-haystack probes (150 retrieval probes, and 100 retrieval-reasoning probes for the R2 variant) with contexts of up to 10K tokens. It needs only forward passes and updates no parameters. These probes serve only to rank heads. They are not part of our training or evaluation data, their needles (natural-language facts) differ from the magic-number/UUID needles used in S-NIAH, and their text does not overlap with the RULER evaluation inputs.

Short-to-long extrapolation. Calibration contexts (up to 10K tokens) are longer than our training prompts (at most 1.5K) but shorter than the evaluation lengths (up to 32K). To check that the selection does not depend on the long probes, we recomputed the scores from probes of at most 2.5K tokens only: 15 of the 16 selected groups on Llama-3.1-8B and 33 of the 36 on Qwen3-8B remain the same.

W2. Long-context evaluation beyond single-needle retrieval

We thank the reviewer for highlighting this limitation. We agree that single-needle retrieval alone is insufficient to assess long-context capabilities requiring the integration of multiple pieces of evidence.

To address this concern, we have selected BABILong, a benchmark that evaluates long-context reasoning by requiring models to retrieve and combine information distributed across lengthy contexts. Unlike single-needle retrieval, BABILong includes tasks involving multi-hop reasoning and the integration of multiple relevant facts, providing a more comprehensive evaluation of long-context capabilities.

We are currently running BABILong experiments to assess whether IGAL preserves these capabilities beyond single-needle retrieval. The experiments are still in progress, and we will report the results once available.

Reviewer_nPnCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IGAL, a framework for converting pretrained LLMs into full-attention-free models by assigning linear attention (LA) to selected GQA groups and sliding-window attention (SWA) to the remaining groups. The allocation is guided by HeadKV-derived group importance scores. The converted model reuses pretrained weights and is adapted through one-stage fine-tuning without a separate attention-distillation stage. The paper also introduces IGAL-R, which determines the LA budget from prompt length and uses a learned prompt-level router to select additional LA groups from a switchable pool. Experiments demonstrate competitive general-language performance, improved long-context retrieval, and substantial reductions in decoding-cache memory.

Critical strengths
  • Simple and well-motivated design : The idea of using pretrained head specialization to guide the assignment of LA and SWA is simple and compelling. Under the same LA budget, importance-guided allocation substantially outperforms random allocation, providing evidence that the gains come from selecting appropriate heads rather than merely introducing recurrent memory.
Critical weaknesses
  • Limited evaluation of end-to-end inference latency : The efficiency evaluation focuses on decoding memory and computational complexity, without reporting measured end-to-end latency or throughput. While the memory savings are clearly demonstrated, the runtime implications of heterogeneous operator assignment and the additional recurrence computation over switchable groups during IGAL-R prefill remain unclear.

  • Unclear benefit of combining static importance ranking with learned routing: IGAL-R uses HeadKV scores to define fixed and switchable groups, then applies learned routing within the switchable pool. However, the experiments do not isolate the benefit of prompt-dependent group selection from that of the length-dependent LA budget. A baseline that uses the same budget schedule but selects groups solely by the static HeadKV ranking would clarify whether learned routing is necessary.

Questions for authors
  • What are the measured end-to-end latency and throughput of IGAL and IGAL-R compared with the evaluated baselines?

  • What is the benefit of combining HeadKV ranking with learned routing in IGAL-R?

4 comments

W1 / Q1. Measured Inference Efficiency and Routing Overhead

We thank the reviewer for raising this concern. We have measured prefill latency, decoding latency, and serving throughput, with particular attention to the overhead of heterogeneous operator assignment and IGAL-R's additional recurrence.

Setup. One NVIDIA RTX A6000 (48 GB), bf16, greedy decoding; full attention uses FlashAttention-2. Prefill is the median of five warmed-up runs (three at ≥16K); decode throughput is measured over 32 warmed-up steps.

1. Prefill cost of heterogeneous operator assignment.

Prefill, batch 1 (s; ratio to full attention)1K4K8K16K32K
Full attention (FA2)0.170.651.353.047.66
SWA-640.18 (1.07×)0.66 (1.01×)1.28 (0.95×)2.54 (0.84×)5.11 (0.67×)
IGAL0.19 (1.12×)0.68 (1.04×)1.31 (0.97×)2.60 (0.86×)5.22 (0.68×)

Our implementation executes the SWA kernel for all heads and applies recurrence only to LA heads, avoiding explicit gathering/scattering of head subsets. This adds 2% to 5% to prefill time over all-SWA. Relative to full attention, IGAL is slower at 1K, reaches parity near 8K, and takes 0.68× its time at 32K.

2. Additional prefill overhead of IGAL-R.

ContextIGAL (s)IGAL-R (s)Time overheadExtra peak memory
1K0.1890.212+12.1%+0.03 GiB
4K0.6760.722+6.9%+0.11 GiB
8K1.3131.399+6.6%+0.22 GiB
16K2.6032.767+6.3%+0.44 GiB
32K5.2155.513+5.7%+0.88 GiB

Computing recurrent states for the switchable candidate pool plus prompt-dependent routing adds 12.1% at 1K, falling to 5.7% at 32K. The extra memory is transient prefill memory, not persistent decoding state.

3. Decoding capacity and throughput.

Decoding (1K / 32K)Max concurrent sequencesTokens/s at 90% of max batchBatch-1 wall time (ms/token)Batch-1 GPU-busy time (ms/token)
Full attention (FA2)240 / 72,358 / 10031.9 / 33.524.3 / 30.0
SWA-642,831 / 2,8312,737 / 2,72141.1 / 41.324.9 / 24.9
IGAL2,057 / 2,0572,640 / 2,59349.1 / 49.325.5 / 25.5
IGAL-R2,055 / 2,0552,558 / 2,54352.6 / 53.626.2 / 26.2

At 32K, IGAL and IGAL-R fit 2,057 and 2,055 concurrent sequences versus 7 for full attention, and reach 2,593 and 2,543 tokens/s versus 100 when each runs at 90% of its own maximum batch. This gap reflects the larger feasible batch as well as runtime efficiency. IGAL stays within 5% of all-SWA throughput, IGAL-R within 2% to 3% of IGAL, and IGAL fits 4.3× to 6.3× as many sequences as LAWCAT, LoLCATs, and Liger.

4. Batch-1 per-token decoding latency. At batch 1, the current implementation is slower: 49.1 ms/token at 1K versus 31.9 ms for FlashAttention-2. GPU-busy time explains only 1.2 ms of this 17.2 ms gap (25.5 vs. 24.3 ms), and at 32K it is lower for IGAL (25.5 vs. 30.0 ms). IGAL launches more kernels per step (1,939 vs. 1,558), which is consistent with launch/host overhead dominating the gap and leaves room for kernel fusion and runtime optimization.

Overall, heterogeneous assignment adds modest prefill cost, and IGAL-R's extra recurrence adds a limited prefill cost while leaving decoding capacity and throughput essentially unchanged.

W2 / Q2. What does learned routing add on top of the HeadKV ranking?

To answer this question, we have now trained a deterministic length-rule baseline (B) with the same short-/long-prompt LA budgets as IGAL-R (C): 3.125% / 6.25%. Instead of using a learned router, B always activates the next-ranked groups from the HeadKV ordering on long prompts. We also evaluate all variants with and without inference-time gate scaling.

RowLA budget (short / long)Group selectionGate scaling βLM-EvalNIAH 2K to 8KNIAH 16K to 32K
A. Static IGAL-GLA6.25 / 6.25HeadKV rankingoff72.0670.0021.67
A. Static IGAL-GLA (new)6.25 / 6.25HeadKV rankingon72.0687.7838.33
B. Length rule (new)3.125 / 6.25HeadKV ranking, no routeroff71.7992.2241.67
B. Length rule (new)3.125 / 6.25HeadKV ranking, no routeron71.7995.5670.00
C. IGAL-R3.125 / 6.25Learned routeroff72.4790.0051.67
C. IGAL-R3.125 / 6.25Learned routeron72.4794.4478.33

Complementary gains from adaptive allocation and gate scaling. The results demonstrate the effectiveness of combining multiple components in IGAL-R. Starting from the static IGAL-GLA baseline (A), gate scaling improves 16K–32K accuracy from 21.67% to 38.33%. Introducing length-dependent allocation (B) further raises accuracy to 70.00% with gate scaling, demonstrating the benefit of adapting the LA budget to context length. Finally, incorporating learned routing (C) achieves 78.33%, yielding an overall improvement of 56.66 percentage points over the original baseline. These results highlight the complementary benefits of the proposed components rather than attributing the entire gain to learned selection alone.

Additional benefit of learned routing. Importantly, learned routing consistently improves long-context retrieval over the deterministic length rule under matched LA budgets and gate-scaling settings. At 16K–32K, IGAL-R achieves 78.33% vs. 70.00% with gate scaling and 51.67% vs. 41.67% without it, corresponding to gains of 8.33 and 10.00 percentage points, respectively. This demonstrates that learned group selection provides additional benefits beyond length-dependent budgeting. Moreover, the router learns to refine the importance-based allocation: it replaces three of the eight additional groups selected by the static ranking, while retaining a largely consistent set of important groups across prompts.

Overall, these results support IGAL-R as a unified framework that combines importance-guided selection, length-adaptive budgeting, and learned routing. The deterministic length rule provides a strong baseline, while learned routing further improves performance on the longest contexts without increasing the LA budget.

Thank you for the additional experiments and clarifications. The runtime measurements and matched-budget routing comparison address my main concerns and help me better understand IGAL’s strengths and limitations. I will raise my rating to 3.

We sincerely thank you for the constructive feedback and raising the score. We will add the addition runtime measurements and the matched-budget routing comparison into the revised paper.

Reviewer_JApkOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper proposes Importance-Guided Attention Linearization (IGAL), a framework for converting pretrained Transformers into architectures that combine linear attention (LA) and sliding-window attention (SWA). The main idea is to reuse pretrained head-importance scores from HeadKV to guide attention operator allocation. High-importance KV groups are assigned to LA for long-range information modeling, while the remaining groups use SWA for local interactions.

Critical strengths
  1. The proposed attention allocation strategy achieves substantial improvements over the evaluated linearization baselines on synthetic long-context retrieval tasks. These results suggest that selective allocation of recurrent attention may be useful for preserving long-range information.
  2. IGAL reduces the persistent decoding cache while maintaining competitive general-language performance. This demonstrates a potentially useful memory-efficiency advantage, especially for long-context inference where KV-cache storage becomes a significant constraint.
Critical weaknesses
  1. The motivation and justification for using head importance to determine attention operator allocation remain insufficient. Section 3.1 explains that the method reuses HeadKV scores and aggregates them within GQA groups, while Table 4 compares several importance metrics against random allocation. However, these experiments do not establish why scores originally designed for KV-cache budget allocation should reliably indicate whether a head is better suited to LA rather than SWA. The paper lacks a direct analysis connecting the importance scores to the relative effectiveness of the two attention operators.
  2. The evaluation is limited to two 8B-scale models. Without experiments on smaller and larger models, it is difficult to assess whether the observed quality improvements generalize across model scales, particularly given potential differences in quality degradation and numerical stability.
  3. Table 3 reports memory consumption only at batch size 1, without measured prefill latency, decoding throughput, or performance under larger batch sizes.
  4. Several closely related approaches discussed in the Introduction and Related Work, particularly recent hybrid-attention and importance-aware methods, are absent from the quality and efficiency comparisons.
Questions for authors

Please see the weaknesses part.

7 comments

We thank the reviewer for the detailed feedback. We have clarified the motivation for importance-guided operator assignment and conducted additional analyses and experiments addressing each concern. We report the new results in this reply first; the revised PDF, with all changes marked in red, will be uploaded before the end of the discussion period.

W1. Why should importance scores designed for KV-cache budgeting decide between LA and SWA?

We agree that the original submission did not sufficiently justify this connection. Our rationale is to preserve long-range access where it matters most while retaining local softmax attention elsewhere.

Motivation. SWA preserves local softmax attention but restricts direct access to tokens outside its window, whereas LA maintains a compressed representation of the entire preceding context. For heads that primarily model local interactions, SWA better preserves the pretrained attention behavior. For heads that rely on distant, answer-relevant information, LA offers a way to retain long-range access, albeit approximately.

This distinction is consistent with prior analyses of the different local and global modeling properties of softmax and linear attention [1]. Moreover, [2] observes that LA branches may be underutilized when both operators are available to every head, motivating explicit operator allocation rather than uniform hybridization.

HeadKV (R) provides a proxy for such dependencies by measuring attention to answer-relevant distant tokens. While HeadKV uses these scores to allocate KV-cache capacity, IGAL uses them to allocate attention operators: high-importance groups receive LA to preserve approximate long-range access, while the remaining groups use SWA to retain local softmax attention.

New controlled evidence. To examine this hypothesis, we compare four selections of 16 KV groups on Llama-3.1-8B: top-ranked, middle-ranked, bottom-ranked, and random. We first examine their dependence on long-range attention in the pretrained model, then evaluate the corresponding LA/SWA conversions under identical training budgets.

LA groups (16 of 256)Attention on needle (original model)ΔNLL if set → SWA-64 (original model)NIAH 2K4K8K16K32K2K to 8K16K to 32K
Top 16 (ranks 1 to 16)0.245+3.0698.6798.6797.3388.0078.6798.2283.33
Random 16 (seed 42)0.062+0.7286.6786.6776.6756.67083.3328.33
Middle 16 (ranks 113 to 128)0.036+0.00176.6770.0070.0060.006.6772.2233.33
Bottom 16 (zero score)0.004+0.00630.006.6703.33012.221.67

From the results we claim:

  1. Higher-ranked groups are more sensitive to losing direct long-range attention. Replacing only the selected groups with SWA-64 in the teacher (all other groups remain full-attention) increases answer-token NLL by 3.06 for the top 16, compared with 0.72 for random and less than 0.01 for middle/bottom. This demostrates the sensitibity to removing their direct long-range attention in the original model..
  2. Assigning LA to these groups better preserves retrieval performance. After conversion, selecting the top-ranked groups achieves 83.33% accuracy at 16K–32K, substantially outperforming random (28.33%), middle-ranked (33.33%), and bottom-ranked (1.67%) selections. All configurations use the same conversion and fine-tuning procedure.

Together, these results support using HeadKV importance to identify groups for which preserving long-range access is particularly valuable.

[1] Dongchen Han et al. Bridging the Divide: Reconsidering Softmax and Linear Attention. NeurIPS 2024.

[2] Martin Benfeghoul et al. Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods. arXiv:2510.05901, 2025.

W2. Evaluation limited to two 8B models

Per your suggestion, we now have evaluated Llama-3.2-1B and Qwen3-14B using the same overall conversion recipe, extending the study to 1B–14B across the Llama and Qwen families.

Table 1: LM-EVAL

Comparison of different linearization methods on LM-EVAL with Llama-3.2-1B, Llama-3.1-8B, Qwen3-8B and Qwen3-14B (fine-tuned on the Alpaca-clean dataset).

ModelMethodPiQA (acc ↑)ARC-e (acc ↑)ARC-c (acc_norm ↑)Hella. (acc_norm ↑)Wino. (acc ↑)Avg. ↑
Llama-3.2-1BOriginal74.268.838.160.760.160.38
LAWCAT71.964.835.153.151.955.36
Liger74.067.437.757.360.059.30
IGAL (Ours)73.867.637.558.059.159.19
Qwen3-14BOriginal80.084.360.278.873.275.30
LAWCAT80.084.260.274.162.972.26
Liger80.285.062.577.072.175.36
IGAL (Ours)79.985.459.877.375.475.54

Table 2: S-NIAH

Accuracy (%) of different linearization methods on S-NIAH across context lengths with Llama-3.2-1B, Llama-3.1-8B, Qwen3-8B and Qwen3-14B (fine-tuned on NIAH dataset).

ModelMethod1K2K4K8K16K32KAvg. ↑
Llama-3.2-1BOriginal100.096.796.7100.0100.0100.098.9
LAWCAT100.0100.096.786.723.30.067.8
Liger0.70.00.00.00.00.00.1
IGAL (Ours)93.383.383.370.063.30.065.6
Qwen3-14BOriginal100.0100.096.7100.0100.0100.099.4
LAWCAT100.0100.096.753.320.06.762.8
Liger0.00.00.00.00.00.00.0
IGAL (Ours)100.0100.0100.086.746.73.372.8

Short-context quality. At 14B, IGAL achieves 75.54 LM-Eval versus 75.30 for the teacher and 72.26 for LAWCAT. At 1B, IGAL with 6.25% LA reaches 59.19, within 1.19 points of the teacher and close to Liger (59.30).

Long-context retrieval. At 14B, IGAL outperforms LAWCAT at 8K (86.7 vs. 53.3) and 16K (46.7 vs. 20.0), but both are weak at 32K. At 1B, IGAL achieves similar performance as LAWCAT while substantially outperforming Liger.

Overall, these results demonstrate that IGAL generalizes beyond the original 8B models, achieving the best overall trade-off between short-context quality and long-context retrieval across model scales and families.

W3. Efficiency metrics beyond batch-1 memory

We have extended our experiments to measure prefill latency, maximum decoding concurrency, and aggregate decode throughput across context lengths from 1K to 32K.

Setup. All measurements use one NVIDIA RTX A6000 (48 GB), bf16 weights, and greedy decoding, with FlashAttention-2 for full attention. Prefill latency is the median of five warmed-up runs (three at ≥16K), while decode throughput is measured over 32 steps after warm-up.

Prefill latency (batch 1, seconds).

Method1K4K8K16K32K
Full attn. (FA2)0.170.651.353.047.66
SWA-640.18 (1.07×)0.66 (1.01×)1.28 (0.95×)2.54 (0.84×)5.11 (0.67×)
LoLCATs0.42 (2.49×)1.57 (2.41×)3.17 (2.35×)6.39 (2.10×)12.84 (1.68×)
LAWCAT0.19 (1.13×)0.71 (1.08×)1.39 (1.02×)2.79 (0.92×)5.68 (0.74×)
Liger0.22 (1.28×)0.79 (1.22×)1.57 (1.16×)3.13 (1.03×)6.34 (0.83×)
IGAL0.19 (1.12×)0.68 (1.04×)1.31 (0.97×)2.60 (0.86×)5.22 (0.68×)
IGAL-R0.21 (1.25×)0.72 (1.11×)1.40 (1.03×)2.77 (0.91×)5.51 (0.72×)

IGAL is slower than FlashAttention-2 at 1K (0.19 vs. 0.17 s), but reaches parity near 8K, and takes 0.68× its time at 32K (5.22 vs. 7.66 s). It is only about 2% to 5% slower than all-SWA.

Decoding concurrency and throughput at 32K. We measure the maximum number of concurrent sequences before OOM, including model weights, persistent states/caches, and decode activations. Throughput is evaluated at 90% of each method's own maximum batch.

MethodMax batchServing batchTokens/s
Full attention (FA2)76100
SWA-642,8312,5472,721
LoLCATs327294808
LAWCAT4744262,354
Liger3312971,580
IGAL2,0571,8512,593
IGAL-R2,0551,8492,543

These results demonstrate that IGAL's bounded decoding state (14.0 MiB/sequence) translates into substantial serving benefits:

  • Serving capacity: At 32K, IGAL supports 294× as many concurrent sequences as full attention (2,057 vs. 7), and 4.3–6.3× as many as LAWCAT, LoLCATs, and Liger. Unlike full attention, its maximum decoding concurrency remains constant across the tested context lengths.
  • Aggregate throughput: At their respective serving capacities, IGAL achieves 25.9× the throughput of full attention at 32K (2,593 vs. 100 tokens/s), reaching approximately 95% of all-SWA's throughput and exceeding LAWCAT by approximately 10%.
  • Small-batch limitation: At batch 1 and 1K, IGAL currently requires 49.1 ms/token, compared with 31.9 ms/token for FA2, despite similar GPU-busy times (25.5 vs. 24.3 ms). This suggests that implementation overhead, including additional kernel launches, contributes to the latency gap.

Overall, IGAL offers substantial improvements in memory-constrained, high-concurrency serving at long context lengths, while maintaining competitive prefill efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.