- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes IGAL, a framework for converting pretrained Transformers into models with linear attention complexity and bounded decoding-state memory. It reuses HeadKV importance scores to assign a small subset of GQA groups to recurrent linear attention and the remaining groups to sliding-window softmax attention. Adaptation uses a single stage of task-based fine-tuning with LoRA and trainable Q/K convolutions. IGAL-R additionally adjusts the recurrent budget by prompt length, learns prompt-conditioned group selection, and optionally rescales forget gates during inference. Evaluation covers Llama-3.1-8B-Instruct and Qwen3-8B, short-context language benchmarks, and single-needle retrieval up to 32K. The paper reports strong retrieval performance, substantial improvements over random allocation at a matched recurrent budget, and a constant 14 MiB decoding cache for the reported Llama configuration. A mixed-data IGAL-R checkpoint also substantially improves retrieval while retaining competitive short-context performance.
Critical strengths
-
Strong evidence for importance-guided recurrent allocation. Table 4 holds the GLA operator and the 6.25% recurrent-group budget fixed while changing the allocation criterion. HeadKV-R improves 16–32K retrieval accuracy from 28.33% under random allocation to 83.33%. This directly supports the paper’s central design choice and demonstrates a consequential benefit beyond simply adding recurrent memory.
-
A compelling bounded-memory retrieval result. The retrieval-specific configuration reaches 89.33% accuracy at 32K on Llama after the reported 1K-length fine-tuning. Table 3 also measures a constant 14 MiB persistent cache, compared with 4096 MiB for full attention at 32K, with an explicit accounting protocol. Together, these results demonstrate a useful retrieval–memory trade-off.
Critical weaknesses
-
The incremental value of learned prompt-conditioned selection remains unclear. IGAL-R changes the length-dependent recurrent budget, learns which groups receive that budget, and applies inference-time gate scaling. Table 5 shows that IGAL-R already improves long retrieval from 21.67% to 51.67% without scaling, and that scaling further increases it to 78.33%. However, there is no deterministic length-conditioned allocation baseline using the same budgets and gate scaling, nor a static baseline with matched scaling. The benefit of the complete configuration is supported, but the necessity of learning prompt-dependent group selection is not established.
-
Long-context validation is restricted to single-needle retrieval. The long-context results cover S-NIAH variants, while the language benchmarks primarily test short-context capabilities. These evaluations do not establish whether the conversion preserves long-context abilities requiring multiple pieces of evidence or broader document understanding. This limits the demonstrated significance of the framework beyond retrieval.
Questions for authors
-
Controlled evidence: Under identical mixed-data training, GLA operators, and inference-time gate scaling, how does IGAL-R compare with a deterministic policy that uses the same short/long recurrent budgets but always selects groups by the fixed importance ranking? What happens when the static IGAL-GLA baseline also receives gate scaling?
-
Training clarification: What are the prompt-length distributions used to train IGAL-R, and what fraction of examples satisfy (T_p \geq 1024)? Appendix A.3 states that only long-regime response losses train the selector. Please clarify how this protocol relates to the reported 1K-length retrieval fine-tuning and whether the learned routes depart from their initialization.
-
Calibration clarification: Were importance scores computed separately for each backbone or transferred from existing scores? What calibration data and context lengths were used, and how were these examples separated from the training and evaluation data? This matters for reproducibility and the interpretation of the short-to-long extrapolation result.
5 comments
Q2. Training regimes, prompt lengths, and learned route changes
Clarifying training vs. inference regimes. We apologize for the ambiguity in the submission. During training, regimes are assigned by data source rather than prompt length: all S-NIAH examples use the long regime (16 LA groups, router active), while Alpaca examples use the short regime (8 groups), regardless of . The threshold applies only at inference. Thus, "only long-regime response losses train the selector" means that the router receives training signals from the response tokens of all S-NIAH examples, but none from Alpaca.
This source-based assignment is intentional: applying the inference-time length rule during training would exclude 80.7% of S-NIAH prompts from router training, leaving only 2,014 long prompts, almost all from S-NIAH-1/2. Source-based assignment exposes the router to the full retrieval training distribution.
Prompt-length distribution. The mixed-data setup interleaves Alpaca-clean and S-NIAH examples one to one. Here, is the position of the first supervised token.
| Source | n | T_p mean ± std | T_p range | Answer length | T_p ≥ 1024 |
|---|---|---|---|---|---|
| Alpaca-clean | 51,760 | 48.0 ± 20.1 | 29–633 | 139.3 ± 127.0 | 0.00% |
| S-NIAH-3 (UUID; "1K" retrieval) | 8,456 | 770.3 ± 50.9 | 661–1,212 | 17–31 | 0.17% (14) |
| S-NIAH-1 (numeric) | 1,000 | 1,226.5 ± 1.7 | 1,223–1,233 | 4 | 100% |
| S-NIAH-2 (numeric) | 1,000 | 1,188.1 ± 75.7 | 1,027–1,474 | 4 | 100% |
| All S-NIAH | 10,456 | 853.9 ± 179.6 | 661–1,474 | 20.0 ± 8.0 | 19.26% (2,014) |
Only 19.26% of S-NIAH examples satisfy (approximately 9.63% of all training examples under one-to-one interleaving). Nevertheless, all 10,456 S-NIAH examples contribute to router training under the source-based protocol.
Relation to 1K retrieval fine-tuning. The "1K" retrieval dataset refers to S-NIAH-3 (mean ), used in the task-specific training setup. The mixed-data setup used by IGAL-R and static IGAL-GLA "Mix" additionally includes 2,000 S-NIAH-1/2 examples with approximately 1.2K-token prompts. Thus, the router is trained on retrieval prompts spanning 661–1,474 tokens, but activated at inference only when . Evaluation at 2K–32K therefore tests extrapolation beyond the training prompt lengths.
Do learned routes depart from initialization? Yes. The router is initialized to reproduce the fixed HeadKV ranking, but its learned selections differ from this initialization on every examined prompt.
- Training prompts: Across 10,456 S-NIAH prompts, the router consistently retains 5 of the 8 initially selected switchable groups, replacing the other 3. We observe 8 distinct routes, with the three most common accounting for 95.7% of prompts (58.3%, 27.9%, and 9.6%). They share two substitutions, and , and differ in the third, where denotes layer and KV group.
- Evaluation prompts: Across various RULER prompts (1K–32K), the router never predicts its initial allocation. We observe two distinct routes, differing in one selected group.
- Router parameters: The prompt-conditioned score term reaches , compared with a maximum absolute bias of 0.999. The learned group ranking has a Spearman correlation of only 0.26 with the initial importance ranking.
Overall, the router learns to refine the initial importance-based allocation rather than simply reproducing the HeadKV ranking. Its selections also vary across prompts, although most groups remain shared.
Q3. Calibration of the importance scores
Computed per backbone. The HeadKV scores are computed separately for each backbone: we ran HeadKV's released head-detection script on the frozen Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct and Qwen3-8B/14B, rather than reusing scores released for other models.
Calibration data. Calibration uses HeadKV's own needle-in-a-haystack probes (150 retrieval probes, and 100 retrieval-reasoning probes for the R2 variant) with contexts of up to 10K tokens. It needs only forward passes and updates no parameters. These probes serve only to rank heads. They are not part of our training or evaluation data, their needles (natural-language facts) differ from the magic-number/UUID needles used in S-NIAH, and their text does not overlap with the RULER evaluation inputs.
Short-to-long extrapolation. Calibration contexts (up to 10K tokens) are longer than our training prompts (at most 1.5K) but shorter than the evaluation lengths (up to 32K). To check that the selection does not depend on the long probes, we recomputed the scores from probes of at most 2.5K tokens only: 15 of the 16 selected groups on Llama-3.1-8B and 33 of the 36 on Qwen3-8B remain the same.
W2. Long-context evaluation beyond single-needle retrieval
We thank the reviewer for highlighting this limitation. We agree that single-needle retrieval alone is insufficient to assess long-context capabilities requiring the integration of multiple pieces of evidence.
To address this concern, we have selected BABILong, a benchmark that evaluates long-context reasoning by requiring models to retrieve and combine information distributed across lengthy contexts. Unlike single-needle retrieval, BABILong includes tasks involving multi-hop reasoning and the integration of multiple relevant facts, providing a more comprehensive evaluation of long-context capabilities.
We are currently running BABILong experiments to assess whether IGAL preserves these capabilities beyond single-needle retrieval. The experiments are still in progress, and we will report the results once available.
W1 / Q1. Controlled evidence for learned routing
We thank the reviewer for suggesting these controlled comparisons. We have now trained a deterministic length-conditioned baseline (B) under the same mixed-data training setup and GLA operators as IGAL-R (C), with identical short-/long-prompt LA budgets (3.125% / 6.25%). Unlike IGAL-R, B selects groups solely according to the fixed HeadKV ranking, without learned routing. We also evaluate the static IGAL-GLA baseline (A) with the same inference-time gate scaling.
Learned routing under matched budgets and scaling (B vs. C). At 16K–32K, IGAL-R improves retrieval accuracy from 70.00% to 78.33% with gate scaling (+8.33 points), and from 41.67% to 51.67% without scaling (+10.00 points). Since B and C share the same training setup, LA budgets, and gate-scaling settings, these comparisons isolate the incremental benefit of learned group selection beyond length-dependent allocation. At 2K–8K, the deterministic baseline performs slightly better with scaling (95.56% vs. 94.44%), indicating that learned routing's advantage is concentrated at longer contexts.
Gate scaling for static IGAL-GLA (A). Applying the same scaling improves 16K–32K retrieval accuracy from 21.67% to 38.33%, confirming that scaling also benefits the static baseline. Nevertheless, this remains below both the deterministic length rule (70.00%) and IGAL-R (78.33%) under matched scaling.
Overall, these controlled comparisons show that learned group selection provides additional long-context retrieval gains beyond length-dependent budgeting and gate scaling, improving 16K–32K accuracy by 8.33 points over the deterministic baseline without increasing the LA budget.