- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LYNX, a lightweight method for linearizing pretrained Transformers. It combines sliding-window attention, timescale-separated recurrent states, and a Landmark cache, with a controller that decouples retention from write allocation. The theoretical analysis characterizes how retention affects spectral concentration and provides sufficient conditions for effective-rank improvement with timescale-separated states. Experiments evaluate general reasoning, knowledge, and long-context performance on 8B-parameter LLaMA-family models.
Critical strengths
The paper addresses the practically relevant problem of linearizing pretrained Transformers under a limited adaptation budget. Its effective-rank analysis provides a useful perspective on recurrent memory utilization and characterizes how retention affects spectral concentration under explicit assumptions. The framework combines exact attention with timescale-separated memory. Experiments cover general reasoning, knowledge, and long-context tasks, with reported improvements over several linearization baselines.
Critical weaknesses
- Effective-rank recovery primarily serves as a diagnostic perspective and a design objective. The method combines timescale-separated recurrent states, retention and write allocation controls, and a Landmark cache. Although the paper discusses related work, introduction does not adequately explain the distinct advantages of this design over existing multi-timescale memory and hybrid attention mechanisms. The theoretical analysis identifies conditions under which retention increases effective rank. However, the theoretical and empirical evidence does not sufficiently explain how this increase improves historical information retrieval and contributes to capability recovery.
- In Figure 7, T+LM8 and LYNX differ in both retention controls and Landmark cache capacity, making it difficult to determine whether the improvement comes from timescale separation or the larger cache. Moreover, these ablations are limited to MMLU, with no corresponding analysis on long-context retrieval tasks. The Liger and GDN baselines reported in Table 2 receive no additional training, as detailed in Appendix C, whereas LYNX undergoes answer-plus-EOS adaptation and retaining-head training. In the LongBench evaluation, GDN and LYNX also differ in training steps, training data, and exact KV budgets.
Questions for authors
see weakness
5 comments
W2
In Figure 7, T+LM8 and LYNX differ in both retention controls and Landmark cache capacity, making it difficult to determine whether the improvement comes from timescale separation or the larger cache. Moreover, these ablations are limited to MMLU, with no corresponding analysis on long-context retrieval tasks. The Liger and GDN baselines reported in Table 2 receive no additional training, as detailed in Appendix C, whereas LYNX undergoes answer-plus-EOS adaptation and retaining-head training. In the LongBench evaluation, GDN and LYNX also differ in training steps, training data, and exact KV budgets.
Author response
Ablations at fixed capacity. We apologize for the incomplete ablation in Figure 7. Changing both the retention policy and the number of Landmarks in the T+LM8 comparison left their contributions unresolved. We therefore repeat the comparison at a common cache capacity and extend it from MMLU to long-context retrieval and question answering. We fix the recurrent-state size, training data, learning-rate schedule, numerical implementation, and evaluation inputs. LYNX uses the Split configuration with separate Medium and Long half-life ranges.
We train each Llama-3.1-8B configuration for 1,000 steps, followed by 200 steps of Answer+EOS adaptation and 200 steps of retaining-head training. Each configuration has 80 exact KV slots (8 sink, 56 recent, 16 Landmark). The table reports percentages on the same 12,350 examples.
| Configuration | S-NIAH 0.5K | 1K | 2K | 4K | S-NIAH mean | Extended RULER (eight axes) | LongBench7 |
|---|---|---|---|---|---|---|---|
| Tied+LM16 | 87.93 | 64.53 | 43.93 | 45.80 | 60.55 | 11.82 | 27.01 |
| Free+LM16 | 88.33 | 39.60 | 34.33 | 33.47 | 48.93 | 10.86 | 26.93 |
| LYNX+LM16 | 92.27 | 67.00 | 52.27 | 51.40 | 65.73 | 14.41 | 27.72 |
The Free–LYNX comparison directly tests timescale separation because both models already control retention independently in each state. Separating the ranges raises mean S-NIAH accuracy from 48.93% to 65.73%, a gain of 16.80 percentage points. LYNX also gains 3.55 points on Extended RULER and 0.79 on LongBench7 over Free. Relative to Tied, the gains are 5.18, 2.59, and 0.70 points, respectively.
Training and budget differences across methods. We apologize for the unclear description of Table 2. The original comparison used each method’s training data and schedule. LYNX trained for 5,000 steps on Alpaca, and our Paper-GDN reproduction trained for 2,441 steps on DCLM-edu. Their additional supervision and exact KV budgets also differed. The original performance gap therefore reflects several changes at once.
Our adaptation builds on Lizard’s experiments with synthetic passkey retrieval and generalization to longer contexts [4]. In LYNX, Answer+EOS supervision trains the model to retrieve an answer from the context, generate it in full, and stop at its end. The retaining head addresses a different decision. It uses evidence labels and K/V features to learn which tokens to keep in the limited historical cache. During this stage, the backbone is frozen and only the retaining heads are updated.
To measure the gain from each adaptation stage, we evaluated successive checkpoints from the same LYNX+LM16 run on a prespecified subset of 2,550 examples. On the S-NIAH portion, accuracy rose from 14.42% after the initial 1,000 steps to 37.92% after Answer200 and 66.67% after Head200. The two stages added 23.50 and 28.75 percentage points, respectively. Free+LM16 also benefited from both stages on the same subset (15.17%, 23.42%, and 49.00%). The LM16 table above reports the corresponding final scores on the full S-NIAH evaluation.
For the new comparison, all three methods use Llama-3.1-8B Base, the checkpoint at step 1,000 on Alpaca-cleaned, 128 exact KV slots, and the same 12,350 evaluation examples. All three receive 200 steps of Answer+EOS adaptation. LYNX and Liger use the same cache allocation and each receives 200 additional steps of retaining-head training. GDN keeps the first 8 and most recent 120 tokens by position, so its adaptation ends after Answer+EOS. Scores are percentages.
| Method | S-NIAH | Extended RULER | LongBench7 |
|---|---|---|---|
| LYNX | 81.23 | 24.19 | 27.95 |
| Liger | 69.10 | 8.67 | 18.30 |
| GDN | 23.60 | 17.39 | 27.96 |
With the same adaptation stages and cache allocation, LYNX exceeds Liger by 12.13 percentage points on S-NIAH, 15.52 on Extended RULER, and 9.65 on LongBench7. Against GDN, the advantage is concentrated in retrieval. LYNX scores 81.23 versus 23.60 on S-NIAH and 24.19 versus 17.39 on Extended RULER. Their seven-task LongBench means are nearly identical (27.95 and 27.96).
LYNX and GDN differ in recurrent memory, historical KV selection, and retaining-head supervision. Their KV128 scores therefore compare the two complete systems under the same cache capacity. The Free–LYNX ablation addresses the narrower question of timescale separation by holding supervision and cache allocation fixed.
References
[1] Philipp Nazari and T. Konstantin Rusch. The Key to State Reduction in Linear Attention: A Rank-based Perspective. arXiv:2602.04852, 2026, §2.2.
[2] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML, 2023.
[3] Ao Sun et al. State Rank Dynamics in Linear Attention LLMs. arXiv:2602.02195, 2026.
[4] Chien Van Nguyen et al. Lizard: An Efficient Linearization Framework for Large Language Models. ACL, 2026, §5.2.
Response to Reviewer exhm
Thank you for asking how timescale separation affects retrieval and how much of the improvement in Table 2 can be attributed to the additional training. We welcome any follow-up questions and would be happy to clarify the comparisons further.
Questions for Authors
W1
Author response
Memory design. LYNX gives each query two ways to access history under fixed memory budgets. The Medium and Long states preserve the full query and key address dimensions and split only the value dimension. Two 128×64 states thus occupy the same space as one 128×128 state. A query can address both states in full, while their value features persist over different timescales. The controller determines how long each state retains information, how much to write in total, and how to allocate that write between the states.
Landmark Attention (Mohtashami and Jaggi, 2023) inserts dedicated landmark tokens to retrieve whole blocks, whereas LYNX retains the original keys and values of individual tokens selected after they leave the sliding window. Their keys and values enter the same causal softmax as those of the sink and recent tokens. The model adds this attention output to the combined recurrent readout. We will clarify in the introduction and related work how these two forms of access connect LYNX to multiscale memory, hybrid attention, and historical KV selection.
Effective rank and task performance. Our theory gives sufficient conditions under which retention redistributes energy across the singular-value spectrum and increases effective rank. Prior work connects the stable rank of a linear attention state to retrieval error under query noise [1]. RankMe found that effective rank can help predict downstream performance for self-supervised representations [2]. State Rank Dynamics also identified low-rank heads that matter for inference and high-rank heads with redundant contributions [3]. These results motivate examining the spectrum alongside a direct measure of task performance.
We compare three retention policies with the same recurrent-state budget and 16 Landmarks. Tied shares a retention control between the two states. Free gives the states independent controls within a common half-life range. LYNX keeps the controls independent and assigns the Medium and Long states nonoverlapping ranges. The Free–LYNX comparison therefore tests the contribution of separating the ranges while holding independent control and cache capacity fixed.
The table pairs a local measurement of normalized effective rank on MMLU inputs with mean MMLU accuracy for each configuration.
With 16 Landmarks and independent retention controls in both models, Free and LYNX differ in whether their two states share the same half-life range. Moving from Free to LYNX increased the locally measured normalized effective rank on MMLU inputs from 0.20494 to 0.23787 and mean MMLU accuracy from 57.69% to 58.29%. We evaluate historical retrieval directly in the fixed-capacity ablation reported under W2.