acceptodds
Under review as a conference paper at ICLR 2027

LYNX: Linearizing Transformers with Timescale-Separated Memory

19 views

Abstract

Linearizing pretrained Transformers reduces long-context inference costs without pretraining from scratch, but preserving their capabilities under limited adaptation remains challenging. Our spectral analysis reveals low effective rank in the linearized baseline's recurrent state, with energy concentrated in a few directions. This limited use of the state's dimensions motivates a memory design with distinct retention timescales. We propose LYNX, a lightweight linearization method that combines sliding-window attention for exact recent access with timescale-separated recurrent states for historical information. The states jointly support association updates and fact retention, while a controller decouples retention from write allocation. A Landmark cache complements recurrent memory with exact access to selected historical tokens. Our analysis characterizes how retention shapes spectral concentration and gives sufficient conditions for the separated states to attain higher effective rank than a shared-retention baseline. Experiments on 8B-parameter LLaMA-family models evaluate general and long-context capabilities. LYNX achieves average accuracy across six reasoning and knowledge tasks, demonstrating general capability recovery. On long-context tasks, LYNX reaches a seven-task LongBench average of 26.07 and outperforms the compared linearization baselines across all eight Extended RULER task groups at 4K.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Linear-Complexity Sequence Modeling, Linear Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_exhmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LYNX, a lightweight method for linearizing pretrained Transformers. It combines sliding-window attention, timescale-separated recurrent states, and a Landmark cache, with a controller that decouples retention from write allocation. The theoretical analysis characterizes how retention affects spectral concentration and provides sufficient conditions for effective-rank improvement with timescale-separated states. Experiments evaluate general reasoning, knowledge, and long-context performance on 8B-parameter LLaMA-family models.

Critical strengths

The paper addresses the practically relevant problem of linearizing pretrained Transformers under a limited adaptation budget. Its effective-rank analysis provides a useful perspective on recurrent memory utilization and characterizes how retention affects spectral concentration under explicit assumptions. The framework combines exact attention with timescale-separated memory. Experiments cover general reasoning, knowledge, and long-context tasks, with reported improvements over several linearization baselines.

Critical weaknesses
  1. Effective-rank recovery primarily serves as a diagnostic perspective and a design objective. The method combines timescale-separated recurrent states, retention and write allocation controls, and a Landmark cache. Although the paper discusses related work, introduction does not adequately explain the distinct advantages of this design over existing multi-timescale memory and hybrid attention mechanisms. The theoretical analysis identifies conditions under which retention increases effective rank. However, the theoretical and empirical evidence does not sufficiently explain how this increase improves historical information retrieval and contributes to capability recovery.
  2. In Figure 7, T+LM8 and LYNX differ in both retention controls and Landmark cache capacity, making it difficult to determine whether the improvement comes from timescale separation or the larger cache. Moreover, these ablations are limited to MMLU, with no corresponding analysis on long-context retrieval tasks. The Liger and GDN baselines reported in Table 2 receive no additional training, as detailed in Appendix C, whereas LYNX undergoes answer-plus-EOS adaptation and retaining-head training. In the LongBench evaluation, GDN and LYNX also differ in training steps, training data, and exact KV budgets.
Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer exhm

Thank you for asking how timescale separation affects retrieval and how much of the improvement in Table 2 can be attributed to the additional training. We welcome any follow-up questions and would be happy to clarify the comparisons further.

Questions for Authors

see weakness

W1

Effective-rank recovery primarily serves as a diagnostic perspective and a design objective. The method combines timescale-separated recurrent states, retention and write allocation controls, and a Landmark cache. Although the paper discusses related work, introduction does not adequately explain the distinct advantages of this design over existing multi-timescale memory and hybrid attention mechanisms. The theoretical analysis identifies conditions under which retention increases effective rank. However, the theoretical and empirical evidence does not sufficiently explain how this increase improves historical information retrieval and contributes to capability recovery.

Author response

Memory design. LYNX gives each query two ways to access history under fixed memory budgets. The Medium and Long states preserve the full query and key address dimensions and split only the value dimension. Two 128×64 states thus occupy the same space as one 128×128 state. A query can address both states in full, while their value features persist over different timescales. The controller determines how long each state retains information, how much to write in total, and how to allocate that write between the states.

Landmark Attention (Mohtashami and Jaggi, 2023) inserts dedicated landmark tokens to retrieve whole blocks, whereas LYNX retains the original keys and values of individual tokens selected after they leave the sliding window. Their keys and values enter the same causal softmax as those of the sink and recent tokens. The model adds this attention output to the combined recurrent readout. We will clarify in the introduction and related work how these two forms of access connect LYNX to multiscale memory, hybrid attention, and historical KV selection.

Effective rank and task performance. Our theory gives sufficient conditions under which retention redistributes energy across the singular-value spectrum and increases effective rank. Prior work connects the stable rank of a linear attention state to retrieval error under query noise [1]. RankMe found that effective rank can help predict downstream performance for self-supervised representations [2]. State Rank Dynamics also identified low-rank heads that matter for inference and high-rank heads with redundant contributions [3]. These results motivate examining the spectrum alongside a direct measure of task performance.

We compare three retention policies with the same recurrent-state budget and 16 Landmarks. Tied shares a retention control between the two states. Free gives the states independent controls within a common half-life range. LYNX keeps the controls independent and assigns the Medium and Long states nonoverlapping ranges. The Free–LYNX comparison therefore tests the contribution of separating the ranges while holding independent control and cache capacity fixed.

The table pairs a local measurement of normalized effective rank on MMLU inputs with mean MMLU accuracy for each configuration.

ConfigurationNormalized effective rankMMLU accuracy (%, mean ± sample SD)
Tied0.1552657.23 ± 1.35
Free0.2049457.69 ± 0.31
LYNX0.2378758.29 ± 0.54

With 16 Landmarks and independent retention controls in both models, Free and LYNX differ in whether their two states share the same half-life range. Moving from Free to LYNX increased the locally measured normalized effective rank on MMLU inputs from 0.20494 to 0.23787 and mean MMLU accuracy from 57.69% to 58.29%. We evaluate historical retrieval directly in the fixed-capacity ablation reported under W2.

W2

In Figure 7, T+LM8 and LYNX differ in both retention controls and Landmark cache capacity, making it difficult to determine whether the improvement comes from timescale separation or the larger cache. Moreover, these ablations are limited to MMLU, with no corresponding analysis on long-context retrieval tasks. The Liger and GDN baselines reported in Table 2 receive no additional training, as detailed in Appendix C, whereas LYNX undergoes answer-plus-EOS adaptation and retaining-head training. In the LongBench evaluation, GDN and LYNX also differ in training steps, training data, and exact KV budgets.

Author response

Ablations at fixed capacity. We apologize for the incomplete ablation in Figure 7. Changing both the retention policy and the number of Landmarks in the T+LM8 comparison left their contributions unresolved. We therefore repeat the comparison at a common cache capacity and extend it from MMLU to long-context retrieval and question answering. We fix the recurrent-state size, training data, learning-rate schedule, numerical implementation, and evaluation inputs. LYNX uses the Split configuration with separate Medium and Long half-life ranges.

We train each Llama-3.1-8B configuration for 1,000 steps, followed by 200 steps of Answer+EOS adaptation and 200 steps of retaining-head training. Each configuration has 80 exact KV slots (8 sink, 56 recent, 16 Landmark). The table reports percentages on the same 12,350 examples.

ConfigurationS-NIAH 0.5K1K2K4KS-NIAH meanExtended RULER (eight axes)LongBench7
Tied+LM1687.9364.5343.9345.8060.5511.8227.01
Free+LM1688.3339.6034.3333.4748.9310.8626.93
LYNX+LM1692.2767.0052.2751.4065.7314.4127.72

The Free–LYNX comparison directly tests timescale separation because both models already control retention independently in each state. Separating the ranges raises mean S-NIAH accuracy from 48.93% to 65.73%, a gain of 16.80 percentage points. LYNX also gains 3.55 points on Extended RULER and 0.79 on LongBench7 over Free. Relative to Tied, the gains are 5.18, 2.59, and 0.70 points, respectively.

Training and budget differences across methods. We apologize for the unclear description of Table 2. The original comparison used each method’s training data and schedule. LYNX trained for 5,000 steps on Alpaca, and our Paper-GDN reproduction trained for 2,441 steps on DCLM-edu. Their additional supervision and exact KV budgets also differed. The original performance gap therefore reflects several changes at once.

Our adaptation builds on Lizard’s experiments with synthetic passkey retrieval and generalization to longer contexts [4]. In LYNX, Answer+EOS supervision trains the model to retrieve an answer from the context, generate it in full, and stop at its end. The retaining head addresses a different decision. It uses evidence labels and K/V features to learn which tokens to keep in the limited historical cache. During this stage, the backbone is frozen and only the retaining heads are updated.

To measure the gain from each adaptation stage, we evaluated successive checkpoints from the same LYNX+LM16 run on a prespecified subset of 2,550 examples. On the S-NIAH portion, accuracy rose from 14.42% after the initial 1,000 steps to 37.92% after Answer200 and 66.67% after Head200. The two stages added 23.50 and 28.75 percentage points, respectively. Free+LM16 also benefited from both stages on the same subset (15.17%, 23.42%, and 49.00%). The LM16 table above reports the corresponding final scores on the full S-NIAH evaluation.

For the new comparison, all three methods use Llama-3.1-8B Base, the checkpoint at step 1,000 on Alpaca-cleaned, 128 exact KV slots, and the same 12,350 evaluation examples. All three receive 200 steps of Answer+EOS adaptation. LYNX and Liger use the same cache allocation and each receives 200 additional steps of retaining-head training. GDN keeps the first 8 and most recent 120 tokens by position, so its adaptation ends after Answer+EOS. Scores are percentages.

MethodS-NIAHExtended RULERLongBench7
LYNX81.2324.1927.95
Liger69.108.6718.30
GDN23.6017.3927.96

With the same adaptation stages and cache allocation, LYNX exceeds Liger by 12.13 percentage points on S-NIAH, 15.52 on Extended RULER, and 9.65 on LongBench7. Against GDN, the advantage is concentrated in retrieval. LYNX scores 81.23 versus 23.60 on S-NIAH and 24.19 versus 17.39 on Extended RULER. Their seven-task LongBench means are nearly identical (27.95 and 27.96).

LYNX and GDN differ in recurrent memory, historical KV selection, and retaining-head supervision. Their KV128 scores therefore compare the two complete systems under the same cache capacity. The Free–LYNX ablation addresses the narrower question of timescale separation by holding supervision and cache allocation fixed.

References

[1] Philipp Nazari and T. Konstantin Rusch. The Key to State Reduction in Linear Attention: A Rank-based Perspective. arXiv:2602.04852, 2026, §2.2.

[2] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML, 2023.

[3] Ao Sun et al. State Rank Dynamics in Linear Attention LLMs. arXiv:2602.02195, 2026.

[4] Chien Van Nguyen et al. Lizard: An Efficient Linearization Framework for Large Language Models. ACL, 2026, §5.2.

Reviewer_3yWaOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows the issue that linearised transformers’ states have a lower effective rank, and proposes an idea that uses timescale-separated linear memory combined with sink and landmark tokens, based on sliding window attention. The authors provide reasons for the low effective rank, an idea to prevent it using this structure, and experimental study that shows solid performance compared with existing linear/linearised transformers.

Critical strengths
  • The authors show that a potential cause of degradation in linearised transformers could be the lowering of the effective rank of the memory state, by comparing the effective ranks of Gated DeltaNet and a linearised transformer.
  • The authors show a way to increase the effective rank of the state and propose timescale-separated memory. Experiments that incorporate the proposed memory combined with sink and landmark tokens based on sliding window attention showed strong retrieval performance compared with linearised transformers and linearly scaling models.
Critical weaknesses
  • Evidence of low rank is insufficient.
    • The effective rank comparison is not an apples-to-apples comparison. I understand that it is difficult to make such a comparison, so instead the authors should show the effective ranks for multiple methods to show that it is a common issue. (Figure 1 compares quite different types of models, which doesn't back the authors' claim enough.)
  • The experimental results don't support the effective rank improvement enough, which is the authors' main claim, nor the performance improvement due to timescale-separated memory.
    • The effective rank improvement solely due to timescale-separated memory is not shown in Figure 7. To support the authors’ claim, the authors should compare models with and without timescale-separated memory, fixing all other hyperparameters, and examine the effective rank improvement and performance improvement. However, the performance improvement given only by timescale-separated memory is not shown (in Figure 7, both the number of landmark tokens and the use of timescale-separated states change), and the difference between T+LM8 and T+LM16+Timescale-separated memory is as small as the variation across random seeds.
Questions for authors
  • What is the positioning of sink tokens and landmark tokens compared with previous works? Which prior work inspired you to use sink/landmark tokens?
  • What was the effective rank for each configuration in the ablation in Figure 7?
  • Looking at Figure 7, the main factor in the improvement seems to come from landmark tokens. How does the landmark token idea relate to LoLA and STILL?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3yWa

Thank you for pointing out that Figure 7 varies both the retention policy and the Landmark budget. Your questions helped us clarify what the fixed-capacity ablations show and how LYNX’s approach to historical KV selection relates to prior work. We welcome further questions and suggestions as we revise the paper.

W1

Evidence of low rank is insufficient.

The effective rank comparison is not an apples-to-apples comparison. I understand that it is difficult to make such a comparison, so instead the authors should show the effective ranks for multiple methods to show that it is a common issue. (Figure 1 compares quite different types of models, which doesn't back the authors' claim enough.)

Author response

On the same retrieval inputs, Liger and Lizard exhibit low effective rank at all four context lengths, while LYNX has a higher effective rank at each length. We evaluated all three models on the same 24 preselected inputs spanning three S-NIAH tasks and four context lengths (0.5K, 1K, 2K, and 4K). We computed the energy-entropy effective rank of each state, averaged within each input, and then gave each input equal weight. The table reports mean absolute effective rank.

Method0.5K1K2K4K
Liger1.2561.1921.1931.187
Lizard1.1551.1471.1531.142
LYNX6.15217.89421.01322.489

At 4K, the leading singular direction accounts for 96.70% of state energy in Liger and 96.39% in Lizard on average, compared with 25.44% in LYNX. This explains the difference in effective rank: nearly all the energy lies along one direction in Liger and Lizard, whereas it is distributed across more directions in LYNX.

From 0.5K to 4K, LYNX's mean effective rank rises from 6.152 to 22.489, with progressively smaller increases. Each length has its own set of preselected inputs, so this trend describes average state spectra across retrieval examples of different lengths.

Liger applies softmax to its queries and keys, and Lizard uses softmax([2Wx, −2Wx]) features. Both form their states through gated decay and the accumulation of outer products. A common component in the nonnegative key features, together with the value directions and decay weights, may concentrate energy in a few directions. LYNX writes to memory using delta-rule corrections, the full key address space, and separate controls for its memory banks.

W2

The experimental results don't support the effective rank improvement enough, which is the authors' main claim, nor the performance improvement due to timescale-separated memory.

The effective rank improvement solely due to timescale-separated memory is not shown in Figure 7. To support the authors’ claim, the authors should compare models with and without timescale-separated memory, fixing all other hyperparameters, and examine the effective rank improvement and performance improvement. However, the performance improvement given only by timescale-separated memory is not shown (in Figure 7, both the number of landmark tokens and the use of timescale-separated states change), and the difference between T+LM8 and T+LM16+Timescale-separated memory is as small as the variation across random seeds.

Author response

Figure 7 varies both the retention policy and the number of Landmarks. In the additional ablation, we fix the historical cache at 16 Landmarks (LM16) and hold the number of recurrent-state elements constant. The table reports local normalized effective rank and mean MMLU accuracy for three retention policies.

Tied uses a shared retention control for both states. Free allows the two states to adjust their retention times independently within a common range. LYNX also controls retention independently but assigns nonoverlapping ranges to the Medium and Long states.

ConfigurationLandmark capacityNormalized effective rankMMLU
Tied160.1552657.23 ± 1.35
Free160.2049457.69 ± 0.31
LYNX160.2378758.29 ± 0.54

At this common capacity, LYNX has a normalized effective rank of 0.23787 in the local state measurement and a mean MMLU accuracy of 58.29%. Relative to Tied, these are gains of 53.21% in local effective rank and 1.06% in accuracy. The table includes the standard deviation of accuracy across runs so that the mean differences can be assessed alongside run-to-run variation.

To examine whether the retrieval gains persist across runs, we conducted three paired long-context experiments. Within each pair, Free and LYNX used the same initialization and training input order, with recurrent-state size and LM16 capacity held fixed. LYNX outscored Free on both S-NIAH and Extended RULER in all three pairs, with mean gains of 11.62 and 3.13 %, respectively.

AuthorsOpenReview ↗

Response to Reviewer 3yWa

Q1

What is the positioning of sink tokens and landmark tokens compared with previous works? Which prior work inspired you to use sink/landmark tokens?

Author response

Our use of sink tokens follows StreamingLLM's observation that initial token positions can attract substantial attention weight. Retaining their keys and values helps stabilize windowed attention. LYNX keeps the first 8 positions and the 56 most recent positions available for exact attention. [1]

Landmarks allow exact access to historical tokens after they leave the recent window. LoLA, STILL, and LYNX all combine such access with recurrent memory. They differ in how they select historical keys and values to retain.

MethodSelection of historical keys and values and use of memory
LoLAUses self-recall error to select keys and values that recurrent memory has difficulty recalling. It combines recent KV pairs, sparse historical KV pairs, and the recurrent state as a training-free extension of linear attention. [2]
STILLComputes a Self-Saliency Score within the sliding window to select important tokens for sparse softmax attention. Its Norm-Preserved Feature Map preserves the magnitudes of pretrained features. [3]
LYNXUses Medium and Long recurrent states with separate controls for retention time and write allocation. The base Landmark score combines the Long bank's write strength and retention priority with key novelty, linking historical KV selection to the controls that govern recurrent memory.

In LYNX, tokens become Landmark candidates when they leave the recent window. Selected historical KV pairs participate in the same causal softmax as sink and recent tokens, and information from these tokens also enters recurrent memory. This gives LYNX both compressed memory at different timescales and exact access to historical tokens selected using its memory controls.

Q2

What was the effective rank for each configuration in the ablation in Figure 7?

Author response

To address the simultaneous changes in retention policy and Landmark count in Figure 7, the W2 table reports a supplementary ablation at fixed capacity. With 16 Landmarks and the same recurrent-state size, the normalized effective ranks in the same local state measurement are 0.1553 for Tied, 0.2049 for Free, and 0.2379 for LYNX.

The comparison between Free and LYNX addresses the choice of timescale ranges directly. Both let the two states adjust their retention times independently. Free uses a common range, while LYNX assigns separate medium- and long-term ranges. At the same memory budget, LYNX has a higher local effective rank, and mean MMLU accuracy rises from 57.69% to 58.29%.

Q3

Looking at Figure 7, the main factor in the improvement seems to come from landmark tokens. How does the landmark token idea relate to LoLA and STILL?

Author response

As described in Q1, LYNX, LoLA, and STILL all retain selected historical KV pairs for exact access. To test whether recurrent-memory mechanisms still affect retrieval at the same Landmark capacity, we compared the full configuration with a joint ablation of these mechanisms using the same checkpoint.

The table reports S-NIAH accuracy (%) at four context lengths with the same weights and LM16 capacity.

Configuration5121K2K4KOverall mean
LYNX+LM1693.4066.6751.5348.7365.08
LYNX+LM16, joint recurrent-memory ablation81.7338.4029.8734.3346.08

At the same Landmark capacity, mean retrieval accuracy is 65.08% for the full configuration and 46.08% for the joint ablation, a difference of 19.00 percentage points. The full configuration scores higher at all four lengths, with gains of 21.66 and 14.40 percentage points at 2K and 4K, respectively. With weights and cache capacity held fixed, these results support a joint contribution of the ablated recurrent-memory mechanisms to S-NIAH retrieval.

References

[1] Efficient Streaming Language Models with Attention Sinks

[2] LoLA: Low-Rank Linear Attention With Sparse Caching

[3] STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

Thank you once more for the time and effort you have devoted to evaluating our work. We have addressed your comments through detailed clarifications and additional experimental results. We would appreciate hearing about any remaining concerns and welcome the opportunity to respond during the remaining discussion period.

Reviewer_gBrUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LYNX converts pretrained LLaMA Transformers into bounded-memory hybrids through lightweight adaptation. Sliding-window attention preserves exact recent access, while tokens leaving the window update two recurrent states with distinct medium- and long-term retention ranges. A controller predicts state-specific retention and dynamically allocates write strength between the states, while a Landmark cache preserves selected historical KV pairs for exact retrieval. Motivated by the low effective rank observed in linearized recurrent states, the paper analyzes how retention affects spectral concentration. Experiments on general reasoning, LongBench, S-NIAH, and Extended RULER show competitive capability recovery and improvements over several linearization baselines on long-context tasks.

Critical strengths
  1. Clear motivation and presentation. The paper is well-written and logically organized. Its spectral analysis provides an interesting perspective on recurrent-state utilization after Transformer linearization and motivates the timescale-separated memory design.

  2. Promising empirical results. With a relatively small adaptation budget, LYNX achieves competitive general-task performance and substantial improvements over several linearization baselines on long-context benchmarks. The broad evaluation provides evidence of its potential for capability recovery.

Critical weaknesses
  1. The core design choices and their contributions are insufficiently justified. The controller is not fully specified, and the Landmark priority and key-only novelty heuristics lack direct theoretical or empirical validation. More importantly, the ablations do not isolate timescale separation from Landmark capacity or the additional long-context supervision, making it difficult to attribute the reported gains to the paper's main architectural contribution. See Questions (1)–(4) and (6).

  2. The accuracy–efficiency advantage is not established under controlled comparisons. Existing baselines use heterogeneous adaptation procedures and cache budgets, particularly for long-context evaluation. The paper also provides no measured training or inference efficiency, despite introducing dual-state updates, a controller, and Landmark selection. See Questions (5) and (7).

Questions for authors

1. Controller parameterization. Section 5.1 defines the retention and write controls but does not fully specify how they are computed from the input representations. Could the authors provide the controller's projection functions and activations, and explain how its outputs are mapped to α\alpha, β\beta, and the routing probabilities pp? In particular, how are the input-dependent half-lives τM\tau^M and τL\tau^L constrained to their predefined ranges?

2. Justification for the Landmark retention priority. Equation (10) estimates a token's long-term importance using βtL(αtL)D\beta_t^L(\alpha_t^L)^D. However, αtL\alpha_t^L controls the decay of the previous state, whereas the persistence of newly written information depends on subsequent retention gates and updates. What justifies using the current coefficient as a proxy for future retention? Is this choice supported theoretically or empirically?

3. Limitations of key-based novelty. Landmark selection measures redundancy solely through key similarity, without considering the associated values. Similar keys may encode distinct and important values, so this criterion may penalize tokens that still merit exact storage. What motivates this design, particularly given that the supervised retaining head retains the same key-based novelty factor?

4. Insufficient ablation of timescale-separated memory. The main novelty of LYNX is its timescale-separated recurrent states, since prior work such as LoLA already combines local attention, recurrent memory, and historical KV caching. However, Figure 7 does not isolate this contribution: the comparison between Tied+LM8 and LYNX also increases the Landmark capacity from 8 to 16. Could the authors compare tied, shared-range, and timescale-separated states using identical state sizes, Landmark budgets, and training settings, particularly on retrieval benchmarks?

5. Fairness of baseline comparisons. Comparisons with existing linearization methods are not fully controlled for adaptation tokens, exact-KV budgets, and training procedures. For example, LYNX uses 80 exact-KV tokens in the general-task evaluation, compared with 64 for Liger and LoLCATs, while the long-context comparisons use different checkpoints and adaptation settings. Could the authors provide budget-matched comparisons, particularly against LoLA and Lizard, which also combine recurrent memory with historical KV caching?

6. Contribution of additional long-context supervision. The long-context results rely on two additional stages: Answer+EOS adaptation and a supervised retaining head trained with gold evidence labels. These stages may improve retrieval independently of the memory architecture, and they are not applied uniformly across comparisons. Could the authors ablate both stages and apply identical supervision to the baselines to clarify how much of the gain comes specifically from timescale-separated memory?

7. Missing efficiency evaluation. A primary motivation for Transformer linearization is to reduce training and inference costs, yet the paper provides no measured efficiency comparison. Could the authors report adaptation throughput, GPU hours, and peak training memory, together with prefill latency, decoding throughput, and memory consumption across context lengths? Comparisons with the source Transformer and existing linearization baselines are needed to assess the accuracy–efficiency trade-off, including the overhead of the controller, dual-state updates, and Landmark selection.

6 comments
AuthorsOpenReview ↗

Response to Reviewer gBrU

Thank you for your careful review and constructive feedback. Your questions have helped us clarify the method and strengthen the experimental evidence. We address each point below.

Q1 — Controller parameterization

Reviewer comment (original)

  1. Controller parameterization. Section 5.1 defines the retention and write controls but does not fully specify how they are computed from the input representations. Could the authors provide the controller's projection functions and activations, and explain how its outputs are mapped to (α\alpha), (β\beta), and the routing probabilities (pp)? In particular, how are the input-dependent half-lives (τM\tau^M) and (τL\tau^L) constrained to their predefined ranges?

Response

The controller reads the hidden representation xtx_t of the token as it leaves the local window. A shared low-rank projection gives

zt=SiLU⁡(Wdxt),zt∈R16.z_t=\operatorname{SiLU}(W_dx_t),\qquad z_t\in\mathbb R^{16}.

Three output heads read this shared feature. One sets the retention timescale, one sets the total write strength, and one divides that write between Medium and Long. We parameterize each half-life as

n_t^M\in[1/16,1],\quad n_t^L\in[2,16].$$ Here, $n_t^b$ varies continuously with the input. For each attention head $h$, memory bank $b$, and decay group $g$, the sigmoid sets the interpolation weight. Interpolating between the bank’s bounds on a log scale keeps the half-life within its assigned range. \$ u_{t,h,b,g}=\sigma\\left(c_b+[W_\alpha z_t]_{h,b,g}\right), \$ $$n_{t,h,b,g}=n_{\min,b} \left(\frac{n_{\max,b}}{n_{\min,b}}\right)^{u_{t,h,b,g}}, \qquad \alpha_{t,h,b,g}=2^{-1/(1024n_{t,h,b,g})}.$$ We initialize $n$ to 1/2 for Medium and 8 for Long. The half-life counts memory updates. The total write strength and routing probabilities for each head are $$\beta_{t,h}=\sigma\left(c_\beta+[W_\beta z_t]_h\right),$$ $$p^L_{t,h}=\sigma([W_pz_t]_h)$$ $$p^M_{t,h}=1-p^L_{t,h}.$$ The routing probabilities sum to one, so each bank receives its share of the total write, $s\beta_{t,h}p^b_{t,h}$. We initialize all three output projections to zero. The fixed biases set the initial half-lives and β, and the two banks initially receive equal shares. ## Q2 — Using current decay to estimate future retention ### Reviewer comment (original) > 2. Justification for the Landmark retention priority. Equation (10) estimates a token's long-term importance using ($\beta_t^L(\alpha_t^L)^D$). However, ($\alpha_t^L$) controls the decay of the previous state, whereas the persistence of newly written information depends on subsequent retention gates and updates. What justifies using the current coefficient as a proxy for future retention? Is this choice supported theoretically or empirically? ### Response You are right that subsequent gates determine how much of a new write survives. At selection time, the controller has access to the current gate. It sets the decay of the existing state and indicates the timescale the controller assigns to the current input. Medium and Long use distinct ranges for this timescale. We use the current Long-bank timescale as a proxy for future decay. A strong Long-bank write receives higher Landmark priority when the current decay is slow. $$P_t=\beta_t^L(\bar\alpha_t^L)^D,\qquad D=2\times1024.$$ We omit the head index above and average the score over heads. If subsequent gates stay near the current value and later writes cause little interference, the decay factor over the next $D$ updates is approximately $(\alpha_t^L)^D$. This local stationarity assumption explains why the current gate can inform a selection made before those updates occur. We tested the proxy directly by replacing input-dependent α with a static α in the priority score. With the same LM16 weights, KV budget, and 96 inputs of length 4K, retrieval hit rate was 48.958% with the original score and 45.833% with static α. The 3.125 percentage point gain shows that the current gate carries information useful for ranking candidates alongside Long-bank write strength.

Q3 — Key-only novelty

Reviewer comment (original)

  1. Limitations of key-based novelty. Landmark selection measures redundancy solely through key similarity, without considering the associated values. Similar keys may encode distinct and important values, so this criterion may penalize tokens that still merit exact storage. What motivates this design, particularly given that the supervised retaining head retains the same key-based novelty factor?

Response

Similar keys can carry different values. The supervised retaining head therefore reads both K and V when scoring a candidate’s content. The novelty factor answers a narrower question about how well the existing cache covers this key address.

We tested whether the novelty factor itself benefits from value information on 24 examples with repeated addresses and fact updates. Key-only novelty, disabling novelty, and a fixed 128-dimensional K/V sketch yielded 11, 11, and 7 correct answers, respectively.

Key-only novelty matched the score with novelty disabled and outperformed the K/V sketch. On these examples, keys alone were sufficient for the diversity term. We therefore keep value information in the retaining head, where it informs content importance.

Q4 — Contribution of timescale separation

Reviewer comment (original)

  1. Insufficient ablation of timescale-separated memory. The main novelty of LYNX is its timescale-separated recurrent states, since prior work such as LoLA already combines local attention, recurrent memory, and historical KV caching. However, Figure 7 does not isolate this contribution: the comparison between Tied+LM8 and LYNX also increases the Landmark capacity from 8 to 16. Could the authors compare tied, shared-range, and timescale-separated states using identical state sizes, Landmark budgets, and training settings, particularly on retrieval benchmarks?

Response

We held recurrent state size, Landmark capacity, and training fixed across Tied, Free, and LYNX. Tied shares retention control across its two states. Free and LYNX both control the states independently. Free uses a common half-life range, while LYNX assigns separate ranges to Medium and Long. Comparing Free with LYNX isolates the choice of range.

All three configurations use Llama-3.1-8B with 1,000 training steps, followed by Answer200 and Head200. The exact KV capacity is 80 slots (8 sink, 56 recent, and 16 Landmark). Scores are percentages on the same 12,350 evaluation examples.

ConfigurationS-NIAH 0.5K1K2K4KS-NIAH meanExtended RULER (eight axes)LongBench7
Tied+LM1687.9364.5343.9345.8060.5511.8227.01
Free+LM1688.3339.6034.3333.4748.9310.8626.93
LYNX+LM1692.2767.0052.2751.4065.7314.4127.72

At the same state budget and LM16 capacity, LYNX improves over Free by 16.80, 3.55, and 0.79 percentage points on S-NIAH, Extended RULER, and LongBench7, respectively.

The following table pairs local normalized effective rank with mean MMLU accuracy for the same LM16 configurations.

ConfigurationNormalized effective rankMMLU accuracy (%, mean ± sample SD)
Tied0.1552657.23 ± 1.35
Free0.2049457.69 ± 0.31
LYNX0.2378758.29 ± 0.54

Local normalized effective rank and mean MMLU accuracy both increase from Tied to Free to LYNX. LYNX exceeds Tied by 1.06 percentage points and Free by 0.60 percentage points in mean accuracy. The rank measurements describe how the state spectrum changes. The fixed-capacity task comparisons measure the performance gain associated with separating the two timescales.

Q5 — Budget-matched baseline comparisons

Reviewer comment (original)

  1. Fairness of baseline comparisons. Comparisons with existing linearization methods are not fully controlled for adaptation tokens, exact-KV budgets, and training procedures. For example, LYNX uses 80 exact-KV tokens in the general-task evaluation, compared with 64 for Liger and LoLCATs, while the long-context comparisons use different checkpoints and adaptation settings. Could the authors provide budget-matched comparisons, particularly against LoLA and Lizard, which also combine recurrent memory with historical KV caching?

Response

We matched the number of exact KV slots for each task group. On the general tasks, LYNX+LM0 retains 8 sink and 56 recent tokens after 1,000 adaptation steps. We use the published LoLCATs and Liger-GLA scores cited in the paper and our local KV64 evaluations for LoLA and Lizard. Adaptation tokens and training procedures follow each method’s own protocol, so the comparison controls cache capacity only.

ConfigurationExact KV (general / long-context)PIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUSix-task meanLongBench7
LoLCATs64 / —80.9081.7054.9079.7074.1052.8070.68—
Liger-GLA64 / —80.3081.1052.5076.3072.0043.4067.60—
LoLA64 / 12880.3682.2854.9577.3374.2738.9568.0223.74
Lizard64 / 12875.1467.0541.9865.0760.0628.1056.2313.02
LYNX64 / 12880.0382.1155.7279.9576.4855.6371.6526.07

Scores are percentages, and the six-task mean is the equally weighted average over the general tasks. The cache column gives the exact KV capacities for general and long-context tasks separately.

With 64 exact KV slots, LYNX achieves the highest six-task mean at 71.65%. Its margins over LoLCATs, Liger, LoLA, and Lizard are 0.97, 4.05, 3.63, and 15.42 percentage points, respectively. LYNX leads on ARC-Challenge, HellaSwag, WinoGrande, and MMLU.

On LongBench7, LYNX achieves 26.07%, ahead of LoLA by 2.33 percentage points and Lizard by 13.05 percentage points at the same exact KV capacity. LYNX thus has the highest reported mean in each task group, using LM0 for general tasks and LM64 for LongBench7.

Q6 — Contribution of additional long-context supervision

Reviewer comment (original)

  1. Contribution of additional long-context supervision. The long-context results rely on two additional stages: Answer+EOS adaptation and a supervised retaining head trained with gold evidence labels. These stages may improve retrieval independently of the memory architecture, and they are not applied uniformly across comparisons. Could the authors ablate both stages and apply identical supervision to the baselines to clarify how much of the gain comes specifically from timescale-separated memory?

Response

We applied the same two supervision stages to Free and LYNX. Starting from base adaptation, Answer200 trains answer generation and stopping behavior. Head200 then uses gold evidence labels to train the retaining head that selects historical KV. Comparing successive stages within each model measures the gain from each stage. Comparing Free and LYNX at the same stage holds supervision fixed.

We use the seed 3 training runs from the long-context comparison in Q4, starting from 1,000 steps of base adaptation. Here we score each stage on the same prespecified subset of 2,550 examples and report single-run percentages. Q4 reports the full evaluation set.

ConfigurationTraining stageS-NIAHExtended RULER (eight axes)LongBench7
Free+LM16Base adaptation15.179.7525.18
Free+LM16Answer20023.4210.3027.90
Free+LM16Answer200+Head20049.0010.3827.87
LYNX+LM16Base adaptation14.429.3526.54
LYNX+LM16Answer20037.929.9626.57
LYNX+LM16Answer200+Head20066.6714.6128.68

On S-NIAH, Answer200 raises accuracy by 8.25 percentage points for Free and 23.50 for LYNX. Head200 adds a further 25.58 and 28.75 percentage points, respectively. Both models benefit from the extra supervision. After both stages, LYNX scores higher than Free on S-NIAH, Extended RULER, and LongBench7 under the same supervision.

Reviewer_BA2aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed LYNX, which is a post-training process for Transformer linearization framework and then applied it to 8B-parameter LLaMA-3 and LLaMA-3.1 models. Motivated by an empirical observation that a linearized Gated DeltaNet baseline exhibits a lower "effective rank" (reffr_{\text{eff}}) of its terminal recurrent state STS_T than a GDN model pretrained from scratch.

Critical strengths
  • Comparing the spectral entropy effective rank (reffr_{\text{eff}}) and state-row cosine similarity of a post-training linearized Transformer against a natively pretrained Gated DeltaNet (Figure 1, Figure 3) is a clean diagnostic perspective on why lightweight LoRA linearizations struggle to utilize recurrent state capacity.
  • Achieving an average accuracy of 71.79±0.33%71.79 \pm 0.33\% across 6 standard benchmarks (PIQA, ARC-e/c, HellaSwag, WinoGrande, MMLU) after only 1,000 steps (~8.2M tokens) of Alpaca LoRA adaptation closely approaches the source LLaMA-3-8B Transformer (72.02%) and improves upon Liger-GLA (67.60%) and LoLCATs (70.68%).
Critical weaknesses
  • The paper does not properly position itself within the literature. There are many missing citations. How the "timescale-separated memory" is different from nested learning, which also targets multi-timescale memory? There is a rich literature on multi-timescale states for SSMs, which almost none of them are cited/discussed. Similarly, it is not clear that how the method is different from "Landmark Attention: Random-Access Infinite Context Length for Transformers" (Mohtashami & Jaggi, 2023). There is a lack of discussion with many other papers with similar sprit as well: For example, RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale (2025), H2O: Heavy-Hitter Oracle (Zhang et al., 2023), Quest (Tang et al., 2024), Titans (Behrouz et al., 2024), and Infini-attention (Munkhdalai et al., 2024).
  • To my understanding, the central narrative of the paper is that timescale-separated recurrent memory (SM,SLS^M, S^L) recovers linearized Transformer capabilities by increasing effective rank. However, Section 6.5 (Figure 7 and Table 7) reveals the opposite! It would be useful if the authors could clarify this.
  • The actual jump in performance comes almost entirely from adding the exact softmax KV Landmark cache (T+LM8 jumps to 56.63±1.09%56.63 \pm 1.09\%, and LYNX with 16 Landmarks reaches 58.29±0.54%58.29 \pm 0.54\%).
  • Even worse, in Figure 7, the comparison between T+LM8 (tied retention + 8 landmarks) and LYNX (timescale-separated retention + 16 landmarks) is completely confounded because the authors doubled the exact Landmark KV cache from 8 to 16 at the same time as enabling timescale separation! There is no ablation comparing T+LM16 vs. LYNX (LM16), nor is there any ablation of Medium/Long states vs. Landmark caching on LongBench, S-NIAH, or Extended RULER!
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer BA2a

Response to Reviewer BA2a

Thank you for these thoughtful questions. They helped us clarify how LYNX relates to prior work and what our ablations show. We welcome further questions and suggestions. If any part of our explanation or evidence still leaves you unconvinced, we would be grateful to hear why and to work through it with you as we revise the paper.

W1

The paper does not properly position itself within the literature. There are many missing citations. How the "timescale-separated memory" is different from nested learning, which also targets multi-timescale memory? There is a rich literature on multi-timescale states for SSMs, which almost none of them are cited/discussed. Similarly, it is not clear that how the method is different from "Landmark Attention: Random-Access Infinite Context Length for Transformers" (Mohtashami & Jaggi, 2023). There is a lack of discussion with many other papers with similar sprit as well: For example, RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale (2025), H2O: Heavy-Hitter Oracle (Zhang et al., 2023), Quest (Tang et al., 2024), Titans (Behrouz et al., 2024), and Infini-attention (Munkhdalai et al., 2024).

Author response

Thank you for pointing out these gaps. We will clarify how LYNX couples two-timescale recurrent memory with exact access to selected historical tokens.

Landmark Attention. Mohtashami and Jaggi insert landmark tokens to retrieve whole blocks. A LYNX landmark is an individual token whose original keys and values (K/V) remain available after it leaves the window. Landmarks, attention sinks, and recent tokens share one exact softmax. [1]

RADLADS. Both methods reuse pretrained Transformer weights. RADLADS initializes RAD-RWKV6/7 from the original query, key, value, and output (QKVO) weights. [2]

ComparisonRADLADSLYNX
AdaptationLayer alignment, logits KL distillation, then long-context cross-entropyCross-entropy on training labels without a teacher pass
Trainable parametersModel-level distillation updates all layers. Freezing the MLP and embeddings or using low-rank training reduced performance.Freezes the backbone. Only the controller, QKVO LoRA parameters, and separately trained retaining head require optimizer state.
Historical accessCompressed recurrent stateExact K/V for recent tokens and a bounded set of historical tokens
Recurrent stateRAD-RWKV6/7 dynamicsMedium and Long share full key addresses, split value dimensions, and have separate retention ranges within a fixed state budget.

H2O and Quest. H2O keeps recent tokens and cumulative-attention heavy hitters. Quest selects KV pages for each query using page statistics. In LYNX, each token leaving the window updates recurrent memory. Its priority for exact caching is the mean, over heads, of its effective Long-bank write weighted by predicted retention. [3][4]

Pi=1H∑h=1Hβt,hL(αˉt,hL)DP_i = \frac{1}{H}\sum_{h=1}^{H}\beta_{t,h}^{L}(\bar{\alpha}_{t,h}^{L})^{D}

For the token ii leaving the window at step tt, HH is the number of heads and DD is the prediction horizon. The Long-bank terms are the effective write and the geometric mean retention across groups, respectively. Multiplying this priority by address novelty gives the selection score. We will ablate the α/β priority, learned K/V retaining head, and novelty separately at fixed weights, KV capacity, and evaluation examples. W3 reports the completed joint S-NIAH ablation.

Titans. Titans updates a neural memory network at test time. LYNX uses the delta rule to update explicit Medium and Long matrix states with distinct half-life ranges. [5]

Infini-attention. Infini-attention gates cross-segment compressed memory and within-segment attention per head. LYNX concatenates the Medium and Long readouts along the value dimension and adds them to exact attention, which can still access selected out-of-window tokens. [6]

Nested Learning and multi-timescale SSMs. Nested Learning organizes learning around different update frequencies and context flows. Multi-timescale SSMs model dependencies of different durations through state dynamics. LYNX brings timescale separation to Transformer linearization under a fixed state budget and uses long-term memory signals to select historical K/V. [7]

References

[1] Landmark Attention: Random-Access Infinite Context Length for Transformers.

[2] RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale.

[3] H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[4] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference.

[5] Titans: Learning to Memorize at Test Time.

[6] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.

[7] Nested Learning: The Illusion of Deep Learning Architectures.

AuthorsOpenReview ↗

Response to Reviewer BA2a

W2

To my understanding, the central narrative of the paper is that timescale-separated recurrent memory (SM,SLS_M, S_L) recovers linearized Transformer capabilities by increasing effective rank. However, Section 6.5 (Figure 7 and Table 7) reveals the opposite! It would be useful if the authors could clarify this.

Author response

We apologize for the confusion caused by our incomplete explanation. You are right to question the interpretation of Figure 7 and Table 7. We linked changes in the state spectrum to task accuracy without explaining what the theory establishes or which factors varied together in the original comparison.

All four configurations use the same local sliding-window attention and the same total number of recurrent state elements. They differ in the number of states and in how retention is controlled.

ConfigurationRecurrent memory
S (Single)One recurrent state.
T (Tied)Two states with shared retention control.
F (Free)Two states with independent retention controls operating within the same timescale range.
LYNXTwo states constrained to non-overlapping Medium and Long timescale ranges.

The original comparison between Tied+LM8 and full LYNX varied both retention and landmark capacity. To compare the memory designs at a fixed historical KV capacity, we evaluated all four with 16 landmarks (LM16). We report mean MMLU accuracy and the sample standard deviation.

ConfigurationMMLU accuracy (%, mean ± sample SD)
S+LM1656.23 ± 0.45
Tied+LM1657.23 ± 1.35
Free+LM1657.69 ± 0.31
LYNX+LM1658.29 ± 0.54

With LM16, LYNX exceeds S, Tied, and Free in mean MMLU accuracy by 2.06, 1.06, and 0.60 percentage points, respectively. Free and LYNX both give the two states independent retention controls. They differ in whether the half-life ranges are separated, so this comparison directly tests the effect of explicit timescale separation at LM16.

Effective rank measures how state energy is distributed across the singular-value spectrum. Our theory gives a sufficient condition for higher effective rank. When weaker spectral components gain proportionally more energy and the spectral ordering condition in the paper holds, the timescale-separated state can have higher effective rank. The result alone does not predict MMLU accuracy. We therefore use the state spectrum to examine changes in the memory representation and the fixed-capacity task ablation to measure performance.

Effective rank and MMLU accuracy

The table pairs normalized effective rank with MMLU accuracy for the same LM16 configurations.

ConfigurationNormalized effective rankMMLU accuracy (%, mean ± sample SD)
Tied0.1552657.23 ± 1.35
Free0.2049457.69 ± 0.31
LYNX0.2378758.29 ± 0.54

From Tied to Free to LYNX, state energy is more evenly distributed, and mean MMLU accuracy increases from 57.23% to 57.69% to 58.29%. This ordering is consistent with the interpretation that timescale separation helps the model use the available state dimensions. These three configurations alone cannot establish how much of the accuracy gain is caused by the increase in effective rank. We will revise our interpretation of Figure 7 and Table 7 accordingly.

AuthorsOpenReview ↗

Response to Reviewer BA2a

W3

The actual jump in performance comes almost entirely from adding the exact softmax KV Landmark cache (T+LM8 jumps to 56.63 ± 1.09, and LYNX with 16 Landmarks reaches 58.29 ± 0.54).

Author response

As you point out, Tied+LM8 and LYNX+LM16 differ in both exact cache capacity and recurrent memory design. Their accuracy difference alone cannot separate the two contributions. We therefore compare three memory designs at the same capacity, then test the contribution of recurrent memory mechanisms to retrieval using a single checkpoint.

On MMLU, all three configurations retain 16 landmarks. LYNX, which separates the retention timescales, has the highest mean accuracy, 1.06 percentage points above Tied with its shared retention control.

ConfigurationMMLU accuracy (%, mean ± sample SD)
Tied+LM1657.23 ± 1.35
Free+LM1657.69 ± 0.31
LYNX+LM1658.29 ± 0.54

On S-NIAH, we use a single Llama-3.1-8B LM16 checkpoint (5,000 steps followed by Answer200 and Head200, distinct from the 1,000-step checkpoint in W4) to compare the full configuration with a joint ablation of recurrent memory mechanisms. The ablated configuration retains the K/V retaining head and the novelty term for token selection. Both configurations use an exact KV budget of 80 tokens and are evaluated on the same 6,000 examples. The table reports accuracy (%) by subtask and context length. Each context-length mean weights the three subtasks equally, and the rightmost column weights the four lengths equally.

ConfigurationS-NIAH subtask0.5K1K2K4KMean across lengths
LYNX+LM16S1100.0099.8099.8099.6099.80
LYNX+LM16S2100.0097.2053.6044.4073.80
LYNX+LM16S380.203.001.202.2021.65
LYNX+LM16Mean93.4066.6751.5348.7365.08
Recurrent memory ablationS179.6059.2049.4058.6061.70
Recurrent memory ablationS2100.0054.8039.4042.2059.10
Recurrent memory ablationS365.601.200.802.2017.45
Recurrent memory ablationMean81.7338.4029.8734.3346.08

The full configuration has higher mean retrieval accuracy at all four lengths. The joint ablation reduces overall accuracy from 65.08% to 46.08%, a drop of 19.00 percentage points. With the weights and exact KV capacity held fixed, this difference measures the joint contribution of recurrent readout and memory-aware token selection to retrieval.

W4

Even worse, in Figure 7, the comparison between T+LM8 (tied retention + 8 landmarks) and LYNX (timescale-separated retention + 16 landmarks) is completely confounded because the authors doubled the exact Landmark KV cache from 8 to 16 at the same time as enabling timescale separation! There is no ablation comparing T+LM16 vs. LYNX (LM16), nor is there any ablation of Medium/Long states vs. Landmark caching on LongBench, S-NIAH, or Extended RULER!

Author response

Following your suggestion, we compare Tied, Free, and LYNX on Llama-3.1-8B with the same 80 exact KV slots (8 sink, 56 recent, and 16 landmarks) and the same 12,350 evaluation examples. Tied shares retention control across its two states. Free and LYNX use independent controls, with a common half-life range in Free and separate Medium and Long ranges in LYNX.

The table reports scores (%) for seed 3. Each S-NIAH length column averages the three subtasks, and the S-NIAH mean weights the four lengths equally.

ConfigurationS-NIAH 0.5K1K2K4KS-NIAH meanExtended RULER (eight axes)LongBench7
Tied+LM1687.9364.5343.9345.8060.5511.8227.01
Free+LM1688.3339.6034.3333.4748.9310.8626.93
LYNX+LM1692.2767.0052.2751.4065.7314.4127.72

With 16 landmarks, LYNX exceeds Tied by 5.18, 2.59, and 0.70 percentage points on S-NIAH, Extended RULER, and LongBench7, respectively. LYNX also outperforms Free at all four S-NIAH lengths. Since both use independent retention controls, this comparison supports the contribution of separate half-life ranges to out-of-window retrieval.

Thank you again for your thoughtful feedback and the time you have dedicated to reviewing our work. We have provided detailed responses and additional results to address your concerns. We welcome any further questions and would be happy to clarify remaining points during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.