acceptodds
Desk rejected at ICLR 2027

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm. However, the O(n²) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requires no pretraining from scratch. LISA integrates two lightweight components in parallel within the original model: (1) a Linear Attention module that provides long-range memory with O(n) time complexity; (2) a Lightning Indexer that selects the top-M important tokens from the full context to feed into a Sparse Self-Attention. The two branches are fused via a gating mechanism, reducing inference complexity from O(n²) to O(nM) (M ≪ n) for generating n tokens. We design a two-stage training pipeline. Stage 1 initializes the model by integrating the linear attention to capture long-range dependencies, complemented by a sliding-window attention mechanism that is optimized via knowledge distillation to approximate the full self-attention distribution of a frozen teacher model. In Stage 2, we further introduce the Indexer to replace the static sliding-window mechanism, enabling dynamic token selection from broader contexts. The Indexer is trained using a novel per-head KL divergence loss, which aligns its selection behavior with the attention patterns of the teacher model. Experiments on DeepSeek-distilled-Qwen models demonstrate that LISA reduces per-token decoding FLOPs by 50.7% at a 16K-token context, while improving the average accuracy by 4.2% over the full-attention baseline across nine benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Linear Attention

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_uiRyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LISA, a hybrid architecture combining linear attention, dynamically indexed sparse attention, learned gating, and inference-time segment correction. For DeepSeek-R1-Distill-Qwen 1.5B and 7B models, the paper reports higher average math accuracy than the full-attention baseline and 23.9% lower eager-mode latency at 1.5B.

Critical strengths

The paper studies an important efficiency challenge in long-CoT reasoning. The combination of linear and dynamically selected sparse attention is potentially useful.

Critical weaknesses

The paper suffers from the following major concerns:

(1) LISA trains 111M (1.5B) or 420M (7B) new parameters on 100K OpenR1-Math samples, taking around 480 H100 hours. But it does not apply to the full-attention baseline. Even though the authors do acknowledge but “+4.2 points over the full-attention baseline” is stated as the result. To have a fair comparison, for the full-attention base model, we should train a LoRA adapter trained on the provided dataset. Without that, the comparison is not meaningful.

(2) One top of point (1), I find this paper very incremental, as without using segment correction Zhang et al. (2026), the own component performs even worse than the baseline. Moreover, the improvement to Zhang et al. (2026) is very marginal, i.e., ~+1.3 points at 1.5B and +2.1 at 7B, from single runs.

Questions for authors

Please check the Critical Weaknesses section.

3 comments

W1: Fairness of the full-attention baseline

Thank you for raising this concern. We first clarify an error in the manuscript that affects the interpretation of this comparison. The full-attention baselines in Table 1 are DeepSeek-R1-Distill-Qwen-1.5B/7B models adapted using standard LoRA fine-tuning, rather than the original checkpoints without additional training. Although Section 4.2 states that the baselines are trained with LoRA, the Table 1 caption and Section 4.4 incorrectly describe them as an “unmodified original model/checkpoint.” We will correct these descriptions and consistently identify the models and their training status.

Converting an existing dense model through post-training into an architecture that supports dynamic retrieval, recurrent memory, and this TTT-style state correction is one of the central contributions of our work.

For the 1.5B experiments, the compared methods use approximately 110M trainable parameters and the same training dataset of approximately 95K examples, controlling for trainable capacity and training data. To distinguish the original checkpoint from our fine-tuned full-attention baseline, we provide their results below and will include this distinction in the revised manuscript. All accuracies are percentages; Avg. is the unweighted mean across the five tasks.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Original checkpoint76.169.052.523.313.346.8
Full-attention LoRA baseline82.178.862.520.016.752.0

W2: Contribution beyond the adopted correction and comparison with Zhang et al.

LISA differs from Zhang et al. in two main respects. First, LISA uses fixed-length segments for state correction and does not require clustering or annotating thinking patterns in training trajectories, or introducing special tokens to delimit reasoning steps. This reduces data preparation and avoids modifying the token format of ordinary CoT text. Second, the Indexer dynamically selects relevant tokens from the full history for explicit retrieval through sparse attention, complementing the information retained in the recurrent state.

Table 2 reports repeated-sampling results for the full-attention baseline and LISA. We have extended this evaluation to Zhang et al. The results below report mean accuracy ± standard deviation over 16 evaluation runs.

MethodAMC23AIME24AIME25Avg.
Full-attention LoRA baseline62.5 ± 2.2419.6 ± 1.6516.1 ± 1.8332.73
LISA70.0 ± 2.7425.8 ± 1.9625.9 ± 2.2940.57
Zhang et al.66.9 ± 2.7325.0 ± 2.3822.5 ± 2.5038.13

The explanation regarding the LoRA-trained baseline addresses my first concern. However, I remain concerned about the limited novelty beyond Zhang et al., as the improvements are relatively modest and the individual contributions are not convincingly isolated. The repeated evaluation runs also do not establish robustness across training seeds. Therefore, my overall concern remains, and I maintain my rating.

Thank you for your follow-up.

Regarding the contribution beyond Zhang et al., we would like to highlight the following differences:

  • Simpler data preparation. LISA does not require clustering the training trajectories or introducing special tokens to mark reasoning steps. It operates directly on ordinary CoT sequences, reducing data-preparation complexity and preprocessing overhead.
  • Simpler adaptation of the linear-attention branch. Zhang et al. jointly optimize an autoregressive objective and an output-distribution distillation objective. LISA trains the LA branch using standard next-token cross-entropy during the cold-start stage. We then introduce a separate stage to train the Indexer through attention supervision. Thus, the training objectives and their roles differ from those in Zhang et al.
  • Explicit retrieval of distant tokens. LISA introduces an Indexer that dynamically selects relevant tokens from the full history for sparse attention. This provides direct access to historical token information that may be lost when compressed into the recurrent LA state, complementing the adopted state-correction mechanism.

We have also provided avg@16 results and additional controlled ablations. The avg@16 results show that LISA’s average performance advantage over Zhang et al. remains positive under repeated-sampling evaluation. Our experiments start from an existing pretrained checkpoint, with the original backbone frozen and approximately 110M trainable parameters in the 1.5B setting. We additionally trained the matched-training sliding-window + LA control and the equal-capacity aggregated-KL variant, which provide evidence for the contributions of dynamic token selection and head-specific supervision, respectively. Although these experiments do not directly quantify training-seed variance, they provide complementary empirical support for our conclusions beyond a single comparison with Zhang et al.

Reviewer_z5i5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LISA, a replacement for standard attention modules in a trained LLM. LISA has two components, a linear attention module and an Indexer that selects the most relevant tokens for a sparse attention module, which are fused to produce the attention output. Training these modules (while freezing the rest of the LLM) on a math reasoning dataset improves performance and inference efficiency relative to other efficient attention methods.

Critical strengths

The paper addresses a relevant problem of reducing the computational cost of long chain-of-thought reasoning. The combination of linear attention with an Indexer-based sparse attention module is intuitive, and LISA shows consistent improvements in inference latency across mathematical reasoning tasks while maintaining or improving accuracy. The ablations also suggest that both attention components are useful.

Critical weaknesses

Although the paper claims that LISA is "plug-and-play" and can be used without pretraining from scratch, significant additional training is still required on a dataset of about 100k reasoning examples (~480 H100 GPU-hours). Comparisons against other methods don't include the amount of compute required during training. Although the authors state that the baselines use comparable numbers of trainable parameters, this doesn't necessarily mean comparable training compute, making it unclear how much of the accuracy improvements comes from the architecture rather than additional training.

The paper is also missing some naive comparisons against other full-attention training methods (e.g. SFT) on the same data and with the same compute budget. Training LISA with a next-token prediction objective on reasoning traces generated by a stronger teacher model could improve performance and change reasoning trace length independently of the architecture. Training LISA on traces generated by the starting model itself, or comparing against a full-attention model trained on the same teacher traces, would help isolate the effect of adapting the model to use the new attention mechanism.

Questions for authors
  1. Can you clarify the amount of compute required to train each method? Especially for Zhang et al. (2026), as it performs closest to LISA.

  2. Did you compare LISA against a full-attention model fine-tuned on the same reasoning data with a similar compute budget? This would help differentiate between improvements from architecture and improvements from training on new data.

  3. How representative are the single-run results across methods? Table 2 shows lower mean scores for AIME24 and AIME25 than the single-run results in Table 1. In particular, the AIME24 mean would put LISA below the reported Zhang et al. result, although this would not be a like-for-like comparison. Could you explain how the representative runs were selected and provide averages across multiple evaluation runs for the other methods?

  4. Can you provide the reasoning length and inference latency for the non-mathematical benchmarks in Table 3? Since efficiency is a central motivation of the paper, it would be useful to see whether the speedups generalize beyond mathematical reasoning tasks.

3 comments

W1: Meaning of “plug-and-play” and adaptation cost

Thank you for pointing out the ambiguity. By “plug-and-play,” we mean converting an existing dense checkpoint into a hybrid architecture combining linear attention and dynamic sparse attention through post-training adaptation. The original model parameters remain frozen while the added components are trained. This avoids pretraining from scratch, but does require adaptation training. We will clarify this meaning and the associated training cost in the revised manuscript.

In the 1.5B experiments, the compared methods use a similar trainable parameter count, approximately 110M, and the same training dataset of approximately 95K examples, to improve the comparability of the adaptation settings.

W2: Full-attention control with additional training

The full-attention baseline has already undergone standard LoRA fine-tuning, as stated in Section 4.2. The descriptions in the Table 1 caption and Section 4.4 are inaccurate and will be corrected. Because this baseline retains the original full-attention architecture while receiving additional training, it provides a direct control for ordinary adaptation without the LISA conversion.

The distinction between the original DeepSeek-R1-Distill-Qwen-1.5B checkpoint and our fine-tuned baseline is shown below. We will add this clarification to the revised manuscript.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Original checkpoint76.169.052.523.313.346.8
Full-attention LoRA baseline82.178.862.520.016.752.0

Q1: Training compute, particularly for Zhang et al.

As noted in W1, we control the trainable parameter count, approximately 110M for the 1.5B experiments, and use the same training dataset of approximately 95K examples across the baselines. Our implementation of Zhang et al. therefore uses a comparable number of trainable parameters and the same training data as LISA.

Q2: Fine-tuned full-attention baseline

Yes. The full-attention row in Table 1 corresponds to a DeepSeek-R1-Distill-Qwen model fine-tuned using standard LoRA on the same training data and with a comparable trainable parameter count. We will correct the erroneous description of this row as the original model without additional training.

Q3: Single-run results versus repeated-sampling results

Table 1 uses greedy decoding. Table 2 instead uses sampling with temperature = 0.6 and top-p = 0.95, following the DeepSeek-R1 sampling settings, and generates 16 responses per question. The table reports the mean and standard deviation of accuracy across the 16 evaluation runs. Because the decoding protocols differ, the greedy result is not expected to equal the repeated-sampling mean. Table 1 does not select a sampled run from Table 2.

We have also evaluated Zhang et al. under the repeated-sampling protocol:

MethodAMC23AIME24AIME25Avg.
Full-attention LoRA baseline62.5 ± 2.2419.6 ± 1.6516.1 ± 1.8332.73
LISA70.0 ± 2.7425.8 ± 1.9625.9 ± 2.2940.57
Zhang et al.66.9 ± 2.7325.0 ± 2.3822.5 ± 2.5038.13

Q4: Output length and latency on non-mathematical tasks

We report the average output length in tokens (Tok) and the mean inference time per sample (ReL, seconds/sample) for the 1.5B models below.

TaskFull-attention TokFull-attention ReL (s/sample)LISA TokLISA ReL (s/sample)
GPQA-Diamond9,242354.89,016262.9
HumanEval7,257279.37,015196.8
LiveCodeBench13,741532.513,017413.7
BigCodeBench12,639503.112,157396.1

I thank the authors for their clarifications and additional experiments. Confirming the full-attention baseline was fine-tuned on the same data addressed my concern that it was an undertrained baseline, and the repeated sampling results add credence to the reported improvements.

I still have some concerns about the efficiency gains in a more realistic setting, as the increased KV-cache memory requirements from LISA could limit batch sizes and throughput, potentially offsetting the per-sample latency improvements reported.

Overall, the rebuttal addresses several of my concerns while leaving my concern about in-practice efficiency, and I will maintain my score of 3.

Reviewer_7pfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Lisa, a reasonable hybrid of linear attention and Indexer selected sparse attention for converting pretrained reasoning models. Training has two stages, with original weights frozen. Stage 1 trains the linear branch (LoRA) with next token cross entropy. Stage 2 replaces the window with a Lightning Indexer style token selector, trained with a per head KL loss against grouped teacher attention distributions. At inference, the method applies the segment level state correction of Zhang et al. (2026).

Critical strengths

What I liked most is how honest the paper is about its own limits. Many efficiency papers oversell asymptotic gains, but here the authors say plainly that the complexity stays quadratic, that the full KV cache is kept, that latency was only measured in eager mode, and that the training budget isn’t matched to the baseline. They also ran 16 evaluations on the small benchmarks to show variance, which I wish more papers did. On the technical side, supervising each Indexer head with a group of four teacher heads is a simple and sensible idea, and the candidate set dropout is a practical way to deal with the hard top M selection.

Critical weaknesses

I have concern about the real contribution of the claimed new architecture. Table 4 reports that LISA without segment correction achieves 50.4% average accuracy, below the unmodified full-attention baseline of 51.6% in Table 1. Full LISA achieves 57.1%, but segment correction is explicitly adopted from Zhang et al. (2026). I am concerned about where the acc improvement comes from, is it from the new architecture or just the adopted component.

Besides, Using the model configurations and the paper's stated Indexer design, I obtain approximately 2.23 times the original KV-cache element count for 1.5B and 2.44 times for 7B, when the additional Indexer key cache is included. This assumes a separate 256-dimensional cache for each Indexer head and the same storage precision. The paper acknowledges the additional cache but does not report peak GPU memory. This is especially relevant because DeepSeek-V3.2 uses a shared Indexer key cache rather than independent keys for every Indexer head.

Also, several references and descriptions require correction. In particular, Chen et al. (2025b) is cited as theoretical support for the test-time training properties of linear attention, although the referenced work concerns overthinking in reasoning models. The MATH-500 benchmark should additionally cite Lightman et al. (2023), and the experimental tables should explicitly identify the DeepSeek-R1-distilled Qwen checkpoints.

Open source code will be highly beneficial.

Questions for authors
  1. Can you report the original full attention model with the same segment correction applied, so the architectural contribution can be separated from the borrowed inference procedure?
  2. Can you report a full attention baseline fine tuned with LoRA of comparable size on the same 100K OpenR1 Math samples and schedule? Without this, it is unclear whether the gains come from the architecture or from the extra training.
  3. Can you include additional comparisons with closely related methods, such as SeerAttention-R (Gao et al., 2026), Quest (Tang et al., 2024), DuoAttention (Xiao et al., 2025), TidalDecode (Yang et al., 2025), and LoLCATs (Zhang et al., 2025), to better establish LISA's advantages over existing sparse attention and post-hoc attention conversion approaches in terms of reasoning accuracy, inference latency, and memory efficiency?
3 comments

W1: Contribution of the architecture relative to segment correction

Thank you for raising this question. LISA converts an existing full-attention reasoning model into a hybrid architecture combining recurrent LA memory with Indexer-guided sparse retrieval. This allows the model to combine access to distant tokens with state-based reasoning correction. Our contribution is this combination and its post-training conversion procedure.

The general idea of test-time training predates Zhang et al.; the specific component we adopt from their work is the segment-correction rule. This rule updates the recurrent state SS of the LA branch. The original full-attention model has no corresponding recurrent state, so the same update rule cannot be directly applied to it.

Converting an existing dense model through post-training into an architecture that supports dynamic retrieval, recurrent memory, and this TTT-style state correction is one of the central contributions of our work.

W2: Additional cache overhead

We acknowledge the additional cache overhead of the current implementation. The cache-capacity estimates at different context lengths are provided below. We will explore GQA-based designs to improve memory efficiency in future work.

ModelTotal context tokensOriginal KV (GiB)Additional Indexer keys (GiB)Combined (GiB)Ratio to original KV
1.5B4,0960.10940.13480.24412.232×
1.5B8,1920.21880.26950.48832.232×
1.5B16,3840.43750.53910.97662.232×
1.5B32,7680.87501.07811.95312.232×
7B4,0960.21880.31450.53322.438×
7B8,1920.43750.62891.06642.438×
7B16,3840.87501.25782.13282.438×
7B32,7681.75002.51564.26562.438×

W3: References and model names

Thank you for identifying these issues. We will correct the relevant references and use the complete model names consistently throughout the revised manuscript.

W4: Code availability

We will release our code upon acceptance to support reproducibility.

Q1: Applying the same correction to the original full-attention model

The adopted segment-correction rule directly updates the recurrent state SS of the LA branch. The original full-attention model has no corresponding state, so this rule cannot be applied to it without changing the architecture.

Q2: Full-attention adaptation control

The full-attention baseline has already undergone standard LoRA fine-tuning, as described in Section 4.2. We will correct the Table 1 caption and Section 4.4, which inaccurately describe its training status. This baseline retains the original attention architecture and provides a control for standard adaptation without the LISA conversion.

For clarity, we distinguish the original checkpoint from the fine-tuned full-attention baseline below and will include this distinction in the revised manuscript.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Original checkpoint76.169.052.523.313.346.8
Full-attention LoRA baseline82.178.862.520.016.752.0

Q3: Related sparse-attention and attention-conversion methods

Thank you for these suggestions. We will expand the comparisons and discussion of these methods in the revised manuscript.

For dynamic sparse attention, Quest selects relevant KV pages using the current query and page-level key extrema, while SeerAttention-R freezes the original model and trains a gate through self-distillation to predict relevant KV blocks. Both reduce attention costs by selectively accessing historical KV entries. DuoAttention separates retrieval heads that retain the full history from streaming heads that attend only to attention sinks and recent tokens, reducing cache and computation. TidalDecode uses correlations between important positions in nearby layers to reuse token selections across layers. For post-training attention conversion, LoLCATs combines attention-output matching and subsequent LoRA adaptation to convert an existing Transformer into a model with linear attention and local sliding-window attention.

LISA shares the goals of exploiting attention sparsity and adapting existing models. Its specific design combines recurrent LA memory with a trainable Indexer that dynamically retrieves relevant distant tokens, extending explicit retrieval beyond a local window, together with segment-level correction of the LA state. This design retains explicit access to relevant tokens throughout the history, while requiring the full KV cache and an additional Indexer-key cache.

Thank you for your response. However, some of my main concerns have not been fully addressed. I was expecting to see experimental results particularly regarding the architectural contribution beyond segment correction, peak GPU memory usage, and comparisons with related methods I listed. So I will maintain my original rating.

Reviewer_1dceOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LISA converts DeepSeek-R1-distilled Qwen models (1.5B, 7B) by replacing 23 of 28 attention blocks with a gated mix of linear attention and sparse attention over 256 tokens chosen by a learned ReLU indexer. Training has two stages: a linear-attention cold start with a sliding window, then indexer training with per-head KL to grouped teacher attention. An inference-time segment correction is adopted from previous work. The full KV cache is kept, so gains are constant-factor. On five math benchmarks (1.5B), accuracy rises from 51.6% to 57.1% and eager-mode latency falls 23.9%; over Zhang et al., accuracy goes from 55.8% to 57.1% (1.5B) and from 68.5% to 70.6% (7B).

Critical strengths
  1. A sensible hybrid with encouraging results. Exact retrieval plus recurrent memory improves both accuracy and latency at two scales, and removing either branch hurts badly.

  2. Transparent cost accounting. The paper reports the extra parameters (111M/420M), the growing indexer-key cache, ~240 H100 GPU-hours per stage, and counts indexer scoring in its FLOP estimate, which I could reproduce (5.44 GFLOPs/token for full attention at 16K).

Critical weaknesses
  1. I find that the gain from LISA's new parts is not isolated. Linear plus restricted attention with gating and state correction comes from previous work, and a ReLU indexer trained with KL comes from DeepSeek-V3.2. Yet "Stage 1 only" (54.3%) also skips Stage 2 training, the single-head ablation (55.6%) changes capacity, there is no aggregated-target or candidate-dropout ablation, the adopted segment correction alone accounts for 6.7 points in Table 4, and the baseline gets no extra training.

  2. Key details of the paper claims are missing or inconsistent. The segment-correction values k and α are not given; Figures 2–3 label the dense layers 7/14/21 as LISA layers; Eq. 1 and Eq. 5 describe the gate differently; and the linear-attention shapes, decoding settings, seeds, prompts, and timing setup are unspecified.

  3. Authors should also revise the presentation of the work.

Questions for authors
  1. How do matched controls compare: full attention fine-tuned on the same data and budget, and sliding-window plus linear attention after the same Stage 2 updates?
  2. At equal capacity, what do aggregated-head KL and no candidate dropout give?
  3. What are k, α, the linear-attention shapes, and the actual dense-layer schedule?
  4. What are the decoding and timing settings, and what are fixed-length latency and peak memory?
  5. Do the gains over prior work (Zhang et al.) and the single-head indexer hold under repeated or paired runs?
2 comments

W1: Isolation of individual components

Thank you for this concern. First, the full-attention baseline in Table 1 has undergone standard LoRA fine-tuning; it is not the original checkpoint without additional training. We will correct the erroneous descriptions in the caption and Section 4.4.

The distinction between the original checkpoint and the fine-tuned full-attention baseline is shown below and will be added to the revised manuscript.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Original checkpoint76.169.052.523.313.346.8
Full-attention LoRA baseline82.178.862.520.016.752.0

Second, we have expanded the ablation study with continued training of the sliding-window + LA model at a matched compute budget and an aggregated-KL control that retains the multi-head Indexer architecture. The expanded results are shown below. The two added controls are highlighted in bold. Accuracies are percentages, and Avg. is the unweighted mean across the five tasks.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Full LISA82.282.070.026.726.757.5
Sliding-window + LA, continued training at matched compute77.979.262.523.320.052.6
Stage 1 only (sliding-window + LA)79.280.665.023.323.354.3
Linear only68.367.855.013.410.042.9
Sparse only66.660.842.56.76.736.7
No gating78.380.265.023.320.053.4
Single-head Indexer79.581.267.526.723.355.6
Multi-head Indexer + aggregated KL79.280.265.026.720.054.2
No segment correction74.977.862.520.016.750.4

W2: Implementation details and layer layout

Thank you for identifying these inconsistencies. The dense-layer positions and counts in Sections 3.2 and 4.1 are incorrect; the layer labels in Figures 2–3 are correct. Under zero-based indexing, full attention is retained at layers 0, 6, 12, 18, 24, and 27, giving six dense layers and 22 LISA layers. We will correct the text and update the theoretical FLOPs and cache-capacity calculations in Appendix A and Table 5 accordingly.

The sparse-attention token budget is M = 256. The 1.5B and 7B models use K = 3 and K = 7 Indexer heads, respectively, with a width of 256 per head. The LA module uses LoRA with rank 256. Inference-time segment correction uses segment length k = 512 and correction coefficient α = 0.3.

For timing, we record the inference time of each sample and take the arithmetic mean over the dataset. ReL is measured in seconds per sample.

W3: Presentation

Thank you for the suggestions. We will revise the affected descriptions and improve the presentation in the revised manuscript.

Q1: Full-attention and sliding-window + LA controls

The full-attention LoRA control is clarified in W1 above. We additionally continued training the sliding-window + LA model under the same training compute budget. Its accuracy decreased relative to the Stage 1 checkpoint. One possible explanation is that adapting the attention module is sensitive to the training distribution and that continued training can lead to overfitting.

Q2: Aggregated-head KL at matched capacity

We have added this control to the ablation table in W1. The Indexer retains the same head count, per-head width, and parameter count; only the supervision objective changes from per-head KL to aggregated KL. The aggregated-KL variant achieves a five-task average of 54.2%, compared with 57.5% for per-head KL in Full LISA. A possible explanation is that aggregation weakens the head-specific alignment constraints between the Indexer and its corresponding teacher distributions.

Q3: Segment-correction parameters and layer configuration

Please see W2 above for the segment-correction parameters, Indexer configuration, and corrected dense-layer layout.

Q4: Decoding and timing settings

The main table uses greedy decoding. Table 2 uses temperature = 0.6 and top-p = 0.95, with 16 sampled responses per question, and reports the mean and standard deviation of accuracy. ReL is the arithmetic mean of the per-sample inference times over the dataset, measured in seconds per sample.

Q5: Repeated-sampling comparison with Zhang et al. and the single-head Indexer

With the corrected values in Table 4, the single-head Indexer and Full LISA achieve five-task average accuracies of 55.6% and 57.5%, respectively. We further evaluated the methods using avg@16 with temperature = 0.6 and top-p = 0.95:

Method (1.5B)AMC23, avg@16AIME24, avg@16AIME25, avg@16Avg.
Full-attention LoRA baseline62.519.616.132.73
LISA70.025.825.940.57
Zhang et al.66.925.022.538.13
Single-head Indexer67.325.422.438.37
Reviewer_rKQ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LISA converts a full-attention reasoning model into a hybrid linear–sparse-full attention model through a two-stage post-training procedure. Its main contribution is the training objectives and conversion strategy.

Critical strengths

Converting existing models without pretraining from scratch is practically relevant. The two-stage training procedure offers a promising approach to maintaining reasoning quality while reducing inference cost.

Critical weaknesses

(1) Architectural novelty is limited relative to existing hybrid-attention approaches.

(2) Several 1.5B baseline rows match the cited state-transition paper numerically, despite conflicting descriptions of whether the full-attention baseline received additional training. The Zhang et al. row also differs in GSM8K accuracy despite matching generation length and latency.

(3) Without matched-training controls, the accuracy gains cannot be attributed clearly to the proposed conversion method rather than additional data and optimization. Acknowledging this confound does not resolve it.

Questions for authors

(1) Which results were reproduced versus reused, and which checkpoints and configurations generated them?

(2) Was the full-attention baseline additionally trained? Please reconcile its description and the numerical discrepancies with the cited work.

(3) Can the authors provide a full-attention control with matched training data, token budget, and comparable trainable parameter count?

(4) Under matched conditions, do the hybrid architecture and proposed training objectives each provide measurable gains?

1 comment

W1: Contribution relative to existing hybrid-attention methods

LISA introduces a modular post-training procedure for converting an existing dense model into a hybrid LA architecture that supports TTT-style state correction. In our evaluations, LISA preserves or improves overall task performance while accelerating inference.

Whereas hybrid architectures trained from scratch require pretraining, our approach adapts an existing dense checkpoint. The Indexer complements recurrent memory with explicit retrieval of relevant distant tokens, helping recover information that may be lost through compression into the LA state. The contribution lies in this combination and its conversion procedure.

W2: Baseline training status and numerical inconsistencies

The full-attention baseline has undergone standard LoRA fine-tuning, as stated in Section 4.2. Its descriptions in the Table 1 caption and Section 4.4 are inaccurate and will be corrected. The model retains the original full-attention architecture while receiving adaptation training, making it a control for standard fine-tuning without the LISA conversion.

To distinguish it from the original DeepSeek-R1-Distill-Qwen-1.5B checkpoint, we provide the following results and will include this distinction in the revised manuscript.

Model (1.5B)GSM8KMATH-500AMC23AIME24AIME25Avg.
Original checkpoint76.169.052.523.313.346.8
Full-attention LoRA baseline82.178.862.520.016.752.0

We will also correct the remaining numerical typographical errors in the manuscript.

W3: Additional data and optimization as possible explanations

As clarified in W2, the full-attention baseline also receives LoRA adaptation. The erroneous descriptions of its training status will be corrected so that the comparison is interpreted consistently.

Q2: Whether the full-attention baseline receives additional training

Yes. The full-attention baseline in Table 1 is a DeepSeek-R1-Distill-Qwen-1.5B/7B model adapted using standard LoRA fine-tuning, as described in Section 4.2. The caption and Section 4.4 incorrectly call it an “unmodified original model/checkpoint.” We will consistently describe it as a baseline that retains the original full-attention architecture and receives standard LoRA adaptation.

Q3: Full-attention control with comparable adaptation settings

As stated in Section 4.2, the baselines use a comparable trainable parameter count, approximately 110M in the 1.5B experiments, and the same training dataset of approximately 95K examples. These controls improve comparability with respect to adaptation capacity and training data.

Q4: Contributions of the hybrid architecture and training objectives

We apologize for the confusion caused by the incorrect description of the baseline. With its training status clarified, the main comparisons evaluate LISA against an adapted full-attention model. The component ablations further examine the roles of the hybrid branches, gating, Indexer design, and inference-time correction, supporting the performance and inference-efficiency benefits of the overall framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.