- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LISA, a hybrid architecture combining linear attention, dynamically indexed sparse attention, learned gating, and inference-time segment correction. For DeepSeek-R1-Distill-Qwen 1.5B and 7B models, the paper reports higher average math accuracy than the full-attention baseline and 23.9% lower eager-mode latency at 1.5B.
Critical strengths
The paper studies an important efficiency challenge in long-CoT reasoning. The combination of linear and dynamically selected sparse attention is potentially useful.
Critical weaknesses
The paper suffers from the following major concerns:
(1) LISA trains 111M (1.5B) or 420M (7B) new parameters on 100K OpenR1-Math samples, taking around 480 H100 hours. But it does not apply to the full-attention baseline. Even though the authors do acknowledge but “+4.2 points over the full-attention baseline” is stated as the result. To have a fair comparison, for the full-attention base model, we should train a LoRA adapter trained on the provided dataset. Without that, the comparison is not meaningful.
(2) One top of point (1), I find this paper very incremental, as without using segment correction Zhang et al. (2026), the own component performs even worse than the baseline. Moreover, the improvement to Zhang et al. (2026) is very marginal, i.e., ~+1.3 points at 1.5B and +2.1 at 7B, from single runs.
Questions for authors
Please check the Critical Weaknesses section.
3 comments
The explanation regarding the LoRA-trained baseline addresses my first concern. However, I remain concerned about the limited novelty beyond Zhang et al., as the improvements are relatively modest and the individual contributions are not convincingly isolated. The repeated evaluation runs also do not establish robustness across training seeds. Therefore, my overall concern remains, and I maintain my rating.
Thank you for your follow-up.
Regarding the contribution beyond Zhang et al., we would like to highlight the following differences:
- Simpler data preparation. LISA does not require clustering the training trajectories or introducing special tokens to mark reasoning steps. It operates directly on ordinary CoT sequences, reducing data-preparation complexity and preprocessing overhead.
- Simpler adaptation of the linear-attention branch. Zhang et al. jointly optimize an autoregressive objective and an output-distribution distillation objective. LISA trains the LA branch using standard next-token cross-entropy during the cold-start stage. We then introduce a separate stage to train the Indexer through attention supervision. Thus, the training objectives and their roles differ from those in Zhang et al.
- Explicit retrieval of distant tokens. LISA introduces an Indexer that dynamically selects relevant tokens from the full history for sparse attention. This provides direct access to historical token information that may be lost when compressed into the recurrent LA state, complementing the adopted state-correction mechanism.
We have also provided avg@16 results and additional controlled ablations. The avg@16 results show that LISA’s average performance advantage over Zhang et al. remains positive under repeated-sampling evaluation. Our experiments start from an existing pretrained checkpoint, with the original backbone frozen and approximately 110M trainable parameters in the 1.5B setting. We additionally trained the matched-training sliding-window + LA control and the equal-capacity aggregated-KL variant, which provide evidence for the contributions of dynamic token selection and head-specific supervision, respectively. Although these experiments do not directly quantify training-seed variance, they provide complementary empirical support for our conclusions beyond a single comparison with Zhang et al.
Thank you for raising this concern. We first clarify an error in the manuscript that affects the interpretation of this comparison. The full-attention baselines in Table 1 are DeepSeek-R1-Distill-Qwen-1.5B/7B models adapted using standard LoRA fine-tuning, rather than the original checkpoints without additional training. Although Section 4.2 states that the baselines are trained with LoRA, the Table 1 caption and Section 4.4 incorrectly describe them as an “unmodified original model/checkpoint.” We will correct these descriptions and consistently identify the models and their training status.
Converting an existing dense model through post-training into an architecture that supports dynamic retrieval, recurrent memory, and this TTT-style state correction is one of the central contributions of our work.
For the 1.5B experiments, the compared methods use approximately 110M trainable parameters and the same training dataset of approximately 95K examples, controlling for trainable capacity and training data. To distinguish the original checkpoint from our fine-tuned full-attention baseline, we provide their results below and will include this distinction in the revised manuscript. All accuracies are percentages; Avg. is the unweighted mean across the five tasks.
LISA differs from Zhang et al. in two main respects. First, LISA uses fixed-length segments for state correction and does not require clustering or annotating thinking patterns in training trajectories, or introducing special tokens to delimit reasoning steps. This reduces data preparation and avoids modifying the token format of ordinary CoT text. Second, the Indexer dynamically selects relevant tokens from the full history for explicit retrieval through sparse attention, complementing the information retained in the recurrent state.
Table 2 reports repeated-sampling results for the full-attention baseline and LISA. We have extended this evaluation to Zhang et al. The results below report mean accuracy ± standard deviation over 16 evaluation runs.