acceptodds
Under review as a conference paper at ICLR 2027

FastRebase: Hybrid Low-Rank States for Efficient Linear-Attention Decoding

Abstract

Linear attention replaces the growing key–value cache of Transformers with a fixed-size recurrent state, but repeatedly accessing this state can still incur substantial memory traffic during batched decoding. Low-rank approximation offers a natural way to reduce this cost. However, maintaining such representations online is challenging due to the cost of repeated singular value decompositions (SVDs). To address this, we introduce FastRebase, which maintains the recurrent states of linear attention models using a hybrid low-rank representation throughout inference without online SVDs. Our key insight is that rank allocation and subspace tracking can be decoupled: FastRebase determines a per-head rank profile offline and tracks only a sequence-dependent subspace during inference using GPU-efficient projections and small-matrix solves. Its hybrid state combines a low-rank history with a residual buffer of recent key–value associations, enabling periodic rebasing that amortizes the cost of subspace updates over multiple decoding steps. We implement FastRebase with custom Triton kernels and evaluate it on models with scalar or diagonal decay and delta-rule state transitions. FastRebase reduces recurrent-state traffic by 61.7–87.3%, while closely matching the dense baselines in zero-shot accuracy and long-context perplexity. At batch size 128 on an H100 GPU, the traffic reduction translates into 1.36–4.73× speedups for linear-attention recurrence and 1.18–2.31× for end-to-end decoding.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
linear attention, recurrent state compression, low-rank approximation, efficient LLM inference, autoregressive decoding, GPU acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_53S9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

FastRebase compresses recurrent states in linear-attention models using low-rank factorization and a recent key-value residual buffer. It combines offline rank calibration, online subspace recycling, and custom Triton kernels. Experiments on six 1.3B model families show reduced memory traffic and faster decoding.

Critical strengths
  1. Addresses an important memory-bandwidth bottleneck.
  2. Combines a practical compression method with efficient GPU kernels.
  3. Evaluates six model families and reports meaningful decoding speedups.
Critical weaknesses
  1. Retrieval quality degrades substantially. For example, DeltaNet FDA accuracy drops from 38.69 to 11.26, and Mamba2 from 35.42 to 17.44. SVD-based initialization recovers much of the loss but adds complexity.
  2. Rank calibration is evaluated on limited WikiText-2 data, leaving robustness to other distributions unclear.
  3. Evaluation is limited to H100 GPUs, and end-to-end speedups vary considerably across models and batch sizes.
  4. Comparisons against alternative methods at matched memory and latency budgets are insufficient.
Questions for authors

Q1. Why does one-shot SVD initialization recover so much retrieval accuracy when the calibrated rank profile remains unchanged? Can the authors identify which aspect of online subspace recycling causes the largest loss? Q2. How much does the calibrated rank profile change when the calibration set, sequence length, or energy threshold changes? Can a profile calibrated on WikiText-2 transfer to substantially different inference distributions? Q3. How does the retained energy of FastRebase compare with the truncated-SVD optimum at each rebase? Is there a systematic relationship between this gap and downstream retrieval degradation? Q4. At what batch sizes and model configurations does FastRebase cease to provide a meaningful end-to-end benefit? Can the authors explain the particularly modest gains for DeltaNet? Q5. Is the one-shot SVD result intended as a diagnostic experiment or a viable deployment option? If the latter, what is its latency and memory cost, and does it offset the gains during realistic serving workloads?

Reviewer_uBTSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FastRebase, which keeps the recurrent state of linear attention in a hybrid form: a low-rank history plus a buffer of recent key–value pairs. Per-head ranks are calibrated offline, and every pp steps the buffer is folded into the low-rank part using a one-step subspace update without SVD. The method is implemented with custom Triton kernels and evaluated on six 1.3B models with scalar decay, diagonal decay, and delta-rule transitions. It reduces recurrent-state traffic by 61.7–87.3% and gives 1.18–2.31× end-to-end decoding speedups at batch size 128 on an H100, while keeping perplexity and zero-shot accuracy close to the dense baselines.

Critical strengths
  1. Figure 1 shows that recurrent-state traffic dominates decoding latency at large batch sizes, and the paper turns the traffic reduction into measured end-to-end speedups with custom kernels on six models, rather than only analytical savings. This is the main reason the paper is of practical value.

  2. Table 3 shows that a fixed subspace degrades quality substantially, while online subspace updates at a fixed rank recover the quality of AdaptiveRebase. This clearly identifies which part of the low-rank approximation must be computed online.

Critical weaknesses
  1. Limited novelty. Each component has a close precedent. Sun et al. (2026) already reported that whether a head is low-rank or high-rank is a stable property acquired during pretraining and consistent across inputs, which largely anticipates the claimed key insight that ranks can be fixed offline. OjaKV (Zhu et al., 2026) combines low-rank history, exact recent entries, and online subspace updates from a calibration-initialized basis. DeltaLog (Lin et al., 2026) and KVBuffer (Zou & Zhong, 2026) use the same buffer-and-merge scheme. Value-subspace recycling is essentially a warm-started one-step block power iteration, as used in streaming PCA (e.g., Mitliagkas et al., 2013), and the delta-rule corrected value coincides with the pseudo-value ut=βt(vt−St−1kt)u_t = \beta_t(v_t - S_{t-1}k_t) already used in DeltaNet (Yang et al., 2024b). The contribution is mainly integration and implementation, and the paper does not discuss these overlaps.

  2. Quality claims are overstated and not tested in the target regime. The abstract omits the clear drop on recall tasks (e.g., from 35.38 to 27.69 for DeltaNet, and from 38.69 to 11.26 on FDA). All quality evaluations are likelihood based or use very short generations, so long autoregressive generation, where rebase errors accumulate, is not evaluated. There is also no analysis of how far the tracked subspace drifts from the optimal rank-rr subspace.

  3. The case against online SVD relies on a weak baseline. Table 14 reports roughly 76–698 ms per decoding step for eigendecompositions of matrices of size at most about 112×112112 \times 112. This more likely reflects the implementation than the intrinsic cost. Since this result is the main motivation for value-subspace recycling, a batched eigensolver kernel or multi-step randomized subspace iteration should be compared.

  4. Incomplete baselines. Table 1 compares DRRQR with AdaptiveRebase rather than with FastRebase. DRRQR collapses only on models where the authors extended it themselves, and it is tested at a single reduction ratio. There is no speed comparison with KVBuffer or DeltaLog, which are the most directly related methods.

  5. Limited scope. Only 1.3B pure linear-attention models are tested, not larger or hybrid models used in practice. At batch size 32, the end-to-end speedup is only 1.01× for DeltaNet and 1.08× for GDN, so the benefit is limited to large-batch serving.

Questions for authors
  1. How is rebasing performed in the chunkwise prefill path, and how does the rebase interval p=16p = 16 relate to the chunk size? The quality results depend on this path.

  2. How many heads reach the rank cap (half the rank limit, up to 96), and is the target η\eta achieved for them?

  3. How does FastRebase perform on long-generation tasks, such as chain-of-thought reasoning or RULER?

  4. Can you report the energy retained by FastRebase relative to the optimal rank-rr truncation as a function of sequence length?

  5. How does AdaptiveRebase compare in speed when using a batched eigensolver kernel?

  6. How does FastRebase compare in speed with KVBuffer and DeltaLog, and can the methods be combined?

  7. What are the end-to-end cost and speedup of dense prefill followed by one-shot SVD, given that this setting recovers most of the recall accuracy?

Reviewer_7BrpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that batched decoding in linear-attention models is heavily bottlenecked by recurrent state (HtH_t) memory traffic, and proposes FastRebase to compress the state without training or expensive online SVDs. The method represents the recurrent state as a low-rank factor pair (UtWt⊤U_t W_t^\top) combined with a small residual token buffer, and combines offline rank calibration with Cholesky-based value-subspace recycling for periodic rebasing. To demonstrate the effectiveness of the proposed method, the authors perform experiments across six 1.3B linear-attention models (GLA, RetNet, HGRN2, Mamba2, DeltaNet, and GDN), evaluating zero-shot accuracy, batched decoding speedups on H100 GPUs, 16K perplexity extrapolation, and recall-intensive benchmarks.

Critical strengths

(S1) SVD-free hybrid low-rank state formulation: By splitting the recurrent state into a low-rank component and a short residual buffer, and replacing online SVD with offline rank calibration and Cholesky-based value-subspace recycling, the authors avoid the severe latency overhead of online SVD (which slows down decoding by 16.57x to 84.69x in Table 2). Without requiring model retraining, this formulation reduces state memory traffic by 61.7% to 87.3% at batch size 128 and achieves measurable decoding speedups across six 1.3B linear-attention architectures.

Critical weaknesses

(W1) Significant degradation on recall tasks, contradictory One-shot SVD Fix, and Missing Long-Context Benchmarks: On the six 2K-context recall-intensive tasks (Table 5 and Table 13), standard FastRebase suffers noticeable accuracy drops across all models due to cumulative information loss during online subspace updates (for example, Mamba2 drops by 6.64 points and DeltaNet drops by 7.69 points, with DeltaNet falling from 38.69 to 11.26 on FDA). To recover this loss, the authors introduce One-shot SVD, which materializes the full dense state during prefill and runs an exact SVD before decoding. However, this directly contradicts the core motivation of avoiding dense state materialization and SVD overhead. Moreover, the 16K long-context evaluation (Figure 4) only reports PG-19 perplexity and omits standard long-context retrieval benchmarks such as Needle-in-a-Haystack (NIAH) or LongBench, leaving it unclear how repeated rebasing affects information retention over truly long contexts.

(W2) Marginal speedups at smaller batch sizes and on delta-rule models: While the main text highlights decoding speedups at a large batch size of 128 (1.18x to 2.31x in Table 4), the practical gains at smaller batch sizes are very limited. According to Table 15 in the appendix, at batch size 32 the end-to-end decoding speedup is only 1.01x on DeltaNet, 1.08x on GDN, and 1.15x on HGRN2, and results for batch sizes 1 to 16 are not reported at all. In particular, the stronger Delta-rule models (DeltaNet and GDN) require higher calibrated ranks and extra value-correction steps, which substantially reduces the speedup benefit.

(W3) Lack of comparison with recent serving/compression baselines and evaluation limited to 1.3B models: Although the introduction and related work (Section 1 and Appendix D) discuss recent linear-attention serving and state-compression methods such as KVBuffer (Zou & Zhong, 2026), DeltaLog (Lin et al., 2026), head pruning (Sun et al., 2026), and OjaKV (Zhu et al., 2026), the experiments do not compare FastRebase against any of them in terms of quality or latency. The only external baseline included is DRRQR (Nazari & Rusch, 2026b; Table 1), which is applied via simple zero-masking to architectures it was not designed for (Appendix B), ignoring both the projection parameter/compute savings of actual channel removal and potential recovery after lightweight fine-tuning. Finally, all experiments are restricted to 1.3B models, so it remains unverified whether the same speedups hold at larger model scales (such as 7B or above) where weight projection and FFN costs become more dominant.

References

  • Junqing Lin, Jingwei Sun, and Guangzhong Sun. DeltaLog: Deferred materialization of recurrent states for linear attention decoding. arXiv preprint arXiv:2608.15533, 2026.
  • Philipp Nazari and T. Konstantin Rusch. The key to state reduction in linear attention: A rank-based perspective. arXiv preprint arXiv:2602.04852, 2026b.
  • Ao Sun, Hongtao Zhang, Heng Zhou, Yixuan Ma, Yiran Qin, Tongrui Su, Yan Liu, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, and Renqing He. State rank dynamics in linear attention LLMs. arXiv preprint arXiv:2602.02195, 2026.
  • Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, and Pin-Yu Chen. OjaKV: Context-aware online low-rank KV cache compression. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 10161–10178, 2026.
  • Longwei Zou and Lin Zhong. KVBuffer: IO-aware serving for linear attention. arXiv preprint arXiv:2605.19049, 2026.
Questions for authors

Please see the critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.