- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
FastRebase compresses recurrent states in linear-attention models using low-rank factorization and a recent key-value residual buffer. It combines offline rank calibration, online subspace recycling, and custom Triton kernels. Experiments on six 1.3B model families show reduced memory traffic and faster decoding.
Critical strengths
- Addresses an important memory-bandwidth bottleneck.
- Combines a practical compression method with efficient GPU kernels.
- Evaluates six model families and reports meaningful decoding speedups.
Critical weaknesses
- Retrieval quality degrades substantially. For example, DeltaNet FDA accuracy drops from 38.69 to 11.26, and Mamba2 from 35.42 to 17.44. SVD-based initialization recovers much of the loss but adds complexity.
- Rank calibration is evaluated on limited WikiText-2 data, leaving robustness to other distributions unclear.
- Evaluation is limited to H100 GPUs, and end-to-end speedups vary considerably across models and batch sizes.
- Comparisons against alternative methods at matched memory and latency budgets are insufficient.
Questions for authors
Q1. Why does one-shot SVD initialization recover so much retrieval accuracy when the calibrated rank profile remains unchanged? Can the authors identify which aspect of online subspace recycling causes the largest loss? Q2. How much does the calibrated rank profile change when the calibration set, sequence length, or energy threshold changes? Can a profile calibrated on WikiText-2 transfer to substantially different inference distributions? Q3. How does the retained energy of FastRebase compare with the truncated-SVD optimum at each rebase? Is there a systematic relationship between this gap and downstream retrieval degradation? Q4. At what batch sizes and model configurations does FastRebase cease to provide a meaningful end-to-end benefit? Can the authors explain the particularly modest gains for DeltaNet? Q5. Is the one-shot SVD result intended as a diagnostic experiment or a viable deployment option? If the latter, what is its latency and memory cost, and does it offset the gains during realistic serving workloads?