- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces LinQuant, a framework for improving the accuracy of low-precision linear recurrent computations through functional invariances. The authors systematically identify and analyze seven mathematical transformations, including rotations, diagonal rescalings, and alternative allocations of decay and write factors, that preserve full-precision behavior while reducing quantization errors. Extensive experiments across various recurrent architectures demonstrate how these transformations interact with recurrence structures and quantization configurations. The authors further integrate selected transformations into the SSDi8 pipeline and evaluate them on Mamba-2 and Nemotron-H models. The results show improved quantization accuracy without increasing measured prefill latency, demonstrating the practical potential of invariance-based quantization for recurrent models.
Critical strengths
-
Rigorous mathematical analysis: The authors provide a comprehensive and systematic characterization of quantization-related functional invariances, supported by rigorous mathematical analysis.
-
Extensive empirical analysis: The paper conducts detailed numerical studies across mainstream recurrent architectures, providing valuable empirical insights through extensive experiments.
Critical weaknesses
-
Limited runtime evaluation across recurrent architectures: While the authors thoroughly analyze the quantization benefits of individual invariances, the runtime overhead of combining multiple transformations across different recurrent architectures remains insufficiently evaluated.
-
Limited production-scale validation. Although the paper covers a broad range of recurrent architectures, its practical deployment experiments primarily focus on Mamba-2 and the SSDi8 framework, with limited evaluation on widely used hybrid architectures.
-
It focuses primarily on chunk-wise parallel prefill, lacking an analysis that extends to the decoding phase.
Questions for authors
-
Applicability to hybrid architectures: Could the authors extend their evaluation to recent hybrid models, such as the Qwen3.5–3.8 series, to further demonstrate the generalizability and practical applicability of LinQuant?
-
Interactions between invariances: Could different invariances interfere with each other when combined, potentially reducing their individual benefits? How does the proposed framework account for such interactions when selecting and composing transformations?
3 comments
LGTM. It’s very interesting to see an analytical paper like this; I’ll keep my score as is.
Response to reviewer
We thank the reviewer for their engagement during the discussion period and maintaining their positive review of the work.
Response to Reviewer tnDw
We thank Reviewer tnDw for their thoughtful comments and positive review of our work. Our general response summarizes the added experiments and manuscript revisions. We address your specific concerns below.
W1: runtime of combined transformations. We agree that the original evaluation did not adequately establish this overhead. We added comparisons of transformation stacks against naive INT8 across all six recurrence families (Appendix D.3, Table 6), plus measurements that include the surrounding transformation passes at production-model shapes (Appendix G.5).
Rescalings and decay/write-gate allocations fold into existing scales and operand passes. Rotations require more care: we fused value transforms into convolution and output normalization, and kept the state in the rotated basis to avoid inverse transforms within the recurrence.
The table shows latency in ms for value convolution through output normalization at batch size 32 and 2048 tokens on an RTX A5000 (Table 28). The fused path adds value rotation for GDN and value rescaling plus rotation for KDA.
Fusion reduces GDN's value-rotation overhead from 17% to 1.7%. KDA's combined value transforms add 0.9%. Both paths remain faster than BF16, reaching 1.22× and 1.29× speedups, respectively, at batch size 1. The six-family comparison also identifies configurations that remain costly: key rotations under diagonal decay and value rotations with canonical state storage.
W2 and Q1: pretrained hybrids, including Qwen3.5. We agree that our original production evaluation was too narrow, focusing on Mamba-2 through SSDi8. We therefore integrated our own INT8 kernels into Nemotron-H (Mamba-2), Qwen3.5-4B and 9B (Gated DeltaNet), and Kimi Linear 48B (Kimi Delta Attention). We quantized only their recurrences, with per-channel value scales. The selected combinations recovered most of the INT8 loss and improved on naive quantization on every calibration draw. This is shown in the table below, which gives WikiText-2 perplexity averaged over five calibration draws.
The best choices differ by model. Adding scalar-decay allocation to query–key rotation raises INT8 gap recovery from 79% to 86% on Nemotron-H Reasoning. Write-gate allocation alone recovers 23% and 24% on Qwen3.5-4B and 9B. We also added INT4 comparisons and longer-context SSDi8 tests: at 8k context, Nemotron-H Base mean RULER accuracy improves from 71.0% to 91.2%.
W3: decoding. We focus on accelerating recurrent prefill, as SageAttention does for attention and SSDi8 for Mamba-2. Chunkwise prefill exposes matrix products that benefit from integer tensor cores. Cached decoding instead updates the state one token at a time and requires separate kernel and quantization analysis. Extending LinQuant to accurate, efficient cached decoding is an important direction for future work.
Q2: interactions and selection. Transformations can help each other, address overlapping errors, or interfere. On Qwen3.5-9B, value rescaling and rotation do little alone, yet together recover most of the quantization loss. Rescaling balances the channel ranges before rotation mixes them. On the flip side, moving the write gate onto the keys also helps alone, but adds no benefit to this combination (Table 24). See the table below which shows the share of the naive INT8 perplexity gap recovered (%), with per-channel value scales.
In the controlled models, combinations usually outperform their strongest member, though the gains are smaller than the sum of their individual gains (Appendix E.2). For example, on Mamba-2 340M at INT6 with per-channel value scales, perplexity falls from 15.722 for the best individual transformation to 15.605 for their combination (Table 7).
We tested combinations in both studies. The zoo's main stacks combine transformations selected for their individual gains. We also directly evaluated rescaling with rotation on both the query–key and value axes. For the hybrids, we selected by held-out loss from a fixed set of individual and combined configurations (listed for INT8 in Table 24). A combination could therefore be selected even if its components did not help alone.