acceptodds
Under review as a conference paper at ICLR 2027

LinQuant: Functional Invariances for Low-Precision Linear Recurrences

71 views

Abstract

Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures. However, despite their rapid emergence, low-precision inference remains much less developed for these models than for softmax attention. During prefill, chunkwise-parallel algorithms cast much of the recurrence as dense matrix multiplications, suggesting a natural path to low-precision execution. Simply quantizing these operations in their native form, however, can be fragile. In this work, we identify functional invariances that preserve the full-precision computation while making the operands more amenable to quantization. Building on this observation, we introduce _LinQuant_, a framework for low-precision recurrent prefill based on seven such invariances, including rotations, diagonal rescalings, and alternative placements of recurrent decay and write factors. We derive when these transformations remain valid across common recurrence families spanning linear, gated, and delta-rule updates, and show that they provide complementary degrees of freedom for improving low-precision accuracy. Finally, we translate this framework to production by augmenting SSDi8, an existing low-precision prefill pipeline for Mamba-2, with the applicable invariances. _LinQuant_ improves perplexity at matched latency on Mamba-2 1.3B and 8B and Nemotron-H 8B, demonstrating the practical utility of invariance-based quantization for recurrent models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, linear recurrence, SSM, efficiency, kernels

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_tnDwOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LinQuant, a framework for improving the accuracy of low-precision linear recurrent computations through functional invariances. The authors systematically identify and analyze seven mathematical transformations, including rotations, diagonal rescalings, and alternative allocations of decay and write factors, that preserve full-precision behavior while reducing quantization errors. Extensive experiments across various recurrent architectures demonstrate how these transformations interact with recurrence structures and quantization configurations. The authors further integrate selected transformations into the SSDi8 pipeline and evaluate them on Mamba-2 and Nemotron-H models. The results show improved quantization accuracy without increasing measured prefill latency, demonstrating the practical potential of invariance-based quantization for recurrent models.

Critical strengths
  1. Rigorous mathematical analysis: The authors provide a comprehensive and systematic characterization of quantization-related functional invariances, supported by rigorous mathematical analysis.

  2. Extensive empirical analysis: The paper conducts detailed numerical studies across mainstream recurrent architectures, providing valuable empirical insights through extensive experiments.

Critical weaknesses
  1. Limited runtime evaluation across recurrent architectures: While the authors thoroughly analyze the quantization benefits of individual invariances, the runtime overhead of combining multiple transformations across different recurrent architectures remains insufficiently evaluated.

  2. Limited production-scale validation. Although the paper covers a broad range of recurrent architectures, its practical deployment experiments primarily focus on Mamba-2 and the SSDi8 framework, with limited evaluation on widely used hybrid architectures.

  3. It focuses primarily on chunk-wise parallel prefill, lacking an analysis that extends to the decoding phase.

Questions for authors
  1. Applicability to hybrid architectures: Could the authors extend their evaluation to recent hybrid models, such as the Qwen3.5–3.8 series, to further demonstrate the generalizability and practical applicability of LinQuant?

  2. Interactions between invariances: Could different invariances interfere with each other when combined, potentially reducing their individual benefits? How does the proposed framework account for such interactions when selecting and composing transformations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer tnDw

We thank Reviewer tnDw for their thoughtful comments and positive review of our work. Our general response summarizes the added experiments and manuscript revisions. We address your specific concerns below.

W1: runtime of combined transformations. We agree that the original evaluation did not adequately establish this overhead. We added comparisons of transformation stacks against naive INT8 across all six recurrence families (Appendix D.3, Table 6), plus measurements that include the surrounding transformation passes at production-model shapes (Appendix G.5).

Rescalings and decay/write-gate allocations fold into existing scales and operand passes. Rotations require more care: we fused value transforms into convolution and output normalization, and kept the state in the rotated basis to avoid inverse transforms within the recurrence.

The table shows latency in ms for value convolution through output normalization at batch size 32 and 2048 tokens on an RTX A5000 (Table 28). The fused path adds value rotation for GDN and value rescaling plus rotation for KDA.

RecurrenceBF16 FLAINT8 without transformsINT8 with fused transforms
Gated DeltaNet19.69418.22518.543
Kimi Delta Attention35.92332.03332.307

Fusion reduces GDN's value-rotation overhead from 17% to 1.7%. KDA's combined value transforms add 0.9%. Both paths remain faster than BF16, reaching 1.22× and 1.29× speedups, respectively, at batch size 1. The six-family comparison also identifies configurations that remain costly: key rotations under diagonal decay and value rotations with canonical state storage.

W2 and Q1: pretrained hybrids, including Qwen3.5. We agree that our original production evaluation was too narrow, focusing on Mamba-2 through SSDi8. We therefore integrated our own INT8 kernels into Nemotron-H (Mamba-2), Qwen3.5-4B and 9B (Gated DeltaNet), and Kimi Linear 48B (Kimi Delta Attention). We quantized only their recurrences, with per-channel value scales. The selected combinations recovered most of the INT8 loss and improved on naive quantization on every calibration draw. This is shown in the table below, which gives WikiText-2 perplexity averaged over five calibration draws.

ModelFull precisionNaive INT8LinQuant INT8
Nemotron-H Base6.0356.1626.054
Nemotron-H Reasoning8.4298.7238.471
Qwen3.5-4B8.0568.1148.066
Qwen3.5-9B6.9837.0296.990
Kimi Linear 48B5.5195.5515.530

The best choices differ by model. Adding scalar-decay allocation to query–key rotation raises INT8 gap recovery from 79% to 86% on Nemotron-H Reasoning. Write-gate allocation alone recovers 23% and 24% on Qwen3.5-4B and 9B. We also added INT4 comparisons and longer-context SSDi8 tests: at 8k context, Nemotron-H Base mean RULER accuracy improves from 71.0% to 91.2%.

W3: decoding. We focus on accelerating recurrent prefill, as SageAttention does for attention and SSDi8 for Mamba-2. Chunkwise prefill exposes matrix products that benefit from integer tensor cores. Cached decoding instead updates the state one token at a time and requires separate kernel and quantization analysis. Extending LinQuant to accurate, efficient cached decoding is an important direction for future work.

Q2: interactions and selection. Transformations can help each other, address overlapping errors, or interfere. On Qwen3.5-9B, value rescaling and rotation do little alone, yet together recover most of the quantization loss. Rescaling balances the channel ranges before rotation mixes them. On the flip side, moving the write gate onto the keys also helps alone, but adds no benefit to this combination (Table 24). See the table below which shows the share of the naive INT8 perplexity gap recovered (%), with per-channel value scales.

TransformationGap recovered (%)
Value rescaling1
Value rotation-1
Rescaling then rotation83
Write gate on keys24
Rescaling + rotation + write-gate allocation80

In the controlled models, combinations usually outperform their strongest member, though the gains are smaller than the sum of their individual gains (Appendix E.2). For example, on Mamba-2 340M at INT6 with per-channel value scales, perplexity falls from 15.722 for the best individual transformation to 15.605 for their combination (Table 7).

We tested combinations in both studies. The zoo's main stacks combine transformations selected for their individual gains. We also directly evaluated rescaling with rotation on both the query–key and value axes. For the hybrids, we selected by held-out loss from a fixed set of individual and combined configurations (listed for INT8 in Table 24). A combination could therefore be selected even if its components did not help alone.

LGTM. It’s very interesting to see an analytical paper like this; I’ll keep my score as is.

AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their engagement during the discussion period and maintaining their positive review of the work.

Reviewer_9yKuOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

[Update after rebuttal]: The authors have adequately addressed my main concerns in their rebuttal. I have therefore increased my rating from 3 to 4.

This paper presents LinQuant, a systematic study of functional invariances in linear recurrences. It identifies seven transformations and analyzes their applicability across different recurrence structures. Through controlled experiments, the authors examine how these invariances interact with quantization settings and state storage choices. They further validate the findings on Mamba-2 and Nemotron-H models, improving quantization quality without increasing measured prefill latency.

Critical strengths
  1. Systematic analysis of functional invariances. The paper provides a thorough and precise characterization of functional invariances across different linear recurrences. The clear applicability conditions and supporting proofs offer a solid foundation for future quantization research and practical deployment.

  2. Recurrence-specific flexibility. Beyond rotations and rescaling, the paper explores decay and write factor absorption, the (σ,τ)(\sigma,\tau) allocation for delta updates, and the choice of state storage basis. It also shows that SSDi8's decay placement is one endpoint of a broader family of equivalent computations.

  3. Well-controlled experiments. The mixer zoo keeps the backbone and training setup fixed while varying recurrence structure, write gating, and QK normalization. The scale granularities reflect practical kernel constraints, helping explain when key and value transformations are most useful.

Critical weaknesses

1. Missing evaluation at longer context lengths. The quality evaluation is limited to sequences of 2048 tokens. In production, the recurrent state is updated in INT8, and Table 7 suggests that quantizing the carried state introduces additional error. This raises the question of whether the quantization gap grows over longer sequences. I would like to see the production configurations evaluated at increasing context lengths, ideally including Nemotron-H 8B. Perplexity or NLL measured across token positions, together with a long context retrieval benchmark such as RULER [1], would help establish whether the improvements remain effective beyond 2048 tokens.

2. Lack of floating point quantization experiments. The evaluation focuses on integer quantization, while FP8 is also an important format on recent GPUs. Its different numerical properties may change the relative benefits of the proposed invariances. For example, FlashAttention 3 has shown that Hadamard rotations can reduce FP8 quantization error, but it is unclear whether the same benefits extend to linear recurrences or whether rescaling and absorption remain equally useful. Since the paper also suggests applicability to floating point formats, adding E4M3 experiments for the main controlled comparisons would help establish the generality of the framework.

[1] Hsieh, Cheng-Ping, et al. "RULER: What's the real context size of your long-context language models?." arXiv preprint arXiv:2404.06654 (2024).

Questions for authors
  1. Delta rule invariances at scale. The (σ,τ)(\sigma,\tau) allocation and delta rule decay absorption are only evaluated in the controlled mixer zoo. Have the authors considered testing these transformations on a larger pretrained model such as Kimi Linear (48B total, 3B active)? Even an evaluation focused on quantizing its KDA recurrence would help show whether these invariances remain useful at production scale.

  2. Value rotation error. Table 21 shows that canonical storage removes the rotated-storage penalty, and Appendix E.1 attributes it to channel mixing under per-(head, value-channel) state scales. Section F.1 further suggests that RMSNorm amplifies this error in later layers with small outputs. However, Fig. 13 only shows a correlation, and all three configurations rise in those layers. Could the authors measure the error after RMSNorm and add canonical storage to Fig. 13, to test this explanation directly?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9yKu

We thank Reviewer 9yKu for their thoughtful review and positive outlook on our work. Our general response summarizes the added experiments and manuscript revisions. We address your specific concerns below.

W1: longer contexts and retrieval. The original evaluation used 2048-token windows. We tested Nemotron-H Base up to its full context length and measured NLL by position (Appendix F.3, Table 22). As you anticipated, the quantization gap grows along the sequence. In the second half of the sequence, LinQuant roughly halves SSDi8's NLL gap to FP16. Its absolute perplexity improvement also increases with context length. The following table shows Nemotron-H Base WikiText-2 perplexity by context length (Table 22).

TokensFP16SSDi8LinQuant
20486.0306.3366.175
40965.6786.0285.843
81925.4715.9205.686

We also added seven RULER tasks with matched prompts. The following table shows RULER mean accuracy (%) on Nemotron-H Base, averaged over the seven tasks. We see a substantial improvement with LinQuant.

ContextFP16W4A16SSDi8LinQuant
4k99.599.593.098.0
8k98.199.471.091.2

Notably, at 8k context, the multi-key task NIAH-MK3 improves from 33% to 95% accuracy.

W2: Lack of FP8. You are correct that this work focuses only on accurate and efficient integer recurrent prefill. But there is precedent for this: the original SageAttention paper develops only INT8 attention kernels, and SSDi8 likewise focuses on integer recurrent prefill. FlashAttention-3 instead develops an FP8 attention path and ignores integer space computation.

Thus, we consider our work the analog of SageAttention for linear recurrences, implementing INT8 and INT4 kernels and evaluating both accuracy and speed. We see FP8 as an important complementary direction: its numerical properties may change which invariances help, as your FlashAttention-3 example suggests. Although the invariances we study are not specific to integer formats, we have not evaluated their benefit under FP8. Studying E4M3 and developing efficient FP8 recurrence kernels is an exciting direction for future work, as noted in our conclusion.

Q1: delta-rule invariances at production scale. You are right that our original evaluation of these transformations was limited to the mixer zoo. We therefore tested write-gate allocation and decay absorption in Qwen3.5-4B and 9B (Gated DeltaNet) and Kimi Linear 48B (Kimi Delta Attention), quantizing only their recurrences with per-channel value scales (Appendix G).

On Qwen3.5, moving the write gate onto the keys alone recovered 23% and 24% of the naive INT8 perplexity gap to full precision on the 4B and 9B models, respectively (Table 24). At INT6, the selected Qwen3.5-4B combination included both decay and write-gate allocation onto the keys, alongside query–key rotation and value rescaling and rotation. This full combination recovered 56% of the gap (Table 25).

On Kimi Linear, balanced write-gate allocation modestly improved INT4 perplexity, although it had little effect at INT8 or INT6 (Appendix G.2). Decay absorption was less useful: in a separate fake-quantization control, it had only a marginal effect on perplexity at each tested precision (Appendix E.1). These experiments establish a benefit from write-gate allocation beyond the mixer zoo, while showing that the gains depend on the recurrence and precision.

In addition, we integrated our INT8 and INT4 kernels for GDN and KDA into these models. With fused value transformations, the INT8 paths achieved 1.22× and 1.29× speedups over BF16 for GDN and KDA, respectively, at batch size 1 on the RTX A5000, timing value convolution through output normalization (Table 28).

Q2: error after RMSNorm and canonical storage. We added both controls. We supplied each layer with full-precision inputs to exclude errors from earlier layers (Table 18, Figure 13). On Mamba-2 1.3B, rotation reduces mean relative error from 1.30% to 1.19% before RMSNorm but increases it from 1.28% to 1.69% afterward. Canonical storage lowers the latter to 1.58%, close to the 1.54% measured with a full-precision state. State quantization therefore contributes to the penalty but does not fully explain it.

Most of the excess error occurs in the first two thirds of each model, rather than in the final layers. Smaller outputs do not fully explain its distribution. We revised Appendix F.1 accordingly: normalization amplifies the error, but this is not specific to late layers. The canonical-storage perplexity comparison is in Table 17.

Thank you for the detailed rebuttal. You have adequately addressed my main concerns, and I find the additional clarifications and results convincing.

This is a well-executed and highly valuable piece of research. I have therefore decided to increase my rating from 3 to 4.

AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their engagement during the discussion period (which has improved our paper as a result) and increasing their rating of our work.

Reviewer_QUYzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies integer execution of the chunkwise prefill in linear recurrent mixers (linear attention, Mamba-2, GLA, DeltaNet, Gated DeltaNet, KDA). It lists seven transformations that leave the full-precision output unchanged but change what the quantizer sees: four changes of basis (QK rotation RQKR_{QK}, QK rescaling DQKD_{QK}, value rotation RVR_V, value rescaling DVD_V) and three absorptions that move decay or write-gate factors between operands. Table 1 and Appendix C say when each one applies, e.g. RQKR_{QK} needs to commute with the decay and DQKD_{QK} breaks the delta-rule erasure term. Experiments have two parts: a "mixer zoo" of 340M and 1.3B models trained from scratch, evaluated mostly at INT6 emulated in INT8 kernels under various scale granularities, and a production setting that adds RQKR_{QK} and DQKD_{QK} to SSDi8 for Mamba-2 (1.3B to 8B) and Nemotron-H 8B, improving WikiText-2 perplexity (e.g. 7.620 to 7.435 on Mamba-2 8B) at the same latency.

Critical strengths

The paper carefully works out which quantization-friendly rewrites survive the recurrent structure, across decay types, delta correction and cross-chunk state, and the kernels actually implement them. Table 1 plus Appendix C is a clear map for practitioners: RQKR_{QK} is only local under diagonal decay, DQKD_{QK} doesn't work with delta writes, value-side transforms apply everywhere, and per-channel value scales absorb DVD_V exactly. Some of the most useful results are negative: rotated state storage hurts (Fig. 3d, Fig. 8, Table 21), value-side transforms add nothing with per-channel value scales, and key-side decay works as well as SSDi8's value-side choice (Table 22). The controlled zoo with a shared backbone makes these comparisons clean.

Critical weaknesses
  1. The practical gain is small and comes from the least novel parts. In production LinQuant is SSDi8 plus a Hadamard QK rotation and a SmoothAttention-style QK rescaling, both known from QuaRot/SpinQuant and QServe. None of the absorptions, which are the most recurrence-specific contribution, is used in production. On Mamba-2 1.3B and 2.7B it closes only ~16% of the gap to FP16 (10.928 to 10.849 vs 10.420). It's about 50% on 8B and Nemotron-H. Adding DQKD_{QK} on top of RQKR_{QK} is within about one std over calibration sets and slightly worse on 2.7B (9.379 vs 9.383, Table 23). Zero-shot averages barely move (0.588 vs 0.588-0.590 on 1.3B, and DQKD_{QK} drops 8B from 0.703 to 0.699). There's no SSDi8 zero-shot row for Nemotron-H. So the "better perplexity at matched latency" claim is true but small, with weak downstream evidence.

  2. The settings where the novel parts help are mostly artificial. The big recoveries in Figs. 3-4 and Tables 10 and 12 are at INT6/INT4 emulated in INT8 kernels with per-tensor value scales. But Sec. 4.1 itself argues per-channel value scales are free, so why deploy per-tensor? In the realistic regime (INT8, per-channel values, Table 9) most differences are around 0.01 perplexity, and for 6 of 15 cells the selected stack is empty. The paper doesn't show any latency or memory benefit from coarse value scales or sub-INT8 recurrence. So for five of the seven invariances the evidence is diagnostic only, and that's the gap between "seven complementary degrees of freedom" and what's actually shown.

Questions for authors
  1. [Clarification] MambaQuant and Quamba2 already use Hadamard or variance-aligned rotations in Mamba. How is RQKR_{QK} different, and can you compare against or combine with them on the same W4A8 pipeline?
  2. [Clarification] Can you report seed or calibration-set variance for the zoo, and check the selected stacks still help when calibrated on another corpus like the Pile? Calibration, the 0.003-nat selection threshold and evaluation all use WikiText-2, and many INT8 differences in Tables 9 and 13 are 0.01-0.05 perplexity.
  3. [Clarification] Is there a real deployment setting where per-tensor value scales or INT6/INT4 recurrence beat per-channel INT8? If so please show the latency benefit.
  4. [Clarification] How do the zoo kernels compare to FP16/BF16 FLA kernels, not just naive INT8 (Table 4, Fig. 5b)? Especially for delta cells, where canonical RVR_V adds up to 14x overhead on A100.
  5. [Future work] Would learned rotations or rescalings (as in SpinQuant) widen the gap over SSDi8 on 1.3B and 2.7B, where gains are smallest?
5 comments
AuthorsOpenReview ↗

Response Part 1

We thank Reviewer QUYz for feedback that substantially improved our paper. Our general response comment above states the added experiments and manuscript revisions. We address W1 here. W2 and your questions follow.

Practical gains and recurrence-specific transformations. We agree that our original production experiments focused on query–key transformations in SSDi8. As such, we integrated our INT8 kernels into Nemotron-H, Qwen3.5, and Kimi Linear, quantizing only their recurrences with per-channel value scales. Every selected combination improves on all five calibration draws (Section 4.2, Table 2a).

The table below shows WT2 PPL.

ModelFPNaive INT8LinQuant INT8Selected invariances
Nemotron-H Base6.0356.1626.054RQK,DQK,RV,DVR_{QK}, D_{QK}, R_V, D_V
Nemotron-H Reasoning8.4298.7238.471RQKR_{QK} + decay on keys
Qwen3.5-4B8.0568.1148.066RV,DVR_V, D_V
Qwen3.5-9B6.9837.0296.990RV,DVR_V, D_V
Kimi Linear 48B5.5195.5515.530RV,DVR_V, D_V

RR and DD denote rotation and rescaling. Subscripts identify the query–key or value axes (Table 1).

The selected Nemotron-H Reasoning config now uses scalar-decay absorption, raising recovery of the naive INT8 perplexity gap from 79% with query–key rotation alone to 86% (Table 24).

Moreover, moving the write gate onto the keys alone helps both Qwen3.5 models. Below is the INT8 perplexity gap recovered (%) with per-channel value scales.

ModelNaive INT8Write gate on keys
Qwen3.5-4B0%23%
Qwen3.5-9B0%24%

At INT6, both allocations also appear in the selected Qwen3.5-4B combination: RQK,DV,RVR_{QK}, D_V, R_V, with decay and the write gate on the keys. This full recipe recovers 56% of the gap (Table 25). Together with the value-axis gains above, we hope these new results convince the reviewer that the practical benefits extend beyond standard query–key transformations.

Novelty and implementation of the transformations. We agree that rotations, diagonal rescalings, and butterfly Hadamard algorithms are established techniques. Our contribution is their recurrence-specific characterization (Table 1, Appendix C) and integration into INT8/INT4 chunkwise kernels, including GDN and KDA. Quamba2 supports Mamba/Mamba-2 and dequantizes operands for floating-point matrix products inside the recurrence. In contrast, we lower the matmuls inside the recurrence into low-precision integer arithmetic. Rotation also requires careful placement: convolution and nonlinearities prevent folding value rotations into projection weights. We fused them into value convolution and output normalization and kept the state in the rotated basis to avoid inverse transforms inside the recurrence. On GDN at batch size 32 on the RTX A5000, fusion reduces the added segment time relative to unrotated INT8 from 17% to 1.7% (Table 28). Diagonal rescaling is a simpler per-channel multiply fused into the same passes.

Interpreting the smaller SSDi8 gains. We agree that the gains on smaller Mamba-2 models are modest. SSDi8 quantizes both weights and activations. Our recurrence transformations target activation quantization. We added W4A16, which uses 4-bit weights and FP16 activations. On Mamba-2 1.3B and 2.7B, 58–59% of SSDi8's gap to FP16 remains even without activation quantization (Table 20). Using W4A16 as a reference, LinQuant closes 37% of the SSDi8–W4A16 gap on both smaller models, 77% on Mamba-2 8B, and 64% and 66% on Nemotron-H Base and Reasoning, without increasing prefill latency.

Query–key rescaling. We agree that it did not consistently help in the original SSDi8 experiments. As such, we compared the Nemotron-H Base combination with and without rescaling. Adding it increased the share of INT8 quantization loss recovered from 79% to 85%, helping on all five calibration draws. Separately, adding rescaling to rotation in SSDi8's computation layout improved perplexity in 11 of 12 comparisons (Table 26). This control simulated activation quantization with BF16 weights. This shows that rescaling can help, but its benefit depends on the configuration.

Downstream evidence. We agree that the short-context tasks did not establish a clear benefit. With the added SSDi8 baseline, Nemotron-H Base mean accuracy is 74.5% for SSDi8, 75.3% for LinQuant, and 75.0% for W4A16 (Table 20). Your concern motivated the new RULER retrieval tests below.

The table below shows RULER mean accuracy (%) on Nemotron-H Base, which shows that LinQuant substantially improves over SSDi8.

ContextFP16W4A16SSDi8LinQuant
4k99.599.593.098.0
8k98.199.471.091.2

At full context, the gains are especially clear on multi-key retrieval: accuracy on NIAH-MK3 rises from 33% to 95%. This provides downstream evidence that the original short-context averages did not.

AuthorsOpenReview ↗

Response Part 2

W2: do the invariances help in a practical quantization regime? We agree that the original evidence for several invariances was mainly diagnostic. As such, we studied the production-scale pretrained hybrids mentioned above using our INT8 kernels and per-channel value scales (Section 4.2). We find that six of the seven invariances help at least one pretrained model, alone or in combination (Table 24). Key-axis transformations and scalar-decay allocation help Nemotron-H. Value-axis combinations give the largest gains on Qwen3.5 and Kimi Linear, and write-gate allocation helps Qwen3.5.

Value-side transformations with per-channel scales. We hope the new combination experiments address this concern. Our original presentation did not clearly distinguish the per-channel and per-tensor settings. Those scales absorb rescaling alone, but rescaling can still improve a subsequent rotation by balancing channel ranges (Figure 3b, Appendix G.6). On Qwen3.5-9B, the combination recovers most of the INT8 loss even though neither transformation helps appreciably alone, as shown in the table below, which measures the share of the naive INT8 perplexity gap recovered (%), with per-channel value scales.

TransformationGap recovered (%)
Value rescaling1
Value rotation-1
Rescaling then rotation83

It also improves on naive quantization and rotation alone in all nine controlled models at INT4. At INT8, most controlled models have little error left to recover: for example, GDN 1.3B has perplexity 9.731 versus 9.727 in full precision. This is why it was important to extend our methods to the production hybrids (as you suggested).

Why retain INT6 and per-tensor value scales? We agree with your point that per-channel value scales are the practical choice because they add no kernel overhead. We revised Section 4.1 accordingly, clearly distinguishing the two regimes and emphasizing the per-channel results. We also used per-channel scales in the production experiments. We retained INT6 and per-tensor scales as diagnostic settings: they expose effects that are hard to see when naive INT8 with per-channel scales is already nearly lossless in the mixer zoo models. We also clarified that the controlled study used fake quantization and that INT6 has no dedicated kernel.

Q3: is there a real sub-INT8 runtime benefit? We implemented packed INT4 kernels for LA, Mamba-2, DeltaNet, Gated DeltaNet, and Kimi Delta Attention, using 4-bit integer tensor cores (Table 5). At the tested batch sizes of four and above on the RTX A5000, these kernels outperform INT8 by up to 1.24× while halving operand storage. The INT8/INT4 timing table in our response to Q4 below shows the runtime benefit at these model shapes (Appendix G.5, Table 27). The storage reduction applies to the operands, not the entire model.

INT4 WikiText-2 perplexity with only the recurrence quantized (Table 25).

ModelFull precisionNaive INT4LinQuant INT4
Nemotron-H Base6.0356,1847.367
Nemotron-H Reasoning8.42856,2979.785
Qwen3.5-4B8.05613.4039.907
Qwen3.5-9B6.98311.5399.575
Kimi Linear 48B5.5198.2888.033

These results show the tradeoff. The transformations substantially reduce INT4 error and prevent accuracy collapse on Nemotron-H, but do not recover INT8-level accuracy. INT4 therefore offers further runtime savings at an accuracy cost. We do not claim comparable speedups for INT6 or per-tensor value scales.

AuthorsOpenReview ↗

Response Part 3

Q1: how does RQKR_{QK} differ from prior rotations, and can they be combined? RQKR_{QK} rotates the recurrence's internal state dimension, not the residual stream. In Mamba notation, it explicitly rotates BB (keys), CC (queries), and thus implicitly the state along that dimension. Quamba2's Hadamards instead rotate residual-stream features and the input to the output projection. MambaQuant is closer: it rotates the computed state and CC for readout. In Mamba-2, scalar decay lets us rotate BB as well, so the state is built in the rotated basis. This changes the operands quantized in the state write and local-score computation, beyond the final readout (Appendix A).

We combined RQKR_{QK} with the projection rotations in SSDi8's W4A8 pipeline. We compared the same SSDi8 configuration with and without RQKR_{QK}, retaining the projection rotations, static per-channel value scales, and value-side decay in both. On Mamba-2 8B, adding RQKR_{QK} reduced WikiText-2 perplexity from 7.620 to 7.456, averaged over four calibration sets (Appendix F.1). This shows that rotating BB and CC helps beyond the projection rotations already present.

Why use SSDi8 as the external baseline? MambaQuant's released method targets Mamba-1, not our Mamba-2 checkpoints. Quamba2 supports Mamba-2, but dequantizes operands for floating-point SSD matrix products. We used SSDi8 because its integer recurrence computations match our target and let us test the added rotation within one pipeline. Comparing against full Quamba2 would also vary the quantization scheme and kernels. Even so, we note that SSDi8, like MambaQuant and Quamba2, restricts its scope to Mamba models only, whereas LinQuant encompasses the full set of relevant linear recurrences.

Q2: calibration and selection. To test sensitivity to calibration data, we recalibrated the controlled models on the Pile without changing the combinations selected on WikiText-2. The selected combinations still improve on naive quantization in 36 of the 37 INT8 model-and-quantization settings tested, and all 58 INT6 settings (Appendix B.1). The exception is a small rescaling gain that becomes essentially neutral.

Repeating calibration with a second draw changed perplexity little. Three random Hadamard sign patterns also gave similar results, except that all improved the two unnormalized LA models relative to the original plain Hadamard. We clarified that the controlled study used disjoint training windows for calibration and selection, with the test split reserved for reporting. The pretrained experiments used the Pile for calibration and selection, with five independent calibration draws and paired naive runs. Every selected INT8 combination improves on every draw (Table 25).

Q4: FLA comparisons and the reported A100 overhead. We added BF16 FLA comparisons for all six families in the mixer zoo (Table 4). For the three recurrences in the large-scale hybrids (Mamba-2, GDN, KDA), we timed the same segment from projection outputs through the recurrence, including input preparation and quantization. Here is the latency (ms) at batch size 1 and 2048 tokens on an RTX A5000. The parentheses give speedup over BF16 FLA (Table 27).

RecurrenceFLA BF16LinQuant INT8LinQuant INT4
Mamba-21.1920.588 (2.03×)0.573 (2.08×)
Gated DeltaNet1.1410.718 (1.59×)0.707 (1.61×)
Kimi Delta Attention1.6341.219 (1.34×)1.196 (1.37×)

INT8 accelerates all three mixer families, and packed INT4 reduces latency further (Figure 5a,b).

Both integer matrix products and fused input preparation contribute to these gains. For Gated DeltaNet at batch size 1, fusing preparation alone reduces the INT8 segment time from 1.141 to 0.775 ms (Table 27). The delta rule still needs a floating-point Gram matrix and triangular solve to prepare each chunk. This limits the overall speedup.

We also revisited the reported 14× A100 overhead. With autotuned launches, canonical storage took at most 3.0× the naive INT8 runtime on A100 across the six families and batch sizes 1, 4, and 8 (Figure 5c, Table 6). Thank you for prompting this check. Autotuning substantially reduces the reported overhead, although the extra matrix products remain costly. Keeping the state in the rotated basis avoids these inverse transforms.

Q5: learned transformations. Our framework permits learned rotations and rescalings: the identities hold for any transformations satisfying Table 1's conditions. We used Hadamard rotations and calibrated rescalings because they are inexpensive to fuse. We have not tested whether learning them increases the gains. Learning these transformations is an interesting direction for future work.

Overall, we hope the revised manuscript and new results address your concerns, and would appreciate your reconsideration of the score in light of this evidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all the reviewers for their thoughtful and thorough feedback. Your comments have prompted us to expand the experimental evaluation and revise the presentation to make the findings and their implications clearer. Below, we summarize the new results and manuscript revisions, highlighted in blue in the paper. All page, table, figure, and equation references refer to the revised manuscript unless stated otherwise.

1. Integer prefill in production-scale models. All three reviewers QUYz, 9yKu, and tnDw asked for stronger production-scale validation. As such, we integrated our existing INT8 kernels into Nemotron-H, Qwen3.5, and Kimi Linear for their Mamba-2, Gated DeltaNet, and Kimi Delta Attention recurrences, respectively. We also implemented packed INT4 kernels to demonstrate the benefit of lower-precision execution. Together, these results more clearly demonstrate the practical value of our invariances and kernels: they recover 65–86% of the naive INT8 perplexity gap and accelerate the timed mixer segments by up to 2.1× over BF16 FLA (Section 4.2).

2. Complementary SSDi8 results and additional controls. Reviewer QUYz questioned the downstream gains of the invariances beyond the metrics we reported, while Reviewer 9yKu asked whether they persist at longer contexts. In response, we extended the SSDi8 evaluation to longer-context perplexity and retrieval, and added the requested baselines and controls. At 8k context on Nemotron-H Base, LinQuant raises mean RULER retrieval accuracy from SSDi8’s 71.0% to 91.2%.

3. Scale granularity and combinations of invariances. Reviewer QUYz rightly emphasized that per-channel value scales are the practical choice because they add no kernel overhead. We therefore reorganized Section 4.1 to emphasize the results in the per-channel setting and clarify which invariances help in each scale regime. We retained the per-tensor results to show what changes when value channels share a scale. Separately, reviewer tnDw asked how invariances interact. To address this, we showed that value rescaling can improve a subsequent rotation by balancing channel ranges, even though per-channel scales absorb rescaling alone.

We address each reviewer's specific concerns in the corresponding response thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.