acceptodds
Under review as a conference paper at ICLR 2027

QReS: Quantizing Recurrent Language Models Is Easier Than You Think

74 views

Abstract

Linear recurrences are becoming an increasingly common building block in large language models, replacing the growing KV cache of softmax attention with a fixed-size state that enables efficient decoding. However, the quantization stack has largely been developed around softmax Transformers, whose numerical properties and inference state differ from those of recurrent models, motivating quantization schemes tailored to recurrent structure. We study how quantizing linear recurrences differs from quantizing softmax attention along two axes. First, extra-state quantization concerns weights and activations outside the recurrent state, whose distributions are shaped indirectly by the sequence mixer during training. In controlled 1.3B models, we find that decayed recurrences develop much weaker residual outlier channels than softmax attention and are therefore easier to quantize without incoherence processing. Second, intra-state quantization concerns the recurrent state itself during decode, where repeated updates and recurrent decay create a different numerical problem from KV cache quantization. We use these insights to develop QReS, a pipeline that combines value-axis scale grouping, decay-based mixed-precision allocation and stochastic rounding for recurrent state quantization. Across Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, QReS supports robust 8-bit state quantization and lower effective precision on shorter decode horizons. We implement QReS in vLLM and demonstrate end-to-end decode speedups.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language modeling, linear attention, quantization, efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6H3WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies post-training quantization of linear recurrent language models, distinguishing weight and activation quantization outside the recurrent state from quantization of the state repeatedly updated during decoding. Using a controlled suite of 1.3B models with a shared backbone and training recipe, the authors find that recurrent mixers with decay develop weaker residual outlier channels and exhibit smaller relative perplexity degradation under unrotated W4A4 quantization. They then analyze state scale grouping, the attenuation of inherited errors through decay, and error accumulation under repeated rounding, leading to QReS, which combines value-axis scale grouping, stochastic rounding, and decay-guided mixed-precision allocation. Generation evaluations on Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B show that INT8 states preserve most of the accuracy, while the feasibility of lower effective precision depends on the task. A vLLM implementation further demonstrates decoding benefits from state quantization that become more pronounced at higher concurrency.

Critical strengths

S1. The controlled architectural comparisons provide informative evidence about quantization behavior in recurrent models. The authors use a shared 1.3B backbone and quantization procedure while systematically varying decay, write gating, and delta correction, reducing the confounding factors associated with comparing independently developed models. Tables 2, 8, and 9 show a consistent relationship between decay and robustness to unrotated W4A4 quantization, while the channel interventions in Tables 3 and 10 provide further evidence that different architectures rely differently on dominant residual channels. This experimental design helps explain how architectural choices influence quantization requirements and constitutes a valuable empirical contribution. S2. The state-quantization approach is validated through actual generation across architectures, with particularly compelling evidence for stochastic rounding. The paper evaluates three large hybrid models on tasks involving short generation, long reasoning, and long inputs. Table 17 shows that stochastic rounding can substantially improve long-reasoning performance under the same INT8 state precision, grouping, and grid settings. For example, Qwen’s AIME accuracy increases from 54.0% with round-to-nearest to 79.5% with stochastic rounding, approaching the 82.3% FP32-state baseline. These results establish practical generation benefits beyond improvements in short-sequence proxy metrics. S3. The principal design choices are supported by substantial ablations, including informative failure cases. The paper compares scale-grouping axes, rounding rules, symmetric and asymmetric grids, rotations, and integer versus floating-point formats. Table 12 additionally includes a random-allocation control with the same precision-allocation proportions, supporting the usefulness of decay information for precision assignment. The authors also report negative results, such as the severe degradation caused by stochastic rounding with key-axis grouping in some models. These findings help establish the method’s operating conditions rather than presenting the final recipe as universally effective. S4. The implementation demonstrates practical decoding gains and includes controls that isolate the contribution of state quantization. The authors implement decoding kernels with fused state quantization and conduct model-level evaluations in vLLM. The appendix reports FP32/INT8 state comparisons at the same weight precision, absolute TPOT measurements, and controls with a fixed attention block size. For example, Table 21 shows that W4A16 Nemotron-H’s TPOT decreases from 9.86 ms to 7.50 ms at concurrency 64. These results demonstrate that state quantization can benefit full-model decoding rather than only isolated kernels.

Critical weaknesses

W1. The incremental benefit over existing state-quantization approaches is not sufficiently established through direct comparisons. The paper discusses related approaches in Q-Mamba, Quamba2, and Nemotron, but the experiments primarily compare selected grouping, rounding, and format variants. A direct comparison against a closely related state-quantization approach under the same model, generation protocol, and resource budget is missing. Since low-precision state storage and stochastic rounding have precedents, this gap limits the assessment of QReS’s incremental contribution and accuracy–efficiency advantages. Including a representative existing approach in at least one comparable Mamba-family setting would substantially strengthen the positioning of the contribution. W2. The low-effective-precision and decay-allocation results do not yet establish a robust configuration-selection rule. The random-allocation control in Table 12 is valuable, but it primarily uses the median proxy loss on short WikiText-2 trajectories. The generation experiments show accuracy curves under decay-based allocation without sufficiently comparing alternative allocations at the same budget. The AIME results in Table 18 use pass@1 on 60 problems and do not report uncertainty, leaving the stability of the minimum bit width satisfying the 5% relative accuracy-loss threshold unclear. It also remains necessary to clarify whether model-specific grid and bit-width choices were made independently of the final test data. W3. Reproducibility details and evidence for the generality of the extra-state findings remain limited. The controlled experiments with a shared backbone are an important strength, but Appendix B.1 does not fully specify the training corpus, optimizer and learning-rate schedule, training sequence length, or random seeds, and variability across repeated training runs is not reported. The relationship between residual outlier structure and quantization robustness is currently established primarily at one scale under a shared training recipe. Providing complete reproduction details and distinguishing observations in this controlled setting from broader architectural regularities would strengthen confidence in this central empirical finding.

Questions for authors

1.Could you compare QReS against an existing state-quantization approach or an FP16+SR configuration on at least one comparable Mamba-family model, reporting accuracy, actual state storage, and decoding latency under the same generation protocol? If such approaches cannot be directly adapted, please explain the specific obstacles and identify which existing results isolate QReS’s additional benefits beyond previously established components. 2.What data are used to determine the decay ranking, grid type, and minimum bit width, and are these choices separated from the final test sets? Could you provide paired uncertainty estimates for generation results near the bit-width thresholds and evaluate at least one alternative precision allocation at the same storage budget, to establish whether the advantage of decay ranking transfers from the proxy loss to actual generation? 3.Please provide the complete training data and training configuration for the mixer zoo, and clarify whether data order and sources of randomness are matched across architectures. Does the relationship between residual outlier channels and W4A4 degradation remain stable across available checkpoints or repeated experiments?

1 comment

We thank Reviewer 6H3W for their thoughtful review. We appreciate their recognition of our controlled mixer-zoo comparisons and “particularly compelling evidence for stochastic rounding”, and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version’s numbering.

W1/Q1: Comparisons and contribution.

Both our analysis and its instantiation, QReS, apply to a general family of linear recurrences. Existing methods, such as Q-Mamba and Quamba2, focus on Mamba-2. We therefore compare the Mamba-2 instantiation of QReS with these baselines. We have clarified this positioning in Section 2.2.

The table below compares QReS on Nemotron-H 8B with Q-Mamba, Quamba2, and the FP16 + SR recipe used in Nemotron 3 Super. We have added this comparison to Appendix H (Table 25).

MethodBitsAIME 64k avg@32 (%)AIME 32k avg@32 (%)GPQA 64k avg@32 (%)AIME Gen. length
Q-Mamba850.446.452.725.8k
Quamba2825.322.844.745.7k
FP16 + SR1650.850.854.012.1k
QReS849.249.253.612.4k

The table above shows that QReS approaches the accuracy of Nemotron 3 Super's FP16 + SR recipe at half the state precision. Quamba2 loses substantial accuracy and produces much longer answers.

Q-Mamba generates roughly twice as many tokens as QReS on average. At a 64k-token cap, its accuracy is comparable to QReS. Reducing the decode cap to 32k, however, lowers Q-Mamba's AIME accuracy, while QReS' remains virtually unchanged.

Decode latency. Since no public implementation was available, we re-implemented Q-Mamba and added a key-axis-only control. The table below reports Mamba-2 INT8 kernel speedups over FP32 on H100 for QReS, "QReS with key-axis scales", and our implementation of Q-Mamba.

BatchQReSQReS with key-axis scalesQ-Mamba
11.22×0.83×0.74×
21.43×1.08×0.98×
41.61×1.20×1.07×
322.74×2.02×1.54×
1283.03×2.34×1.65×

The table above shows that, in our implementation, key-axis scaling already reduces the speedup, with Q-Mamba's combined row and column scales adding further overhead. This supports the point in Appendix G.1: value-axis scales are local to decode kernel tiles, whereas key-axis scales require reduction across tiles. We have added this comparison to Appendix H (Figure 16).

Finally, QReS INT8 reduces measured state storage per layer and sequence from 4.19 MB (FP32) to 1.08 MB, including scales.

W2/Q2: Uncertainty and configuration selection.

To better estimate the minimum precision that meets the 5% relative accuracy-loss rule, we reran the critical bit widths with avg@16. The table below reports FP32 avg@32 accuracy (%) and quantized avg@16 accuracy with standard errors across seeds.

ModelTaskFP326 bits6.5 bits7 bits7.5 bits8 bits
Qwen-3.5AIME82.3—75.4±1.080.1±1.080.8±0.7—
GPQA80.168.6±0.578.0±0.579.7±0.4——
Nemotron-HAIME51.2——46.4±0.849.7±1.149.4±1.1
GPQA54.1—50.6±0.453.3±0.652.5±0.6—

In the avg@16 evals, the minimum precision meeting the 5% criterion on AIME is lowered from 7.5 to 7 bits for Qwen and from 8 to 7.5 bits for Nemotron-H. The rest is the same. Overall, our promotion findings stay consistent. We thank Reviewer 6H3W for pointing out the missing error bars. We have added the uncertainty results and updated promotion plot in Appendix F.3.

Calibration and configuration selection. All calibration is done using held-out wikitext-2 samples. We chose the model-specific grids for the main evaluations and promotion sweeps from the AIME comparison in Table 18, then kept them fixed for the other benchmarks.

Alternative allocation. To test whether the decay ranking improves generation, we compare it with one random assignment, giving both the same mix of half INT6 and half INT4 heads. The table below shows that decay-based allocation retains substantially more GPQA accuracy in both models.

ModelDecay avg@16 (%)Random avg@16 (%)
Qwen-3.560.5±0.64.8±0.4
Nemotron-H37.2±0.716.9±0.6

W3/Q3: Reproducibility and checkpoints.

Training setup. We thank Reviewer 6H3W for pointing this out. We have added the training details in Appendix B.1. All models used the same seed with matched data order and randomness.

Extra State. The table below reports the kurtosis–W4A4 Spearman correlation across all models at the intermediate and final checkpoints, confirming a consistent relationship. Training ends at 190k steps.

Training stepSpearman correlation
100k0.855
190k0.864

We again thank Reviewer 6H3W for their thoughtful review and hope to have addressed their questions.

Reviewer_sS36OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes QReS, a recurrent-state quantization framework for efficient inference of linear-recurrent large language models, and evaluates QReS across several linear-recurrent architectures. Beyond algorithmic accuracy, the authors integrate QReS into vLLM and evaluate end-to-end decoding efficiency on H100 GPUs.

Critical strengths

QReS is evaluated beyond isolated accuracy metrics and is actually integrated into vLLM for end-to-end inference. The evaluation on H100 further strengthens the practical relevance of the proposed method by connecting the proposed state-quantization scheme to real serving throughput rather than only reporting algorithmic compression or perplexity improvements.

Critical weaknesses

While the proposed combination of value-axis grouping, stochastic rounding, and decay-based precision allocation is empirically effective, the paper could more clearly distinguish the novelty of these individual components from existing recurrent-state quantization techniques.

Questions for authors

For large hybrid models, how does QReS compare with existing recurrent-state quantization methods, particularly Q-Mamba’s decoupled-scale quantization and Quamba2’s INT8 state quantization. The proposed INT5 allocation strategy ranks recurrent heads according to their average decay measured during calibration. But the robustness of the resulting ranking with respect to the calibration data distribution is not sufficiently investigated?

1 comment

We thank Reviewer sS36 for their thoughtful review, and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version’s numbering.

W1: Contribution and positioning.

We thank Reviewer sS36 for the question about existing methods. Both our analysis and its instantiation, QReS, apply to a general family of linear recurrences. Existing methods, such as Q-Mamba and Quamba2, focus on Mamba-2. We therefore compare the Mamba-2 instantiation of QReS with these baselines. We have clarified this positioning in Section 2.2.

Low-precision state storage and SR already have precedents in Mamba models. Our contribution is the systematic analysis, with theory and controlled ablations, showing that grouping and rounding must be chosen together at low precision. In particular, SR can fail with key-axis grouping, while value-axis grouping supports accurate INT8 states. We further use recurrent decay to allocate precision across heads or channels, a component absent from the compared methods.

Q1: Existing methods.

The table below compares QReS on Nemotron-H 8B with Q-Mamba, Quamba2, and the FP16 + SR recipe used in Nemotron 3 Super. We have added this comparison to Appendix H (Table 25).

MethodBitsAIME 64k avg@32 (%)AIME 32k avg@32 (%)GPQA 64k avg@32 (%)AIME Gen. length
Q-Mamba850.446.452.725.8k
Quamba2825.322.844.745.7k
FP16 + SR1650.850.854.012.1k
QReS849.249.253.612.4k

The table above shows that QReS approaches the accuracy of Nemotron 3 Super's FP16 + SR recipe at half the state precision. Quamba2 loses substantial accuracy on both reasoning tasks and produces much longer answers.

Q-Mamba generates roughly twice as many tokens as QReS on average. At a 64k-token decode cap, its accuracy is comparable to QReS. Reducing the decode cap to 32k, however, lowers Q-Mamba's AIME accuracy, while QReS' remains virtually unchanged.

Decode latency. Since no public implementation was available, we re-implemented Q-Mamba and added a key-axis-only control. The table below reports Mamba-2 INT8 kernel speedups over FP32 on H100 for QReS, "QReS with key-axis scales", and our implementation of Q-Mamba.

BatchQReSQReS with key-axis scalesQ-Mamba
11.22×1.22\times0.83×0.83\times0.74×0.74\times
21.43×1.43\times1.08×1.08\times0.98×0.98\times
41.61×1.61\times1.20×1.20\times1.07×1.07\times
82.09×2.09\times1.57×1.57\times1.36×1.36\times
322.74×2.74\times2.02×2.02\times1.54×1.54\times
1283.03×3.03\times2.34×2.34\times1.65×1.65\times

The table above shows that, in our implementation, key-axis scaling already reduces the speedup, with Q-Mamba's combined row and column scales adding further overhead. This supports the point in Appendix G.1: value-axis scales are local to decode kernel tiles, whereas key-axis scales require reduction across tiles. We have added this comparison to Appendix H (Figure 16).

Q2: Calibration robustness.

To address calibration robustness, we recomputed decay rankings on 512 training tokens each from C4 (allenai) and MATH (eleutherai). The table below reports promoted-half overlap with WikiText-2 calibration, showing that the promoted heads remain very stable across these tested domains.

ModelC4 (%)MATH (%)
Qwen-3.597.496.6
Nemotron-H97.796.1

We thank Reviewer sS36 for this question, which strengthens the paper. We have added these results in Appendix E.2 (Table 14).

We again thank Reviewer sS36 for their thoughtful review and hope to have addressed their questions.

Reviewer_qFgBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies quantization of recurrent language models from a unified perspective, covering both extra-state weight/activation quantization and intra-state recurrent-state quantization. Through controlled experiments across different sequence mixers, the authors identify differences in activation outliers and quantization robustness. They further propose QReS, combining value-axis scale grouping, decay-based mixed precision, and stochastic rounding. Experiments on modern hybrid LLMs demonstrate promising accuracy preservation and decoding speedups.

Critical strengths
  1. Unified analysis of recurrent-model quantization. Rather than focusing on a specific recurrent architecture, the paper systematically investigates how different sequence mixers affect quantization robustness. The controlled experiments and theoretical analysis provide useful insights into activation outliers, error accumulation, and recurrent decay.

  2. Practical validation across modern hybrid LLMs. The experiments cover multiple recurrent architectures and large-scale models. The vLLM integration further demonstrates that recurrent-state quantization can deliver practical decoding speedups with limited accuracy degradation.

Critical weaknesses
  1. Insufficient system-level evaluation of mixed precision. While decay-based mixed precision achieves promising accuracy at 5–6 effective bits, the serving experiments primarily focus on INT8 states. It remains unclear whether these configurations provide practical memory savings and decoding speedups after accounting for packing and conversion overhead.

  2. Lack of direct comparisons with existing methods. The paper discusses prior recurrent-state quantization methods, such as Q-Mamba and Nemotron 3 Super, but lacks direct comparisons under matched settings. This makes it difficult to assess the practical advantages of QReS over existing approaches.

Questions for authors
  1. Can the authors provide actual GPU memory usage and end-to-end decoding performance for the proposed INT5/INT6 mixed-precision configurations?

  2. How does QReS compare with existing recurrent-state quantization methods under matched accuracy, memory, and throughput settings?

1 comment

We thank Reviewer qFgB for their thoughtful review. We appreciate their recognition of our unified analysis and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version’s numbering.

W1/Q1: Mixed precision.

Our serving experiments run on H100 GPUs, which lack native microformats below eight bits. This means INT4/INT6 kernels require explicit packing and unpacking. In our experiments, this overhead offsets the benefit of smaller memory transfers, so these kernels offer no meaningful speed advantage over INT8. This is why we use INT8 in the serving evaluation.

However, our accuracy results show that decay-based mixed precision can reduce state precision further.

At the same time, these low-bit configurations do still reduce state memory. The table below reports measured packed state storage per layer and sequence, including scales.

State layoutFP32INT8INT65 bit
Gated DeltaNet2.10 MB0.54 MB0.41 MB0.34 MB
Mamba-24.19 MB1.08 MB0.82 MB0.69 MB

Making these lower precisions faster during decode is an interesting direction for future work, including on newer hardware with native low-precision microformats.

W2/Q2: Matched baseline comparison.

We thank Reviewer qFgB for the question about existing methods. Both our analysis and its instantiation, QReS, apply to a general family of linear recurrences. Existing methods, such as Q-Mamba and Quamba2, focus on Mamba-2. We therefore compare the Mamba-2 instantiation of QReS with these baselines. We have clarified this positioning in Section 2.2.

The table below compares QReS on Nemotron-H 8B with Q-Mamba, Quamba2, and the FP16 + SR recipe used in Nemotron 3 Super. We have added this comparison to Appendix H (Table 25).

MethodBitsAIME 64k avg@32 (%)AIME 32k avg@32 (%)GPQA 64k avg@32 (%)AIME Gen. length
Q-Mamba850.446.452.725.8k
Quamba2825.322.844.745.7k
FP16 + SR1650.850.854.012.1k
QReS849.249.253.612.4k

The table above shows that QReS approaches the accuracy of Nemotron 3 Super's FP16 + SR recipe at half the state precision. Quamba2 loses substantial accuracy on both reasoning tasks and produces much longer answers.

Q-Mamba generates roughly twice as many tokens as QReS on average. At a 64k-token decode cap, its accuracy is comparable to QReS. Reducing the decode cap to 32k, however, lowers Q-Mamba's AIME accuracy, while QReS' remains virtually unchanged.

Decode latency. Since no public implementation was available, we re-implemented Q-Mamba and added a key-axis-only control. The table below reports Mamba-2 INT8 kernel speedups over FP32 on H100 for QReS, "QReS with key-axis scales", and our implementation of Q-Mamba.

BatchQReSQReS with key-axis scalesQ-Mamba
11.22×1.22\times0.83×0.83\times0.74×0.74\times
21.43×1.43\times1.08×1.08\times0.98×0.98\times
41.61×1.61\times1.20×1.20\times1.07×1.07\times
82.09×2.09\times1.57×1.57\times1.36×1.36\times
322.74×2.74\times2.02×2.02\times1.54×1.54\times
1283.03×3.03\times2.34×2.34\times1.65×1.65\times

The table above shows that, in our implementation, key-axis scaling already reduces the speedup, with Q-Mamba's combined row and column scales adding further overhead. This supports the point in Appendix G.1: value-axis scales are local to decode kernel tiles, whereas key-axis scales require reduction across tiles. We have added this comparison to Appendix H (Figure 16).

We again thank Reviewer qFgB for their thoughtful review and hope to have addressed their questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.