- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies post-training quantization of linear recurrent language models, distinguishing weight and activation quantization outside the recurrent state from quantization of the state repeatedly updated during decoding. Using a controlled suite of 1.3B models with a shared backbone and training recipe, the authors find that recurrent mixers with decay develop weaker residual outlier channels and exhibit smaller relative perplexity degradation under unrotated W4A4 quantization. They then analyze state scale grouping, the attenuation of inherited errors through decay, and error accumulation under repeated rounding, leading to QReS, which combines value-axis scale grouping, stochastic rounding, and decay-guided mixed-precision allocation. Generation evaluations on Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B show that INT8 states preserve most of the accuracy, while the feasibility of lower effective precision depends on the task. A vLLM implementation further demonstrates decoding benefits from state quantization that become more pronounced at higher concurrency.
Critical strengths
S1. The controlled architectural comparisons provide informative evidence about quantization behavior in recurrent models. The authors use a shared 1.3B backbone and quantization procedure while systematically varying decay, write gating, and delta correction, reducing the confounding factors associated with comparing independently developed models. Tables 2, 8, and 9 show a consistent relationship between decay and robustness to unrotated W4A4 quantization, while the channel interventions in Tables 3 and 10 provide further evidence that different architectures rely differently on dominant residual channels. This experimental design helps explain how architectural choices influence quantization requirements and constitutes a valuable empirical contribution. S2. The state-quantization approach is validated through actual generation across architectures, with particularly compelling evidence for stochastic rounding. The paper evaluates three large hybrid models on tasks involving short generation, long reasoning, and long inputs. Table 17 shows that stochastic rounding can substantially improve long-reasoning performance under the same INT8 state precision, grouping, and grid settings. For example, Qwen’s AIME accuracy increases from 54.0% with round-to-nearest to 79.5% with stochastic rounding, approaching the 82.3% FP32-state baseline. These results establish practical generation benefits beyond improvements in short-sequence proxy metrics. S3. The principal design choices are supported by substantial ablations, including informative failure cases. The paper compares scale-grouping axes, rounding rules, symmetric and asymmetric grids, rotations, and integer versus floating-point formats. Table 12 additionally includes a random-allocation control with the same precision-allocation proportions, supporting the usefulness of decay information for precision assignment. The authors also report negative results, such as the severe degradation caused by stochastic rounding with key-axis grouping in some models. These findings help establish the method’s operating conditions rather than presenting the final recipe as universally effective. S4. The implementation demonstrates practical decoding gains and includes controls that isolate the contribution of state quantization. The authors implement decoding kernels with fused state quantization and conduct model-level evaluations in vLLM. The appendix reports FP32/INT8 state comparisons at the same weight precision, absolute TPOT measurements, and controls with a fixed attention block size. For example, Table 21 shows that W4A16 Nemotron-H’s TPOT decreases from 9.86 ms to 7.50 ms at concurrency 64. These results demonstrate that state quantization can benefit full-model decoding rather than only isolated kernels.
Critical weaknesses
W1. The incremental benefit over existing state-quantization approaches is not sufficiently established through direct comparisons. The paper discusses related approaches in Q-Mamba, Quamba2, and Nemotron, but the experiments primarily compare selected grouping, rounding, and format variants. A direct comparison against a closely related state-quantization approach under the same model, generation protocol, and resource budget is missing. Since low-precision state storage and stochastic rounding have precedents, this gap limits the assessment of QReS’s incremental contribution and accuracy–efficiency advantages. Including a representative existing approach in at least one comparable Mamba-family setting would substantially strengthen the positioning of the contribution. W2. The low-effective-precision and decay-allocation results do not yet establish a robust configuration-selection rule. The random-allocation control in Table 12 is valuable, but it primarily uses the median proxy loss on short WikiText-2 trajectories. The generation experiments show accuracy curves under decay-based allocation without sufficiently comparing alternative allocations at the same budget. The AIME results in Table 18 use pass@1 on 60 problems and do not report uncertainty, leaving the stability of the minimum bit width satisfying the 5% relative accuracy-loss threshold unclear. It also remains necessary to clarify whether model-specific grid and bit-width choices were made independently of the final test data. W3. Reproducibility details and evidence for the generality of the extra-state findings remain limited. The controlled experiments with a shared backbone are an important strength, but Appendix B.1 does not fully specify the training corpus, optimizer and learning-rate schedule, training sequence length, or random seeds, and variability across repeated training runs is not reported. The relationship between residual outlier structure and quantization robustness is currently established primarily at one scale under a shared training recipe. Providing complete reproduction details and distinguishing observations in this controlled setting from broader architectural regularities would strengthen confidence in this central empirical finding.
Questions for authors
1.Could you compare QReS against an existing state-quantization approach or an FP16+SR configuration on at least one comparable Mamba-family model, reporting accuracy, actual state storage, and decoding latency under the same generation protocol? If such approaches cannot be directly adapted, please explain the specific obstacles and identify which existing results isolate QReS’s additional benefits beyond previously established components. 2.What data are used to determine the decay ranking, grid type, and minimum bit width, and are these choices separated from the final test sets? Could you provide paired uncertainty estimates for generation results near the bit-width thresholds and evaluate at least one alternative precision allocation at the same storage budget, to establish whether the advantage of decay ranking transfers from the proxy loss to actual generation? 3.Please provide the complete training data and training configuration for the mixer zoo, and clarify whether data order and sources of randomness are matched across architectures. Does the relationship between residual outlier channels and W4A4 degradation remain stable across available checkpoints or repeated experiments?
We thank Reviewer 6H3W for their thoughtful review. We appreciate their recognition of our controlled mixer-zoo comparisons and “particularly compelling evidence for stochastic rounding”, and hope to address the remaining questions below.
We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version’s numbering.
W1/Q1: Comparisons and contribution.
Both our analysis and its instantiation, QReS, apply to a general family of linear recurrences. Existing methods, such as Q-Mamba and Quamba2, focus on Mamba-2. We therefore compare the Mamba-2 instantiation of QReS with these baselines. We have clarified this positioning in Section 2.2.
The table below compares QReS on Nemotron-H 8B with Q-Mamba, Quamba2, and the FP16 + SR recipe used in Nemotron 3 Super. We have added this comparison to Appendix H (Table 25).
The table above shows that QReS approaches the accuracy of Nemotron 3 Super's FP16 + SR recipe at half the state precision. Quamba2 loses substantial accuracy and produces much longer answers.
Q-Mamba generates roughly twice as many tokens as QReS on average. At a 64k-token cap, its accuracy is comparable to QReS. Reducing the decode cap to 32k, however, lowers Q-Mamba's AIME accuracy, while QReS' remains virtually unchanged.
Decode latency. Since no public implementation was available, we re-implemented Q-Mamba and added a key-axis-only control. The table below reports Mamba-2 INT8 kernel speedups over FP32 on H100 for QReS, "QReS with key-axis scales", and our implementation of Q-Mamba.
The table above shows that, in our implementation, key-axis scaling already reduces the speedup, with Q-Mamba's combined row and column scales adding further overhead. This supports the point in Appendix G.1: value-axis scales are local to decode kernel tiles, whereas key-axis scales require reduction across tiles. We have added this comparison to Appendix H (Figure 16).
Finally, QReS INT8 reduces measured state storage per layer and sequence from 4.19 MB (FP32) to 1.08 MB, including scales.
W2/Q2: Uncertainty and configuration selection.
To better estimate the minimum precision that meets the 5% relative accuracy-loss rule, we reran the critical bit widths with avg@16. The table below reports FP32 avg@32 accuracy (%) and quantized avg@16 accuracy with standard errors across seeds.
In the avg@16 evals, the minimum precision meeting the 5% criterion on AIME is lowered from 7.5 to 7 bits for Qwen and from 8 to 7.5 bits for Nemotron-H. The rest is the same. Overall, our promotion findings stay consistent. We thank Reviewer 6H3W for pointing out the missing error bars. We have added the uncertainty results and updated promotion plot in Appendix F.3.
Calibration and configuration selection. All calibration is done using held-out wikitext-2 samples. We chose the model-specific grids for the main evaluations and promotion sweeps from the AIME comparison in Table 18, then kept them fixed for the other benchmarks.
Alternative allocation. To test whether the decay ranking improves generation, we compare it with one random assignment, giving both the same mix of half INT6 and half INT4 heads. The table below shows that decay-based allocation retains substantially more GPQA accuracy in both models.
W3/Q3: Reproducibility and checkpoints.
Training setup. We thank Reviewer 6H3W for pointing this out. We have added the training details in Appendix B.1. All models used the same seed with matched data order and randomness.
Extra State. The table below reports the kurtosis–W4A4 Spearman correlation across all models at the intermediate and final checkpoints, confirming a consistent relationship. Training ends at 190k steps.
We again thank Reviewer 6H3W for their thoughtful review and hope to have addressed their questions.