- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PRQuant introduces a training-free quantization approach designed to boost the accuracy of MXFP4 FFN down-projections without breaking regular low-precision compute patterns. By combining channel scaling with a permutation strategy that clusters high-error weight columns, the method incorporates offline weight residuals and activation duplication into a single expanded MXFP4 GEMM operation. Empirical results across five multiple-choice benchmarks show that PRQuant achieves the highest average accuracy among evaluated quantization methods. However, BF16 accuracy was not fully recovered on average. On Ascend 950PR, the targeted down-projection kernels achieve up to 2.6x and 1.8x speedups over BF16 on Qwen3-4B and Qwen3-30B-A3B, respectively. And the computation overhead, measured by end-to-end decoding latency comparing with vanilla MXFP4 model, is approximately 6% and 4% of the two models tested.
Critical strengths
-
The method is clear and has a concrete implementation rationale. Moving selected channels into a contiguous tail makes activation reuse regular, while an offline weight residual allows compensation through an augmented MXFP4 GEMM without constructing activation residuals online or requiring a separate high-precision correction GEMM. This is a useful connection between quantization and efficient execution.
-
The paper evaluates actual inference performance as well as numerical accuracy. Table 1 reports average accuracy improvements over plain MXFP4 of 1.24 percentage points on Qwen3-4B and 0.55 on Qwen3-30B, while Table 4 reports decoding latency close to plain MXFP4 in the tested configuration. These results provide evidence of practical utility.
Critical weaknesses
- The channel-selection strategy is insufficiently validated after microscaling blocks are regrouped.
Algorithm 1 selects channels using column-wise MXFP4 quantization errors before permutation. However, these errors depend on the shared scales of the original blocks. Moving the selected channels to the tail changes block membership not only for those channels, but also for the remaining channels. Channels that were easy to quantize in their original blocks may consequently incur larger errors in the new blocks, yet they receive no residual compensation.
The w/o Perm ablation in Figure 2 is insufficiently specified to resolve this concern. However, it is unclear whether this variant compensates the same top-k high-error channels through gathering, or simply compensates the last k channels in the original ordering. The column-error score, channel-ordering rules, and whether alpha is held fixed or refitted also need clarification. A controlled comparison tracking the same selected and unselected channels before and after permutation would better substantiate Section 3's claim that regrouping protects regular blocks from outliers.
- Experimental settings, parameter ablations, and generation evaluation are incomplete.
The candidate set for alpha is not specified, and the calibration description does not give the sample count and sequence lengths used. The paper does report k=128 for Qwen3-30B-A3B and k=1024 for Qwen3-4B on page 8, but does not explain their selection or study the method's sensitivity to k. Since k controls both residual capacity and additional computation, an accuracy-latency tradeoff across tail widths is important to assess the claimed low overhead. The component ablations report layer reconstruction error, leaving their effects on downstream accuracy unclear.
The accuracy evaluation is limited to five multiple-choice benchmarks, long sequence generation served only as latency benchmarking task in the manuscript. A task requiring substantial generated output, together with additional input/output length configurations, would make the evaluation more representative.
Questions for authors
-
Clarification and targeted validation of regrouping: In w/o Perm, are the compensated channels the same top-k high-error channels as in full PRQuant, the last k channels in the original ordering, or another set? Is alpha held fixed or retuned? With scaling and channel selection fixed, please compare the errors of the same selected and unselected channels before and after permutation, before residual compensation, to establish whether the selected set still captures the dominant errors after regrouping.
-
Reproducibility and parameter validation: What alpha candidates, calibration sample count, and calibration sequence lengths are used? How were the reported tail widths selected? Please provide an accuracy-latency comparison across k values and downstream accuracy results for the component ablations, in addition to layer reconstruction error.
-
Generation validation: Can you provide quality results on at least one task requiring substantial autoregressive generation, and latency or throughput results across additional input and output lengths? This would help determine whether the benefits shown on multiple-choice tasks extend to longer generation.
3 comments
Response to Reviewer ww4E (2/3): Calibration, Tail-Width Selection, and Ablations
Q2: How are calibration and tail width configured, and how do tail width and individual components affect downstream accuracy and latency?
A2: Thank you for encouraging us to provide reproducible settings and explain our configuration choices. We have added these to the revised manuscript.
Calibration settings. We search over , with 21 candidates, and independently select the value minimizing down-projection output reconstruction error for each layer or expert. Calibration uses 100 diverse prompts from mix_calib.jsonl, covering code generation, mathematical reasoning, and general instruction following. Captured sequences are padded to 4,096 tokens. For sequences beyond the token budget, we subsample at evenly spaced indices so at most 2,048 tokens per sequence contribute to calibration statistics.
Tail-width selection. We use tail widths aligned with MXFP4 blocks (multiples of 32) and select separately for each model by comparing downstream accuracy and down-projection operator latency across candidate widths. The default configurations expand the reduction dimension by 10.5% on 4B and 16.7% on 30B. For each candidate , we recalibrate .
Table R2. Sensitivity to tail width , excerpted from Table 8 in Appendix B.3. Mean accuracy is the unweighted average over the same five benchmarks. denotes the number of input tokens to the operator. Latencies are in μs.
| Model | Configuration | Mean accuracy (%) | Latency at (μs) | Latency at (μs) |
|---|---|---|---|---|
| Qwen3-4B | Plain MXFP4 | — | 13.65 | 54.68 |
| Qwen3-4B | 81.95 | 14.12 | 56.86 | |
| Qwen3-4B | 82.20 | 14.37 | 59.86 | |
| Qwen3-4B | 81.99 | 14.80 | 64.19 | |
| Qwen3-30B | Plain MXFP4 | — | 9.70 | 12.80 |
| Qwen3-30B | 89.89 | 9.83 | 16.26 | |
| Qwen3-30B | 89.90 | 10.10 | 17.20 | |
| Qwen3-30B | 89.88 | 10.78 | 23.26 |
Qwen3-4B. Increasing from 768 to 1024 improves mean accuracy by 0.25 percentage points, with additional operator latencies of 0.25 μs at and 3.00 μs at . Increasing further to 2048 lowers the observed mean accuracy by 0.21 percentage points and adds another 0.43 μs and 4.33 μs, respectively. We therefore select , which gives the highest mean accuracy in this comparison and lower latency than at both input sizes.
Qwen3-30B-A3B. Mean accuracy varies by only 0.02 percentage points across the three widths, while individual benchmarks move in different directions. From to , WinoGrande accuracy rises from 74.90% to 76.01%, whereas ARC-C accuracy falls from 94.24% to 92.54%. Given the small accuracy differences on 30B, we conservatively select : it gives the highest observed mean within this narrow range and adds only 0.27 μs at and 0.94 μs at over . Doubling the width to 256 adds another 6.06 μs at without improving mean accuracy.
Increasing compensates more channels but expands the GEMM reduction dimension from to and increases activation copying. These results support selecting a limited tail width based on the accuracy–latency tradeoff.
Downstream component ablations. We additionally evaluate downstream accuracy on the five benchmarks using Qwen3-30B-A3B. All three ablated variants have lower mean accuracy than the full PRQuant reference.
Table R3. Downstream component ablations on Qwen3-30B-A3B. Accuracy is the unweighted mean over the five benchmarks. Ablation results report the mean ± sample standard deviation across three runs.
| Configuration | Mean accuracy (%) | Change relative to full PRQuant (pp) |
|---|---|---|
| Full PRQuant | 89.90 (reference) | — |
| w/o Perm | 89.75 ± 0.02 | -0.15 |
| w/o Res | 89.79 ± 0.08 | -0.11 |
| w/o AWQ | 89.63 ± 0.21 | -0.27 |
In Table 1, the mean accuracies of BF16, plain MXFP4, and full PRQuant on Qwen3-30B are 90.62%, 89.35%, and 89.90%, respectively. Plain MXFP4 has a 1.27-percentage-point gap to BF16, while full PRQuant improves accuracy by 0.55 percentage points, recovering approximately 43.3% of this quantization-induced accuracy gap.
Downstream accuracy and layer-wise NRMSE measure different aspects of performance, which is why we supplement the reconstruction results with these downstream evaluations. NRMSE directly measures the fidelity of local output reconstruction, whereas downstream accuracy also depends on whether errors propagated through subsequent layers change the final answer. Improvements in local reconstruction error therefore need not translate proportionally into accuracy gains. This distinction informs our interpretation of the magnitudes of the observed differences.
We thank the reviewer again for the detailed suggestions, which have helped us clarify the configuration choices and strengthen the experimental analysis.
Response to Reviewer ww4E (3/3): Generation Quality and Sequence-Length Evaluation
Q3: How does PRQuant affect autoregressive generation quality, and how does runtime vary with input and output sequence lengths?
A3: Thank you for this suggestion. We have added GPQA-Diamond accuracy evaluation and end-to-end performance results covering additional input and output sequence lengths.
GPQA-Diamond accuracy. We evaluate Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 on GPQA-Diamond.
Table R4. GPQA-Diamond accuracy. All values are percentages.
| Model | BF16 | Plain MXFP4 | PRQuant |
|---|---|---|---|
| Qwen3-4B | 49.49 | 45.58 | 48.23 |
| Qwen3-30B-A3B | 69.19 | 65.15 | 68.08 |
PRQuant improves GPQA-Diamond accuracy over plain MXFP4 by 2.65 percentage points on Qwen3-4B and 2.93 percentage points on Qwen3-30B-A3B, reducing the remaining gaps to BF16 to 1.26 and 1.11 percentage points, respectively.
Performance across additional input and output lengths. We evaluate Qwen3-4B and Qwen3-30B on Ascend 950PR with batch size 8, covering all four combinations of input lengths 4096/8192 and output lengths 1024/2048.
Table R5. End-to-end performance on Ascend 950PR at batch size 8. ISL and OSL are input and output sequence lengths in tokens. TTFT is in ms; TPOT is in ms/token.
| ISL | OSL | Method | 4B TTFT | 4B TPOT | 30B TTFT | 30B TPOT |
|---|---|---|---|---|---|---|
| 4096 | 1024 | BF16 | 632 | 12.07 | 913.0 | 25.12 |
| 4096 | 1024 | Plain MXFP4 | 381.1 | 8.87 | 606.8 | 13.91 |
| 4096 | 1024 | PRQuant | 386.4 | 9.39 | 751.4 | 14.12 |
| 4096 | 2048 | BF16 | 634 | 12.34 | 916.6 | 25.29 |
| 4096 | 2048 | Plain MXFP4 | 384.2 | 9.27 | 604.9 | 13.95 |
| 4096 | 2048 | PRQuant | 388.3 | 9.58 | 754.0 | 14.31 |
| 8192 | 1024 | BF16 | 1248 | 15.38 | 1815.5 | 25.82 |
| 8192 | 1024 | Plain MXFP4 | 772.5 | 11.93 | 1194.9 | 14.61 |
| 8192 | 1024 | PRQuant | 780.5 | 12.23 | 1436.3 | 14.91 |
| 8192 | 2048 | BF16 | 1244 | 15.42 | 1815.4 | 26.07 |
| 8192 | 2048 | Plain MXFP4 | 770.7 | 12.08 | 1195.3 | 14.74 |
| 8192 | 2048 | PRQuant | 781.4 | 12.34 | 1440.8 | 15.04 |
Across these configurations, PRQuant achieves 1.25–1.29× and 1.73–1.78× decoding speedups over BF16 on Qwen3-4B and Qwen3-30B, respectively, with 2.2%–5.9% and 1.5%–2.6% additional TPOT relative to plain MXFP4. Together with the accuracy results, these measurements quantify the runtime cost of accuracy recovery: PRQuant retains decoding efficiency close to plain MXFP4 in tests, while both prefill and decoding remain faster than BF16 across all reported configurations.
We sincerely thank the reviewer for the thoughtful questions and careful attention to technical details.
Response to Reviewer ww4E (1/3): Controlled Analysis of Channel Regrouping
We sincerely thank the reviewer for the careful reading and technically detailed, constructive feedback. Your questions have helped us clarify the implementation of the ablation, isolate the effects of channel regrouping through controlled comparisons, and explain the calibration settings and tail-width selection more precisely. They have also prompted us to examine the scope of our evaluation more carefully. Addressing these points has strengthened both the technical clarity of the manuscript and the supporting experimental analysis. We respond to your concerns in the following three parts.
Q1: How does channel regrouping affect quantization error in selected and unselected channels, and how is the w/o Perm ablation implemented?
A1: Thank you for highlighting the need to clarify the role of channel regrouping, particularly whether permutation protects regular channels that are not selected for compensation. Our additional controlled experiment shows that permutation reduces overall weight quantization error while concentrating a larger share of the remaining error in channels selected for residual compensation. Following your suggestion, we fix the scaling factors and selected channel set and compare the errors of exactly the same selected and unselected channels before and after permutation, prior to residual compensation.
Purpose and setup of the original w/o Perm ablation. Our primary motivation for introducing permutation is to reduce online processing overhead. By absorbing the channel permutation offline into adjacent projection weights, scattered channels requiring compensation directly form a contiguous tail, replacing runtime indexed gathering of scattered channels with contiguous tail copying . The original w/o Perm ablation additionally examines whether permutation can further reduce reconstruction error within the complete quantization pipeline.
Specifically, w/o Perm retains the original channel order and recalibrates α . Residual compensation is applied through indexed gathering to the channels selected under that configuration, rather than to the last k channels in the original order. The main quantization branch retains its original block grouping. Thus, this ablation compares complete quantization configurations that are each recalibrated separately. We have clarified this setup in Section 5.3.
Controlled experiment with fixed scaling and channel selection. To further isolate the effect of block regrouping itself, we fix the scaling factors and selected channel set obtained by full PRQuant and track exactly the same selected and unselected channels before and after permutation, prior to residual compensation. Across all 48 layers, the unselected channels' error share decreases, while the selected channels' share increases. Although the selected channels constitute only one-sixth of all channels, they account for approximately 40% of the weight quantization error after regrouping.
Table R1. Weight quantization error shares before residual compensation on Qwen3-30B. Scaling factors and the selected channel set are fixed. Values are averaged equally across the 48 layers; pp denotes percentage points. Complete layer-wise results are provided in Table 9 of Appendix C.
Meanwhile, permutation also reduces the total squared weight quantization error before residual compensation in all 48 layers. The corresponding weight NRMSE decreases by 1.58%-11.97% across layers, with an average relative reduction of 4.19%. Thus, the increased concentration of error in the selected channels accompanies a reduction in overall weight quantization error.
These results support the numerical role of channel regrouping: protecting regular blocks from outliers while concentrating a larger share of the remaining error in channels eligible for targeted residual compensation. They also show that the contiguous channel layout introduced to reduce online overhead can simultaneously improve the distribution of quantization error. We thank you for these detailed suggestions. We have added this analysis to Section 5.3 and provided the complete layer-wise error-share statistics in Appendix C.