acceptodds
Under review as a conference paper at ICLR 2027

PRQuant: Permutation Residual Quantization for Low-Overhead Inference

Abstract

Low-bit quantization of linear layers is often dominated by a small number of outlier channels. Existing smoothing, rotation, and residual-based methods can mitigate this issue, but may shift the quantization bottleneck to weights or introduce costly online operations. To address these limitations, we propose PRQuant (Permutation Residual Quantization), a training-free framework that combines channel permutation with offline weight residual compensation. PRQuant identifies the scaled-weight columns with the largest quantization errors and permutes them into contiguous tail blocks. This structure allows the corresponding weight residuals to be precomputed entirely offline, while replacing scattered activation gathering with simple contiguous access during inference, yielding a single regular MXFP4 GEMM for compensated computation. Experiments show that PRQuant substantially reduces down-projection reconstruction error, with scaling and residual compensation providing the main numerical gains while permutation enables a hardware-friendly contiguous layout. Comprehensive experiment results on Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 illustrate that PRQuant achieves up to averagelly 2.6× and 1.8× operator speedup over BF16 respectively, while preserving near plain MXFP4 end-to-end decoding efficiency. Across five downstream benchmarks, PRQuant achieves the best average accuracy among the quantized methods, improving accuracy over MXFP4 by 1.24 and 0.55, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Low-Bit Quantization, MXFP4, Microscaling Formats, Hardware-Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ww4EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRQuant introduces a training-free quantization approach designed to boost the accuracy of MXFP4 FFN down-projections without breaking regular low-precision compute patterns. By combining channel scaling with a permutation strategy that clusters high-error weight columns, the method incorporates offline weight residuals and activation duplication into a single expanded MXFP4 GEMM operation. Empirical results across five multiple-choice benchmarks show that PRQuant achieves the highest average accuracy among evaluated quantization methods. However, BF16 accuracy was not fully recovered on average. On Ascend 950PR, the targeted down-projection kernels achieve up to 2.6x and 1.8x speedups over BF16 on Qwen3-4B and Qwen3-30B-A3B, respectively. And the computation overhead, measured by end-to-end decoding latency comparing with vanilla MXFP4 model, is approximately 6% and 4% of the two models tested.

Critical strengths
  1. The method is clear and has a concrete implementation rationale. Moving selected channels into a contiguous tail makes activation reuse regular, while an offline weight residual allows compensation through an augmented MXFP4 GEMM without constructing activation residuals online or requiring a separate high-precision correction GEMM. This is a useful connection between quantization and efficient execution.

  2. The paper evaluates actual inference performance as well as numerical accuracy. Table 1 reports average accuracy improvements over plain MXFP4 of 1.24 percentage points on Qwen3-4B and 0.55 on Qwen3-30B, while Table 4 reports decoding latency close to plain MXFP4 in the tested configuration. These results provide evidence of practical utility.

Critical weaknesses
  1. The channel-selection strategy is insufficiently validated after microscaling blocks are regrouped.

Algorithm 1 selects channels using column-wise MXFP4 quantization errors before permutation. However, these errors depend on the shared scales of the original blocks. Moving the selected channels to the tail changes block membership not only for those channels, but also for the remaining channels. Channels that were easy to quantize in their original blocks may consequently incur larger errors in the new blocks, yet they receive no residual compensation.

The w/o Perm ablation in Figure 2 is insufficiently specified to resolve this concern. However, it is unclear whether this variant compensates the same top-k high-error channels through gathering, or simply compensates the last k channels in the original ordering. The column-error score, channel-ordering rules, and whether alpha is held fixed or refitted also need clarification. A controlled comparison tracking the same selected and unselected channels before and after permutation would better substantiate Section 3's claim that regrouping protects regular blocks from outliers.

  1. Experimental settings, parameter ablations, and generation evaluation are incomplete.

The candidate set for alpha is not specified, and the calibration description does not give the sample count and sequence lengths used. The paper does report k=128 for Qwen3-30B-A3B and k=1024 for Qwen3-4B on page 8, but does not explain their selection or study the method's sensitivity to k. Since k controls both residual capacity and additional computation, an accuracy-latency tradeoff across tail widths is important to assess the claimed low overhead. The component ablations report layer reconstruction error, leaving their effects on downstream accuracy unclear.

The accuracy evaluation is limited to five multiple-choice benchmarks, long sequence generation served only as latency benchmarking task in the manuscript. A task requiring substantial generated output, together with additional input/output length configurations, would make the evaluation more representative.

Questions for authors
  1. Clarification and targeted validation of regrouping: In w/o Perm, are the compensated channels the same top-k high-error channels as in full PRQuant, the last k channels in the original ordering, or another set? Is alpha held fixed or retuned? With scaling and channel selection fixed, please compare the errors of the same selected and unselected channels before and after permutation, before residual compensation, to establish whether the selected set still captures the dominant errors after regrouping.

  2. Reproducibility and parameter validation: What alpha candidates, calibration sample count, and calibration sequence lengths are used? How were the reported tail widths selected? Please provide an accuracy-latency comparison across k values and downstream accuracy results for the component ablations, in addition to layer reconstruction error.

  3. Generation validation: Can you provide quality results on at least one task requiring substantial autoregressive generation, and latency or throughput results across additional input and output lengths? This would help determine whether the benefits shown on multiple-choice tasks extend to longer generation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ww4E (1/3): Controlled Analysis of Channel Regrouping

We sincerely thank the reviewer for the careful reading and technically detailed, constructive feedback. Your questions have helped us clarify the implementation of the ablation, isolate the effects of channel regrouping through controlled comparisons, and explain the calibration settings and tail-width selection more precisely. They have also prompted us to examine the scope of our evaluation more carefully. Addressing these points has strengthened both the technical clarity of the manuscript and the supporting experimental analysis. We respond to your concerns in the following three parts.

Q1: How does channel regrouping affect quantization error in selected and unselected channels, and how is the w/o Perm ablation implemented?

A1: Thank you for highlighting the need to clarify the role of channel regrouping, particularly whether permutation protects regular channels that are not selected for compensation. Our additional controlled experiment shows that permutation reduces overall weight quantization error while concentrating a larger share of the remaining error in channels selected for residual compensation. Following your suggestion, we fix the scaling factors and selected channel set and compare the errors of exactly the same selected and unselected channels before and after permutation, prior to residual compensation.

Purpose and setup of the original w/o Perm ablation. Our primary motivation for introducing permutation is to reduce online processing overhead. By absorbing the channel permutation offline into adjacent projection weights, scattered channels requiring compensation directly form a contiguous tail, replacing runtime indexed gathering of scattered channels with contiguous tail copying . The original w/o Perm ablation additionally examines whether permutation can further reduce reconstruction error within the complete quantization pipeline.

Specifically, w/o Perm retains the original channel order and recalibrates α\alpha . Residual compensation is applied through indexed gathering to the channels selected under that configuration, rather than to the last kk channels in the original order. The main quantization branch retains its original block grouping. Thus, this ablation compares complete quantization configurations that are each recalibrated separately. We have clarified this setup in Section 5.3.

Controlled experiment with fixed scaling and channel selection. To further isolate the effect of block regrouping itself, we fix the scaling factors and selected channel set obtained by full PRQuant and track exactly the same selected and unselected channels before and after permutation, prior to residual compensation. Across all 48 layers, the unselected channels' error share decreases, while the selected channels' share increases. Although the selected channels constitute only one-sixth of all channels, they account for approximately 40% of the weight quantization error after regrouping.

Table R1. Weight quantization error shares before residual compensation on Qwen3-30B. Scaling factors and the selected channel set are fixed. Values are averaged equally across the 48 layers; pp denotes percentage points. Complete layer-wise results are provided in Table 9 of Appendix C.

Channel setError share before permutationError share after permutationMean change
Selected (1/6 of channels)33.15%39.33%+6.18 pp
Unselected66.85%60.67%-6.18 pp

Meanwhile, permutation also reduces the total squared weight quantization error before residual compensation in all 48 layers. The corresponding weight NRMSE decreases by 1.58%-11.97% across layers, with an average relative reduction of 4.19%. Thus, the increased concentration of error in the selected channels accompanies a reduction in overall weight quantization error.

These results support the numerical role of channel regrouping: protecting regular blocks from outliers while concentrating a larger share of the remaining error in channels eligible for targeted residual compensation. They also show that the contiguous channel layout introduced to reduce online overhead can simultaneously improve the distribution of quantization error. We thank you for these detailed suggestions. We have added this analysis to Section 5.3 and provided the complete layer-wise error-share statistics in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer ww4E (2/3): Calibration, Tail-Width Selection, and Ablations

Q2: How are calibration and tail width kk configured, and how do tail width and individual components affect downstream accuracy and latency?

A2: Thank you for encouraging us to provide reproducible settings and explain our configuration choices. We have added these to the revised manuscript.

Calibration settings. We search over α∈{0,0.05,0.10,…,1.00}\alpha \in \{0,0.05,0.10,\ldots,1.00\}, with 21 candidates, and independently select the value minimizing down-projection output reconstruction error for each layer or expert. Calibration uses 100 diverse prompts from mix_calib.jsonl, covering code generation, mathematical reasoning, and general instruction following. Captured sequences are padded to 4,096 tokens. For sequences beyond the token budget, we subsample at evenly spaced indices so at most 2,048 tokens per sequence contribute to calibration statistics.

Tail-width selection. We use tail widths aligned with MXFP4 blocks (multiples of 32) and select kk separately for each model by comparing downstream accuracy and down-projection operator latency across candidate widths. The default configurations expand the reduction dimension by 10.5% on 4B and 16.7% on 30B. For each candidate kk, we recalibrate α\alpha.

Table R2. Sensitivity to tail width kk, excerpted from Table 8 in Appendix B.3. Mean accuracy is the unweighted average over the same five benchmarks. MM denotes the number of input tokens to the operator. Latencies are in μs.

ModelConfigurationMean accuracy (%)Latency at M=16M=16 (μs)Latency at M=1024M=1024 (μs)
Qwen3-4BPlain MXFP4—13.6554.68
Qwen3-4Bk=768k=76881.9514.1256.86
Qwen3-4Bk=1024k=102482.2014.3759.86
Qwen3-4Bk=2048k=204881.9914.8064.19
Qwen3-30BPlain MXFP4—9.7012.80
Qwen3-30Bk=64k=6489.899.8316.26
Qwen3-30Bk=128k=12889.9010.1017.20
Qwen3-30Bk=256k=25689.8810.7823.26

Qwen3-4B. Increasing kk from 768 to 1024 improves mean accuracy by 0.25 percentage points, with additional operator latencies of 0.25 μs at M=16M=16 and 3.00 μs at M=1024M=1024. Increasing kk further to 2048 lowers the observed mean accuracy by 0.21 percentage points and adds another 0.43 μs and 4.33 μs, respectively. We therefore select k=1024k=1024, which gives the highest mean accuracy in this comparison and lower latency than k=2048k=2048 at both input sizes.

Qwen3-30B-A3B. Mean accuracy varies by only 0.02 percentage points across the three widths, while individual benchmarks move in different directions. From k=64k=64 to k=256k=256, WinoGrande accuracy rises from 74.90% to 76.01%, whereas ARC-C accuracy falls from 94.24% to 92.54%. Given the small accuracy differences on 30B, we conservatively select k=128k=128: it gives the highest observed mean within this narrow range and adds only 0.27 μs at M=16M=16 and 0.94 μs at M=1024M=1024 over k=64k=64. Doubling the width to 256 adds another 6.06 μs at M=1024M=1024 without improving mean accuracy.

Increasing kk compensates more channels but expands the GEMM reduction dimension from CC to C+kC+k and increases activation copying. These results support selecting a limited tail width based on the accuracy–latency tradeoff.

Downstream component ablations. We additionally evaluate downstream accuracy on the five benchmarks using Qwen3-30B-A3B. All three ablated variants have lower mean accuracy than the full PRQuant reference.

Table R3. Downstream component ablations on Qwen3-30B-A3B. Accuracy is the unweighted mean over the five benchmarks. Ablation results report the mean ± sample standard deviation across three runs.

ConfigurationMean accuracy (%)Change relative to full PRQuant (pp)
Full PRQuant89.90 (reference)—
w/o Perm89.75 ± 0.02-0.15
w/o Res89.79 ± 0.08-0.11
w/o AWQ89.63 ± 0.21-0.27

In Table 1, the mean accuracies of BF16, plain MXFP4, and full PRQuant on Qwen3-30B are 90.62%, 89.35%, and 89.90%, respectively. Plain MXFP4 has a 1.27-percentage-point gap to BF16, while full PRQuant improves accuracy by 0.55 percentage points, recovering approximately 43.3% of this quantization-induced accuracy gap.

Downstream accuracy and layer-wise NRMSE measure different aspects of performance, which is why we supplement the reconstruction results with these downstream evaluations. NRMSE directly measures the fidelity of local output reconstruction, whereas downstream accuracy also depends on whether errors propagated through subsequent layers change the final answer. Improvements in local reconstruction error therefore need not translate proportionally into accuracy gains. This distinction informs our interpretation of the magnitudes of the observed differences.

We thank the reviewer again for the detailed suggestions, which have helped us clarify the configuration choices and strengthen the experimental analysis.

AuthorsOpenReview ↗

Response to Reviewer ww4E (3/3): Generation Quality and Sequence-Length Evaluation

Q3: How does PRQuant affect autoregressive generation quality, and how does runtime vary with input and output sequence lengths?

A3: Thank you for this suggestion. We have added GPQA-Diamond accuracy evaluation and end-to-end performance results covering additional input and output sequence lengths.

GPQA-Diamond accuracy. We evaluate Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 on GPQA-Diamond.

Table R4. GPQA-Diamond accuracy. All values are percentages.

ModelBF16Plain MXFP4PRQuant
Qwen3-4B49.4945.5848.23
Qwen3-30B-A3B69.1965.1568.08

PRQuant improves GPQA-Diamond accuracy over plain MXFP4 by 2.65 percentage points on Qwen3-4B and 2.93 percentage points on Qwen3-30B-A3B, reducing the remaining gaps to BF16 to 1.26 and 1.11 percentage points, respectively.

Performance across additional input and output lengths. We evaluate Qwen3-4B and Qwen3-30B on Ascend 950PR with batch size 8, covering all four combinations of input lengths 4096/8192 and output lengths 1024/2048.

Table R5. End-to-end performance on Ascend 950PR at batch size 8. ISL and OSL are input and output sequence lengths in tokens. TTFT is in ms; TPOT is in ms/token.

ISLOSLMethod4B TTFT4B TPOT30B TTFT30B TPOT
40961024BF1663212.07913.025.12
40961024Plain MXFP4381.18.87606.813.91
40961024PRQuant386.49.39751.414.12
40962048BF1663412.34916.625.29
40962048Plain MXFP4384.29.27604.913.95
40962048PRQuant388.39.58754.014.31
81921024BF16124815.381815.525.82
81921024Plain MXFP4772.511.931194.914.61
81921024PRQuant780.512.231436.314.91
81922048BF16124415.421815.426.07
81922048Plain MXFP4770.712.081195.314.74
81922048PRQuant781.412.341440.815.04

Across these configurations, PRQuant achieves 1.25–1.29× and 1.73–1.78× decoding speedups over BF16 on Qwen3-4B and Qwen3-30B, respectively, with 2.2%–5.9% and 1.5%–2.6% additional TPOT relative to plain MXFP4. Together with the accuracy results, these measurements quantify the runtime cost of accuracy recovery: PRQuant retains decoding efficiency close to plain MXFP4 in tests, while both prefill and decoding remain faster than BF16 across all reported configurations.

We sincerely thank the reviewer for the thoughtful questions and careful attention to technical details.

Reviewer_PGAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRQuant is quantization framework that reduces errors caused by outlier channels while maintaining low inference overhead. It first applies channel scaling, then selects the input channels with the largest quantization errors. The weight quantization residuals for these channels are computed offline, quantized to MXFP4, and appended to the original weight matrix. Result in increasing both the number of stored weight values and the total weight storage. Experiments on Qwen3-4B and Qwen3-30B-A3B report speedups of up to approximately 2.6× and 1.8× over BF16, respectively, while maintaining end-to-end decoding efficiency close to plain MXFP4.

Critical strengths
  1. The proposed method addresses the accuracy limitations of MXFP4 and compensate quantization error.
  2. The paper demonstrates reductions in TTFT and TPOT for the down-projection kernel across a range of batch sizes on the Ascend 950PR.
Critical weaknesses
  1. Incomplete characterization of residual-compensation overhead. Although Section 5.4 reports dimension expansions of 10.5% and 16.7% and provides latency measurements, the paper does not fully quantify the additional deployed-model storage and inference memory required by residual compensation, including residual weights and duplicated activations. It also lacks a latency breakdown separating the expanded GEMM from tail-copy operations.

  2. Limited evidence for other device. The runtime evaluation is conducted exclusively on Ascend 950PR, leaving the portability of the efficiency results to other architectures unclear. The paper would benefit from explicit documentation of the native MXFP4 execution paths and kernel optimizations used by both PRQuant and the plain MXFP4 baseline. In particular, it remains unverified whether the gains over BF16/ARCQuant and the overhead relative to plain MXFP4 would remain comparable on NVIDIA Blackwell GPUs.

Questions for authors
  1. Could the authors quantify the amount of data required and computation cost for residual compensation of the weights and clarify its impact on latency?
  2. Could the authors clarify whether both PRQuant and the plain MXFP4 baseline use native MXFP4 matrix-multiplication instructions on Ascend 950PR? How would the reported gains over BF16/PRQuant and the overhead changes relative to plain MXFP4 on NVIDIA Blackwell GPUs?
  3. Given that offline quantization is typically a one-time cost per exported checkpoint, in which deployment scenarios is the reduction in Table 2 practically significant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PGAK (1/3): Residual-Compensation Overhead

We thank the reviewer for the careful review and constructive feedback. In the revised manuscript, we have added a quantitative analysis of residual-compensation overhead in Appendix B.2 and clarified the native MXFP4 execution path and the practical relevance of offline quantization costs. We address these concerns in the following three responses.

Q1: What data, storage, and computation does residual compensation require, and what is its latency impact?

A1: Residual compensation adds modest storage and arithmetic, while most measured operator overhead comes from the expanded GEMM. We have added these analyses in Appendix B.2, "Overhead of residual compensation," and summarized the latency findings in Section 5.4, "Efficiency and Speed."

Storage and activation memory. PRQuant reuses the calibration data in Section 5.1 and requires no residual-specific dataset. Residual weights are computed and quantized offline. The additional weight payload is 47.8 MiB for Qwen3-4B and 0.80 GiB for Qwen3-30B, covering all experts. These correspond to approximately 1.6% and 4.9% of the estimated total mixed-precision model weight payload.

Table R1. Additional weight payload, including E8M0 block scales. Reproduced from Table 6 in Appendix B.2. HH, CC, and kk denote the hidden dimension, intermediate dimension, and tail width; NprojN_{\mathrm{proj}} is the number of compensated projections.

Model(H,C,k)(H,C,k)NprojN_{\mathrm{proj}}Residual valuesStorage
Qwen3-4B(2560, 9728, 1024)369.44×1079.44 \times 10^747.8 MiB
Qwen3-30B(2048, 768, 128)48×12848 \times 1281.61×1091.61 \times 10^90.80 GiB

Online copy additionally requires 544M544M and 68M68M bytes of packed activation-tail storage per down-projection invocation for 4B and 30B, respectively, where MM is the number of activation rows. Peak inference memory also depends on temporary buffers and workspace.

Computation. PRQuant extends only the down-projection GEMM reduction dimension from CC to C+kC+k, without a separate correction GEMM. Down-projection FLOPs increase by 10.5% and 16.7% on 4B and 30B. Since up/gate dimensions remain unchanged, the increases across the three FFN projection GEMMs are only 3.5% and 5.6%.

Operator latency. Figure 3 reports operator-level speedups based on complete down-projection timings. Appendix B.2 additionally profiles Qwen3-4B at M=16,256,4096M=16,256,4096. GEMM latency increases by 8.3%-9.8%, while the sum of preprocessing and GEMM times increases by 6.5%-7.8% relative to plain MXFP4. Activation quantization and tail copying are fused into one preprocessing kernel, whose latency remains close to plain MXFP4. Most incremental time in these profiled workloads is associated with the expanded GEMM.

Table R2. Down-projection latency breakdown on Qwen3-4B, reproduced from Table 7 in Appendix B.2. All latencies are in microseconds (μs). Preprocessing comprises activation quantization for plain MXFP4 and fused activation quantization plus quantized tail/scale copying for PRQuant.

MMMethodPreprocessingGEMMSum
16BF16—17.5517.55
16Plain MXFP42.7911.9614.75
16PRQuant (online)2.7812.9515.72
256BF16—44.3344.33
256Plain MXFP45.7215.0020.72
256PRQuant (online)6.0616.2722.33
4096BF16—583.91583.91
4096Plain MXFP479.51150.39229.89
4096PRQuant (online)79.84165.08244.92

End-to-end latency.

During decoding (Table 4), the maximum additional TPOT relative to plain MXFP4 is approximately 6.1% on 4B and 3.1% on 30B. Speedups over BF16 are 1.34×-1.43× and 1.21×-1.80×, respectively. During prefill (Table 3), TTFT increases by at most 3.3% on 4B and by 5.5%-38.7% on 30B relative to plain MXFP4. Compared with ARCQuant, PRQuant reduces TTFT by 4.5%-38.8% on 4B and by up to 14.9% on 30B.

These results quantify the costs of the accuracy improvement: PRQuant retains decoding efficiency close to plain MXFP4, and both prefill and decoding remain faster than BF16 across all reported configurations.

AuthorsOpenReview ↗

Response to Reviewer PGAK (2/3): Native MXFP4 Execution and Hardware Portability

Q2: Do PRQuant and plain MXFP4 both use native MXFP4 instructions on Ascend 950PR, and how might PRQuant's speedups and overhead change on NVIDIA Blackwell?

Ascend 950PR. Yes. Both the plain MXFP4 baseline and PRQuant execute FFN matrix multiplications through the native MXFP4 GEMM path on Ascend 950PR.

The two kernels differ only in the operands of this GEMM:

  • Plain MXFP4 multiplies MXFP4 activations by MXFP4 down-projection weights, with a reduction length of CC.

  • PRQuant quantizes the permuted activations once, appends the quantized tail blocks with their original scales, and multiplies the augmented activations by the offline-prepared MXFP4 matrix Q([W~down,RW])Q([\widetilde W_{\mathrm{down}},R_W]), with a reduction length of C+kC+k.

Section 4 specifies the conditions under which this is numerically equivalent to directly quantizing [Z~,Z~tail][\widetilde Z,\widetilde Z_{\mathrm{tail}}]: the concatenation boundaries must align with MXFP4 blocks, and the copied blocks must retain their original scales. Both CC and kk satisfy this alignment. The baseline and PRQuant use the same vllm-ascend 0.20.2 RC1 stack, run on the same Ascend 950PR system, and use the same MXFP8 attention configuration. The 5–10% operator latency gap on Qwen3-4B is precisely the difference expected when the reduction lengths of the same type of MXFP4 GEMM differ by 10.5%. If one path used native FP4 MMA while the other performed BF16 GEMM on dequantized operands, the latency difference would not exhibit this pattern.

Ascend 950PR exposes this operation through MX-type matrix instructions—MX-mode Mmad, with fp4x2_e2m1 / fp4x2_e1m2 inputs and one fp8_e8m0 scale per 32 elements along the KK dimension. Our kernels use this execution path. We also specify the instruction name in the revised manuscript so that confirming instruction consistency does not require inspecting the kernel source code.

Blackwell. We have not evaluated PRQuant on Blackwell and therefore do not directly extrapolate the Ascend speedups as measured Blackwell results. Blackwell provides native block-scaled MXFP4 matrix multiplication, and the algorithm itself is portable; once both methods use native FP4 MMA, the overhead has a clear upper bound.

Blackwell Tensor Cores implement MXFP4, with published dense throughput of 2×2\times FP8 and 4×4\times BF16. This leads to two observations.

1. Speedup over BF16. Our 2.6×2.6\times and 1.8×1.8\times operator speedups, together with the 1.211.21–1.80×1.80\times decoding speedups in Table 4, characterize Ascend 950PR rather than establish an upper bound for Blackwell. During large-batch, compute-bound prefill, if Blackwell MXFP4 GEMMs fully utilize the FP4 Tensor Cores, linear-layer speedups over BF16 can approach 4×4\times more closely than our Ascend kernels. Small-batch MoE decoding remains constrained by weight movement and tile sizes on both Blackwell and Ascend, so end-to-end decoding speedups over BF16 will be substantially below this peak. In our configuration, attention uses MXFP8 and the KV cache remains unchanged, which limits end-to-end speedups on either device.

2. Overhead relative to plain MXFP4. This overhead is determined by matrix shapes rather than the Ascend kernel. When both methods use the same type of MMA:

Compute-bound execution. Down-projection latency scales with (C+k)/C(C+k)/C, corresponding to approximately +10.5% for Qwen3-4B and +16.7% for Qwen3-30B. Across the three FFN GEMMs, the corresponding increases are approximately +3.5% and +5.6%, respectively.

Memory-bound decoding. Additional data movement occurs only in the down-projection, with a relative increase of k/Ck/C. This corresponds to approximately 2.4% of the Qwen3-4B parameter count and 3% of the active parameters in Qwen3-30B-A3B, which is why the overhead in Table 4 remains within 6%.

Tail copying. Tail copying only reads a contiguous k/Ck/C fraction of the already-quantized activations, and its data volume is negligible compared with that of GEMM. The Qwen3-30B TTFT results in Table 3 are measured on Ascend 950PR. Blackwell FP4 Tensor Cores provide dense throughput of 4×4\times BF16, and movement of the contiguous tail blocks and scales can overlap with the main loop. The same PRQuant formulation could therefore be faster on Blackwell: linear-layer speedups over BF16 could exceed the Ascend results and approach this 4×4\times peak more closely. Relative to plain MXFP4, only the longer reduction dimension would remain, corresponding to approximately +16.7% for the Qwen3-30B down-projection and a few percentage points end to end.

AuthorsOpenReview ↗

Response to Reviewer PGAK (3/3): Practical Relevance of Offline Quantization Costs

Q3: When are the offline quantization savings in Table 2 practically relevant, given that quantization is usually a one-time cost per checkpoint?

We thank the reviewer for pointing out that offline quantization costs can usually be amortized over subsequent serving. We agree: for a single checkpoint that is quantized once and then served for a long period, the reduction in offline cost reported in Table 2 is not the primary reason to choose PRQuant. The ongoing costs after deployment are the additional weight storage and expanded down-projection quantified in our response to Question 1, together with the prefill and decoding latency differences reported in Tables 3 and 4. In this scenario, choosing PRQuant over plain MXFP4 is primarily an accuracy–runtime tradeoff: in Table 1, average accuracy improves by 0.55 and 1.24 percentage points on Qwen3-30B and Qwen3-4B, respectively, with approximately 0–6% additional decoding latency relative to plain MXFP4.

Table 2 compares different offline quantization workflows that achieve comparable accuracy. On Qwen3-30B, the training-based and Hessian-based baselines require 7.4–9.1 hours and 22.9–59.2 GiB, whereas PRQuant completes quantization in 0.4 hours using 4.4 GiB. These reductions have practical significance in the following scenarios.

Repeated exports. When quantization is part of an iterative model-development cycle rather than the final step of a single release, the approximately 18–23× difference in quantization time per export has a substantial cumulative effect on preparation costs. Examples include continually updated post-training checkpoints, intermediate snapshots from RL or preference optimization, models fine-tuned for different tenants, and comparisons across quantization configurations. Algorithm 1 already includes a search over α\alpha; comparing additional values of kk or different layer quantization configurations introduces further calibration and export work. An overnight job lasting several hours may be acceptable once, but when it must be repeated for every updated checkpoint, export time affects iteration efficiency and delivery schedules.

Working memory during calibration and export. The reported 4.4 GiB is the peak accelerator working memory of the quantization job, not the serving memory or total memory requirement of the 30B model. One copy of the Qwen3-30B-A3B BF16 weights occupies 57 GiB. PRQuant neither constructs Hessians nor trains learnable transformations. Instead, it processes the targeted expert down-projections using shared calibration statistics, maintaining a small quantization working set.

In comparison, the 59.2 GiB required by MR-GPTQ and the 22.9–36.1 GiB required by methods with learnable transformations increase accelerator-memory requirements during quantization. With appropriate loading in chunks and host-side weight management, lower working-memory requirements facilitate local exports, CI jobs, and requantization after fine-tuning, lowering the resource barrier to producing checkpoints. This benefit occurs during model preparation.

Larger MoE models. Qwen3-30B-A3B already contains 48×128=614448 \times 128 = 6144 expert down-projections. As the number of matrices to process increases, the offline computation required for Hessian construction or learnable-transformation optimization accumulates. At this scale, an approximately 20× difference in quantization time and a working-memory difference of tens of GiB already affect device selection and resource budgeting for export jobs. Our measurements of these costs cover only the 30B model; this discussion identifies scenarios in which the costs become operationally significant, rather than reporting results for larger models.

Comparison with plain MXFP4. Plain MXFP4 does not require the search or training procedures described above and is therefore not included in Table 2. PRQuant’s 0.4 hours represents an offline cost paid to improve accuracy over that baseline, not a cost that every MXFP4 model must incur. On Qwen3-30B, the average accuracies of BATQuant, FlatQuant, and MR-GPTQ are all within approximately 0.4 percentage points of PRQuant. Thus, for a single long-lived checkpoint, accuracy and runtime overhead remain the deciding factors. Table 2 demonstrates that the accuracy improvements in Table 1 can be obtained through a quantization workflow with low time and working-memory costs.

The revised manuscript will explicitly define the memory column in Table 2 as peak accelerator working memory during quantization, clarify that the table compares the offline quantization costs of PRQuant against methods using learnable transformations or Hessians, and direct readers to the additional-storage analysis in Question 1 and the runtime latencies in Tables 3 and 4 when assessing post-export deployment costs.

We thank the reviewer again for the thoughtful and constructive questions.

Reviewer_1RWnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PRQuant, a post-training quantization (PTQ) method and execution scheme tailored for block-scaled formats and MXFP4 hardware operators. It shifts residual compensation from the activation side at runtime to the weight side at calibration time, so that inference only requires copying k activation columns. The method is applied to FFN down-projection layers. Evaluations on the Ascend 950PR report a ~2× layer-wise speedup over BF16 with latency close to plain MXFP4, while improving accuracy over MXFP4.

Critical strengths
  • The problem definition and the proposed methodology are clearly stated and well-structured.

  • The experiments cover accuracy, kernel-level speedup, and end-to-end TTFT/TPOT across batch sizes and context lengths, giving a comprehensive picture of the method's real-world behavior.

Critical weaknesses

– The runtime overhead reduction relies on absorbing the permutation and scaling into adjacent weights. This is only possible for layers whose input is produced element-wise by a preceding projection, i.e., the down-projection of a gated FFN. It does not extend to attention projections or non-gated FFNs. For the same reason, the accuracy comparison is on a partial configuration (only down-projections modified, attention in MXFP8), which limits what Table 1 can show.

– Figure 2 shows that AWQ scaling and residual compensation are the key factors of error reduction, and that permutation contributes little numerically. The paper explains how these two existing components can be merged to run on a single MXFP4 GEMM, but beyond this arrangement there is no new component.

Questions for authors

• The paper introduces two activation-side variants, online copy and offline copy, but evaluates only the online copy, which incurs a runtime tail-copy cost. Can the authors provide latency numbers for both variants and explain why the online copy was chosen?

• Given that the proposed method cannot be fused into other layers, what would the end-to-end accuracy be if PRQuant were applied (without fusion) to the entire model? This would clarify how much of the remaining accuracy gap is attributable to the down-projection restriction, and would allow a comparison with baselines under the same full-model setting.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1RWn (1/3): Online vs. Offline Copy

We thank the reviewer for the careful review and constructive feedback. In the revised manuscript, we have added latency comparisons of the two activation-copy variants and controlled analyses of permutation in Appendix B.1 and Appendix C, respectively. We address these concerns in the following three responses.

Q1: How do online and offline copy compare in latency, and why was online copy chosen?

A1: Thank you for this suggestion. Our choice of Online-copy considers both complete FFN latency and weight storage: although Offline-copy can reduce down-projection latency, Online-copy generally achieves lower complete FFN latency in our experiments while avoiding additional Up/Gate weight storage and computation. We have added latency comparisons between the two variants and a breakdown of online preprocessing in the appendix.

Tradeoff between down-projection and complete FFN latency. Across the tested range of MM, Offline-copy reduces down-projection latency by up to 12.06% relative to Online-copy. However, Offline-copy requires expanding the Up/Gate output dimensions from CC to C+SC+S. For our Qwen3-4B configuration, this increases both Up/Gate GEMM arithmetic and weight storage by 10.53%, together with additional gated-activation computation.

We therefore further compared complete FFN latency across 15 configurations, covering M=1,2,4,...,16384M=1,2,4,...,16384. Online-copy is faster, reducing latency by 0.19%–8.61% relative to Offline-copy. Table R1 presents representative results; the complete sweep is provided in the appendix.

Table R1. Representative latency comparison between Online-copy and Offline-copy on Qwen3-4B. All latencies are in μs\mu\mathrm{s}. Down denotes the down-projection operator, while FFN includes Up/Gate, gated activation, and down-projection.

MMDown: OnlineDown: OfflineFFN: OnlineFFN: Offline
113.1813.1937.5039.34
1614.3714.0539.5240.63
25620.8619.2160.1261.25
51232.8329.3096.1795.10
4096245.15231.93777.57817.12
163841016.34995.223075.533300.97

To further quantify online Tail copy overhead, Table R2 compares the preprocessing latencies of Plain MXFP4 and PRQuant. Plain MXFP4 preprocessing performs activation quantization only, whereas PRQuant additionally copies the already-quantized tail blocks and their original scales, with both operations fused into one kernel. Thus, the additional preprocessing operation is the copying of quantized tail blocks and scales. Preprocessing latency is essentially unchanged at M=16M=16. At M=256M=256 and M=4096M=4096, the measured preprocessing latency differences are only 0.340.34 and 0.33 μs0.33\,\mu\mathrm{s}, respectively, corresponding to 1.64%1.64\% and 0.14%0.14\% of the complete plain MXFP4 down-projection latency.

These show that PRQuant and Plain MXFP4 achieve comparable online preprocessing performance, indicating that fused online Tail copy introduces only a small additional overhead. This mainly results from PRQuant placing residual computation on the offline weight side and absorbing the permutation offline to avoid runtime irregular gathers, leaving only contiguous tail copying in the online stage.

Table R2. Online preprocessing latency on Qwen3-4B, excerpted from Table 7 in Appendix B.2.

(M)Plain MXFP4: quantizationPRQuant: quantization + tail/scale copyDifference
162.792.78−0.01
2565.726.06+0.34
409679.5179.84+0.33

Finally, the end-to-end experiments using Online-copy show that PRQuant retains decoding latency close to Plain MXFP4 (Table 4). Considering complete FFN computational efficiency and storage overhead together, these results motivate our choice of Online-copy as the default implementation.

AuthorsOpenReview ↗

Response to Reviewer 1RWn(2/3): Permutation and Method Contributions

Q2: Given the modest numerical contribution of permutation in Figure 2, what is its role, and where does PRQuant’s novelty lie?

A2: We thank the reviewer for their attention to the role of permutation and the novelty of our method. The key contribution of permutation is to organize scattered channels requiring compensation into a layout that supports offline fusion and contiguous access. It also redistributes quantization error, concentrating a larger share in channels that can subsequently receive residual compensation. We agree that scaling and residual compensation provide the main numerical gains in Figure 2, but permutation contributes in the following two ways.

Numerical contribution. MXFP4 uses shared scales within blocks, so channel regrouping affects quantization error. Appendix C presents a controlled comparison before residual compensation, with the scaling factors and selected channel set fixed. Table 9 shows that the selected channels’ error share increases from 33.15% to 39.33% after permutation: although these channels constitute only 16.7% of all channels, they account for approximately 40% of the remaining weight quantization error after permutation. Meanwhile, weight NRMSE decreases in all 48 layers, with an average relative reduction of 4.19%. Thus, permutation reduces the overall error while concentrating the remaining error in channels eligible for subsequent residual compensation.

Runtime overhead and methodological contribution. PRQuant absorbs permutation offline into the adjacent up/gate weights, so the selected channels directly form a contiguous tail. This avoids irregular gathering at runtime, allowing online activation augmentation through contiguous tail-block copying alone. Table 7 in the appendix shows that this tail copying introduces only a small additional preprocessing cost.

ARCQuant performs activation channel reordering and constructs and quantizes activation residuals online. In contrast, through outlier migration, PRQuant moves residual computation and quantization to the offline weight side, making online processing more lightweight. In Figure 3, PRQuant achieves approximately 1.8× and 1.6× operator speedups over ARCQuant on Qwen3-4B and Qwen3-30B, respectively.

Therefore, our contribution is the joint design of error redistribution, offline weight residual compensation, and native low-precision execution tailored to MXFP4’s shared-scale properties and the gated FFN structure. Permutation is a key structural design that connects numerical compensation with efficient execution.

AuthorsOpenReview ↗

Response to Reviewer 1RWn (3/3): Scope and Down-Projection Focus

Q3: What is the end-to-end accuracy of applying PRQuant to the full model without fusion, and how much of the remaining accuracy gap is attributable to restricting compensation to down-projections?

A3: We thank the reviewer for raising this important question. Our decision to prioritize down-projections reflects a joint consideration of quantization sensitivity, accuracy recovery, and hardware execution efficiency. Our goal is to recover accuracy lost under W4A4 quantization with modest additional overhead while retaining the efficiency of native MXFP4 execution.

First, a systematic study found that MXFP8 generally maintains near-lossless accuracy, whereas accuracy recovery under MXFP4 remains challenging (Zhang et al., ACL 2026). This motivates our focus on recovering W4A4 accuracy with low overhead.

Second, down-projections are sensitive components that warrant prioritizing the compensation budget. The analysis and experiments of Xie et al. (Findings of ACL 2025) show that down-projections are more sensitive to quantization than up-projections in the MoE models they studied. Zhou et al. (ICML 2025, FloE) similarly observe more pronounced accuracy degradation from quantizing down-projections in their low-bit experiments, while up-projections are comparatively less sensitive. These findings support prioritizing down-projections.

Our experiments further demonstrate that compensating only these projections yields clear end-to-end benefits. Table 1 shows that PRQuant improves average accuracy over plain MXFP4 by 0.55 and 1.24 percentage points on Qwen3-30B and Qwen3-4B, respectively, recovering 43.3% and 39.9% of the accuracy gap to BF16 and achieving the highest average accuracy among the compared quantized methods. Meanwhile, PRQuant adds low-bit weight residuals only for selected channels. Amortized over the original down-projection weight elements, the average storage, including E8M0 shared scales, is approximately 4.96 and 4.70 bits, respectively, while computation still uses native MXFP4 GEMM. These accuracy gains therefore require only a limited compensation budget, without promoting the entire projection to W8A8.

More importantly, the gated FFN structure enables efficient execution of this compensation. Table 7 in the appendix shows that tail copying introduces only a small additional preprocessing cost. Figure 3 and Table 4 further demonstrate the operator-level efficiency advantage over ARCQuant and end-to-end decoding efficiency close to plain MXFP4, respectively. Down-projections thus combine meaningful accuracy recovery with structural properties suitable for low-overhead execution.

We thank the reviewer again for encouraging us to clarify the rationale and scope of this design choice.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.