acceptodds
Under review as a conference paper at ICLR 2027

Gamma-Aware Block Rotations for MXFP4 LLM Quantization

Abstract

Rotation-based post-training quantization reduces outliers by inserting orthogonal transforms and fusing them into adjacent weights. For the residual-stream rotation R1, however, the learned scaling parameter of RMSNorm does not commute with R1 in general, so existing offline methods first fold it into the weights of subsequent linear layers. This operation can amplify magnitude disparities within quantization blocks and degrade the quantized model. We propose Gamma-aware Block Rotation (GaBR). GaBR first permutes channels to group similar values within quantization blocks. It then decomposes the scaling parameters into a block-constant retained in RMSNorm and a residual fused into weights, and alternately optimizes and a block-diagonal R1. With a block-diagonal rotation, retaining one scale per 32 channels in RMSNorm remains exactly equivalent in full precision. Extensive experiments on seven Llama and Qwen models under MXFP4 W4A4 quantization show that GaBR consistently improves upon existing rotation-based PTQ methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, MXFP4, rotation-based quantization, block-scaled quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JaRBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that residual-stream rotation R1 requires fully fusing the RMSNorm scale γ into the following weights, which can enlarge within-block magnitude disparities and increase MXFP4 quantization error. The proposed GaBR permutes channels by their cross-layer mean |γ| so that similarly scaled channels fall into the same 32-channel block, factors γ into a block-constant scale G retained in RMSNorm and a residual D fused into the weights, which stays exactly equivalent under 32×32 block-diagonal rotations, and alternately optimizes G and the block rotation. Applied to OSTQuant and PeRQ, GaBR is evaluated on seven Llama and Qwen3 models under MXFP4 W4A4 and reports improvements in WikiText-2 perplexity and seven-task average accuracy over its parent methods on most models.

Critical strengths

The paper identifies a concrete, format-specific issue: fusing the RMSNorm scale γ into downstream weights can increase MXFP4 quantization error. This observation is relevant to any offline rotation method targeting MXFP4.

The proposed factorization is exactly function-preserving and adds no inference-time operators, since a block-constant scale commutes with block-diagonal rotations. This makes the method easy to plug into existing pipelines such as OSTQuant and PeRQ.

Critical weaknesses

Compared with OSTQuant, GaBR-OST simultaneously applies block rotation, channel permutation, and G, but no result isolates the effect of block rotation alone. The ablations are also limited to four and three models, and their results are inconsistent. It is therefore unclear whether the improvements come from GaBR’s components or from block rotation itself, which weakens the paper’s main claim.

Questions for authors
  1. Tables 4 and 7 show that the benefit of the retained block scale G is not consistent across models. Since the Γ = GD factorization ablation is conducted on only four models, please extend it to all seven models and explain in which cases retaining G is beneficial.

  2. Tables 5 and 8 show that the effect of the gamma-aware permutation is not consistent across models. Since the gamma-permutation comparison is conducted on only three models, please report the results for all seven models.

  3. Compared with OSTQuant + Qronos, GaBR-OST simultaneously introduces the 32 × 32 block-diagonal R1, the gamma-aware permutation, and the retained block scale G, so the source of the improvement cannot be identified. Please add results for OSTQuant + Qronos with only the block-diagonal R1 to separate the contribution of block rotation itself from that of the GaBR components.

  4. DuQuant++ is excluded from the main comparison (Table 3) because its additional online activation transforms result in a different inference cost, but the paper reports no measurements of latency, FLOPs, or quantization time. Please measure the inference latency and offline quantization time of GaBR and DuQuant++ on the same hardware.

4 comments
AuthorsOpenReview ↗

Extended Ablations and Efficiency Evaluation

Thank you for your helpful suggestions on the ablations and efficiency evaluation. We provide additional results and clarify the experimental settings below.

Q1. Effect of retaining GG across models

We now report the GG ablation on all seven models under W4A4 MXFP4 with Qronos.

ModelWithout GG: PPL ↓ / Mean7 (%) ↑With GG: PPL ↓ / Mean7 (%) ↑
Llama-2-7B5.916 / 63.225.870 / 63.04
Llama-3-8B7.188 / 69.627.177 / 69.62
Llama-3.2-1B12.141 / 50.8312.073 / 52.11
Llama-3.2-3B8.956 / 63.008.909 / 62.66
Qwen3-1.7B18.059 / 55.9917.073 / 59.18
Qwen3-4B14.669 / 66.3113.953 / 66.88
Qwen3-8B10.591 / 71.8210.059 / 72.26

Retaining GG improves performance on most of the tested models.

Q2. Gamma-permutation comparison across models

We expanded Table 5 under W4A4 MXFP4 with Qronos to cover all four orderings on all seven models.

ModelIdentityZigzagGamma-MassDiffGamma-sort
Llama-2-7B5.816 / 64.165.852 / 63.935.874 / 64.065.870 / 63.04
Llama-3-8B7.175 / 69.747.231 / 69.427.226 / 69.387.177 / 69.62
Llama-3.2-1B12.099 / 50.7312.203 / 50.8912.166 / 51.4812.073 / 52.11
Llama-3.2-3B8.979 / 62.438.933 / 62.688.950 / 62.888.909 / 62.66
Qwen3-1.7B18.101 / 56.8917.708 / 57.3617.439 / 57.3117.073 / 59.18
Qwen3-4B13.990 / 66.3813.407 / 67.4714.071 / 67.9113.207 / 67.04
Qwen3-8B9.829 / 71.3810.071 / 72.0110.094 / 72.2310.059 / 72.26

Each entry reports PPL ↓ / Mean7 (%) ↑.

Although performance varies across orderings, Gamma-sort remains competitive across models and achieves the best Mean7 on several models.

Q3. OSTQuant + Qronos with only block-diagonal R1R_1

We added the requested control by replacing only dense R1R_1 with independent 32×3232\times32 orthogonal blocks in OSTQuant + Qronos. The control keeps the original 100-step training procedure, channel order, and full γ\gamma fusion, without Gamma-sort or retained GG.

ModelOSTQuant + QronosOSTQuant + Qronos: block R1R_1 onlyGaBR-OST + Qronos
Llama-2-7B5.986 / 62.665.857 / 63.195.870 / 63.04
Llama-3-8B7.242 / 69.717.152 / 69.447.177 / 69.62
Llama-3.2-1B12.358 / 50.8912.247 / 51.4112.073 / 52.11
Llama-3.2-3B9.131 / 62.868.923 / 62.938.918 / 62.97
Qwen3-1.7B17.751 / 57.5517.437 / 56.6217.173 / 56.52
Qwen3-4B14.239 / 66.8213.680 / 65.1413.207 / 67.04
Qwen3-8B10.301 / 72.5810.091 / 70.9410.059 / 72.26

Each entry reports PPL ↓ / Mean7 (%) ↑.

Full GaBR achieves lower PPL on five of seven models and higher Mean7 on five of seven, indicating benefits beyond block rotation.

Q4. Same-hardware latency and offline quantization time

We evaluated Qwen3-1.7B on the same hardware.

Offline timing (min) ↓DuQuant++GaBR-OST + Qronos
Weight quantization/reconstruction6.9929.51
Complete offline preparation8.3842.59

Although GaBR requires longer preparation, this cost is incurred entirely offline. These timings include different reconstruction backends, rather than isolating GaBR’s incremental cost.

GaBR’s residual-stream R1R_1 is fused into the weights offline, adding no independent online operator. We measured additional online-transform overhead using the same BF16 model graph and KV-cache settings, excluding down-projection transforms from both paths.

Batch-1 metricOffline-R1R_1 referenceDuQuant++ online transforms
TPOT (ms/token) ↓17.1221.15
Decode throughput (tokens/s) ↑58.4247.27

Online timing uses a 512-token input prefix and 32 cached one-token decoding steps. TPOT excludes prefix prefill and reports the median of five trials. This software comparison measures additional online-transform overhead, not native W4A4 deployment latency.

Thanks for the additional experiments. The additional results show that G lowers PPL on all seven models. However,m the effect of the permutation is still inconsistent across models, and on some models block rotation alone gives similar or better results, so the contribution of the GaBR components seems limited. Regarding DuQuant++, GaBR avoids online transforms, but its offline preparation takes about five times longer, and its Mean 7 is still lower on most models. The reported inference advantage is also measured in BF16 rather than in W4A4. I will keep my score.

Thank you for your feedback. GaBR fuses its R1R_1 rotation offline, adding no online R1R_1 computation to its parent PeRQ or OSTQuant pipeline. Our BF16 comparison isolates online-transform overhead, while efficient native W4A4 kernels can provide further inference acceleration.

Although offline preparation takes about five times longer in our comparison, this one-time cost is amortized over deployment and does not affect inference latency.

Your subsequent comment discusses RFS, EWGS, and Fourier-derived gradients, which are not part of our submission. Could you please check whether it was intended for another paper?

Sorry for the confusion. My previous comment about RFS, EWGS, and Fourier-derived gradients was meant for another paper and was posted here by mistake. Please disregard it.

Reviewer_7z6KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a novel strategy to balance block rotations and their corresponding RMSNorm gamma scaling aimed to improve the quantization performance for block-wise quantization. The main idea consist of a re-parametrization of the gamma scale, into a per-group scalar, which commutes with the rotation and intra-group component, which needs to be fused into the model weights. Combined with a smart policy for channel permutation, and an alternating optimization procedure the paper aims to demonstrate the effectiveness of the proposed Gamma-aware Block Rotation (GaBR) method.

Critical strengths
  • The paper provides a simple method that can be applied to any block-rotation-based quantization method, which is quite relevant considering the increasing popularity of MXFP/NVFP4 formats.
  • Overall the paper is well written and easy to follow. The Gamma re-parametrization is simple but quite elegant.
Critical weaknesses

Main concerns

  • Although the proposed method is intuitive and simple, the experimental result report very small improvements when compared to existing methods, with gains in the order of 10−310^{-3} to 10−110^{-1} in PPL and ~ 1% on task accuracy in table 4 and 5. Considering that no standard deviation is reported, it is quite difficult to assess the significance of the reported results, which undermines the central contribution of the paper.
  • The negative impact of fusing the γ\gamma parameter is presented as a major issue and a motivating factor for the paper. However, the deterioration in performance reported in Table 2 seems quite limited and Table 1 reports only an illustrative example. Additional intuition or empirical motivation would further strengthen the submission.
  • The novelty of the proposed approach is quite limited. I don't believe this factor alone is sufficient to justify rejection, but I would expect more substantial empirical evidence to support the proposed method and claims.

Minor issues

  • Table 1 reports a very specific example. This is not necessarily representative of the problem arising from fusing γ\gamma into the weight. As it stands, the table seems more like a cherrypicked example and does not provide meaningful support to the claims, especially considering that an illustrative example is already provided in Table 1.
  • The "Fuse-γ\gamma then rotate" Illustration is reported in both Figure 2 and Figure 3.
  • Description for methods such as PerQ* and PerQ$^\dagger$ is not contained in the paper. A short paragraph to summarize the difference would improve clarity.
  • The experimental setup for Table 5 is quite unclear.
Questions for authors
  1. What is the effect of the proposed alternating minimization procedure? How many alternative iterations are required for convergence and how does the error evolve as a function of the number of iterations?
  2. How data-dependent is the proposed GaBR method? Considering that the "Phase R" includes a Cross-entropy term, it would be interesting to assess GaBR performance when calibrated on a different dataset.
  3. Can the authors clarify the experimental setup used to produce Table 5? Does it include the iterative alternating minimization for GG and RR?
  4. Are the gains observed using GaBR consistent with other block-based quantization formats such as NVFP4 or other block size?
4 comments
AuthorsOpenReview ↗

Additional Experiments and Clarifications

Thank you for your constructive feedback and detailed questions. We address your concerns and provide additional experimental evidence below.

W1. Effect sizes and missing standard deviations

The gains in the original Tables 4 and 5 are modest in absolute terms, but retaining GG and using Gamma-sort recover 10.8% and 19.4%, respectively, of the average Mean7 gap between BF16 and their corresponding W4A4 MXFP4 controls. On Qwen3-8B, retaining GG reduces PPL from 10.591 to 10.059, recovering 61.5% of the increase over BF16 (9.726).

To address the missing standard deviations, we report the mean ± sample standard deviation for each configuration across three matched training seeds.

Table 4: Effect of retaining GG

ModelGaBR-OST with GG: PPL ↓GaBR-OST with GG: Mean7 (%) ↑Without GG: PPL ↓Without GG: Mean7 (%) ↑
Qwen3-1.7B17.768 ± 0.86158.514 ± 0.72918.012 ± 0.57857.357 ± 1.184
Qwen3-8B10.271 ± 0.11871.594 ± 0.59410.335 ± 0.16771.673 ± 0.472

W2–W3. Empirical motivation for avoiding full γ\gamma fusion

Full γ\gamma fusion can increase within-block magnitude variation, worsening shared-scale MXFP4 quantization. Table 2 already shows increased PPL and logit MSE; we additionally measured weight NMSE across all seven models using the same block-32 quantizer and full-precision reference.

ModelBefore fusion: NMSE ↓After fusion: NMSE ↓NMSE increase
Llama-2-7B0.0133810.0136091.70%
Llama-3-8B0.0138120.0147166.55%
Llama-3.2-1B0.0138610.0148327.01%
Llama-3.2-3B0.0136470.0143655.26%
Qwen3-1.7B0.0148290.01912828.99%
Qwen3-4B0.0139880.0143002.23%
Qwen3-8B0.0145980.01815024.33%

Fusion increases weight NMSE on all seven tested checkpoints by 1.70–28.99%, supporting the motivation beyond Table 1’s illustrative example.

Q1. Alternating optimization and convergence

On Qwen3-1.7B, we fixed the budget at 90 RR steps and 30 final SS steps, varying the number of R/GR/G rounds.

R/GR/G roundsRR steps per roundFinal validation KL ↓PPL ↓Mean7 (%) ↑
1900.503317.13358.00
2450.520418.65459.53
3300.460117.07359.18
5180.497717.30658.34

Three rounds achieved the lowest validation KL and PPL, supporting this practical setting.

Q2. Sensitivity to the transform-training corpus

We varied only the transform-training corpus, keeping the training budget, teacher-KL objective, Qronos calibration, and evaluation fixed.

ModelTraining corpusDense R1R_1: PPL ↓ / Mean7 (%) ↑GaBR: PPL ↓ / Mean7 (%) ↑
Qwen3-1.7BWikiText-218.643 / 57.7517.073 / 59.18
Qwen3-1.7BGSM8K19.342 / 56.5417.818 / 58.41
Qwen3-8BWikiText-211.074 / 70.7610.375 / 72.27
Qwen3-8BGSM8K10.966 / 71.4110.587 / 72.42
Llama-3.2-1BWikiText-212.262 / 51.0612.084 / 51.31
Llama-3.2-1BGSM8K12.354 / 50.1112.471 / 51.18

GaBR achieves higher Mean7 on all three models with either training corpus, while lowering PPL in five of six comparisons.

Q3. Training protocol in Table 5

For each model, we vary only the initial channel permutation. Once determined, the permutation remains fixed throughout training, and all four orderings follow the same subsequent G/R/SG/R/S optimization schedule, quantization procedure, and evaluation settings.

Q4. Transfer to NVFP4

We followed the same experimental protocol, replacing MXFP4 with NVFP4 and adapting GaBR’s rotation and scale blocks to size 16.

ModelOSTQuant + QronosGaBR-OST + Qronos
Llama-3.2-1B11.467 / 51.4511.391 / 51.77
Llama-3.2-3B8.870 / 63.138.607 / 62.80
Qwen3-1.7B21.626 / 56.9418.925 / 56.67
Qwen3-4B14.924 / 66.6214.484 / 67.97
Qwen3-8B10.660 / 70.8910.312 / 71.86

Each entry reports PPL ↓ / Mean7 (%) ↑. OSTQuant uses its original 100-step training schedule; GaBR uses three 30-step R/GR/G rounds followed by 30 SS steps. These are software quantization results, not native deployment measurements.

GaBR shows an overall advantage under NVFP4, supporting its applicability beyond MXFP4.

Reviewer_7z6KOpenReview ↗

Response to the Authors

I want to thank the authors for their detailed response and the additional experiments which help me understand the applicability of their approach.

Regarding the standard deviation analysis, my main concern is that, especially on Qwen 3 8B, the effect of GG is well within standard deviation, indicating a very minor effect. This is considering that, from the reported table, Qwen 3 1.7B and Qwen 3 8B are the models that are most sensitive to the fusion of the γ\gamma from the RSMNorm. This makes me think that the impact will be even lower on the other reported models.

For this reason I currently keep my score, but I am open to revise my evaluation whenever the authors provides sufficient evidence by reporting standard deviation for more models (e.g. LLama family).

Extended Three-Seed Analysis Across All Seven Models

Thank you for your follow-up. Following your suggestion, we extended the retained-GG analysis to all four Llama models and Qwen3-4B, completing the seven-model comparison using three matched training seeds.

ModelWith GG: PPL ↓Without GG: PPL ↓With GG: Mean7 (%) ↑Without GG: Mean7 (%) ↑
Llama-2-7B5.902 ± 0.0345.961 ± 0.02863.326 ± 0.10163.522 ± 0.562
Llama-3-8B7.212 ± 0.0147.215 ± 0.06869.278 ± 0.60969.183 ± 0.572
Llama-3.2-1B12.073 ± 0.04112.164 ± 0.04651.442 ± 0.61351.145 ± 0.132
Llama-3.2-3B8.920 ± 0.0458.940 ± 0.03862.840 ± 0.21062.584 ± 0.453
Qwen3-1.7B17.768 ± 0.86118.012 ± 0.57858.514 ± 0.72957.357 ± 1.184
Qwen3-4B14.169 ± 0.20114.982 ± 0.36066.823 ± 0.67866.727 ± 0.618
Qwen3-8B10.271 ± 0.11810.335 ± 0.16771.594 ± 0.59471.673 ± 0.472

Retaining GG achieves lower average PPL on all seven models and higher average Mean7 on five of seven models.

Reviewer_7z6KOpenReview ↗

Response to the Authors

I thank the authors for providing the standard deviation for their main results. The standard deviations indicate that the effect of the presented method passes standard significance tests only on a few models. Given the limited impact of the key contribution I will keep my score, but I will consider the author insightful responses when discussing with the other reviewer and the AC.

Reviewer_1ogxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gamma-aware Block Rotation (GaBR) for W4A4 MXFP4 quantization. It observes that fully folding RMSNorm’s channel-wise scale into downstream weights can increase within-block variation and quantization error. GaBR groups channels with similar scale magnitudes, factorizes the RMSNorm scale as ΓP = G D, retains the block-constant component G in RMSNorm, and fuses only D into the weights. Since G commutes with the aligned block-diagonal rotation, full-precision equivalence is preserved. Experiments on seven Llama and Qwen models show generally improved perplexity over PeRQ and OSTQuant variants.

Critical strengths
  1. Identifies a concrete issue in offline rotation-based MXFP4 quantization.
  2. The factorization is simple, mathematically exact, and easy to integrate.
  3. Evaluates two parent frameworks across seven models.
  4. Includes useful component and permutation ablations.
Critical weaknesses
  1. The core technical novelty is too limited and straightforward to justify acceptance.
  2. The contribution is relatively incremental: gamma sorting, block-wise scaling, and alternating rotation optimization are straightforward extensions of existing scaling and rotation methods.
  3. Improvements are not fully consistent across downstream tasks, and several PPL gains are small.
Questions for authors

See Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Contributions and Empirical Evidence

Thank you for reviewing our work and sharing your comments. We clarify our contributions and address your concerns below.

W1. Core technical novelty

Many quantization methods improve accuracy through techniques such as rotations, codebook design, and scaling; their contributions often lie in the distinct observations and problems that motivate these techniques. Our paper focuses on the impact of RMSNorm scale fusion in global residual-stream rotation and investigates it through both mathematical analysis and controlled experiments.

Specifically, Introduction paragraphs 2–3 explain why existing offline R1R_1 pipelines first fuse RMSNorm’s learned scale into downstream weights and why this can harm shared-scale MXFP4 quantization. We observed this fusion-before-rotation procedure in the OSTQuant and PeRQ implementations we examined. Sections 3.1–3.2 analyze the mechanism, Table 1 illustrates how fusion distorts within-block magnitudes, and Table 2 demonstrates the resulting increases in PPL and logit MSE. Thus, our contribution is not simply another rotation or scaling operation, but identifying, analyzing, and addressing the quantization consequences of this design choice. We will revise the wording and organization of the main text to make this contribution clearer.

W2. Relationship to existing sorting, scaling, and rotation methods

Our proposed gamma-aware sorting and block-rotation design are motivated by two observations developed in Sections 3 and 4:

  1. Full γ\gamma fusion can worsen shared-scale quantization. With a generic dense rotation RR, the channel-wise scale generally does not commute with the rotation: RΓR⊤≠ΓR\Gamma R^\top \ne \Gamma. Existing offline pipelines therefore fully fuse γ\gamma into downstream weights to preserve exact equivalence, but Tables 1–2 show that this can increase within-block magnitude variation and quantization error. Motivated by this observation, Section 4.1 groups channels by cross-layer mean ∣γ∣|\gamma| to reduce variation in the residual scale factors. This sorting targets RMSNorm scale fusion, rather than redistributing outliers as in DuQuant or balancing activation-derived block mass as in PeRQ.
  2. Aligned block rotations allow exact partial fusion. Section 4.2 shows that a block-constant scale commutes with its aligned rotation: (gbI)Rb=Rb(gbI)(g_b I)R_b=R_b(g_b I). We therefore factor ΓP=GD\Gamma_P=GD, retain GG in RMSNorm, and fold only the signed residual DD into the weights, preserving full-precision equivalence while keeping R1R_1 offline. Building on this construction, Section 4.3 searches GG under hard MXFP4 at fixed RR, compensates through DD, and then updates RR at fixed GG. The sorting, block rotation, and scale optimization thus work together to realize this commutation-based partial-fusion design.

W3. Modest PPL gains and mixed downstream results

Existing MXFP4 methods already preserve much of BF16 performance, leaving relatively limited room for further improvement. GaBR improves upon these methods by reducing the remaining quantization gap. The table below reports the average gap to BF16 across the seven models in Table 3 and the percentage recovered by GaBR.

Matched comparisonMetricBaseline gap to BF16GaBR gap to BF16Quantization gap reduction (%) ↑
PeRQ$^*$ → GaBR-PeRQPPL1.3331.2337.5%
PeRQ$^*$ → GaBR-PeRQMean7 (pp)6.784.0640.1%
OSTQuant + Qronos → GaBR-OSTPPL1.1020.74132.8%
OSTQuant + Qronos → GaBR-OSTMean7 (pp)4.023.893.3%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.