acceptodds
Under review as a conference paper at ICLR 2027

OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Abstract

NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration, but preserving accuracy requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97% of vendor NVFP4 prefill throughput on the measured workloads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, NVFP4, Dither, Diagonal Smoothing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gziPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents OSFP4, a post-training quantization method for NVFP4 LLM inference. It jointly optimizes a diagonal smoothing transformation and block scales to reduce matrix-product error under either RTN or GPTQ-style SIC. A multiplicatively dithered FP4 quantizer provides a differentiable surrogate loss, followed by discrete E4M3 scale selection. Experiments on Llama, Qwen, and Gemma models evaluate accuracy, perplexity, ablations, and throughput. OSFP4 generally improves over NVFP4 baselines while retaining 94–97% of vendor NVFP4 prefill throughput on the tested workloads.

Critical strengths

The problem is practically important. The paper focuses on NVFP4, which is directly supported by NVIDIA hardware. The method is integrated into vLLM, and the overhead introduced by quantization is discussed, with some of its extra computation fused into normalization layers.

The paper's evaluation is thorough. The authors test several model families, quantization settings, and error metrics, and also report real B200 throughput. OSFP4 improves over the main NVFP4 baselines while retaining 94–97% of the vendor implementation’s prefill throughput.

Critical weaknesses

Presentation and Experimental Organization. The paper is generally understandable but harder to follow than necessary. The full OSFP4 pipeline is split across several sections and appendices, without a clear overview of the randomized surrogate, scale optimization, and deployment procedure. The main experimental section is also overly compressed, with most setup details, ablations, additional models, and throughput results deferred to the appendix. A high-level algorithm and a more self-contained discussion of the experiments would substantially improve readability.

The contribution of the dithered surrogate is not directly ablated. The paper shows that the surrogate closely matches the error of the corresponding randomized quantizer, but does not compare it against alternative optimization objectives while holding the smoothing, scale search, and rounding procedure fixed. Therefore, it remains unclear how much of the final improvement is specifically due to the proposed dithered formulation.

Questions for authors
  • The paper shows that the dithered surrogate predicts the randomized quantization error well, but it is less clear whether it improves the final deterministic model. Could the authors compare it with another optimization objective while keeping the final scale search and RTN/SIC procedure unchanged?

  • In the W4A4 setting, the activation scales ΓX\Gamma_X are optimized together with A, but are discarded at deployment, where absmax-to-6 scaling is used instead. Could the authors explain why the optimized A should still work well with this different scaling rule?

  • The throughput results compare OSFP4 with vendor NVFP4 and an A=I control, but not with the main methods used in the accuracy comparison. Could the authors provide same-platform throughput results for representative baselines so that the accuracy–throughput trade-off can be assessed fairly?

2 comments

Thank you for highlighting the readability issues. This was a recurring comment from all reviewers, and we address it in our general response above. We are grateful for the detailed account to what exactly should be improved in the presentation, and have done our best to address them in the revision.

Questions

  1. Alternative objectives with fixed scale search and RTN/SIC:

    The reviewer is questioning the importance of using the smooth relative MSE function ϕ\phi rather than the exact relative MSE function ϕexact\phi_{\mathrm{exact}}. First, we would like to point out that no prior work has used either of those expressions for the relative MSE function, so a big part of the innovation here is simply to recognize that the approximate MatMul loss with diagonal smoothing takes the form of our equations (3)-(4), regardless of whether it is with ϕ\phi or with ϕexact\phi_{\mathrm{exact}}.

    Conceptually, the starting point of our derivation is eq. (1), where we assume that ZZ is statistically independent of yy. This simplified model is what enables the derivation of the approximate loss functions. Without randomization (in the form of multiplicative dither), modeling ZZ as statistically independent of yy is strange, because ZZ is a deterministic function of yy. On the other hand, under the multiplicative-dither quantizer, ZZ is stochastic even given yy. In fact, for $y\in\mathcal{I}_{\mathrm{FP} 4} $ we have that ZZ is statistically independent of yy (see Ordentlich & Polyanskiy (2026a)). Though it is not exactly zero-mean. This is the main reason to introduce the multiplicative-dither quantizer - to give a better justification for the additive noise model.

    It terms of optimization, in Appendix J (Table 12) we added a ϕexact\phi_{\mathrm{exact}} ablation, replacing ϕ\phi with ϕexact(x)=(x−QFP4(x))2/x2\phi_{\mathrm{exact}}(x)=(x-Q_{\mathrm{FP4}}(x))^2/x^2

    in the loss function. The optimization and final scale search remain unchanged. The dithered objective generally achieves slightly lower MSE, but the difference is not too large. We note that another conceptual benefit of using ϕ\phi over ϕexact\phi_{\mathrm{exact}} is that it makes the objective of diagonal smoothing and block scaling very clear - we simply want to place most entries of smoothed and scaled WW and XX in IFP4\mathcal{I}_{\mathrm{FP} 4}.

  2. Optimized versus deployed activation scales:

    Roughly speaking, the goal of the optimization is to find AA that ensures that all groups of 16 entries of WW or XX that are quantized together have dynamic range not greater than $ 7/(7/4)=4$. The block scales then “match” the group's dynamic range to the FP4 dynamic range. The main role of ΓW\Gamma_W and ΓX\Gamma_X used for minimizing the approximate loss function is for quantifying what dynamic ranges are good, through the shape of ϕ\phi.

    As the reviewer observes, ΓX\Gamma_X computed in the optimization is based on calibration data, and is not expected to work well for fresh activations. We have added Figures 10 and 11 in Appendix G, that show that this is indeed the case. The green histograms there correspond to the error of the Q~FP4\tilde{Q}_{\mathrm{FP}4} quantizer with the approximate loss optimizing scales. The full line corresponds to the MSE of the calibration data (to which ΓX\Gamma_X was optimized) and the dashed line to fresh XX but with the ΓX\Gamma_X found for the calibration data. Evidently, this ΓX\Gamma_X is terrible for the fresh data. However, once we apply absmax-to-6 (red lines) scaling for the activations, the performance for the calibration and fresh XX is close. The reason is that AA has too few parameters to overfit, so optimizing it for calibration or for test data makes a small difference.

  3. Same-platform throughput comparisons with baselines: We understand the reviewer's frustration with lack of efficiency comparisons to competing methods. However, we feel it is impossible to provide a fair comparison of this type. The reason is that efficiency depends on how good is the kernel implementing the scheme. We do not feel confident enough in implementing truly optimized kernels for the competing schemes. For this reason we only show how close we are to the vendor NVFP4 efficiency, but avoid making quantitative claims about how much OSFP4 is more efficient than competing schemes.

Thank you for the rebuttal. I appreciate authors' efforts to improve the readability of the manuscript. The additional explanations address part of my concerns about proposed technique, but the performance gains are still not substantial enough and not fully ablated. After carefully reading the authors' comments as well as other reviewers' feedbacks, I decided to keep my original score.

Reviewer_xGDTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes OSFP4, a post-training quantization method that jointly optimizes diagonal smoothing and block scales for NVFP4. It uses a multiplicative-dither error model to derive optimization objectives that account for either RTN or GPTQ-style error compensation, followed by discrete weight-scale selection. Experiments show improved average accuracy over the evaluated NVFP4 baselines with a small inference throughput overhead.

Critical strengths
  1. The error model accounts for NVFP4’s limited dynamic range and the rounding procedure, providing a principled basis for jointly optimizing smoothing and block scales.

  2. The experiments cover downstream tasks and actual inference throughput, showing accuracy gains with only a small runtime overhead.

Critical weaknesses
  1. The paper lacks a main figure presenting the method and key insights. Much of the presentation consists of lengthy formulas, making the paper difficult to follow and understand.

  2. The main evaluation focuses heavily on Llama-3.1-8B. Although additional models are included in the appendix, the Qwen3-30B-A3B experiments cover only W4A16. More complete W4A4 comparisons on larger models would strengthen the results. And why not put results of newer models in main paper rather than appendix?

  3. The main benchmark comparison is limited to four tasks. AIME, GPQA Diamond, and ARC-Challenge are evaluated on other models with different baseline sets, leaving the broader comparison fragmented. Evaluating these tasks consistently across models and adding instruction-following benchmarks such as IF-Bench would provide stronger evidence.

  4. The ablations do not clearly isolate the benefit of joint optimization. A controlled comparison against separately optimized smoothing and scales, using the same final quantizer, would help establish whether joint optimization is actually necessary.

Questions for authors
  1. Could you provide W4A4 results for Qwen3-30B-A3B and use a consistent set of reasoning and instruction-following benchmarks across models to verify the broader applicability of OSFP4?

  2. There are 25 formulas in section2. Is it really necessary to use so many formulas to explain loss functions?

2 comments

Thank you for highlighting the readability issues. This was a recurring comment from all reviewers, and we address it in our general response above. We are particularly grateful for the reviewer's comment about lack of illustration figures. We used this suggestion and added Figure 2 for the overall scheme and Figure 3 for the motivation and intuition. We also streamlined the equations and derivations in the main text, moving some details to the appendix.

Weaknesses

  1. Larger-model W4A4 evaluation and reasoning and instruction-following benchmarks: We agree that broader evaluation was needed. The revision we uploaded now includes (Appendix F), in addition to Llama3.1-8B-Instruct, also Qwen3-8B, Mistral-7B-Instruct-v0.3, OLMo-3-7B-Instruct, Qwen3-30B-A3B-Instruct (a mixture of experts model), and Gemma-4-31B-IT. For all experiments in Appendix F we now also cover GPQA Diamond, ARC-Challenge and IFbench. For all the models we texted OSFP4 attained the highest average among all compared schemes in the W4A4 setup, and for W4A16 it attained the highest average for most models. We would love to move more of these tables to the main text, but we were not able to free any real-estate for this.

    Beyond our experiments, independent community work has used adapted OSFP4 for GLM-5.3-Flash (320B parameters)1 Its authors report higher next-token agreement with BF16 predictions and lower NLL than NVIDIA's NVFP4 checkpoint on their evaluation set.

  2. Joint versus separate smoothing and scale optimization: Interesting question. To isolate the benefit of joint optimization, Appendix J (Table 12) includes an αonly\alpha_{\mathrm{only}} variant: smoothing is optimized with block scales fixed at their absmax-to-6 initialization, followed by the same deterministic scale selection and quantization procedure. Joint optimization generally achieves better results, and should not be avoided, but the differences are typically not substantial. This behavior is consistent with the structure of the FP4 quantization objective: diagonal smoothing can redistribute the relative magnitudes of entries within each quantization block, whereas adjusting a block scale changes all normalized entries by a common factor. Since absmax-to-6 already places the largest entry of each block within the minimum-distortion interval of ϕ\phi, optimizing AA addresses most of the remaining imbalance. Importantly, the αonly\alpha_{\mathrm{only}} ablation retains our final deterministic E4M3 scale-selection procedure. Thus, the comparison isolates the contribution of joint continuous scale optimization, rather than the contribution of block-scale optimization altogether.

Questions

  1. Qwen3-30B-A3B W4A4 results and consistent benchmarks: Done. See item 1 in weaknesses above.

  2. Simplifying Section 2 and reducing formulas: We have shortened section 2 and kept only what is absolutely necessary for understanding the essential ingredients of our approximate loss derivation.

Footnotes

  1. https://huggingface.co/Bizuayeu/GLM-5.3-Flash-XSFP4 ↩

Thanks for reply. After improving readability, I have raised my score accordingly.

Reviewer_nAtNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

OSFP4 jointly optimizes a diagonal smoothing transformation and NVFP4 block scales to minimize matrix-product quantization error. Because deterministic FP4 rounding makes the optimization objective discontinuous, the paper introduces a randomized multiplicative-dither FP4 quantizer that smooths the quantization error. Based on this formulation, the authors derive optimization objectives for both RTN and SIC/GPTQ-style weight quantization. After optimizing the smoothing factors and continuous scales, OSFP4 searches for hardware-valid deterministic E4M3/E2M1 scales for deployment. The method improves NVFP4 accuracy while retaining approximately 94–97% of the baseline NVFP4 prefill throughput.

Critical strengths
  • The randomized multiplicative-dither FP4 formulation provides a principled way to obtain a smooth optimization objective despite the discontinuity of deterministic FP4 rounding.
  • The ablation studies provide useful evidence on the contribution of smoothing, scale optimization, and SIC to the final accuracy improvement.
  • The method remains compatible with the standard NVFP4 representation and achieves throughput close to the vendor NVFP4 implementation.
Critical weaknesses
  • The paper is difficult to follow. The paper introduces many notations before providing sufficient high-level intuition. For example, the SIC section quickly moves into the derivation without first explaining the overall SIC/GPTQ procedure or why its objective differs from RTN. SIC is also introduced before being clearly explained.
  • The main experimental evaluation is heavily centered on Llama-3.1-8B. Although additional models are evaluated in the appendix, the evaluation is less comprehensive, and not all models use the same W4A4/W4A16 configurations. A more systematic evaluation across model families and sizes would strengthen the generalizability claim.
  • The advantage of SIC over RTN is not consistent across all evaluated models. In some of the additional experiments, RTN performs better than SIC
Questions for authors
  • Could the authors provide a short high-level explanation of SIC/GPTQ before presenting the derivation in Section 2? In particular, explaining that previously incurred quantization errors are compensated through subsequent coordinates would make Eq. (21) substantially easier to understand.
  • How sensitive is OSFP4, especially the SIC-based objective, to the choice and distribution of the calibration dataset? Have the authors evaluated calibration-domain mismatch?
  • What explains the cases where OSFP4 with SIC performs worse than the corresponding RTN version on the additional models? Are there identifiable activation statistics or model properties that predict when SIC is beneficial?
  • For W4A4, how important is the mismatch between optimizing in the continuous surrogate and using runtime absmax-to-6 activation scales during actual inference? Could the authors provide an ablation where the optimization more directly reflects the deployed activation scaling rule?
2 comments

Thank you for highlighting the readability issues. This was a recurring comment from all reviewers, and we address it in our general response above.

Weaknesses

  1. Evaluation across model families: We agree that broader evaluation was needed. The revision we uploaded now includes (Appendix F), in addition to Llama3.1-8B-Instruct, also Qwen3-8B, Mistral-7B-Instruct-v0.3, OLMo-3-7B-Instruct, Qwen3-30B-A3B-Instruct (a mixture of experts model), and Gemma-4-31B-IT. For all these models OSFP4 attained the highest average among all compared schemes in the W4A4 setup, and for W4A16 it attained the highest average for most models. Beyond our experiments, independent community work has used adapted OSFP4 for GLM-5.3-Flash (320B parameters)1 Its authors report higher next-token agreement with BF16 predictions and lower NLL than NVIDIA's NVFP4 checkpoint on their evaluation set.

Questions

  1. High-level SIC/GPTQ explanation: We have added a high-level explanation of SIC/GPTQ in Section 2.2. Hopefully it is clear now.

  2. Calibration sensitivity and domain mismatch: The reviewer raised an excellent question. GPTQ/SIC rely on calibration data for estimating the second-order statistics of activation vectors that will be encountered in inference runtime. The empirical covariance matrix ΣX\Sigma_X of the calibration samples is used for computing the Cholesky factor UU, which in turn dictates how past quantization errors are fed back before quantizing the next weights in each projection. the W-SIC variants of OSFP4 also need these second order statistics. However, OSFP4 makes further use of the calibration samples beyond second-order statistics: It uses the empirical distribution of activations in order to tune the smoothing matrix AA. Recall that AA aims to place most weights and activations in the interval IFP4\mathcal{I}_{\mathrm{FP} 4}, and this requires more delicate information on the distribution of XX than its covariance matrix (see Figure 3 in the revision).

    In our submission, all experiments used FineWeb-Edu for calibration. We now added Appendix H (Table 10) which compares FineWeb-Edu, C4, and MathQA with equal calibration token budgets under W4A4 and W4A16. Table 10 shows two things: First, OSFP4 dominates the competing GPTQ and MR-GPTQ among all three calibration sets. Second, as expected, it is also more sensitive than GPTQ and MR-GPTQ to the choice of the calibration set. In fact, it's gap from the competing scheme is smallest for FineWeb, which we used for the experiments in this paper.

  3. When and why SIC underperforms RTN: In a perfect world, where the calibration statistics and the test activation statistics are identical, it should never happen that RTN outperforms SIC. This follows since their high-resolution loss expressions differ only in the coefficients {Uℓ2(r)}\{U_\ell^2(r)\} (for SIC) and {σXℓ2(r)}\{\sigma^2_{X_\ell}(r)\} (for RTN). As we explain below eq. (23), it always holds that Uℓ2(r)≤σXℓ2(r)U_\ell^2(r)\leq \sigma^2_{X_\ell}(r). When the calibration set does not predict well the statistics of activations encountered in test time, it can certainly happen that SIC is worse than RTN. Intuitively, SIC aims to “push” the weights quantization error eWe_W to directions in Rn\mathbb{R}^n where a random activation XX has small energy, so that E(eW⊤X)2=ew⊤ΣXew\mathbb{E}(e_W^\top X)^2=e_w^\top \Sigma_X e_w is small. But to do this, we must increase the squared ℓ2\ell_2 error E∥eW∥2\mathbb{E}\|e_W\|^2, which is close to the objective RTN aims to minimize. In some cases, if the calibration and population are sufficiently different, RTN may end up better.

    In our experiments, we see this only in Qwen models and we do not have a good answer to why second-order statistics estimation is harder for these models. We do note however, that for some of the Qwen models FP-Quant's RTN performs better than FP-Quant's GPTQ, indicating that this unexpected phenomenon of RTN sometimes beating SIC/GPTQ is not special to OSFP4.

  4. Continuous optimization versus deployed activation scaling: We agree that this mismatch requires an ablation and we added Figures 10 and 11 in Appendix G to address this concern. The figures show that, as the reviewer recognized, if we simply used the ΣX\Sigma_X optimized from the calibration data on fresh activations, performance will be bad. However, this is not what OSFP4 does. Instead, it uses absmax-to-6 scaling for the (smoothed) activations. This correspond to the red histograms in the figures, and those are quite similar on the calibration and on the fresh activations.

Footnotes

  1. https://huggingface.co/Bizuayeu/GLM-5.3-Flash-XSFP4 ↩

Thank you for the additional experiments and clarifications. While the expanded evaluation addresses my concern about model coverage, the gains over the strongest baselines are generally small and inconsistent across tasks. Given the evaluation results, along with the conern of an extension of diagonal smoothing to NVFP4 with joint scale optimization, I will keep my score.

Comments on the paper

1 comment

All reviewers pointed out that the paper was too hard to follow, and we sincerely apologize for that. We took this criticism very seriously, and we have completely rewritten the introduction, and made substantial revisions to the rest of the paper as well. We believe the paper is much easier to understand now, and we invite the reviewer to take a look at the new introduction, which (hopefully) now gives a full overview to the OSFP4 scheme and the philosophy behind it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.