- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents OSFP4, a post-training quantization method for NVFP4 LLM inference. It jointly optimizes a diagonal smoothing transformation and block scales to reduce matrix-product error under either RTN or GPTQ-style SIC. A multiplicatively dithered FP4 quantizer provides a differentiable surrogate loss, followed by discrete E4M3 scale selection. Experiments on Llama, Qwen, and Gemma models evaluate accuracy, perplexity, ablations, and throughput. OSFP4 generally improves over NVFP4 baselines while retaining 94–97% of vendor NVFP4 prefill throughput on the tested workloads.
Critical strengths
The problem is practically important. The paper focuses on NVFP4, which is directly supported by NVIDIA hardware. The method is integrated into vLLM, and the overhead introduced by quantization is discussed, with some of its extra computation fused into normalization layers.
The paper's evaluation is thorough. The authors test several model families, quantization settings, and error metrics, and also report real B200 throughput. OSFP4 improves over the main NVFP4 baselines while retaining 94–97% of the vendor implementation’s prefill throughput.
Critical weaknesses
Presentation and Experimental Organization. The paper is generally understandable but harder to follow than necessary. The full OSFP4 pipeline is split across several sections and appendices, without a clear overview of the randomized surrogate, scale optimization, and deployment procedure. The main experimental section is also overly compressed, with most setup details, ablations, additional models, and throughput results deferred to the appendix. A high-level algorithm and a more self-contained discussion of the experiments would substantially improve readability.
The contribution of the dithered surrogate is not directly ablated. The paper shows that the surrogate closely matches the error of the corresponding randomized quantizer, but does not compare it against alternative optimization objectives while holding the smoothing, scale search, and rounding procedure fixed. Therefore, it remains unclear how much of the final improvement is specifically due to the proposed dithered formulation.
Questions for authors
-
The paper shows that the dithered surrogate predicts the randomized quantization error well, but it is less clear whether it improves the final deterministic model. Could the authors compare it with another optimization objective while keeping the final scale search and RTN/SIC procedure unchanged?
-
In the W4A4 setting, the activation scales are optimized together with A, but are discarded at deployment, where absmax-to-6 scaling is used instead. Could the authors explain why the optimized A should still work well with this different scaling rule?
-
The throughput results compare OSFP4 with vendor NVFP4 and an A=I control, but not with the main methods used in the accuracy comparison. Could the authors provide same-platform throughput results for representative baselines so that the accuracy–throughput trade-off can be assessed fairly?
2 comments
Thank you for the rebuttal. I appreciate authors' efforts to improve the readability of the manuscript. The additional explanations address part of my concerns about proposed technique, but the performance gains are still not substantial enough and not fully ablated. After carefully reading the authors' comments as well as other reviewers' feedbacks, I decided to keep my original score.
Thank you for highlighting the readability issues. This was a recurring comment from all reviewers, and we address it in our general response above. We are grateful for the detailed account to what exactly should be improved in the presentation, and have done our best to address them in the revision.
Questions
Alternative objectives with fixed scale search and RTN/SIC:
The reviewer is questioning the importance of using the smooth relative MSE function ϕ rather than the exact relative MSE function ϕexact. First, we would like to point out that no prior work has used either of those expressions for the relative MSE function, so a big part of the innovation here is simply to recognize that the approximate MatMul loss with diagonal smoothing takes the form of our equations (3)-(4), regardless of whether it is with ϕ or with ϕexact.
Conceptually, the starting point of our derivation is eq. (1), where we assume that Z is statistically independent of y. This simplified model is what enables the derivation of the approximate loss functions. Without randomization (in the form of multiplicative dither), modeling Z as statistically independent of y is strange, because Z is a deterministic function of y. On the other hand, under the multiplicative-dither quantizer, Z is stochastic even given y. In fact, for $y\in\mathcal{I}_{\mathrm{FP} 4} $ we have that Z is statistically independent of y (see Ordentlich & Polyanskiy (2026a)). Though it is not exactly zero-mean. This is the main reason to introduce the multiplicative-dither quantizer - to give a better justification for the additive noise model.
It terms of optimization, in Appendix J (Table 12) we added a ϕexact ablation, replacing ϕ with ϕexact(x)=(x−QFP4(x))2/x2
in the loss function. The optimization and final scale search remain unchanged. The dithered objective generally achieves slightly lower MSE, but the difference is not too large. We note that another conceptual benefit of using ϕ over ϕexact is that it makes the objective of diagonal smoothing and block scaling very clear - we simply want to place most entries of smoothed and scaled W and X in IFP4.
Optimized versus deployed activation scales:
Roughly speaking, the goal of the optimization is to find A that ensures that all groups of 16 entries of W or X that are quantized together have dynamic range not greater than $ 7/(7/4)=4$. The block scales then “match” the group's dynamic range to the FP4 dynamic range. The main role of ΓW and ΓX used for minimizing the approximate loss function is for quantifying what dynamic ranges are good, through the shape of ϕ.
As the reviewer observes, ΓX computed in the optimization is based on calibration data, and is not expected to work well for fresh activations. We have added Figures 10 and 11 in Appendix G, that show that this is indeed the case. The green histograms there correspond to the error of the Q~FP4 quantizer with the approximate loss optimizing scales. The full line corresponds to the MSE of the calibration data (to which ΓX was optimized) and the dashed line to fresh X but with the ΓX found for the calibration data. Evidently, this ΓX is terrible for the fresh data. However, once we apply absmax-to-6 (red lines) scaling for the activations, the performance for the calibration and fresh X is close. The reason is that A has too few parameters to overfit, so optimizing it for calibration or for test data makes a small difference.
Same-platform throughput comparisons with baselines: We understand the reviewer's frustration with lack of efficiency comparisons to competing methods. However, we feel it is impossible to provide a fair comparison of this type. The reason is that efficiency depends on how good is the kernel implementing the scheme. We do not feel confident enough in implementing truly optimized kernels for the competing schemes. For this reason we only show how close we are to the vendor NVFP4 efficiency, but avoid making quantitative claims about how much OSFP4 is more efficient than competing schemes.