- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SCOPE, a forward NVFP4 quantization method for LLM training that performs data-dependent per-block scale selection. It derives an analytical oracle for the continuous optimal scale and uses it to construct a small fixed candidate set searched online. The experiments show that SCOPE reduces the BF16 gap to 1.01%, a 30% relative gap reduction over Quartet-II and 47% over the NVIDIA NVFP4 recipe.
Critical strengths
- The paper derives a closed-form, piecewise analytical oracle for the optimal NVFP4 block scale, providing a principled basis for the subsequent candidate-search approximation.
- SCOPE is integrated with existing NVFP4 training recipes and evaluated in end-to-end training, making the comparison with prior methods such as Quartet-II practically relevant.
- The paper has included the kernel and end-to-end runtime overhead for the SCOPE techniques.
Critical weaknesses
- While the analytical oracle is novel, the actual runtime algorithm is essentially a small candidate sweep with reconstruction-error minimization for each block.
- Candidate-search overhead is still nontrivial for candidate set=4 for weights and 8 for activations, adding additional quantization overhead for training.
Questions for authors
- Compared with Quartet-II/4over6, SCOPE searches more forward scale candidates and therefore appears to introduce additional forward quantization overhead. Could the authors break down the 16.5% end-to-end training-time improvement and quantify how much comes from the different backward estimator versus the proposed forward scale-selection method?
- The candidate scales are calibrated once and reused across training runs and models. How sensitive is the comparison with Quartet-II to the calibration dataset/model, and how well do these candidates generalize across model architectures and different stages of training?
- Figure 4(a) uses RTN as baselines, while Figure 4(b) switches to NVIDIA. Could the authors clarify the rationale for using different baseline sets and, if possible, make the comparison more consistent across the two panels?