acceptodds
Under review as a conference paper at ICLR 2027

SCOPE: From Analytical Scale Oracles to Candidate Scale Search for NVFP4 Quantized Training

Abstract

Recent hardware support for microscaled FP4 formats has made NVFP4 a compelling target for reducing the cost of large language model pre-training. However, existing NVFP4 training recipes still show a noticeable quality gap to BF16, primarily due to forward weight-activation quantization under the sparse FP4 codebook and shared block scale. We propose **SCOPE**, a forward-centered NVFP4 quantization framework that combines an analytical scale oracle with a deployable candidate search. SCOPE first introduces an analytical per-block scale oracle based on FP4 rounding breakpoints to characterize the reconstruction improvement recoverable by adaptive scale selection. For deployment, SCOPE replaces online breakpoint enumeration with a compact fixed-candidate search, preserving data-dependent scale selection while keeping the online path simple. Backward gradient-to-quantization-noise diagnostics further show that lightweight SR/Had-SR estimators remain well above the theoretically motivated diagnostic reference in our setting, allowing us to use a low-overhead backward path and focus the main optimization effort on forward scale selection. On NanoChat pre-training up to 1.9B parameters and 38B tokens, SCOPE reduces the BF16 BPB gap to 1.01%, a relative gap reduction of \(~30%\) over Quartet-II and \(~47%\) over the NVIDIA NVFP4 recipe. In a matched 500M experiment, SCOPE reduces the measured complete-recipe training time by 16.5% relative to Quartet-II while achieving better validation quality. Additional PTQ and pre-training experiments show transfer across dense LLMs, MoE models, and vision-language settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM quantization, NVFP4 training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NzBGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SCOPE, a forward NVFP4 quantization method for LLM training that performs data-dependent per-block scale selection. It derives an analytical oracle for the continuous optimal scale and uses it to construct a small fixed candidate set searched online. The experiments show that SCOPE reduces the BF16 gap to 1.01%, a 30% relative gap reduction over Quartet-II and 47% over the NVIDIA NVFP4 recipe.

Critical strengths
  • The paper derives a closed-form, piecewise analytical oracle for the optimal NVFP4 block scale, providing a principled basis for the subsequent candidate-search approximation.
  • SCOPE is integrated with existing NVFP4 training recipes and evaluated in end-to-end training, making the comparison with prior methods such as Quartet-II practically relevant.
  • The paper has included the kernel and end-to-end runtime overhead for the SCOPE techniques.
Critical weaknesses
  • While the analytical oracle is novel, the actual runtime algorithm is essentially a small candidate sweep with reconstruction-error minimization for each block.
  • Candidate-search overhead is still nontrivial for candidate set=4 for weights and 8 for activations, adding additional quantization overhead for training.
Questions for authors
  • Compared with Quartet-II/4over6, SCOPE searches more forward scale candidates and therefore appears to introduce additional forward quantization overhead. Could the authors break down the 16.5% end-to-end training-time improvement and quantify how much comes from the different backward estimator versus the proposed forward scale-selection method?
  • The candidate scales are calibrated once and reused across training runs and models. How sensitive is the comparison with Quartet-II to the calibration dataset/model, and how well do these candidates generalize across model architectures and different stages of training?
  • Figure 4(a) uses RTN as baselines, while Figure 4(b) switches to NVIDIA. Could the authors clarify the rationale for using different baseline sets and, if possible, make the comparison more consistent across the two panels?
Reviewer_8DRFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCOPE for NVFP4 quantized training. It derives an analytical oracle for block-wise scale selection and approximates it with a small calibrated candidate set to improve forward weight and activation quantization. The backward pass uses lightweight stochastic rounding. Pretraining experiments show a smaller validation-loss gap to BF16 than the compared NVFP4 baselines.

Critical strengths
  1. The experiments cover both PTQ and pretraining, providing a reasonably comprehensive assessment of the quality loss from NVFP4 quantization.
  2. The scale-oracle analysis offers useful insight into how scale selection affects quantization error and could guide future work on NVFP4 quantization.
Critical weaknesses
  1. The scale-search approach is closely related to prior work such as 4over6. Although SCOPE searches a richer set of scales, the overall methodological novelty is limited.

  2. The presentation needs further polishing. Some figures, particularly Figure 3, would benefit from more careful layout and labeling.

  3. The PTQ evaluation should include newer models and a broader set of benchmarks. Even within QA, the coverage is limited. Additional experiments on reasoning and agent tasks would help assess whether the gains extend to more demanding settings.

Questions for authors

Section 4.4 reports 12.7% higher end-to-end training time than the NVIDIA recipe on a 500M model. How would this relative overhead change for larger models, such as 3B, 10B, or 100B+? While Appendix H discusses the scaling of forward-search costs, could the authors extend this analysis to overall training time under representative model widths, batch sizes, and sequence lengths? An analytical estimate would be sufficient without requiring full training runs at these scales.

Reviewer_JugTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCOPE studies block-wise scale selection for NVFP4 quantized training. It first derives an analytical oracle for the continuous effective dequantization factor by partitioning the scale range at FP4 rounding thresholds and solving a scalar least-squares problem in each interval. For deployment, it replaces this oracle with fixed candidate sets calibrated offline: four scales for weights and eight for activations, selected online by block reconstruction error. The complete recipe combines this forward quantizer with a lightweight SR/Hadamard-assisted SR backward path. Evaluation includes PTQ, controlled LLaMA-family pre-training, and NanoChat pre-training up to 1.9B parameters and 38B tokens. At the largest scale, validation BPB is 0.6995 versus 0.6925 for BF16 and 0.7025 for Quartet-II, corresponding to a 30% reduction in the excess BPB over BF16. A separate 500M timing experiment reports 16.5% less training time than Quartet-II, although SCOPE remains 12.7% slower than the NVIDIA recipe.

Critical strengths
  1. SCOPE connects an analytical scale oracle with a practical fixed-candidate search. Scale-dynamics analysis and equal-budget reconstruction ablations support this design (Appendices B and F).
  2. Controlled forward/backward ablations help isolate the source of improvement (Tables 1 and 8). Gains across LLaMA-family and NanoChat pre-training, including 1.9B/38B, provide evidence beyond a single training setup.
Critical weaknesses
  1. Reproducibility is limited by missing details on E4M3 factor rounding, candidate scoring after rounding, and calibration. The ambiguous "Greedy/Frequency-TopK" label also leaves the candidate-construction rule unclear (Section 3.3, Appendix B).
  2. Practical efficiency remains uncertain: SCOPE is 12.7% slower than NVIDIA, without a matched-time quality comparison or complete BF16 training timing. Equal-budget candidate comparisons measure only reconstruction error, not training quality (Section 4.4, Table 9).
Questions for authors
  1. Please clarify the deployed E4M3 rounding and error-evaluation procedure and provide the exact calibration sample size, candidate pool, and selection rule behind "Greedy/Frequency-TopK."
  2. Under the same execution stack, how do SCOPE, NVIDIA, Quartet-II, and BF16 compare in validation quality at matched wall-clock budgets, including the cost of candidate search?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.