- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Q-PACE (Precision Allocation via Curvature Estimation for Quantization-Aware Training), a dynamic mixed-precision allocation method that aims to reduce quantization-induced performance degradation under a fixed precision budget.
The central idea is to approximate the loss increase from quantization as a sum of layerwise quantization errors weighted by curvature-based sensitivity coefficients. These coefficients are estimated through forward-pass weight perturbations, avoiding expensive Hessian computation. A multiple-choice knapsack solver then allocates precision across layers, with periodic recalibration to accommodate changing sensitivities during training.
Critical strengths
S1. A practically important and well-motivated problem. The paper addresses efficient precision allocation during low-bit LLM training. Its focus on dynamically distributing a limited precision budget is relevant to improving the accuracy–resource trade-off of quantization-aware training.
S2. A principled and computationally efficient allocation strategy. Q-PACE derives a curvature-weighted quantization error model from a second-order loss approximation. Estimating layer sensitivity through forward-only perturbations avoids explicit Hessian computation and reduces calibration memory requirements.
S3. Strong empirical improvements across precision budgets. Experiments demonstrate consistently lower pretraining loss than the NVIDIA heuristic and SNIP across the reported model scales, bit budgets, and precision choices. The comparisons with uniform NVFP4 and MXFP8 provide clear measures of the recovered quantization loss gap.
Critical weaknesses
W1. The theoretical model relies on restrictive quantization-noise assumptions. Section 3; Equations 3–5. The derivation assumes independent, zero-mean, isotropic quantization errors and approximates loss degradation as additive across layers. However, practical quantization errors can be biased, structured, and correlated. Although the empirical results support the approximation, its validity across different quantization regimes and training stages remains insufficiently established.
W2. The quantization-error approximation may oversimplify different numerical formats. Section 3; Appendix A. Q-PACE uses an analytical error model proportional to (4^{-b}), derived from uniform quantization assumptions. However, NVFP4, MXFP6, and MXFP8 differ in their scaling mechanisms and numerical representations. The paper shows that measured errors offer limited improvement, but more analysis is needed to explain why a format-independent approximation remains effective.
W3. The specific contribution of dynamic reallocation needs stronger validation. Section 5; Figure 5b. The paper demonstrates that periodic reallocation improves performance over a single allocation, but this analysis is mainly conducted on the 350M model. It remains unclear how much dynamic adaptation contributes independently of identifying a strong initial allocation, particularly for larger models. Additional comparisons with static allocations across model scales would strengthen this claim.
Questions for authors
Q1. Validity of the sensitivity approximation How well do the independent, zero-mean, isotropic noise assumptions hold for actual low-precision training? Can the authors directly measure cross-layer quantization-error correlations and evaluate their influence on allocation quality?
Q2. Generality of the analytical quantization-error model Why does the simple (4^{-b}) approximation work across numerical formats with different scaling and rounding mechanisms? How would Q-PACE behave with other hardware-supported formats or substantially different quantization schemes?
Q3. Contribution of dynamic reallocation Can the authors compare dynamic Q-PACE with fixed initial, intermediate, and final precision allocations under identical training budgets? Does the improvement from periodic reallocation remain significant for the 800M and 4B models?
2 comments
Q3 + W3. We have done the experiments with static allocations on the 4B model. As reported in Table R4, doing only one calibration at the beginning of training costs +0.0042 in the final test loss and yields the highest loss in Table R4. As shown in Fig. 6b in the paper, the allocation changes rapidly in early training, showing why the initial allocation can improve training performance in the early stages but not afterward. Overall, Table R4 suggests that reusing middle or final allocation produced by Q-PACE improves the loss, but still not to the level of full Q-PACE. Also note while intermediate and final fixed allocations help, they can only be obtained by first running the full Q-PACE training.
Table R4. 4B model, 6.5 bits, {4,16}, 100k steps: Q-PACE vs. static allocations.
| Allocation | Final test loss | Difference vs. Q-PACE |
|---|---|---|
| Q-PACE | – | |
| Q-PACE frozen final allocation | ||
| Q-PACE frozen allocation @ 30% of training | ||
| Q-PACE frozen start allocation |
The 4B runs use a single seed, but across seeds for 800M model standard deviation of the final loss is small, 5e-4. Taking this as the noise level, the +0.0042 gap for the initial allocation is roughly 6 times the noise expected for a difference between two runs, and the gaps for the intermediate and final allocations are about 2x. We have not run static allocations for the 800M model, but since the 350M (Fig. 5b) and 4B results agree, we expect the same behavior in between.
We will include these results in the paper to show that the improvements come from both training under the Q-PACE allocation and the periodic recalibration.
We appreciate the reviewer's points and see that addressing them would strengthen the paper. We will update the manuscript accordingly.
Thank you for the detailed review! We address the questions and concerns below.
W1 + Q1. First, we revised the assumptions in derivation in Sec. 3, and one of them can be relaxed: noise does not have to be isotropic with VarδWi=σi2I. Having second and third moments of δW to be finite suffices for the loss approximation to hold. We will include a formal proof in the revised version.
To justify two other modelling assumptions, we check if they hold empirically. We tested each assumption on the true quantization error Ei=Q(Wi)−Wi of a uniform-FP4 800M checkpoint in the middle of training (step 40k of 76k). i index layers.
Zero-mean assumption holds: mean value of Ei is ⟨Ei⟩=(−0.9±7.5)×10−7. Independence across layers holds: we compute the Pearson correlation between errors for different layers r(Ei,Ej)=Var(Ei)Var(Ej)Corr(Ei,Ej) and average across layer pairs ⟨∣r(Ei,Ej)∣⟩=5⋅10−4, maxij∣r(Ei,Ej)∣=4⋅10−3. Therefore errors are effectively uncorrelated between layers. In addition, we evaluated inter-layer correlation directly as a part of the sensitivity model, see Quadratic Q-PACE variant (App. F, Eq. 15). It adds pairwise cross-layer terms to the sensitivity model, and it does not improve over our main method.
W2 + Q2.
About using Gaussian noise regardless of the actual numerical format: we compared analytical (Gaussian) noise with true measured noise in App.A. On small scale models the difference was within the noise interval. We decided on Gaussian noise to keep the method general. As a hypothetical example, for vector quantization with a large codebook cost of forward passes for recalibration might become significant and analytical noise would be cheaper. More importantly, forward quantization is deterministic in our case, and sampling noise from a distribution gives more points for sensitivity fit.
To verify that analytical noise is not hindering performance, we conducted additional experiments on the 800M model, see Table R1. Models trained with the analytic and with the measured quantization error reach the same final loss within seed noise. This is consistent with the fact that analytic and true noise probes produce similar allocations: see fit quality and allocation differences in Table R2.
The sensitivity fits are good overall: averaged over training, R² is 0.80 with analytic noise and 0.81 with true noise, and the two variants give very close sensitivities and allocations. The fit is better for more sensitive layers (MLPs vs Q proj.).
Table R1. Test loss for the 800M model (80B tokens) trained with dynamic precision allocation.
Table R2. Sensitivity fit analysis for 800M model, 6.0 bits/param, {4,16}. "Analytic" refers to Gaussian noise probes, "true" to measured quantization noise. Sensitivity is reported relative to the mean at corresponding timestep. All metrics are averaged over training.
In addition, we test Q-PACE in a different numerical format: fake integer quantization (weight-only) for INT{2,3,4,8}, see Table R3. Q-PACE with analytic noise outperforms other methods.
Table R3. Test loss for 350M (18B tokens) QAT with INT quantization.