acceptodds
Under review as a conference paper at ICLR 2027

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

11 views

Abstract

Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final performance. In order to assign precision to different layers, common approaches rely on quantization error proxies. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization-aware training, mixed-precision training, FP4, large language models, efficiency, pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_BkPFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Q-PACE (Precision Allocation via Curvature Estimation for Quantization-Aware Training), a dynamic mixed-precision allocation method that aims to reduce quantization-induced performance degradation under a fixed precision budget.

The central idea is to approximate the loss increase from quantization as a sum of layerwise quantization errors weighted by curvature-based sensitivity coefficients. These coefficients are estimated through forward-pass weight perturbations, avoiding expensive Hessian computation. A multiple-choice knapsack solver then allocates precision across layers, with periodic recalibration to accommodate changing sensitivities during training.

Critical strengths

S1. A practically important and well-motivated problem. The paper addresses efficient precision allocation during low-bit LLM training. Its focus on dynamically distributing a limited precision budget is relevant to improving the accuracy–resource trade-off of quantization-aware training.

S2. A principled and computationally efficient allocation strategy. Q-PACE derives a curvature-weighted quantization error model from a second-order loss approximation. Estimating layer sensitivity through forward-only perturbations avoids explicit Hessian computation and reduces calibration memory requirements.

S3. Strong empirical improvements across precision budgets. Experiments demonstrate consistently lower pretraining loss than the NVIDIA heuristic and SNIP across the reported model scales, bit budgets, and precision choices. The comparisons with uniform NVFP4 and MXFP8 provide clear measures of the recovered quantization loss gap.

Critical weaknesses

W1. The theoretical model relies on restrictive quantization-noise assumptions. Section 3; Equations 3–5. The derivation assumes independent, zero-mean, isotropic quantization errors and approximates loss degradation as additive across layers. However, practical quantization errors can be biased, structured, and correlated. Although the empirical results support the approximation, its validity across different quantization regimes and training stages remains insufficiently established.

W2. The quantization-error approximation may oversimplify different numerical formats. Section 3; Appendix A. Q-PACE uses an analytical error model proportional to (4^{-b}), derived from uniform quantization assumptions. However, NVFP4, MXFP6, and MXFP8 differ in their scaling mechanisms and numerical representations. The paper shows that measured errors offer limited improvement, but more analysis is needed to explain why a format-independent approximation remains effective.

W3. The specific contribution of dynamic reallocation needs stronger validation. Section 5; Figure 5b. The paper demonstrates that periodic reallocation improves performance over a single allocation, but this analysis is mainly conducted on the 350M model. It remains unclear how much dynamic adaptation contributes independently of identifying a strong initial allocation, particularly for larger models. Additional comparisons with static allocations across model scales would strengthen this claim.

Questions for authors

Q1. Validity of the sensitivity approximation How well do the independent, zero-mean, isotropic noise assumptions hold for actual low-precision training? Can the authors directly measure cross-layer quantization-error correlations and evaluate their influence on allocation quality?

Q2. Generality of the analytical quantization-error model Why does the simple (4^{-b}) approximation work across numerical formats with different scaling and rounding mechanisms? How would Q-PACE behave with other hardware-supported formats or substantially different quantization schemes?

Q3. Contribution of dynamic reallocation Can the authors compare dynamic Q-PACE with fixed initial, intermediate, and final precision allocations under identical training budgets? Does the improvement from periodic reallocation remain significant for the 800M and 4B models?

2 comments

Thank you for the detailed review! We address the questions and concerns below.

W1 + Q1. First, we revised the assumptions in derivation in Sec. 3, and one of them can be relaxed: noise does not have to be isotropic with Var⁡δWi=σi2I\operatorname{Var} \delta W_i = \sigma_i^2 I. Having second and third moments of δW\delta W to be finite suffices for the loss approximation to hold. We will include a formal proof in the revised version.

To justify two other modelling assumptions, we check if they hold empirically. We tested each assumption on the true quantization error Ei=Q(Wi)−WiE_i = Q(W_i) − W_i of a uniform-FP4 800M checkpoint in the middle of training (step 40k of 76k). ii index layers.

Zero-mean assumption holds: mean value of EiE_i is ⟨Ei⟩=(−0.9±7.5)×10−7\langle E_i \rangle = (-0.9 \pm 7.5)\times 10^{-7}. Independence across layers holds: we compute the Pearson correlation between errors for different layers r(Ei,Ej)=Corr⁡(Ei,Ej)Var⁡(Ei)Var⁡(Ej)r(E_i, E_j) = \frac{\operatorname{Corr} (E_i, E_j) }{\sqrt{\operatorname{Var}(E_i)} \sqrt{\operatorname{Var}(E_j)} } and average across layer pairs ⟨∣r(Ei,Ej)∣⟩=5⋅10−4\langle |r (E_i, E_j)| \rangle= 5\cdot 10^{-4}, max⁡ij∣r(Ei,Ej)∣=4⋅10−3\max_{ij} |r (E_i, E_j) |= 4 \cdot 10^{-3}. Therefore errors are effectively uncorrelated between layers. In addition, we evaluated inter-layer correlation directly as a part of the sensitivity model, see Quadratic Q-PACE variant (App. F, Eq. 15). It adds pairwise cross-layer terms to the sensitivity model, and it does not improve over our main method.

W2 + Q2.

About using Gaussian noise regardless of the actual numerical format: we compared analytical (Gaussian) noise with true measured noise in App.A. On small scale models the difference was within the noise interval. We decided on Gaussian noise to keep the method general. As a hypothetical example, for vector quantization with a large codebook cost of forward passes for recalibration might become significant and analytical noise would be cheaper. More importantly, forward quantization is deterministic in our case, and sampling noise from a distribution gives more points for sensitivity fit.

To verify that analytical noise is not hindering performance, we conducted additional experiments on the 800M model, see Table R1. Models trained with the analytic and with the measured quantization error reach the same final loss within seed noise. This is consistent with the fact that analytic and true noise probes produce similar allocations: see fit quality and allocation differences in Table R2.

The sensitivity fits are good overall: averaged over training, R² is 0.80 with analytic noise and 0.81 with true noise, and the two variants give very close sensitivities and allocations. The fit is better for more sensitive layers (MLPs vs Q proj.).

Table R1. Test loss for the 800M model (80B tokens) trained with dynamic precision allocation.

ModelBudget, bits/paramBit choicesMethodMean test loss (n=3)Std. of test loss (n=3)
800M6.0{4,16}NVIDIA2.60682.60682⋅10−42 \cdot 10^{-4}
6.0{4,16}SNIP2.60462.6046$3 \cdot 10^{-3} $
6.0{4,16}Q-PACE, analytic noise2.60292.60296⋅10−46 \cdot 10^{-4}
6.0{4,16}Q-PACE, true noise2.60282.60285⋅10−45 \cdot 10^{-4}

Table R2. Sensitivity fit analysis for 800M model, 6.0 bits/param, {4,16}. "Analytic" refers to Gaussian noise probes, "true" to measured quantization noise. Sensitivity is reported relative to the mean at corresponding timestep. All metrics are averaged over training.

Layer typeR² (analytic)R² (true)Sensitivity (analytic)Sensitivity (true)Layers in bf16 (analytic)Layers in bf16 (true)
Q0.680.690.290.290.1 / 170.0 / 17
K0.600.600.200.206.3 / 175.5 / 17
V0.860.871.041.0417.0 / 1717.0 / 17
O0.880.881.161.1412.4 / 1712.6 / 17
MLP up0.920.922.152.171.0 / 171.0 / 17
MLP down0.880.891.161.160.0 / 170.0 / 17
All layers0.800.811.001.0036.7 / 10236.1 / 102

In addition, we test Q-PACE in a different numerical format: fake integer quantization (weight-only) for INT{2,3,4,8}, see Table R3. Q-PACE with analytic noise outperforms other methods.

Table R3. Test loss for 350M (18B tokens) QAT with INT quantization.

MethodBudgetBitsLoss mean (n=3)Loss std (n=3)
fixed INT33{3}2.88080.0008
Q-PACE analytic noise2.5{2, 3, 4, 8}2.91170.0014
SNIP2.5{2, 3, 4, 8}2.91980.0007
Q-PACE true noise2.5{2, 3, 4, 8}2.92090.0018
NVIDIA2.5{2, 3, 4, 8}2.95380.0007
fixed INT22{2}2.97060.0014

Q3 + W3. We have done the experiments with static allocations on the 4B model. As reported in Table R4, doing only one calibration at the beginning of training costs +0.0042 in the final test loss and yields the highest loss in Table R4. As shown in Fig. 6b in the paper, the allocation changes rapidly in early training, showing why the initial allocation can improve training performance in the early stages but not afterward. Overall, Table R4 suggests that reusing middle or final allocation produced by Q-PACE improves the loss, but still not to the level of full Q-PACE. Also note while intermediate and final fixed allocations help, they can only be obtained by first running the full Q-PACE training.

Table R4. 4B model, 6.5 bits, {4,16}, 100k steps: Q-PACE vs. static allocations.

AllocationFinal test lossDifference vs. Q-PACE
Q-PACE2.26562.2656–
Q-PACE frozen final allocation2.26732.2673+0.0017+0.0017
Q-PACE frozen allocation @ 30% of training2.26722.2672+0.0016+0.0016
Q-PACE frozen start allocation2.26982.2698+0.0042+0.0042

The 4B runs use a single seed, but across seeds for 800M model standard deviation of the final loss is small, 5e-4. Taking this as the noise level, the +0.0042 gap for the initial allocation is roughly 6 times the noise expected for a difference between two runs, and the gaps for the intermediate and final allocations are about 2x. We have not run static allocations for the 800M model, but since the 350M (Fig. 5b) and 4B results agree, we expect the same behavior in between.

We will include these results in the paper to show that the improvements come from both training under the Q-PACE allocation and the periodic recalibration.

We appreciate the reviewer's points and see that addressing them would strengthen the paper. We will update the manuscript accordingly.

Reviewer_iKP6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-PACE, which estimates layer-wise quantization sensitivity through forward-pass perturbations and periodically reallocates precision during training. Under a global bit budget, it improves the loss-memory trade-off of low-precision training.

Critical strengths
  1. Across the evaluated pretraining settings, Q-PACE achieves lower final loss than NVFP4 heuristic and SNIP at the same bit budget, recovering up to 76% of the loss gap between FP4 and FP8 training.
  2. The proposed sensitivity estimation approach avoids explicit Hessian computation and additional backward passes. The analysis explains how sensitivity varies with layer type, depth, and training stage. Its relatively slow evolution makes infrequent dynamic reallocation practical.
Critical weaknesses
  1. The curvature-weighted quantization error model is closely related to prior work such as HIGGS and HAWQ, while dynamic bit-width allocation has already been explored by SNIP. Q-PACE mainly introduces a more effective sensitivity estimation method within an existing framework. This incremental novelty weakens the paper's contribution.
  2. An average bit budget does not directly reflect actual throughput or total training memory, which also depend on format switching and microscaling metadata. Evidence of end-to-end training speedups on real hardware remains insufficient.
Questions for authors
  1. Given that curvature-weighted quantization error models have been explored in HIGGS and HAWQ, and dynamic bit-width allocation has already been introduced by SNIP, could the authors clarify the key methodological novelty of Q-PACE beyond combining and adapting these existing ideas to QAT?
  2. Can the authors provide detailed ablations that separately evaluate the benefits of the proposed sensitivity estimation and dynamic reallocation?
1 comment

We thank the reviewer for their effort. We address the questions and concerns below.

W1 + Q1. Our contribution is to show that a curvature weighted error model (as in HIGGS) is the right sensitivity model for periodic bit reallocation during QAT. This does not follow from prior work: those models consider PTQ, whereas in QAT quantization also distorts gradient matmuls. It is the reason why SNIP adds a backward-pass term to its scores, but we show that this term are not needed. The forward weight-error model alone predicts final test loss with r = 0.96–0.98, even when the entire backward pass runs in FP4 (Fig. 2, App. B).

SNIP and Q-PACE both solve a knapsack problem and differ in the sensitivity model. Q-PACE reaches lower final loss in every pretraining and SFT setting. Q-PACE also provides cheaper calibration. Our coefficients need only forward passes and one noise tensor of extra memory. HAWQ’s Hessian-trace estimates need Hessian-vector products, and SNIP needs backward passes during calibration, which raises peak memory (Table R1: 177.2 vs. 154.7 GiB).

We also provide new findings about sensitivity during training. Sensitivity is structured by layer type and depth and evolves much more slowly than the weights, so about five reallocations per run suffice (Fig. 5b). For example, in a fixed allocation, Q projections can stay in low precision, while V projections should be kept in higher precision.

W2.

We chose format’s memory cost as a proxy of inference cost, not training. We do not claim that it is an optimal choice of cost function, rather, it is one of the choices. An alternative could be a matmul latency of a layer at every bitwidth, which would constrain training throughput and inference time instead of memory.

Regarding end-to-end training comparisons between methods, we provide them below in Table R1 for the 4B model. Methods reach similar training speeds between uniform FP4 and FP8. Q-PACE has lower overall throughput due to recalibration procedure, but same memory cost as NVIDIA. The recalibration cost of Q-PACE can be further reduced by freezing the allocation after the first third of training (see Q2). SNIP’s calibration, however, needs multiple backward passes and using the same micro-batch size as training led to OOM error.

Table R1. 4B model, 6.5 bits / parameter: end-to-end efficiency on 16x B200 (Batch size 1.05M tokens, micro BS 8, context 4096 tokens).

MethodBudgetBit choicesfwd+bwd (s)Optim step (s)Tok/s/gpu (train only)Tok/s/gpu (with recalibration)NVML mean / max (GiB)
FP4–{4}1.0500.18451.2k51.2k154.2 / 154.7
FP8–{8}1.4220.18040.1k40.1k172.6 / 173.1
NVIDIA6.5{4,16}1.1490.19147.0k47.0k152.8 / 154.7
SNIP6.5{4,16}1.1050.18448.8k48.5k165.5 / 177.2
Q-PACE6.5{4,16}1.1300.18447.7k44.3k154.2 / 154.7

Q2.

Table R2 compares Q-PACE with its static variants on the 4B model. A single calibration at the start, with no reallocation, is on par with SNIP. Frozen allocation at 30% of training or frozen final allocation improve on it but still lose to original Q-PACE.

In summary, most of the benefit of dynamic reallocation is concentrated in early training stages, which is consistent with our sensitivity analysis in Sec. 5. This also means the allocation can be frozen after the first third of training to save recalibration cost.

Table R2. 4B model, 6.5 bits, {4,16}, 100k steps: Q-PACE vs. static allocations.

AllocationFinal test lossDifference vs. Q-PACE
Q-PACE2.26562.2656–
Q-PACE frozen @ 30%2.26722.2672+0.0016+0.0016
Q-PACE frozen final allocation2.26732.2673+0.0017+0.0017
Q-PACE frozen at the start2.26982.2698+0.0042+0.0042

For reference, same setting and seed: SNIP 2.26992.2699 (+0.0043+0.0043), NVIDIA 2.27092.2709 (+0.0053+0.0053).

We appreciate the reviewer's points and see that addressing them would strengthen the paper. We will update the manuscript accordingly.

Reviewer_h6soOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-PACE, a dynamic layer-wise precision allocation method for low-precision LLM training. It estimates sensitivity through Gaussian weight perturbations and forward loss evaluations, then periodically adjusts precision assignments under an average-weight-bitwidth budget. Experiments cover pretraining from 350M to 4B parameters and supervised fine-tuning of Qwen3-4B. The reported results show improved loss over the evaluated mixed-precision baselines.

Critical strengths
  1. The paper addresses an important quality–precision trade-off in low-precision LLM training. The core second-order derivation is reasonable under the stated assumptions on quantization noise. Gaussian perturbations provide a practical way to estimate the resulting curvature coefficients without explicit Hessian computation or calibration backward passes.

  2. The results show lower pretraining test loss than the evaluated NVIDIA and SNIP baselines across the reported configurations. The paper also includes useful ablations and analyzes how sensitivity varies with layer type, depth, and training time. Together, these experiments provide meaningful evidence of the method’s empirical effectiveness.

Critical weaknesses
  1. The paper clearly distinguishes Q-PACE from HIGGS and SNIP. However, its main methodological contribution is adapting forward-only sensitivity estimation to dynamic low-precision LLM training. I view this as an incremental advance, whose significance depends largely on the strength of the empirical improvements.

  2. Although Q-PACE consistently improves pretraining loss, some gains over stronger baselines are modest. For example, Table 4 reports 2.6670 versus 2.6696 for SNIP weight-gradient-only at 500M and a 5.5-bit budget. The 500M, 800M, and 4B mixed-precision pretraining results use one seed, and SFT does not consistently improve downstream accuracy over SNIP. Additional evidence would strengthen confidence in the robustness and practical significance of these gains.

Questions for authors

The loss improvements over SNIP weight-gradient-only appear relatively modest in some settings, while the larger mixed-precision pretraining experiments use only a single seed. Could the authors provide additional evidence across random seeds under matched training settings to assess the robustness of these gains?

3 comments

We thank the reviewer for their time and effort. We address the questions and concerns below.

Q1. We agree that results over multiple random seeds would be more convincing. To address this, we provide measurements for 500M and 800M models at 3 random seeds in Table R1. Dataloader, model init, and Q-PACE noise draws all use a new random seed.

Table R1. Test Loss of the 500M (74B tokens) and 800M (80B tokens) models trained with mixed precision, 3 seeds each.

ModelBudget (bits/param)Bit choicesMethodMean loss (n=3)Std. (n=3)Δ vs. Q-PACE (Δ/σ_Δ)
500M–{4}Uniform FP42.69262.69261⋅10−31 \cdot 10^{-3}+0.0167+0.0167 (15.5)(15.5)
–{8}Uniform FP82.65652.65652⋅10−42 \cdot 10^{-4}−0.0194-0.0194 (43.4)(43.4)
6.5{4,16}NVIDIA2.68302.68309⋅10−49 \cdot 10^{-4}+0.0071+0.0071 (7.2)(7.2)
6.5{4,16}SNIP2.68082.68089⋅10−49 \cdot 10^{-4}+0.0049+0.0049 (5.0)(5.0)
6.5{4,16}SNIP (w·g)2.67982.67988⋅10−48 \cdot 10^{-4}+0.0039+0.0039 (4.6)(4.6)
6.5{4,16}Q-PACE2.67592.67594⋅10−44 \cdot 10^{-4}—
800M6.0{4,16}NVIDIA2.60682.60682⋅10−42 \cdot 10^{-4}+0.0039+0.0039 (6.2)(6.2)
6.0{4,16}SNIP2.60462.60465⋅10−45 \cdot 10^{-4}+0.0017+0.0017 (2.2)(2.2)
6.0{4,16}Q-PACE2.60292.60296⋅10−46 \cdot 10^{-4}—

As shown above, Q-PACE improves over SNIP by 2 (800M) to 5 (500M) standard deviations. It improves over SNIP w-g by 4.4 standard deviations at 500M. Therefore, our original claims hold in a statistically significant way.

Moreover, Q-PACE has the lowest loss in every individual seed at both sizes. The new multi-seed means match the single-seed results in the paper to within 1e-3.

W1. We would like to argue that empirical improvements of the method are indeed significant. For 350M, 500M, and 4B models, at 5.5 bits/parameter (only 1 bit over uniform NVFP4) our method recovers 68-71% of the gap between NVFP4 and FP8.

We care about FP8 to FP4 loss gap for the following reason: FP8 training is practically lossless relative to BF16, but uniform FP4 training features a persistent loss gap to a full precision model. Moreover, uniform FP4 training faces instabilities, which makes it unattainable at larger training horizons [1, 2]. Thus some layers have to be promoted to higher precision to resume training. If a certain budget has to be spent anyway, we provide an algorithm to spend it in an efficient way, achieving better loss than previous baselines.

Techniques such as the NVIDIA heuristic cannot make use of extra bit formats. SNIP can, but it has higher loss than Q-PACE in every setting and has a substantial memory overhead from storing 2 sets of gradients.

In addition, our analysis of how sensitivity varies with layer type, depth, and how it evolves over time, reveals interesting new insights. For example, if searching for a fixed allocation to do training with, Q proj. could be left in lower precision, but V proj. should be kept in a higher precision.

W2. You are right to notice that some of the improvements are larger than others. In particular, one characteristic of the method is that, at a fixed total budget, Q-PACE performs better when more bit choices are available. This highlights the flexibility of the method when given a larger search space. Specifically, Figure 4 shows that 4B model at 6.5 bits/parameter Q-PACE with choices {FP4, BF16} gives 44% recovery to FP8 level, and adding FP8 format allows the gap to recover 76% at the same 6.5 bits/parameter.

Also note that Q-PACE outperforms both NVIDIA and SNIP for all sets of formats tested. Even in the most restricted (for Q-PACE) setting with only 2 formats available Q-PACE shows the best loss.

Regarding SFT: we agree that Q-PACE and SNIP are close on downstream accuracy in SFT, as the differences (0.1 to 0.4 points) are within seed noise. Held-out loss is the more sensitive metric, and Q-PACE has better loss at every budget, recovering 47–74% of the FP4-to-FP8 gap versus 41–64% for SNIP. We will state this explicitly in the revision.

[1] https://arxiv.org/pdf/2610.24343

[2] https://arxiv.org/pdf/2509.25149

We would like to provide more evidence on Q-PACE performance on downstream evals.

In the time between submission and rebuttal, we had the opportunity to piggy-back on an ongoing FP4 pretraining experiment at larger scale than what we could do for the paper: a 12 billion parameter dense transformer trained with Muon on 64 B300 GPUs. This training run first tested full FP4 allocation, which resulted in divergence after about 600B tokens, and then switched to (an approximation of) the NVIDIA method, running the last 10% of transformer blocks in MXFP8.

While replicating the full run in Q-PACE was out of budget for us, we could take two of the intermediate checkpoints to gather further evidence of the stability and usefulness of Q-PACE at larger scales. First, we took the checkpoint before the NVFP4 divergence, at which the original run switched to the NVIDIA recipe, and instead continued for ~100B tokens with Q-Pace, validating that no divergence occurs and that the loss improves over the NVIDIA allocation. At about 0.65T tokens, NVIDIA improves over uniform NVFP4 by 0.0046 in validation loss, and Q-PACE by 0.0060, a roughly 30% larger improvement at the same budget.

Second, we rerun the mid-training phase (100B tokens of Dolmino [3] dataset) with Q-PACE instead of NVIDIA. As mid-training results in strong uplifts of downstream performance within a modest token budget (compared to pre-training), this provides a way to estimate how Q-Pace affects downstream results.

We evaluate produced checkpoints on the OLMES suite at different inference formats, results shown in Table R2.

Table R2. Results at the 4T checkpoint (step 2,543,200), after midtraining is complete. We report test loss, OLMES score, and bits per byte (BPB). Both methods use 4.875 bits / parameter, which corresponds to 10% of the last layers in FP8 for the NVIDIA method. FP4 cooldown for the last 0.5% steps is used to improve FP4 inference. Single seed.

Trained withServed withTest loss ( ↓\downarrow)OLMES (↑\uparrow)BPB ( ↓\downarrow)
NVIDIAFP42.121181.930.6440
NVIDIA + FP4 cooldownFP42.118782.200.6395
Q-PACEFP42.123882.020.6415
Q-PACE + FP4 cooldownFP42.118582.480.6411
NVIDIANVIDIA2.113182.050.6419
NVIDIAQ-PACE2.114282.180.6412
Q-PACEQ-PACE2.111582.750.6388

Q-PACE served with its own bit allocation achieves the best result on all three metrics. Under uniform FP4 serving the two methods are close, with cooldown Q-PACE has the higher OLMES score, and NVIDIA the lower BPB.

Applying Q-PACE allocation to the NVIDIA-midtrained model does not improve over trained+served NVIDIA much. It allows us to separate the effect of Q-PACE on training dynamics from the effect of Q-PACE allocation applied post training. If Q-PACE’s gains came only from better allocation at inference, NVIDIA + Q-PACE would match Q-PACE + Q-PACE, but it does not. We therefore conclude that large part of the gain comes from training with the Q-PACE allocation, not only from the allocation alone applied at inference.

We appreciate the reviewer's points and see that addressing them would strengthen the paper. We will update the manuscript accordingly.

[3] https://arxiv.org/abs/2512.13961

Thanks for the additional experiments. The multi-seed results at 500M and 800M help address my concern about the robustness of the improvements over SNIP. The new 12B results are also encouraging. I still think the contribution is somewhat incremental, but the additional evidence makes the results more convincing. I will raise my rating to 3.

Reviewer_rrqHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic mixed-precision allocation for LLM quantization-aware training under a fixed precision budget. Q-PACE models the quantization-induced loss increase as curvature-weighted layer-wise errors, estimates sensitivities through Gaussian perturbations and forward evaluations, and periodically updates precision assignments using a multiple-choice knapsack solver.

Critical strengths

1.The method addresses a practical limitation of uniform and manually assigned precision. Layer-wise sensitivity estimation captures differences across layers, while periodic reallocation adapts to changes during training. 2.Forward-only calibration avoids explicit Hessian computation and calibration backward passes. Combined with an efficient knapsack solver, this provides a simple and interpretable allocation procedure.

Critical weaknesses

1.The theoretical model describes local loss changes at fixed weights under independent, zero-mean, isotropic weight noise. It does not establish how these estimates predict the effects of precision choices on the training trajectory. Activation and backward quantization are not explicitly modeled, and applying the same 4−b4^{-b} error curve across numerical formats introduces another simplifying assumption. 2.Average weight bitwidth is an incomplete proxy for practical efficiency, particularly because master weights and optimizer states remain in high precision. Despite reporting approximately 5% calibration overhead and inference memory curves, the paper lacks matched end-to-end comparisons with SNIP and the NVIDIA strategy in training time, peak memory, and throughput. 3.The benefit of continued layer-wise recalibration over simpler strategies remains unclear. Figure 5b supports multiple allocations over initialization-only allocation, but does not establish whether similar gains could be achieved by freezing the allocation after warmup or sharing sensitivity scores by layer type and depth. 4.Minor Issue: In Section 4.1 L254-L255, ‘matmix multiplications’ should be ‘matrix multiplications’.

Questions for authors

1.How are the points in Figures 2 and 7 constructed? Are sensitivities fitted at the final weights, and can estimates from intermediate checkpoints predict the effects of unseen allocations? 2.Including calibration and format conversion overhead, how does Q-PACE compare with SNIP and the NVIDIA strategy in training throughput, peak memory, and total GPU time to reach the same loss? 3.How much improvement does periodic recalibration provide over allocation frozen after warmup or scores shared by layer type and depth? Does this improvement justify the additional calibration cost?

3 comments

We thank the reviewer for their time and consideration. We address the questions and concerns below.

W1. Modelling assumptions of quantization noise model.

First, we revised the assumptions in derivation in Sec. 3, and one of them can be relaxed: noise does not have to be isotropic with Var⁡δWi=σi2I\operatorname{Var} \delta W_i = \sigma_i^2 I. Having second and third moments of δW\delta W to be finite suffices for the loss approximation to hold. We will include a formal proof in the revised version.

To justify two other modelling assumptions, we check if they hold empirically. We tested each assumption on the true quantization error Ei=Q(Wi)−WiE_i = Q(W_i) − W_i of a uniform-FP4 800M checkpoint in the middle of training (step 40k of 76k). ii index layers.

Zero-mean assumption holds: mean value of EiE_i is ⟨Ei⟩=(−0.9±7.5)×10−7\langle E_i \rangle = (-0.9 \pm 7.5)\times 10^{-7}. Independence across layers holds: we compute the Pearson correlation between errors for different layers r(Ei,Ej)=Corr⁡(Ei,Ej)Var⁡(Ei)Var⁡(Ej)r(E_i, E_j) = \frac{\operatorname{Corr} (E_i, E_j) }{\sqrt{\operatorname{Var}(E_i)} \sqrt{\operatorname{Var}(E_j)} } and average across layer pairs ⟨∣r(Ei,Ej)∣⟩=5⋅10−4\langle |r (E_i, E_j)| \rangle= 5\cdot 10^{-4}, max⁡ij∣r(Ei,Ej)∣=4⋅10−3\max_{ij} |r (E_i, E_j) |= 4 \cdot 10^{-3}. Therefore errors are effectively uncorrelated between layers.

Regarding model predictions of training trajectory: indeed, our loss model does not include interaction with an optimizer. As we mentioned in the paper, solving the full training trajectory problem (Eq. 2) is very costly. We justify our loss model empirically: see Figure 2, for example.

As for the activation and gradient-based loss models, see ablations in App.F Table 6. Sensitivity models using MSE of activations or gradients did not provide larger gains over our main method, so we did not extend these experiments past 350M models.

Regarding using Gaussian noise regardless of the actual numerical format: we compared analytical (Gaussian) noise with true measured noise in App.A. On small scale models the difference was within the noise interval. We decided on Gaussian noise to keep the method general. As a hypothetical example, for vector quantization with a large codebook cost of forward passes for recalibration might become significant and analytical noise would be cheaper. More importantly, forward quantization is deterministic in our case, and sampling noise from a distribution gives more points for sensitivity fit.

To verify that analytical noise is not hindering performance, we conducted additional experiments on the 800M model, see Table R1. Models trained with the analytic and with the measured quantization error reach the same final loss within seed noise. This is consistent with the fact that analytic and true noise probes produce similar allocations: on average, bit allocation for analytic vs true noise would only differ for 6 out of 102 model layers. The variation within one method across random seeds gives 7 layers on average. For reference, 29 to 35 of the 102 layers are in BF16 at this budget.

Table R1. Test loss for the 800M model (80B tokens) trained with dynamic precision allocation.

ModelBudget, bits/paramBit choicesMethodMean test loss (n=3)Std. of test loss (n=3)
800M6.0{4,16}NVIDIA2.60682.60682⋅10−42 \cdot 10^{-4}
6.0{4,16}SNIP2.60462.6046$3 \cdot 10^{-3} $
6.0{4,16}Q-PACE, analytic noise2.60292.60296⋅10−46 \cdot 10^{-4}
6.0{4,16}Q-PACE, true noise2.60282.60285⋅10−45 \cdot 10^{-4}

W2 + Q2.

We do not claim that memory is an optimal choice of cost function when allocating layers. Rather, it is one of the choices. An alternative could be a matmul latency of a layer at every bitwidth, which would constrain training throughput and inference time instead of memory. Note that in typical large-scale distributed training runs, optimizer state precision is fully decoupled from matmul precision, and optimizer states are sharded across many GPUs, reducing their impact on memory requirements.

Regarding end-to-end training comparisons between methods, we provide them below in Table R2 for the 4B models with recalibration period of 10k steps. Methods reach similar training speeds between uniform FP4 and FP8. Q-PACE has lower overall throughput due to recalibration procedure, but same memory cost as NVIDIA. The recalibration cost of Q-PACE can be further reduced by freezing the allocation after the first third of training (see Q3). SNIP’s calibration, however, needs multiple backward passes and using the same micro-batch size as training led to OOM error.

Table R2. 4B model, 6.5 bits / parameter: end-to-end efficiency on 16x B200 (Batch size 1.05M tokens, micro BS 8, context 4096 tokens).

MethodBudgetBit choicesfwd+bwd (s)Optim step (s)Tok/s/gpu (train only)Tok/s/gpu (with recalibration)Time to val loss 2.40NVML mean / max (GiB)
FP4–{4}1.0500.18451.2k51.2k29.2 h154.2 / 154.7
FP8–{8}1.4220.18040.1k40.1k32.0 h172.6 / 173.1
NVIDIA6.5{4,16}1.1490.19147.0k47.0k31.5 h152.8 / 154.7
SNIP6.5{4,16}1.1050.18448.8k48.5k30.6 h165.5 / 177.2
Q-PACE6.5{4,16}1.1300.18447.7k44.3k33.8 h154.2 / 154.7

W3 + Q3. We ran both ablations the reviewer suggests: freezing the allocation early in training, and sharing sensitivities across layer types (6 groups: Q, K, V, O, up, down) or across blocks, results are reported in Table R3.

Table R3. 4B model, 6.5 bits, {4,16}, 105B tokens: Q-PACE vs. shared sensitivities. Single seed.

AllocationFinal test lossDifference vs. Q-PACE
Q-PACE2.26562.2656–
Q-PACE per layer type2.26702.2670+0.0014+0.0014
Q-PACE frozen @ 30%2.26722.2672+0.0016+0.0016
Q-PACE per block2.26952.2695+0.0039+0.0039

For reference, same setting and seed: SNIP 2.26992.2699 (+0.0043+0.0043), NVIDIA 2.27092.2709 (+0.0053+0.0053). At this budget the variants have small separation: seed standard deviation in our multi-seed runs (350M to 800M) is at most 1e-3, so sharing per layer type (+0.0014) is at the edge of noise, while sharing per block (+0.0039) is a clear degradation.

To get more resolution we repeated the comparison at a much tighter budget: 350M models with fake integer quantization, 2.5 bits/parameter, formats INT{2,3,4,8}, 3 seeds each.

Table R4. 350M model, 3 seeds, fake integer quantization with group size 32.

ModelBudget, bits/paramBit choicesMethodMean test loss (n=3)Std. of test loss (n=3)Increase vs Q-PACE
350M2.5{2,3,4,8}Q-PACE2.91171.4⋅10−31.4 \cdot 10^{-3}—
350M2.5{2,3,4,8}Q-PACE frozen @ 30% of training2.91731.5⋅10−31.5 \cdot 10^{-3}+0.0056
350M2.5{2,3,4,8}Q-PACE per layer type2.91941.0⋅10−31.0 \cdot 10^{-3}+0.0077
350M2.5{2,3,4,8}SNIP2.91987⋅10−47 \cdot 10^{-4}+0.0081
350M2.5{2,3,4,8}Q-PACE per block2.93498⋅10−38 \cdot 10^{-3}+0.023
350M2.5{2,3,4,8}NVIDIA2.95387⋅10−47 \cdot 10^{-4}+0.042
350M2{2}fixed INT22.97061.4⋅10−31.4 \cdot 10^{-3}+0.059

We interpret the results the following way: simplified Q-PACE variants recover most of the benefit, which is consistent with our finding in Section 5 that sensitivity is structured by layer type and depth and changes slowly. Overall, Q-PACE with individual sensitivities for each layer remains the best method. Since time overhead is the main cost of Q-PACE (memory is negligible), it can be reduced via using sensitivity per layer type, or by using a frozen allocation at a small performance cost.

W4. Thanks for notiсing, we will fix it.

Q1. Each point in the plot corresponds to a fully trained model from Table 1. The x-axis shows predicted loss increase due to precision allocation, ∑iαiei\sum_i \alpha_i e_i, based on the analytic noise error 4−bi4^{-b_i} in Fig. 2 and measured error in Fig. 7b. The y-axis shows the actual loss increase due to bit allocations after training is done.

can estimates from intermediate checkpoints predict the effects of unseen allocations?

Q-PACE fits sensitivities that predict loss increase for any bit allocation for a static model. As the training progresses, absolute sensitivity values decrease, but relative to each other they stay stable. This leads to optimal allocation also being stable over time.

We appreciate the reviewer's points and see that addressing them would strengthen the paper. We will update the manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.