acceptodds
Under review as a conference paper at ICLR 2027

Beyond Shadow Weights: Quantization-Aware Training as Quantized-Endpoint Descent

11 views

Abstract

Quantization-aware training (QAT) updates a full-precision shadow weight but deploys the quantized endpoint . Existing explanations for QAT largely view its success through the lens of shadow weights: QAT can move toward flatter basins, gain robustness from oscillations, or balance the shadow loss against the quantization error . These perspectives do not directly explain the empirical observation that the deployed endpoint loss improves even as the shadow loss increases substantially. In this paper, we offer a different explanation by treating QAT as finite-grid endpoint dynamics. Motivated by the approximate normality of rescaled pretrained weights, we propose an idealized residual-phase model that yields a crossing law that determines which coordinates cross quantization boundaries after a shadow update. We also find that signal imbalance creates asymmetric oscillations that favor weaker-gradient endpoints. This motivates QAR (Quantization with Amplified Routing), an algorithmic framework that directly operates on the quantization code. We further prove feasible-gradient guarantees for QAR with a family of power amplifier. Post-training experiments on three large language models support the endpoint view and show that QAR can achieve performance comparable to QAT with up to 24.1% peak memory saving.

Primary area:
optimization
Keywords:
quantization-aware training, large language models, straight-through estimator, crossing law, signal imbalance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GddKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the weight distribution dynamics during QAT. They observe that while the loss for the quantized model improves during QAT, the loss for the full-precision model gets worse. They analyze the residual of weight changes and raise a new perspective that "signal imbalance is a source of asymmetric oscillations that favor weaker-gradient endpoints." From that analysis, they propose QAR, a method for QAT that favors strong signals and uses a smaller memory footprint. On three models, they achieve similar loss and accuracy while using less memory compared to the traditional QAT method.

Critical strengths
  1. I think formalizing the weight dynamics during QAT is informative for some communities. I see that this paper belongs to the optimization community and could potentially be interesting to people with more theoretical interests.
  2. A memory-efficient QAT method will be useful for many applications. QAT is necessary for some downstream tasks. But on non-industry-level GPUs, memory is indeed a big issue. QAR will be useful to enable some workflows on, for example, personal computers.
Critical weaknesses
  1. The paper is quite hard to follow. I would suggest making the main text a coherent story and introducing the analysis through words instead of equations and symbols. I feel like you don't need to talk about all the assumptions and modeling in the main text, as it just confuses people and gets worse with a lot of definitions/terminologies that are not commonly used in other literature (see my questions below, for example).
  2. The evaluation to confirm the efficiency of QAR is not complete. Table 2 shows that QAR saves memory, but we don't know if it saves time. The sampling and code manipulation in the QAR might introduce extra latency that is not disclosed in the current experiments. Also, the current PTQ method is RTN only. From my experience, I don't remember the degradation of PPL/accuracy being that big for 4-bit weight-only quantization for those better methods, and I wonder: a. how QAR works together with different PTQ methods and b. how QAR works with weight+activation quantization.
Questions for authors

Below are some questions that I seek clarification on regarding the paper:

  1. Do I understand correctly that "signals" are weight magnitudes? Strong signals mean big weights, and weak signals mean small weights.
  2. Do I understand correctly that QAR is more memory-efficient than standard QAT in that they do not work on full-precision weights, but instead works on quantized weights only? So the benefit over QAT is that QAR does not need to store a copy of the shadow weight.

Below are some technical questions:

  1. The paper establishes that QAR-0 recovers DQT under the stated conditions. However, the main experiments compare QAR primarily against conventional QAT. Could the authors provide a direct comparison between QAR-0 and QAR-(gamma) on three LLMs, using comparable hyperparameter tuning budgets?

  2. What are QAR's actual computational costs relative to QAT? Please report the time used to train QAT vs. QAR.

  3. If we change the PTQ method, what happens to the two QAT model variants?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GddK (1/3)

We thank Reviewer GddK for the review and the helpful questions. We address your concerns below and changes in the revised PDF are marked in green.

Q1: Are "signals" weight magnitudes?

The signal of a weight is the size of its gradient (more precisely, of the optimizer update) at the deployed quantized weight, not the size of the weight. A strong signal means a large update, regardless of the size of the weight. Section 4.1 defines the signal as the update magnitude.

Q2: Is the memory benefit that QAR stores no shadow weight?

Yes. QAR stores the integer codes, scales, and optimizer moments but no full-precision shadow weight, which saves 3 to 3.5 bytes per quantized weight at 4 and 8 bits compared with FP32-shadow QAT (Appendix B.10). In our measurements, this lowers the persistent memory at 4 bits by 27–29% on all four models (Table 2).

Q3: QAR-0 vs QAR-γ on the LLMs with comparable tuning budgets.

Table 1 now includes DQT on all four LLMs, tuned on the same grid with three seeds; the γ ablation (Appendix B.6, Fig. 12) also includes γ = 0, which coincides with tuned DQT at these learning rates. C4 loss at 4 bits:

Qwen-2.5-0.5BLlama-3.2-3BLlama-2-7BDeepSeek-MoE-16B
DQT3.317 ± 0.0022.691 ± 0.0032.138 ± 0.0022.422 ± 0.000
QAR-γ3.297 ± 0.0022.675 ± 0.0022.135 ± 0.0022.414 ± 0.000
Gain0.020 ± 0.0000.017 ± 0.0020.003 ± 0.0020.008 ± 0.000
  • At 4 bits, QAR-γ has a lower C4 and a lower WikiText-2 loss than DQT on all four models; on Qwen-2.5-0.5B the gain also holds at a matched number of transitions (Appendix B.7).
  • At 8 bits, the grid is fine enough that both methods recover the full-precision model, and they differ by at most 0.003.

W2 & Q4: Training time of QAT vs QAR.

Table 2 now reports the persistent memory and the time per training step. To make the comparison fair, we give QAT the same memory-saving implementation as QAR-γ: it applies a fused AdamW step to each weight during the backward pass, which by itself lowers QAT's peak memory by 8–22%. On DeepSeek-MoE-16B, both methods update all experts of a layer in one batched call, and the clock stops after all GPUs finish. QAR-γ compared with QAT at 4 bits:

Persistent memoryTime per step
Qwen-2.5-0.5B27% less5.6% faster
Llama-3.2-3B28% less7.2% slower
Llama-2-7B29% less (one GPU instead of two)5.2% slower
DeepSeek-MoE-16B29% less (same three GPUs)4.6% faster
  • The peak memory per GPU is 26–28% lower on Qwen-2.5-0.5B, Llama-3.2-3B, and DeepSeek-MoE-16B, and Llama-2-7B fits on one GPU with QAR-γ but needs two with QAT (Appendix B.10), so QAR-γ needs 47% fewer GPU-seconds per step on this model (1.693 s on one GPU against 1.610 s on two).
  • QAR-γ computes its transition probabilities, samples the transitions, and updates the codes inside one fused kernel (Appendix B.10), so there is no separate sampling pass. The difference in time comes from the part of a step that scales with the number of weights rather than with the computation: forming the forward weights and applying the update (Appendix B.10). On the Llama models the matrix multiplications dominate, and QAR-γ's update kernel costs 5–7% of a step more than QAT's fused AdamW. On DeepSeek-MoE-16B each token activates only about 2.8B of the 16.4B weights, so the per-weight part is a larger share of a step, and QAR-γ, which forms its forward weights from codes instead of fake-quantizing an FP32 shadow, is about 5% faster.
AuthorsOpenReview ↗

Response to Reviewer GddK (2/3)

W2 & Q5: Other PTQ methods.

"If we change the PTQ method, what happens to the two QAT model variants?"

Thank you for this suggestion. The revision adds GPTQ as a PTQ baseline for every model at 4 and 8 bits (Table 1, three calibration sets).

The two kinds of training also start from different points. QAT keeps a full-precision shadow weight, so it starts from the pretrained weights and uses the local information they carry. DQT and QAR-γ are endpoint training: they keep no shadow weight and update the quantized weights directly. On the other hand, this lets them start directly from a PTQ solution and leverage what the PTQ method has already achieved, so in the revision both start from GPTQ and keep improving it. The comparison at the same start and the same memory is therefore QAR-γ against DQT (Q3). On Llama-2-7B at 4 bits, for example, GPTQ reaches a C4 loss of 2.209, and QAR-γ lowers it to 2.135; at 4 bits QAR-γ improves on its GPTQ start in C4 loss, WikiText-2 loss, and zero-shot accuracy on all four models.

Appendix B.9 compares an RTN and a GPTQ start on Qwen-2.5-0.5B, each method at its best learning rate (Table B.9, tuning C4 loss, three seeds):

RTN, 4 bitsGPTQ, 4 bitsRTN, 8 bitsGPTQ, 8 bits
PTQ only (no training)3.6823.2152.9942.992
DQT3.299 ± 0.0023.177 ± 0.0032.977 ± 0.0002.976 ± 0.000
QAR-γ3.258 ± 0.0033.157 ± 0.0012.976 ± 0.0012.976 ± 0.001

A better start helps, most clearly at 4 bits. At 4 bits QAR-γ improves on DQT from either start (by 0.042 from RTN and 0.020 from GPTQ); at 8 bits the two are within 0.001.

W2: Weight and activation quantization.

"… and b. how QAR works with weight+activation quantization."

Our work focuses on weight-only quantization: the deployed weights live on a fixed grid, and the full-precision shadow weight is the memory cost that QAR removes. The Limitations name joint weight-activation quantization as a direction for extending the endpoint view.

QAR only changes how the weights are updated, so it combines directly with an activation quantizer in the forward pass. As a quick check, we quantized the input of every quantized linear layer to 8 bits (dynamic symmetric scale per token, straight-through gradients, the same quantization at evaluation) and trained Qwen-2.5-0.5B with 4-bit weights at the learning rates of Table B.2, with QAT started from the full-precision weights and QAR-γ from GPTQ, as in Table 1. We show the W4A8 evaluation loss here:

C4WikiText-2
QAT3.3613.161
QAR-γ3.3183.118

With 8-bit activations, QAR-γ keeps a lead of about 0.04 over QAT in both C4 and WikiText-2 loss. Lower-bit activations usually call for rotations or smoothing against activation outliers, such as QuaRot; these act on the activations and can be combined with QAR's weight update.

AuthorsOpenReview ↗

Response to Reviewer GddK (3/3)

W1: Readability of the main text.

"I would suggest making the main text a coherent story and introducing the analysis through words instead of equations and symbols."

Thank you for this suggestion. We would like to clarify the storyline of the paper and summarize the changes we made for readability.

The main text follows one storyline, where each step leads to the next (the new Contributions and roadmap paragraph of the Introduction gives the section or theorem behind each contribution):

  1. Observation: the deployed loss falls while the shadow loss rises (Section 1, Fig. 1). During QAT, the shadow loss can rise substantially while the deployed loss falls. Existing explanations look at the shadow weights and cannot account for this, so we turn to the deployed weights.
  2. Mechanism: residuals turn STE updates into one-cell moves (Section 3). To see how the deployed weights move, we track the residual of each shadow weight inside its cell. A straight-through estimator (STE) step smaller than a cell moves the deployed weight by at most one cell, and when this residual is close to uniform, the probability of the move is set by the size of the update. This one-cell mechanism is the basis for the rest of the paper.
  3. Signal imbalance motivates us to favor stronger signal for descent (Section 4.1). Through the same mechanism, we find that QAT tends to settle at endpoints with weak gradients: a weak signal lets the shadow drift slowly across a boundary, while a strong signal pulls it back quickly. This suggests giving strong signals more transitions, which is the idea behind our algorithm.
  4. Algorithmic framework: QAR (Sections 4.2 and 4.3). We therefore sample the one-cell moves directly on the quantized weights, without a shadow weight, and amplify the transitions of strong signals. QAR comes with convergence guarantees on the finite grid (Theorem 4).
  5. Evidence (Section 4.4). Experiments on four LLMs, including a 16B mixture-of-experts model, confirm this picture: at 4 and 8 bits, QAR-γ matches or outperforms QAT, and at 4 bits it stores 27–29% less persistent memory.

Changes for readability in the revision:

  • The contributions in the Introduction are stated in more detail, each with the section or theorem where it is developed.
  • Section 3 now explains in words how the residual phase follows from the pretrained weights.
  • The notation in Sections 2 and 4 is streamlined.
  • Terms such as the signal of a weight are defined in plain words (Section 4.1).
  • The experiments state their findings in plain language.

We keep the two assumptions and the two crossing models in the main text because each of them carries a step of this storyline:

  • Two classic assumptions, used in the main results. Assumption 1 (bounded update direction) and Assumption 2 (coordinate-wise smoothness, standard in sign-gradient analyses such as signSGD) are the standing assumptions of the analysis. Proposition 2, Theorem 3, and Theorem 4 all rest on them, so we state them once in Section 2 and the main results refer back to them.
  • Two models that form one pipeline. Model 1 states the crossing law in general form, the probability that a step moves a deployed weight by one cell, and yields the endpoint descent bound (Theorem 3). Model 2 is its special case under a near-uniform residual, where this probability becomes η∣ui∣/si\eta|u_i|/s_i. QAR with the uniform amplifier samples exactly this law, and the power amplifier reweights it toward strong signals (Section 4.2), so Model 2 is the bridge from the analysis of QAT to the algorithm.
  • The residual analysis is kept short. The main text gives the residual phase in words, and its derivation is in Appendix C.

We would also welcome more specific suggestions: if there are parts of Sections 2–4 that you would prefer to see moved to the appendix or rephrased, we are glad to revise them accordingly.

We hope these answers address your concerns and are happy to clarify further.

References

[1] Zhao, K., Tabaru, T., Kobayashi, K., Honda, T., Yamazaki, M., and Tsuruoka, Y. Direct Quantized Training of Language Models with Stochastic Rounding. ACML, 2025.

[2] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.

[3] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. NeurIPS, 2024.

[4] Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed Optimisation for Non-Convex Problems. ICML, 2018.

Reviewer_fmsQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes QAR, which is a method to remove storing shadow weights. The authors present an extensive analysis of Quantization Aware Training, and then propose directly updating quantized weights using power amplification. By removing shadow weights, they are able to reduce GPU memory usage significantly.

Critical strengths
  1. Authors provide a nice general observation that QAT can improve the quantized model's loss even when the shadow model loss becomes worse.
  2. Relatively substantial reduction in memory footprint, which seems like a worthwhile result
Critical weaknesses
  1. The novelty/contribution does not appear to be that significant -- the DQT paper seems to have most of these details. The main difference being the power amplification, leaving the contribution a little narrow.
  2. The paper relies on the detailed explanation of QAT for its novelty.
  3. The QAT comparison is confounded by numerical precision and optimizer choices. Appendix B.3 shows that BF16 shadow weights cause severe late-training suppression of QAT’s endpoint updates. FP32 shadows remove this behavior, but the corresponding accuracy comparison against QAR is not reported. QAT and QAR also use different AdamW coefficients without a controlled ablation. The reported accuracy and learning rate robustness gains do not clearly separate the benefit of amplified routing from avoiding numerical stalling or changing optimizer settings.
  4. Model 2 assumes coordinate level crossing probabilities conditional on the endpoint and update direction, whereas the supporting diagnostics primarily examine pooled residual distributions and aggregate crossing rates. Agreement in these averages does not establish the gradient weighted crossing behavior required by the descent analysis. Moreover, the uniform approximation is least accurate at 2 bits, where the strongest amplification benefit is reported. The connection between the idealized model and the regime motivating the method is incomplete.
  5. The QAR-0/DQT ablation is limited to a single-seed sweep on Qwen-0.5B, with no corresponding baseline for the larger Llama models. Furthermore, amplification increases the total expected number of endpoint transitions as well as changing their allocation across coordinates.
Questions for authors
  1. How does QAR perform across groups of weights with different gradient magnitudes? Does power amplification consistently improve the updates for these different groups?
  2. Is the proposed memory saving specific to QAR or does this also apply to DQT?
  3. Could the authors evaluate QAR on a larger, non-dense LLM to show that their results continue to generalize?
  4. Could the authors compare QAR against FP32-shadow QAT using matched or systematically tuned AdamW settings? Do the accuracy and learning rate robustness advantages persist after eliminating the BF16 update stalling effect documented in Appendix B.3?
  5. Could the authors validate Model 2 after stratifying coordinates by endpoint code, update sign, and update magnitude, particularly at 2 bits? How do the observed deviations from the assumed crossing probabilities affect the endpoint descent explanation?
  6. Could the authors compare amplified QAR with QAR-0/DQT on the Llama models using multiple seeds and comparable tuning budgets? Does amplification retain an advantage when the overall transition count is controlled?
7 comments
AuthorsOpenReview ↗

Response to Reviewer fmsQ (1/4)

We thank Reviewer fmsQ for the careful review and concrete suggestions. We address your concerns and provide further clarifications below.

W1 & W2: Novelty relative to DQT, and the role of the explanation of QAT.

"… the DQT paper seems to have most of these details. The main difference being the power amplification, leaving the contribution a little narrow."

We respectfully disagree with this assessment. Starting from a one-cell analysis of how straight-through estimator (STE) updates move the deployed weights, the paper derives a more general and principled algorithmic framework, QAR, together with finite-grid guarantees.

On the other hand, in the one-cell regime DQT is only a narrow special case of this framework: QAR with the uniform amplifier (γ=0\gamma = 0). The power amplifier, which moves the transitions to strong signals, is the new algorithmic component, and the one-cell analysis tells us both to amplify and to keep each amplified transition to one cell: rounding a full step across several cells can cycle on a finite codebook (Proposition 11), and for the amplified steps of QAR-γ, multi-cell rounding is worse than one-cell transitions on Qwen-2.5-0.5B at 2 bits (Appendix B.8).

The paper develops this in four steps:

1. A new explanation of QAT: the one-cell mechanism. Existing explanations of QAT focus on its shadow weights (flat regions, proximal regularization, oscillations). We instead follow the deployed weights: an STE step smaller than a cell acts on the deployed weight as a random move of at most one cell, and when the residual of the shadow weight inside its cell is close to uniform, the probability of this move is proportional to the update (Section 3). To our knowledge, no prior work derives the crossing law of STE-based QAT from the residuals of its shadow weights and turns it into a descent bound for the deployed weights. This view explains what the shadow-weight views do not: the deployed loss falls while the shadow loss rises, which you call "a nice general observation" (Fig. 1, Fig. 5).

2. What the mechanism implies on a finite grid. Every move is a full cell, so its curvature cost is O(η)O(\eta) rather than O(η2)O(\eta^2), and a smaller learning rate alone does not guarantee descent (Theorem 3); what matters is which weights move. The same view shows how signal imbalance makes QAT favor weak-gradient endpoints (Section 4.1).

3. From the mechanism to an algorithm. QAR samples the one-cell moves directly, with a probability that factors into a route and an amplifier, and the power amplifier moves the transitions to strong signals. Theorem 4 gives a convergence guarantee in terms of a feasible endpoint gradient that accounts for the edges of the codebook, and Proposition 11 shows that rounding a full step across several cells can instead cycle, which is why QAR keeps every transition to one cell.

4. Evidence. At 4 bits, from the same GPTQ start and at the same memory, the power amplifier lowers the C4 and WikiText-2 loss of DQT on all four models (Table 1; paired gains in Q6), and on Qwen-2.5-0.5B the gain also holds at a matched number of transitions (Q6).

The one-cell transition also beats multi-cell rounding of the same amplified step. On Qwen-2.5-0.5B at 2 bits (Appendix B.8; tuning C4 loss, three seeds), multi-cell rounding raises the loss as soon as the cap binds. At 1e-4, where the cap binds on only 0.58% of the transitions, multi-cell rounding raises the loss by 0.019, i.e., a 1.9% higher perplexity:

Learning rateTransitions cappedOne-cell (QAR-γ)Multi-cell rounding
2e-5<0.01%5.7285.734
5e-5 (selected)0.06%5.6975.702
1e-40.58%5.7125.731
AuthorsOpenReview ↗

Response to Reviewer fmsQ (2/4)

Q1: Does amplification help groups of weights with different gradient magnitudes?

Yes. Amplification moves the transitions toward the weights with the largest updates, and every group of layers we amplified improves on DQT (Appendix B.7).

  • Where the transitions go. For a weight ii in row GG, let νi=∣ui∣/max⁡j∈G∣uj∣\nu_i = |u_i| / \max_{j \in G} |u_j| be the size of its update relative to the largest one in the row. Amplification moves the transitions to the weights with the largest νi\nu_i: at 2 bits, 96% of the transitions of QAR-γ come from the 10% of weights with the largest νi\nu_i, against 23% for DQT (94% against 23% at 4 bits).
  • Amplifying one group at a time. On Qwen-2.5-0.5B at 2 bits, we apply QAR-γ to one group of layers and train all other layers with DQT (Table B.8; tuning C4 loss, three seeds):
Amplified layersShare of weightsC4 lossvs. DQT
None (DQT)0%6.053 ± 0.025–
Attention (q, k, v, o)12%5.900 ± 0.021−0.153
MLP (gate, up, down)88%5.894 ± 0.012−0.159
Blocks 1–833%5.883 ± 0.039−0.170
Blocks 9–1633%5.993 ± 0.026−0.059
Blocks 17–2433%5.901 ± 0.010−0.151
All (QAR-γ)100%5.697 ± 0.015−0.355

Every group improves on DQT. Amplifying only the attention projections, 12% of the weights, already gives 43% of the full gain. The gain also holds at a matched number of transitions (see our answer to Q6).

Q2: Does the memory saving also apply to DQT?

Yes. DQT also trains the quantized weights directly without a shadow weight, so DQT has the same memory saving as QAR-γ (Appendix B.10). At the same memory, QAR-γ has a lower C4 and WikiText-2 loss than DQT at 4 bits on every model (Q6).

Q3: A larger, non-dense LLM.

We have added DeepSeek-MoE-16B: 16.4B parameters, about 2.8B active per token, and 2 shared plus 64 routed experts per MoE layer with top-6 routing, evaluated with the same protocol (Table 1; C4 loss / average zero-shot accuracy, three seeds):

4-bit8-bit
QAT2.425 / 62.872.374 / 64.24
DQT2.422 / 63.332.374 / 64.16
QAR-γ2.414 / 63.382.372 / 64.32
  • QAR-γ has the lowest C4 loss and the highest accuracy at both bit widths.
  • QAT needs three GPUs. At 4 bits on the same three GPUs, QAR-γ stores 29% less persistent state, its peak memory per GPU is 27% lower, and a step is about 5% faster (Table 2, Appendix B.10).
AuthorsOpenReview ↗

Response to Reviewer fmsQ (3/4)

W3 & Q4: FP32-shadow QAT with matched or tuned AdamW settings.

We have rerun all QAT baselines with an FP32 shadow, shared AdamW settings, the same learning-rate grid, and three seeds (see the general response and our answer to Reviewer aa1C, W1 & Q1).

  • At 4 bits, QAR-γ has lower C4 and WikiText-2 loss and higher zero-shot accuracy than FP32-shadow QAT on all four models (Table 1).
  • At 8 bits, all trained methods match the full-precision model.
  • With a BF16 shadow, QAT is worse than with an FP32 shadow at every bit width (Appendix B.4).
  • Fig. 13 shows each method's learning-rate sweep on Qwen-2.5-0.5B, and Table B.2 lists the selected rates for all models.

W4 & Q5: Validating Model 2 after stratifying by endpoint code, update sign, and update magnitude.

"How do the observed deviations from the assumed crossing probabilities affect the endpoint descent explanation?"

Thank you for this suggestion, which led to a new stratified analysis (Appendix B.5, Fig. 11, Table B.7). Model 2 is a simplified model of how STE updates move the deployed weights, and the new analysis measures how closely QAT follows it in each group of weights.

  • Weights are grouped by endpoint code and update direction (toward or away from the grid centre), and each group is binned by the predicted crossing probability, which is proportional to the update magnitude. This tests the "gradient weighted crossing behavior" you point out, not only the pooled average. Counts are pooled over three seeds.
  • At 4 and 8 bits, the observed crossing frequency is proportional to the prediction in every group of Table B.7 (log-log slopes 0.98–1.11). Groups differ only by a constant factor that depends on the code and the direction of the move, not on the size of the update.
  • At 8 bits, every group is within 7% of the model. At 4 bits, the centre code is within 1%, and moves toward or away from the grid centre cross 1.39 and 0.76 times as often as predicted.
  • Effect on the descent explanation: because the factor does not depend on the update size, crossings still move against the update and the first-order term of Theorem 3 keeps the sign it has under Model 2; at 4 bits its size changes by a factor of 0.76–1.39.
  • At 2 bits, the deviation is the one Section 3 predicts. Section 3 derives the residual phase from the pretrained weights: at 3 bits and above it is close to uniform, which is where Model 2 applies, while at 2 bits the only non-saturated cell is the zero cell, so the phase is a truncated normal concentrated at the cell centre. This truncated normal, built from the pretrained weights alone, captures both the direction and the size of the 2-bit deviation: it predicts crossing factors of 0.24 for the centre code and 7.9 for the other codes, against measured 0.32 and 7.1. Model 2 therefore captures the crossing law at 4 and 8 bits up to constant factors (within 7% at 8 bits and 0.76–1.39 at 4 bits), and the truncated-normal phase of Section 3 accounts for the 2-bit factors.
Reviewer_aa1COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies QAT through the dynamics of deployed quantized weights rather than full-precision shadow weights. It introduces an idealized conditional crossing model, derives endpoint-descent bounds, and motivates QAR, which directly samples discrete weight transitions with signal-dependent power amplification. The analysis establishes feasible-gradient bounds with a finite-grid error floor for gradient and sign-gradient variants, including stochastic extensions. Experiments with an AdamW implementation on three language models report broadly comparable quality to QAT and up to 24.1% lower peak memory.

Critical strengths
  1. A useful analysis of quantized endpoint dynamics. Figures 1 and 7–8 demonstrate that endpoint loss can improve while shadow loss increases, motivating the proposed perspective. Theorem 3, particularly Eq. (6), distinguishes first-order progress from the curvature cost of finite grid jumps: both terms scale linearly with the learning rate under the crossing model. This clarifies why decreasing the learning rate alone does not guarantee endpoint descent. Theorem 4 further accounts for feasibility at codebook boundaries.

  2. A concrete and transparently measured memory benefit. For 4-bit Llama-2-7B, Table 2 reports a reduction from 37.43 to 28.42 GiB (24.1%) in the controlled memory configuration. Table 1 separately reports improved C4 perplexity (10.07 → 9.28) and mean downstream accuracy (60.00% → 61.22%) relative to the reported QAT baseline. Appendix B.5 explains the packed weight representation and streams updates for both methods, strengthening the fairness of the memory comparison.

Critical weaknesses
  1. The quality comparison is confounded by documented numerical stalling in the QAT baseline. Appendix B.3 attributes the late-training collapse of QAT boundary crossings to BF16 rounding discarding small shadow-weight updates. Figure 12 shows that an FP32-shadow control removes this collapse, but reports crossing rates rather than final model quality. Furthermore, Table B.1 uses different AdamW parameters: (0.9, 0.95) for QAT versus (0.95, 0.9) for QAR. Consequently, the reported quality and learning-rate-robustness advantages cannot yet be cleanly attributed to the proposed routing mechanism. This does not undermine the measured memory saving, but it limits the optimization conclusions.

  2. The incremental benefit of power amplification over existing direct quantized training is insufficiently established on the larger models. The paper appropriately recognizes that QAR-0 recovers DQT under the stated fixed-grid, one-cell conditions (§4.2). DQT already eliminates persistent shadow weights through stochastic rounding. However, the amplification ablation is restricted to Qwen-2.5-0.5B with one seed (Appendix B.4, Fig. 13): the benefit is substantial at 2 bits, small at 4 bits, and absent at 8 bits, where γ\gamma = 0 performs best. Table 1 does not include QAR-0 for the 3B/7B models. Thus, the larger-model results do not isolate the contribution of amplification from direct quantized updates, which already provide the structural memory saving.

Questions for authors
  1. Baseline control: Does QAR retain its reported quality advantage against FP32-shadow QAT when AdamW parameters and hyperparameter-search budgets are controlled? Please report final loss/perplexity and downstream performance, beyond the crossing-rate diagnostic.

  2. Amplification contribution: How does QAR-0/DQT compare with QAR-γ\gamma on the Llama models under matched training budgets? Please specify the selected learning rates and γ\gamma values. For the 4-bit Llama-2-7B result, additional seeds would help determine whether the reported 1.22-point downstream improvement is consistent.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aa1C (1/2)

We thank Reviewer aa1C for the detailed and constructive review. We sincerely appreciate your positive assessment of our theoretical and empirical contributions, and address your concerns below.

W1 & Q1: Comparison with FP32-shadow QAT under a controlled protocol.

"Does QAR retain its reported quality advantage against FP32-shadow QAT when AdamW parameters and hyperparameter-search budgets are controlled?"

Thank you for this suggestion. All methods now follow a controlled protocol:

  • QAT keeps an FP32 shadow weight.
  • QAT, DQT, and QAR-γ use AdamW with (β1,β2)=(0.95,0.95)(\beta_1, \beta_2) = (0.95, 0.95), FP32 moments, and the same schedule.
  • Each method's learning rate is tuned on the same grid (Table B.1, selected rates in Table B.2, and Fig. 13 shows the sweeps on Qwen-2.5-0.5B), and the grid is extended by the same rule for every method whenever the best rate lies on its edge, so all methods follow the same tuning protocol.
  • Every trained row of Table 1 uses three seeds (Table B.1).

Your comment also led to an ablation of the shadow precision (Appendix B.4, Fig. 10): with a BF16 shadow, QAT stops moving once its updates fall below half of the local BF16 spacing, and its best tuning C4 loss is worse than with an FP32 shadow by 0.34, 0.041, and 0.005 at 2, 4, and 8 bits. We therefore use the FP32 shadow throughout.

QAT starts from the pretrained weights, which its full-precision shadow stores directly; DQT and QAR-γ have no shadow weight and start from a GPTQ solution of the quantized point.

Final quality at 4 bits (Table 1, C4 loss / nine-task zero-shot accuracy, mean over three seeds):

Qwen-2.5-0.5BLlama-3.2-3BLlama-2-7BDeepSeek-MoE-16B
Full precision3.130 / 51.662.610 / 64.242.113 / 64.632.411 / 63.97
QAT (FP32 shadow)3.350 / 47.232.693 / 61.362.143 / 62.352.425 / 62.87
QAR-γ3.297 / 48.082.675 / 61.832.135 / 62.602.414 / 63.38
  • At 4 bits, QAR-γ has lower C4 and WikiText-2 loss and higher accuracy than FP32-shadow QAT on all four models.
  • At 8 bits, QAT, DQT, and QAR-γ all match the full-precision model: their zero-shot averages are within 0.05 points of it or above, and their C4 losses are within 0.003 of one another.
AuthorsOpenReview ↗

Response to Reviewer aa1C (2/2)

W2 & Q2: Amplification vs QAR-0 / DQT on the larger models; selected learning rates and γ; more seeds on Llama-2-7B.

"How does QAR-0/DQT compare with QAR-γ on the Llama models under matched training budgets?"

Table 1 now includes DQT on all four models, tuned on the same grid with three seeds. C4 loss at 4 bits:

Qwen-2.5-0.5BLlama-3.2-3BLlama-2-7BDeepSeek-MoE-16B
QAT3.350 ± 0.0012.693 ± 0.0002.143 ± 0.0012.425 ± 0.001
DQT3.317 ± 0.0022.691 ± 0.0032.138 ± 0.0022.422 ± 0.000
QAR-γ3.297 ± 0.0022.675 ± 0.0022.135 ± 0.0022.414 ± 0.000
Gain over DQT0.020 ± 0.0000.017 ± 0.0020.003 ± 0.0020.008 ± 0.000

(Mean ± standard deviation over three seeds; the gain is paired by seed.)

  • At 4 bits, QAR-γ has a lower C4 and WikiText-2 loss than DQT on all four models. Its C4 loss is lower by 0.008–0.020 on Qwen-2.5-0.5B, Llama-3.2-3B, and DeepSeek-MoE-16B and by 0.003 on Llama-2-7B, and its WikiText-2 loss is lower on all four.
  • At 8 bits, both methods recover the full-precision model. The 8-bit grid is fine enough that every trained method matches the full-precision model (Table 1), which leaves little room for any update rule: QAR-γ and DQT differ by at most 0.003 in C4 and WikiText-2 loss on every model.
  • The gain grows on coarser grids. In the γ ablation on Qwen-2.5-0.5B (Appendix B.6, Fig. 12), γ = 16 lowers the loss of the uniform amplifier (γ = 0, which coincides with tuned DQT at these learning rates) by 0.36 in tuning loss at 2 bits. Under the protocol of Table 1, QAR-γ also lowers the 2-bit C4 loss of DQT by 0.34–0.75 on all four models, from the same GPTQ start (Appendix B.12, Table B.11).
  • At 2 bits, QAR-γ has a lower C4 loss than QAT on three of the four models: Llama-3.2-3B (5.227 vs 5.413), Llama-2-7B (4.606 vs 5.258), and DeepSeek-MoE-16B (4.320 vs 4.700); on Qwen-2.5-0.5B, QAT is lower (5.700 vs 5.799) (Table B.11). We do not base the method comparison on 2 bits, where Model 2 no longer describes QAT: its residual phase is a truncated normal rather than near-uniform (Section 3).
  • The gain comes from which weights move. On Qwen-2.5-0.5B, at the same number of transitions per step, QAR-γ is better than DQT by 0.26–0.38 at 2 bits and 0.020–0.023 at 4 bits in tuning C4 loss (Appendix B.7, Fig. 14).
  • Selected γ and learning rates. One γ per bit width for all models, chosen by the ablation in Appendix B.6: γ = 16 at 4 bits and γ = 1 at 8 bits. Each method's initial learning rate is selected on the same 1-2-5 grid (Table B.2; 4,000 steps at 4 bits and 2,000 at 8 bits):
4 bits / 8 bitsQwen-2.5-0.5BLlama-3.2-3BLlama-2-7BDeepSeek-MoE-16B
QAT5e-6 / 1e-62e-6 / 1e-62e-6 / 5e-75e-6 / 1e-6
DQT2e-7 / 1e-62e-7 / 1e-62e-7 / 5e-71e-6 / 1e-6
QAR-γ1e-7 / 5e-75e-8 / 5e-75e-8 / 2e-72e-7 / 1e-6
  • Llama-2-7B at 4 bits with three seeds. QAR-γ reaches a C4 loss of 2.135 ± 0.002, against 2.143 ± 0.001 for FP32-shadow QAT and 2.138 ± 0.002 for DQT, and an average accuracy of 62.60 ± 0.23, against 62.35 ± 0.08 and 62.73 ± 0.09. The gap to QAT is smaller than in the submission, whose QAT used a BF16 shadow; on Qwen-2.5-0.5B, such a shadow raises QAT's best 4-bit tuning C4 loss by 0.041 (Appendix B.4).

We hope these answers address your concerns and are happy to clarify further.

References

[1] Zhao, K., Tabaru, T., Kobayashi, K., Honda, T., Yamazaki, M., and Tsuruoka, Y. Direct Quantized Training of Language Models with Stochastic Rounding. ACML, 2025.

[2] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.

Thank you for further experiments. The FP32 controls and full model DQT comparisons address the corresponding concerns in my review. However, QAT starts from pretrained shadow weights while DQT/QAR start from GPTQ endpoints, so this comparison with QAT does not isolate the effect of the update rule. The additional 2-bit results further strengthen the evidence for amplification, but its Llama-2-7B benefit on 4 bits remains limited: the C4-loss improvement over DQT is (0.003), for example reaching (62.60 vs 62.73). Also, DQT already gets rid of shadow weights, so the memory saving alone does not establish the incremental contribution of amplification. Still, while this revised evidence is better overall, it still does not convince me that the incremental contribution is enough for acceptance and I keep my original rating.

AuthorsOpenReview ↗

Follow up on Reviewer aa1C

Thank you for your follow-up, and we are glad that the FP32 controls and the full-model DQT comparisons resolved those concerns. Below we respond to the remaining points.

1. What if QAT also starts from GPTQ?

This is a fair question, so we ran it. We initialized QAT's FP32 shadow at the GPTQ endpoint and trained it under the same protocol and learning-rate grid, with three seeds. C4 loss / nine-task average accuracy:

Qwen-2.5-0.5BLlama-3.2-3BLlama-2-7BDeepSeek-MoE-16B
4-bit, QAT from FP3.350 / 47.232.693 / 61.362.143 / 62.352.425 / 62.87
4-bit, QAT from GPTQ3.291 / 47.802.676 / 62.082.140 / 62.872.425 / 63.20
8-bit, QAT from FP3.110 / 51.942.517 / 64.462.057 / 64.612.374 / 64.24
8-bit, QAT from GPTQ3.112 / 51.952.524 / 64.442.059 / 64.632.375 / 64.04

The picture is mixed. At 4 bits the GPTQ start does help QAT, lowering its C4 loss; at 8 bits it slightly hurts on all four models (by 0.001–0.007). When the model gets larger, the difference of the initial point becomes less. From the same GPTQ start, QAR-γ reaches a comparable loss without any shadow weight, with 27–29% less persistent memory.

2. Where we see the main contribution.

We may not have made this clear enough: amplification is the last step of the paper, not its center. The center is a mechanism of QAT itself, from which both DQT and QAR-γ follow.

  • QAT moves weights one cell at a time (Section 3). Looking at QAT through the residual inside each quantization cell, a step smaller than a cell moves a deployed weight by at most one cell, with a crossing probability proportional to the update. The law holds within every stratum of code and update direction at 4 and 8 bits (log-log slopes 0.98–1.11, Appendix B.5).
  • Descent on a finite grid works differently (Theorem 3). The cost of a grid jump is first order in the learning rate rather than second order. This explains why the deployed loss can fall while the shadow loss rises, and why a smaller learning rate alone does not guarantee descent. To our knowledge, neither result has been stated before.
  • The mechanism tells us how to design endpoint training.
    • One-cell transitions can be sampled on the codes directly, without a shadow weight. DQT is the uniform instance of this framework, so the theory also explains why shadow-free training descends at all.
    • The one-cell cap matters: rounding the full step can overshoot and cycle on a finite codebook (Proposition 11), and Appendix B.8 compares the two empirically.
    • Signal imbalance tilts uniform transitions toward weak-gradient endpoints (Section 4.1), and amplification is the correction this suggests.

So the memory saving belongs to the whole shadow-free family that this view describes, DQT included. What the paper adds is the account of why such training works, together with a better update rule within the family at the same memory.

Thank you again for engaging with the paper, we would greatly appreciate it if you could reconsider your rating.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive suggestions. The reviewers recognized two strengths of the paper:

  1. The endpoint view of QAT is informative.
    • A phenomenon that the shadow-weight view does not explain. During QAT, the deployed loss falls while the shadow loss rises (Reviewer aa1C); Reviewer fmsQ calls this "a nice general observation".
    • Why a smaller learning rate alone does not guarantee descent (Reviewer aa1C). In continuous optimization, the curvature cost of a step is O(η2)O(\eta^2), so a small enough step always descends. On a finite grid, every move is a full cell: a smaller η\eta makes a jump less likely but not shorter, so the curvature cost is also O(η)O(\eta). Theorem 3 shows that descent then depends on whether the signal outweighs the curvature cost of the grid, not on η\eta.
    • A formal account of QAT's weight dynamics, which Reviewer GddK finds informative for readers with theoretical interests.
  2. The memory saving is relatively substantial and useful.
    • Reviewer aa1C: "concrete and transparently measured", with a fair comparison.
    • Reviewer fmsQ: "a worthwhile result".
    • Reviewer GddK: on non-industry-level GPUs, where "memory is indeed a big issue", QAR "will be useful to enable some workflows".

Following the reviewers' suggestions, we have uploaded a revised PDF, with all changes marked in green. The main changes are:

  1. FP32-shadow QAT with a shared protocol (Reviewers aa1C and fmsQ). All trained methods share the AdamW settings, the schedule, and the learning-rate grid, and every trained row of Table 1 and Appendix B.12 uses three seeds. Appendix B.4 compares QAT with an FP32 and a BF16 shadow.
  2. DQT on every model (Reviewers aa1C, fmsQ, and GddK). Table 1 reports DQT for every model and bit width, and Appendix B.7 compares QAR-γ with DQT at a matched number of transitions on Qwen-2.5-0.5B.
  3. GPTQ as a baseline and as a warm start (Reviewer GddK). GPTQ is reported for every model at 4 and 8 bits, DQT and QAR-γ start from it, and Appendix B.9 compares an RTN and a GPTQ start.
  4. Time per training step (Reviewer GddK). Table 2 reports the time per step next to the persistent memory, with QAT given the same memory-saving implementation as QAR-γ (Appendix B.10).
  5. A mixture-of-experts model (Reviewer fmsQ). DeepSeek-MoE-16B, with 16.4B parameters and about 2.8B active per token, is added to Tables 1 and 2.
  6. New diagnostics of the crossing model and the amplifier (Reviewer fmsQ). Appendix B.5 checks the crossing model within strata of code, update direction, and update magnitude; Appendix B.7 amplifies one group of layers at a time; and Appendix B.8 compares the one-cell cap with multi-cell rounding.
  7. A main text that introduces the analysis through words (Reviewer GddK). The Introduction now ends with a contributions-and-roadmap paragraph that points to the section or theorem behind each contribution, Section 3 describes the residual phases in words with fewer symbols, the notation in Sections 2 and 4 is streamlined, and terms such as the signal of a weight are defined in plain language.

With these changes, the revision shows that:

  • QAR-γ makes full use of a PTQ start. Without a shadow weight, QAR-γ trains the quantized weights directly and can start from a GPTQ solution. From this start, it has the lowest C4 and WikiText-2 loss of all quantized models on all four LLMs at 4 bits, including the 16B mixture-of-experts model. At 8 bits, all trained methods match the full-precision model.
  • The gain over DQT comes from amplification. From the same start and at the same memory, QAR-γ has a lower C4 and WikiText-2 loss than DQT on every model at 4 bits, and on Qwen-2.5-0.5B the gain also holds at a matched number of transitions (Appendix B.7).
  • QAR-γ trains with much less memory at a comparable speed. At 4 bits it stores 27–29% less persistent memory than FP32-shadow QAT, trains Llama-2-7B on one GPU instead of two, and its step time ranges from 5.6% faster to 7.2% slower than QAT's.
  • Stratified checks support the crossing model. Within every group of endpoint code and update direction in Table B.7, the observed crossing frequency of QAT is proportional to the predicted probability at 4 and 8 bits (log-log slopes 0.98–1.11). The groups differ only by constant factors, which the distribution of the pretrained weights approximately predicts (Appendix B.5).

Details are in the individual responses, and we will continue to improve the paper based on the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.