acceptodds
Under review as a conference paper at ICLR 2027

REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent

Abstract

Post-training quantization (PTQ) is essential for deploying large language models (LLMs) under strict resource constraints. State-of-the-art PTQ methods quantize each layer with a single closed-form second-order solver: to remain analytically tractable, they heavily approximate the global loss (dropping cross-channel coupling, pooling output rows into groups), and they then freeze the resulting Hessian across the entire layer, with no way to refresh it as the loss landscape shifts column by column—a phenomenon we call information misalignment. We propose REAL-Q (Real-time E2E-loss Aligned LLM Quantization), a novel PTQ paradigm that breaks this compromise: instead of diluting the objective for the sake of analytic tractability, REAL-Q targets an end-to-end-aligned surrogate of the global loss and refines it via fine-grained, dynamic Block-wise Gradient Descent applied after every column block (128 columns). By coupling this fine-grained correction with a sliding window mechanism for smooth cross-layer transitions, REAL-Q effectively mitigates error propagation across the network. Experiments on LLaMA-3.1-8B-Instruct and Qwen3-4B/8B/32B cover twelve weight-only and weight–activation settings, with REAL-Q achieving the lowest KL loss and ranking first on downstream tasks in most settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Model Compression, Fisher Information, Gradient-Based Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FtDYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REAL-Q, a post-training quantization framework that combines GPTQ-style analytical compensation with dynamic gradient-based updates. The method introduces an Aggregated Fisher MSE objective to incorporate downstream sensitivity and cross-channel dependencies, applies Block-wise Gradient Descent (Block-GD) after each column block, and uses a loss sliding window to smooth the transition between consecutive Transformer blocks. Experiments across four LLMs and twelve quantization settings demonstrate improvements in KL divergence and downstream performance over several existing quantization methods.

Critical strengths
  1. Strong empirical performance. REAL-Q achieves the lowest KL divergence across all twelve evaluated quantization settings.

  2. Interesting gradient-based correction approach. The integration of gradient-based corrections into the GPTQ quantization pipeline is an interesting approach.

  3. Broad and comprehensive experimental evaluation. The paper evaluates multiple LLM architectures, model sizes, and quantization settings, including both weight-only and weight–activation quantization.

Critical weaknesses
  1. The paper does not isolate the contribution of the proposed full Aggregated Fisher MSE, (L=12Δy⊤FΔy\mathcal{L}=\frac{1}{2}\Delta y^\top F\Delta y). It would be better for understanding the role of full Fisher if there is comparision with standard MSE (F=IF=I) and diagonal Fisher (F=diag⁡(F)F=\operatorname{diag}(F)) under the same Block-GD framework

  2. REAL-Q combines GPTQ compensation with Block-GD, but does not evaluate whether gradient-based updates alone could achieve comparable performance (e.g., using Adam updates without analytical compensation).

  3. I've found that KronQ [1] also uses the gradient-based Hessian for pre-processing. It can also be a comparison baseline on weight-only quantization configuration.

  4. Despite reporting low quantization costs, the paper does not provide a breakdown of Fisher precomputation, analytical compensation, and Block-GD.

[1] Lee, Donghyun, et al. "KronQ: LLM Quantization via Kronecker-Factored Hessian." arXiv preprint arXiv:2607.07964 (2026).

Questions for authors
  1. Figure 2 illustrates error accumulation in static solvers and its correction by REAL-Q, but this is only a schematic illustration. Can the authors provide an empirical plot of the reconstruction loss or KL divergence across column-wise quantization steps, comparing GPTQ and REAL-Q?

  2. Can the authors evaluate whether the actual Adam update directions are aligned with the true end-to-end KL gradients, rather than only evaluating the surrogate gradients in Figure 3?

4 comments

We thank the reviewer for recognizing the results and the gradient-correction idea, and for the concrete suggestions.

W1: Full Fisher vs. standard MSE vs. diagonal Fisher under the same Block-GD framework. Fixed pipeline, calibration, seeds, and budget; only the loss changes, LR tuned separately. The GPU-h column confirms full Fisher costs only ~0.04 GPU-h more than standard MSE (Qwen3-4B):

SettingLossKL↓QA↑Reason. mean↑GPU-h
W3A16Full Fisher14.9860.8657.810.371
Diagonal Fisher15.2660.3153.680.337
Standard MSE15.1360.8252.060.334
W4A4KV4Full Fisher33.0157.7351.980.368
Diagonal Fisher33.4455.7647.44—
Standard MSE33.2556.5048.310.333

Full Fisher wins on KL, QA, and reasoning mean, so cross-channel coupling matters. On W3A16 standard MSE even underperforms Block-GD-off (52.06 vs. 52.32 reasoning): per Section 5, a loss misaligned with the true KL need not decrease it.

W2: Can gradient updates alone (no analytical compensation) reach comparable performance? REAL-Q corrects GPTQ-style analytical compensation, not replaces it: the compensation absorbs the bulk of each column's error in closed form, and Block-GD handles what it cannot (static-Hessian mismatch, accumulated upstream error). The Main LR = 0 row (compensation only, Block-GD off) is one endpoint: reasoning mean drops 57.81→52.32 (W3A16), 51.98→37.08 (W4A4KV4). We measured the opposite endpoint too — a three-way ablation of plain RTN, RTN+Block-GD (compensation off, H−1=IH^{-1}=I, LR tuned per model), and full REAL-Q. WikiText-2 KL at W3A16 (×10−2\times 10^{-2}):

ModelPlain RTNRTN+Block-GD (best LR)Full REAL-Q
Qwen3-4B56450.2 (1e-4)14.98
Qwen3-8B87.025.1 (5e-5)11.57
Qwen3-32B77.022.8 (5e-5)12.94
Llama-3.1-8B10714.7 (5e-5)9.71

Both contribute: Block-GD cuts plain-RTN KL 3.4–11.2× (564→50.2), the compensation a further 1.5–3.4× (50.2→14.98), combined 6–38×. So gradient descent alone recovers plain RTN substantially but, even per-model-tuned, stays well short of the full method; RTN+Block-GD's best LRs (5e-5–1e-4) are ~10× full REAL-Q's.

W3: KronQ (uniform bit-width, no mixed precision), under the main-table setup. W3A16:

ModelMethodKL↓Reason. mean↑
Qwen3-4BKronQ16.951.89
REAL-Q15.057.81
Qwen3-8BKronQ12.656.13
REAL-Q11.659.15
Qwen3-32BKronQ13.862.35
REAL-Q12.966.82
Llama-3.1-8BKronQ11.148.26
REAL-Q9.747.62

W2A16:

ModelMethodKL↓Reason. mean↑
Qwen3-4BKronQ140.80.57
REAL-Q66.01.15
Qwen3-8BKronQ111.70.39
REAL-Q56.52.46
Qwen3-32BKronQ64.86.70
REAL-Q48.215.42
Llama-3.1-8BKronQ69.61.40
REAL-Q54.52.16

W4: Breakdown of quantization cost. Qwen3-4B, single B200, all end-to-end quantization time:

StageW3A16 GPU-hW4A4KV4 GPU-h
Fisher pre-computation0.0750.075
Analytical compensation (incl. rotation, loading, fixed overhead)0.1480.156
Block-GD0.1480.137
Total0.3710.368

Q1: The measured curve for Figure 2.

Done. Setup: Qwen3-4B W3A16, main-table config. After each column block we record the transformer block's Fisher-weighted block-output loss for REAL-Q vs. Block-GD-off (static compensation only, Main LR = 0), same mini-batch per step; the last block is excluded. Over 7035 steps across 35 blocks, REAL-Q is lower on 96.5% of steps and at 33/35 block-ends (median ratio 0.75), the largest gaps where the static loss rises (e.g. block 6: 1.79→4.52 vs. REAL-Q 0.73–0.75). The revision uses this measured figure.

Q2: Whether Adam's actual update direction aligns with the true KL gradient.

Figure 3 measures whether the surrogate gradient descends the true loss — Section 5's condition. Adam's update m^/(v^+ϵ)\hat m/(\sqrt{\hat v}+\epsilon) is not meant to align with the raw gradient (momentum smooths noise, the second moment preconditions), so even Adam on the true KL would not give cosine 1. On REAL-Q's real W3A16 trajectory we recompute gg at each refresh step and compare with the applied update; since W←W−updateW \leftarrow W - \text{update}, descent means ⟨update,g⟩>0\langle \text{update}, g\rangle > 0. Positive-inner-product fraction (five models, the Figure 3 set):

ModelPositive fractionSteps
Qwen3-0.6B0.964140
Qwen3-1.7B0.841252
Qwen3-4B0.833468
Qwen3-8B0.880648
Llama-3.1-8B0.939608

So the update descends the true KL in 83–96% of steps.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer FtDY

Dear Reviewer FtDY,

Thank you again for your review. We wanted to follow up on our response and would welcome any further feedback or questions.

Best regards,
The Authors

Thank you for the thorough responses and additional experiments, which address most of my concerns. I will raise my score accordingly. One remaining question: Fisher pre-computation alone takes 0.075 GPU-h, yet the Standard MSE run is only 0.037 GPU-h faster than Full Fisher. Could the authors explain this discrepancy?

AuthorsOpenReview ↗

Follow-up: Fisher pre-computation vs. the Full–MSE GPU-h gap

We thank the reviewer for the willingness to raise the score and for this careful follow-up question. The two numbers are in fact consistent, and the apparent discrepancy comes from what the Standard MSE arm actually does.

This is a controlled ablation, so all three arms share the Stage-0 Fisher pre-computation. The Full Fisher, Diagonal Fisher, and Standard MSE arms run the identical pipeline — the same calibration data, seeds, number of samples and steps, and the same Stage-0 Fisher pre-computation. The Standard MSE arm does not skip Stage-0; it still computes the Fisher and only replaces the weighting inside the Block-GD block-output loss with the identity (i.e., the pre-computed F is simply not applied in the loss).

With the cost broken out by stage, the two arms match exactly:

ArmFisher pre-computationAnalytical compensationBlock-GDTotal
Full Fisher0.0750.1480.1480.371
Standard MSE0.0750.1480.1110.334

Both arms pay the same 0.075 (Fisher pre-computation) and 0.148 (analytical compensation); they differ only in Block-GD (0.148 vs. 0.111). The 0.037 GPU-h gap is therefore the marginal cost of evaluating the full-matrix Fisher quadratic form Δy⊤FΔy\Delta y^\top F \Delta y at every Block-GD step, relative to the unweighted sum of squares — it is not the cost of the entire Fisher path. The Diagonal Fisher arm (0.337), sitting right next to Standard MSE (0.334), corroborates this: once F is cached, applying only its diagonal is almost free, and only the full off-diagonal matrix multiplication adds a measurable 0.037. We will add a one-line footnote to the cost ablation making this breakdown explicit.

Reviewer_BzmPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes REAL-Q, a post-training quantization (PTQ) framework that combines GPTQ-style error compensation with Dynamic Block-wise Gradient Descent. Instead of relying only on a fixed second-order approximation, REAL-Q introduces gradient-based corrections during sequential quantization to reduce accumulated errors.

The method also uses an aggregated Fisher-based reconstruction loss to better reflect the end-to-end behavior of the model. A loss sliding window is introduced to make optimization transitions between Transformer blocks smoother.

The paper provides a theoretical analysis of error accumulation and gradient alignment. Experiments across LLaMA and Qwen models demonstrate improvements in KL divergence and competitive downstream performance under several weight-only and weight-activation quantization settings.

Critical strengths

1. Interesting approach to dynamic quantization error correction.

Integrating Dynamic Block-wise Gradient Descent into the sequential GPTQ pipeline is a promising idea. Unlike methods that rely on a fixed second-order approximation, REAL-Q introduces gradient-based corrections during quantization, which may help reduce accumulated errors.

The theoretical analysis of error accumulation under static approximations and the conditions for gradient alignment provides useful motivation for this approach.

2. Promising experimental results.

REAL-Q achieves strong results across different models and quantization settings. In particular, the reported improvements in KL divergence suggest that the method can better preserve the output distributions of full-precision models.

The ablation studies also provide evidence that the update frequency and other design choices affect quantization performance. These findings support the potential effectiveness of dynamic gradient-based correction.

Critical weaknesses

1. Incomplete related work and missing comparisons with closely related methods.

My main concern is that several highly relevant studies are not adequately discussed or compared, including QEP (NeurIPS 2025), LPCD, LoaQ, Qronos (ICLR 2026), FOEM (AAAI 2026), and OAC (AAAI 2025).

These studies address closely related problems, such as quantization error propagation, optimization across layers or submodules, output-aware reconstruction, and first-order error compensation.

Although their approaches differ from REAL-Q, the paper should explain more clearly how its technical contributions differ from these methods. In particular, comparisons with the most relevant baselines under matched quantization settings would help establish whether REAL-Q provides a meaningful improvement.

Without a more complete discussion and evaluation of this related work, it is difficult to assess the novelty and empirical advantages of the proposed framework.

2. Insufficient isolation and validation of the main contributions.

The paper introduces several components, but the experiments do not fully isolate their individual contributions.

In particular, the ablation study does not directly compare full Fisher and diagonal Fisher objectives under the same Dynamic Block-wise Gradient Descent procedure. Such a comparison would help determine whether preserving cross-channel information provides a substantial benefit beyond the gradient-based corrections themselves.

Furthermore, the gradient-alignment analysis is limited to Qwen3-0.6B under W4A16 quantization. Since the quality of the Fisher-based objective is important to the paper's motivation, evaluating its alignment with the true end-to-end gradient on larger models and more challenging quantization settings would strengthen the evidence.

These limitations make it difficult to determine how much of the improvement comes from the proposed Fisher objective and how reliably the findings generalize.

Questions for authors

1. Relationship to closely related methods.

Could the authors clarify the main technical differences between REAL-Q and related methods such as QEP, LPCD, LoaQ, Qronos, FOEM, and OAC?

In particular, could the authors provide comparisons with the most relevant methods under similar model architectures, quantization settings, and computational budgets? Such evidence would help me better assess the novelty and practical advantages of REAL-Q.

2. Contribution of the full Fisher objective.

Could the authors compare full Fisher and diagonal Fisher objectives while keeping the Dynamic Block-wise Gradient Descent procedure and all other settings unchanged?

This would help establish whether the performance improvements come mainly from the richer Fisher-based objective or from the dynamic gradient corrections.

3. Generality of gradient alignment.

Could the authors extend the gradient-alignment analysis beyond Qwen3-0.6B under W4A16?

In particular, results on larger models and lower-bit settings would help determine whether the proposed surrogate objective remains well aligned with the true end-to-end loss in more challenging scenarios.

2 comments

We thank the reviewer for recognizing the dynamic error-correction idea and the results.

W1 / Q1: Relationship to QEP, LPCD, LoaQ, Qronos, FOEM, and OAC. The GPTQ-family layer-wise objective has two mismatches: upstream, it assumes full-precision inputs and ignores error from earlier quantized layers; downstream, it weights all output channels equally, whereas the true KL couples them nonlinearly through later layers and the LM head. It also freezes the layer Hessian, so error accumulates during quantization. REAL-Q fixes both: (1) the full block-output Fisher (from the second-order expansion of the end-to-end KL) as objective, capturing downstream sensitivity and cross-channel coupling; (2) after each 128-column block, one Adam step on a gradient recomputed on the current partially quantized model, whose input (the quantized upstream forward pass) carries accumulated upstream error, raising the correction from per-layer to per-block.

All six improve the local objective or the solver, but none has both "end-to-end gradient" and "per-column-block online correction":

  • QEP (NeurIPS 2025): closed-form fix of upstream drift once before quantization, then a standard layer-wise quantizer — upstream only, no downstream sensitivity, no update during quantization.
  • LPCD (arXiv 2025): cyclic block coordinate descent over weight/activation/KV submodules, each projected back to the layer-wise quantizer — still local reconstruction; QEP is its one-step case.
  • LoaQ (arXiv 2025): output alignment extended to a sub-block (residual + RMSNorm), solved with LDLQ (= GPTQ) — larger scope, still static local reconstruction.
  • Qronos (ICLR 2026): least-squares under mismatched input, alternating column correction and error diffusion — column-granular, but each step is a layer-wise closed-form update.
  • FOEM (AAAI 2026): keeps GPTQ's first-order term, latent-weight gradient approximated as β(W−W)H\beta(W-\mathbb{W})H — analytical, from the layer Hessian, not end-to-end.
  • OAC (AAAI 2025): output-adaptive Hessian from output-CE gradient outer products — adds downstream sensitivity, but pre-computed once, layer-local, static.

Empirical comparison. QEP has official code and a quantizer-agnostic correction, so we ported it to the main-table setup on 4 models (W3A16/W2A16):

W3A16:

ModelMethodKL↓Reason. mean↑
Qwen3-4BQEP19.549.31
REAL-Q15.057.81
Qwen3-8BQEP13.651.47
REAL-Q11.659.15
Qwen3-32BQEP14.565.70
REAL-Q12.966.82
Llama-3.1-8BQEP11.142.62
REAL-Q9.747.62

W2A16:

ModelMethodKL↓Reason. mean↑
Qwen3-4BQEP110.20.47
REAL-Q66.01.15
Qwen3-8BQEP85.01.90
REAL-Q56.52.46
Qwen3-32BQEP61.914.31
REAL-Q48.215.42
Llama-3.1-8BQEP71.40.89
REAL-Q54.52.16

W2 / Q2: Full Fisher vs. diagonal Fisher under the same Block-GD pipeline. Fixed pipeline/calibration/seeds/budget; only the block-output loss changes (full Fisher, diagonal Fisher, plain MSE), LR tuned separately (Qwen3-4B):

SettingLossKL↓QA↑Reason. mean↑
W3A16Full Fisher14.9860.8657.81
Diagonal Fisher15.2660.3153.68
Plain MSE15.1360.8252.06
W4A4KV4Full Fisher33.0157.7351.98
Diagonal Fisher33.4455.7647.44
Plain MSE33.2556.5048.31

Full Fisher wins on KL, QA, and reasoning mean, so cross-channel coupling matters. On W3A16 plain MSE even underperforms Block-GD-off (52.06 vs. 52.32 reasoning): per Section 5, a loss misaligned with the true KL need not decrease it.

Q3: Extending the gradient-alignment analysis to more models. We extend Figure 3 beyond Qwen3-0.6B to the larger Qwen3-8B and, from a different family, Llama-3.1-8B. Mean cosine between each surrogate's gradient and the true end-to-end KL gradient (attention / MLP):

ModelREAL-Q (Fisher)GuidedQuant (saliency MSE)GPTQ/GPTAQ (module MSE)
Qwen3-0.6B0.518 / 0.3440.046 / 0.0350.047 / 0.033
Qwen3-8B0.270 / 0.1340.049 / 0.0230.049 / 0.022
Llama-3.1-8B0.284 / 0.1960.034 / 0.0140.032 / 0.013

Across scales and families, REAL-Q's alignment is 5.5–11× layer-wise MSE's on attention and 6–15× on MLP. Layer-wise MSE and saliency-weighted MSE are both near 0 and nearly identical, so saliency re-weighting does not fix the layer-wise objective's directional deviation from the true KL gradient. We take the cosine from the post-RTN error, not any specific algorithm's, for fairness — on REAL-Q's own trajectory the residual would be inflated, since it already compensates along its update direction. As RTN collapses below 4 bits, this is fixed at W4A16.

Thank you for your clear explanation and additional experiments. These have alleviated some of my concerns, so I plan to increase my score.

Reviewer_ZjB7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Many post-training quantization methods use a fixed approximation of the loss while processing a layer. As more weights are quantized, this approximation can become less accurate and leave errors that affect later layers. REAL-Q addresses this by adding a gradient update after each small block of columns, using a Fisher-weighted block-output loss and a sliding window over neighboring transformer blocks. The authors evaluate Llama-3.1-8B-Instruct and Qwen3-4B/8B/32B across different weight-only and weight–activation configurations. They measure output KL divergence, perplexity, general task accuracy, and reasoning performance. REAL-Q achieves the lowest KL divergence in all settings and the highest reasoning average in most settings, with some large gains when activations are also quantized.

Critical strengths
  1. The paper addresses a reasonable limitation of sequential quantization. Updating the remaining weights during quantization allows the method to respond to errors in the current model. The ablations show that the gradient updates and sliding window help in the tested settings.
  2. The evaluation includes several model sizes and measures both output-distribution error and downstream performance. Some gains are substantial.
Critical weaknesses
  1. The unusually poor baseline results need more careful validation. In Table 2, YAQA has a perplexity of 50.43 on Llama-3.1-8B-Instruct at W3A16, compared with 7.21 for the original model and roughly 8 for several other methods. TurboBoA also has a perplexity of 385.14 on Qwen3-32B at W3A16, compared with 7.61 for the original model. These are large failures, rather than ordinary performance gaps, which need further clarification.
  2. The baseline configurations do not all provide equally strong comparisons. EfficientQAT is evaluated without QuaRot, while REAL-Q and the other PTQ methods use it. The paper also states that its calibration budget is much smaller than YAQA’s official setup and therefore uses the Sketch-A estimator. A shared data budget is a reasonable comparison, but it does not establish that REAL-Q outperforms these methods under their recommended settings.
  3. The ablations do not directly establish the value of the Fisher loss. They change the column-block size, backward sample count, sliding window, and learning rate. However, the paper should also keep the update procedure and compute budget fixed while replacing the Fisher loss with ordinary block MSE or a diagonal Fisher loss.
Questions for authors

See weakness above.

4 comments

We thank the reviewer for the careful reading and for recognizing the dynamic-correction idea and the breadth of our evaluation. We respond point by point below.

W1: YAQA and TurboBoA are abnormally weak in a few settings.

Both results come from each method's official default hyperparameters. To check whether they reflect genuine limitations of the methods, we tuned only the officially provided hyperparameters, selected by validation KL, and then re-evaluated under the same test protocol as the main table. For TurboBoA we swept α∈{0, 0.1, 0.25, 0.5} × percdamp∈{0.01, 0.1}; for YAQA we swept sigma_reg∈{0.001, 0.01, 0.1, 1, 3}.

TurboBoA on Qwen3-32B W3A16:

MethodKL↓QA↑Reason. mean↑
TurboBoA (default α=0.25, d=0.01)428.8734.361.58
TurboBoA (retuned α=0.1, d=0.1)14.4971.0966.96
REAL-Q12.9471.4166.82

YAQA on Llama-3.1-8B-Instruct W3A16:

MethodKL↓QA↑Reason. mean↑
YAQA (default σ=0.01)193.5450.672.29
YAQA (σ=1)70.4160.1015.70
REAL-Q9.7164.7847.62

Retuned TurboBoA is a strong competitor: REAL-Q is better on KL and QA, while TurboBoA is slightly higher on the reasoning mean (+0.14). We further applied this retuning procedure to all 12 TurboBoA settings. Compared with retuned TurboBoA, REAL-Q is better on KL in 12/12 and on the reasoning mean in 11/12 (the only exception being Qwen3-32B W3A16 in the table above).

In the revised main table, we will annotate the originally collapsing cells with the retuned numbers and their hyperparameters, and place the full comparison for the remaining settings in the appendix.

W2: EfficientQAT does not use QuaRot; the shared data budget differs from YAQA's recommended setting.

EfficientQAT + QuaRot. We added the main-table QuaRot rotation and training-time A/K/V awareness to EfficientQAT and re-searched the LR over all 12 settings. Under W4A4KV4, QuaRot brings it back from collapse (PPL 29–29100 → 7.85–14.81); under weight-only quantization it helps nothing (all four W3A16 models worse than unrotated). The revised main table takes the better of rotated/unrotated per setting. REAL-Q still leads on KL, QA, and reasoning mean across all four W4A4KV4 models:

Model (W4A4KV4)MethodKL↓QA↑Reason. mean↑
Qwen3-4BEfficientQAT + QuaRot38.155.2622.27
REAL-Q33.057.7351.98
Qwen3-8BEfficientQAT + QuaRot32.559.0610.57
REAL-Q25.463.2959.25
Qwen3-32BEfficientQAT + QuaRot28.668.4159.07
REAL-Q25.668.7064.80
Llama-3.1-8BEfficientQAT + QuaRot27.357.5720.75
REAL-Q19.862.2243.20

YAQA's large-budget setting. We adopt the smallest large-data configuration from the YAQA paper: Sketch-A, 6 power iterations, 2440×8192 RedPajama sequences, about 20M tokens. REAL-Q uses exactly the same tokens, compared on Qwen3-4B:

SettingMethodKL↓QA↑Reason. mean↑
W3A16YAQA13.0261.2852.83
REAL-Q12.4461.6154.37
W2A16YAQA124.1733.311.27
REAL-Q60.8246.931.90

KL is on a held-out RedPajama split; QA/reasoning use the main-table benchmarks. At YAQA's recommended scale REAL-Q still leads on most W3A16 metrics and at W2A16 halves KL with +13.6 QA. Cost: YAQA needs 295 GPU-h for Hessian estimation alone (300 GPU-h for both settings); REAL-Q needs none, 24 GPU-h total.

W3: The ablation does not directly show the role of the Fisher loss.

Keeping the Block-GD pipeline, calibration, seeds, and sample/step budget fixed, we replace only the block-output loss — full Fisher, diagonal-only Fisher, and plain block MSE (Fisher = identity) — tuning the LR separately for each. Full Fisher costs only ~0.04 GPU-h more than plain MSE (Qwen3-4B):

SettingLossKL↓QA↑Reason. mean↑
W3A16Full Fisher14.9860.8657.81
Diagonal Fisher15.2660.3153.68
Plain MSE15.1360.8252.06
W4A4KV4Full Fisher33.0157.7351.98
Diagonal Fisher33.4455.7647.44
Plain MSE33.2556.5048.31

Full Fisher wins on KL, QA, and reasoning mean, so cross-channel coupling cannot be ignored. On W3A16, plain MSE even underperforms Block-GD off (52.06 vs. 52.32, the Main LR = 0 row): a loss poorly aligned with the true KL does not guarantee the true loss decreases, matching the descent-condition analysis in Section 5.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer ZjB7

Dear Reviewer ZjB7,

Thank you again for your review. We wanted to follow up on our response and would welcome any further feedback or questions.

Best regards,
The Authors

Reviewer_ZjB7OpenReview ↗

response

Thanks for the reply. My original concern specifically referred to the unusually high PPL values. However, the response reports KL and downstream accuracy without providing the corresponding re-tuned PPL results. This means it is still unclear whether the baseline setup is reasonable.

On the other hand, the large recovery suggests that the original comparison used failed baseline runs, and these checks should have been performed before submission, instead of in the rebuttal. Updating the results of all important baselines is a major revision to the paper, and the corrected results substantially change the reported performance gaps. Therefore, I will maintain my rating.

AuthorsOpenReview ↗

Follow-up: Re-tuned PPL for the baselines

We thank the reviewer for the continued engagement, and we apologize for the omission that caused this: the re-tuned PPL values were in fact part of our original response, but when we compressed the rebuttal tables to fit the length limit we dropped the PPL column and kept only KL / QA / reasoning. That was a presentation mistake on our side, not missing data. We give the complete re-tuned PPL below, which directly addresses whether the baseline setup is reasonable.

1. Re-tuned PPL (WikiText-2, lower is better)

TurboBoA, all 12 settings (selected by argmin validation KL over α∈{0,0.1,0.25,0.5} × percdamp∈{0.01,0.1}, then evaluated under the identical test protocol):

ModelSettingTurboBoA default PPLTurboBoA re-tuned PPLREAL-Q PPLbf16
LLaMA-3.1-8BW3A168.058.027.947.21
W2A1618.1517.6612.327.21
W4A4KV48.879.248.727.21
Qwen3-4BW3A1616.5615.1914.0813.66
W2A1665.9729.1219.1013.66
W4A4KV420.9617.4615.1613.66
Qwen3-8BW3A1610.9110.3410.359.72
W2A161013.2123.8014.999.72
W4A4KV412.6311.5511.359.72
Qwen3-32BW3A16385.147.987.767.61
W2A1615.8310.999.837.61
W4A4KV49.359.758.287.61

YAQA, the headline collapsed cell (selected over sigma_reg∈{0.001,0.01,0.1,1,3}):

ModelSettingYAQA default (σ=0.01) PPLYAQA re-tuned (σ=1) PPLREAL-Q PPLbf16
LLaMA-3.1-8BW3A1650.4314.657.947.21

(EfficientQAT, for completeness: adding QuaRot brings its W4A4KV4 PPL back from the collapsed 29–29100 range to 7.85–14.81, i.e. the normal range.)

2. What the re-tuned PPL shows

The baseline setup is reasonable. After re-tuning within each method's own official hyperparameter grid, every PPL returns to the normal range — the collapsed cells (TurboBoA Qwen3-32B W3A16 385.14 → 7.98; TurboBoA Qwen3-8B W2A16 1013.21 → 23.80; YAQA Llama W3A16 50.43 → 14.65) land right next to bf16 and the non-collapsing baselines. A genuinely broken harness would not recover to competitive PPL simply by changing percdamp or sigma_reg; the fact that it does is itself evidence that the pipeline, calibration, rotation, and evaluation are correct and that the collapse was hyperparameter sensitivity, not a setup error. Concretely, the same α=0.25 that collapses with percdamp=0.01 on Qwen3-32B (PPL 385) is healthy with percdamp=0.1 (PPL 7.98) — the collapse is governed by the (α, percdamp) combination.

The re-tuned PPL does not change the paper's conclusions. On PPL, REAL-Q is still lower than re-tuned TurboBoA in 11/12 settings; the single exception is Qwen3-8B W3A16, where the two are effectively tied (10.35 vs 10.34, a 0.01 difference). Re-tuned YAQA recovers its PPL but remains well above REAL-Q (14.65 vs 7.94) and still trails sharply on reasoning. So the corrected PPL narrows the gaps only on the specific cells that had collapsed, while REAL-Q's ranking is unchanged.

3. On "failed runs" and the scope of the revision

The original runs were not failed runs. Each baseline was reported at its method's official default hyperparameters — a standard, reproducible protocol for PTQ comparisons, where re-tuning every baseline per model/bit-width is not the usual expectation. That said, we accept the reviewer's point that including the hyperparameter sweep strengthens the comparison, and we have completed it for all 12 TurboBoA settings and the YAQA grid.

The revision is bounded and transparent, not a change to the findings. Only the originally-collapsing cells are annotated with their re-tuned values and the hyperparameters used; the full per-setting comparison (now including PPL) goes in the appendix. The table structure, the shared evaluation protocol, and every headline claim are unchanged — REAL-Q leads on KL in 12/12, on PPL in 11/12, and on reasoning in 11/12. We view this as an increase in rigor rather than a revision that alters the paper's conclusions, and we would be grateful if the reviewer would reconsider in light of the complete PPL results above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.