- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes REAL-Q, a post-training quantization framework that combines GPTQ-style analytical compensation with dynamic gradient-based updates. The method introduces an Aggregated Fisher MSE objective to incorporate downstream sensitivity and cross-channel dependencies, applies Block-wise Gradient Descent (Block-GD) after each column block, and uses a loss sliding window to smooth the transition between consecutive Transformer blocks. Experiments across four LLMs and twelve quantization settings demonstrate improvements in KL divergence and downstream performance over several existing quantization methods.
Critical strengths
-
Strong empirical performance. REAL-Q achieves the lowest KL divergence across all twelve evaluated quantization settings.
-
Interesting gradient-based correction approach. The integration of gradient-based corrections into the GPTQ quantization pipeline is an interesting approach.
-
Broad and comprehensive experimental evaluation. The paper evaluates multiple LLM architectures, model sizes, and quantization settings, including both weight-only and weight–activation quantization.
Critical weaknesses
-
The paper does not isolate the contribution of the proposed full Aggregated Fisher MSE, (). It would be better for understanding the role of full Fisher if there is comparision with standard MSE () and diagonal Fisher () under the same Block-GD framework
-
REAL-Q combines GPTQ compensation with Block-GD, but does not evaluate whether gradient-based updates alone could achieve comparable performance (e.g., using Adam updates without analytical compensation).
-
I've found that KronQ [1] also uses the gradient-based Hessian for pre-processing. It can also be a comparison baseline on weight-only quantization configuration.
-
Despite reporting low quantization costs, the paper does not provide a breakdown of Fisher precomputation, analytical compensation, and Block-GD.
[1] Lee, Donghyun, et al. "KronQ: LLM Quantization via Kronecker-Factored Hessian." arXiv preprint arXiv:2607.07964 (2026).
Questions for authors
-
Figure 2 illustrates error accumulation in static solvers and its correction by REAL-Q, but this is only a schematic illustration. Can the authors provide an empirical plot of the reconstruction loss or KL divergence across column-wise quantization steps, comparing GPTQ and REAL-Q?
-
Can the authors evaluate whether the actual Adam update directions are aligned with the true end-to-end KL gradients, rather than only evaluating the surrogate gradients in Figure 3?
4 comments
Follow-up on our response to Reviewer FtDY
Dear Reviewer FtDY,
Thank you again for your review. We wanted to follow up on our response and would welcome any further feedback or questions.
Best regards,
The Authors
Thank you for the thorough responses and additional experiments, which address most of my concerns. I will raise my score accordingly. One remaining question: Fisher pre-computation alone takes 0.075 GPU-h, yet the Standard MSE run is only 0.037 GPU-h faster than Full Fisher. Could the authors explain this discrepancy?
Follow-up: Fisher pre-computation vs. the Full–MSE GPU-h gap
We thank the reviewer for the willingness to raise the score and for this careful follow-up question. The two numbers are in fact consistent, and the apparent discrepancy comes from what the Standard MSE arm actually does.
This is a controlled ablation, so all three arms share the Stage-0 Fisher pre-computation. The Full Fisher, Diagonal Fisher, and Standard MSE arms run the identical pipeline — the same calibration data, seeds, number of samples and steps, and the same Stage-0 Fisher pre-computation. The Standard MSE arm does not skip Stage-0; it still computes the Fisher and only replaces the weighting inside the Block-GD block-output loss with the identity (i.e., the pre-computed F is simply not applied in the loss).
With the cost broken out by stage, the two arms match exactly:
| Arm | Fisher pre-computation | Analytical compensation | Block-GD | Total |
|---|---|---|---|---|
| Full Fisher | 0.075 | 0.148 | 0.148 | 0.371 |
| Standard MSE | 0.075 | 0.148 | 0.111 | 0.334 |
Both arms pay the same 0.075 (Fisher pre-computation) and 0.148 (analytical compensation); they differ only in Block-GD (0.148 vs. 0.111). The 0.037 GPU-h gap is therefore the marginal cost of evaluating the full-matrix Fisher quadratic form at every Block-GD step, relative to the unweighted sum of squares — it is not the cost of the entire Fisher path. The Diagonal Fisher arm (0.337), sitting right next to Standard MSE (0.334), corroborates this: once F is cached, applying only its diagonal is almost free, and only the full off-diagonal matrix multiplication adds a measurable 0.037. We will add a one-line footnote to the cost ablation making this breakdown explicit.
We thank the reviewer for recognizing the results and the gradient-correction idea, and for the concrete suggestions.
W1: Full Fisher vs. standard MSE vs. diagonal Fisher under the same Block-GD framework. Fixed pipeline, calibration, seeds, and budget; only the loss changes, LR tuned separately. The GPU-h column confirms full Fisher costs only ~0.04 GPU-h more than standard MSE (Qwen3-4B):
Full Fisher wins on KL, QA, and reasoning mean, so cross-channel coupling matters. On W3A16 standard MSE even underperforms Block-GD-off (52.06 vs. 52.32 reasoning): per Section 5, a loss misaligned with the true KL need not decrease it.
W2: Can gradient updates alone (no analytical compensation) reach comparable performance? REAL-Q corrects GPTQ-style analytical compensation, not replaces it: the compensation absorbs the bulk of each column's error in closed form, and Block-GD handles what it cannot (static-Hessian mismatch, accumulated upstream error). The Main LR = 0 row (compensation only, Block-GD off) is one endpoint: reasoning mean drops 57.81→52.32 (W3A16), 51.98→37.08 (W4A4KV4). We measured the opposite endpoint too — a three-way ablation of plain RTN, RTN+Block-GD (compensation off, H−1=I, LR tuned per model), and full REAL-Q. WikiText-2 KL at W3A16 (×10−2):
Both contribute: Block-GD cuts plain-RTN KL 3.4–11.2× (564→50.2), the compensation a further 1.5–3.4× (50.2→14.98), combined 6–38×. So gradient descent alone recovers plain RTN substantially but, even per-model-tuned, stays well short of the full method; RTN+Block-GD's best LRs (5e-5–1e-4) are ~10× full REAL-Q's.
W3: KronQ (uniform bit-width, no mixed precision), under the main-table setup. W3A16:
W2A16:
W4: Breakdown of quantization cost. Qwen3-4B, single B200, all end-to-end quantization time:
Q1: The measured curve for Figure 2.
Done. Setup: Qwen3-4B W3A16, main-table config. After each column block we record the transformer block's Fisher-weighted block-output loss for REAL-Q vs. Block-GD-off (static compensation only, Main LR = 0), same mini-batch per step; the last block is excluded. Over 7035 steps across 35 blocks, REAL-Q is lower on 96.5% of steps and at 33/35 block-ends (median ratio 0.75), the largest gaps where the static loss rises (e.g. block 6: 1.79→4.52 vs. REAL-Q 0.73–0.75). The revision uses this measured figure.
Q2: Whether Adam's actual update direction aligns with the true KL gradient.
Figure 3 measures whether the surrogate gradient descends the true loss — Section 5's condition. Adam's update m^/(v^+ϵ) is not meant to align with the raw gradient (momentum smooths noise, the second moment preconditions), so even Adam on the true KL would not give cosine 1. On REAL-Q's real W3A16 trajectory we recompute g at each refresh step and compare with the applied update; since W←W−update, descent means ⟨update,g⟩>0. Positive-inner-product fraction (five models, the Figure 3 set):
So the update descends the true KL in 83–96% of steps.