- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
HeRo-Q is a post-training quantization framework that combines Hessian-proxy-based diagonal conditioning with orthogonal rotation optimization. The conditioning targets effective curvature, while rotation is updated by minimizing output reconstruction error induced by weight quantization. The paper provides a local second-order analysis and evaluates weight-only and weight-activation quantization across dense LLMs, MoE models, ViTs, and a VLM. Compatibility with existing quantization algorithms (GPTQ, AWQ, and RTN) is evaluated, along with composition with QuIP. The experiments broadly cover PPL, downstream accuracy, calibration cost, and decoding throughput.
Critical strengths
-
Evaluation broadly covers dense LLMs, MoE models, ViTs, and a VLM under multiple quantization settings. The results generally show improvements over the reported baselines across these model families (Tables 1–3 and additional experiments in Appendix B).
-
The framework is compatible with different base quantizers, including GPTQ, AWQ, and RTN (Table 7). Combining it with QuIP also improves over both standalone methods in the reported settings (Table 21).
-
The paper includes component ablations and controlled comparisons rather than relying only on comparisons between complete methods. Tables 5 and 18 provide evidence for the conditioning’s contribution within the evaluated settings, although their scope remains limited.
-
Calibration time, peak calibration memory, and measured decoding throughput are reported (Tables 4 and 16). The deployment description also distinguishes transformations absorbed offline from those remaining online (Table 15), making the implementation assumptions more explicit.
-
The Hessian-based analysis provides a useful distinction between diagonal conditioning and orthogonal rotation. This offers a principled motivation for combining the two operations, with a spectral-compression result under the stated diagonal-proxy assumptions (Theorem 2).
Critical weaknesses
-
Though the Hessian-based analysis and conditioning method are valuable, HeRo-Q fundamentally combines channel-wise scaling with learned rotations, leaving its methodological novelty limited. Furthermore, activation quantization is excluded during optimization (Appendix C.4), while the component ablation uses W4A16 (Table 5), so the conditioning's additional contribution under W4A4/W4A8 remains unclear.
-
The comparisons in Table 1 do not cover recent transformation-based methods, such as OSTQuant [1], DartQuant [2], and OptRot [3], leaving competitiveness against these methods unclear.
-
In Table 16(b), HeRo-Q's throughput is close to the transformation-free baseline, despite retaining online R4 at the SwiGLU–down interface (Table 15). Further clarification of the exact inference setup, including the W4A16 kernel used for each method and a timing ablation isolating the online transform, is needed to establish its overhead and the competitiveness of the inference baseline.
-
Gains over QTIP in Table 20 are small (0.48 pp on MMLU and 0.31 pp on GSM8K). Without uncertainty estimates or repeated calibration and optimization runs, it is unclear whether these gains are reliable and consistent.
-
Although GSM8K (Table 1), HumanEval, and MathQA (Table 9) results are reported, HeRo-Q's robustness on recent reasoning models and challenging long-generation tasks is not evaluated. This leaves its applicability to these increasingly important workloads unclear.
Questions for authors
I have a few questions for the authors, roughly aligned to the weaknesses:
-
What does the proposed Hessian-based conditioning contribute beyond existing scaling–rotation approaches? Please extend the component ablation in Table 5 to W4A4/W4A8, isolating conditioning-only, rotation-only, and combined effects, and broaden the conditioning-rule comparison in Table 18 beyond Llama-3.1-8B W3A16.
-
Can the authors compare against recent transformation-based methods, such as OSTQuant [1], DartQuant [2], and OptRot [3], using matched checkpoints, quantization settings, calibration budgets, and evaluation protocols?
-
Which W4A16 kernels and library versions are used for each method in Table 16(b)? Please provide a matched comparison against an optimized transformation-free baseline and a timing ablation isolating the online R4 transform listed in Table 15.
-
Are the small gains over QTIP in Table 20 consistent across calibration samples and optimization seeds? Please provide repeated-run results and appropriate uncertainty estimates for the downstream accuracy differences.
-
Do the gains reported in Tables 1 and 9 hold on recent reasoning models and challenging long-generation tasks, such as harder math (AIME 2025), science (GPQA Diamond), and coding benchmarks (LiveCodeBench v5)?
-
The code URL provided in Appendix C.1 is expired. Can the authors provide an accessible anonymized repository with the implementation and scripts needed to reproduce the main results?
7 comments
Rebuttal by Authors
Q3
All W4A8 experiments use the Marlin kernel in vLLM v0.17.1, with symmetric INT4 weight quantization at group size 128 and dynamic per-token symmetric INT8 activation quantization. All methods share the same GEMM kernel, and end-to-end latency includes activation quantization and the required online transformations. Timing excludes prefill and includes the remaining online transformations. After three warm-up runs, we synchronize the GPU before and after the timed region.
To isolate the cost of online rotation, we add a timing ablation at the SwiGLU–down interface, comparing “pre-rotated input + down GEMM” with “online R4 + down GEMM”:
| Model | Pre-rotated input + down GEMM (ms/step) | Online R4 + down GEMM (ms/step) | Increment (ms/step) | Fraction of total decoding time |
|---|---|---|---|---|
| Llama-3.2-1B | 1.160±0.027 | 1.420±0.029 | 0.260±0.007 | 1.97% |
| Llama-3.2-3B | 2.080±0.046 | 2.520±0.048 | 0.440±0.016 | 2.31% |
| Llama-3.1-8B | 4.320±0.094 | 4.890±0.124 | 0.570±0.032 | 1.61% |
The pre-rotated-input path is used to isolate the timing cost of the online transformation; it does not imply that R4 can simply be removed from the full deployment. This rotation is still executed in actual deployment.
Under the tested configurations, the online R4 path adds 0.260–0.570 ms/step, accounting for 1.61%–2.31% of total decoding time. This is consistent with the small overall throughput differences reported in the paper. Our efficiency conclusion is that diagonal conditioning at compatible interfaces can be absorbed offline, avoiding a separate online scaling operator; the remaining online rotations still incur costs, which are included in the decoding measurements.
Q4
The small accuracy differences in the original Table 20 require support from repeated experiments. Under the Llama-3.2-1B W3A16 setting, the results of 15 paired evaluation runs are as follows:
| Task | QTIP | HeRo-Q | Paired difference (mean±SD, percentage points) | Runs where HeRo-Q leads |
|---|---|---|---|---|
| MMLU | 42.02±0.15 | 42.50±0.16 | +0.48±0.13 | 15/15 |
| GSM8K | 28.79±0.14 | 29.10±0.23 | +0.31±0.13 | 15/15 |
HeRo-Q achieves higher accuracy in all 15 paired evaluation runs on both tasks. Its average gains on MMLU and GSM8K are 0.48 and 0.31 percentage points, respectively.
Q5
We extend the evaluation to Qwen3-8B and Qwen3-14B and add AIME, GPQA Diamond, and LiveCodeBench v5. The compared methods include FP16, GPTQ, AWQ, FlatQuant, and HeRo-Q.
Each cell below reports scores on AIME / GPQA Diamond / LiveCodeBench v5, in that order.
| Method | Qwen3-8B | Qwen3-14B |
|---|---|---|
| FP16 | 67.3 / 62.0 / 57.5 | 70.4 / 64.0 / 63.5 |
| GPTQ | 64.5 / 59.9 / 53.5 | 69.1 / 63.5 / 61.2 |
| AWQ | 63.1 / 59.0 / 55.0 | 68.1 / 62.1 / 62.1 |
| FlatQuant | 65.5 / 60.7 / 54.4 | 69.5 / 63.1 / 61.7 |
| HeRo-Q | 66.4 / 61.7 / 55.6 | 70.3 / 63.9 / 62.8 |
HeRo-Q achieves the highest scores among the listed quantization methods on all three tasks for both models. Compared with the strongest quantization baseline on each task, the gains are 0.9, 1.0, and 0.6 percentage points on Qwen3-8B and 0.8, 0.4, and 0.7 percentage points on Qwen3-14B, respectively.
Q6
We have updated Appendix C.1.
Follow up on rebuttal
Dear Reviewer KN6t,
We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up to ensure that our previous response and revisions have fully addressed your concerns.
We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We are joyous to engage in further discussion if needed.
Best regards, The Authors
Rebuttal by Authors
Q1
Regarding methodological novelty, please refer to our responses to Reviewer CoaR's Weaknesses 2 and Question 1. Here, we focus on the requested component ablations and expanded conditioning comparisons.
We extend the component ablation in the original Table 5 to W4A4 and W4A8 settings across three models. The following table reports WikiText-2 perplexity.
Under W4A4, HeRo-Q reduces perplexity by 24.8%–29.8% compared with rotation only and by 12.1%–17.5% compared with SmoothQuant+SpinQuant.
We also extend the conditioning-rule comparison in the original Table 18 to additional models and precisions. Each cell below reports WikiText-2 PPL↓ / C4 PPL↓ / MMLU↑ / GSM8K↑, in that order.
Q2
We have added comparisons with OSTQuant, DartQuant, and OptRot. Each cell below reports WikiText-2 PPL↓ / C4 PPL↓ / MMLU↑ / GSM8K↑, in that order.
The W4A16 results are as follows:
The W3A16 results are as follows:
HeRo-Q achieves the best mean performance on all four metrics in the comparisons above. Under W3A16, compared with the best result among the listed baselines for each metric, HeRo-Q reduces WikiText-2 perplexity by 7.7% and 12.8% and improves GSM8K accuracy by 1.47 and 4.13 percentage points on Llama-3.1-8B and Llama-3.2-1B, respectively.