- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CAPQuant studies channel ordering for dynamic asymmetric group-wise INT4 activation quantization. It derives a within-group dispersion surrogate from a range-based error model and shows that, for equally sized groups, the sum of individual-channel variance terms is independent of the partition; mean differences and cross-channel covariances determine the ordering-dependent part. The method combines balanced clustering with improving channel swaps and folds the inverse permutation into adjacent weights while fusing the forward ordering into existing activation kernels. On frozen OSTQuant, QuaRot, and SpinQuant checkpoints, it improves perplexity; over contiguous grouping, ordering reduces perplexity in 15 of 16 reported comparisons. The reference prefill setup shows a 3.6–6.9% added latency for the permutation.
Critical strengths
- The paper introduces a covariance-aware channel reordering objective, which provides a principled explanation of why ordering based only on marginal channel statistics, such as range or mean, can be insufficient.
- The method is integrated through offline weight-column permutation and kernel fusion, avoiding additional matrix multiplications or separate activation passes. This is a technically sound and practically appealing design.
- The experiments cover multiple models, W4A4KV16 and W4A4KV4 settings, and multiple base quantizers, with contiguous, random, and mean-sorted ordering baselines.
Critical weaknesses
- The method optimizes a dispersion surrogate for the dynamic group-wise quantization objective rather than the actual quantization error itself. Although the paper provides theoretical bounds, the empirical evidence is still insufficient to establish that this approximation is consistently reliable across layers, models, and quantization configurations.
- While the experiments include Qwen3 and Llama models, they are all dense decoder-only Transformers. In addition, the QuaRot plug-in evaluation is reported only on Llama-3-8B. The generality across model architectures, calibration samples, and random seeds remains insufficiently validated.
- Runtime is measured with simulated activation quantization and BF16 matrix multiplication. Therefore, the practical overhead under native INT4 inference kernels, especially during decoding, remains unclear.
Questions for authors
- Could the authors report, across more settings, the correlation between the surrogate objective and the actual group range, activation reconstruction MSE, and downstream perplexity? How consistently do the actual range and MSE improve after optimization?
- Can the method be evaluated on additional architectures, such as Mistral, or mixture-of-experts models? Are the ordering gains and accuracy improvements stable across different calibration samples and random seeds?
- Could the authors report prefill and decoding latency, throughput, and memory overhead using native INT4 kernels, in order to better characterize the practical deployment trade-offs?
- My main question is: if it works, why not incorporate permutation-related learning during training (permutation itself is a type of orthogonal transformation)?
4 comments
Thank you again for your constructive feedback!
- We have expanded the surrogate analysis across 14 model/KV settings, finding a 0.91–0.99 correlation with actual reconstruction error. Across 7,440 site/seed measurements, CAPQuant never increases reconstruction error over contiguous grouping. We also report strong correlations with squared group range and downstream PPL, directly strengthening the empirical support for our objective.
- We completed Mistral-7B evaluations, additional calibration seeds, C4 calibration, and 2/8/64-window experiments. The gains persist across calibration samples, with the five-seed mean PPL improving over contiguous grouping in all comparisons.
- We also added native NVFP4 runtime experiments on B300, covering prefill, decode, throughput, and memory under the same settings as our initial evalutions.
- Regarding learning permutations during training: we agree this is an interesting direction! CAPQuant demonstrates that existing quantized checkpoints can benefit from better grouping through offline calibration alone. Jointly learning rotations and permutations would be out of scope for now, but it remains a great next direction for us.
We believe these additions address your concerns and strengthen the paper. Would you consider revising your score in light of the new evidence before the rebuttal phase deadline? Thank you again for all your time—your feedback helped us improve the evaluation!
Thanks for the follow-up. The checks across 14 model/KV settings answer my question about how well the dispersion surrogate tracks group range, reconstruction error, and perplexity. The Mistral-7B results and the five-seed runs give me more confidence that the gains hold across dense models and calibration samples. I also see why you reorder offline. With the current per-row scales, the training loss is the same for every channel order.
The B300 NVFP4 timings show the cost of the permutation on a native 4-bit backend, including during decoding. I'd still like to know the overhead of a native INT4 implementation of the dynamic asymmetric group-wise quantizer used in the accuracy tests. MoE and broader coverage with QuaRot also remain open. I'll keep my rating at 2.
Thank you for the follow-up. We ran both remaining experiments: CAPQuant on QuaRot across six models, and the native INT4 overhead of the group-wise quantizer.
QuaRot acts on the rotation layers, and CAPQuant does not affect them. We therefore assumed that the Llama-3-8B QuaRot rows in Table 1 and the multi-model OSTQuant results in Table 2 were sufficient. For native INT4, we did not have access to an Ada or Ampere GPU. H200 and B300, the available cloud GPUs, have no INT4 tensor-core paths. We thus responded with experiments on native NVFP4. We have since rented a cloud L40S (Ada).
1. QuaRot: lower perplexity in all 24 settings. QuaRot (Hadamard rotation, GPTQ W4, asymmetric A4) with g = 128 activation groups, on six models from 1B to 13B. Evaluation tokens are fixed. We use 5 seeds per model.
Mean perplexity, QuaRot + contiguous → QuaRot + CAPQuant (reduction):
| Model | KV16 Wiki | KV16 C4 | KV4 Wiki | KV4 C4 |
|---|---|---|---|---|
| Llama-3.2-1B | 12.230 → 11.933 (2.4%) | 18.938 → 18.507 (2.3%) | 13.174 → 12.821 (2.7%) | 20.538 → 19.992 (2.7%) |
| Qwen3-4B | 15.821 → 15.266 (3.5%) | 22.560 → 21.927 (2.8%) | 16.283 → 15.651 (3.9%) | 22.785 → 22.147 (2.8%) |
| Llama-2-7B | 5.859 → 5.810 (0.8%) | 7.940 → 7.889 (0.6%) | 5.917 → 5.856 (1.0%) | 8.026 → 7.965 (0.8%) |
| Mistral-7B | 5.491 → 5.468 (0.4%) | 8.788 → 8.754 (0.4%) | 5.523 → 5.501 (0.4%) | 8.837 → 8.809 (0.3%) |
| Llama-3-8B | 7.207 → 7.127 (1.1%) | 11.683 → 11.583 (0.9%) | 7.331 → 7.252 (1.1%) | 11.913 → 11.819 (0.8%) |
| Llama-2-13B | 5.194 → 5.170 (0.5%) | 7.220 → 7.192 (0.4%) | 5.229 → 5.209 (0.4%) | 7.271 → 7.242 (0.4%) |
CAPQuant outperforms MeanSort with QuaRot again. The pattern matches Table 2: largest gains at 1B–4B, smaller but consistent at 7B–13B.
2. Native INT4: CAPQuant adds 1.0–2.7%. On the L40S, INT4 runs 2.00× faster than INT8 on the tensor cores. Three paired arms: per-token INT4 (the OSTQuant baseline), g = 128 contiguous, g = 128 with CAPQuant.
| Model | Prefill, 2,048 tokens | Decode, largest batch that fits |
|---|---|---|
| Qwen3-4B | +8.66 ms (15.0%) / +1.76 ms (2.7%) | 64 users: +8.4% / +0.6% |
| Llama-2-7B | +14.33 ms (18.7%) / +1.48 ms (1.6%) | 64 users: +3.4% / +0.3% |
| Llama-2-13B | +32.37 ms (23.9%) / +1.68 ms (1.0%) | 32 users: +4.6% / +0.3% |
| Llama-1-30B | +53.91 ms (12.8%) / +6.88 ms (1.4%) | 16 users: +7.0% / +0.6% |
Each cell reports format / CAPQuant, where format is g = 128 contiguous over per-token and CAPQuant is its increment over contiguous. The format cost is the price of group-wise activation scales on INT4 tensor cores. Every g = 128 scheme pays it, including Atom and QuaRot-128G. CAPQuant's own cost is 1.0–2.7% at prefill and 0.3–1.4% at decode, in line with our B300 NVFP4 measurements, in which the tensor core natively handles block scales.
An in-depth analysis of CapQuant on MoEs remains future work, as mentioned before. For MoE models, we would like to note that an expert is a dense FFN and is reordered exactly as one, with its own channel order. We expect the same behavior as with dense models, but defer in-depth experiments to future work due to the lack of an OSTQ checkpoint and a lack of exploration in related work, which limited our foresight to prepare these checkpoints in advance. We hope these results close the remaining questions you had.
We would like to thank reviewer U7SZ for recognizing the covariance-aware objective and the fused integration. Your questions led us to validate the surrogate on every layer of every model. Short version: the surrogate tracks the real quantization error everywhere we measured, the gain holds on a new architecture and across seeds, and the native-kernel overhead is small.
1. The surrogate tracks the real quantization error across layers, models and KV settings
2. New architecture, stable across calibration samples and seeds
Mistral-7B at KV16 and KV4: 5 seeds, C4 calibration, 2/8/64 windows, nine-task accuracy. Llama-30B is complete.
14 settings: CAPQuant beats contiguous in 28/28 five-seed mean comparisons, in 28/28 with C4 calibration, and in all 140 individual seed/corpus comparisons.
Zero-shot accuracy is on par across seeds and orderings, within lm-eval noise. On Llama-2-13B, Llama-30B and Mistral-7B CAPQuant is +0.3 to +0.7 points over per-token OSTQuant.
QuaRot on more models, MoE and 70B are the natural next steps. The revised conclusion lists MoE and 70B as future work.
3. Native kernels: +3.7–5.5% prefill, +0.6–3.4% decode, a few MiB of memory
4. Why not learn the permutation during training?
Please also see (posted above, visible to all reviewers)
We hope the broader error analysis, calibration results, and native runtime measurements address your concerns. We would like to ask if the reviewer would be open to revising their score. Please let us know if you would like any additional experiments.