acceptodds
Under review as a conference paper at ICLR 2027

CAPQuant: Covariance-Aware Partitioning of Activation Channels for 4-bit LLM Quantization

Abstract

Low-bit post-training quantization (PTQ) reduces the memory and latency footprint of large language models (LLMs), but activation outliers can compromise accuracy. Group-wise activation quantization limits their reach by assigning one scale to each fixed-size block of channels, and rotation-based methods flatten the outliers before quantization. Which channels share a scale is then a free choice that these methods leave to the arbitrary channel order. We introduce CAPQuant, a training-free method that selects this order from the activation means and cross-channel covariances of an already-transformed model. Starting from the expected error of asymmetric group-wise quantization, we bound the per-group range by the within-group dispersion and reduce the dispersion to a sum of pairwise channel distances. Across balanced partitions the per-channel variances contribute a constant, so only mean differences and covariances decide the grouping, and sorting channels by their mean is already optimal when they are uncorrelated. CAPQuant minimizes the remaining objective with balanced clustering and pairwise swaps at calibration time, then folds the permutation into the adjacent weights and fuses it into the activation kernels, so inference needs no additional matrix multiplication or activation pass. On top of OSTQuant, converting the per-token activation sites to groups and ordering their channels reduces perplexity by 1.5–9.0% and raises zero-shot accuracy by 0.9–2.3 points across Qwen3-4B, Llama-3.2-1B, Llama-3-8B, and Llama-2-7B, with gains that persist at 13B and 30B parameters and transfer to frozen QuaRot and SpinQuant checkpoints. Ordering alone accounts for up to 2.5% of perplexity beyond contiguous grouping, and on Qwen3-4B its share grows as groups shrink, reaching 2.6% at the MXFP4 group size . The fused permutation adds 3.6–6.9% to prefill latency in a BF16 reference implementation. These results support channel order as an inexpensive calibration-time improvement for group-wise quantization.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, Training-Free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U7SZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAPQuant studies channel ordering for dynamic asymmetric group-wise INT4 activation quantization. It derives a within-group dispersion surrogate from a range-based error model and shows that, for equally sized groups, the sum of individual-channel variance terms is independent of the partition; mean differences and cross-channel covariances determine the ordering-dependent part. The method combines balanced clustering with improving channel swaps and folds the inverse permutation into adjacent weights while fusing the forward ordering into existing activation kernels. On frozen OSTQuant, QuaRot, and SpinQuant checkpoints, it improves perplexity; over contiguous grouping, ordering reduces perplexity in 15 of 16 reported comparisons. The reference prefill setup shows a 3.6–6.9% added latency for the permutation.

Critical strengths
  1. The paper introduces a covariance-aware channel reordering objective, which provides a principled explanation of why ordering based only on marginal channel statistics, such as range or mean, can be insufficient.
  2. The method is integrated through offline weight-column permutation and kernel fusion, avoiding additional matrix multiplications or separate activation passes. This is a technically sound and practically appealing design.
  3. The experiments cover multiple models, W4A4KV16 and W4A4KV4 settings, and multiple base quantizers, with contiguous, random, and mean-sorted ordering baselines.
Critical weaknesses
  1. The method optimizes a dispersion surrogate for the dynamic group-wise quantization objective rather than the actual quantization error itself. Although the paper provides theoretical bounds, the empirical evidence is still insufficient to establish that this approximation is consistently reliable across layers, models, and quantization configurations.
  2. While the experiments include Qwen3 and Llama models, they are all dense decoder-only Transformers. In addition, the QuaRot plug-in evaluation is reported only on Llama-3-8B. The generality across model architectures, calibration samples, and random seeds remains insufficiently validated.
  3. Runtime is measured with simulated activation quantization and BF16 matrix multiplication. Therefore, the practical overhead under native INT4 inference kernels, especially during decoding, remains unclear.
Questions for authors
  1. Could the authors report, across more settings, the correlation between the surrogate objective and the actual group range, activation reconstruction MSE, and downstream perplexity? How consistently do the actual range and MSE improve after optimization?
  2. Can the method be evaluated on additional architectures, such as Mistral, or mixture-of-experts models? Are the ordering gains and accuracy improvements stable across different calibration samples and random seeds?
  3. Could the authors report prefill and decoding latency, throughput, and memory overhead using native INT4 kernels, in order to better characterize the practical deployment trade-offs?
  4. My main question is: if it works, why not incorporate permutation-related learning during training (permutation itself is a type of orthogonal transformation)?
4 comments

We would like to thank reviewer U7SZ for recognizing the covariance-aware objective and the fused integration. Your questions led us to validate the surrogate on every layer of every model. Short version: the surrogate tracks the real quantization error everywhere we measured, the gain holds on a new architecture and across seeds, and the native-kernel overhead is small.

1. The surrogate tracks the real quantization error across layers, models and KV settings

  • Scope: all 1,488 activation sites of 14 model/KV settings. Real error is the relative error of the paper's 4-bit, g = 128 quantizer.
  • Ψ vs real error: median per-site Spearman 0.91–0.99. Ψ vs squared group range: 0.92–0.99 (14 settings).
  • After optimization: CAPQuant lowers the real error at every site, for orderings built from five calibration seeds. Four of the five never saw the measured tokens. 0 of 7,440 cases is worse than contiguous. The swap refinement alone lowers the real error at 97–100% of sites.
  • Downstream: across all 14 settings, ranking 24 orderings each by total Ψ and by perplexity, the Spearman correlation is positive in all 28 model/corpus cases (median 0.68 WikiText-2, 0.67 C4), as strong as the real 4-bit error's own correlation with perplexity (Appendix F.7, Figure 6). On Llama-3-8B, eight orderings give Spearman 0.83 and Pearson 0.96 (Appendix A.11).
CAPQuant vs contiguous, Llama-3-8BCalibration activationsIndependent calibration draw
Dispersion Ψ24% lower15% lower
Mean group range12% lower7% lower
Relative reconstruction error1.00% → 0.77%1.00% → 0.86%
  • The guarantee is on dispersion. These measurements connect it to error and quality in every evaluated setting. This evidence is now in the main text (Section 4.1), with all 14 settings in Appendix F.7.

2. New architecture, stable across calibration samples and seeds

  • Mistral-7B at KV16 and KV4: 5 seeds, C4 calibration, 2/8/64 windows, nine-task accuracy. Llama-30B is complete.

  • 14 settings: CAPQuant beats contiguous in 28/28 five-seed mean comparisons, in 28/28 with C4 calibration, and in all 140 individual seed/corpus comparisons.

  • Zero-shot accuracy is on par across seeds and orderings, within lm-eval noise. On Llama-2-13B, Llama-30B and Mistral-7B CAPQuant is +0.3 to +0.7 points over per-token OSTQuant.

  • QuaRot on more models, MoE and 70B are the natural next steps. The revised conclusion lists MoE and 70B as future work.

3. Native kernels: +3.7–5.5% prefill, +0.6–3.4% decode, a few MiB of memory

  • 36 paired NVFP4 workloads on B300, four models from 4B to 30B, full embedding-to-logits graph with CAPQuant's real permutations.
  • Decode, one step: +0.55–3.38%, and +0.55–1.22% at batch 64. Prefill: +3.67–5.46%. Throughput drops by about the same percentage.
  • Memory: indices take 2.04–7.15 MiB, at most 0.12% of the 4-bit weights. Peak GPU memory rises by less than 0.1 GiB.
  • NVFP4 is the native 4-bit path on current GPUs. The prefill cost in ms is similar on INT8, FP8 and NVFP4, because CAPQuant does not touch the matmul.

4. Why not learn the permutation during training?

  • Agreed: a permutation is orthogonal, and joint learning is a promising extension.
  • Training gives no signal for the order today. At the three sites CAPQuant reorders, OSTQuant uses one scale per row, so its loss is identical across all channel orders.
  • CAPQuant answers the complementary question: can grouping improve after the rotation and weights are fixed? Yes. Offline, no backprop, no retraining, 82 s for Llama-2-7B and 231 s for Llama-30B (Table 6).
  • So it improves existing checkpoints such as OSTQuant, QuaRot, and SpinQuant (Table 1).
  • Joint optimization of rotation and permutation under group-wise scales is future work.

Please also see (posted above, visible to all reviewers)

  • General response (3/4): surrogate table per setting.
  • General response (1/4): five-seed table, and the list of manuscript changes, including one corrected Table 2 entry (contiguous accuracy on Qwen3-4B KV4 is 58.76, was 57.89).
  • General response (4/4): setup, all three kernels, memory.
  • The revised PDF is uploaded. Changes are in red.

We hope the broader error analysis, calibration results, and native runtime measurements address your concerns. We would like to ask if the reviewer would be open to revising their score. Please let us know if you would like any additional experiments.

Thank you again for your constructive feedback!

  • We have expanded the surrogate analysis across 14 model/KV settings, finding a 0.91–0.99 correlation with actual reconstruction error. Across 7,440 site/seed measurements, CAPQuant never increases reconstruction error over contiguous grouping. We also report strong correlations with squared group range and downstream PPL, directly strengthening the empirical support for our objective.
  • We completed Mistral-7B evaluations, additional calibration seeds, C4 calibration, and 2/8/64-window experiments. The gains persist across calibration samples, with the five-seed mean PPL improving over contiguous grouping in all comparisons.
  • We also added native NVFP4 runtime experiments on B300, covering prefill, decode, throughput, and memory under the same settings as our initial evalutions.
  • Regarding learning permutations during training: we agree this is an interesting direction! CAPQuant demonstrates that existing quantized checkpoints can benefit from better grouping through offline calibration alone. Jointly learning rotations and permutations would be out of scope for now, but it remains a great next direction for us.

We believe these additions address your concerns and strengthen the paper. Would you consider revising your score in light of the new evidence before the rebuttal phase deadline? Thank you again for all your time—your feedback helped us improve the evaluation!

Thanks for the follow-up. The checks across 14 model/KV settings answer my question about how well the dispersion surrogate tracks group range, reconstruction error, and perplexity. The Mistral-7B results and the five-seed runs give me more confidence that the gains hold across dense models and calibration samples. I also see why you reorder offline. With the current per-row scales, the training loss is the same for every channel order.

The B300 NVFP4 timings show the cost of the permutation on a native 4-bit backend, including during decoding. I'd still like to know the overhead of a native INT4 implementation of the dynamic asymmetric group-wise quantizer used in the accuracy tests. MoE and broader coverage with QuaRot also remain open. I'll keep my rating at 2.

Thank you for the follow-up. We ran both remaining experiments: CAPQuant on QuaRot across six models, and the native INT4 overhead of the group-wise quantizer.

QuaRot acts on the rotation layers, and CAPQuant does not affect them. We therefore assumed that the Llama-3-8B QuaRot rows in Table 1 and the multi-model OSTQuant results in Table 2 were sufficient. For native INT4, we did not have access to an Ada or Ampere GPU. H200 and B300, the available cloud GPUs, have no INT4 tensor-core paths. We thus responded with experiments on native NVFP4. We have since rented a cloud L40S (Ada).

1. QuaRot: lower perplexity in all 24 settings. QuaRot (Hadamard rotation, GPTQ W4, asymmetric A4) with g = 128 activation groups, on six models from 1B to 13B. Evaluation tokens are fixed. We use 5 seeds per model.

Mean perplexity, QuaRot + contiguous → QuaRot + CAPQuant (reduction):

ModelKV16 WikiKV16 C4KV4 WikiKV4 C4
Llama-3.2-1B12.230 → 11.933 (2.4%)18.938 → 18.507 (2.3%)13.174 → 12.821 (2.7%)20.538 → 19.992 (2.7%)
Qwen3-4B15.821 → 15.266 (3.5%)22.560 → 21.927 (2.8%)16.283 → 15.651 (3.9%)22.785 → 22.147 (2.8%)
Llama-2-7B5.859 → 5.810 (0.8%)7.940 → 7.889 (0.6%)5.917 → 5.856 (1.0%)8.026 → 7.965 (0.8%)
Mistral-7B5.491 → 5.468 (0.4%)8.788 → 8.754 (0.4%)5.523 → 5.501 (0.4%)8.837 → 8.809 (0.3%)
Llama-3-8B7.207 → 7.127 (1.1%)11.683 → 11.583 (0.9%)7.331 → 7.252 (1.1%)11.913 → 11.819 (0.8%)
Llama-2-13B5.194 → 5.170 (0.5%)7.220 → 7.192 (0.4%)5.229 → 5.209 (0.4%)7.271 → 7.242 (0.4%)

CAPQuant outperforms MeanSort with QuaRot again. The pattern matches Table 2: largest gains at 1B–4B, smaller but consistent at 7B–13B.

2. Native INT4: CAPQuant adds 1.0–2.7%. On the L40S, INT4 runs 2.00× faster than INT8 on the tensor cores. Three paired arms: per-token INT4 (the OSTQuant baseline), g = 128 contiguous, g = 128 with CAPQuant.

ModelPrefill, 2,048 tokensDecode, largest batch that fits
Qwen3-4B+8.66 ms (15.0%) / +1.76 ms (2.7%)64 users: +8.4% / +0.6%
Llama-2-7B+14.33 ms (18.7%) / +1.48 ms (1.6%)64 users: +3.4% / +0.3%
Llama-2-13B+32.37 ms (23.9%) / +1.68 ms (1.0%)32 users: +4.6% / +0.3%
Llama-1-30B+53.91 ms (12.8%) / +6.88 ms (1.4%)16 users: +7.0% / +0.6%

Each cell reports format / CAPQuant, where format is g = 128 contiguous over per-token and CAPQuant is its increment over contiguous. The format cost is the price of group-wise activation scales on INT4 tensor cores. Every g = 128 scheme pays it, including Atom and QuaRot-128G. CAPQuant's own cost is 1.0–2.7% at prefill and 0.3–1.4% at decode, in line with our B300 NVFP4 measurements, in which the tensor core natively handles block scales.

An in-depth analysis of CapQuant on MoEs remains future work, as mentioned before. For MoE models, we would like to note that an expert is a dense FFN and is reordered exactly as one, with its own channel order. We expect the same behavior as with dense models, but defer in-depth experiments to future work due to the lack of an OSTQ checkpoint and a lack of exploration in related work, which limited our foresight to prepare these checkpoints in advance. We hope these results close the remaining questions you had.

Reviewer_YwgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a quantization technique, CAPQuant. This is a training-free method that picks the channel ordering for group-wise asymmetric activation quantization after a rotation-based quantizer such as spinquant.

Critical strengths

The paper is well written and clear. CapQuant bounds the group-range error by within-group dispersion, notes that the variance term is constant across balanced partitions, and minimizes a mean-difference plus covariance objective with hierarchical initialization and balanced swap refinement.

Critical weaknesses

Headline claims should be framed around the ordering contribution, and OSTQuant should be compared at an equal metadata budget.The headline 1.5–9.0% improvement over OSTQuant mostly comes from adding g=128 scales, which also adds 0.16 bits of metadata per activation. Ordering alone adds only up to 2.5% and is often within 0.05–0.25% of mean-sorting (Llama-2-7B, Qwen3-4B). Every result uses a single calibration seed with only two 2,048-token WikiText-2 windows, and the random-order baseline is a single draw, why? The largest model is the old Llama-1-30B, and larger models report perplexity only. The paper should add 70B models (Llama-3-70B, Qwen2.5/3-72B), MoE models (Mixtral or others), and other families (Mistral, Gemma). There is no comparison with or integration into DuQuant’s zigzag permutation, Atom’s reordering, PermuQuant, or FlatQuant. The conclusion’s claim of an “exact formulation” of the quantization error overstates what is only a bound.

Questions for authors

See weakness section.

4 comments

We would like to thank reviewer YwgY for the detailed feedback. You are right about the framing, and we rebuilt the evaluation around it. Short version: at equal metadata the ordering gain holds in 28/28 comparisons over five seeds, and CAPQuant is the best ordering against DuQuant, Atom and PermuQuant.

1. The headline is now the ordering contribution, at equal metadata

  • Agreed. The 1.5–9.0% figure combines group-wise scales and ordering. The revised abstract, introduction and conclusion lead with CAPQuant vs contiguous grouping (Table 5).
  • "OSTQuant + contiguous grouping" is OSTQuant at the same metadata budget: same g = 128, same scale and zero-point storage (0.16 bits per activation), same frozen transform and quantized weights. The permutation adds no metadata. Its indices take 2.04–7.15 MiB.
  • At this equal budget CAPQuant lowers the five-seed mean PPL in 28/28 model/corpus comparisons, and in all 140 individual seed/corpus comparisons.

2. Five seeds, five random draws, two calibration domains, three calibration sizes

  • You are right that one seed and one random draw were not enough. Now: 5 calibration seeds and 5 random-order draws on 14 model/KV settings.
  • vs mean-sort: 25/28, one tie, the other two within 0.002 PPL. Random order stays at the contiguous level.
  • C4 calibration instead of WikiText-2: 28/28 over contiguous, 26/28 over mean-sort.
  • 2, 8 and 64 calibration windows on 12 settings: same quality.

3. CAPQuant is the best ordering against DuQuant zigzag, Atom and PermuQuant

  • All orderings run on the same frozen OSTQuant checkpoints, so only the channel order differs. Zigzag and Atom follow the official code. PermuQuant follows its Eq. 9–10, since no code is released.
  • CAPQuant has the lowest PPL in 26/28 comparisons, including all 8 on Mistral-7B and Llama-30B. The two exceptions, on Qwen3-4B KV16, are within its seed spread.
  • FlatQuant has no ordering stage. A faithful integration relearns its clipping thresholds (Appendix B.7), which goes beyond that of a training-free plug-in. The revised conclusion states this as future work.

4. New model family, and accuracy for larger models

WikiText-2 / C4, five-seed meanContiguousCAPQuant
Mistral-7B KV165.426 / 8.7275.416 / 8.721
Mistral-7B KV45.447 / 8.7815.434 / 8.767
Llama-30B KV164.381 / 6.5144.366 / 6.496
Llama-30B KV44.366 / 6.4214.357 / 6.405
  • Mistral-7B is evaluated at KV16 and KV4 with the full protocol. Llama-30B is complete.
  • Nine-task accuracy is now reported for Llama-2-13B (66.68 / 65.88) and Llama-30B (68.62 / 68.61) at KV16 / KV4. Accuracy is preserved under ordering: all orderings are within 0.6 points. Baselines are in general response (3/4).
  • 70B, MoE and further families are the natural next step. The revised conclusion states this scope.

5. Theory wording

  • Agreed. The conclusion now reads "exact decomposition of a dispersion surrogate ... with bounds that relate dispersion to the group range".

Please also see (posted above, visible to all reviewers)

  • General response (1/4): five-seed table for all 14 settings, the setup note on checkpoints, and the list of manuscript changes, including one corrected Table 2 entry (contiguous accuracy on Qwen3-4B KV4 is 58.76, was 57.89).
  • General response (2/4): C4-calibration and calibration-size tables.
  • General response (3/4): ordering-baseline table for all 14 settings and the accuracy table.
  • The revised PDF is uploaded. Changes are in red (Table 5, Section 5.4, Appendix F).

We appreciate the suggestions. They improved the attribution, the robustness and the baselines. We would like to ask if the reviewer would be open to revising their score. Please let us know if you would like any additional experiment.

Thank you again for your feedback! We agree that the headline should isolate the contributions from ordering.

  • The gain of up to 2.5% uses the same group size and scale budget. Across five seeds, CAPQuant improves mean PPL over contiguous grouping in 28/28 comparisons and mean-sort in 25/28, plus one tie. We've densely evaluated our method across more seeds, random draws, and window sizes.
  • We also completed Mistral-7B evaluations and larger-model accuracy tests, which were previously only perplexity-based.
  • Similar layers and our mathematical derivation motivate the expectation that benefits will extend to other architectures; however, due to resource constraints, detailed runs on MoEs and 70B evaluations remain future work, given the lack of OSTQuant checkpoints for models in that range and cloud GPU availability constraints.
  • We added method comparisons against DuQuant, Atom, and PermuQuant ordering rules on a shared backend. CAPQuant again wins 26/28 comparisons. FlatQuant train-time learning and integrations are future work and currently remain out of scope, as they could very well be extensions of our method, including the rotation matrix.
  • We have corrected “exact formulation.” Beyond the bounds, our expanded analysis shows a correlation of 0.91–0.99 with the actual reconstruction error, supported by downstream PPL analysis.

We hope these additions address your concerns. We would appreciate a response before the deadline. Thank you for your time!

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Thank you for the detailed response and the additional experiments. The multi-seed and calibration studies, the equal-metadata reframing, the added ordering baselines, and the corrected “exact formulation” wording address most of my concerns. However, scale remains unresolved (no 70B or MoE models, FlatQuant deferred), and the ordering gain is small, with no zero-shot accuracy benefit. I will maintain my score.

Reviewer_1T7KOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CAPQuant addresses the effect of channel assignment on shared scales in group-wise activation quantization. It constructs pairwise distances from channel means and covariances, then uses balanced clustering and swap optimization to select channel groups and orderings. The authors derive that, for balanced groups, the within-group variance term is constant, leaving mean differences and covariances to determine their dispersion objective. CAPQuant is an offline calibration plug-in for methods including OSTQuant, QuaRot, and SpinQuant. Experiments on several LLaMA/Qwen models under W4A4KV16/KV4 report perplexity and accuracy gains and measure simulated permutation overhead; native INT4 end-to-end speed is not shown.

Critical strengths
  1. The decomposition of asymmetric group-quantization error explains why means and covariances affect grouping, and the objective is derived directly from the problem.
  2. The method is training-free, plugs into existing quantization pipelines, and is accompanied by code and reproduction materials.
Critical weaknesses
  1. Grouping uses only two 2,048-token calibration sequences and may be sensitive to samples or domain. Sensitivity to calibration size and random seed should be reported, especially given the stability of high-dimensional covariance estimates with limited samples.
  2. The paper reports that the PPL differences from the covariance term relative to mean-sorted order are within 0.25% on Qwen3-4B and 0.05% on Llama-2-7B. These figures bound the difference; they do not mean every comparison improves (for example, Qwen3-4B WikiText-2 PPL is 14.209 versus 14.182 for mean sorting). The paper should clarify on which metrics the covariance term provides consistent gains and its independent contribution beyond regrouping itself.
  3. The gains are often modest, and latency is measured with BF16 matrix multiplication and simulated quantization; the authors acknowledge this does not establish native INT4 speedups. For a deployment-motivated paper, native low-bit kernels and end-to-end latency/throughput and memory measurements remain important evidence gaps.
Questions for authors
  1. Are two calibration sequences sufficient for stable covariance estimates? How do grouping and model quality change with more tokens or a different calibration domain?
  2. Can you provide end-to-end prefill/decode latency and throughput with native INT4 kernels, or explain why the current simulated measurements reliably predict their trend?
4 comments

We would like to thank reviewer 1T7K for recognizing the derivation and the training-free plug-in design, and for the precise suggestions. We ran each experiment you asked for. Short version: the gain holds across seeds, domain and calibration size, the covariance term helps beyond mean-sort, and the overhead on native kernels is small.

1. Two calibration windows are enough

  • Seeds: 5 calibration seeds on 14 model/KV settings. CAPQuant beats contiguous in 28/28 five-seed mean comparisons and in all 140 individual seed/corpus comparisons.
  • Domain: C4 calibration instead of WikiText-2. Still 28/28.
  • Size: 2, 8 and 64 windows on 12 settings. CAPQuant beats contiguous at every size (72/72). Example: Mistral-7B KV16 stays within 5.414–5.418 WikiText-2 PPL.
  • Grouping: the exact groups change with the sample (adjusted Rand index 0.03–0.15 across seeds). The quality does not. Many partitions are near-equally good and CAPQuant finds one every time. Groups converge with more data: 8-vs-64-window agreement is 0.16–0.32.
  • Why few samples suffice: CAPQuant never forms the d × d covariance. It clusters channels in the top r ≤ 128 directions of 4,096 calibration tokens (Section 4.2).

2. The covariance term gives consistent gains over mean-sort

  • Thank you for the clarification. Agreed: the percentages in the paper only bound the difference. The revised Section 5.2 adds matched comparisons.
  • Five-seed means at the same checkpoint, g = 128 and metadata: CAPQuant beats mean-sort in 25/28 comparisons, with one tie and the other two within 0.002 PPL.
  • Your example reverses with more seeds: Qwen3-4B KV16 WikiText-2 is 14.115 for CAPQuant and 14.168 for mean-sort.
  • Metrics: the covariance term helps on perplexity (above) and on reconstruction error, where CAPQuant is below mean-sort (Appendix A.11). Zero-shot accuracy is on par across orderings, within lm-eval noise.
  • Beyond regrouping itself: random regrouping (5 draws) stays at the contiguous level. The gain comes from the criterion.
WikiText-2 / C4, five-seed meanMean-sortCAPQuant
Qwen3-4B KV1614.168 / 21.72014.115 / 21.611
Qwen3-4B KV414.361 / 21.96014.239 / 21.821
Llama-30B KV164.372 / 6.5114.366 / 6.496
Llama-30B KV44.365 / 6.4184.357 / 6.405

3. Native low-bit kernels: +3.7–5.5% prefill, +0.6–3.4% decode

  • 36 paired NVFP4 workloads on B300, four models from 4B to 30B, full embedding-to-logits graph with CAPQuant's real permutations.
  • Prefill: +3.67–5.46%. One decode step: +0.55–3.38%, and +0.55–1.22% at batch 64. Index memory: 2.0–7.2 MiB.
  • Why NVFP4: measured INT4 is 0.08× and 0.98× of INT8. NVFP4 is the native 4-bit path. FP8 and INT8 are reported too.
  • Why simulation predicts the trend: CAPQuant does not touch the matmul, so its prefill cost is a near-fixed number of ms on any kernel. Llama-30B: +4.30 ms (INT8), +4.42 ms (FP8), +4.17 ms (NVFP4). The percentage follows from the kernel speed.
  • On gain size: the gain needs no extra metadata, no retraining and no backprop. It grows as groups shrink, from 0.5% at g = 128 to 2.6% at g = 32 on Qwen3-4B (Table 4).

Please also see (posted above, visible to all reviewers)

  • General response (1/4): five-seed table for all 14 settings, the setup note on checkpoints, and the list of manuscript changes, including one corrected Table 2 entry (contiguous accuracy on Qwen3-4B KV4 is 58.76, was 57.89).
  • General response (2/4): full C4-calibration and calibration-size tables.
  • General response (4/4): setup, all three kernels, memory.
  • The revised PDF is uploaded. Changes are in red (Table 5, Section 5.4, Appendix F).

We hope this addresses your concerns. We would like to ask if the reviewer would be open to revising their score. Please let us know if you would like any additional experiments.

Thank you again for your feedback! We have attached additional calibration results for five seeds each, using two 2048-token windows. These results further point to the strength of our method across domains (C4, Wikitext) and window sizes (2, 8, 64). Groupings change moderately with calibration seeds, but the gains persist! We outperform continuous grouping across all comparisons (28/28) and over most of the mean sort (25/28, including one tie), and demonstrate that our training-free method achieves gains over various quantization channel-ordering techniques.

We also added runtime experiments, as per your recommendation, on a B300 with native NVFP4 kernels, covering prefill, decode, throughput, and memory. We use full forward graphs with architecture-matched random weights and actual CAPQuant permutations, mimicking the quality of our asymmetric INT4 evaluation.

We believe these additions, together with the mathematical grounding that motivated CAPQuant in the first place, address the concerns raised in your review. We would be happy to clarify any remaining questions. In light of the rebuttal period deadline, would you consider revising your score? Thank you for your time and consideration once again! Your feedback pushed us and helped us improve the paper's claims.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

The multi-seed / multi-domain / calibration-size studies and the native NVFP4 runtime results are helpful and address much of my concern on calibration stability and deployment measurement. My remaining reservation is mainly about effect size: even with the clearer five-seed mean-sort comparisons (25/28 wins), the absolute PPL gains from the covariance term remain small, so the independent contribution beyond mean sorting is still limited relative to the paper’s main claim. I therefore maintain my rating.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/4): new experiments and changes to the manuscript

We thank all three reviewers for the constructive feedback. It made the paper stronger. Summary of the new experiments first.

Main result: the ordering gain is robust. 14 model/KV settings (the 12 in the paper plus Mistral-7B at KV16 and KV4), 5 calibration seeds, 2 calibration domains. Every comparison uses the same frozen checkpoint, the same g = 128 and the same metadata. Only the channel order changes.

  • vs contiguous grouping: CAPQuant lowers the five-seed mean PPL in 28/28 model/corpus comparisons. Every individual seed/corpus comparison improves: 140/140.
  • vs mean-sort (isolates the covariance term): 25/28, one tie, the other two within 0.002 PPL.
  • vs random regrouping (5 draws): random stays at the contiguous level. CAPQuant is ahead in 28/28.
  • vs DuQuant zigzag, Atom and PermuQuant orderings: CAPQuant has the lowest PPL in 26/28.
  • C4 calibration instead of WikiText-2: 28/28 over contiguous, 26/28 over mean-sort.
  • Calibration size: 2, 8 and 64 windows give the same quality (12 settings).
  • Surrogate: Ψ tracks the real 4-bit error with Spearman 0.91–0.99. The real error drops at all 1,488 sites for all five seeds (0 of 7,440 worse than contiguous).
  • Native low-bit kernels (NVFP4 on B300): prefill +3.67–5.46%, decode +0.55–3.38%. Index memory 2.0–7.2 MiB.
  • Accuracy: nine-task zero-shot added for Llama-2-13B, Llama-30B and Mistral-7B. Accuracy is preserved under every ordering (within 0.6 points).

Five-seed mean perplexity, WikiText-2 / C4 (lower is better, best in bold)

SettingContiguousRandomMean-sortCAPQuant
Qwen3-4B KV1614.285 / 21.92314.308 / 21.88214.168 / 21.72014.115 / 21.611
Qwen3-4B KV414.462 / 22.21814.487 / 22.18214.361 / 21.96014.239 / 21.821
Llama-3.2-1B KV1611.433 / 17.91211.430 / 17.89911.443 / 17.91811.315 / 17.734
Llama-3.2-1B KV411.950 / 18.98011.946 / 18.91711.971 / 18.95111.774 / 18.748
Llama-3-8B KV166.954 / 11.3136.954 / 11.2996.938 / 11.2926.903 / 11.214
Llama-3-8B KV47.111 / 11.7097.111 / 11.7057.094 / 11.6827.080 / 11.662
Llama-2-7B KV165.725 / 7.8295.721 / 7.8305.708 / 7.8195.706 / 7.820
Llama-2-7B KV45.809 / 7.9085.805 / 7.9105.793 / 7.9005.795 / 7.899
Llama-2-13B KV165.150 / 7.1975.155 / 7.2025.147 / 7.1975.128 / 7.168
Llama-2-13B KV45.173 / 7.2465.175 / 7.2425.169 / 7.2355.156 / 7.222
Mistral-7B KV165.426 / 8.7275.426 / 8.7345.416 / 8.7265.416 / 8.721
Mistral-7B KV45.447 / 8.7815.449 / 8.7815.436 / 8.7695.434 / 8.767
Llama-30B KV164.381 / 6.5144.379 / 6.5054.372 / 6.5114.366 / 6.496
Llama-30B KV44.366 / 6.4214.367 / 6.4204.365 / 6.4184.357 / 6.405

Within a row, all orderings share one checkpoint. On Llama-3.2-1B the ordering gain holds on both checkpoints: 1.0% on WikiText-2 at KV16 in each. Llama-3-8B and Llama-2-13B are rebuilt from the released transforms and match the paper within 0.014 PPL.

Revised PDF uploaded. Changes are in red. Table and section numbers in our replies refer to it. The revision:

  • leads with the ordering contribution at equal metadata (abstract, introduction, conclusion).
  • adds the new results. Main text: Table 5 and Section 5.4. Appendix F: seeds, calibration domain and size, ordering baselines, Mistral-7B, larger-model accuracy, surrogate analysis on 14 settings, native-kernel runtime (Tables 10–15, Figures 4–7).
  • adds matched five-seed comparisons to the mean-sort discussion (Section 5.2).
  • rewords "exact formulation" to "exact decomposition of a dispersion surrogate ... with bounds that relate dispersion to the group range" (Section 6).
  • states the scope: 70B, MoE, FlatQuant integration and joint learning of rotation and permutation are future work (Section 6).
  • corrects one Table 2 entry: contiguous accuracy on Qwen3-4B KV4 is 58.76 (was 57.89). The accuracy counts in Section 5.2 follow. Perplexity entries are unchanged. Five-seed CAPQuant mean in that setting: 59.02 (Table 13).
AuthorsOpenReview ↗

General response (2/4): calibration domain and calibration size

C4 calibration instead of WikiText-2 (seed 42, WikiText-2 / C4 PPL, best in bold). CAPQuant beats contiguous in 28/28 and mean-sort in 26/28. The gain does not depend on calibrating on the evaluation domain.

SettingContiguousMean-sortCAPQuant
Qwen3-4B KV1614.285 / 21.92314.117 / 21.59814.063 / 21.508
Qwen3-4B KV414.462 / 22.21814.325 / 21.87114.286 / 21.832
Llama-3.2-1B KV1611.433 / 17.91211.409 / 17.80411.338 / 17.744
Llama-3.2-1B KV411.950 / 18.98011.914 / 18.85311.848 / 18.774
Llama-3-8B KV166.954 / 11.3136.940 / 11.2686.924 / 11.211
Llama-3-8B KV47.111 / 11.7097.091 / 11.6647.089 / 11.659
Llama-2-7B KV165.725 / 7.8295.714 / 7.8215.711 / 7.814
Llama-2-7B KV45.809 / 7.9085.795 / 7.8975.805 / 7.900
Llama-2-13B KV165.150 / 7.1975.148 / 7.2005.135 / 7.168
Llama-2-13B KV45.173 / 7.2465.174 / 7.2395.158 / 7.226
Mistral-7B KV165.426 / 8.7275.418 / 8.7245.415 / 8.719
Mistral-7B KV45.447 / 8.7815.442 / 8.7685.436 / 8.765
Llama-30B KV164.381 / 6.5144.378 / 6.5064.363 / 6.499
Llama-30B KV44.366 / 6.4214.369 / 6.4204.357 / 6.401

Calibration size: CAPQuant with 2, 8 and 64 windows of 2,048 tokens (seed 42, WikiText-2 / C4 PPL). CAPQuant beats contiguous at every size: 72/72 comparisons. 32× more calibration data changes PPL by under 0.4% on five of the six models. Qwen3-4B moves by up to 1.5%, in both directions.

Setting2 windows (paper)8 windows64 windows
Qwen3-4B KV1614.209 / 21.68314.142 / 21.59414.097 / 21.613
Qwen3-4B KV414.101 / 21.69914.318 / 21.91714.163 / 21.745
Llama-3.2-1B KV1611.318 / 17.74411.299 / 17.72011.306 / 17.707
Llama-3.2-1B KV411.772 / 18.77111.797 / 18.73211.770 / 18.707
Llama-3-8B KV166.906 / 11.2176.900 / 11.2236.887 / 11.194
Llama-3-8B KV47.086 / 11.6627.077 / 11.6677.073 / 11.657
Llama-2-7B KV165.704 / 7.8185.712 / 7.8135.705 / 7.811
Llama-2-7B KV45.795 / 7.9035.790 / 7.8945.798 / 7.895
Llama-2-13B KV165.123 / 7.1685.125 / 7.1675.131 / 7.167
Llama-2-13B KV45.157 / 7.2235.156 / 7.2175.155 / 7.225
Mistral-7B KV165.416 / 8.7205.418 / 8.7215.414 / 8.722
Mistral-7B KV45.431 / 8.7695.438 / 8.7595.429 / 8.770

Group agreement (adjusted Rand index, 0 = chance, 1 = identical): 0.03–0.15 between two-window seeds, 0.16–0.32 between the 8- and 64-window partitions. The exact groups vary with the sample and converge with more data. The quality does not depend on which grouping is found.

AuthorsOpenReview ↗

General response (3/4): ordering baselines, accuracy, surrogate

Ordering baselines on the same frozen OSTQuant checkpoints (seed 42, WikiText-2 / C4 PPL, best in bold). Only the channel order differs. Zigzag and Atom follow the official code. PermuQuant follows its Eq. 9–10. CAPQuant is lowest in 26/28.

SettingCAPQuantDuQuant zigzagAtomPermuQuant
Qwen3-4B KV1614.209 / 21.68314.338 / 21.97114.168 / 21.70914.203 / 21.671
Qwen3-4B KV414.101 / 21.69914.513 / 22.22614.270 / 21.93114.245 / 21.995
Llama-3.2-1B KV1611.318 / 17.74411.427 / 17.85911.473 / 17.92611.450 / 17.894
Llama-3.2-1B KV411.772 / 18.77111.973 / 18.88911.956 / 18.93511.929 / 18.920
Llama-3-8B KV166.906 / 11.2176.951 / 11.3006.933 / 11.2746.937 / 11.252
Llama-3-8B KV47.086 / 11.6627.103 / 11.7037.089 / 11.6897.102 / 11.711
Llama-2-7B KV165.704 / 7.8185.725 / 7.8325.712 / 7.8295.719 / 7.830
Llama-2-7B KV45.795 / 7.9035.809 / 7.9185.801 / 7.9085.801 / 7.911
Llama-2-13B KV165.123 / 7.1685.150 / 7.2025.131 / 7.1755.144 / 7.201
Llama-2-13B KV45.157 / 7.2235.180 / 7.2425.164 / 7.2365.169 / 7.240
Mistral-7B KV165.416 / 8.7205.426 / 8.7365.425 / 8.7365.424 / 8.733
Mistral-7B KV45.431 / 8.7695.449 / 8.7815.444 / 8.7775.447 / 8.779
Llama-30B KV164.367 / 6.5004.379 / 6.5064.374 / 6.5064.385 / 6.504
Llama-30B KV44.352 / 6.4074.369 / 6.4224.363 / 6.4084.366 / 6.416

Nine-task zero-shot accuracy, % (seed 42). Accuracy is preserved under ordering: all group-wise orderings are within 0.6 points.

SettingOSTQuant (per-token)ContiguousMean-sortCAPQuant
Llama-2-13B KV1666.0766.3066.7166.68
Llama-2-13B KV465.5865.7765.6165.88
Llama-30B KV1668.1668.7568.9268.62
Llama-30B KV467.9068.6568.7368.61
Mistral-7B KV1667.1967.7867.9667.63
Mistral-7B KV467.4367.6567.8767.89

Surrogate Ψ vs real 4-bit error (Spearman, median site, all 1,488 sites). Last column: share of sites where CAPQuant lowers the real error against contiguous.

SettingΨ vs real errorΨ vs squared rangeSites improved (seed 42 / held-out seeds)
Qwen3-4B KV160.980.98100% / 100%
Qwen3-4B KV40.980.98100% / 100%
Llama-3.2-1B KV160.980.98100% / 100%
Llama-3.2-1B KV40.980.98100% / 100%
Llama-3-8B KV160.970.97100% / 100%
Llama-3-8B KV40.950.95100% / 100%
Llama-2-7B KV160.970.97100% / 100%
Llama-2-7B KV40.940.94100% / 100%
Llama-2-13B KV160.940.94100% / 100%
Llama-2-13B KV40.930.94100% / 100%
Mistral-7B KV160.990.99100% / 100%
Mistral-7B KV40.990.99100% / 100%
Llama-30B KV160.910.92100% / 100%
Llama-30B KV40.930.94100% / 100%
AuthorsOpenReview ↗

General response (4/4): Native low-bit runtime (reviewers U7SZ and 1T7K)

On native kernels CAPQuant costs +3.67–5.46% in prefill and +0.55–3.38% per decode step (NVFP4, B300). In prefill the cost is a near-constant number of ms across kernels.

Setup

  • Kernels: NVFP4 (4-bit) on B300 with FlashInfer's FP4 matmul. FP8 and INT8 on H200 (measured INT4 throughput is 0.08× INT8 on H200 and 0.98× on B300).
  • Two arms of the same model: contiguous and CAPQuant. Same kernels, same quantizer. The only difference is the reordering, so the measured difference is its cost.
  • Whole model, embedding to logits, as a CUDA graph. CAPQuant's real per-layer orders at the 3 sites of every layer. Weights are random with the real shapes, since speed does not depend on values.
  • Prefill: one prompt of 256 to 4,096 tokens. Decode: one step for 1, 4, 16 and 64 users with a 1,024-token KV cache. We report the median of paired differences over alternating rounds.

Added latency, NVFP4 / FP8 / INT8

ModelPrefill, 2,048 tokensDecode, 1 userDecode, largest batch (64 users)
Qwen3-4B+1.07 ms (4.7%) / +1.06 ms (3.3%) / +1.36 ms (2.1%)+0.124 ms (3.4%) / +0.109 ms (1.9%) / +0.114 ms (1.7%)+0.100 ms (1.2%) / +0.119 ms (1.5%) / +0.116 ms (1.2%)
Llama-2-7B+1.27 ms (5.0%) / +1.01 ms (2.4%) / +1.10 ms (1.4%)+0.062 ms (1.8%) / +0.102 ms (1.7%) / +0.130 ms (2.0%)+0.101 ms (0.6%) / +0.112 ms (0.8%) / +0.125 ms (0.8%)
Llama-2-13B+1.97 ms (4.8%) / +1.55 ms (2.0%) / +1.71 ms (1.2%)+0.127 ms (2.5%) / +0.203 ms (2.2%) / +0.189 ms (2.0%)+0.151 ms (0.6%) / +0.179 ms (0.9%) / +0.197 ms (0.9%)
Llama-30B+4.17 ms (5.0%) / +4.42 ms (2.5%) / +4.30 ms (1.4%)+0.278 ms (3.1%) / +0.647 ms (3.3%) / +0.406 ms (2.1%)+0.255 ms (0.6%) / +0.138 ms (0.4%) / +0.384 ms (1.3%) (FP8 and INT8 at 32 users)
  • NVFP4, all 36 workloads: prefill +3.67–5.46%, decode +0.55–3.38%, and +0.55–1.22% at 64 users. Throughput drops by about the same percentage.
  • Trend: in prefill the extra ms is similar on INT8, FP8 and NVFP4, because CAPQuant does not touch the matmul. Llama-30B: +4.30 / +4.42 / +4.17 ms. The percentage is this near-fixed cost over the model time.
  • Memory: channel orders take 2.04–7.15 MiB, at most 0.12% of the 4-bit weights. Peak GPU memory rises by less than 0.1 GiB. No extra activation buffers.
  • NVFP4 is a different number format from the paper's asymmetric INT4 quality setting. The overhead transfers because it is independent of the matmul format.
  • Every token length, batch size and kernel is plotted in Figure 7 of the revised PDF. Table 15 adds the absolute latency and the memory (Appendix F.8).

Native INT4: CAPQuant adds 1.0–2.7%. On the L40S, INT4 runs 2.00× faster than INT8 on the tensor cores. Three paired arms: per-token INT4 (the OSTQuant baseline), g = 128 contiguous, g = 128 with CAPQuant.

ModelPrefill, 2,048 tokensDecode, largest batch that fits
Qwen3-4B+8.66 ms (15.0%) / +1.76 ms (2.7%)64 users: +8.4% / +0.6%
Llama-2-7B+14.33 ms (18.7%) / +1.48 ms (1.6%)64 users: +3.4% / +0.3%
Llama-2-13B+32.37 ms (23.9%) / +1.68 ms (1.0%)32 users: +4.6% / +0.3%
Llama-1-30B+53.91 ms (12.8%) / +6.88 ms (1.4%)16 users: +7.0% / +0.6%

Each cell reports format / CAPQuant, where format is g = 128 contiguous over per-token and CAPQuant is its increment over contiguous. The format cost is the price of group-wise activation scales on INT4 tensor cores. Every g = 128 scheme pays it, including Atom and QuaRot-128G. CAPQuant's own cost is 1.0–2.7% at prefill and 0.3–1.4% at decode, in line with our B300 NVFP4 measurements, in which the tensor core natively handles block scales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.