acceptodds
Under review as a conference paper at ICLR 2027

DUALQUANT: Dual-Scale Calibration-Free Weight-and-Activation Quantization for LLMs

15 views

Abstract

Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error also provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. By treating the quantizer as a black box and projecting scales onto their format-specific admissible set after each update, DUALQUANT provides a unified algorithm supporting a wide range of formats, including INT4, INT8, the full MX family, and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with particularly large gains for MX and NV block formats. We further validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61× memory reduction and up to 2.46× throughput speedup over BF16. Our anonymized codebase is available https://anonymous.4open.science/r/dualquant-69B3/README.md.

Primary area:
optimization
Keywords:
post-training quantization, dual-scale quantization, calibration-free, low-bit inference, efficient inference, block coordinate descent, weight-reconstruction error, block data format

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Js8NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DualQuant utilises a dual scaling mechanism derived from SINQ, but optimises dual scaling factors by minimising the mean squared error of weight reconstruction rather than minimising the activation-weight product error, thereby enabling competitive activation quantisation without the need for calibration data.

Critical strengths
  1. DualQuant achieves better perplexity and accuracy in most popular W4A4 percision formats compared to other SOTA solutions, while it is calibration-free.
  2. The observation that minimising the weights-reconstruction error can also achieve effective and accurate activation rescaling is interesting.
Critical weaknesses
  1. The observation on “minimising the weights-reconstruction error can also achieve effective and accurate activation rescaling” is empirical and lacks clear boundaries regarding its applicability. Although author provides evaluation results on different models, it is unclear when this approach might fail, or whether it is generally applicable.
Questions for authors
  1. It is good to see that Tables 11 and 12 present comparisons with FP16; please consider including further comparisons with other quantisation schemes in terms of end-to-end memory usage and throughput/latency.
  2. The phrase "calibration-free" might be ambiguous, the process of finding α\alpha and β\beta could also be considered as calibration, please consider use more precise description.
  3. Please clarify the details of the baseline configuration. Although the author provides some information in Appendix F, there remain some concerns regarding the baseline comparison. For example, QuaRot should support quantisation across the entire computation path (including the KV cache, weights and activations), whilst Table 1 only illustrates the quantisation precision for weights and activations. Does the evaluation of QuaRot uses 4-bit precision KV cache quantisation, but compare it to your work without using KV cache quantisation? If so the comparision is unfair.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Js8N: Activation Rescaling, End-to-End Memory and Throughput Comparisons, and Experimental Clarifications

We thank the reviewer for the thoughtful questions. We address them below and highlight the corresponding additions to the revised paper.

W1: Applicability of weight-derived activation rescaling.

The activation-side benefit of DualQuant is a key empirical observation of our work, supported by broad evidence across models, formats, and tasks. DualQuant's scales play two complementary roles:

  • Weight side (optimization). The row and column scales minimize a weight-reconstruction MSE under format-specific scale constraints. Each projected closed-form update exactly minimizes its block with the quantized weights fixed. Our revised Theorem 1 guarantees finite termination for fixed grids and conditional convergence for adaptive grids once the quantized matrix stabilizes. Importantly, this optimization uses only model weights, with no activation statistics or calibration data.

  • Activation side (empirical). The same column scales β\beta rescale layer inputs before activation quantization. Their benefit is not theoretically guaranteed, but we provide three complementary forms of evidence:

    (i) Isolated activation study (Appendix E, Table 7): Holding quantized weights fixed and varying only β\beta, activation SQNR improves across all seven projection types of Llama-3.1-8B, averaging +0.33 dB, with the largest improvements in k_proj (+0.81 dB) and v_proj (+0.65 dB).

    (ii) Scale ablation (Table 4): Adding β\beta to row-only scaling consistently reduces W+A perplexity across the evaluated models and formats.

    (iii) End-to-end evaluation: We evaluate six Llama/Qwen models, four 4-bit formats, perplexity, five zero-shot tasks, and newly added reasoning/code-generation benchmarks, demonstrating broad empirical effectiveness.

We hypothesize that weight-derived column scales rescale activations in a way that benefits activation quantization. Validating such benefits empirically is common practice in PTQ, as in QuaRot (NeurIPS 2024) and ResQ (ICML 2025); following this practice, we evaluate DualQuant broadly: six models, four 4-bit formats plus 8-bit, perplexity, zero-shot, and reasoning/code tasks, and native INT4/NVFP4 deployment. Across these settings, DualQuant is the strongest calibration-free method in almost all cases and matches or surpasses calibrated SmoothQuant in most settings, all without any calibration data.

Q1: Memory and throughput vs. other quantization schemes.

Beyond the BF16 comparisons (Tables 11–12 of the original submission, now Tables 15–16), we extended packed INT4 inference to native NVFP4 W4A4 on RTX 5090 (extended Tables 3 and new table 6), and directly compare DualQuant against calibrated SmoothQuant and AWQ+ using the same packed FP4 GEMM backend.

Llama-3.1-8B: Native NVFP4 inference (RTX 5090, batch 16).

MethodWikiText-2 PPL ↓Prefill (tok/s) ↑Decode (tok/s) ↑Peak prefill memory (GB) ↓
BF165.6212,82290216.62
SmoothQuant*7.3236,1341,2696.51
AWQ+*22.7936,3521,2636.53
DualQuant6.8336,1181,2636.53

*Calibrated. DualQuant achieves better PPL at matched efficiency, without calibration data.

Additional Qwen3-14B native deployment (rebuttal-only), packed NVFP4 W4A4 on RTX 5090. Model-weight and runtime-overhead memory.

ComponentBF16NVFP4 W4A4
Linear weights26.426.61
FP8 group scales + β–0.84
Embedding + lm_head (BF16)3.113.11
Runtime/allocator overhead–0.30
Subtotal29.5410.85

Qwen3-14B: End-to-end GPU memory and throughput vs. BF16. (Memory includes workload activations and 16-bit KV cache where applicable):

WorkloadPeak GPU memory, BF16 → W4A4 (GB)Memory reductionThroughput speedup
Prefill b=129.64 → 10.962.70×1.87×
Prefill b=1630.02 → 11.342.65×3.07×
Decode b=129.65 → 10.972.70×1.46×
Decode b=1630.14 → 11.462.63×1.48×

Overall, DualQuant scales to Qwen3-14B with up to 3.07× speedup and 2.70× memory reduction over BF16, while the activation quantization, including fused β\beta, uses only 1.9–4.2% of GPU time.

Q2: Definition of “calibration-free.”

In DualQuant, calibration-free means calibration data-free: both α\alpha and β\beta are determined exclusively from model weights, without input samples, forward passes, or activation statistics. Although we optimize these scales, the optimization requires no calibration data. We clarified this distinction in the revised Introduction.

Q3: QuaRot and KV-cache quantization.

In all experiments, only linear-layer weights and activations are quantized; KV caches and attention computations remain 16-bit. For QuaRot, we disable its optional KV-cache quantization and use the same W4A4 setting as DualQuant and other baselines. We now state this explicitly in Section 5.1 and Appendix F.

Thank you to the authors for their response. The authors' response addresses some of my concerns but does not fully resolve my concerns regarding the applicability of the proposed method. Therefore, I maintain my rating of Weak Accept.

AuthorsOpenReview ↗

Follow-up Response to Reviewer Js8N: Guaranteed Weight Optimization, General Applicability, and When the Activation Benefit Is Largest

We would like to thank the reviewer for the follow-up. We understand the remaining concern as: where does DualQuant apply, and under which conditions does its weight-derived activation rescaling help? We address this in three parts as below:

  1. The weight-side optimization is guaranteed. DualQuant solves an explicit optimization problem: it minimizes the weight-reconstruction MSE under the format's scale constraints. Every projected update is an exact block minimizer (Lemmas 1–2). On a fixed grid, the error decreases monotonically and the iterates terminate finitely at a coordinate-wise stationary point (Theorem 1(a)). On the adaptive grid used in our experiments, the same holds once the quantized weights stabilize, which can be checked at run time and occurs within about 15 iterations in practice (Theorem 1(b), Proposition 1, Figures 2–3).

  2. DualQuant is generally applicable by construction. Its only inputs are a weight matrix and the target quantizer, which every linear layer provides. It therefore applies to any linear layer in any architecture, with no calibration data, no architecture-specific design, and no tuning beyond the iteration count. In particular, DualQuant is applicable wherever established PTQ methods such as GPTQ, SmoothQuant, QuaRot, ResQ, AdpQ, EasyQuant, and many more are applicable: like DualQuant, they act on the linear layers of the model, but most additionally require calibration data (GPTQ, SmoothQuant) or architecture-specific rotation fusion (QuaRot, ResQ), whereas DualQuant needs only the weights themselves. Unlike approaches that rely on activation statistics (SmoothQuant), proxy heuristics (SINQ's matrix imbalance), or rotations alone (QuaRot), DualQuant optimizes an explicit error criterion under the format constraints. It also cannot fail by diverging: the format is built into the optimization problem itself, so every scale is projected onto the admissible set (E8M0, UE4M3, or BF16) at every step and is representable by construction. This is precisely the failure mode behind the divergence of SINQ+ and QuaRot on NVFP4 (Tables 1, 2, 8), which DualQuant avoids by design across all formats, models, and bit-widths we evaluated.

  3. When the activation-side benefit is largest. The same column scales β attenuate input channels with large-magnitude weight columns. This benefits activation quantization most when activation outliers are channel-aligned and persistent across tokens, which is the regime documented for LLM linear-layer inputs:

    • 4-bit block formats with constrained scales (MXINT4, MXFP4, NVFP4), where the shared block scale cannot absorb channel outliers. Here DualQuant is the strongest calibration-free method and often surpasses calibrated SmoothQuant (Tables 1, 2, 8). These are precisely the native formats of current accelerators.
    • Projections with channel-wise outliers, e.g., k_proj, v_proj, q_proj (+0.81, +0.65, +0.25 dB activation SQNR; Table 7). The gain is more modest where outliers are token-specific rather than channel-aligned, e.g., down_proj inputs with massive activations (+0.06 dB, Table 7), a case that no static per-channel scale, including SmoothQuant's, can target. Even there, β never reduced activation SQNR in any projection we measured (Table 7).

We would be glad to evaluate any specific scenario the reviewer has in mind, e.g., a particular architecture, format, or bit-width.

Reviewer_ZxxVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DUALQUANT performs calibration-free weight-and-activation quantization by optimizing row and column scales to minimize weight reconstruction error. The column scales also rescale activations. Experiments cover Llama/Qwen and several low-bit formats, with a packed INT4 deployment demonstrating practical speedups.

Critical strengths
  1. Useful approach: deriving activation rescaling from weights alone removes the need for calibration data.
  2. Broad evaluation across six models, four 4-bit formats, perplexity benchmarks, and downstream tasks; scale ablations support using both row and column scales.
  3. Concrete deployment evidence: packed INT4 closely matches simulated accuracy and reports 1.39–2.46× speedups over BF16.
Critical weaknesses
  1. Convergence claims exceed the proof. Quantizer stabilization is assumed rather than established. Appendix C’s compactness argument is also flawed: its frozen-quantizer objective need not diverge when a scale tends to infinity. Convergence of the full projected algorithm therefore remains insufficiently justified.
  2. Comparative and deployment evidence has limits. SINQ+ is an author-created extension, and QuaRot uses RTN without weight-error compensation. Their MX/NV failures do not establish superiority over stronger implementations. Actual deployment tests cover only INT4 on one GPU with short prompts.
Questions for authors
  1. Can you correct the convergence proof and establish conditions ensuring quantizer stabilization and stationarity under discrete scale constraints?
  2. Do the gains persist against stronger baselines and in native MX/NV deployment with longer contexts?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ZxxV: Revised Convergence Analysis, Stronger Baselines, and Native NVFP4 Deployment

We thank the reviewer for the careful assessment; we have revised the proof, clarified baselines, and added native NVFP4 evaluation.

W1, Q1: Convergence, quantizer stabilization, and compactness.

Following the reviewer's comment, we revised the proof of Theorem 1 step by step (Sec. 4.3 and Appendix C): it now treats fixed and adaptive grids separately, uses a direct finite-state argument instead of compactness, and gives conditions for stabilization.

Under Assumption 1 (nonzero quantized rows/columns), Lemma 1 proves that projected closed-form scale updates exactly minimize each block of the frozen-Q^\hat Q objective over finite admissible inverse-scale sets. For fixed quantization grids, re-quantization is also an exact Q^\hat Q-block minimization (Lemma 2). Together, these ensure non-increasing MSE. Finiteness, Lemma 1's equality conditions, and deterministic rounding exclude nontrivial equal-MSE cycles, proving unconditional finite termination at a coordinate-wise stationary point for fixed grids, without compactness (Theorem 1(a)).

For adaptive grids (our implementation), changing internal quantizer scales prevents unconditional descent guarantees. Theorem 1(b) establishes descent and finite termination if Q^\hat Q remains constant at both re-quantizations of every subsequent iteration; stationarity is with respect to the frozen-Q^\hat Q objective. Proposition 1 establishes (i) eventual periodicity in general, (ii) eventual constancy of the internal grid as a sufficient condition for stabilization (hence freezing the block scales after a few iterations gives a fixed-grid run covered by Theorem 1(a)), and (iii) a runtime check: if one complete iteration leaves (α,β)(\alpha,\beta) unchanged, subsequent iterates and Q^\hat Q remain fixed.

Empirical behavior. Figures 2–3 show MSE plateauing and relative Q^\hat Q changes approaching zero by ~15 iterations on Qwen2.5-7B/Llama-3.1-8B (MXINT4), confirming the stabilization hypothesis of Theorem 1(b) in practice.

W2, Q2: Baselines, native deployment, and long generation.

Why SINQ+. Original SINQ primarily targets weight-only W4A16 (its only activation-quantized evaluation is W4A8). For a fair W4A4 comparison, we extend SINQ's column scales to activations (SINQ+, Appendix D). Without our extension, Appendix H (Table 13) compares against original SINQ in W4A16, where DualQuant achieves lower PPL in most settings.

Why QuaRot with RTN. DualQuant is calibration-free, so we compare against calibration-free baselines. QuaRot-RTN is QuaRot's own calibration-free variant; QuaRot-GPTQ is not calibration-free, since GPTQ estimates a Hessian from calibration data. GPTQ is also complementary; it could be applied on top of DualQuant's scales as an additional error-compensation step.

Stronger, calibrated baselines. We conducted additional evaluations on (1) accuracy and (2) end-to-end memory and throughput.

(1) Accuracy vs. NVIDIA ModelOpt. We evaluated ModelOpt v0.35.1 using its default NVFP4 max calibration (512 CNN/DailyMail samples), matching NVIDIA's Qwen3-14B-NVFP4 recipe. Both methods use identical source models, ModelOpt NVFP4 quantizer, and 16-bit KV cache (WikiText-2 / C4 PPL ↓):

MethodCalib.Qwen3-14BLlama-3.1-8BLlama-3.2-1BQwen3-0.6B
BF16–7.54 / 11.065.62 / 8.828.67 / 13.5318.14 / 27.30
ModelOpt512 samples7.77 / 13.526.62 / 10.9310.25 / 16.4322.64 / 38.04
DualQuantnone7.75 / 13.476.23 / 10.4410.02 / 16.0122.10 / 35.36

DualQuant achieves lower PPL in all eight cases without calibration data. PPLs differ from Table 1 because both methods here run inside ModelOpt's NVFP4 implementation; BF16 references are identical, so the evaluation pipeline is the same.

(2) Native NVFP4 W4A4 deployment. We extended packed INT4 inference to native NVFP4 W4A4 on RTX 5090 for Llama-3.1-8B (extended Table 3, new Table 6), achieving 2.53–2.63× memory reduction and 1.23–2.80× throughput gains versus BF16. Additional rebuttal-only Qwen3-14B deployment achieves 2.63–2.70× memory reduction, 1.87×/3.07× prefill and 1.46×/1.48× decode speedups (batch 1/16), with activation quantization, including β\beta scaling, accounting for only 1.9–4.2% of GPU time. (Please see the response to Reviewer Rz6P (Q2) for exact numbers/tables.)

Long-context behavior. DualQuant quantizes linear-layer weights and activations, while KV caches and attention remain 16-bit. Thus, longer contexts introduce no additional KV-cache storage quantization error; weight scales and the activation quantizations are sequence-length independent. Although quantization effects may still influence long-context accuracy, no additional cache precision reduction occurs as context grows. As a long-generation test (up to 32,764 tokens) thinking-mode AIME24 results are provided in new Table 11.

Reviewer_Rz6POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DualQuant, a calibration-free post-training quantization method that optimizes row and column scaling factors to minimize weight reconstruction error. It uses alternating closed-form scale updates with format-specific projection, and reuses the column scales to rescale activations before quantization. Experiments cover Llama and Qwen models across several quantization formats. The paper also implements packed INT4 W4A4 inference for Llama-3.1-8B on an RTX 4090, reporting close agreement with fake-quantization accuracy and memory and throughput improvements over BF16.

Critical strengths

The method provides a relatively simple, data-free optimization procedure for low-bit quantization. Reusing weight-derived column scales for activation quantization is an interesting observation, and avoiding calibration forward passes and Hessian computation could simplify the quantization procedure. The packed INT4 implementation also provides useful deployment validation beyond simulation. The integration of activation rescaling into the quantization kernel demonstrates that the proposed transformation can be implemented with limited additional runtime overhead.

Critical weaknesses
  • The evaluation does not include reasoning and code-generation benchmarks. The model suite includes Qwen3, but the reported evaluation consists of PPL and conventional zero-shot benchmarks rather than long-form reasoning or code generation. Improvements in WikiText-2/C4 perplexity do not fully establish that the method preserves model capability after quantization. Representative evaluations on AIME, LiveCodeBench, and GPQA-Diamond, etc., would be more informative. This could be investigated on the already included Qwen3-14B or a more recent model such as Qwen3.8-27B. Furthermore, Tables 2 and 8 do not include the BF16 accuracies, making absolute capability regression difficult to assess.

  • The practical accuracy-cost advantage over relevant alternatives is still unclear. The paper does not compare against a strong NVFP4 PTQ baseline such as NVIDIA Model Optimizer. Public ModelOpt checkpoints and evaluation results already exist, including Qwen3 families. The efficiency argument is also insufficiently quantified. The paper provides no corresponding offline quantization wall-clock or peak-memory measurements over AWQ and SmoothQuant. The packed INT4 throughput improvements over BF16 do not establish an advantage over competing W4A4 implementations or validate native NVFP4 deployment.

Questions for authors
  1. Can you provide reasoning or code-generation results, for example on AIME, GPQA Diamond, and LiveCodeBench, for an included Qwen3 model or another contemporary reasoning model? Comparisons should use the same prompts, thinking configuration, generation budgets, and sampling protocol for BF16 and quantized models.

  2. How does DualQuant compare with a well-configured NVIDIA Model Optimizer NVFP4 PTQ baseline on the same source model?

  3. What are the measured quantization wall-clock time and peak GPU memory relative to RTN, AWQ, SmoothQuant, or representative calibration-based methods on the same hardware? Please distinguish optimization/calibration time from model loading and packing, and report the iteration counts used.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Rz6P: Reasoning/Code Benchmarks, ModelOpt Comparison, Native NVFP4 Deployment, and Offline Cost

We thank the reviewer for the constructive suggestions. The revision adds reasoning/code benchmarks, native NVFP4 deployment, and offline costs; ModelOpt and Qwen3-14B deployment results are given below.

W1, Q1: Reasoning and code generation.

We added Qwen3-14B W4A4 evaluations on AIME24, GPQA-Diamond, and LiveCodeBench v6 (new Table 10), using matched prompts, chat templates, non-thinking mode, greedy decoding, and an 8,192-token limit, including BF16 references. Selected results (accuracy/pass@1, %):

FormatMethodAIME24GPQA-DLCB v6
BF16-33.350.026.9
MXINT4RTN20.042.428.0
SmoothQuant*26.743.427.4
DualQuant26.746.528.0
MXFP4RTN26.744.925.7
SmoothQuant*20.046.026.9
DualQuant20.046.026.9

Bold: best/tied-best calibration-free; *calibrated. Full results, including NVFP4 and AWQ+, are in Table 10. DualQuant leads/ties the calibration-free baseline in 8/9 settings. Thinking-mode AIME24 results are in new Table 11; BF16 references were added to Tables 2 and 9 (originally 2 and 8).

Q2: NVIDIA Model Optimizer (ModelOpt).

We evaluated ModelOpt v0.35.1 with its default NVFP4 max calibration (512 CNN/DailyMail samples), matching the format/dataset of NVIDIA's Qwen3-14B-NVFP4 checkpoint. Both methods use identical source models, environment, ModelOpt NVFP4 quantizer, and 16-bit KV cache; DualQuant is data-free. (WikiText-2 / C4 PPL ↓):

MethodCalib.Qwen3-14BLlama-3.1-8BLlama-3.2-1BQwen3-0.6B
BF16–7.54 / 11.065.62 / 8.828.67 / 13.5318.14 / 27.30
ModelOpt (default)512 samples7.77 / 13.526.62 / 10.9310.25 / 16.4322.64 / 38.04
DualQuantnone7.75 / 13.476.23 / 10.4410.02 / 16.0122.10 / 35.36

DualQuant has better PPL in all eight comparisons versus the tested default recipe. PPLs differ from Table 1 because both methods here run inside ModelOpt's NVFP4 implementation; BF16 references are identical, so the evaluation pipeline is the same, and DualQuant uses no data.

W2, Q2: Native NVFP4 deployment.

We added a packed NVFP4 W4A4 deployment of DualQuant on Llama-3.1-8B on RTX 5090 to the revised paper (Tables 3, 6, 15, and 16).

Additional Qwen3-14B results (rebuttal only), packed NVFP4 on RTX 5090. Memory breakdown (GB):

Weights (all workloads)BF16W4A4
Quantized linear weights26.426.61
FP8 group scales + β\beta–0.84
Embedding + lm_head (BF16)3.113.11
Runtime/allocator overhead–0.30
Weights subtotal29.5410.85

End-to-end (memory includes activations and 16-bit KV cache for decode; speedups are versus BF16):

WorkloadPeak memory BF16 → W4A4 (GB)ReductionSpeedup
Prefill b=129.64 → 10.962.70×1.87×
Prefill b=1630.02 → 11.342.65×3.07×
Decode b=129.65 → 10.972.70×1.46×
Decode b=1630.14 → 11.462.63×1.48×

Overall, DualQuant scales to Qwen3-14B with up to 3.07× speedup and 2.70× memory reduction over BF16 (Llama-3.1-8B: up to 2.80× / 2.63×, Table 3), while the activation quantization, including fused β\beta, uses only 1.9–4.2% of GPU time.

W2, Q3: Comparison with calibration-based methods

We run the comparison in two parts: (1) end-2-end accuracy, throughput, and memory and (2) offline quantization time and peak GPU memory.

(1): Native inference (new Table 6). On Llama-3.1-8B (RTX 5090), all methods share the Nunchaku/SVDQuant FP4 GEMM backend, differing only in per-channel scaling. At batch 16, prefill throughput is 36,118 / 36,134 / 36,352 tok/s, peak memory 6.53 / 6.51 / 6.53 GB, and decode throughput 1,263 / 1,269 / 1,263 tok/s for DualQuant / SmoothQuant / AWQ+; b=1 is similarly matched, while WikiText-2 PPL is 6.83 / 7.32 / 22.79. DualQuant thus improves accuracy without sacrificing packed-W4A4 efficiency.

(2): Offline cost (new Table 14). We measured calibration preparation and optimization; model loading and packing are excluded (Appendix I):

MethodCalibrationTimePeak GPU memoryOptimization
DualQuantnone41.0 s2.22 GB15 BCD iterations; 224 linears
SmoothQuant128 × 512 tokens60.3 s5.45 GB3 λ\lambda × 64 points
AWQ+128 × 512 tokens375.2 s21.85 GB20-point search; 224 linears

For SmoothQuant/AWQ+, data preparation takes 3.0/3.0 s, forward passes 11.8/12.3 s, and scale search 45.5/359.9 s, respectively. DualQuant requires 41.0 s for weight-only optimization, making it 1.47×/9.15× faster with lower peak memory. With the lowest perplexity too (Table 6), DualQuant offers the best accuracy–cost trade-off of the three.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.