acceptodds
Under review as a conference paper at ICLR 2027

AdaptQuant: Hardware-Aligned Per-Tile MX Format Selection for Post-Training Quantization on NVIDIA Blackwell

21 views

Abstract

Post-Training Quantization cuts the cost of serving Large Language Models, and NVIDIA Blackwell's native support for microscaling (MX) formats pushes the speedup further. Yet uniform MXFP4 still incurs substantial quantization error, and existing fixes assign higher-precision formats to sensitive regions using calibration data: a cost repeated for every new model before it can be served at all. We propose AdaptQuant, which assigns an MX element format per tile, the granularity at which Blackwell's block-scaled MMA reads its operands, from a single statistic computed in one pass over the tile. The rule mapping this statistic to a format is derived once, offline, on a single reference model against NVFP4 and reused unchanged on every other model. Across 13 open-weight LLMs on a single RTX 5090, AdaptQuant raises mean perplexity over BF16 by only on WikiText-2 and on C4 while spending fewer effective weight bits on average than the strongest mixed-precision baseline and quantizing faster, cutting stored weight memory by up to over BF16 and reaching BF16's prefill throughput.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Quantization, model acceleration, model compression, microscaling quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_w7dvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaptQuant, which selects MXFP4/6/8 per 16x32 tile using a statistic based on amax/RMS. The thresholds are derived once on a reference model and reused across models, avoiding per-model calibration while targeting Blackwell's block-scaled MMA granularity. Weight format selection is performed offline once per model, while activation format selection is performed dynamically during inference. Experiments on 13 LLMs report comparable accuracy to MicroMix while using slightly fewer effective weight bits, approximately 2.9x lower model quantization time, and similar overall inference throughput. The reported geometric mean prefill throughput is 1.8x over BF16.

Critical strengths
  • The 16x32 tile granularity matches Blackwell's block-scaled MMA operand shape, making the format-selection granularity well motivated.
  • The method uses a simple per-tile statistic and reuses thresholds across models rather than recalibrating for each model.
  • The paper details the hardware implementation including the storage, and conversion.
Critical weaknesses
  • Activation format selection requires computing both amax and RMS for every tile on every forward pass, but the paper does not isolate the runtime cost of this operation from quantization and GEMM.
  • AdaptQuant has limited throughput advantage over MicroMix. AdaptQuant's largest advantage is reducing one-time quantization time from 211.6s to 72.3s, while steady-state throughput is similar to MicroMix and sometimes lower. Given that quantization is typically a one-time offline cost, reducing quantization time from a few hundred seconds to roughly one minute is not the main bottleneck.
Questions for authors
  • For Table 1, what tensors/models are used to compute the reported SQNR values? Are they weights or activations, and are the values averaged across layers or models? *Could you provide a runtime breakdown of activation statistic computation, activation quantization, weight conversion, and GEMM?
  • Is my understanding correct that AdaptQuant stores weights in mixed FP4/FP6 formats but widens them to a uniform format (upcast MXFP4 to MXFP6) before the actual MMA? If so, could this distinction be made clearer? If the advantage is storage, a bigger LLM model, like MoE, would be a better use case.
  • Could you show accuracy versus effective-bit curves for MXFP4, MXFP6, AdaptQuant, MicroMix, and other baselines to demonstrate whether AdaptQuant lies on the Pareto frontier?
Reviewer_fPVgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes AdaptQuant, a mixed-precision post-training quantization (PTQ) method targeting NVIDIA Blackwell GPUs' native microscaling (MX) format support. The key observation is that existing mixed-precision MX methods (e.g., MicroMix, ARCQuant) pay a per-model calibration cost that grows with model size, before a single token can be served. AdaptQuant instead:

  1. Assigns an MX element format (MXFP4 vs. MXFP6) per 16×32 tile — exactly the operand granularity of Blackwell's block-scaled MMA instruction (mma.sync.kind::mxf8f6f4.block_scale) — using a single statistic stat(τ) = log₂(a_max(τ)/rms(τ)) computable in one pass over the tile (Sec. 4.2). The paper derives why this statistic predicts FP4 distortion: it tracks the range-loss term (values flushed to zero below the format's lower bound), not the format-invariant rounding term.
  2. Derives thresholds once, offline (Sec. 4.3, Algorithm 1): a rate–distortion program constrained to match NVFP4's per-layer distortion, solved exactly over a threshold grid using prefix sums, on a single reference model (Llama-3.1-8B, one 2048-token sequence). The resulting 14 numbers (per operand × projection type) are reused unchanged on all models — no calibration data needed for new models. The derived optimum sets t₂ = CAP, so FP8 is never selected.
  3. Resolves the hardware's "one format per operand" constraint (Sec. 4.4, Appendix C): mixed-width tensors are stored in per-format contiguous pools with a per-tile index; a byte-level cast kernel converts each operand to a uniform container once per layer on a side stream, so a stock CUTLASS block-scaled collective runs the GEMM. The conversion costs only 2.4–3.0% of the GEMM it feeds.

Results: Across 13 open-weight LLMs (3B–32B) on a single RTX 5090, AdaptQuant attains the lowest mean perplexity increase over BF16 of any method compared (5.0% WikiText-2 / 3.9% C4, vs. 5.6%/4.5% for MicroMix), matches MicroMix's downstream accuracy (98.6% zero-shot retention), quantizes 2.9× faster (mean 72.3s vs. 211.6s), stores smaller weights on 10 of 13 models (geomean 2.65× reduction, 5.17 vs. 5.33 effective bits), and reaches 1.81× BF16 prefill throughput (geomean), never slower than BF16 on any measured shape — while serving two 32B models that BF16 cannot load at all.

Critical strengths
  • S1 — Genuinely solves the calibration-cost pain point. The one-time threshold derivation replaces per-model calibration entirely: quantizing a new model requires only one pass over its weights with no text data. Figure 1 and Figure 4 demonstrate the cost scales with model size for all prior methods (MicroMix 2.9×, ARCQuant 3.1×, GPTQ 4.4×, AWQ 4.1× growth from 8B to 32B), while AdaptQuant's cost stays lowest and fastest on every model (2.18×–3.47× over MicroMix).
  • S2 — Principled, not purely heuristic, format-selection rule. The connection stat(τ) = −log₂ rms(z) (Eq. 7) ties a cheaply computable statistic to the specific range-loss mechanism of MX formats (cutoffs 1/24 for FP4 vs. 1/120 for FP6, derived in Appendix A), and the error attribution in Table 4 honestly verifies that stat tracks exactly the term that varies across tiles (+0.81 on activations, +0.59 on weights for the low-magnitude regime).
  • S3 — Hardware-aligned design with a clean interface solution. The tile granularity matches the MMA's native operand shape; the pool-storage + one-pass cast + side-stream prefetch design (two-event ordering, 3-buffer depth) lets a stock CUTLASS collective run unmodified. This is a realistic, deployable engineering contribution, and the 1.81× geomean prefill speedup with no configuration slower than BF16 (min 1.53×) substantiates it.
  • S4 — Accuracy matches calibrated methods at fewer bits. With zero per-model calibration, AdaptQuant achieves the lowest mean perplexity increase over BF16 among all compared methods, beats MicroMix on the largest models (Qwen2.5-32B/Qwen3-32B), and spends fewer effective weight bits (5.17 vs. 5.33, storing smaller weights on 10/13 models) — a favorable point on the rate–distortion trade-off.
  • S5 — Above-average experimental breadth and transparency. 13 models across 3 families, full per-configuration throughput tables (Table 8), OOM configurations explicitly marked, and an appendix that self-reports the weaknesses of the statistic (Table 4) rather than hiding them.
Critical weaknesses
  • W1 — No ablation on threshold generalization. Thresholds are derived on a single reference model (Llama-3.1-8B) over a single WikiText-2 sequence (Appendix B, lines 734–741). The paper never tests how thresholds drift when the reference model or corpus changes. The Llama-2-7B MMLU deficit vs. MicroMix (41.84 vs. 43.66, −1.82 points) suggests some model families are more sensitive to fixed thresholds, but this is neither explained nor quantified.
  • W2 — The "exact optimum" rests on an unproven sufficient ordering. Eq. 10 is solved exactly over the threshold grid, but the objective's quality depends on stat(τ) ordering tiles by total distortion. The authors' own Table 4 shows 88% of weight-side distortion comes from terms stat cannot change (rounding 74.4% + clamping 13.7%), with only +0.31 Spearman correlation with total error on weights. The optimality claim therefore applies to the search procedure, not the underlying selection problem.
  • W3 — Throughput geomean is computed over non-aligned configuration sets. Table 8 contains multiple "—" (did not complete) entries, notably Qwen2.5-32B and Qwen3-32B at large batch sizes for both methods. Figure 5's geomean is taken only over batch sizes completed by every method, systematically excluding large-model/large-batch samples, which weakens the 1.81× headline figure.
  • W4 — Single consumer-grade GPU; portability claims untested. The core selling point is alignment with Blackwell's native MX hardware, and the related-work section touts OCP MX as a cross-vendor standard — yet all results are on one RTX 5090 (sm_120). No datacenter GPU (B200) or non-NVIDIA MX-capable hardware (e.g., AMD MI300) result is provided, so the interoperability advantage over the NVIDIA-proprietary NVFP4 baseline remains an assertion.
  • W5 — Missing decode-phase evaluation and online-overhead breakdown. Only prefill throughput is reported (Fig. 5, Table 8); decode is often the serving bottleneck and is memory-bound, where the method's bandwidth savings could matter most (or where the non-contiguous mixed-pool layout could hurt). Additionally, the online activation-path cost (per-tile stat computation + encoding + conversion) is never separately measured, making it hard to explain the speedup variation across batch sizes (e.g., 1.57× at batch 1 vs. 1.53× at batch 32 on Qwen3-4B).
Questions for authors
  • Q1 (threshold generalization): If thresholds are re-derived using a different reference model (e.g., Mistral-7B) or a different corpus (e.g., C4), how much do the resulting (t₁, t₂) pairs and downstream perplexities drift? Can you explain the Llama-2-7B MMLU deficit relative to MicroMix in terms of threshold fit?
  • Q2 (FP6 share and speedup attribution): Using Table 7's W-bits and Eq. 3, the average effective bit width (5.17) implies roughly 46% of tiles are promoted to FP6 — far from a "small sensitive subset." Can you report the actual per-projection-type FP4/FP6 tile distribution, and correlate FP6 share with the per-model/per-batch-size speedup variation, to attribute the 1.81× geomean?
  • Q3 (hardware breadth and decode): Can you reproduce prefill results on a datacenter Blackwell GPU (B200), and can you report decode throughput (at least single-batch) to show whether the mixed-pool storage layout and per-layer conversion remain profitable in the memory-bound regime?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.