acceptodds
Under review as a conference paper at ICLR 2027

AWSLQ: Activation-Wise Sparse-Ladder Quantization for Accurate W4A4 Single-Load Decoding

51 views

Abstract

On FP4-native GPUs, deployed serving stacks already run W4A4 — FP4 weights times FP4 activations — at decode, yet the quantization literature retreats to 16-bit decode because W4A4 is where accuracy collapses. Small-batch decode is bound by streaming the weights, so its tensor cores sit mostly idle. AWSLQ, a training-free W4A4 scheme that requires no external calibration corpus, spends that idle compute: it multiplies each weight tile by several FP4 activation terms instead of one, here a ladder of N structured-sparse NVFP4 operands, peeled online, that share one weight load. Two-operand ablations had concluded that more sparse computation does not recover the loss; the knee is at N=3, and by N≈4 the ladder is on the W4A16 floor, to within +0.005 perplexity on ten of eleven models. A selective reverse-smoothing fold, in the direction SmoothQuant advises against, lowers the weight-only floor itself; the ladder absorbs the activation outliers the fold creates and lands on the lowered floor, while a single dense term stays above it; whether to fold is decided per model from its own generated text at preparation time. A fused Blackwell kernel runs the ladder as one M-stacked sparse GEMM into a single accumulator and beats cuBLASLt's FP4 GEMM across a Llama-3.1-8B decode block at M=16–64 (1.26× to 1.07× on an RTX 5090, 1.08× at M=16 on a B200). AWSLQ matches or beats every published W4A4 method, calibrated or not, on 16 of the 18 model–dataset cells with a published comparator, at coarser block-32 scales, and on GSM8K it matches the accuracy of retreating to 16-bit decode activations. The accuracy comes from the number of FP4 terms that share one weight load. A sparse rung performs half the multiply-accumulates of a dense term, so the ladder also offers an intermediate arithmetic budget, three rungs at one and a half dense terms, which chooses the refined half of each token's channels online.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Post-training quantization, Efficient LLM inference, Activation quantization, Autoregressive decoding, Sparse MMA, Sparse tensor cores, NVFP4, GPU kernels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8jsHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AWSLQ, a training-free W4A4 quantization method for small-batch LLM decoding. Instead of representing each activation with a single dense FP4 tensor, AWSLQ decomposes it online into several structured-sparse FP4 terms that share one weight load. A selective reverse-smoothing transform improves weight quantization, while a self-probe based on model-generated text decides whether to apply the transform without external calibration data. The authors also develop an M-stacked Blackwell kernel that executes the sparse terms within one GEMM. Experiments across Llama, Qwen, and Gemma models show that four sparse terms approach the W4A16 accuracy floor and are competitive with existing W4A4 methods.

Critical strengths

Hardware-aligned sparse residual decomposition. AWSLQ decomposes activations into several structured-sparse FP4 residual terms that reuse the same weight loading, providing a neat way to use otherwise idle compute to improve W4A4 accuracy during small-batch decoding.

Evaluation. The paper implements the sparse ladder on both RTX 5090 and B200 GPUs and includes activation preparation in the latency measurements. The results show that the method can improve W4A4 accuracy without giving up small-batch decode speed.

Critical weaknesses

Limited end-to-end performance validation. The speed evaluation covers individual GEMMs and a chained group of linear layers, but not complete model serving. Since the gain on B200 drops to only 2–6% against a similarly launched vendor baseline, it is unclear how much of the improvement would remain in end-to-end token throughput.

Unclear advantage over dense multi-term quantization. SL4 and two dense residual terms achieve similar perplexity and small-batch latency, but the paper does not compare them on downstream tasks or end-to-end performance. It is therefore unclear how much of the gain comes from structured sparsity instead of simply using multiple activation terms with M-stacking.

Questions for authors
  • Could the authors report full-model decoding speed, including activation processing, attention, KV-cache operations, and runtime overhead?

  • Could the authors provide more ablation / analysis on the gain comes from structured sparsity?

  • Figure 7(b) is somewhat confusing because the bars show latency, while the annotations report inverse latency ratios as speedups.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8jsH: end-to-end speed, the sparsity ablation, Figure 7(b)

Thank you for the review. Table and figure numbers are those of the revised PDF.

End-to-end speed (W1, Q1). The revision reports full-model decoding (Figure 5b, Table 24, Appendix D.8): a full Llama-3.1-8B decode loop with activation preparation, attention, the KV cache and every launch inside the measurement.

  • Batch 16, context 512: SL4 takes 7.18 ms per output token against 7.88 ms for the vendor FP4 path, 1.10× faster; 1.04× with transformers' stock attention path.
  • Batch 32: 1.07× faster. Batch 64: the two are level.
  • The linear layers are 55 to 58% of a decode step, so the block's 1.30× becomes 1.10× per token.
  • B200: your concern is justified. With the exact per-token scale the ladder is 1.10× slower than the vendor block at batch 16, and the revision scopes the speed claim to the RTX 5090 (Sections 6 and 7).

What structured sparsity contributes (W2, Q2). The revision adds the ablation you ask for, in accuracy and in time.

  • Accuracy. Sparsity adds a half term whose channels each token chooses. With the same weights, the same first term and the same arithmetic, the per-token half is more accurate than a half fixed in advance in all 24 cells on block-32 weights and all 42 cells on block-16 weights (Tables 1 and 7; our comment to all reviewers gives the design).
  • Time. Per GEMM, one sparse term is 1.17× faster than one dense term on the same kernel, and SL4 is 1.16× faster than a stacked dense pair at M=16 (Table 5). With preparation included the decode block is level through M=64 (Table 20b), so today our speed advantage is over the vendor's kernel.
  • At two terms the accuracy comes from the number of terms, as the submission states. On GSM8K, SL4 and the dense pair tie on the same block-32 weights on all three models, with and without the fold (Table 3); on block-16 weights the pair ties on two models and leads on Qwen3-14B. We time the linear layers of the stacked pair, since that is the only part of a decode step where the two differ.

Figure 7(b) of the submission, now Figure 5(a) (Q3). The labels were the vendor's latency divided by ours, printed over latency bars without saying so. They now read "1.30× faster" or "1.29× slower", and the caption defines them.

Thank you for reading the new experiments, and for confirming that the end-to-end results address your first concern.

On SL4 and the dense pair we agree, and that tie is a ceiling. At two terms both sit on W4A16, the best any activation quantizer can reach on those weights: SL4 is within 0.01 perplexity of it on all twelve models (Table 1), and two dense terms are level with it (Tables 6 and 7). No gap is left there for either form to win.

What the sparse path adds is per-token channel selection. A sparse term refines half of the channels, and each token chooses its half; dense terms have no such choice. That can only show below the ceiling, at one and a half terms, where a gap remains. There, with the same weights, the same first term and the same arithmetic:

  • SL3 is closer to W4A16 than the dense configuration in all 24 cells of twelve models (Table 1), and in all 42 cells on block-16 weights (Table 7, simulated);
  • on the six-model task suite SL3 is within 0.3 points of W4A16 and leads the dense configuration by 0.3 to 1.0 points, significantly on four models (Table 8).

In practice, one and a half terms of sparse arithmetic are enough for W4A16 task accuracy, and the sparse path gives nothing back at two terms: per GEMM SL4 is 1.16× faster than the stacked dense pair (Table 5), and with preparation included the block is level (Table 20b). On the RTX 5090 at decode batch sizes the saved quarter of the arithmetic buys no time yet; Section 7 says so and lists what would price it.

We hope the comparison at one and a half terms speaks to your second question, on what structured sparsity contributes: its two arms differ only in the per-token selection. We would be glad to hear whether it does.

Reviewer_CNDgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes AWSLQ, an activation quantization method for LLM inference on Blackwell GPUs. It represents 16-bit activations using multiple structured-sparse FP4 terms obtained through iterative residual quantization. The authors design a fused GEMM kernel to compute the resulting linear layers with NVFP4 weights. The method is evaluated on multiple LLMs and downstream tasks for quantization accuracy, and its latency is benchmarked on Blackwell GPUs against several baselines.

Critical strengths
  • Accurate and efficient low-precision inference is an important problem for practical LLM deployment.
  • AWSLQ demonstrates a favorable accuracy-latency trade-off against the evaluated baselines in the targeted small-batch decoding regime.
  • The source code and raw evaluation logs are provided as supplementary material, improving the transparency and reproducibility of the experiments.
Critical weaknesses

W1. Poor presentation and organization.

The paper is difficult to follow. The terminology (e.g., ladder, rung, fold) is not always introduced or explained clearly, and Figures 2-6 are difficult to interpret. Sections 3 and 4 are particularly challenging to understand without repeatedly consulting later sections or the appendix. Several important experimental results and implementation details are relegated to the appendix, even though they are essential for evaluating the main contributions. For example, Section 4 discusses the sequentially fused kernel extensively, while the main performance results use the M-stacked kernel, and the performance comparison between these implementations is mainly provided in the appendix. The main paper would benefit substantially from a more coherent and self-contained presentation.

W2. Insufficient motivation for structured-sparse multi-term FP4 quantization over simpler alternatives.

The benefits of using structured-sparse multi-term FP4 quantization instead of simpler activation representations are not sufficiently established. The paper acknowledges that four sparse terms (SL4) and two dense FP4 residual terms achieve nearly identical perplexity and small-batch latency (Section 3.2, around Line 215). The main distinctive advantage of the sparse representation therefore appears to be the intermediate arithmetic budget offered by three sparse terms (SL3). However, SL3 and SL4 have almost identical measured latency (Figure 7), suggesting that this theoretical advantage does not translate into meaningful latency savings in the evaluated regime.

Furthermore, a more straightforward alternative to multiple FP4 terms is to increase the activation precision, for example, using FP8 activations. Although Appendix C.5 includes a W4A8 baseline, it uses a different weight format (MXFP4 rather than NVFP4), making the comparison difficult to interpret. A controlled comparison with higher-precision activation quantization would help justify the additional complexity of the proposed representation.

W3. The latency evaluation does not sufficiently isolate the benefits of the proposed quantization format.

Figure 1 shows that the dense NVFP4 implementation using cuBLASLt is slower than the W4A16 Marlin kernel at small batch sizes (up to approximately 16), despite using lower-precision Tensor Core computation. This raises concerns about whether the dense NVFP4 baseline is sufficiently optimized for the targeted operating regime.

Moreover, Marlin was originally optimized for earlier GPU architectures and uses INT4 weight storage with 16-bit computation. Although the authors port it to Blackwell, the resulting implementation is not necessarily an optimized W4A16 baseline for this architecture. Overall, the latency comparisons do not convincingly distinguish the benefits of the proposed sparse representation from those of the specific kernel implementations.

W4. Formatting and readability issues.

The paper appears to use a different font from the one produced by the standard ICLR template, resulting in noticeable typographical differences, particularly in the headings, and slightly increased text density. In addition, some tables and figures contain text that is too small to read comfortably. The authors should ensure that the submission follows the official formatting requirements and improve the readability of the figures and tables.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer CNDg: presentation, alternatives, baselines, formatting

Thank you for the review. Table and figure numbers are those of the revised PDF; the terms are defined in our comment to all reviewers.

W1, presentation. We rewrote the parts you name.

  • Section 3 opens by defining the four terms the method uses (rung, ladder, floor, fold) and explains the NVFP4 format and the two tensor-core instructions before either is used, including why the sparse instruction implies block-32 weights.
  • Section 4 leads with the M-stacked kernel, which produces every decode result; Figure 4 shows that kernel first, as panel (a), and the fused kernel for full tiles as panel (b).
  • The results needed to judge the claims are in the main text: the single-operand control (Figure 5a), end-to-end time per token (Figure 5b), GSM8K with the dense pair (Table 3), the half step on twelve models (Table 1), RULER (Table 4) and the per-GEMM latencies of the stacked and fused kernels with the single-operand controls (Table 5).
  • Figures 2, 3 and 4 are redrawn, and the captions of Figures 2 to 5 can be read on their own.

W2, simpler alternatives. Both are now controls in the paper.

  • Dense terms. Our comment to all reviewers gives the controlled experiment: at the same arithmetic, the half term whose channels each token chooses is closer to W4A16 than a fixed half in all 24 cells of Table 1 and all 42 cells of Table 7.
  • FP8. The comparison is on NVFP4 weights, as you ask (Tables 1 and 7). One FP8 term costs the tensor time of two FP4 terms, and at that cost two FP4 terms (SL4 or the dense pair) are closer to W4A16 on every model. SL3 matches FP8 at three quarters of that tensor time: within 0.03 perplexity in all 24 cells and within 0.3 points on a six-model task suite (Table 8).
  • SL3 against SL4. You are right that they have the same latency at decode batch sizes (Figure 7(b) of the submission, now Figure 5(a)): both fit in the rows the batch leaves empty, so the fourth term is free and SL4 is our operating point. Against dense terms, the sparse form is the faster one per GEMM (SL4 is 1.16× faster than a stacked dense pair at M=16, Table 5) and level on the decode block once preparation is included (Table 20b). Section 7 states this scope.

W3, latency baselines. The revision separates the kernel from the format, as you ask, with two single-operand columns added to Table 5.

  • The dense baseline. On our own stacked kernel, one dense NVFP4 term is 1.29× faster than cuBLASLt at M=16, so you are right that the vendor kernel leaves time unused at small batch. On that same kernel one sparse term is 1.17× faster than the dense term: with the kernel held fixed, the sparse format is the faster one per GEMM.
  • W4A16 at small batch. All paths move the same weight bytes at M≤16 and none is limited by tensor-core arithmetic, which is why 4-bit activations buy no time over 16-bit ones there.
  • Marlin. It is the fastest published W4A16 kernel, and the paper discloses the two fixes its port needed. The revision adds a lower bound that holds for any W4A16 kernel (Appendix D.5): at M=128 that arithmetic alone, with no time allowed for memory, takes longer than our measured FP4 GEMMs on all five shapes, and at M=32 to 64 Marlin is 1.1 to 4.0× above it, so a better W4A16 kernel could narrow the gap there. End to end, Marlin is 1.03× faster at batch 16 (7.00 against 7.18 ms per token), and SL4 is 1.05× and 1.15× faster at batch 32 and 64 (Table 24).

W4, formatting. The submission was compiled with XeTeX, which substituted TeX Gyre Termes for the template's Times. The revision is compiled with pdfLaTeX and the template's fonts, and its main text is ten pages. New tables are set without scaling, the saturation table and the two perplexity ledgers are reset at larger type, and Figures 5 and 9 are redrawn with larger text.

The authors have posted their rebuttal replies, but they are not visible to the public (not selecting readers "Everyone"). The authors' confidence in their work is questionable, and the reliability of their work is unclear.

Thank you for the detailed reply.

My main concern (Weakness 2) about the practical benefit of structured sparsity remains unresolved. Four sparse rungs (SL4) and two dense FP4 residual terms achieve nearly identical accuracy and latency when including activation preparation (Section 3.2 Lines 212-216, Section 6 Lines 455-461, Table 20, etc.). The authors also acknowledge that the baseline kernels are not optimal for small batches and for the tested GPUs (Weakness 3), making the necessity of sparse rungs less convincing.

The changes in the revised PDF are not highlighted, making it difficult to verify the revisions.

Reviewer_ckrhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets W4A4 (NVFP4 weights × NVFP4 activations) at small-batch decode on Blackwell GPUs. At this batch size the GEMM is bound by streaming the weights, so the tensor cores are mostly idle. AWSLQ spends that idle compute on extra activation terms. Each activation tile is peeled online into a ladder of N 4:8 structured-sparse NVFP4 operands: each rung keeps the two largest of every four adjacent pairs of the running residual and quantizes them under its own block scale. All rungs multiply the same weight tile. No calibration data is used. Contributions and validation:

  1. Saturation. The ladder converges to the W4A16 floor by N≈4: within +0.02 PPL on all 11 models and within +0.005 on 10 (Table 4), and also on a GPTQ weight track (Table 6). A lemma (App. A.2) shows that two greedy rungs have the same worst-case error bound as one dense term. Together with the coarser block-32 scale that the sparse instruction imposes, this explains why earlier two-operand studies saw no gain over dense NVFP4.
  2. Selective reverse fold. A channel-norm rescaling in SmoothQuant's α=0 direction, applied to up, down, o and v, lowers the weight-only floor. The ladder absorbs the activation outliers that the fold creates. A preparation-time probe on text generated by the model itself decides per model whether to fold.
  3. Kernel. An M-stacked CUTLASS kernel places the rungs in the rows a decode tile leaves empty, with bit-exact output. On an RTX 5090 it runs a Llama-3.1-8B decode block 1.26× to 1.07× faster than cuBLASLt FP4 at M=16–64, and 1.88–2.36× faster than Marlin W4A16 at M=32–64. On a B200 the gain is 1.08× at M=16.
  4. Accuracy. Without calibration and at block-32 scales, it matches or beats every published W4A4 method on 16 of 18 WikiText-2/C4 cells. With dense NVFP4 prefill and ladder decode, GSM8K accuracy is within 0.1 point of 16-bit decode activations on three models. Zero-shot, MMLU, MATH-500 and RULER are also reported.
Critical strengths

S1. A correct diagnosis of why multi-term sparse activations appeared not to help, backed by strong controls. The two-rung lemma (App. A.2) shows that one dense NVFP4 term and two greedy sparse rungs both bound every element of a block within 17m/96. The bound is tight: the largest measured ratio is 0.1764 against 0.1771. On average, two rungs are somewhat better than one dense term on the same block-32 weights (Qwen3-8B 10.0646 vs 10.1224; Llama-3.2-3B 8.6471 vs 8.8388; Tables 4 and 7), but no better than the block-16 dense NVFP4 that is deployed (10.0537 and 8.6120; Table 13). The lemma and the block-32 scale together explain SharQ's result that two sparse operands do not improve on one dense term (Section 2): prior work stopped at the depth where, against the deployed format, the second sparse term buys nothing. The paper then shows that the floor is set by the number of terms that share one weight load. The ladder saturates on 11 models and on a GPTQ track, and a dense residual pair at block 32 lands within 0.008 PPL of SL4 (Table 4).

The same reasoning makes the fold work. Under SmoothQuant's α=0 transform on Llama-3.2-3B, dense NVFP4 sits 1.51 PPL above that transform's weight-only floor, while SL4 sits 0.05 above it (Table 7). With the channel-norm fold, the ladder follows the lowered floor to within 0.003 PPL, while one dense term keeps a gap of 0.19–0.48. The controls are strong: no calibrated SmoothQuant α beats the zero-calibration fold beyond the 0.02 tie rule, and a learned transform (OmniQuant LET) regresses every arm.

S2. Accuracy and speed are measured on the same mechanism, under conditions that work against the method. M-stacking makes an extra rung almost free at decode batch sizes. At M≤32, SL3 and SL4 are within about one timer tick on every shape (Table 15), and SL4 is within 2% of SL3 on the chained block. On the RTX 5090 the ladder is 1.07–1.26× faster than cuBLASLt FP4 at M=16–64 and 1.88–2.36× faster than Marlin W4A16 at M=32–64. On GSM8K it is within 0.1 point of 16-bit decode activations on all three models.

The comparisons work against the method. Table 1 pits block-32, calibration-free AWSLQ against block-16 baselines, two of which (RaZeR, ARCQuant) are calibrated. Latency is measured with cold L2, and the vendor block uses the cheaper static per-tensor scale while the ladder computes the exact per-token scale. The authors also disclose the two fixes they had to make to Marlin on sm_120.

Critical weaknesses

W1. The sparse form is motivated by speed, but that benefit is only partly realized in the evaluated configurations, while its cost is certain. The elements that set AWSLQ apart from prior work are not the ones that produce its gains.

  • Motivation, and where it is realized. A sparse rung performs half the multiply-accumulates of a dense term (§3.2), so sparsity should make each extra term cheaper. In the evaluated configurations:
    • SL4, the operating point of the accuracy tables, has the same arithmetic budget as a dense residual pair (4 × 0.5 = 2 terms), so it saves no arithmetic. Only SL3 (1.5 terms) does.
    • At decode M≤32 arithmetic is free: SL3 is within 2% of SL4, and a matched dense build (two terms on 2M rows) is within 3% of SL4 on the Llama-3.1-8B block (§4).
    • At compute-bound shapes (App. D.7, M=4096–8192, fused kernel), three rungs cost 0.87–0.90 of a dense pair (0.75 ideal) and four rungs 1.40–1.50 (1.00 ideal), because the stock sparse mainloop keeps the tensor pipe active 50% of the time against 76% for the dense one.
    • At M=64–128, where arithmetic starts to matter during decode, no dense-pair latency is reported, and from M≈96 the dispatch returns single-term dense NVFP4.
    • On B200 the block is bound by launches and preparation; the third rung costs 0.06 µs per GEMM (App. D.4).
  • The cost. The sparse instruction carries only four scale factors, which forces block-32 weights (App. D.6). At block 32 a dense pair ties SL4 (6.6844 vs 6.6843 on Llama-3.1-8B; within 0.008 PPL on all three models tested). At block 16, which only the dense instruction allows, the dense pair is lower on Llama-3.1-8B (6.6275) and Qwen3-8B (9.9238 vs 9.9488), though not on Gemma 4 (10.3208 vs 10.1711).
  • What depends on sparsity. The weight-only floor (Table 4), the fold (two dense terms follow the lowered floor to within 0.002 PPL of four rungs, §3.3) and M-stacking ("indifferent to the operand's structure", §4) depend on the number of terms, not their structure. Only the two-rung lemma, the half step (SL3 leads fixed-half dense configurations on 6 of 8 models by up to 0.07 PPL, Table 5) and the block-32 constraint are specific to sparsity. The dense pair is not compared at task level (Section 7).

W2. The headline deployment claim rests on kernel-level and narrow task-level evidence.

  • Scope of the speed measurements. Only the linear layers are accelerated and measured (one chained block of four projections). There is no end-to-end tokens/s or time-per-output-token in a serving stack, so attention, KV-cache reads and launch overhead are not counted.
  • Datacenter GPU. On B200, the 1.08× gain at M=16 is for SL3 only. SL4, the configuration used in the accuracy tables, ties the vendor. The SL3 arm reuses the previous step's per-tensor activation scale; a variant using the current step's scale trails the vendor by 8%, and a barrier-free variant by 5% (App. D.4). There is no W4A16 baseline on B200. On the datacenter part, the fast configuration and the accurate configuration are therefore not the same.
  • Attribution of the speedup. Table 15 has no single-operand (N=1) column for the stacked kernel on sm_120. On B200, the single-operand mainloop already beats the vendor on three of five shapes, so part of the lead over cuBLASLt may come from the authors' kernel itself rather than from the ladder.
  • Decode-only accuracy. The paper's motivation is decode-specific, but dense prefill plus ladder decode is evaluated only on GSM8K (3 models), MATH-500 (2) and RULER (1). The paper gives 1.28 points as the standard error of a difference between two arms on the full GSM8K test set (App. C.2, Qwen3-8B). Against that, the ladder separates from dense NVFP4 only on Qwen3-8B (87.41 vs 83.85). On Qwen3-14B (86.66 vs 85.90) and Qwen2.5-32B (89.99 vs 89.31) the difference is within noise. So is the 16-bit-decode retreat, which on Qwen3-14B (85.67) is below dense.
Questions for authors
  1. Single-operand baseline (W2). On sm_120, what is the latency of your stacked kernel with a single operand (N=1, dense and sparse) for the Table 15 shapes at M=16–64? This would separate the gain from M-stacked rungs from the gain of your mainloop over cuBLASLt's kernels.
  2. Sparse ladder vs. M-stacked dense pair (W1). Which weight block does the matched dense build ("two terms on 2M rows", within 3% of SL4 at M≤32) use? Please report stacked SL3 and SL4 against a stacked dense residual pair with block-16 weights at M=16–128: latency including preparation (the dense pair needs no pair selection or 4:8 metadata) and GSM8K accuracy. This would show what the sparse realization contributes in accuracy and in latency, and from which M the half step's arithmetic saving turns into time.
  3. B200 configuration (W2). Please report SL4 with the exact per-token scale against the vendor block with its quantizer issued as a programmatic dependent launch (75–78 µs, App. D.6), and add a W4A16 baseline on B200. That way a single configuration would carry both the accuracy claim and the speed claim.
  4. MATH-500 claim. The introduction says the ladder beats uniform dense NVFP4 on MATH-500 on both models. This holds under greedy decoding with math_verify, but not in the other settings you report. Under sampling on Qwen3-8B, fold+SL4 is 48.1 ± 1.0 against dense 48.3 ± 0.8. Under strict extraction, both ladder arms on Qwen3-8B are below dense (40.2 and 38.6 vs 41.4, Table 11). Please qualify the claim, and consider adding a paired significance test for ladder vs dense on GSM8K for Qwen3-14B and Qwen2.5-32B.
  5. End-to-end throughput. Report decode throughput (tokens/s and time per output token) in a serving stack at M=16–64, including the prepare kernel and attention.
  6. MoE. The introduction motivates the work with MoE FP4 deployments, but no MoE model is evaluated; App. A.7 surveys weight statistics only. With the fold on the up projection, each expert has its own prescale s, so a token's hidden state would seem to need a separate peel for every routed expert. Without the fold, one peel of the shared input could serve all of them. How do the prepare cost and the grouped-GEMM latency scale with the number of active experts?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ckrh (1/2): W1 and Q1–Q3

Thank you for the careful reading. Table and figure numbers are those of the revised PDF.

W1, what the sparse form contributes. The controlled experiment is in our comment to all reviewers. You are right that Table 5 of the submission mixed the weight format with the channel selection; Tables 1 and 7 of the revision change only the selection, on block-32 and on block-16 weights. On speed, Q1 and Q2 below give what the sparse form contributes per GEMM and where the dense pair wins it back. At compute-bound shapes, where you note the stock sparse mainloop's lower pipe occupancy, the revision re-measures the fused kernels (Appendix D.7): three rungs run 1.2 to 1.3× faster than a dense pair fused the same way, and four rungs are within 6% of it.

Q1, single-operand baseline. Added to the per-GEMM tables (Table 15 of the submission, now Tables 5 and 19). At M=16 on the RTX 5090, as geometric means over the five shapes:

  • our stacked kernel with one dense operand is 1.29× faster than cuBLASLt;
  • with one sparse operand it is 1.51× faster.

So you are right: most of the lead over cuBLASLt is the kernel's own. The sparse form adds two things. One sparse term is 1.17× faster than one dense term on the same kernel. And further sparse terms are nearly free: on the chained block with preparation, one sparse term takes 106.5 µs, SL3 109.1 µs and SL4 110.6 µs, against 143.4 µs for cuBLASLt (Table 20, Figure 5a). The extra terms add 2.6 and 4.1 µs to the block, all of it preparation, and no GEMM time through M=32; they start to cost at M=64.

Q2, the stacked dense pair. The matched dense build of the submission used block-16 weights and had no preparation of its own. The revision adds what you ask for: a stacked dense pair on block-16 weights with its own preparation, at M=16 to 128 (Tables 5, 19 and 20b).

  • Latency. With preparation included, the pair, SL3 and SL4 run the block level through M=64, within 3 µs of each other, and the pair leads at M=128. At the batch sizes we claim, our speed advantage today is over the vendor's kernel, not over two dense terms. Per GEMM the ladder is the faster of the two, by 1.16× at M=16; the pair wins that back in preparation (10.9 µs per block against 21.2 µs), since it selects no pairs. The arithmetic advantage grows with M: at M=256, outside our operating point, it outweighs the preparation and SL3 runs the block 1.10× faster than the fastest pair (Table 20b).
  • GSM8K (Table 3, tests paired over questions). On the same block-32 weights the pair and the ladder tie on all three models, with and without the fold. On block-16 weights the pair ties on two models and leads on Qwen3-14B by 2.4 to 3.3 points with the fold. That lead follows the weight format: block-16 weights are 0.16 perplexity better on this model, and the ladder runs on block-32 weights.

Q3, B200. We re-measured the B200 in one session with the vendor block, the ladder and W4A16 side by side (Section 6, Appendix D.4).

  • With the exact per-token scale, SL4 takes 84 µs at M=16 against 76 µs for the vendor block with its quantizer issued as a programmatic dependent launch: 1.10× slower.
  • With the previous token's scale, SL3 is 1.03× faster than that vendor block.
  • So on the B200 one configuration does not carry both claims; the revision says so, and the abstract no longer quotes a B200 speedup.
  • W4A16 on the B200: Marlin is slower than cuBLASLt's FP4 GEMM in all 23 measured cells (1.24× at M=16, 2.5× at M=64). Against Marlin, SL4 with the exact scale is level at M=16 and 1.36× and 1.45× faster at M=32 and 64.
AuthorsOpenReview ↗

Response to Reviewer ckrh (2/2): Q4–Q6

(Continued from our first comment.)

Q4, MATH-500 and paired tests.

  • MATH-500. The introduction no longer makes the claim, and the bold is removed (Table 11 of the submission, now Table 14). Section 5.2 states what holds: under the equivalence scorer at greedy decoding both ladder arms are above dense NVFP4 on both models, but on Qwen3-8B every difference is inside one standard error, and strict extraction reads both arms below dense.
  • GSM8K, paired. We re-ran every arm of a model on one GPU type with per-question logs and paired the tests (Table 3, Appendix C.2). Every deployed ladder row is within 1.6 points of W4A16 decode on all three models, and no difference is significant. Against dense NVFP4, SL4 is +2.3 points on Qwen3-14B (p = 0.03) and +2.1 on Qwen2.5-32B (p = 0.005) under flexible extraction, and not significant under strict extraction; part of the flexible gap is a scoring effect (Appendix C.2).
  • More power. On identical block-32 weights, where only the activation side differs, we also ran the same arms on the 7,473-question GSM8K training split (Table 8, Appendix C.2): SL3 leads the fixed-half dense configuration of the same arithmetic by 1.6 points on Llama-3.1-8B and 0.8 on Qwen3-8B under both extraction rules (p ≤ 0.014) and stays within 0.7 points of W4A16 decode (p ≥ 0.07). Only the paired comparison between arms on the same split is claimed.
  • Breadth. Decode-only accuracy, dense prefill with the ladder at decode, now covers GSM8K on seven models (the n=250 grid of Table 13) and on the full test set for three, the training split for two, MATH-500 for two, RULER for one, and the mixture-of-experts model.

Q5, end-to-end throughput. Added (Figure 5b, Table 24, Appendix D.8): a full Llama-3.1-8B decode loop, with attention, the KV cache, our preparation and every launch inside the measurement.

  • Batch 16, context 512: SL4 takes 7.18 ms per output token (2,229 tokens/s) against 7.88 ms (2,030 tokens/s) for the vendor FP4 path, 1.10× faster; 1.04× with transformers' stock attention path.
  • Batch 32: 1.07× faster. Batch 64: the two are level.
  • The linear layers are 55 to 58% of a step, which is why the block's 1.30× becomes 1.10×.
  • The fold needs a second preparation at two sites of each block, and the folded configuration is 1.03× slower than the vendor end to end; the speed claim is the plain ladder's.

Q6, MoE. We evaluated Qwen3-30B-A3B, which has 128 experts per layer and routes 8 per token (Appendix C.6, Table 18).

  • Accuracy. The plain ladder behaves as on dense models: SL4 is on the W4A16 perplexity of its weights and within 0.3 GSM8K points of W4A16 decode. The fold loses accuracy on this model (0.10 perplexity and 4.6 GSM8K points) although the self-probe selects it, so we run the plain ladder on MoE models and the paper states this exception.
  • Prepare cost. Your reading is right: with the fold on the up projection each routed expert needs its own peel of the shared input, 24,576 elements per token and layer against 8,192 without it.
  • GEMM cost. Each active expert is one weight load: 53 experts are active at M=16 and 85 at M=128, so an expert's GEMM sees 2 to 12 rows on average. We timed single GEMMs of the three expert shapes at 1 to 16 rows (Table 18, lower panel): SL4 takes 6.1 to 6.7 µs, the same as a single sparse term, and at 8 and 16 rows, the smallest sizes cuBLASLt's FP4 GEMM runs at these shapes, it is 1.34 to 1.55× faster than cuBLASLt.
  • The plain ladder needs no calibration data, which is convenient for MoE models: in a standard 128-sequence calibration set, 13% of this model's experts see fewer than 64 tokens.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Initial Author Response: Sparsity, Performance Attribution, and Planned Revisions

We thank the reviewers for careful readings.

Regarding the question of whether sparsity is at the center of this work. Our starting point was a single mechanism that serves the whole of inference with one knob: the number of FP4 terms placed on one weight load, where the structured-sparse instruction lets that number move in half steps and lets every token choose where its half step goes.

The accuracy limit of this family is set by the number of terms, as the submission states, so a dense residual pair reaches the same limit at two terms; we report that pair beside the ladder, at task and kernel level, including where it leads. Dense can also spend one and a half terms, by giving a second term to a half of the channels fixed in advance, and it comes close. On identical block-32 weights, across the models of Table 5 and both datasets, SL3 is closer to W4A16 than that dense configuration, paired over evaluation windows, and is level with FP8 per token; at task level it ties W4A16 and FP8 per token and leads the dense configuration on the models tested.

That is what the per-token choice buys at equal arithmetic, and it is the sparse form's own contribution: one knob, finer steps, and the better operating point at one and a half terms of arithmetic budget. The dense form needs a block size, a residual half and a per-layer rule chosen in advance to approach it.

Regarding whether the half step's arithmetic saving becomes time. On current GPUs it does not reach decode latency, and the revision will say so. The rungs add no GEMM time at decode; the cost is the preparation that selects and packs the kept pairs in software. The revision will report where the time goes, on the 5090 and on a B200-class part, where the saving is measured, which is where arithmetic rather than weight traffic sets the time, and what hardware would have to provide for it to appear at decode.

On attribution. We timed the four linear layers of one Llama-3.1-8B decoder block at batch 16 on the RTX 5090, quantization included. Our kernel with a single FP4 operand takes 117 to 125 µs against cuBLASLt's 143 µs; with three and four sparse operands, SL3 and SL4, it takes 113 and 116 µs. The lead over the vendor is the kernel's; the extra rungs cost no measurable time.

We will provide the full response and a revised manuscript by November 13, including the dense pair at task and kernel level with paired tests, end-to-end decode throughput, and B200 measurements with W4A16 on the same hardware. We will also report the controlled W4A8 comparison on NVFP4 weights, MoE evaluation, and the MATH-500 clarification requested by Reviewer ckrh. The revision will make Sections 3 and 4 self-contained, improve figure and table readability, and restore the official ICLR template typography.

AuthorsOpenReview ↗

Response to all reviewers: what structured sparsity contributes (revised PDF uploaded)

Thank you for three careful reviews. The revised PDF contains every experiment you asked for; table and figure numbers below refer to it. All three reviews ask what structured sparsity contributes beyond placing several FP4 terms on one weight load (ckrh W1, 8jsH W2 and Q2, CNDg W2). This comment answers that; the other points are answered per reviewer.

Terms used in all our replies

  • Rung, ladder, SLN: a rung is one 4:8 structured-sparse NVFP4 term, at half the arithmetic of a dense term. The ladder SLN runs N rungs on one weight load: SL3 costs one and a half dense terms, SL4 two.
  • Dense 1.5: one dense NVFP4 term plus a half term on a fixed half of the channels; the dense configuration at SL3's arithmetic.
  • Dense pair: two dense NVFP4 terms on one weight load; the dense configuration at SL4's arithmetic.
  • W4A16: the same 4-bit weights with 16-bit activations, the best an activation quantizer on those weights can reach.
  • Vendor kernel, vendor block: cuBLASLt's FP4 GEMM, and the four linear projections of one Llama-3.1-8B decoder block run with it.

Short answer: per-token channel selection. A rung covers half of the channels, and the sparse instruction lets each token choose which half. With dense terms the choice is the same for every token.

What the new experiments show

  1. The per-token half is more accurate than a fixed half on all twelve models (new Table 7). Table 5 of the submission compared SL3 on block-32 weights with dense 1.5 on block-16 weights, so it mixed the weight format with the channel selection, as Reviewer ckrh points out. The new experiment puts both on block-16 weights: the same dense first term with the exact block-16 scales, plus half a term spent either on a fixed half of the channels or on one sparse rung, whose GEMM reads each two adjacent 16-element scales merged into one. Across twelve models, WikiText-2 and C4, with and without the fold, the sparse half is closer to W4A16 in all 42 cells. Of the gap between one dense term and W4A16, the fixed half recovers 43% to 84% and the sparse half 68% to 100%, including on Qwen3-4B and Qwen3-14B, where the submission's Table 5 had the ladder behind. These cells are simulated, since our kernels issue one instruction kind per GEMM, and the revision labels them so.
  2. The ladder shows the same on its own weights (new Table 1). On identical block-32 weights SL3 is closer to W4A16 than dense 1.5 in all 24 cells of twelve models and two datasets, and it matches FP8 activations, the simpler alternative Reviewer CNDg raises. At task level it matches W4A16 and leads dense 1.5 on all six models tested, significantly on four, and on GSM8K it leads dense 1.5 on both models over the 7,473-question training split (new Table 8).
  3. At two terms the number of terms sets the accuracy. SL4 and the dense pair converge, as the submission states, and on block-16 weights the dense pair remains the most accurate configuration on most models; the revised conclusion says so.
  4. Time, on the RTX 5090. The ladder's speed advantage today is over the vendor's kernel: 1.30× on the decode block at M=16. Against the dense pair on our own kernel the block is level through M=64 with preparation included (new Table 20b), as Reviewers ckrh and CNDg observe. Per GEMM the sparse form is the faster one, one sparse term 1.17× faster than one dense term and SL4 1.16× faster than the stacked pair at M=16 (new Table 5); the pair's cheaper preparation closes that gap, so the half step's saving of a quarter of the arithmetic is not yet priced.

Why the sparse form is needed. A half term is only worth spending where the token needs it, and that place changes from token to token. The sparse instruction's 2:4 metadata lets every token choose its own channels and the tensor core applies the choice directly; a dense kernel would need a different weight read for every token. That gives the ladder its single knob: the number of rungs N moves the same kernel, on the same weight load, from one dense term of arithmetic to two in half steps. Dense reaches one and a half terms only with a block size, a residual half and a per-layer rule fixed in advance, at the lower accuracy shown above.

What we changed in the claims

  • Speed (ckrh W2, 8jsH W1). Scoped to the RTX 5090 at M≤64. The B200 speedup is out of the abstract: there the accurate configuration is 1.10× slower than the vendor block (Appendix D.4).
  • Half step (ckrh W1, CNDg W2). Presented as an accuracy point at one and a half terms, not as a latency saving; Section 7 states what would price it.
  • MATH-500 (ckrh Q4). The introduction's claim is withdrawn; Section 5.2 states what holds under each scorer.
Area_Chair_DCamOpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.