- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AWSLQ, a training-free W4A4 quantization method for small-batch LLM decoding. Instead of representing each activation with a single dense FP4 tensor, AWSLQ decomposes it online into several structured-sparse FP4 terms that share one weight load. A selective reverse-smoothing transform improves weight quantization, while a self-probe based on model-generated text decides whether to apply the transform without external calibration data. The authors also develop an M-stacked Blackwell kernel that executes the sparse terms within one GEMM. Experiments across Llama, Qwen, and Gemma models show that four sparse terms approach the W4A16 accuracy floor and are competitive with existing W4A4 methods.
Critical strengths
Hardware-aligned sparse residual decomposition. AWSLQ decomposes activations into several structured-sparse FP4 residual terms that reuse the same weight loading, providing a neat way to use otherwise idle compute to improve W4A4 accuracy during small-batch decoding.
Evaluation. The paper implements the sparse ladder on both RTX 5090 and B200 GPUs and includes activation preparation in the latency measurements. The results show that the method can improve W4A4 accuracy without giving up small-batch decode speed.
Critical weaknesses
Limited end-to-end performance validation. The speed evaluation covers individual GEMMs and a chained group of linear layers, but not complete model serving. Since the gain on B200 drops to only 2–6% against a similarly launched vendor baseline, it is unclear how much of the improvement would remain in end-to-end token throughput.
Unclear advantage over dense multi-term quantization. SL4 and two dense residual terms achieve similar perplexity and small-batch latency, but the paper does not compare them on downstream tasks or end-to-end performance. It is therefore unclear how much of the gain comes from structured sparsity instead of simply using multiple activation terms with M-stacking.
Questions for authors
-
Could the authors report full-model decoding speed, including activation processing, attention, KV-cache operations, and runtime overhead?
-
Could the authors provide more ablation / analysis on the gain comes from structured sparsity?
-
Figure 7(b) is somewhat confusing because the bars show latency, while the annotations report inverse latency ratios as speedups.
2 comments
Thank you for reading the new experiments, and for confirming that the end-to-end results address your first concern.
On SL4 and the dense pair we agree, and that tie is a ceiling. At two terms both sit on W4A16, the best any activation quantizer can reach on those weights: SL4 is within 0.01 perplexity of it on all twelve models (Table 1), and two dense terms are level with it (Tables 6 and 7). No gap is left there for either form to win.
What the sparse path adds is per-token channel selection. A sparse term refines half of the channels, and each token chooses its half; dense terms have no such choice. That can only show below the ceiling, at one and a half terms, where a gap remains. There, with the same weights, the same first term and the same arithmetic:
- SL3 is closer to W4A16 than the dense configuration in all 24 cells of twelve models (Table 1), and in all 42 cells on block-16 weights (Table 7, simulated);
- on the six-model task suite SL3 is within 0.3 points of W4A16 and leads the dense configuration by 0.3 to 1.0 points, significantly on four models (Table 8).
In practice, one and a half terms of sparse arithmetic are enough for W4A16 task accuracy, and the sparse path gives nothing back at two terms: per GEMM SL4 is 1.16× faster than the stacked dense pair (Table 5), and with preparation included the block is level (Table 20b). On the RTX 5090 at decode batch sizes the saved quarter of the arithmetic buys no time yet; Section 7 says so and lists what would price it.
We hope the comparison at one and a half terms speaks to your second question, on what structured sparsity contributes: its two arms differ only in the per-token selection. We would be glad to hear whether it does.
Response to Reviewer 8jsH: end-to-end speed, the sparsity ablation, Figure 7(b)
Thank you for the review. Table and figure numbers are those of the revised PDF.
End-to-end speed (W1, Q1). The revision reports full-model decoding (Figure 5b, Table 24, Appendix D.8): a full Llama-3.1-8B decode loop with activation preparation, attention, the KV cache and every launch inside the measurement.
What structured sparsity contributes (W2, Q2). The revision adds the ablation you ask for, in accuracy and in time.
Figure 7(b) of the submission, now Figure 5(a) (Q3). The labels were the vendor's latency divided by ours, printed over latency bars without saying so. They now read "1.30× faster" or "1.29× slower", and the caption defines them.