acceptodds
Under review as a conference paper at ICLR 2027

LSMXQ: Layerwise Salience-Aware Mixed-Precision Quantization for Efficient LLM Inference

Abstract

Post-training quantization (PTQ) is an effective approach for reducing the memory footprint of large language models (LLMs), enabling larger models to fit within limited GPU memory and improving inference efficiency under memory-bound workloads. While recent sub-4-bit quantization methods can achieve high accuracy compared to FP16 baselines, they do not necessarily leverage modern hardware-supported FP4 Tensor Core architectures, limiting their ability to fully utilize the computational throughput of modern GPUs, particularly at larger batch sizes. In this work, we introduce LSMXQ, a hardware-aware mixed-precision post-training quantization approach that allocates low-bit representations under an average bit budget while minimizing layer-wise propagated quantization error. LSMXQ is designed for coalesced memory access and efficient NVFP4 Tensor Core execution, supporting both memory-bound low-batch inference and high-throughput large-batch workloads. Across a broad evaluation of LLMs and zero-shot benchmarks, LSMXQ retains up to 98.5% of FP16 accuracy at 2-bit precision, while matching FP16 accuracy at 2.5- and 3-bit precision. For inference efficiency, LSMXQ achieves up to 3.71× the throughput of FP16 in batch-1 inference, while delivering substantial throughput improvements at larger batch sizes in vLLM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training quantization, mixed-precision quantization, hardware-aware, tensor core

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wcTAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a mixed-precision quantization method built around the NVFP4 format, using a widely adopted GPTQ-style Hessian sensitivity metric to guide bit-width allocation. However, several key details of the proposed representation are insufficiently specified. In particular, the paper refers to its configurations as 1-, 2-, 3-, and 4-bit quantization, but these nominal bit-widths do not appear to reflect the actual storage cost. In addition to the weight indices, the representation requires a 2-bit bit-width code for every 32 weights, an E4M3 scale for every 16 weights, and an FP16 scale for every 256 weights, together with the additional scaling metadata required by the execution path. Moreover, if different groups are allowed to select different E2M1 subsets, a subset identifier must also be stored for each group. Under the assumption that arbitrary subsets are allowed, the effective storage would be approximately 1.84375 bits/weight for the nominal 1-bit case, 2.96875 bits/weight for 2-bit, 4.0625 bits/weight for 3-bit, and 4.625 bits/weight for 4-bit, excluding any additional alignment or fixed metadata overhead. More importantly, the paper does not clearly explain how the 1-, 2-, and 3-bit E2M1 subsets are constructed, selected, stored, and decoded. If a bb-bit representation selects 2b2^b levels from the 16 E2M1 values, there are (162)\binom{16}{2}, (164)\binom{16}{4}, and (168)\binom{16}{8} possible subsets for the 1-, 2-, and 3-bit cases, respectively. It is therefore important to clarify whether the selected subsets are shared globally or can vary across 32-weight groups. If they vary across groups, the corresponding subset identifiers constitute non-negligible metadata and should be included in the reported effective bits per weight. The paper should provide a complete storage breakdown and a precise description of the subset-generation and encoding scheme.

Critical strengths

None

Critical weaknesses
  1. The paper does not clearly explain how the 1-, 2-, and 3-bit E2M1 subsets are generated, selected, and encoded. In particular, it is ambiguous whether the selected subsets are shared globally or can vary across 32-weight groups. If different groups can use different subsets, the runtime must additionally encode and recover the corresponding subset identity. This ambiguity makes the proposed representation difficult to reproduce and leaves a key part of the method underspecified.
  2. The 2-, 2.5-, and 3-bit configurations appear to count only the packed weight indices, while the actual representation additionally requires per-group bitwidth metadata and hierarchical scaling factors. As a result, the nominal bitwidth can substantially underestimate the effective bits per weight, making the compression claims and comparisons harder to interpret.
Questions for authors
  1. Are the E2M1 subsets used for the 1-, 2-, and 3-bit representations shared globally, shared per layer/tensor, or selected independently for each 32-weight group?
  2. Could the authors provide the exact candidate E2M1 subsets considered for the 1-, 2-, and 3-bit representations, as well as the subsets selected for each target average bitwidth?
  3. Could the authors report the effective bits per weight of the 2-, 2.5-, and 3-bit configurations after including all representation overhead?
  4. Could the authors clarify how the bit-width configurations are set for each compared method?
  5. Since the throughput comparison is mainly between LSMXQ 2-bit and native NVFP4, could the authors also report the corresponding accuracy/perplexity of NVFP4 under the same models and evaluation setup? This would make the accuracy–memory–throughput trade-off directly comparable.
1 comment
AuthorsOpenReview ↗

The subsets are shared globally and not stored per group, the full storage cost is 2.625 to 3.625 bits per weight, and the compared settings and the NVFP4 accuracy are now reported.

Thank you for the review. The points you raise were not stated clearly enough in the original submission, and the revised paper describes them explicitly. We say where each point is addressed in the revised PDF (L. = line numbers in the margin; appendix items by section and table number), and refer to our response to Reviewer LPU5 (LPU5) where the answer is the same.

Weakness 1 and Q1 (subsets: shared globally or per group; how they are built, selected, stored and decoded). The subsets are shared by all groups of a model and bit width. They are not stored per group, so no subset identifier is stored (L. 139--142). A subset is defined by a step and an offset into the E2M1 magnitudes: the raw code cc selects magnitude index c⋅step+offsetc\cdot\mathrm{step}+\mathrm{offset} (A.8). The 4-bit width uses the full E2M1 grid and the 1-bit width the single magnitude 0.5, so only 2 and 3 bits have a choice. Selection is by WikiText-2 perplexity among a few candidates (A.3). The subset is a minor design choice: we tried several candidates for exploration, the selected one is at most 11% better in perplexity than the third-best at 2 bits and at most 4% at 2.5 and 3 bits, and the same subset is chosen for all nine models at 2 and 2.5 bits and for six of nine at 3 bits (Table 35). The decoder reads the subset from a fixed per-width lookup table (L. 295--297). See also LPU5, Q3 and Q7.

Q2 (candidate and selected subsets). These are now listed in the paper: candidates, selected subset per model and bit width, and the gap to the third-best in Table 35 (A.3); the subsets and their magnitudes in Table 44 (A.8). See also LPU5, Q3.

Weakness 2 and Q3 (effective bits per weight). We agree that nominal widths understate storage. The stored cost per weight is the indices, a 2-bit width code per 32 weights (0.0625), an E4M3 scale per 16 weights (0.5) and a BF16 scale per 256 weights (0.0625): 2.625, 3.125 and 3.625 bits for the nominal 2, 2.5 and 3 (Table 1). The figures you computed (2.97, 4.06 bits for 2 and 3 bits) assume a subset identifier per group, which we do not store. Total weight memory including embeddings and LM head: Table 36 (A.4). Matched-storage comparison with QTIP: Table 37 (A.4). The full storage breakdown is new in the revised paper; see also LPU5, Q1.

Q4 (bit-width configuration of the compared methods). GPTQ and SliM-LLM are run at their nominal widths with group size 128, and QTIP with its public 2- and 3-bit codes, all with the same calibration (A.9). GPTQ and SliM-LLM add per-group scale and zero-point metadata (about 0.2 bits per weight) that we do not add to their nominal widths, while QTIP's side information is negligible; LSMXQ is reported at its nominal and at its effective widths. Table 37 compares at matched storage. See also LPU5, Q4 for the evaluation protocol.

Q5 (accuracy of NVFP4 under the same setup). Qwen3-8B: NVFP4 perplexity 9.970 (W4A16) and 10.120 (W4A4), zero-shot average 0.7294 served in vLLM, against 0.6318 for LSMXQ 2-bit (Table 40, A.6). Qwen3-32B: 0.7733 (W4A4) and 0.7786 (Marlin, W4A16), against 0.7420 (Table 41). Llama-3.1-405B: perplexity 2.22 and average accuracy 82.0% for NVFP4 against 4.68 and 79.9% for LSMXQ (L. 481--485). The accuracy cost of LSMXQ against NVFP4 is stated at L. 473--474. This comparison is new in the revised paper; see also LPU5, Q6.

Regarding our contribution and novelty. Sub-4-bit weight formats currently have no kernels that run on the NVFP4 tensor-core path of Blackwell GPUs, and a quantization algorithm that is dynamic in its allocation of bits per group only helps if it is designed together with the hardware: an exact bit budget, a coalesced packing, and a layout that NVFP4-compatible kernels can execute. This is what LSMXQ addresses. The paper now shows that the resulting format runs through vLLM on FP4 tensor cores (Section 3.3), and serves Llama-3.1-405B on two GPUs where stock NVFP4 does not fit (Table 5). We thank the reviewer for the careful reading, which led us to specify the representation precisely.

Reviewer_D8feOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LSMXQ, post-training quantization method for sub-4-bit LLM inference. It assigns 1-, 2-, 3-, or 4-bit representations to groups of 32 weights using Hessian-weighted reconstruction error and dynamic programming under an average bit budget and GPU memory-packing constraints. The compressed weights are decoded into NVFP4 E2M1 values at runtime, allowing the method to use Blackwell FP4 Tensor Cores while reducing weight memory.

Critical strengths

LSMXQ jointly considers quantization error, mixed-bit allocation, memory packing, and NVFP4 execution. The compressed representation can be decoded into hardware-supported FP4 values at runtime, making the method more practical than approaches evaluated only through simulated quantization.

Critical weaknesses

LSMXQ combines several ideas, including Hessian-based salience, subset selection, dynamic programming, and hardware-aware packing, but does not show which parts actually matter. A comparison with simpler allocation methods using the same quantization format and kernel would make the main contribution clearer.

The speed results compare mainly with FP16/BF16 and native NVFP4, instead of existing optimized low-bit systems. LSMXQ is slower than NVFP4 at the same KV-cache size. This advantage may diminish under large-scale serving scenarios on cloud GPUs.

Questions for authors

What is the runtime overhead of decoding the mixed-bit weights into E2M1 values? Could you ablate this overhead from the NVFP4 GEMM?

How is the progressive mixed-bit decoding implemented in the vLLM execution path? Is it fused with the NVFP4 GEMM or launched as a separate kernel?

1 comment
AuthorsOpenReview ↗

The ablations show which components matter, the decoding overhead is measured and separated from the GEMM, and we state where LSMXQ is slower than NVFP4 and where its advantage lies.

Thank you for the review. Most of your points overlap with Reviewer LPU5's, so we point to our response to Reviewer LPU5 where the answer is the same, and say where it is addressed in the revised PDF (L. = line numbers in the margin; appendix items by section and table number).

Weakness 1 (which components matter, simpler allocation with the same format and kernel). We replaced the exact dynamic program by a greedy exchange and by a pointer heuristic, with the same format and kernel, and removed or changed the packing constraint, the level subsets and the Hessian weighting: Tables 38 and 45 (A.5, A.10). The exact DP is best; the heuristics worsen perplexity by 7.6% and 3.8%. See also LPU5, Q2.

Weakness 2 (comparison with optimized low-bit systems; slower than NVFP4 at the same KV; large-scale serving). We compare with Marlin, the optimized W4A16 kernel, per layer for M=1 to 2048 (Figure 2 / Table 48) and in serving (Table 4, text: L. 423--427), and with stock NVFP4. You are right that LSMXQ is slower than NVFP4 at the same KV size and under saturated load (0.76 to 1.03$\times$; 0.82 to 0.84$\times$ at matched KV in Tables 50--51; L. 423--427, 468--473); we state this and name large-batch decoding the main open issue (L. 512--513). The advantage is at low concurrency, in memory (serving KV sizes at which both 4-bit baselines run out of memory) and in fitting larger models. At larger scale: Llama-3.1-405B runs on two GPUs, where stock NVFP4 does not fit, and on four GPUs reaches 0.94 to 1.02$\times$ of NVFP4 with 60% more KV tokens (Table 5; text: L. 475--480). Limit: we measured only on the RTX 5090 and the RTX PRO 6000 Blackwell Server Edition; we could not run on B200-class datacenter GPUs, whose architecture differs, so our kernels and numbers may not transfer to them.

Q1 (decode overhead, separated from the GEMM). Decoding one Qwen3-8B layer takes 83 μ\mus, 39% of the layer latency at M=256 and 11% at M=2048 (A.10, Table 47). The GEMM itself is the stock one and costs the same: 151.4 μ\mus against 152.3 μ\mus for stock NVFP4 at M=704 (Table 46), so the decode accounts for the difference. See LPU5, Weakness 1.

Q2 (progressive decoding in vLLM; fused or separate kernel). It depends on the number of tokens M per step (L. 299--306, Figure 1, A.1). For small M, one fused GEMV decodes in registers and feeds FP4 tensor-core MMAs; for intermediate M, a fused decode GEMM decodes in dedicated warps into a shared-memory tile consumed by the MMA warps; for large M, a separate decode kernel writes one layer into a shared scratch buffer and the stock NVFP4 GEMM consumes it. A router picks the path per layer from latencies measured offline.

Reviewer_LPU5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LSMXQ combines Hessian-weighted reconstruction costs, 32-weight mixed-precision groups, and dynamic programming under coalesced-packing constraints. Weights use subsets of E2M1 levels with 1–4-bit indices. A native batch-one path and an unpacking/scaling conversion path for NVFP4 GEMM aim to connect sub-4-bit storage with hardware-supported execution. Evaluation covers Llama and Qwen models, perplexity and six zero-shot tasks, batch-one decoding, and vLLM serving under different KV-cache budgets.

Critical strengths
  1. The paper addresses a real gap: sub-4-bit weight formats usually cannot use Blackwell's FP4 Tensor Cores, and LSMXQ is built so that they can. The abstract states the gap (L. 016–017), and the format is designed for "coalesced memory access and efficient NVFP4 Tensor Core execution" (L. 022). Three choices make this work. First, the 1–3-bit levels are subsets of E2M1, so expanding an index to a 4-bit code is exact and "the packed E2M1 weight values remain unchanged" (L. 212). Second, the scales are converted offline to the per-16 E4M3 plus FP32 layout that the NVFP4 GEMM expects, including merged QKV and gate–up projections executed as "a single NVFP4 GEMM invocation" (L. 202–207). Third, the bit allocation respects 1024-bit packing units, so groups are never split across memory boundaries (L. 252–264). As a result, the vLLM path can reuse the optimized NVFP4 GEMM "without requiring a separate kernel" (L. 069–070). Codebook and trellis formats, such as the QTIP baseline used here, cannot map directly onto this path. Co-designing the quantizer with the execution format is the paper's most consequential idea.
  2. The system results are measured on real Blackwell hardware, not inferred from perplexity. With the 2-bit format, batch-one decoding is 2.1–3.7× faster than FP16 (Tables 5–6, L. 389, 409). LLaMA-30B fits in 9.93 GiB (L. 404). In vLLM, the smaller weight footprint allows a larger KV cache before running out of memory (Tables 7–8; L. 431–434). These measurements make the hardware-aware claim concrete, even though, as argued below, they do not isolate the cost of decoding.
Critical weaknesses
  1. The cost of decoding to NVFP4 at run time is never measured, and it is not compared with dequantizing on CUDA cores. In vLLM, "an additional decode step integrated into the execution pipeline … progressively decodes the weights into the E2M1 representation required by the optimized NVFP4 GEMM kernel" (L. 243–246). Because the model is never held in decoded form, this step presumably runs for every layer in every forward pass. The paper reports neither its time, nor whether the decoded codes are written to global memory or staged on chip, nor the size of any temporary buffer, nor whether decoding overlaps with the GEMM. This matters for memory-bound decoding. If the decoded E2M1 tensor is materialized in global memory, a forward pass at the 2-bit setting would move roughly 2.3–2.5× the weight bytes of native NVFP4: about 2.1–2.6 bits per weight read as packed indices and metadata, 4 bits written, and 4.5 bits read by the GEMM (codes plus E4M3 scales), against 4.5 bits for native NVFP4. That traffic alone would remove the bandwidth advantage of sub-4-bit storage. The only quantitative evidence is end-to-end. At matched KV budgets, LSMXQ reaches 83.9–87.5% of NVFP4 throughput on Qwen3-8B and 61.9% on Qwen3-32B (Tables 7–8). This gap is consistent with a decode cost that is not amortized at low concurrency, but it is not separated from the GEMM or from other overheads. The text calls it "a modest throughput delta" and a "moderate throughput reduction" (L. 428, 435), and it describes 87.5% of NVFP4's serving tok/s as "the raw compute throughput" (L. 427). The natural alternative is never evaluated: decoding the same format in registers on CUDA cores, fused into the GEMM main loop, with BF16 Tensor Core multiplication (a W4A16-style kernel such as Marlin, which the paper cites at L. 129) or a LUT-based GEMV. That design avoids a separate pass and keeps activations in 16 bits. Without a comparison at matched effective bits across batch sizes and both phases (decoding and prefill), it is unknown where decoding to NVFP4 plus FP4 MMA wins and where CUDA-core dequantization is faster. This trade-off is the paper's central systems claim, because the case for using Blackwell's FP4 Tensor Cores rests on it.
  2. The system evaluation does not yet reflect a realistic kernel environment. Section 4.4, "Kernel-Level Performance" (L. 399–406), contains no kernel-level measurements: no per-layer or per-kernel latency, no achieved memory bandwidth, and no tensor-pipe utilization. The batch-one kernel is described only up to the point where decoded values are "consumed by the subsequent GEMM" (L. 277–279). The paper does not say whether that GEMV runs on FP4 Tensor Cores or on CUDA cores. The batch-one tables compare only with FP16. They include no NVFP4 or 4-bit weight-only kernel (e.g., Marlin) and no other sub-4-bit decoding kernel at matched effective bits, although a format that stores about 2.6 bits per weight should beat FP16 under almost any efficient kernel. One signal suggests that the batch-one kernel is not purely bandwidth-bound. Moving from the RTX 5090 to the RTX PRO 6000, FP16 throughput drops by 8.5–10.1%, but LSMXQ-2b drops by 16.4–23.1% (Tables 5–6), even though the two GPUs have nominally similar memory bandwidth. The serving study tests one concurrency limit (B = 256, L. 422) and one prompt set per model. It does not discuss several deployment questions: compatibility with CUDA graphs and torch.compile; continuous batching and chunked prefill, where the per-step decode cost is amortized over a varying number of tokens; tensor-parallel sharding of the 1024-bit packing units; and the memory taken by decode buffers, which reduces exactly the KV headroom that the paper presents as LSMXQ's advantage.
Questions for authors
  1. Please give exact effective bits/weight, total weight residency, temporary decode buffers, and retained high-precision tensors for LSMXQ and the strongest baseline. At matched downstream quality and total GPU memory, what throughput advantage remains?
  2. Can you isolate DP versus a heuristic allocation, selected versus fixed E2M1 subsets, and constrained versus unconstrained packing? Please include quantization time and peak host/GPU memory for the search.
  3. What are the explicit DP segment-state update, scale candidates, and subset-selection scope? Which held-out data select subsets, and how are the reported evaluation corpora kept separate from that selection?
  4. Could you re-examine the ARC-Easy results for QTIP and SliM-LLM on the LLaMA-1/2 models and for the QTIP/QuIP# checkpoints in Table 29? Please report the harness version, metric variant (acc vs. acc_norm), prompt format, and BOS/tokenizer settings, and include per-task standard errors. Do the conclusions of Sec. 4.3 hold on the other five tasks? If corrected ArcE scores are in line with the other methods, LSMXQ's 2-bit lead over QTIP would hold on only about half of the models.
  5. Why does in-domain WikiText-2 calibration raise LSMXQ's WikiText-2 perplexity by 28–31% on LLaMA-7b at 2.5 and 3 bits (Table 32), when GPTQ and QTIP change by ≤1.4% at 3 bits? Was the E2M1 subset re-selected for each calibration set, with the same number of samples and sequence length? Could you report the main comparison under both protocols, or under a protocol fixed independently of the zero-shot results?
  6. What is the activation precision in the vLLM path and in the batch-one kernel? Please report WikiText-2 perplexity and zero-shot accuracy for the model exactly as served (after scale fusion, with the activation format actually used), together with the NVFP4 baseline's accuracy. That would let Tables 7–8 be read as accuracy–throughput trade-offs.
  7. I think LSMXQ's ranking depends on a calibration protocol selected on the evaluation benchmarks, and LSMXQ is markedly more calibration-sensitive than the baselines. Sec. 4.2 adopts the five-corpus mixture "for the remaining experiments" because it "generally produces better zero-shot results" (L. 318–321), i.e., on the same benchmarks later used for the comparison. Tables 30–36 show that the choice matters most for LSMXQ. Over the six model–bitwidth cells available for all methods (LLaMA-7b, LLaMA2-7b, and Qwen2.5-7b at 2 and 3 bits), switching between the mixture and WikiText-2 changes LSMXQ's average accuracy by 4.05 points on average, versus 1.04 for GPTQ, 2.28 for SliM-LLM, and 0.89 for QTIP. At 2.5 bits on LLaMA-7b, the change reaches 10.72 points. On LLaMA-7b, in-domain WikiText-2 calibration raises LSMXQ's WikiText-2 perplexity by 28.5% at 3 bits and 31.4% at 2.5 bits (Table 32). At 3 bits the corresponding changes are +0.9% for GPTQ, −1.4% for QTIP, and +12.5% for SliM-LLM. Under that protocol, LSMXQ-3b on LLaMA-7b ranks third in average accuracy (0.6011 vs. GPTQ 0.6562 and QTIP 0.6206) and has the worst perplexity of the four methods (7.81 vs. 5.96–7.49); at 2 bits it trails QTIP by 7.9 points (0.5282 vs. 0.6072). Across the three models, LSMXQ has the best 3-bit average in all three under the mixture but in only one under WikiText-2. The paper discloses these comparisons, which is commendable (Critical Strength 2). It does not discuss these implications, or whether selecting E2M1 subsets by calibration perplexity (L. 139–140) adapts too closely to the calibration set.
  8. Table 6's "resident VRAM footprint" appears to count only the quantized decoder linear layers, and the implied effective bitwidths sharpen the storage concern in Weakness 1. For Qwen3-8B, the BF16 embedding and untied LM head alone would occupy ≈2.32 GiB, more than the 2.14 GiB reported for the whole 2-bit model (L. 415). Dividing each footprint by the decoder linear-layer parameter count (computed from the public model configurations) gives 2.64–2.66 bits/weight at nominal 2 bits, ≈3.17 at 2.5 bits, and 3.72–3.73 at 3 bits, consistently across all six models. This agrees with the ≈0.625-bit scale/metadata estimate in Weakness 1 plus a small remainder, and indicates that embeddings, LM head, KV cache, and runtime buffers are excluded. The statement "enabling LLaMA-30B to run within 9.93 GiB at 2-bit" (L. 404–405) therefore understates the memory needed to run the model. If these rates are right, LSMXQ-2.5b (≈3.17 bits/weight) is closer in storage to the nominal 3-bit baselines than LSMXQ-3b (≈3.73) is. At that point, QTIP-3b has lower WikiText-2 perplexity than LSMXQ-2.5b on all nine controlled models (Tables 10–28). It also has higher average accuracy on the four models not affected by the ArcE anomaly in Weakness 3 (LLaMA3-8b and Qwen2.5-7b/14b/32b). Because the baselines' effective rates are not reported, this is an inference to verify, not an established result.

minors:

  • L. 002–004: the title hyphenates "INFERENCE" across two lines ("IN-"/"FERENCE"); a manual line break would avoid this.
  • Notation is overloaded. B is the GPTQ block size (L. 164) and the vLLM batch limit (L. 422). Calligraphic B denotes both the current column block (L. 167) and the candidate-bitwidth set in Algorithm 1 (L. 217). q is both the DP budget state and the per-group bit cost q_{g,b} (L. 217–224). "NP" (L. 422–423) is typeset as a product of N and P, and the "M" in "M=1" (L. 389, 409) is undefined.
  • Algorithm 1 does not state the terminal condition (an exact total of Q bits, cf. L. 186) or which states are "feasible" (L. 222), and line 13 backtracks "from the optimal feasible state" (L. 231) without defining it.
  • L. 187–191: "Specifically, …" restates L. 183–186, and C_{g,b} = E_{g,b} introduces a second name for the same quantity; the two passages can be merged.
  • L. 255: "the fixed-width representation" suggests that all groups use 2 bits in the 2.0-bit configuration, whereas Table 1 and the per-group width codes imply mixed widths. Please clarify.
  • Sec. 3.2 says each thread loads "the 32-bit portion of the weights it processes" (L. 260–261), while Sec. 3.3 says each lane decodes "its assigned group" (L. 270–271); please state the thread-to-group mapping once. At L. 272–275, the weighted popcount sum yields the sum of width codes, not the bit offset itself. If code c encodes c + 1 bits, the offset of group i is 32·(i + Σ_{k<i} c_k); please give the mapping and the formula.
  • L. 362: "as a notable exception Table 41" → "(Table 41)". L. 1171–1173: "reported by (Tseng et al., 2024b)" should be a textual citation, "reported by Tseng et al. (2024b)", ideally pointing to the specific result.
  • Naming is inconsistent. Models appear as "LLaMA-7b"/"LLaMA2-7b" (tables), "LLaMA-2-70B" (L. 369), "LLaMa 2/3" (L. 341), and "Llama" (L. 284), with "b" and "B" mixed. GPUs appear as "RTX 5090" (L. 389), "GPU-RTX 5090" (L. 440, 451), "GPU-RTXPRO6000" (L. 409), and "RTX PRO 6000" (L. 402). The unquantized Qwen3 baseline is labeled FP16 in Tables 5–6 but BF16 in Table 7.
  • L. 427: "the raw compute throughput of native, hardware-accelerated NVFP4" refers to end-to-end serving throughput (tok/s), not raw compute throughput.
  • Table 2 (L. 325–336) reports only which calibration set wins, not by how much, although the magnitudes in Tables 30–36 are what matter (Weakness 4). The calibration names also vary: mix/wikitext2 (caption), "wt2" (cells), "WikiText2" (Tables 10–36), and "WikiText-2" (text).
  • Table 3 (L. 340–350) duplicates nine rows of Table 43. In both tables, Qwen2.5-7b's 2.5-bit retention is 0.7156/0.7413 = 96.5%, but it is printed as 96.6 (L. 345, 1218). A few printed averages also differ slightly from the mean of the printed task scores (e.g., Table 25, LSMXQ-2b: 0.6937 vs. 0.6939, L. 944).
1 comment
AuthorsOpenReview ↗

We measured the decoding cost and compared it with Marlin, added kernel-level, serving, memory and accuracy evidence, and state the remaining limits.

Thank you for the detailed review. For each point we say where it is addressed in the revised PDF (L. = line numbers in the margin; appendix items by section and table number). Where a point exposed an error or a missing measurement, we measured it and corrected the paper.

Weakness 1 (decode cost, CUDA-core alternative). Measured: decoding one Qwen3-8B layer takes 83 μ\mus, 39% of the layer latency at M=256 and 11% at M=2048 (A.10, Table 47). Decoded codes are written to a shared scratch buffer only in the large-M path, where decoding is not overlapped with the GEMM, which is why LSMXQ is slower than NVFP4 there (L. 419--422); the fused GEMV and fused decode GEMM decode on chip (L. 299--306), and buffer sizes are in A.4. Per-layer latency of LSMXQ, stock NVFP4 and Marlin for M=1 to 2048, including batch one, is in Figure 2 / Table 48, and in serving in Table 4. Limit: Marlin reads the 4.5-bit NVFP4 checkpoint, so this is not a matched-bit comparison. Our decoding already runs on CUDA cores in registers, fused into the MMA loop (L. 299--303; A.1); what we did not build is a W4A16 (BF16 MMA) variant of our format in vLLM.

Weakness 2 (kernel measurements, deployment). Beyond the latency measurements above, Nsight Compute utilization (DRAM, tensor pipe, SM, issue) is in A.10, Table 46; it shows the batch-one kernel is limited by decode instruction throughput, consistent with the larger RTX 5090 to PRO 6000 drop (L. 413--415), and that the GEMV runs on FP4 tensor cores (A.1). Serving: concurrency 1/4/16 and saturation on five datasets (Table 4, Table 52 in A.12) and KV-budget sweeps (Tables 50--51 in A.11). LSMXQ is integrated into vLLM as a custom operator behind its normal linear-layer interface (L. 304--307), so vLLM's continuous batching, chunked prefill, CUDA graphs and compilation passes apply unchanged and are used in all runs (L. 332--333). Tensor parallelism and 128-channel sharding: A.1; Llama-3.1-405B on TP=2 and TP=4: Table 5.

Q1 (effective bits, memory, matched quality). Table 1; Tables 36 and 37 (A.4), including embeddings and LM head and a matched-storage comparison with QTIP.

Q2 (ablations, quantization cost). DP versus heuristics, packing constraint, subsets, Hessian weighting: Tables 38 and 45 (A.5, A.10); time and peak memory: Table 39 (A.5); the quantization time is also stated at L. 211.

Q3 (DP details, subset selection scope, held-out data). Algorithm 1 with the exact-budget terminal condition, and A.8. Subsets are global, not stored per group, and chosen by WikiText-2 perplexity on the reported split; there is no separate held-out set (A.3, Table 35). See Q7 for a check on this.

Q4 (ARC-Easy). We re-evaluated all methods with lm-eval 0.4.4 and corrected a tokenization error that had lowered ARC-Easy of QTIP and SliM-LLM (A.9). The conclusions were rewritten: LSMXQ at 2 bits is close to QTIP, not ahead (L. 366--368). Numbers: Table 2, and Table 26 for the LLaMA2-70B public checkpoints.

Q5 (WikiText-2 calibration and perplexity; subset re-selection; both protocols). We re-ran both calibrations under one fixed protocol: the same level subsets (not re-selected), 64 samples and sequence length 2048 (Table 27, A.2). The gap is at most 1.7 zero-shot points and 3.4% perplexity, except one outlier (LLaMA2-7B at 2.5 bits) explained there. Per-method effect of the calibration set on the baselines: Table 49.

Q6 (accuracy as served). Qwen3-8B and Qwen3-32B through the real vLLM kernels, next to stock NVFP4 and Marlin: Tables 40 and 41 (A.6); the accuracy cost is stated at L. 473--474. Activations are NVFP4 in both the vLLM path and the batch-one kernel (L. 268, 299--303).

Q7 (ranking depends on a protocol chosen on the benchmarks; subset adaptation). We agree the mixture was adopted after seeing zero-shot results, and we state this (L. 351--352); text: L. 347--353. LSMXQ's sensitivity under a fixed protocol is in Table 27 and the baselines' in Table 49. Subsets are chosen by WikiText-2 perplexity, not by zero-shot tasks (A.3). Additional check, not in the paper: on LLaMA3-8B at 2 bits, six candidate subsets ranked by WikiText-2 and by C4 perplexity give the same top three, with the selected subset (step 3, offset 1; subset A in Tables 35 and 44) best on both (11.89 and 18.98).

Q8 (resident footprint). Table 36 gives total weight memory including embeddings and LM head (A.4), and Table 1 the effective bits.

Minor points. All fixed in the revised text: title, notation, Algorithm 1, thread-to-group mapping and bit-offset formula, citation forms, naming, FP16/BF16 labels, retention percentages, the ``raw compute throughput'' wording, calibration names, and the duplicate summary table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.