- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a mixed-precision quantization method built around the NVFP4 format, using a widely adopted GPTQ-style Hessian sensitivity metric to guide bit-width allocation. However, several key details of the proposed representation are insufficiently specified. In particular, the paper refers to its configurations as 1-, 2-, 3-, and 4-bit quantization, but these nominal bit-widths do not appear to reflect the actual storage cost. In addition to the weight indices, the representation requires a 2-bit bit-width code for every 32 weights, an E4M3 scale for every 16 weights, and an FP16 scale for every 256 weights, together with the additional scaling metadata required by the execution path. Moreover, if different groups are allowed to select different E2M1 subsets, a subset identifier must also be stored for each group. Under the assumption that arbitrary subsets are allowed, the effective storage would be approximately 1.84375 bits/weight for the nominal 1-bit case, 2.96875 bits/weight for 2-bit, 4.0625 bits/weight for 3-bit, and 4.625 bits/weight for 4-bit, excluding any additional alignment or fixed metadata overhead. More importantly, the paper does not clearly explain how the 1-, 2-, and 3-bit E2M1 subsets are constructed, selected, stored, and decoded. If a -bit representation selects levels from the 16 E2M1 values, there are , , and possible subsets for the 1-, 2-, and 3-bit cases, respectively. It is therefore important to clarify whether the selected subsets are shared globally or can vary across 32-weight groups. If they vary across groups, the corresponding subset identifiers constitute non-negligible metadata and should be included in the reported effective bits per weight. The paper should provide a complete storage breakdown and a precise description of the subset-generation and encoding scheme.
Critical strengths
None
Critical weaknesses
- The paper does not clearly explain how the 1-, 2-, and 3-bit E2M1 subsets are generated, selected, and encoded. In particular, it is ambiguous whether the selected subsets are shared globally or can vary across 32-weight groups. If different groups can use different subsets, the runtime must additionally encode and recover the corresponding subset identity. This ambiguity makes the proposed representation difficult to reproduce and leaves a key part of the method underspecified.
- The 2-, 2.5-, and 3-bit configurations appear to count only the packed weight indices, while the actual representation additionally requires per-group bitwidth metadata and hierarchical scaling factors. As a result, the nominal bitwidth can substantially underestimate the effective bits per weight, making the compression claims and comparisons harder to interpret.
Questions for authors
- Are the E2M1 subsets used for the 1-, 2-, and 3-bit representations shared globally, shared per layer/tensor, or selected independently for each 32-weight group?
- Could the authors provide the exact candidate E2M1 subsets considered for the 1-, 2-, and 3-bit representations, as well as the subsets selected for each target average bitwidth?
- Could the authors report the effective bits per weight of the 2-, 2.5-, and 3-bit configurations after including all representation overhead?
- Could the authors clarify how the bit-width configurations are set for each compared method?
- Since the throughput comparison is mainly between LSMXQ 2-bit and native NVFP4, could the authors also report the corresponding accuracy/perplexity of NVFP4 under the same models and evaluation setup? This would make the accuracy–memory–throughput trade-off directly comparable.
The subsets are shared globally and not stored per group, the full storage cost is 2.625 to 3.625 bits per weight, and the compared settings and the NVFP4 accuracy are now reported.
Thank you for the review. The points you raise were not stated clearly enough in the original submission, and the revised paper describes them explicitly. We say where each point is addressed in the revised PDF (L. = line numbers in the margin; appendix items by section and table number), and refer to our response to Reviewer LPU5 (LPU5) where the answer is the same.
Weakness 1 and Q1 (subsets: shared globally or per group; how they are built, selected, stored and decoded). The subsets are shared by all groups of a model and bit width. They are not stored per group, so no subset identifier is stored (L. 139--142). A subset is defined by a step and an offset into the E2M1 magnitudes: the raw code c selects magnitude index c⋅step+offset (A.8). The 4-bit width uses the full E2M1 grid and the 1-bit width the single magnitude 0.5, so only 2 and 3 bits have a choice. Selection is by WikiText-2 perplexity among a few candidates (A.3). The subset is a minor design choice: we tried several candidates for exploration, the selected one is at most 11% better in perplexity than the third-best at 2 bits and at most 4% at 2.5 and 3 bits, and the same subset is chosen for all nine models at 2 and 2.5 bits and for six of nine at 3 bits (Table 35). The decoder reads the subset from a fixed per-width lookup table (L. 295--297). See also LPU5, Q3 and Q7.
Q2 (candidate and selected subsets). These are now listed in the paper: candidates, selected subset per model and bit width, and the gap to the third-best in Table 35 (A.3); the subsets and their magnitudes in Table 44 (A.8). See also LPU5, Q3.
Weakness 2 and Q3 (effective bits per weight). We agree that nominal widths understate storage. The stored cost per weight is the indices, a 2-bit width code per 32 weights (0.0625), an E4M3 scale per 16 weights (0.5) and a BF16 scale per 256 weights (0.0625): 2.625, 3.125 and 3.625 bits for the nominal 2, 2.5 and 3 (Table 1). The figures you computed (2.97, 4.06 bits for 2 and 3 bits) assume a subset identifier per group, which we do not store. Total weight memory including embeddings and LM head: Table 36 (A.4). Matched-storage comparison with QTIP: Table 37 (A.4). The full storage breakdown is new in the revised paper; see also LPU5, Q1.
Q4 (bit-width configuration of the compared methods). GPTQ and SliM-LLM are run at their nominal widths with group size 128, and QTIP with its public 2- and 3-bit codes, all with the same calibration (A.9). GPTQ and SliM-LLM add per-group scale and zero-point metadata (about 0.2 bits per weight) that we do not add to their nominal widths, while QTIP's side information is negligible; LSMXQ is reported at its nominal and at its effective widths. Table 37 compares at matched storage. See also LPU5, Q4 for the evaluation protocol.
Q5 (accuracy of NVFP4 under the same setup). Qwen3-8B: NVFP4 perplexity 9.970 (W4A16) and 10.120 (W4A4), zero-shot average 0.7294 served in vLLM, against 0.6318 for LSMXQ 2-bit (Table 40, A.6). Qwen3-32B: 0.7733 (W4A4) and 0.7786 (Marlin, W4A16), against 0.7420 (Table 41). Llama-3.1-405B: perplexity 2.22 and average accuracy 82.0% for NVFP4 against 4.68 and 79.9% for LSMXQ (L. 481--485). The accuracy cost of LSMXQ against NVFP4 is stated at L. 473--474. This comparison is new in the revised paper; see also LPU5, Q6.
Regarding our contribution and novelty. Sub-4-bit weight formats currently have no kernels that run on the NVFP4 tensor-core path of Blackwell GPUs, and a quantization algorithm that is dynamic in its allocation of bits per group only helps if it is designed together with the hardware: an exact bit budget, a coalesced packing, and a layout that NVFP4-compatible kernels can execute. This is what LSMXQ addresses. The paper now shows that the resulting format runs through vLLM on FP4 tensor cores (Section 3.3), and serves Llama-3.1-405B on two GPUs where stock NVFP4 does not fit (Table 5). We thank the reviewer for the careful reading, which led us to specify the representation precisely.