acceptodds
Under review as a conference paper at ICLR 2027

AMS-Quant: Adaptive Mantissa Sharing for Floating-Point Quantization

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in various kinds of tasks, while the billion or even trillion parameters bring storage and efficiency bottlenecks for inference. Quantization, particularly floating-point quantization, is known to be capable of speeding up LLM inference by reducing memory footprint and data movement during the inference process. For the first time, we advance the floating-point quantization exploration from integer bit-widths to non-integer bit-widths, namely AMS-Quant, to further approach the quantization sweet spot. AMS-Quant incorporates two novel techniques to put it into effect: (1) it proposes Mantissa-bit Sharing, which groups k quantized weights and lets them share the least significant mantissa bit, allowing us to further approach the minimum quantization bit-width without accuracy loss. (2) It introduces Adaptive Searching, which employs an offline optimization strategy to minimize the accuracy degradation introduced by sharing. Moreover, AMS-Quant is also prototyped as efficient CUDA Linear kernels, which translates memory savings into wall-clock latency reduction by reducing memory access. Extensive experiments on large-scale datasets and models show that AMS-Quant can quantize the model to FP-5.33-e2m3 and FP4.25-e2m2, and significantly speed up the LLM decoding over FP16 inference (2.8 and 3.2), with negligible accuracy loss.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UxKMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes AMS-Quant, a weight-only floating-point quantization scheme in which groups of k weights share their least significant mantissa bit. Starting from channel-wise RTN quantization, the method chooses the shared bit by minimizing group-wise weight reconstruction error, yielding nominal rates such as 5.33 and 4.25 bits per weight. The authors also implement packed CUDA linear kernels. Experiments on five Llama/Qwen models report near-FP16 average accuracy at 5.33 bits and improved accuracy over RTN FP4 at 4.25 bits. Selected memory-bound linear layers obtain up to 2.77× and 3.30× speedups over FP16.

Critical strengths
  1. Sharing one mantissa bit gives a concrete mechanism for intermediate storage rates with a lightweight offline search and straightforward bit reconstruction.
  2. The packed representation and CUDA results provide evidence that fractional nominal rates can translate into useful linear-layer latency reductions, beyond merely reducing an abstract bit count
  3. Table 2 reports instruction following, mathematical reasoning, and knowledge accuracy separately. The 5.33-bit results are consistently close to the FP16 averages across the tested models.
Critical weaknesses
  1. The accuracy baselines are limited to RTN quantization. Comparisons with other floating-point PTQ methods, such as AFPQ and LLM-FP4, are needed.
  2. Table 2 covers too few downstream tasks. More challenging benchmarks, such as MATH, GPQA, and HumanEval, should be added.
  3. The kernel evaluation is limited and lacks end-to-end speed comparisons. The gains in Table 3 seem mainly due to reduced memory traffic at lower bit widths. They do not isolate the benefits of the kernel design.
Questions for authors
  1. Could you clarify the mathematical rationale and steps of Adaptive Searching?
  2. Could you compare FP4.25 with MXFP4 at a matched storage budget, including scale overhead?
Reviewer_JbTQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AMS-Quant, a weight-only floating-point quantization method for LLMs. It first applies round-to-nearest quantization and then lets groups of weights share the least significant mantissa bit. A search selects the shared bit to minimize weight reconstruction error. Experiments on Llama and Qwen models show that the FP5.33 configuration largely preserves FP16 accuracy. The authors also implement CUDA kernels that accelerate memory-bound linear layers.

Critical strengths

None

Critical weaknesses
  1. Insufficient comparison with established low-bit quantization methods. The main accuracy experiment (Table 2) compares AMS-Quant only with FP16 and vanilla round-to-nearest (RTN) floating-point quantization formats. However, RTN is a basic quantization baseline. Although the paper discusses established PTQ methods such as GPTQ and AWQ, it does not include experimental comparisons with them. The paper also lacks discussion of other relevant approaches, such as OmniQuant and rotation-based quantization methods like SpinQuant. I consider this a major limitation because demonstrating a better accuracy–storage trade-off within a restricted set of RTN-based formats does not establish the practical advantages of AMS-Quant over existing methods.

  2. Insufficient justification for the proposed search. AMS-Quant first applies RTN and then optimizes only the shared least significant mantissa bit, keeping the remaining bits fixed. However, the initial RTN assignment may no longer be optimal once the shared-bit constraint is imposed. The search only selects the better of two candidates with fixed high bits.The paper neither analyzes this potential error gap nor compares the proposed search with re-rounding each weight under the same constraint. Therefore, it remains unclear how much avoidable quantization error results from fixing the initial RTN assignment.

Questions for authors
  1. The paper does not provide an ablation that isolates the contribution of Adaptive Searching. Could the authors compare it with simpler strategies, such as always setting the shared mantissa bit to 0 or 1, while keeping the quantization format and group size unchanged?

  2. The paper does not report perplexity results. Could the authors include perplexity evaluations on language-modeling datasets such as WikiText-2 and C4 to complement the downstream task results?

  3. End-to-end inference latency. Section 4.2 reports only kernel-level performance for MLP-down linear layers. These results do not establish end-to-end inference speedups. Could the authors also report full-model inference latency and throughput?

  4. Figure 6 is difficult to read because the legend is too small. It is also unclear how many layers were tested and whether the results were averaged over all MLP-down layers or measured using representative matrix shapes. Could the authors clarify the measurement procedure and enlarge the legend? The titles and axis labels in Figure 5 should also be enlarged.

  5. Table 2 does not clearly distinguish AMS-Quant configurations from the baselines. Without carefully reading the main text, it is difficult to identify which results were obtained using the proposed method. Please explicitly label the AMS-Quant rows in the table.

  6. In Section 4.1, the authors state that FP5.33 is a better quantization bit-width than both FP6 and FP5 in terms of model accuracy and inference efficiency. However, Table 3 shows that FP5 is faster than FP5.33 in every reported configuration. Could the authors clarify or revise this statement?

Reviewer_6cNCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes mantissa-bit sharing in floating point representations of neural net/LLM weights. Groups of k weights share the mantissa's least significant bit, thereby reducing the total number of bits. It chooses how that shared bit is selected via an offline search. The paper also presents numerical experiments to illustrate the performance of the proposed idea.

Critical strengths
  • The idea of bit-sharing is very reasonable and interesting in the context of floating point weights.
Critical weaknesses
  • The idea, while natural, is not sufficient to make this a strong ICLR contribution. For example, there is no meaningful theoretical analysis of the approach, and no comparison against PTQ methods like GPTQ either theoretically or numerically.
  • The ``adaptive search" part of the contribution is rather basic, and done on an MSE loss targeting W-Q, with no emphasis on X(W-Q) which is a better proxy for the error the method will cause.
Questions for authors
  • This is a suggestion for improvement: Figure 2 is not very convincing. If it is supposed to show that the distribution of FP6 formats resembles that of the actual weights of neural networks, more experiments are needed, and a better visualization.
  • Why focus on MSE of the weights, and not on the error associated with XW? I understand it is easier, but the latter should be better.
  • How well does the idea empirically perform with more sophisticated quantization methods than round-to-nearest like GPTQ, GPFQ, etc... ?
  • In table 2, some of the lower bit representation (e.g., FP5.33, FP4.5) outperforms FP6 on some tasks, and on average. This seems odd. Do you have a good explanation?
  • In the paragraph starting "We noted that it is possible for GPTQ to realize non-integer quantization by performing per-group quantization and adjusting the grid size...", it is natural to ask why you didn't compare against these methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.