acceptodds
Under review as a conference paper at ICLR 2027

AdaPivot: An Adaptive Core-Tail Representation for Low-Bit LLM Quantization

Abstract

Post-training quantization (PTQ) of Large Language Models (LLMs) often suffers substantial accuracy degradation because fixed numerical formats poorly accommodate the pronounced outliers in weights and activations. Through extensive profiling across diverse models, we observe a distributional structure: most values concentrate in a compact core, while the outliers, which account for only a small fraction of the values, form a sparse tail, typically separated from the core by a sharp magnitude transition and a nearly unoccupied interval. Building on this observation, we introduce AdaPivot, an adaptive low-bit numerical format that partitions the representable range into distinct core and tail regions. AdaPivot independently adapts the representation granularity of the two regions, together with their boundary and dynamic ranges, to match the underlying tensor distribution within a compact parametric form. We further formulate format fitting as a joint discrete–continuous reconstruction problem and develop a tailored solver that efficiently co-optimizes discrete level selection and continuous format parameters. An integer-constrained variant further reduces metadata overhead while preserving the adaptive structure. Experiments on various models from 7B to 70B parameters show that AdaPivot consistently outperforms representative PTQ baselines. Under W4A4 quantization, it improves average zero-shot accuracy by up to 7.5% over the state-of-the-art baseline, while W4A8 performance closely approaches the FP16 baselines. Our code is available at https://anonymous.4open.science/r/AdaPivot-A.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Post-Training Quantization, Low-Bit Quantization, Adaptive Quantization, Non-uniform Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kZXyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

I have the following questions about this paper. Please help me analyze whether these are genuine issues based on the paper:

  1. The segmented scale format similar to Core-Tail is not original and has been explored in prior work, which limits the novelty of this paper.
  2. What advantages are brought by the constraints of segmented parameterization in AdaPivot?
  3. For AdaPivot's non-uniform quantization, how to design acceleration kernels? Considering the long-tail effect, why is there no comparison with fp4 quantization?
  4. What is the independent contribution of the additional endpoints proposed in this paper?
  5. The so-called Core–Tail Gap lacks sufficient statistical and causal evidence.
  6. Does the improvement under W4A4 truly come from the new format? The authors integrated AdaPivot into a relatively complete PTQ pipeline.
  7. There is a lack of real inference acceleration and hardware execution analysis.
Critical strengths
  • The paper addresses an important problem in low-bit LLM quantization: efficiently representing long-tailed weight and activation distributions.

  • The proposed format is compact and interpretable, and its parameter optimization is relatively straightforward.

  • The W4A4 results on larger Llama2 models are encouraging, especially the improvement over SliderQuant on Llama2-70B.

Critical weaknesses
  1. Limited novelty and insufficient justification of the proposed parameterization.

The core–tail representation appears closely related to existing distribution-adaptive and piecewise non-uniform quantization techniques. Previous methods, including LLM-FP4, BiE, and BlockDialect, have already explored flexible numerical formats for accommodating non-uniform distributions and outliers. Moreover, the proposed fitting procedure primarily relies on discrete candidate enumeration and alternating least-squares updates, making its algorithmic novelty relatively limited.

More importantly, the paper does not sufficiently justify why the particular piecewise parameterization offers a favorable trade-off compared with simpler fixed piecewise formats or more general learned non-uniform codebooks. Tables 4 and 5 show that fixed choices such as C=4 or S=6 individually achieve performance close to the full method, raising questions about the necessity of the proposed adaptive search.

A comparison with simpler piecewise quantizers and learned non-uniform codebooks under matched effective bit-width, calibration budgets, and reconstruction pipelines would help establish the distinct contribution of AdaPivot.


  1. Insufficient empirical evidence for the proposed core–tail gap motivation.

A central motivation of the paper is the claim that LLM weight and activation distributions exhibit a dense core and a sparse tail separated by a distinct low-density gap. However, the evidence is primarily based on selected distribution visualizations in Figure 1, without systematic statistics across layers, tensors, and quantization groups.

Furthermore, Table 3 suggests that adapting the tail stride contributes substantially more than introducing an explicit core–tail gap. Removing the gap only increases WikiText-2 perplexity from 7.15 to 7.55, while restricting the tail stride leads to considerably larger degradation.

Therefore, the current experiments more convincingly establish the usefulness of non-uniform level spacing than the necessity of explicitly modeling a core–tail gap. A quantitative characterization of gap prevalence, together with an analysis of its correlation with quantization improvements, would strengthen the proposed motivation.


  1. Unclear attribution of W4A4 improvements to the proposed numerical format.

AdaPivot is integrated into a relatively sophisticated PTQ pipeline incorporating GPTQ-style sequential error compensation and optional components such as LET, LWC, and activation-error compensation. Although the paper includes ablations on individual format parameters, it does not adequately isolate the contribution of the numerical format from the surrounding calibration and reconstruction procedures under W4A4.

The supplementary W4A16 comparison provides some evidence of the format's effectiveness, but it does not fully explain the substantially larger improvements observed under W4A4.

A more convincing evaluation would compare uniform INT4, competitive FP4 formats, and AdaPivot under the same calibration and reconstruction pipeline. The authors should also clarify which auxiliary components are enabled for each reported configuration.


  1. Missing FP4 comparisons and insufficient evidence of practical inference efficiency.

Given that AdaPivot is motivated by the limitations of fixed formats in representing long-tailed distributions, comparisons with FP4-based approaches are particularly important. However, methods such as LLM-FP4 and BlockDialect are discussed in Related Work but are not included in the main quantitative comparisons.

In addition, AdaPivot uses non-standard reconstruction levels with group-specific format parameters. Although the integer-constrained representation may permit integer-based execution, it remains unclear whether these reconstructed values can be efficiently processed by existing low-bit GEMM kernels without additional decoding, conversion, or arithmetic overhead.

The paper does not report kernel-level latency, end-to-end inference throughput, or actual memory consumption. Consequently, while the numerical accuracy improvements are promising, the practical inference benefits remain insufficiently established. At minimum, the authors should clarify the intended execution path, effective storage cost, and computational overhead of the proposed format.

Questions for authors

1.Contribution of the extra negative endpoint. The paper compares assigning the redundant negative-zero code to positive and negative tail endpoints. However, it does not isolate the contribution of endpoint reuse itself relative to a format without the additional level. Given the connection to previous redundant-zero remapping methods such as RaZeR, a targeted ablation would help clarify its independent contribution.

2.Generalization of the experimental comparisons. The Qwen3 W4A4 experiments omit several strong baselines included in the Llama2 evaluations. More comprehensive comparisons would be useful to support the claimed improvements across model families.

Reviewer_z83rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AdaPivot, an adaptive nonuniform quantization format for LLM weights and activations. It adjusts the allocation and spacing of quantization levels for the dense core and sparse tails. Experiments on Llama-2 and Qwen3 report gains over the evaluated baselines, with strong W4A4 results on larger models and near-FP16 accuracy under W4A8.

Critical strengths

The paper combines discrete parameter search with closed-form updates for continuous parameters. It also explains how to integrate the format with activation calibration and GPTQ error compensation. The integer-constrained version reduces the metadata needed to store the format. The methods and motivations are easy to understand, and the figures are well-drawn.

Critical weaknesses
  1. The paper presents uncovering a “consistent core–tail distributional structure” as a contribution. However, such an observation is not a novel insight in LLM quantization. Many existing works have studied this observation like BiE and SqueezeLLM.
  2. The comparison with baselines is insufficient. This work proposes a new non-uniform quantization format. The main baselines should therefore include other non-uniform quantization methods or commonly used formats such as NVFP4 and MXFP4. However, the experiments only compare against methods that use uniform integer formats. The experiments also use coarse-grained, per-channel quantization. Coarser quantization groups tend to have a wider dynamic range. Prior work (arXiv:2510.25602) shows that floating-point formats can outperform integer formats in this setting. Therefore, the reported gains may simply come from the baselines using less suitable quantization formats, rather than from the specific design proposed in this paper.
  3. The baselines in this paper use uniform quantization and could share the same optimized inference kernels in principle. However, this work proposes a new nonuniform quantization format that cannot directly use these kernels. An optimized GEMM implementation and inference speed benchmarks are therefore essential. Without them, readers cannot assess the practical value of the proposed quantization format. But the paper provides neither.
  4. The experiments lack sufficient comparisons with strong baselines such as AWQ (smoothing-based), QuaRot (rotation-based), or SVDQuant (svd-based). Although I notice that the authors report GPTQ results in Appendix Table S1, this comparison only covers Llama2-13B. This is not enough to support their claim to “evaluate AdaPivot across diverse models.” I consider these comparisons important for two reasons. First, these methods can address the wide dynamic ranges observed in the paper. Second, AdaPivot itself builds on GPTQ. Without more thorough comparisons, readers cannot tell whether most of the reported gains come from the proposed method or from GPTQ. Furthermore, AWQ is listed as a baseline in Table S1 without any results reported.
Questions for authors
  1. What is new about the core–tail observation compared with the findings in existing works?
  2. Does AdaPivot retain its advantage over non-uniform methods or formats?
  3. Please evaluate the actual inference cost.
  4. Please demonstrate the advantages over strong baselines.
Reviewer_fRN5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AdaPivot, an adaptive low-bit numerical format for LLM post training quantization. Based on the observation of dense cores and sparse tails existing in weight and activation distributions, it assigns separate representation granularity, boundary and dynamic range to these regions. The method further formulates format fitting as an importance-weighted reconstruction problem, combining discrete search with alternating parameter optimization, and develops an integer-constrained variant to reduce metadata overhead. The method is intergrated into the PTQ pipeline, including activation offline calibration and weights quantization via GPTQ. W4A4 experiments on Llama2 and Qwen3 show improvements over the baseline and comparative methods in most datasets. Additionally, W4A8 and W4A16 results are also reported.

Critical strengths
  1. The core-tail design has a clear motivation. The dilema between normal values and outliers is always a bottleneck in PTQ. This design offers a way to balance precision for most values with coverage of outliers by separating the regions. This could be useful for low-bit quantization format design.
  2. The reported results on W4A4 show substantial improvements on downstream tasks. Compared with SliderQuant, AdaPivot improves average task accuracy from 59.82% to 64.09% on Llama2-13B and from 63.28% to 70.76% on Llama2-70B.
Critical weaknesses

However, I do have some concerns regarding these parts:

  1. The evaluation does not establish the advantage of AdaPivot over existing Micro-scaling FP4 formats (e.g. MXFP4 or NVFP4, which is supported by existing hardware). The main results use per-output-channel and per-token quantization for weights and activations, respectively. They do not compare against MXFP4 or NVFP4 using their native grouping (32 or 16, respectively). It remains unclear whether the observed core-tail structure and the downstream task accuracy gain persist in smaller groups. These comparisons matter because this work proposes a new numerical format, but its benefits over existing FP4 formats are not demonstrated.
  2. The paper does not explain how its custom representation maps to existing low-bit matrix multiplication units. It is difficult to tell whether the accuracy gain justifies the cost of adopting this format in hardware.
Questions for authors
  1. Comparison with existing FP4 formats: could you provide accuracy comparisons with MXFP4 and NVFP4 using their native group size(32 for MXFP4 and 16 for NVFP4)? Please also specify the preprocessing and calibration settings.
  2. At what granularity are the distributions in Figure 1 collected? Per-tensor, per-channel, or another granularity? Does the core-tail structure remain within small groups of 16 or 32 elements after scaling? Could you provide W4A4 results with these smaller groups to show whether the accuracy gain remains?
  3. What datatypes and kernels are used for matrix multiplication in the reported experiments? Please clarify whether a kernel implementation is available and, if so, what the latency or throughput results are. If the experiments only evaluate accuracy through quantization simulation, please also state this explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.