acceptodds
Under review as a conference paper at ICLR 2027

Density-Matched Power-of-Two Quantization: Allocating Logarithmic Levels for Multiplier-Free LLM Inference

Abstract

Large language model (LLM) inference spends most of its compute in the multiply-accumulate (MAC) operations of linear projections. Post-training quantization (PTQ) reduces this cost with low-bit integer weights and activations, leaving a multiplier in every MAC. Power-of-Two (PoT) weight quantization replaces that multiplier with a shifter but spaces its levels a factor of two apart, degrading accuracy at low bit-widths. Finer fixed-ratio grids narrow the gap but remain suboptimal, since they spend the same number of levels per doubling of magnitude, wherever the weights lie. We introduce **Density-Matched Power-of-Two (DMPoT)** quantization, which formulates logarithmic level allocation as a continuous point-density optimization problem. We derive the density that minimizes quantization mean squared error (MSE) and use it to guide the construction of a multi-piece family of fractional PoT grids whose deployable members support folded shift-add hardware. Three results follow. (i) The derived density concentrates quantization resolution on the upper flank of the weight distribution while leaving the bottom of the range sparse. (ii) The optimal grids at successive bit-widths nest as sets of levels into one chain, and two bit-width laws fix the operating regime at four to five bits: a one-adder-per-weight bound and a width-minimality result at INT-matched MSE. (iii) Every deployed 4-bit level is an integer shift of or of , so a folded datapath computes the single addition once per activation, shared by all output channels. Across seven models and two configurations, DMPoT matches the uniform integer baseline in all fourteen cells, never more than perplexity above it, with mean gaps of and . The synthesized MAC array with 4-bit weights and activations occupies the area of its integer counterpart.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLMs), Post-Training Quantization (PTQ), Power-of-Two (PoT) Quantization, Hardware-Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4FbDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DMPoT, a weight quantization method that adjusts the allocation of logarithmic levels under hardware constraints. Its deployed 4-bit grid allows a shared 3x computation at the array input, followed by selection, shifts, and accumulation in each processing element. The method uses learned rotation and smoothing with GPTQ calibration, while activations and the KV cache use uniform INT. Experiments on seven models report accuracy close to uniform INT and lower perplexity than FPoT. The hardware evaluation reports smaller synthesized MAC-tile area.

Critical strengths

The grid supports sharing computation across output channels. For the 4-bit design, 3x is computed once per input lane and reused across the tile. This provides a concrete way to reduce repeated arithmetic (Sec. 5; App. E).

The reported accuracy is close to uniform INT. The main results cover seven models and two quantization configurations, with three calibration seeds. DMPoT has lower perplexity than FPoT in all fourteen comparisons and similar perplexity to APoT (Table 1).

The ablations help explain the design choices. The paper examines preprocessing, rounding, grid approximation, and calibration settings. It also evaluates different combinations of preprocessing and deployed grids, which helps distinguish their effects (App. H).

Critical weaknesses

The theoretical and format novelty is limited. The point-density analysis follows existing quantization theory, the deployed 4-bit values match scaled E2M1, and calibration and grid alignment are inherited from FPoT. The main contribution is the grid selection under hardware constraints and its hardware implementation.

Table 1 should include a full-precision baseline and comparisons with other INT quantization methods. BF16 accuracy is reported in the appendix, but the main table should include the full-precision accuracy and perplexity to make the absolute quantization loss clear. Most comparisons use the same preprocessing and GPTQ pipeline, so the comparison with other INT quantization approaches remains limited.

Synthesis area alone is insufficient to establish the implemented area savings. The paper reports Yosys mapped-cell area without timing constraints, placement and routing, or clock-tree synthesis. The area comparison should use the die area obtained after completing the full physical design flow. The current results do not establish whether the savings in Figures 3(a) and 3(c) will hold after physical implementation.

The proposed ASIC design lacks implementation-level validation. The hardware evaluation stops at tile synthesis. There are no post-layout simulation results, silicon measurements, or FPGA results. Practical power, performance, and area (PPA), end-to-end numerical accuracy, and inference throughput are therefore not validated in an implemented system.

The area savings are sensitive to the INT baseline. With the Booth INT baseline reported in the appendix, the W4A4/W4A8 savings decrease from 29.8%/38.2% to approximately 17.2%/25.4%. This dependence should be made clearer in the main results.

Questions for authors

Please address weaknesses above.

Also, were the accuracy and perplexity results in Table 1 obtained entirely through PyTorch inference, or was inference evaluated using the synthesized netlist? Please clarify how the hardware's numerical behavior was checked against the software evaluation.

Reviewer_HET2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focus on weight-and-activation post-trained quantisation for LLM that that can be run on pure-integer, shift-add only, multiplier-free chips. The core contribution falls in the research of Power-of-2 method, which limit the weights and activation to be only the power-of-2, so that all computaiton can be done by SHIFT and ADD. One problem of PoT method is that the representation is far too caurse, resulting in huge quanlity degradation.

Previous work FPoT offered a solution to represting data udner base-2\sqrt{2} instead of base-22, with a trick of using x + x >> 1 = 1.5x to approximate sqrt(2)x=1.414x. However, FPoT still adopted the log-uniform assumption: levels are placed uniformly in each octave. DMPoT challenged this assumption, saying that levels should be placed according to the weight distribution. After thoropgh theoretical analysis, the authors derived that whtn the model weights follow Gaussian Distribution, the MSE-optimal log-domain level density is concentrated on its upper flank rather than at the mode of the weight distribution.

Concequently, the less levels should be places on the bottom octave and more should be assigned to the top octave. For example, under 4-bit settings, the schedule should be {1, 2, 3, 4, 6, 8, 12}, against the FPoT log-uniform schedule {1, 1.5, 2, 3, 4, 6, 8} .

Under the hardware constraint of "one-adder-per-weight" budget, the deployable bit-witdth should be b≤5b \le 5, and b=4b=4 is "the efficiency sweet spot", whose MAC tile reports 0.702×0.702\times the area of an equivalent INT4 tile, and 77% less MAC tile savings against APoT, its most competitive rival.

Critical strengths

The motivation of the idea is clean and straight-forward. Following a clear line of PTQ, PoT, APoT, FPoT, the authors guide readers to challenge and the log-uniform assumption of FPoT and begin to state the analysis and method of DMPoT. The contribution is well-scoped. Ideas and methods are backed with theoretical analysis and derivation.

In constrast of providing empirical observation of heuristcs, a theory is derived to not only explain the motivation and design choices, but also offer theoretical guide to the log-level domain allocation problem.

Reproducibility is highly respected, where checkpoint revisions, learning rates, seeds, harness versions, Triton kernel description, and runtime are reported in Appendix.

Critical weaknesses

From the empirical experiments, whether PPL or 0-shot accuracy, DMPoT does surpass its predecessor FPoT, but does not perform as good as APoT in most cases. In temrs of hardware efficiency, DMPoT does save 7% less area than APoT. However, such comparison is relatively weak, we'd like to see end-to-end chip power or time evidence, rather than simply MAC array savings.

Although the author derived a theory to explain and guide log-level domain allocation, it still lacks predictive power. As acknowledged in Appendix B.10, the deriveation only serve as a theoretical guide or a design pronciple. The exact schedule sitll requires an exhaustive search. And under the 4-bit setting, "the efficiency sweet spot" and the focus of this paper, exhaustive search remains cheap. This is a pitty that weakens the predicative value of the nice theoretical resules.

Furthermore, as the Limitation section and Appendix B.10 confessed, the density result of the theory derivation is for unweighted MSE under a Gaussian proxy, while the deployed pipeline minimizes a different objective (GPTQ + OSTQuant KL). We noticed the authors use the proxy only to rank grids and the measured density peak (+0.795) matching theory (+0.79), but this is still suggestive yet does not close the gap.

Models in Experiments are a bit outdated. In the Expertiment section, the author used 7 models from 1.5 to 14B, but they are from Llama2/3, Mistral, and Qwen2.5 series. There are far newer open-source models, like Llama3.1, Ministral 3, Qwen3 and even Qwen3.5/3.8 series. We believe reporting experiment results on latest models makes the conclusion cutting-edge. What's more, all the models used in this paper are standard attention Dense Transformer. Current open-source models tends to have hybrid attention modules, like Sliding-Window Attention, Native Sparse Attention, and Linear Attention. Could the model perform robust as well in all these trending variants? That remains a question.

Questions for authors
  1. Could the author report power or time savings, in addition to MAC tile area savings of DMPoT against APoT udner the save budget to quantify the relative hardware advantage of DMPoT?

  2. As the theory optimised Gaussian MSE objective while the experiments uses the GPTQ+ OSTQuant KL objective, could the authors also report the exhaustive schedule search schedule results in the Experiment section under the deployed objective GPTQ + OSTQuant KL?

  3. As discussed in the previous sections, the authors are suggested to report power or time savings of DMPoT and complete the experiments with newer models, like Llama3.1, Ministral 3, Qwen3.5/3.8 series.

  4. Additional feedback:As an important citation (perhaps the most important one), FPoT should not be cited as (Anonymous, 2026).

1 comment

Thanks for the detailed follow-up. The Q2 experiment — running all 19 b=4 schedules end-to-end under GPTQ + KL — is exactly the evidence I asked for, and the Spearman 0.99 agreement between the proxy and deployed ordering resolves my central concern that the theory might be lack of predictive power. The post-route data in Q1 also addresses my earlier point that the hardware evidence was gate-level only.

The practical case nevertheless remains narrower than the framing: the theory predicts a schedule that a cheap search would find anyway, and that schedule ties APoT on accuracy; on ASAP7, which is the relative more advanced node, the throughput advantage is 0.99×, showing that the advantage does not hold; and the near-lossless claim remains for the standard attention transformer models, which may not be applicable for future hybrid attention models (at least the author did not explain). Therefore I will maintain my score.

Reviewer_TB3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

DMPoT revisits power-of-two weight quantization through distribution-matched allocation of logarithmic reconstruction levels. A high-resolution log-domain MSE analysis motivates placing more grid levels where large-magnitude weights make quantization error expensive. The deployed four-bit levels are proportional to {0, ±1, ±2, ±3, ±4, ±6, ±8, ±12}; an odd-core {1, 3} representation permits shifts and a shared 3x activation precomputation rather than per-weight multipliers. Coupled with OSTQuant-style preprocessing and GPTQ, DMPoT is near-lossless against uniform INT across seven models and two four-bit configurations, with lower perplexity than FPoT in all 14 cells. The synthesized W4A4 MAC tile occupies 0.702 times the area of the matched integer tile.

Critical strengths
  1. The paper presents a relatively complete co-design framework. Starting from the optimal point-density analysis, it constructs a discrete logarithmic quantization grid, develops a single-adder approximation, and proposes a folded hardware implementation. The overall technical progression is coherent.
  2. The evaluation is broad and generally convincing. The main experiments cover seven language models, two activation/KV-cache configurations, and three calibration seeds. DMPoT achieves competitive results across all fourteen main experimental settings.
  3. The hardware analysis and reproducibility efforts are relatively thorough. Moving the coefficient-product computation to the edge of each activation lane and sharing it across multiple output channels is a practically meaningful design. The paper also provides SAT-based equivalence verification, area evaluations across different array sizes, and cross-checks using multiple synthesis-tool versions and standard-cell libraries.
  4. The paper is clearly written and appropriately qualifies its conclusions. It distinguishes between results derived theoretically, obtained through discrete search, and supported empirically. The appendix is also comprehensive.
Critical weaknesses
  1. The hardware validation remains incomplete. The current results are based on untimed, unrouted combinational-logic area synthesis. The effects of additional broadcasting and fan-out on post-layout area, operating frequency, and power have not yet been evaluated.
  2. The independent contribution of density matching to end-to-end performance requires further validation. The paper explains its relationship to FPoT and E2M1 and uses a fixed-ratio control matched in span and zero fraction to demonstrate the value of non-uniform density allocation under the Gaussian proxy. However, this key control has not been extended to end-to-end model evaluation. Consequently, direct controlled evidence is still needed to determine how much of the observed perplexity improvement is attributable specifically to density allocation, as opposed to differences in range, clipping, or preprocessing adaptation.
  3. The distributional assumption is validated on a limited set of models. The Gaussian proxy is directly examined primarily on Llama-3-8B. Results on the density peak, level occupancy, or proxy-MSE ordering for additional models would provide stronger evidence for the generality of the proposed approach.
Questions for authors
  1. Could the authors provide timing-constrained synthesis or preliminary place-and-route results, and discuss how the additional broadcasting and fan-out affect area, operating frequency, and power?
  2. Could the authors extend the fixed-ratio control matched in span and zero fraction from the Gaussian-proxy analysis to end-to-end model experiments, reporting perplexity and downstream accuracy under the same preprocessing transform and calibration protocol? This would help determine whether the advantage of density allocation itself translates into practical model-level improvements.
  3. Beyond Llama-3-8B, do the other evaluated models exhibit similar optimal density peaks, level occupancy patterns, and proxy-MSE rankings? Could the authors provide corresponding results to support the generality of the method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.