acceptodds
Withdrawn from ICLR 2027

Breaking the Even Split: Code Allocation for 4-Bit Post-GELU Quantization in Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Static 4-bit post-training quantization leaves each activation tensor only sixteen codes, making their allocation a first-order design choice. In transformer MLPs, GELU confines negative outputs to a narrow, bounded interval, so nearly all of the output range lies on the long, unbounded positive side. Two-branch quantizers fit one range per sign yet split the codes evenly (), spending half of them on the narrow negative interval. Prior methods address this asymmetry but change the code allocation jointly with the grid, region boundaries, or scale search, leaving it without a principled rule or an isolated measurement. We treat code allocation as a design variable and ask whether it can be predicted from unlabeled calibration statistics alone. Minimizing the two branches' uniform-quantization error yields a closed-form rule that sets the allocation from each sign's calibrated mass and range, without labels, gradients, Hessians, or search. Across 84 post-GELU sites, the rule tracks the reconstruction-optimal allocation (Spearman , always within one code) and leave-one-setting-out aggregation recovers a model-wide 4/12 allocation (four negative, twelve nonnegative codes) in all seven settings, the split prior work fixed by design. Held fixed across 19 vision settings (five model families, three resolutions, two datasets), improves top-1 accuracy over the even split in 17, by points on average at identical BitOps, and outperforms promoting three sites to 8 bits at higher cost. Without retuning, it lowers WikiText-2 perplexity versus single-range and even-split quantizers on all six language models tested when only post-GELU activations are reduced to 4 bits. The same allocation at two-sided attention inputs degrades performance in both domains, tying the gain to GELU's bounded negative lobe. Code allocation can thus be read off calibration statistics, and changing it costs no additional BitOps.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
post-training quantization, low-bit quantization, activation quantization, transformer quantization, vision transformers, model compression, efficient inference, GELU

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nnm6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The writing quality is significantly below the bar. I would advise authors to make a complete work and carefully review before submission. This work doesn't follow usual scientific writing rules. The claims are just claims in this work not supported by evidences.

Critical strengths

NA

Critical weaknesses

This work's writing quality doesn't follow usual scientific writing rules and practices and significantly under the bar. Spending large amount of time for reading this work is difficult for reviewer. I managed to ignore the unusual presentation and focus on content/its idea. Unfortunately, after the reading the introduction, I still can't understand what is the method.

Questions for authors
  1. Presentation quality is significantly under the bar.

For example in abstract:

tatic 4-bit post-training quantization leaves each activation tensor only sixteen codes, making their allocation a first-order design choice.

In transformer MLPs, GELU confines negative outputs to a narrow, bounded interval, so nearly all of the output range lies on the long, unbounded positive side.

... there are more.

I would advise the authors to have LLM/native writer for reviewing the manuscript before submission.

  1. What is "Two-branch quantizers"?

Two-branch quantizers fit one range per sign yet split the codes evenly (8/8), spending half of them on the narrow negative interval.

  1. Abstract is not the usual abstract in top conferences. The work wrapers almost everthing into the abstract making it worse for reading.

  2. L39

Static post-training quantization (PTQ)

What is Static PTQ?

  1. L44-46

The GELU output, which feeds the second linear layer (fc2) of every transformer MLP, is a clear example. GELU is bounded below by−0.169971, so all negative outputs lie in an interval narrower than 0.17, whereas positive outputs are unbounded.

Why assume GELU?

The experiment (light) here is presented not well for scientific papers.

  1. Lacking experiments and baselines.
Reviewer_Rt3iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • 4-bits format provides 16 codes. This paper studies how to divide the 16 codes between negative and nonnegative post-GELU activations. They derive a closed-form allocation rule using calibrated probability masses and ranges under a uniform quantization MSE approximation. This paper focuses on integer-coded 4-bit quantizer, not FP4 or NVFP4.

  • The rule predicts local reconstruction-optimal allocations and, after aggregation across layers, generally recovers the 4/12 split already used in AMP-ViT.

  • Experiments compare fixed 4/12 against 8/8 across 19 vision settings and six language models. 4/12 improves top-1 accuracy by 1.09 points on average at identical BitOps.

Critical strengths
  • The allocation rule is simple and interpretable. In each layer, negative and non-negative sides received codes according to its probability mass and range. Table 8 shows that this align well will local reconstruction optima (compared with fixed k=4 configuration).
  • The experiments show that allocation matters in non-uniform post-GELU activation quantization. Fixed 4/12 allocation improves accuracy over 8/8 allocation in 17 of 19 vision settings, averaging 1.09 points. The paper also tested on many models across vision and languages that involves GELU.
Critical weaknesses
  • I think the evidence is insufficient to establish the contribution of the proposed allocation predictor beyond the existing fixed 4/12 allocation used in AMP-ViT. The accuracy results use fixed 4/12 and show improvements over 8/8, while Table 8 shows that the predictor is closer to the local MSE optimum. However, Table 4 shows that per-site MSE allocation does not consistently improve accuracy over fixed 4/12. The paper derives a shared allocation through aggregation across layers, but the motivation for this choice is unclear. These comparisons do not separate the contribution of the prediction rule from the effect of sharing an allocation across layers. It remains unclear whether the rule provides useful allocation selection beyond explaining the existing 4/12 choice.

  • The practical relevance of the proposed quantizer remains unclear. The current results do not establish when and why a user should prefer this quantizer over existing 4-bit alternatives, such as NVFP4, in terms of accuracy–latency tradeoffs.

Questions for authors
  1. Could the authors clarify the motivation for model-wide aggregation and separate its contribution from that of the prediction rule? How do prediction and exhaustive calibration-MSE search compare when both use one shared allocation, and when both allow per-site allocations?

  2. Do the prediction rule and aggregation heuristic generalize to other activation distributions or bit widths? Are there settings where the predictor selects a different allocation and provides an e2e accuracy benefits over fixed 4/12?

  3. Could the authors clarify when and why a user should prefer this quantizer over existing 4-bit formats, such as NVFP4? Is there evidence of a better accuracy–latency trade-off in a deployed implementation?

Reviewer_XvgROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose to assign more bits to positive values in post-GELU activations than to negative ones in order to get lower quantization error at equal memory cost.

Critical strengths

None

Critical weaknesses
  • Limited scope

    • The idea is very limited in scope, only considering post-GELU activations.
  • Missing end to end evaluation

    • There is no assessment of the end-to-end inference speed impact of the proposed method. How does a 4-bit uneven post-GELU compare to a 6-bit even split in terms of speed and output quality for example?

    • The proposed method complicates the execution framework, as uneven levels need to be supported. The main advantage of activation quantization is compute cost reduction. This is not shown here.

  • Lack of comparison to standard non-uniform quantizers

    • The most widely used 'uneven' level distribution would be a non-uniform quantizer with e.g. k-means level discovery. This is a baseline that is missing here. (fully non-uniform levels also generalize to any layer, not only post-GELU.) The proposed method has some potential advantage of being easier to execute on some hardware, but this needs to be shown. In short, we are missing a proper comparison to learned non-uniform levels both in terms of inference speed and output quality.
Questions for authors

How is this better than k-means-learned non-uniform quant. levels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.