acceptodds
Under review as a conference paper at ICLR 2027

Softmax Reparameterization For Output-Head Quantization

Abstract

Large vocabularies make language-model output heads a substantial inference cost in small language models. We introduce softmax reparameterization, a post-training method that searches over functionally equivalent output heads before quantization. The method subtracts a scalar multiple of the vocabulary-row mean from every output row and selects the coefficient by validation KL. For linear- softmax heads, these shifts preserve full-precision predictions exactly and require no decoder retraining; a rank-one correction extends the construction to nonlinear logit paths. Across seven output heads and three quantizers, W4 gains are largest where baseline quantization substantially distorts predictions: test KL falls by 93% on XGLM under RTN and by 73–77% on Phi, BLOOM, and BLOOMZ under activation-weighted MSE. Heads with low baseline error change little; at W2, used as a compression stress test, benefits extend more broadly. On Phi, the gains persist under stronger GPTQ calibration, and independent Phi and BLOOM holdouts reproduce the improvements. Frozen WikiText-selected coefficients also transfer to C4 and OpenWebMath. Residual analysis on Phi shows how fidelity can improve despite greater total logit error: the selected representative reduces error on likely outputs and lowers its Fisher-weighted cost. For shift-compatible heads, the shift adds no inference operation. With the decoder held in BF16, a packed W4 Phi output head reduces batch-one generation latency by 10.8%, and reparameterization preserves this speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Inference, Quantization, PTQ

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1fFxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose that the shift invariance of weights in a linear layer with softmax non-linearity can be used to improve quantization quality of this layer.

Critical strengths
  • The central idea is sound. The proposed shift leaves outputs equivalent (if used without quantization) and has the potential to change quantization results (similar to other ~preconditioning like rotations).
  • The selection methods are sensible: Simple centering as a baseline and validation KL based as a straight-forward improvement.
  • KL-divergence evaluations are more sensitive than the (unfortunately) commonly used multiple-choice accuracy evaluations.
Critical weaknesses
  • Is the impact of lm-head quantization relevant?
    • The method applies only to the lm-head.

    • Baselines are given only at few bitwidths, what about 2,3,4,5,6,7,8 bits w/o shift. How many bits does the shift save?

    • More generally speaking, the evaluation scope leaves a critical blindspot: It is unclear how relevant the contribution is. To be able to assess this, the authors must supply a comparison of bits saved vs. kl-damage inflicted under different quantization approaches. Currently, many methods simply leave the lm-head at 6 or 8 bits (or maybe even 16) and assume further marginal space savings (compared to the savings from quantizing everything else) are not worth the trouble. A pareto-plot of model size vs KL to baseline under different encoder and lm-head quantizations (and not only at 4, but also at 6 or 8 bits) would be very informative. This would help answer questions like: How much can the authors shrink e.g. a model quantized with encoder@4bits, lm-head@6 bits and does it retain its quality?

Questions for authors

My key concern is that it is difficult to assess the relevance of the contribution (see above). How important an issue is lm-head quantization? Explain in detail how many bits / parameter (of the whole model!) you can save at equal quality thanks to this idea. My ball-park estimation is that it is irrelevant, but I am ready to be convinced otherwise.

15 comments

We thank the reviewer for raising the practical relevance of output-head quantization. The revised manuscript now includes a whole-model size–quality comparison spanning W2–W8 output-head precision with fixed quantized decoders (Section 3.3, Table 3, (new) Figure 3 w/ full detailed run results in Appendix K.2).

Practical relevance: In our packed Phi experiment, replacing the BF16 output head with a W4 AW-MSE head reduces batch-one 64-token generation latency from 1136.6 to 1013.4 ms, a 10.8% reduction, and batch-16 latency from 1335.8 to 1210.3 ms, a 9.4% reduction. Reparameterization uses the same packed W4 inference path and preserves this speedup. These measurements establish an end-to-end decode benefit and we do not claim the same speedup for prefill-dominated workloads or arbitrary quantized decoders. For context, output-head cost can be substantial in current SLMs because reducing decoder size does not require a proportional reduction in vocabulary. Phi-4-mini, for example, uses a roughly 200K vocabulary, and prior scaling-law work also identifies vocabulary size as an independent quality-relevant scaling dimension. A dense output projection reads all (Vd) weights at every generated token, unlike an input embedding lookup that accesses selected rows.

Whole-model storage qualification: The reviewer is correct that head bitwidth alone does not establish model-level savings, especially for tied embeddings. In our evaluated Phi layout, the original BF16 input/output embedding is shared. Quantizing the output head independently therefore retains the BF16 input embedding and adds a packed output matrix. We do not present this configuration as a model-size reduction; its demonstrated benefit is reduced output-projection traffic and generation latency.

To quantify the storage–fidelity trade-off directly, we added a size–quality comparison with fixed GPTQ decoders and W2–W8 GPTQ output heads. The accounting includes retained embeddings, duplicated output matrices, and head scales. Test KL is measured against the original BF16 model.

For Phi with the BF16 decoder, validation-selected W4 GPTQ lowers test KL from 0.0929 to 0.0349, while unshifted W5 reaches 0.0250 and W6 reaches 0.0065. We therefore do not claim a two- or four-bit saving at W6/W8 fidelity. The supported result is that representative selection improves the W4 fidelity operating point at unchanged W4 payload.

With a fixed W8 Phi decoder, selected W4 reduces end-to-end KL from 0.1989 to 0.1338, while unshifted W5 gives 0.1255. For an illustrative end-to-end KL budget of 0.15, selected W4 satisfies the budget whereas unshifted W4 does not, replacing W5 with one fewer head bit. Under this weight-payload accounting, that corresponds to 0.160 whole-model bits/parameter, or 73.3 MiB, approximately 1.6% of the W8-decoder/W5-head weight payload. The 0.15 threshold is a descriptive reading of the measured curve, not a pre-specified quality threshold or a claim of identical KL.

Thus, with retained tied embeddings, the demonstrated benefit is reduced output-projection traffic and generation latency. Among independently stored output heads, representative selection improves the W4 fidelity point and can reduce the precision required under a specified error budget.

Taken together, the measured 10.8% batch-one latency reduction and the new model-level size–quality comparison show that output-head quantization can have a measurable systems impact, while the storage benefit depends on the tied-weight layout and target error budget. The manuscript also makes these deployment constraints explicit, including tied-weight separation, packed-head storage, and the distinction between reduced output traffic and resident model memory (Appendix G). Prior scaling-law work further shows that compute-optimal vocabulary size can increase with model scale, and Figure 1 illustrates that several contemporary SLM families already use substantially larger vocabularies than earlier designs. We hope these additions address the reviewer’s concern about the practical relevance of the contribution.

Reviewer_1fFxOpenReview ↗

quick follow up

The response does not address my concerns. Please clarify the following:

  1. I think there is a misunderstanding. The issue is not that you need to sweep the head bits in a more fine-grained way; rather, it is not clear whether reducing bits specifically in the head layer is really better than in some other layer. Does the proposed method meaningfully save bits / get a speedup when we could quantize elsewhere?

  2. The Table in app K2 shows that your models are larger than the original (because the embedding table and head need to be stored separately?). This further pushes the proposed method into a very setup-specific corner. Agreed?

  3. Please describe under what constraints your method can be expected to yield a benefit, and list when this applies in practice. E.g. say among typical devices in {phone, laptop, desktop, server} and serving bottleneck in {disk space, compute speed, memory space, memory bandwidth} (or any other relevant), especially also list in which of these cases it is not applicable.

AuthorsOpenReview ↗

Complementary to mixed-precision decisions about the rest of the network, rather than a replacement for them.

Point 1: If we understand correctly, the reviewer is asking whether quantizing the output head is the best place to spend a global bit or latency budget. Rather, our motivation is that output heads are often left at higher precision because they can be unusually sensitive to quantization, even when the rest of the model is compressed. Softmax reparameterization reduces the fidelity cost of quantizing this remaining component.

This motivation also comes from our experience deploying SLMs under strict inference-time SLAs, where reducing output-projection cost provided useful latency headroom beyond decoder-side optimization. In the controlled deployment experiment reported in the paper, with the decoder held fixed in BF16, quantizing the output head reduces end-to-end generation latency by 10.8% at batch 1 and 9.4% at batch 16. Thus, our claim is when a deployment chooses to quantize the output projection, softmax reparameterization can make that choice substantially less costly in fidelity without changing the head bitwidth or, for a shift-compatible linear head, adding an inference operation. It is complementary to mixed-precision decisions about the rest of the network, rather than a replacement for them.

Point 2: We agree that tied embeddings change the storage accounting. In our implementation, independently quantizing the output projection requires retaining the input embedding and storing a separate packed output matrix. The W4 output head adds approximately 0.30 GiB of weight storage. Thus, this configuration is not primarily a model-size optimization.

The trade-off is modest additional resident storage for lower per-token output-projection traffic. In our measured Phi deployment, that trade gives approximately 10% lower generation latency. Appendix H describes how we separate the tied embedding and output projection and pack the latter for serving in vLLM. Importantly, the reparameterization itself adds no storage relative to an ordinarily quantized output head; the storage overhead comes from separating the tied weights.

Point 3: The applicable use case is therefore latency-sensitive autoregressive serving where decode-time memory bandwidth or output-projection cost is significant, especially server-side query processing, interactive generation, and low-to-moderate-batch serving. Large-vocabulary SLMs are particularly relevant because the output projection can remain large even after the decoder has been substantially compressed, while the additional packed head is modest in absolute size.

The same trade-off can be useful for local inference on desktops and laptops when an efficient low-bit output-projection kernel is available. It is less attractive on tightly memory-constrained devices such as phones when breaking weight tying materially affects the memory budget. We will add this distinction in the paper and also note this limitation for memory constrained devices. It is also less relevant for prefill-dominated workloads, very high-batch serving, small output heads, or systems where attention, KV-cache traffic, decoder computation, or communication dominates latency. We will add this distinction in the paper and also note this limitation for memory constrained devices.

Thus, the intended regime is not universal model compression. It is a targeted optimization for deployments where the output projection remains a meaningful decode-time bottleneck and where a small additional packed representation is worthwhile in exchange for lower bandwidth and latency.

We provide the table as requested by reviewer that follow our comments above:

DeploymentPrimary bottleneckExpected benefitApplicability
Phone / constrained edgeMemory / package sizeSeparate packed head may increase footprintLimited / conditional
Laptop, local inferenceInteractive decode latencyLower output-head bandwidth at low batchGood use case
Desktop / workstation GPUDecode bandwidth / latencyEfficient packed W4 projection can reduce per-token costGood use case
Server, latency-sensitive queriesDecode latency / memory bandwidthLarge output projection is read every generated tokenStrongest use case
Server, high-batch throughputCompute / attention / KV / communicationHead traffic is more heavily amortizedConditional
Reviewer_A4XNOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a Softmax reparameterization method designed for the quantization of output heads in large language models (LLMs). By leveraging the translation invariance of the Softmax function, the method identifies a weight representation better suited for quantization—through a one-dimensional parameter search—while preserving the original full-precision prediction distribution, all without the need for model retraining. The authors conducted experiments across seven types of model output heads and three quantization methods, demonstrating that this approach effectively reduces the prediction distribution errors introduced by quantization. Furthermore, the study evaluates the method’s performance and underlying mechanisms through practical deployment and error analysis. Overall, the proposed method features a concise design, a clear theoretical foundation, and robust experimental validation, offering significant practical utility.

Critical strengths
  1. The method is concise and grounded in a clear theoretical basis. By leveraging the translation invariance of the Softmax function, the approach improves the quantization fidelity of the output head through a simple one-dimensional parameter search, all while strictly preserving the original prediction distribution. This method requires neither model retraining nor modifications to the existing decoder.
  2. The method offers clear practical utility and a well-explained mechanism. It significantly reduces quantization errors for certain models without increasing the computational cost of inference for standard linear Softmax output heads, thereby preserving the inference speedups gained from low-bit quantization in practical INT4 deployments. Furthermore, the authors provide valuable theoretical insight by explaining the improvement in prediction fidelity through Fisher-weighted error analysis.
Critical weaknesses
  1. The core method of the paper is: $ W_t = W - t\mathbf{1}\mu^\top $ The benefits of this newly introduced operation exhibit significant model-dependent variation. In XGLM, even without additional search, fixed-mean centering already achieves the full improvement. This raises a research-level question: has the author discovered a universally meaningful quantization reparameterization strategy, or simply performed an effective hyperparameter search on a few quantization-sensitive output heads?
  2. The paper primarily keeps the Transformer Decoder in BF16 and only quantizes the Output Head. This experimental design is reasonable as it allows for a very clean study of output head quantization. However, practical deployment often requires quantizing both the Decoder and the Output Head simultaneously. In such cases, we have: $ h_{\mathrm{BF16}} \neq h_{\mathrm{W4}} $. Although the full-precision Softmax equivalence still holds for any hidden state, the quantized probability distribution error depends on the hidden state. Therefore, my concern is that the optimal $ t^* $ selected on the BF16 Decoder may not remain applicable to the quantized Decoder.
Questions for authors
  1. I suggest that the authors to further investigate: why is the optimal coefficient 1 for some models, 4 for others, and even negative for some? This would not only strengthen the novelty argument but also move the method from empirical search toward more interpretable parameter selection.
  2. Could there be a simple full-model quantization experiment, comparing the original quantization, fixed centering, and the proposed method, and considering two coefficient selection schemes: directly reusing the $ t^*$ selected from BF16 activations, and re-selecting on the quantized Decoder activations. This would help distinguish the effectiveness of the method itself from the sensitivity of parameter selection to the hidden state distribution.
2 comments

We thank the reviewer for the questions about coefficient variation and selection under decoder quantization. The revised manuscript now reports quantized-decoder experiments in Section 3.3 and Appendix K.3.

Why does the selected coefficient vary:

The exact equivalence class is general, but the best quantized representative need not be. For a linear-softmax head,

Wt=W−t1μ⊤W_t = W - t\mathbf{1}\mu^\top

preserves the full-precision output distribution for every (t)(t). However, changing (t) changes quantization ranges, clipping and rounding residuals, while the predictive cost of those residuals depends on hidden states and source probabilities. These quantities vary across models and quantizers. Thus (t=1)(t=1), which minimizes the full-precision Frobenius norm along this family, need not minimize post-quantization predictive error, as we demonstrate in our paper.

Furthermore, we do not currently claim an analytic predictor of �0��0� We deliberately use a one-dimensional scalar search because it is simple, inexpensive, and sufficient to expose large differences among exactly equivalent heads. The equivalence class itself is much higher-dimensional as shown in Appendix J, which explores a higher-capacity group-wise parameterization with separate coefficients aligned to quantization groups and finds additional gains under RTN and AW-MSE, including in a matched-budget SmolLM3 comparison. Thus the scalar sweep should be viewed as a practical low-cost search strategy rather than a claim that one scalar is the optimal parameterization of the equivalence class. Developing analytic or learned predictors of �1��1� is an interesting direction beyond the present work.

XGLM is a useful control: its selected coefficient is (t=1), so fixed mean-centering already obtains the full gain. Phi differs: under W4 AW-MSE, test KL is 0.936 at (t=0)(t=0), 0.577 at (t=1)(t=1), and 0.256 at the validation-selected coefficient. Negative coefficients are likewise valid members of the same equivalence class rather than evidence for a universal negative-shift mechanism.

Selection with a quantized decoder: We tested the reviewer’s proposed comparison using a Phi W8A16 GPTQ decoder and a Qwen3.5 W4A16 GPTQ decoder. For each fixed decoder, we compare (t=0)(t=0), (t=1)(t=1), the coefficient selected from BF16-decoder states, and a coefficient reselected on quantized-decoder validation states. The head quantizer itself is fitted using states from the fixed quantized decoder in both cases; only the scalar coefficient is transferred. We use W8 for Phi because W4 Phi decoder constructions were already severely degraded even prior to output-head quantization; these failed controls are documented in Appendix K.1.

For Phi, both procedures select (t=4)(t=4) for RTN, AW-MSE and GPTQ. With the GPTQ head, end-to-end KL falls from 0.1989 at (t=0) to 0.1338 at (t=4), compared with 0.1659 under fixed mean-centering. The paired 95% interval for selected-minus-raw KL is ([-0.0719,-0.0585]). Thus, in this setting, the BF16-selected coefficient transfers to the quantized decoder without reselection.

For Qwen, both procedures select (t=−0.5)(t=-0.5) for all three head quantizers. The GPTQ head is already highly faithful and end-to-end KL changes from 0.05038 to 0.05006, with a paired 95% interval of ([-0.00082,0.00020]), so we do not claim a statistically resolved GPTQ improvement. RTN gives a larger resolved reduction.

Reselection minimizes head KL relative to the original head on the same quantized-decoder states; the reported end-to-end KL instead uses the original BF16 model as reference. Fitting, validation, and test articles are disjoint.

These results show coefficient transfer in the two tested decoder settings, while also reinforcing the paper's broader pattern i.e. representative selection has the largest practical effect when the base head quantizer leaves substantial distortion.

My concerns have been addressed. The current score already effectively reflects the strengths of the paper; therefore, I will maintain the original score.

Reviewer_Fks1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a reparameterization for the vocabulary of the output head with the aim of preserving the output distribution in quantization scenarios. In order to do so, the authors introduce a post-hoc process with a flexible recentering mechanism. The results support the reasonably successful preservation.

Critical strengths
  • Wide set of experiments in different heads and quantizers.
  • Article is clear and the proposal is simple. This last feature is important because it allows to be easily accommodated in real architectures.
Critical weaknesses
  • The KL objective minimization is a legit quantization objective but as a global score it may obfuscate different effects, such as rare token prediction or tail effects.
Questions for authors
  • In general the study is correctly performed and convincing. It would be interesting to explicitly state the trade-offs of the technique in quantization scenarios as well as to emphasize its limitations.
  • The main question I would pose regards the weakness formerly described; the KL objective minimization is a legit quantization objective but as a global score it may obfuscate different effects, such as rare token prediction or tail effects. In this respect, how the authors may address such effects?
2 comments

We thank the reviewer for raising the concern that average KL could hide degradation on rare or low-probability outputs. We therefore added three complementary tail evaluations, all with the BF16 decoder fixed so that they isolate output-head quantization (Section 4.2 and Appendix K.4).

Low-probability outputs: The Fisher analysis in Section 4.2 lets us localize where the fidelity improvement comes from. For Phi W4 AW-MSE, outputs with (p≥10−2)(p\ge10^{-2}) carry 84.8% of the source probability mass and explain 71.8% of the reduction in Fisher cost; outputs with (10−6≤p<10−2)(10^{-6}\le p<10^{-2}) explain another 27.9%. Together, these regions account for 99.7% of the reduction. In contrast, visible logit error more than doubles for (p<10−6)(p<10^{-6}), but this region carries only 0.20% of source probability mass and contributes just 0.3% of the Fisher-cost reduction. RTN shows nearly the same attribution, 73.2% and 26.5%. Thus the selected representative reduces error on outputs to which the source model assigns meaningful probability while allowing more reconstruction error in extremely low-probability coordinates. This attribution is exact for the Fisher quadratic, which closely tracks measured KL here: at (t=4), it differs from actual KL by only 6.2% for AW-MSE and 1.4% for RTN.

Rare ground-truth tokens: Low conditional probability and corpus rarity are different notions, so we separately stratify held-out ground-truth NLL by token frequency in the fitting articles: unseen, rare (1–9 occurrences), medium (10–99), and common (100+). These are fitting-corpus counts, not estimates of pretraining frequency. For Phi W4 GPTQ, the 1,018 rare-token positions improve from 3.107 to 3.011 NLL. Selected-minus-raw NLL is (-0.096) nats, with a paired article-bootstrap 95% interval of ([-0.153,-0.050]). For the 394 unseen-token positions, the interval includes zero, so we do not claim a resolved improvement there.

Prediction tails: We also measure the distribution of position-level KL rather than only its mean. On Phi GPTQ, the selected representative reduces the 99th-percentile KL from 0.559 to 0.244 and raises top-1 agreement with the source model from 82.58% to 89.11%. Thus the fidelity improvement is present in the tail of per-position distortion as well as in mean KL.

The effect is not uniform across models or token subsets. On Qwen, changes are small and mixed; under AW-MSE, for example, mean KL improves slightly while common-token NLL worsens significantly. We therefore do not claim that lower mean KL guarantees improvement for every frequency stratum, preservation of every rare token, rare-token recall, or improved downstream rare-event accuracy. This is precisely why we report these tail-sensitive diagnostics in addition to mean KL.

Trade-offs: Selection adds offline search cost, requiring up to 14 candidate head constructions per base quantizer. For shift-compatible linear-softmax heads, however, the selected representative adds no inference operation. The gains are also concentrated where baseline head quantization introduces substantial distortion and we observe that heads that are already well preserved generally change little.

Taken together, the Phi results show that the observed KL gains are not merely an average-score artifact: they are independently supported by probability-weighted error attribution, rare-token likelihood, and the tail of per-position KL. The Qwen results provide the complementary limitation: improved average fidelity should not be interpreted as a guarantee of improvement for every token subset.

We have incorporated these analyses into the revised manuscript, which now reports tail-sensitive fidelity alongside mean KL and makes the method’s practical costs and limitations explicit.

My questions are addressed. I ratify the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for reviewing and providing feedback towards improving the paper

We thank the reviewers for their constructive feedback. We were encouraged that the reviewers generally found the central idea sound and the paper clear, while raising three important questions: whether output-head compression matters at the whole-model level, whether the selected coefficient remains useful when the decoder is quantized, and whether aggregate KL can hide undesirable behavior on rare or low-probability tokens. To address these, we added quantized-decoder experiments, W2–W8 precision–quality sweeps with whole-model storage accounting, and frequency-stratified likelihood and prediction-tail evaluations. The revised manuscript reports these in Section 3.3, Section 4.2, and Appendix K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.