- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose that the shift invariance of weights in a linear layer with softmax non-linearity can be used to improve quantization quality of this layer.
Critical strengths
- The central idea is sound. The proposed shift leaves outputs equivalent (if used without quantization) and has the potential to change quantization results (similar to other ~preconditioning like rotations).
- The selection methods are sensible: Simple centering as a baseline and validation KL based as a straight-forward improvement.
- KL-divergence evaluations are more sensitive than the (unfortunately) commonly used multiple-choice accuracy evaluations.
Critical weaknesses
- Is the impact of lm-head quantization relevant?
-
The method applies only to the lm-head.
-
Baselines are given only at few bitwidths, what about 2,3,4,5,6,7,8 bits w/o shift. How many bits does the shift save?
-
More generally speaking, the evaluation scope leaves a critical blindspot: It is unclear how relevant the contribution is. To be able to assess this, the authors must supply a comparison of bits saved vs. kl-damage inflicted under different quantization approaches. Currently, many methods simply leave the lm-head at 6 or 8 bits (or maybe even 16) and assume further marginal space savings (compared to the savings from quantizing everything else) are not worth the trouble. A pareto-plot of model size vs KL to baseline under different encoder and lm-head quantizations (and not only at 4, but also at 6 or 8 bits) would be very informative. This would help answer questions like: How much can the authors shrink e.g. a model quantized with encoder@4bits, lm-head@6 bits and does it retain its quality?
-
Questions for authors
My key concern is that it is difficult to assess the relevance of the contribution (see above). How important an issue is lm-head quantization? Explain in detail how many bits / parameter (of the whole model!) you can save at equal quality thanks to this idea. My ball-park estimation is that it is irrelevant, but I am ready to be convinced otherwise.
15 comments
quick follow up
The response does not address my concerns. Please clarify the following:
-
I think there is a misunderstanding. The issue is not that you need to sweep the head bits in a more fine-grained way; rather, it is not clear whether reducing bits specifically in the head layer is really better than in some other layer. Does the proposed method meaningfully save bits / get a speedup when we could quantize elsewhere?
-
The Table in app K2 shows that your models are larger than the original (because the embedding table and head need to be stored separately?). This further pushes the proposed method into a very setup-specific corner. Agreed?
-
Please describe under what constraints your method can be expected to yield a benefit, and list when this applies in practice. E.g. say among typical devices in {phone, laptop, desktop, server} and serving bottleneck in {disk space, compute speed, memory space, memory bandwidth} (or any other relevant), especially also list in which of these cases it is not applicable.
Complementary to mixed-precision decisions about the rest of the network, rather than a replacement for them.
Point 1: If we understand correctly, the reviewer is asking whether quantizing the output head is the best place to spend a global bit or latency budget. Rather, our motivation is that output heads are often left at higher precision because they can be unusually sensitive to quantization, even when the rest of the model is compressed. Softmax reparameterization reduces the fidelity cost of quantizing this remaining component.
This motivation also comes from our experience deploying SLMs under strict inference-time SLAs, where reducing output-projection cost provided useful latency headroom beyond decoder-side optimization. In the controlled deployment experiment reported in the paper, with the decoder held fixed in BF16, quantizing the output head reduces end-to-end generation latency by 10.8% at batch 1 and 9.4% at batch 16. Thus, our claim is when a deployment chooses to quantize the output projection, softmax reparameterization can make that choice substantially less costly in fidelity without changing the head bitwidth or, for a shift-compatible linear head, adding an inference operation. It is complementary to mixed-precision decisions about the rest of the network, rather than a replacement for them.
Point 2: We agree that tied embeddings change the storage accounting. In our implementation, independently quantizing the output projection requires retaining the input embedding and storing a separate packed output matrix. The W4 output head adds approximately 0.30 GiB of weight storage. Thus, this configuration is not primarily a model-size optimization.
The trade-off is modest additional resident storage for lower per-token output-projection traffic. In our measured Phi deployment, that trade gives approximately 10% lower generation latency. Appendix H describes how we separate the tied embedding and output projection and pack the latter for serving in vLLM. Importantly, the reparameterization itself adds no storage relative to an ordinarily quantized output head; the storage overhead comes from separating the tied weights.
Point 3: The applicable use case is therefore latency-sensitive autoregressive serving where decode-time memory bandwidth or output-projection cost is significant, especially server-side query processing, interactive generation, and low-to-moderate-batch serving. Large-vocabulary SLMs are particularly relevant because the output projection can remain large even after the decoder has been substantially compressed, while the additional packed head is modest in absolute size.
The same trade-off can be useful for local inference on desktops and laptops when an efficient low-bit output-projection kernel is available. It is less attractive on tightly memory-constrained devices such as phones when breaking weight tying materially affects the memory budget. We will add this distinction in the paper and also note this limitation for memory constrained devices. It is also less relevant for prefill-dominated workloads, very high-batch serving, small output heads, or systems where attention, KV-cache traffic, decoder computation, or communication dominates latency. We will add this distinction in the paper and also note this limitation for memory constrained devices.
Thus, the intended regime is not universal model compression. It is a targeted optimization for deployments where the output projection remains a meaningful decode-time bottleneck and where a small additional packed representation is worthwhile in exchange for lower bandwidth and latency.
We provide the table as requested by reviewer that follow our comments above:
| Deployment | Primary bottleneck | Expected benefit | Applicability |
|---|---|---|---|
| Phone / constrained edge | Memory / package size | Separate packed head may increase footprint | Limited / conditional |
| Laptop, local inference | Interactive decode latency | Lower output-head bandwidth at low batch | Good use case |
| Desktop / workstation GPU | Decode bandwidth / latency | Efficient packed W4 projection can reduce per-token cost | Good use case |
| Server, latency-sensitive queries | Decode latency / memory bandwidth | Large output projection is read every generated token | Strongest use case |
| Server, high-batch throughput | Compute / attention / KV / communication | Head traffic is more heavily amortized | Conditional |
We thank the reviewer for raising the practical relevance of output-head quantization. The revised manuscript now includes a whole-model size–quality comparison spanning W2–W8 output-head precision with fixed quantized decoders (Section 3.3, Table 3, (new) Figure 3 w/ full detailed run results in Appendix K.2).
Practical relevance: In our packed Phi experiment, replacing the BF16 output head with a W4 AW-MSE head reduces batch-one 64-token generation latency from 1136.6 to 1013.4 ms, a 10.8% reduction, and batch-16 latency from 1335.8 to 1210.3 ms, a 9.4% reduction. Reparameterization uses the same packed W4 inference path and preserves this speedup. These measurements establish an end-to-end decode benefit and we do not claim the same speedup for prefill-dominated workloads or arbitrary quantized decoders. For context, output-head cost can be substantial in current SLMs because reducing decoder size does not require a proportional reduction in vocabulary. Phi-4-mini, for example, uses a roughly 200K vocabulary, and prior scaling-law work also identifies vocabulary size as an independent quality-relevant scaling dimension. A dense output projection reads all (Vd) weights at every generated token, unlike an input embedding lookup that accesses selected rows.
Whole-model storage qualification: The reviewer is correct that head bitwidth alone does not establish model-level savings, especially for tied embeddings. In our evaluated Phi layout, the original BF16 input/output embedding is shared. Quantizing the output head independently therefore retains the BF16 input embedding and adds a packed output matrix. We do not present this configuration as a model-size reduction; its demonstrated benefit is reduced output-projection traffic and generation latency.
To quantify the storage–fidelity trade-off directly, we added a size–quality comparison with fixed GPTQ decoders and W2–W8 GPTQ output heads. The accounting includes retained embeddings, duplicated output matrices, and head scales. Test KL is measured against the original BF16 model.
For Phi with the BF16 decoder, validation-selected W4 GPTQ lowers test KL from 0.0929 to 0.0349, while unshifted W5 reaches 0.0250 and W6 reaches 0.0065. We therefore do not claim a two- or four-bit saving at W6/W8 fidelity. The supported result is that representative selection improves the W4 fidelity operating point at unchanged W4 payload.
With a fixed W8 Phi decoder, selected W4 reduces end-to-end KL from 0.1989 to 0.1338, while unshifted W5 gives 0.1255. For an illustrative end-to-end KL budget of 0.15, selected W4 satisfies the budget whereas unshifted W4 does not, replacing W5 with one fewer head bit. Under this weight-payload accounting, that corresponds to 0.160 whole-model bits/parameter, or 73.3 MiB, approximately 1.6% of the W8-decoder/W5-head weight payload. The 0.15 threshold is a descriptive reading of the measured curve, not a pre-specified quality threshold or a claim of identical KL.
Thus, with retained tied embeddings, the demonstrated benefit is reduced output-projection traffic and generation latency. Among independently stored output heads, representative selection improves the W4 fidelity point and can reduce the precision required under a specified error budget.
Taken together, the measured 10.8% batch-one latency reduction and the new model-level size–quality comparison show that output-head quantization can have a measurable systems impact, while the storage benefit depends on the tied-weight layout and target error budget. The manuscript also makes these deployment constraints explicit, including tied-weight separation, packed-head storage, and the distinction between reduced output traffic and resident model memory (Appendix G). Prior scaling-law work further shows that compute-optimal vocabulary size can increase with model scale, and Figure 1 illustrates that several contemporary SLM families already use substantially larger vocabularies than earlier designs. We hope these additions address the reviewer’s concern about the practical relevance of the contribution.