acceptodds
Under review as a conference paper at ICLR 2027

SoftWater: The Rate-Distortion Geometry of Softmax Quantization

Abstract

Post-training quantization methods typically optimize weighted mean-squared error (WMSE) on linear-layer outputs, yet the final softmax layer is fundamentally different: its relevant distortion is the change in the predictive distribution. We formulate softmax-layer quantization as a rate-distortion problem under KL divergence and show that, to second order, its geometry is class-aware, coupling feature covariance with class-specific softmax curvature. This distinction matters because output heads are often left in high precision even though, in modern language models, they can account for 15–30% of all parameters, and over half of the stored bytes once the body is quantized. We introduce SoftWater, a practical quantizer derived from this KL geometry. A separable approximation reduces the otherwise prohibitive joint class-feature weighting to a single feature-side factorization together with inexpensive class-dependent rescaling, allowing the head to be quantized using statistics collected in one calibration pass. The resulting allocation assigns precision according to the output class frequency discounted by its variance across inputs, which is especially important in highly imbalanced output spaces such as LLM vocabularies. Across five language models from 1B to 32B parameters, SoftWater outperforms the released WaterSIC quantizer, near-optimal under WMSE, at matched head rates on 59 of 60 evaluation points, reducing head-induced KL by 6.5–8.3 at 2 bits despite using none of WaterSIC's additional refinements. Because the class-side statistic comes from calibration data, matching calibration to the deployment domain gives the lowest KL on that domain throughout. On Llama-3.2-1B-Instruct with quantized bodies, where the tied head is also the embedding, a 2-bit head removes 45–60% of stored bytes for a 2.9–3.7% perplexity increase and a 4-bit head is near-lossless. These results show that treating the softmax layer according to its own distortion geometry can make aggressive end-to-end model quantization practical.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, LLM, softmax

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KGHVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The study treats quantization of the final softmax (LM head) layer as a rate-distortion problem under output KL divergence rather than weighted MSE. It shows that the second-order error weight is class-aware: feature covariance coupled with per-class softmax curvature. Overall Paper is well written and explores an interesting direction.

Critical strengths
  1. The formulation is principled and simple.

  2. Results are consistent, large, and well ablated. Gains hold across five models, three head rates, two body quantizers (GuidedQuant, VPTQ), and zero-shot tasks, where LAMBADA improves by 5–7 points.

Critical weaknesses
  1. The only comparison is WaterSIC with its refinements removed. Missing are GPTQ/RTN/AWQ group-wise on the head, QTIP/QuIP# head quantization, LFQ, GuidedQuant applied to the head, and a YAQA-style Hessian-sketch head.

  2. Entropy-coded, variable-rate rows and columns are not a standard inference format. There is no decode or inference latency, kernel support, or runtime memory analysis, and no comparison at fixed-rate hardware-friendly formats.

  3. Calibration is mostly on WikiText-2 and evaluation is in-domain, and there is no variance across calibration seeds.

  4. The second-order KL expansion is assumed valid at 2 bits, where KL increases are large (e.g., +34.9% on a 2-bit body), but expansion accuracy is never measured at those operating points.

  5. The paper claims the method applies to vision and speech classifiers and MoE routers, but all are not tested.

Questions for authors

See Weakness.

Reviewer_FsY2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates softmax-layer quantization as a rate-distortion problem under KL divergence, in contrast to the standard weighted mean-squared error (WMSE) objective used for linear layers. The key insight is that the softmax layer's relevant distortion is the change in the predictive distribution, not the output norm. A second-order Taylor expansion of the KL divergence yields an error weight E[(diag(p) - pp^T) ⊗ XX^T], which couples feature covariance with class-specific softmax curvature. This makes the correct rate allocation class-aware: frequent, low-variance classes get fine grids, while rare ones get coarse grids. The paper introduces SoftWater, a practical algorithm that uses a separable approximation (Assumption 1) to reduce the otherwise prohibitive joint class-feature weighting to a single feature-side factorization with inexpensive class-dependent rescaling. A smoothing prior caps grid spacing on classes absent from calibration data, keeping the Taylor expansion valid. The method requires only one additional K-vector statistic collected in the same calibration pass. Across five models from 1B to 32B parameters, SoftWater outperforms the released WaterSIC quantizer on 59 of 60 evaluation points, reducing head-induced KL by 6.5-8.3x at 2 bits. On Llama-3.2-1B with quantized bodies, a 2-bit head removes 45-60% of stored bytes for a 2.9-3.7% perplexity increase, and a 4-bit head is near-lossless.

Critical strengths

The paper's central contribution — recognizing that softmax layers require a fundamentally different quantization objective than linear layers — is both correct and practically important. The derivation of the class-aware error weight from the KL divergence's second-order expansion is clean and well-motivated. The separability assumption (Assumption 1) is a reasonable tractability shortcut, and the paper provides both a theoretical justification (the dither argument showing cross-class terms vanish) and an empirical validation (Section 5.6). The smoothing prior that interpolates between data-driven allocation and uniform allocation (recovering WaterSIC at epsilon = 1) is an elegant design that prevents breakdown on uncovered classes. The experimental results are strong and consistent: 59/60 wins over WaterSIC, with large KL reductions (6.5-8.3x) at 2 bits. The finding that a 2-bit SoftWater head outperforms a 3-bit WaterSIC head on WT2 KL is particularly compelling. The domain-targeted calibration experiment (Section 5.3) showing that matching calibration to deployment domain gives the lowest KL is a practically useful insight.

Critical weaknesses

(1) The separability assumption (Assumption 1) is the method's critical enabling simplification, and while the paper provides empirical validation, the theoretical justification is thin. Equation (16) shows separability preserves total curvature per class, but the anisotropic structure that is dropped could matter in practice. The paper tests this empirically (Section 5.6), but only on a limited set of models and rates. A more systematic study of when separability breaks — and what happens to SoftWater's allocation when it does — would strengthen the contribution. (2) The second-order Taylor expansion (Eq. 5) is only valid for small logit perturbations delta. The paper acknowledges this and introduces the smoothing prior to keep perturbations in the valid regime, but does not quantify how often the expansion is violated in practice, especially at aggressive rates (2 bits). If the smoothing prior's floor is set too high, SoftWater effectively reverts to uniform allocation on the rarest classes, potentially losing its advantage exactly where the class distribution is most skewed. (3) The comparison with WaterSIC is not fully fair: SoftWater uses none of WaterSIC's additional refinements (LMMSE shrinkage, rescaler optimization, finetuning), and the paper attributes the gains "to the allocation alone." While this isolates the contribution, it leaves open whether WaterSIC with the class-aware allocation would perform comparably, or whether the gains are truly from the KL geometry rather than from the specific encoding strategy. (4) The method requires calibration data to estimate the class-side statistic (lambda_bar), which means the allocation is deployment-specific. While the paper frames this as a feature (domain-targeted quantization), it also means the quantized model is not transferable across domains without re-calibration. For multi-domain deployment, this could be a significant practical limitation. (5) The evaluation is limited to perplexity and KL divergence. No downstream task results (e.g., zero-shot accuracy, code generation, math reasoning) are reported, which would better demonstrate that the KL reduction translates to practical quality preservation. The claim that a 4-bit head is "near-lossless" is based on perplexity alone. (6) The method's complexity is O(n^3) per class for the Cholesky (though shared across classes), plus K secant searches for rate assignment. For very large vocabularies (e.g., 256K tokens), even storing the K-vector lambda_bar and performing K rescaling operations could introduce non-trivial calibration overhead, which is not discussed.

Questions for authors

Q1. The separability assumption (Assumption 1) is the critical enabling simplification. You provide empirical validation on a limited set of models and rates, but can you characterize the conditions under which separability breaks? Specifically, what properties of the feature covariance or class distribution would cause the dropped anisotropic structure to matter? Can you provide a quantitative bound on the approximation error introduced by separability? Q2. The second-order Taylor expansion is central to the derivation, but its validity at 2-bit quantization (where logit perturbations can be large) is questionable. Can you empirically measure the fraction of calibration tokens where the higher-order terms (third-order and above) exceed a given threshold at 2 bits? How does the smoothing prior's floor value (epsilon) interact with the expansion's validity — is there a principled way to set epsilon based on the expansion order? Q3. The evaluation is limited to perplexity and KL divergence. Can you provide results on downstream tasks (e.g., zero-shot accuracy on MMLU, HumanEval code generation, GSM8K math reasoning)? Does the KL reduction translate to measurable quality preservation on tasks where the softmax distribution matters most? Q4. The comparison with WaterSIC isolates the allocation contribution by disabling WaterSIC's additional refinements (LMMSE, rescaler optimization, finetuning). Have you tried applying the class-aware allocation on top of a fully-featured WaterSIC? Would the gains compound, or are they subsumed by the refinements? Q5. The calibration is deployment-specific (domain-targeted). How much does the allocation change between domains? If a model is calibrated on Wikipedia and deployed on code, what is the KL degradation? Is there a universal allocation that works reasonably across domains, or is re-calibration always necessary? Q6. For models with very large vocabularies (e.g., 256K tokens), how many classes are typically uncovered in calibration data? What fraction of the vocabulary receives the smoothing prior's default allocation, and does this dominate the KL budget at large vocabulary sizes?

Reviewer_a4vCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SoftWater quantizes LLM output heads by allocating precision according to softmax curvature and feature covariance. It extends WaterSIC with class-dependent scaling to better preserve the output distribution. The paper reports substantial improvements at matched bit rates, including when combined with quantized model bodies.

Critical strengths
  • The motivation is convincing: output heads can account for a substantial share of storage after the rest of the model is quantized.
  • The method is simple and requires only one additional vector of calibration statistics. The reported 6.5–8.3× reduction in WikiText2 KL at 2 bits across five models is a strong result.
  • The experiments include tied and untied heads, domain-specific calibration, and useful ablations of the class statistic and smoothing.
Critical weaknesses

The justification for the implemented approximation needs stronger validation. Equation (7) removes cross-class terms in expectation over subtractive dither, but the final algorithm is undithered. Table 15 checks diagonal separability on one model; it does not measure the omitted cross-class contribution or Taylor approximation error. Could the authors compare actual KL, the full quadratic approximation, and the separable proxy at 2 bits? This would also support the claim that smoothing keeps perturbations within the quadratic regime.

The domain experiment changes both feature covariance and class statistics when switching calibration domains. It therefore does not isolate the claim that class statistics provide additional domain information beyond covariance. Holding one factor fixed while varying the other would make this conclusion more convincing.

The practical evidence is strongest for storage and distribution preservation. Downstream evaluation covers one small model, with gains concentrated in LAMBADA, while inference memory and decoding overhead are not reported. Clarifying these limits would help establish the method’s deployment value.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.