acceptodds
Under review as a conference paper at ICLR 2027

ESPTQ: Post-Training Ternarization of Weights and KV Cache for Eliminating Per-Element Multiplications

Abstract

Reading the weights and KV cache dominates large language model inference. Ternary quantization to -1, 0, +1 shrinks both and removes per-element multipliers: each weight product becomes a sign flip or skip, so a signed adder tree suffices. Ternary weights alone, however, leave multipliers in attention's and , whose operands are both run-time floating-point tensors. We present ESPTQ (element-sparse-plane ternary post-training quantization), which quantizes the weights and KV cache into one ternary storage format, removing per-element multipliers from every matrix product inside the Transformer layers except the rotation of layer inputs. By design, ESPTQ caps the residual multiplication ratio at the weight-side budget of two multiplications per 128-element reduction (). For weights, after a learnable butterfly rotation within each 128-column group, a rate-distortion criterion selects, at a continuously variable second-plane assignment ratio , which elements receive a sparse second ternary plane. For the KV cache, static tables replace per-token offsets and the V-side scaling factor, meeting the same budget. We compare methods on effective bits per weight, serializing each in its own storage format with losslessly compressed discrete codes, so scaling factors, rotation angles and other auxiliary metadata count. On LLaMA-2-7B, ESPTQ matches TWLA in perplexity and average zero-shot accuracy at less than half the effective bits, and at comparable bits lowers perplexity by 36% against PT2-LLM. Ternarizing the KV cache as well shrinks it to about one fifth of its FP16 size, at less cost in perplexity and zero-shot accuracy than ternarizing the weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training quantization, ternary quantization, KV cache quantization, large language models, model compression, multiplication-free inference, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SmKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ESPTQ combines learned butterfly rotations, sparse secondary ternary planes, and calibration for weight quantization, and extends a constrained ternary format to the KV cache. It reports model quality and serialized storage, with analytical multiplication counts for a proposed shared datapath.

Critical strengths
  1. Extending ternary computation from linear projections to attention addresses a relevant limitation of weight-only quantization.
  2. Well-organized paper and figure.
Critical weaknesses
  1. Table 2 compares against FP16 KV without an empirical low-bit KV baseline, and Appendix J.1 limits evaluation to at most 2048 tokens. More directly, its two 7B comparisons differ in code version and batch size between weight-only and weight-plus-KV runs, as the authors acknowledge, weakening attribution of the measured difference to KV quantization.
  2. Figure 2(b) changes several factors between element- and group-sparse variants. The final pipeline therefore does not cleanly isolate rate–distortion selection from simpler alternatives at equal storage.
  3. Appendices C.1 and J.2 explicitly acknowledge that activation flattening targets xR^T, whereas inference uses xR; the default experiments retain this mismatch. What influence will happen?
  4. Appendix B.3 states that evaluation uses FP32 rotation angles, whereas serialization stores FP16 angles without evaluating the resulting model. Why the author do like that?
  5. Appendix F.5 reports a successful 512-token packing/unpacking check, but the evaluation itself uses floating-point KV reconstruction. Thus, quality with the serialized parameters and execution costs remain unverified. The disclosed analytical counts and file sizes should not directly be interpreted as measured runtime savings.
Questions for authors

Questions are listed in weakness please refer.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SmKP

Thank you for reading the appendix in detail. You asked us to measure the points that the paper had only disclosed; we measured what can be measured. All experiments use the same A6000 machine as Tables 1 and 2, with LLaMA-2-7B and ρ = 0.12. Decision margin (2√2 times the standard deviation over 3 seeds): ±0.17% on WikiText-2 and ±0.26% on C4.

W1 (code version and batch size). We re-measured the weight-only and weight + KV rows with the same code and batch size:

WikiText-2C40-shot⁸GSM8K
ESPTQ-w (KV in FP16)6.65589.722361.303.64
ESPTQ-wkv (ternary KV)7.1834 (+7.93%)10.8267 (+11.36%)57.74 (−3.56 pt)2.12 (−1.52 pt)

Disclosure: the ESPTQ-wkv values for LLaMA-2-7B in Table 2 (7.13, 58.12 and 2.81) were measured with an older version of the code; the current code gives the values above. The cost of ternarizing the KV cache is therefore slightly larger than Table 2 suggests (+7.05%, −3.21 points). We have not identified which change had the effect. Your point was correct. The Qwen2.5-7B row reproduces Table 2 with the current code (9.0528 against 9.05).

W1 (low-bit KV baseline). The weights are identical byte for byte; only the KV treatment differs.

KVbits/elementWikiText-2C40-shot⁸GSM8K
FP16166.65589.722361.303.64
Ternary (ESPTQ-wkv)3.287.183410.826757.742.12
4-bit integer, per-token scale4.1256.976410.248159.213.26
3-bit integer, per-token scale3.12512.746421.043145.310.68

The ternary KV cache is far better than 3-bit integers and slightly below 4-bit integers. Its advantage is not accuracy but fewer bits and meeting the multiplication condition of the paper, which the integer KV cache does not meet. This is an integer baseline built within our framework, not a reproduction of KIVI. The integer rows were measured earlier than the ternary row, and we have not directly confirmed that the code changes in between leave the KV numbers unchanged.

W1 (2048 tokens). At sequence length 4096, WikiText-2 perplexity is 5.1163 (FP16), 6.2129 (ESPTQ-w) and 6.9629 (ESPTQ-wkv). The cost of the ternary KV cache grows from +7.93% to +12.07% but does not collapse. Long-context tasks were not measured; the limitation in Appendix J.1 remains.

W2 (Figure 2(b)). As you noted, several factors are mixed there. When only the selection criterion is changed (same element-sparse pipeline, storage format and ρ), the proposed criterion uses the fewest bits (2.6443) and the alternatives are not more accurate. With ρ lowered to align the bits (2.6424), the gain without the code-length term is worse by 0.23% (WikiText-2) and 0.82% (C4), beyond the margin. The differences are small; we read most of the difference in Figure 2(b) as coming from element-sparse versus group-sparse placement.

W3 (xRᵀ and xR). The direction in the implementation was an error. In the setting of Table 1 it has no effect: removing the term changes WikiText-2 by −0.17% (within the margin), and correcting the direction gives −0.01% on WikiText-2, +0.43% on C4 and −1.05 points on 0-shot⁸ (1 seed). The 0.84% stated in Appendix I.3 was not reproduced. The mismatch enters only 1 term of the objective for learning the angles; after learning, the weights, the inference input and the GPTQ Hessian use the same rotation.

W4 (FP16 angles). We assembled the model from the compressed file alone, with FP16 angles, and evaluated it. Perplexity is the same to the 4th decimal place (6.6558 and 9.7223); 0-shot⁸ is 61.30 against 61.33 and MMLU is 28.19 against 28.31, within the fluctuation of re-runs. Qwen2.5-7B agrees as well. At submission we did not have these numbers.

W5 (quality and cost in the stored form). Quality: over all 166 evaluation windows, all 10624 packed K and V tensors matched the values used in evaluation bit for bit, and perplexity is the same (7.1834). Cost: we agree. The number of multiplications is analytical and the file size is storage; neither is execution time. End-to-end decoding with the reference implementation (LLaMA-2-7B, batch 1) is 37.75 tok/s for FP16, 4.83 for ESPTQ-w and 2.80 for ESPTQ-wkv. Folding the rotation into the weights at load time gives ESPTQ-w the speed of FP16, but with dense 16-bit weights; ESPTQ-wkv stays at 0.19× FP16 even with the CPU–GPU waits removed. On the GPU we obtained no speed gain and claim none. The gain from removing multipliers is to be shown with a dedicated circuit (an accelerator), a remaining direction in §6.

Final version. How to reflect the above (the Table 2 row, appendix tables, the activation term) is under consideration. Full tables for each item can be posted on request.

Reviewer_18PuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ESPTQ, a post-training ternary quantization framework for both LLM weights and the KV cache. For weights, it combines group-local learned butterfly rotations with rate–distortion-based allocation of a sparse second ternary plane, followed by GPTQ error compensation and two-stage calibration. For the KV cache, it uses a compatible two-plane representation and reorganizes scales, offsets, and cache updates to limit general multiplications in attention reductions. The paper evaluates accuracy against effective storage costs, including auxiliary metadata, across multiple model families and sizes. Its efficiency claims are supported by storage accounting and analytical multiplier counts rather than measured kernel or end-to-end inference speedups.

Critical strengths
  1. The paper provides a detailed accounting of effective storage costs, including auxiliary parameters, and evaluates the method across a broad range of model families and sizes. The results support the usefulness of the sparse second ternary plane for improving the accuracy–storage trade-off.
  2. The KV-cache extension is particularly interesting and the attempt to use compatible ternary representations for weights and the KV cache under a common multiplier budget is a compelling aspect of the work.
  3. Furthermore, Figure 2(a) and the text indicate that blockfly remains largely unimodal.
Critical weaknesses
  1. Ablation study is not enough. In particular, the benefit of the rate–distortion criterion over simpler residual- or gain-based selection is unclear. Furthermore, Figure 2(a) and the text indicate that blockfly remains largely unimodal. The paper therefore does not establish whether its gains arise from the trimodal shaping objective or from rotation more generally.

  2. No end-to-end and kernel latency measurement. It remains unclear whether the proposed representations improve prefill or decoding performance once these costs are included.

  3. The second-plane allocation in Figure 1(d) uses the diagonal Hessian approximation (h_j=H_{jj}), whereas GPTQ compensation uses richer Hessian information.

  4. The presentation could be improved. Tables 1 and 2 use very small text, making the dense numerical comparisons difficult to read at normal viewing size. The abstract is also overloaded with implementation details and accounting terminology and some explanatory phrasing is conversational rather than precise.

Questions for authors

If the author has solved the concerns, I would consider raise my score.

  1. The second-plane allocation in Figure 1(d) only uses the diagonal Hessian approximation (h_j=H_{jj}). Explain what motivates this design and demonstrate with ablation results.
  2. Can you provide end-to-end prefill and decoding latency for at least one representative model, comparing the baseline, ESPTQ-w, and ESPTQ-wkv under matched hardware and workload settings? Kernel benchmarks over at least three representative workload configurations should also be covered.
  3. Please report the wall-clock time of block-wise calibration and end-to-end logit distillation separately for the weight and KV-cache pipelines. 4.Under matched effective-bit budgets and otherwise identical settings, how does the proposed allocation criterion compare with residual-only selection, activation-weighted residual selection, and gain-only selection without the code-length term? These comparisons would help isolate the benefits of activation weighting and rate–distortion optimization.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 18Pu

Thank you for the detailed review and for specifying what to align and what to compare against. All additional experiments use the same A6000 machine and code as Table 1 (LLaMA-2-7B, ρ = 0.12 unless noted). Decision margin (2√2 times the standard deviation over 3 seeds): ±0.17% on WikiText-2 and ±0.26% on C4.

Q4 / W1 (selection criterion). Same pipeline, storage format and ρ; only the criterion changes.

Criterion#W*WikiText-2C40-shot⁸
Proposed (gain / code length)2.64436.65589.722361.33
Residual only2.65586.65389.733560.69
Activation-weighted residual2.65986.66869.725361.41
Gain without code-length term2.67106.65049.750961.45
Same, ρ lowered to align bits2.64246.67149.801761.39

The proposed criterion uses the fewest bits, and no alternative is more accurate. At aligned bits, the gain without the code-length term is worse by 0.23% (WikiText-2) and 0.82% (C4), both beyond the margin. Only this alternative was measured at aligned bits; this is 1 model and 1 seed.

W1 (trimodal objective or rotation in general).

Rotation#W*WikiText-2C40-shot⁸
Table 1 (trimodal + activation term)2.64436.65589.722361.33
Activation term removed2.64466.64489.725361.63
Trimodal term removed2.63426.69099.831860.34
Fixed Hadamard (no learning)2.62436.70809.910860.21
No rotation2.592712.827710.1914—

Most of the gain comes from the rotation itself; learning adds a small gain that comes from the trimodal objective. As you noted, the distribution stays unimodal: "trimodal" names the shape of the objective, not of the distribution. The activation-flattening term gives no measurable gain.

Q2 / W2 (latency). End-to-end, reference implementation (no fused kernel), LLaMA-2-7B, prefill 1024 and decode 64 tokens, 48 GB. The lower 3 rows are the largest batch that fits.

Conditionbatchprefill (s)decode (s)decode tok/s
FP1610.2471.69537.75
ESPTQ-w10.85013.2594.83
ESPTQ-wkv11.54122.8302.80
FP16589.29210.771344.63
ESPTQ-w5723.62413.975261.03
ESPTQ-wkv9386.801235.92225.23

ESPTQ is slower than FP16. Folding the rotation into the weights at load time gives ESPTQ-w the speed of FP16, but with dense 16-bit weights. ESPTQ-wkv stays at 0.19× FP16 at batch 1 even with the rotation folded and the CPU–GPU waits removed. We have no fused kernel; GPU kernels for the KV-cache path also left decoding slower than FP16, and those numbers will follow. The FP16 baseline uses the library's default cache; without its per-step re-concatenation, FP16 itself becomes 2.23× faster at the largest batch. On the GPU we obtained no speed gain and claim none. The gain from removing multipliers is to be shown with a dedicated circuit (an accelerator), a remaining direction in §6.

Q1 / W3 (diagonal Hessian). With a diagonal Hessian the distortion separates over elements, so they can be ordered by a single threshold on "gain / code length" (as in Choi et al. 2017); the off-diagonal information is used by the GPTQ compensation afterwards. We measured (a) the column weight 1/d_j² from the Cholesky factor of H⁻¹ and (b) selection inside the GPTQ loop on the compensated weights. Differences from Table 1 with the same seed:

FormModel#W*WikiText-2C40-shot⁸
(a)LLaMA-2-7B (3 seeds)−0.0157−0.52%−0.18%−0.47 pt
(b), bits alignedLLaMA-2-7B (3 seeds)+0.0006−0.74%−0.47%−0.27 pt
(b), bits alignedQwen2.5-7B−0.0089−0.92%−1.16%+0.34 pt
(a)+(b), bits alignedLLaMA-2-7B (seed 0)+0.0008−1.56%−0.77%+0.01 pt
(a)+(b), bits alignedQwen2.5-7B−0.0134−2.23%−1.85%+0.57 pt

Your point is correct: richer Hessian information lowers perplexity by about 1–2% at the same or fewer bits. 0-shot⁸ shows no consistent gain: with (a), 0-shot⁸ of Qwen2.5-7B dropped on all 3 seeds (−3.39, −0.64 and −0.70 points). (b) also needs ρ re-tuned for each model. Tables 1 and 2 keep the diagonal.

Q3 (calibration time). LLaMA-2-7B, wall-clock seconds. Weights: learning the rotation 2055.6, GPTQ and second-plane selection 2509.9, block-wise calibration 1405.4, logit distillation 1867.3, other 1000.7; total 8838.9 (2.46 hours). KV cache: static tables 7131.5 (includes 1 optimization stage run on the CPU), logit distillation 415.0, scales 17.1; total 7563.6 (2.10 hours).

W4. You are right about the text size of the tables, the abstract and the colloquial wording.

Final version. How to reflect the above (appendix tables, presentation) is under consideration. Full tables for each item can be posted on request.

Reviewer_hmwVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ESPTQ that quantizes the weights and KV caches to 1.58-bit in order to avoid per-element multiplications except the rotations. The authors show that ESPTQ can achieve comparable perplexity and zero-shot accuracy to TWLA across different models.

Critical strengths
  • The authors aim at ternary quantization of weights and KV caches, which leads to significant memory savings.
  • The authors evaluate ESPTQ across multiple models (e.g., Llama2, Llama3.2, Qwen2.5, Qwen3)
Critical weaknesses
  • Either PPL or CSR accuracy is reported. So, it is hard to determine whether ESPTQ is effective or not. It would be more instructive if the authors also explore more challenging tasks (e.g., MMLU and/or its variants) and generation tasks (e.g., IFEval, MATH500).
  • Given that there is no measurement of real inference speed-up on hardwares (e.g., end-to-end GPU latency), it is difficult to determine whether ESPTQ is practical in real world applications.
  • As large models (e.g., 30B-sized or larger) needs extremely low-bit quantization more than small models, it would be beneficial if the authors evaluate ESPTQ on large models.
Questions for authors
  • Comparison of ESPTQ with extremely low-bit QAT (e.g., ParetoQ)
  • ESPTQ application to MoE models
1 comment
AuthorsOpenReview ↗

Response to Reviewer hmwV

Thank you for the review. The tasks and baselines you named let us run the experiments below.

Novelty. (1) Per-element multiplications are removed in attention as well: the KV cache is quantized into the weights' storage format, keeping the residual multiplication rate, cache write included, within the weight-side budget (1.5625%) without retraining. (2) A second ternary plane, placed sparsely per element by a rate-distortion criterion, makes bits and accuracy move continuously with ρ. (3) Methods are compared in measured bits including all auxiliary data; the "comparable to TWLA" result in your summary is obtained at less than half of TWLA's measured bits.

W1 (harder tasks). We added 5-shot MMLU and MATH500 (string match / verified as expressions). Different machine from Table 1; compare only within this table.

ModelMethod#W*MMLU 5-shotMATH500
LLaMA-2-7BFP161645.982.20 / 3.20
LLaMA-2-7BGPTQ W3 g1283.1741.604.40 / 6.00
LLaMA-2-7BESPTQ-w₀.₁₂2.6434.092.00 / 2.80
Qwen2.5-7BFP161674.2139.40 / 51.20
Qwen2.5-7BGPTQ W3 g1283.1467.9721.20 / 27.80
Qwen2.5-7BESPTQ-w₀.₁₂2.6462.4716.60 / 19.20

ESPTQ-w does not collapse, but it is below FP16 and below 3-bit GPTQ, which uses about 0.5 bits more. Harder tasks drop more, a limit of the operating point of about 2.6 bits; §6 lists multi-step reasoning as future work. We did not measure IFEval, which targets instruction-tuned models; the paper covers base models only.

W2 (speed). We do not claim speed. End-to-end decoding speed of our reference implementation (no fused kernel; A6000 with 48 GB; prefill 1024, decode 64 tokens):

ModelConditiontok/s, batch 1tok/s, largest batch (batch size)
LLaMA-2-7BFP1637.75344.63 (58)
LLaMA-2-7BESPTQ-w4.83261.03 (57)
LLaMA-2-7BESPTQ-wkv2.8025.23 (93)
Qwen2.5-7BFP1640.182261.93 (364)
Qwen2.5-7BESPTQ-w5.461292.92 (328)
Qwen2.5-7BESPTQ-wkv3.34212.96 (459)

ESPTQ is slower than FP16. ESPTQ-w is slow because the rotation is computed at run time; folding it into the weights at load time gives FP16 speed, but with dense 16-bit weights, so nothing becomes faster or smaller on a GPU. ESPTQ-wkv stays at 0.19× and 0.22× of FP16 at batch 1 even with the rotation folded and the CPU–GPU waits removed from the KV-cache path. GPU kernels for the KV-cache path also left decoding slower than FP16 at every batch we measured; those numbers will follow. The FP16 baseline uses the library's default cache; without its per-step re-concatenation, FP16 itself becomes 2.23× and 1.67× faster at the largest batch.

On the GPU, none of the implementations we built gave a speed gain, and we claim none. The paper's efficiency claims are stored bits and an analytical multiplication count in a circuit model (§4.4). The gain from removing multipliers is to be shown with a dedicated circuit (an accelerator), a remaining direction in §6; there we expect smaller circuit size and power. For reference, in FPGA synthesis of 1 inner product (64 terms, FP16 activations), the ternary design uses about 48% of the LUTs and 53% of the estimated dynamic power of a LUT-only FP16 design; the slowest path does not become shorter.

W3 (30B and larger). Not measured: the FP16 weights of a 30B model (about 60 GB) do not fit on our largest GPU (48 GB). Weight-only results reach 13B and 14B models (Appendix H.1). Whether the method works equally at 30B and above is not shown.

Q1 (ParetoQ). We applied ESPTQ-w to ParetoQ's full-precision parents and evaluated both in the same way (different machine from Table 1). WikiText-2 perplexity / 0-shot⁸:

SizeParetoQ (QAT, 1.80–1.86 bits)ESPTQ-w, ρ = 0.12 (2.65–2.72 bits)
125M21.10 / 42.1023.66 / 39.14
350M15.65 / 47.0416.00 / 44.38
600M12.62 / 51.0213.60 / 49.05
1B11.25 / 54.5211.04 / 52.47
1.5B10.28 / 57.9810.79 / 55.08

QAT is better: ParetoQ has higher zero-shot accuracy at all 5 sizes with fewer bits, and ESPTQ-w comes close only in perplexity. ParetoQ reports 120,000 updates on 16 GPUs; ESPTQ needs 128 calibration windows and about 2.5 hours on 1 GPU for a 7B model, so it is an option when retraining cannot be afforded.

Q2 (MoE). Not yet tried. We expect the method to apply as it is: weight quantization is closed within each linear layer, KV-cache quantization within attention, and the router would stay in FP16. Prior work (CAT-Q, ScaleQ-1.58) already applies ternary post-training quantization to MoE models; CAT-Q reports them as more sensitive than dense models. Accuracy on MoE is not shown.

Final version. How to reflect the above (appendix tables, and limitations including model size and MoE) is under consideration. Full tables for each item can be posted on request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.