- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ESPTQ combines learned butterfly rotations, sparse secondary ternary planes, and calibration for weight quantization, and extends a constrained ternary format to the KV cache. It reports model quality and serialized storage, with analytical multiplication counts for a proposed shared datapath.
Critical strengths
- Extending ternary computation from linear projections to attention addresses a relevant limitation of weight-only quantization.
- Well-organized paper and figure.
Critical weaknesses
- Table 2 compares against FP16 KV without an empirical low-bit KV baseline, and Appendix J.1 limits evaluation to at most 2048 tokens. More directly, its two 7B comparisons differ in code version and batch size between weight-only and weight-plus-KV runs, as the authors acknowledge, weakening attribution of the measured difference to KV quantization.
- Figure 2(b) changes several factors between element- and group-sparse variants. The final pipeline therefore does not cleanly isolate rate–distortion selection from simpler alternatives at equal storage.
- Appendices C.1 and J.2 explicitly acknowledge that activation flattening targets
xR^T, whereas inference usesxR; the default experiments retain this mismatch. What influence will happen? - Appendix B.3 states that evaluation uses FP32 rotation angles, whereas serialization stores FP16 angles without evaluating the resulting model. Why the author do like that?
- Appendix F.5 reports a successful 512-token packing/unpacking check, but the evaluation itself uses floating-point KV reconstruction. Thus, quality with the serialized parameters and execution costs remain unverified. The disclosed analytical counts and file sizes should not directly be interpreted as measured runtime savings.
Questions for authors
Questions are listed in weakness please refer.
Response to Reviewer SmKP
Thank you for reading the appendix in detail. You asked us to measure the points that the paper had only disclosed; we measured what can be measured. All experiments use the same A6000 machine as Tables 1 and 2, with LLaMA-2-7B and ρ = 0.12. Decision margin (2√2 times the standard deviation over 3 seeds): ±0.17% on WikiText-2 and ±0.26% on C4.
W1 (code version and batch size). We re-measured the weight-only and weight + KV rows with the same code and batch size:
Disclosure: the ESPTQ-wkv values for LLaMA-2-7B in Table 2 (7.13, 58.12 and 2.81) were measured with an older version of the code; the current code gives the values above. The cost of ternarizing the KV cache is therefore slightly larger than Table 2 suggests (+7.05%, −3.21 points). We have not identified which change had the effect. Your point was correct. The Qwen2.5-7B row reproduces Table 2 with the current code (9.0528 against 9.05).
W1 (low-bit KV baseline). The weights are identical byte for byte; only the KV treatment differs.
The ternary KV cache is far better than 3-bit integers and slightly below 4-bit integers. Its advantage is not accuracy but fewer bits and meeting the multiplication condition of the paper, which the integer KV cache does not meet. This is an integer baseline built within our framework, not a reproduction of KIVI. The integer rows were measured earlier than the ternary row, and we have not directly confirmed that the code changes in between leave the KV numbers unchanged.
W1 (2048 tokens). At sequence length 4096, WikiText-2 perplexity is 5.1163 (FP16), 6.2129 (ESPTQ-w) and 6.9629 (ESPTQ-wkv). The cost of the ternary KV cache grows from +7.93% to +12.07% but does not collapse. Long-context tasks were not measured; the limitation in Appendix J.1 remains.
W2 (Figure 2(b)). As you noted, several factors are mixed there. When only the selection criterion is changed (same element-sparse pipeline, storage format and ρ), the proposed criterion uses the fewest bits (2.6443) and the alternatives are not more accurate. With ρ lowered to align the bits (2.6424), the gain without the code-length term is worse by 0.23% (WikiText-2) and 0.82% (C4), beyond the margin. The differences are small; we read most of the difference in Figure 2(b) as coming from element-sparse versus group-sparse placement.
W3 (xRᵀ and xR). The direction in the implementation was an error. In the setting of Table 1 it has no effect: removing the term changes WikiText-2 by −0.17% (within the margin), and correcting the direction gives −0.01% on WikiText-2, +0.43% on C4 and −1.05 points on 0-shot⁸ (1 seed). The 0.84% stated in Appendix I.3 was not reproduced. The mismatch enters only 1 term of the objective for learning the angles; after learning, the weights, the inference input and the GPTQ Hessian use the same rotation.
W4 (FP16 angles). We assembled the model from the compressed file alone, with FP16 angles, and evaluated it. Perplexity is the same to the 4th decimal place (6.6558 and 9.7223); 0-shot⁸ is 61.30 against 61.33 and MMLU is 28.19 against 28.31, within the fluctuation of re-runs. Qwen2.5-7B agrees as well. At submission we did not have these numbers.
W5 (quality and cost in the stored form). Quality: over all 166 evaluation windows, all 10624 packed K and V tensors matched the values used in evaluation bit for bit, and perplexity is the same (7.1834). Cost: we agree. The number of multiplications is analytical and the file size is storage; neither is execution time. End-to-end decoding with the reference implementation (LLaMA-2-7B, batch 1) is 37.75 tok/s for FP16, 4.83 for ESPTQ-w and 2.80 for ESPTQ-wkv. Folding the rotation into the weights at load time gives ESPTQ-w the speed of FP16, but with dense 16-bit weights; ESPTQ-wkv stays at 0.19× FP16 even with the CPU–GPU waits removed. On the GPU we obtained no speed gain and claim none. The gain from removing multipliers is to be shown with a dedicated circuit (an accelerator), a remaining direction in §6.
Final version. How to reflect the above (the Table 2 row, appendix tables, the activation term) is under consideration. Full tables for each item can be posted on request.