- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Transformer-FSQ, a post-training KV-cache compression method that replaces conventional per-token K/V storage with a shared wide low-bit discrete latent. The latent is produced from the attention input hidden state using finite scalar quantization (FSQ), and keys/values are reconstructed through learned linear projections; the value path further exploits linearity to aggregate directly in latent space before applying the value projection. The method can retrofit pretrained Transformers while freezing the backbone. Experiments on Qwen3 models show that Transformer-FSQ is particularly competitive in the extreme-compression regime, outperforming the evaluated eviction, quantization, and low-rank baselines beyond roughly 12× compression, with extensive ablations and byte-level memory accounting.
Critical strengths
-
S1. Novel wide low-bit latent formulation for extreme KV-cache compression. The paper combines latent dimensionality reduction with low-bit quantization under a fixed memory budget, allowing the two compression axes to compound. This design pushes KV-cache compression substantially beyond the regimes typically explored by either low-rank projection or quantization alone.
-
S2. Clever algorithmic and implementation design. FSQ avoids the explicit codebooks, nearest-neighbor lookup, and associated metadata overhead of conventional VQ, allowing a larger fraction of the memory budget to store useful latent information. Moreover, the method exploits linearity in the value path, computing the attention-weighted reduction directly over cached latents and applying the value projection only afterward, thus avoiding explicit reconstruction of the full value cache and reducing value-side memory traffic and computation.
-
S3. Strong results in the extreme-compression regime with a practical post-training conversion procedure. Transformer-FSQ maintains substantially better quality than the evaluated baselines at very high compression ratios, while requiring only post-training conversion and finetuning rather than pretraining a new architecture from scratch. The paper also provides extensive ablations and careful byte-level accounting of cache storage, including auxiliary metadata and other method-specific overheads.
Critical weaknesses
-
W1. The favorable accuracy--compression and efficiency regimes barely overlap. Transformer-FSQ mainly outperforms baselines in the extreme-compression regime, where absolute quality has already degraded substantially. The paper itself states that at , VecInfer and Palu retain of uncompressed RULER accuracy, while Transformer-FSQ becomes advantageous mainly beyond (Sec. 5.2, RQ1). In Tables 13--14, Transformer-FSQ already drops from 0.71 to 0.58 at on Qwen3-0.6B and from 0.77 to 0.65 at on Qwen3-1.7B. Meanwhile, its decode throughput is only 0.12--0.13$\times$ of uncompressed at (Tables 11--12), due to repeated key reconstruction and RoPE. Speedups appear only at much more aggressive compression (Fig. 6 / Sec. G.2), where accuracy is already severely degraded. Thus, the method advances the extreme-compression frontier, but its benefit in a practically usable serving regime remains unclear.
-
W2. The end-to-end serving benefit is insufficiently evaluated. The paper reports decode throughput over several batch sizes and context lengths, but does not provide TTFT, TPOT, latency percentiles, or throughput-vs-concurrency saturation curves. This is important because Transformer-FSQ trades memory traffic for substantial additional computation: Sec. G.2 explicitly gives the key-reconstruction cost as per layer and decoding step. Prefill is also not characterized in latency: Appendix D.5 reports about 11.1 GiB transient memory per sequence at 8K context and OOM at batch 4 in the reference pipeline, while chunked prefill/FlashAttention are only suggested as remedies. Moreover, the long-context capacity claim is largely extrapolated: Sec. 5.3 explicitly states that the 3.8M-token projection extrapolates 8,192-token scores and “does not establish million-token retrieval accuracy.” Longer-context and full serving-capacity evaluations are therefore needed.
-
W3. The numerical-precision setting conflicts with the edge-device motivation. The main RULER evaluations use float32, whereas practical LLM inference typically relies on BF16/FP16 or lower precision. Appendix A reports that a BF16 evaluation caused one 128 B/token/layer configuration to collapse from 0.221 to 0.000 because small activation perturbations crossed FSQ quantization boundaries and accumulated across layers. This raises concerns for the proposed consumer/edge setting. In addition, the value-side reformulation changes the floating-point accumulation order, although it is exact in real arithmetic. Its numerical effect under low-precision accumulation and long contexts is not evaluated.
Questions for authors
Can Transformer-FSQ be adapted to improve the practically useful 5--10$\times$ compression regime, where near-lossless accuracy is still achievable? For example, could a larger latent budget be paired with a directly cached low-bit key representation to avoid per-step key reconstruction, while retaining the compressed/fused value path? Such an operating point may be more practically meaningful than the regime, where absolute accuracy has already degraded substantially. How would this variant compare with KIVI and Palu in both accuracy and serving throughput at matched memory budgets?
2 comments
ACK to response
Thanks to author's response. I think the revised paper and response can address my concern, I will increase the rating from 2->3.
Response to Reviewer nH4H
Thank you for the careful reading, the generous account of the wide low-bit latent and fused value path, and the hybrid proposal, which we built.
With Transformer-FSQ, extreme compression (12x–100x) is viable where others collapse; below 5x others win, as the paper states (RQ1). In brief: 32k saturation (batch 256 at 32x, >=1,024 at 97.5x vs 8 uncompressed) and prefill (1.0 vs 4.6 GB at 42 B at equal TTFT; +10 ms TTFT at 8k; App. G.1) plus 8k batch-16 TPOT/p99 (run for this response) are measured; accuracy and speed overlap from 32x (1.38x, 1.34x, 1.01x native SDPA; App. E.8); the hybrid loses to the shared latent at 8x (0.145 vs 0.557) and 4x (App. G.5); RULER is in bf16 (App. D.5), NF4 8B keeps 0.464 at 32x (E.3), and RULER reaches 128k (E.4–E.5).
"Does not provide TTFT, TPOT, latency percentiles, or throughput-vs-concurrency saturation curves ... Prefill is also not characterized."
At 32k tokens on L40S, FSQ raises max batch from 8 uncompressed to 256 at 32x and >=1,024 at 97.5x and cuts batch-1 unchunked prefill peak memory at 42 B from 4.6 to 1.0 GB above weights on 0.6B at equal TTFT (+~10 ms at 8k; new, App. G.1, Fig. 19), while cutting 8k batch-16 TPOT from 73.5 ms (p99 81.2) to 52.4 ms (p99 53.0) at 32x and 23.4 ms (p99 23.8) at 97.5x (run for this response). Peak 8k throughput is 226, 356 and 911 tok/s (run for this response), and 8k batch-16 TTFT rises 9% (App. G.1). In 32k prefill at 42 B (App. G.1; seconds run for this response), the 11.1 GiB transient was the reference pipeline's: unchunked takes 1.12 s vs 1.12 s, while 2k chunks cut peak to 0.7 GB and add 25%–33% to TTFT (2.63 s vs 1.98 s). So FSQ raises 32k concurrency 32x–128x and cuts batch-16 TPOT for ~10 ms batch-1 TTFT.
"The favorable accuracy–compression and efficiency regimes barely overlap."
From 32x FSQ leads or ties SnapKV (0.22, 0.26 and 0.49 RULER vs 0.72, 0.77 and 0.92 uncompressed; Tables 14–16, new, App. G.2) and decodes at 1.38x, 1.34x and level 1.01x [0.98, 1.05] native SDPA on 0.6B, 1.7B and 8B (1.54x at 97.5x on 8B; 0.84x–0.93x at 12x–16x; new, App. E.8, Table 9). Re-timed against native SDPA, FSQ beats SnapKV on 1.7B (+0.05) and 8B (+0.21) and ties on 0.6B at 32x (see our reply to Reviewer Wfe3). So at 12x–16x the gain is memory/accuracy, and from 32x decode is faster on 0.6B/1.7B and level on 8B (faster at 97.5x).
"Could a larger latent budget be paired with a directly cached low-bit key representation ... compare with KIVI and Palu?"
We built the hybrid you propose at 8x and tested its ceiling at 4x on 0.6B (3 seeds; new, App. G.5): at 8x it scores 0.145 RULER vs 0.557 for FSQ (0.72 uncompressed), trailing Palu (0.364) and 2-bit KIVI at 4.9x (0.32, Table 14), so we report a negative result. At 8x (512 B), 2-bit KIVI keys leave 96 B for values, and the hybrid trails FSQ on all tasks. At 4x (1,024 B) we trained the ceiling first, uncompressed keys plus a value-only latent at over twice the 1,024 B budget (App. G.5): it still scores 0.065 below FSQ's 0.566, so no hybrid beats FSQ there and none was trained. Without a fused kernel the hybrid has no speed figure. So the shared latent wins at both 4x and 8x.
"The numerical-precision setting conflicts with the edge-device motivation."
Every RULER result in the revision is bfloat16, equivalent to float32 in 28 of 32 groups (new, App. D.5), FSQ keeps 0.464 vs 0.493 at 32x on an NF4 8B backbone (0.92 uncompressed; App. E.3), and the fused value path's error is one bf16 rounding (App. E.7). The bf16 root cause and NF4 details are in our reply to Reviewer wRXh. Against float64 (App. E.7), the fused value path's median error is one bf16 rounding across 8k–131k tokens, vs 2x–4x larger for reconstruct-then-attend. So the model runs in bf16 and on NF4 weights, and the fused value path is more accurate than the standard one.
"The long-context capacity claim is largely extrapolated."
RULER is now measured to 128k tokens: FSQ keeps 87% of uncompressed (0.517 vs 0.594) at 32k on Qwen3-0.6B at 8x (new, App. E.4), and on Llama-3.2-1B-Instruct at 128k keeps 74% of control (0.35 vs 0.48) at 4x and 31% (0.15) at 32x (new, App. E.5, Table 7). Baselines at 32k (Palu 0.511 at 5.2x, PyramidKV 0.291, TransMLA 0.000; run for this response) and 8k/32k Llama scores are in our reply to Reviewer wRXh. 3.8M tokens stays a projection; measured capacity is 32x the uncompressed batch at 32k tokens (App. G.1, Fig. 19). So RULER is measured to 128k, and at 32k FSQ leads the baselines we ran at its bytes.
From 32x FSQ leads or ties SnapKV and decodes at or above native SDPA, with measured TTFT, TPOT and 32k saturation, in bf16; at 97.5x at 8k none of our eviction settings fit; below 5x we leave the field to others.
Changes made: serving/prefill (App. G.1); decode (E.8); bf16 (D.5); NF4 (E.3); 128k RULER (E.4–E.5); hybrid, value path (G.5, E.7).
We hope these address W1–W3 and the question, and would be grateful if you would reconsider your score.