acceptodds
Under review as a conference paper at ICLR 2027

Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

Abstract

Private, personalised language models must run on local consumer devices, where frontier models cannot fit and capability depends on what the model retains in context. However, context memory—the key-value (KV) cache—grows linearly with retained cache entries, quickly dwarfing the model itself: for Qwen3-1.7B-Base, 200,000 retained entries require 21 GiB, 6.6 times more than the model weights. While high token capacity is urgently needed on edge hardware, the KV-cache compression methods we measured fail beyond modest compression ratios. We introduce *Transformer-FSQ*, addressing extreme KV-cache compression by replacing standard key and value projections with a finite scalar quantisation bottleneck. Rather than storing large vectors, the cache retains a packed vector of discrete integer coordinates per retained token per layer, requiring no per-token quantization metadata. Converting a pretrained transformer requires training these projections on 0.25 tokens per original parameter.We evaluate Transformer-FSQ against five baseline families: token eviction, scalar and vector quantisation, low-rank projection, and latent attention on models from 0.6B to 8B parameters. On long-context retrieval (RULER), the methods fall into two regimes. Up to about 5, vector quantisation and low-rank projection stay close to the uncompressed model; from 12 to 97.5, Transformer-FSQ is the most accurate method among our primary evaluated implementations, and leads at extreme compression across scales. Extreme KV-cache compression is therefore a practical way to extend how much context a model on a fixed device can hold.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, finite scalar quantisation, discrete latents, long-context inference, memory-efficient inference, on-device language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nH4HOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Transformer-FSQ, a post-training KV-cache compression method that replaces conventional per-token K/V storage with a shared wide low-bit discrete latent. The latent is produced from the attention input hidden state using finite scalar quantization (FSQ), and keys/values are reconstructed through learned linear projections; the value path further exploits linearity to aggregate directly in latent space before applying the value projection. The method can retrofit pretrained Transformers while freezing the backbone. Experiments on Qwen3 models show that Transformer-FSQ is particularly competitive in the extreme-compression regime, outperforming the evaluated eviction, quantization, and low-rank baselines beyond roughly 12× compression, with extensive ablations and byte-level memory accounting.

Critical strengths
  • S1. Novel wide low-bit latent formulation for extreme KV-cache compression. The paper combines latent dimensionality reduction with low-bit quantization under a fixed memory budget, allowing the two compression axes to compound. This design pushes KV-cache compression substantially beyond the regimes typically explored by either low-rank projection or quantization alone.

  • S2. Clever algorithmic and implementation design. FSQ avoids the explicit codebooks, nearest-neighbor lookup, and associated metadata overhead of conventional VQ, allowing a larger fraction of the memory budget to store useful latent information. Moreover, the method exploits linearity in the value path, computing the attention-weighted reduction directly over cached latents and applying the value projection only afterward, thus avoiding explicit reconstruction of the full value cache and reducing value-side memory traffic and computation.

  • S3. Strong results in the extreme-compression regime with a practical post-training conversion procedure. Transformer-FSQ maintains substantially better quality than the evaluated baselines at very high compression ratios, while requiring only post-training conversion and finetuning rather than pretraining a new architecture from scratch. The paper also provides extensive ablations and careful byte-level accounting of cache storage, including auxiliary metadata and other method-specific overheads.

Critical weaknesses
  • W1. The favorable accuracy--compression and efficiency regimes barely overlap. Transformer-FSQ mainly outperforms baselines in the extreme-compression regime, where absolute quality has already degraded substantially. The paper itself states that at ≤5×\leq5\times, VecInfer and Palu retain >93%>93\% of uncompressed RULER accuracy, while Transformer-FSQ becomes advantageous mainly beyond 12×12\times (Sec. 5.2, RQ1). In Tables 13--14, Transformer-FSQ already drops from 0.71 to 0.58 at 4×4\times on Qwen3-0.6B and from 0.77 to 0.65 at 4×4\times on Qwen3-1.7B. Meanwhile, its decode throughput is only 0.12--0.13$\times$ of uncompressed at 4×4\times (Tables 11--12), due to repeated key reconstruction and RoPE. Speedups appear only at much more aggressive compression (Fig. 6 / Sec. G.2), where accuracy is already severely degraded. Thus, the method advances the extreme-compression frontier, but its benefit in a practically usable serving regime remains unclear.

  • W2. The end-to-end serving benefit is insufficiently evaluated. The paper reports decode throughput over several batch sizes and context lengths, but does not provide TTFT, TPOT, latency percentiles, or throughput-vs-concurrency saturation curves. This is important because Transformer-FSQ trades memory traffic for substantial additional computation: Sec. G.2 explicitly gives the key-reconstruction cost as O(TGnkvdh)O(TG n_{kv}d_h) per layer and decoding step. Prefill is also not characterized in latency: Appendix D.5 reports about 11.1 GiB transient memory per sequence at 8K context and OOM at batch 4 in the reference pipeline, while chunked prefill/FlashAttention are only suggested as remedies. Moreover, the long-context capacity claim is largely extrapolated: Sec. 5.3 explicitly states that the 3.8M-token projection extrapolates 8,192-token scores and “does not establish million-token retrieval accuracy.” Longer-context and full serving-capacity evaluations are therefore needed.

  • W3. The numerical-precision setting conflicts with the edge-device motivation. The main RULER evaluations use float32, whereas practical LLM inference typically relies on BF16/FP16 or lower precision. Appendix A reports that a BF16 evaluation caused one 128 B/token/layer configuration to collapse from 0.221 to 0.000 because small activation perturbations crossed FSQ quantization boundaries and accumulated across layers. This raises concerns for the proposed consumer/edge setting. In addition, the value-side reformulation ∑tatWvzt=Wv(∑tatzt)\sum_t a_tW_vz_t=W_v(\sum_ta_tz_t) changes the floating-point accumulation order, although it is exact in real arithmetic. Its numerical effect under low-precision accumulation and long contexts is not evaluated.

Questions for authors

Can Transformer-FSQ be adapted to improve the practically useful 5--10$\times$ compression regime, where near-lossless accuracy is still achievable? For example, could a larger latent budget be paired with a directly cached low-bit key representation to avoid per-step key reconstruction, while retaining the compressed/fused value path? Such an operating point may be more practically meaningful than the ∼100×\sim100\times regime, where absolute accuracy has already degraded substantially. How would this variant compare with KIVI and Palu in both accuracy and serving throughput at matched memory budgets?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nH4H

Thank you for the careful reading, the generous account of the wide low-bit latent and fused value path, and the hybrid proposal, which we built.

With Transformer-FSQ, extreme compression (12x–100x) is viable where others collapse; below 5x others win, as the paper states (RQ1). In brief: 32k saturation (batch 256 at 32x, >=1,024 at 97.5x vs 8 uncompressed) and prefill (1.0 vs 4.6 GB at 42 B at equal TTFT; +10 ms TTFT at 8k; App. G.1) plus 8k batch-16 TPOT/p99 (run for this response) are measured; accuracy and speed overlap from 32x (1.38x, 1.34x, 1.01x native SDPA; App. E.8); the hybrid loses to the shared latent at 8x (0.145 vs 0.557) and 4x (App. G.5); RULER is in bf16 (App. D.5), NF4 8B keeps 0.464 at 32x (E.3), and RULER reaches 128k (E.4–E.5).

"Does not provide TTFT, TPOT, latency percentiles, or throughput-vs-concurrency saturation curves ... Prefill is also not characterized."

At 32k tokens on L40S, FSQ raises max batch from 8 uncompressed to 256 at 32x and >=1,024 at 97.5x and cuts batch-1 unchunked prefill peak memory at 42 B from 4.6 to 1.0 GB above weights on 0.6B at equal TTFT (+~10 ms at 8k; new, App. G.1, Fig. 19), while cutting 8k batch-16 TPOT from 73.5 ms (p99 81.2) to 52.4 ms (p99 53.0) at 32x and 23.4 ms (p99 23.8) at 97.5x (run for this response). Peak 8k throughput is 226, 356 and 911 tok/s (run for this response), and 8k batch-16 TTFT rises 9% (App. G.1). In 32k prefill at 42 B (App. G.1; seconds run for this response), the 11.1 GiB transient was the reference pipeline's: unchunked takes 1.12 s vs 1.12 s, while 2k chunks cut peak to 0.7 GB and add 25%–33% to TTFT (2.63 s vs 1.98 s). So FSQ raises 32k concurrency 32x–128x and cuts batch-16 TPOT for ~10 ms batch-1 TTFT.

"The favorable accuracy–compression and efficiency regimes barely overlap."

From 32x FSQ leads or ties SnapKV (0.22, 0.26 and 0.49 RULER vs 0.72, 0.77 and 0.92 uncompressed; Tables 14–16, new, App. G.2) and decodes at 1.38x, 1.34x and level 1.01x [0.98, 1.05] native SDPA on 0.6B, 1.7B and 8B (1.54x at 97.5x on 8B; 0.84x–0.93x at 12x–16x; new, App. E.8, Table 9). Re-timed against native SDPA, FSQ beats SnapKV on 1.7B (+0.05) and 8B (+0.21) and ties on 0.6B at 32x (see our reply to Reviewer Wfe3). So at 12x–16x the gain is memory/accuracy, and from 32x decode is faster on 0.6B/1.7B and level on 8B (faster at 97.5x).

"Could a larger latent budget be paired with a directly cached low-bit key representation ... compare with KIVI and Palu?"

We built the hybrid you propose at 8x and tested its ceiling at 4x on 0.6B (3 seeds; new, App. G.5): at 8x it scores 0.145 RULER vs 0.557 for FSQ (0.72 uncompressed), trailing Palu (0.364) and 2-bit KIVI at 4.9x (0.32, Table 14), so we report a negative result. At 8x (512 B), 2-bit KIVI keys leave 96 B for values, and the hybrid trails FSQ on all tasks. At 4x (1,024 B) we trained the ceiling first, uncompressed keys plus a value-only latent at over twice the 1,024 B budget (App. G.5): it still scores 0.065 below FSQ's 0.566, so no hybrid beats FSQ there and none was trained. Without a fused kernel the hybrid has no speed figure. So the shared latent wins at both 4x and 8x.

"The numerical-precision setting conflicts with the edge-device motivation."

Every RULER result in the revision is bfloat16, equivalent to float32 in 28 of 32 groups (new, App. D.5), FSQ keeps 0.464 vs 0.493 at 32x on an NF4 8B backbone (0.92 uncompressed; App. E.3), and the fused value path's error is one bf16 rounding (App. E.7). The bf16 root cause and NF4 details are in our reply to Reviewer wRXh. Against float64 (App. E.7), the fused value path's median error is one bf16 rounding across 8k–131k tokens, vs 2x–4x larger for reconstruct-then-attend. So the model runs in bf16 and on NF4 weights, and the fused value path is more accurate than the standard one.

"The long-context capacity claim is largely extrapolated."

RULER is now measured to 128k tokens: FSQ keeps 87% of uncompressed (0.517 vs 0.594) at 32k on Qwen3-0.6B at 8x (new, App. E.4), and on Llama-3.2-1B-Instruct at 128k keeps 74% of control (0.35 vs 0.48) at 4x and 31% (0.15) at 32x (new, App. E.5, Table 7). Baselines at 32k (Palu 0.511 at 5.2x, PyramidKV 0.291, TransMLA 0.000; run for this response) and 8k/32k Llama scores are in our reply to Reviewer wRXh. 3.8M tokens stays a projection; measured capacity is 32x the uncompressed batch at 32k tokens (App. G.1, Fig. 19). So RULER is measured to 128k, and at 32k FSQ leads the baselines we ran at its bytes.

From 32x FSQ leads or ties SnapKV and decodes at or above native SDPA, with measured TTFT, TPOT and 32k saturation, in bf16; at 97.5x at 8k none of our eviction settings fit; below 5x we leave the field to others.

Changes made: serving/prefill (App. G.1); decode (E.8); bf16 (D.5); NF4 (E.3); 128k RULER (E.4–E.5); hybrid, value path (G.5, E.7).

We hope these address W1–W3 and the question, and would be grateful if you would reconsider your score.

Reviewer_nH4HOpenReview ↗

ACK to response

Thanks to author's response. I think the revised paper and response can address my concern, I will increase the rating from 2->3.

Reviewer_Wfe3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for compressing the key–value (KV) cache using finite scalar quantization (FSQ), originally introduced for learning discrete representations in VQ-VAEs and related generative models. The method learns encoder and decoder projections that map KV-cache entries to and from a compact discrete representation whose implicit codebook is a Cartesian product of finite scalar quantization levels. These projections are trained on a representative text corpus. The method is evaluated on RULER and a task from LongBench.

Critical strengths
  • The proposed method achieves a strong trade-off between KV-cache compression and model quality, particularly at high compression ratios.
Critical weaknesses
  • The evaluation could be more comprehensive. RULER includes many needle-in-a-haystack retrieval tasks, although it also covers multi-hop tracing, aggregation, and question answering. In retrieval tasks where only a small portion of the context is relevant to the answer, competitive token-eviction methods may achieve substantial compression with limited quality degradation. Direct comparisons would help establish the benefits of FSQ in these settings. Broader evaluation on reasoning benchmarks, such as AIME, GPQA Diamond, and MATH-500, as well as context-intensive tasks such as Text2JSON [1], would provide a more complete assessment. Text2JSON requires extracting structured information from across the input, making it useful for testing whether compression preserves information beyond a small set of relevant tokens.

  • Training the compression modules requires substantial data: 125 million tokens for the 0.6B model and 2 billion tokens for Qwen3-8B. The associated computational cost may make training-free alternatives preferable in some deployment scenarios.

  • The method introduces substantial decoding overhead, particularly at lower compression ratios. At very high compression ratios, FSQ provides some speedup over the uncompressed baseline. However, existing KV-cache quantization methods, such as KIVI [2] and KVQuant [3], and token-eviction methods, such as SnapKV [4], Ada-KV [5], and PyramidKV [6], have reported much more significant decoding speed-ups. Given the decoding overhead, comparisons with systems that use KV-cache offloading, such as ShadowKV [7], PQCache [8], and MagicPIG [9], would also be informative, as the inference latency is likely to be comparable.

  • At a moderate compression ratio of 4×, FSQ appears to underperform the baselines in model quality, with an already noticeable degradation relative to the uncompressed model. This limits its appeal at compression ratios that may be sufficient for many practical applications.

[1] Bocharnikov, A., Ermakov, I., Kuznedelev, D., Zhdanovskiy, V., and Yershov, Y. (2026). KV Cache Offloading for Context-Intensive Tasks. arXiv:2604.08426. Paper · Google Scholar

[2] Liu, Z., et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv:2402.02750. Paper · Google Scholar

[3] Hooper, C., et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv:2401.18079. Paper · Google Scholar

[4] Li, Y., et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation. arXiv:2404.14469. Paper · Google Scholar

[5] Feng, Y., et al. (2024). Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. arXiv:2407.11550. Paper · Google Scholar

[6] Cai, Z., et al. (2024). PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv:2406.02069. Paper · Google Scholar

[7] Sun, H., et al. (2024). ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. arXiv:2410.21465. Paper · Google Scholar

[8] Zhang, H., et al. (2024). PQCache: Product Quantization-based KVCache for Long Context LLM Inference. arXiv:2407.12820. Paper · Google Scholar

[9] Chen, Z., et al. (2024). MagicPIG: LSH Sampling for Efficient LLM Generation. arXiv:2410.16179. Paper · Google Scholar

Questions for authors
  • Could using the Muon optimizer instead of Adam accelerate convergence when training the FSQ encoder and decoder?

  • Could you include a comparison with AQUA-KV [10]? This method exploits dependencies between KV caches in adjacent layers using learned predictors and quantizes the prediction residuals. Such a comparison would establish whether the use of FSQ is the main advantage of the method, or the use of learned projections.

[10] Shutova, A., et al. (2025). Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models. Introduces AQUA-KV. arXiv:2501.19392. Paper · Google Scholar

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wfe3

Thank you for noting the strong compression-quality trade-off at high ratios.

With Transformer-FSQ, extreme compression (12x–100x) is viable; we do not claim 4x (RQ1). In brief: eviction is compared (FSQ beats PyramidKV from 12x, leads SnapKV in 6 of 8 cells and ties in 2, Ada-KV within 0.022 of SnapKV; App. G.2); AQUA-KV wins at 4x and FSQ leads from 8x (App. G.3); from 32x FSQ decodes at 1.38x and 1.34x native SDPA on 0.6B/1.7B, level on 8B (0.84x–0.93x at 12x–16x; App. E.8); Muon does not help and 1/10 data keeps 90% RULER at 16x on 1.7B (App. G.4); on MATH-500 at 12x FSQ is level with PyramidKV (App. G.6); offloading is not benchmarked.

"Direct comparisons [with token eviction] would help."

The submission compared PyramidKV (layer-wise SnapKV), and we added SnapKV and Ada-KV (new, App. G.2); FSQ beats PyramidKV from 12x and leads SnapKV by 0.05–0.21 in 6 of 8 cells across 4x–32x, tying in 2 (0.6B at 32x, 8B at 12x; Ada-KV within 0.022 of SnapKV). FSQ leads PyramidKV by 0.12 at 12x and 0.10 at 32x on 0.6B (0.72 uncompressed) and 0.25 at 32x on 8B (0.92 uncompressed; RQ1, bf16). At matched bytes (App. G.2): FSQ leads SnapKV at 4x and 12x on 0.6B and 1.7B and at 32x on 1.7B (+0.05; 0.77 uncompressed) and 8B (+0.21). Eviction wins non-needle tasks, FSQ single-needle ones (Fig. 20); at 97.5x (42 B) at 8k, none of our eviction settings fit. So from 12x FSQ matches or beats every eviction method we ran.

"whether the use of FSQ is the main advantage, or the use of learned projections"

The grid alone does not appear to be the advantage. To separate grid from projections, we kept FSQ's learned projections and replaced only its grid with HIGGS, the quantiser AQUA-KV uses (Malinovskii et al., 2025). FSQ minus HIGGS is +3.8 points [-12, +19] at 512 B and +0.4 [-5, +6] at 256 B (3 seeds): not distinguishable. AQUA-KV wins near 4x (0.722 vs 0.566 on 0.6B, 0.72 uncompressed) and FSQ leads from 8x (0.557 vs ~0.40, with no AQUA-KV setting at >=12x; new, App. G.3, Fig. 21), while RQ3 (Table 2, App. F.3) shows both the learned projection and QAFT on FSQ's metadata-free grid contribute (+0.045 over 4-bit min-max QAFT at 28x).

"Could using the Muon optimizer ... accelerate convergence?"

We tried Muon as suggested (new, App. G.4); it does not accelerate convergence, ending 0.0012 bits/byte worse in loss and 0.064 [-0.104, -0.023] lower on RULER than AdamW .

"requires substantial data"

To address the reviewer’s concern, we ran new data-efficiency ablations: reducing finetuning data 10× causes no statistically detectable RULER drop at 4×–8× on 0.6B and retains 90% of full-data performance at 16× on 1.7B (93% with one-third the data; App. G.3). Full finetuning takes 2.6 GPU-hours on 0.6B and 138 on 8B (Table 5). Closed-form conversion (Table 12) keeps 1.02 of finetuned RULER at 8x on 0.6B and 1.04 at 4x on 1.7B (0.02 at 8x on 1.7B). Thus, substantially less data suffices at moderate compression, while additional training benefits extreme ratios where competing baselines already degrade sharply.

"At ... 4×, FSQ appears to underperform."

Agreed, and the submission says so (RQ1): at 4x on 0.6B (0.72 uncompressed), Palu (0.709) and AQUA-KV (0.722) beat FSQ (0.566). So the claim starts at 12x, where it holds.

"reasoning benchmarks ... Text2JSON"

On MATH-500 (new, App. G.6), Qwen 8B FSQ at 12x scores 19.4% vs PyramidKV's 17.0% (+2.4 [-1.8, +6.6]; 55.4% uncompressed); Text2JSON, AIME and GPQA were not run. On Llama-3.2-1B (run for this response), MATH-500 falls from 18.2% uncompressed to 9.8% at 4x. So FSQ is level with eviction on MATH-500, and we make no reasoning claim. Reasoning performance degrades with other baselines as well as ours.

"systems that use KV-cache offloading"

We have not benchmarked offloading systems, we consider it an orthogonal improvement. Device memory is bounded by the fetched subset and latency by transfer; the comparison remains open.

"substantial decoding overhead"

We provide a new kernel for significantly improved decoding throughput. From 32x FSQ decodes faster than uncompressed on 0.6B (1.38x) and 1.7B (1.34x), level on 8B (1.01x [0.98, 1.05]), and ~2x at 97.5x (1.54x on 8B), vs 0.84x–0.93x at 12x–16x, beating Palu's kernel in 88%–100% of matched-byte settings (new, App. E.8, Table 9). At 4x on 0.6B FSQ is 0.27x, and KIVI's kernel is faster in 85% of settings. So below 12x decode is an overhead, at 12x–16x within 7%–16% of native SDPA, and from 32x a speed-up too. This is a result of an engineering task, which could be further optimised.

Below 5x others win, as we state; by 8x FSQ passes AQUA-KV; at 12x–32x it beats PyramidKV and leads or ties SnapKV; from 32x it decodes at or above native SDPA; at 97.5x at 8k none of our eviction settings fit.

Changes made: decode (E.8); eviction, AQUA-KV, data, Muon, MATH-500 (G.2–G.6).

We hope these address your concerns and would be grateful if you reconsidered your score.

Thank you for your response and the new results. While your comments and the newly added results address some of my concerns, I have decided to maintain my original score after considering your responses to me and the other reviewers.

AuthorsOpenReview ↗

Further considerations

Thanks for considering. For information and for your considerations we have also provided updated responses to the other reviewers. Are the particular concerns you had that you felt were unaddressed? We feel that your comments have significantly helped to shape the improvements to the paper, and we would hope that we have satisfied the majority of your concerns.

Reviewer_wRXhOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper moves KV-cache compression upstream of the key and value projections. Each layer maps the residual stream through a linear encoder into a wide low-bit latent, rounds every coordinate onto FSQ's fixed grid, and caches only the packed integer code, one per token per layer, shared by all KV heads. A single linear decoder reconstructs all keys and values, and values are decoded after attention pooling because the value projection commutes with the weighted sum. Pretrained models are converted in closed form from calibration activations, and only the inserted tensors are finetuned on 0.25 tokens per original parameter with the backbone frozen. The evaluation covers Qwen3 0.6B, 1.7B and 8B against five baseline families under one byte budget; the main metric is RULER at 8,192 tokens, supported by LongBench, decode throughput and memory measurements. The finding is that baselines are stronger up to about 5x, while Transformer-FSQ is the most accurate evaluated configuration from 12x to 97.5x.

Critical strengths
  1. The compression point is well chosen and the design is clean. Caching one discrete code per token per layer that serves all heads and both K and V removes codebooks, per-token scale metadata and per-head duplication. This makes equal-byte comparison meaningful and lets the method compose with token eviction. At 12x it reaches RULER 0.39, 0.45 and 0.62 on the three models, ahead of eviction and continuous latents at matched budgets.
  2. The accounting and disclosure are unusually careful. Every cache is priced by one byte solver, measured device memory matches the analytic value within 0.05 percent, and the appendix reports per-task tables, baseline deviations and limitations including single-seed 8B runs and the 8k-only retrieval protocol. The reported numbers are auditable.
Critical weaknesses
  1. The precision requirement undermines the deployment story. Appendix A reports that a 128 byte configuration falls from 0.221 to 0.000 in bfloat16, because activation noise in the frozen backbone flips coordinates across grid boundaries, with a mean flip rate of 0.064 by layer 14; keeping only the encoder in float32 does not prevent the collapse. The whole model must therefore run in float32. That doubles weight memory, from 1.25 GB to 2.50 GB for Qwen3-0.6B and to roughly 30.6 GiB for Qwen3-8B, so the 8B model cannot fit on the 16 GiB device that motivates the paper. The headline capacity figures in Figure 1 and the abstract do not hold at the only precision at which the method works. No net memory accounting including float32 weights is given, and no quantisation-aware training or boundary-regularisation result is reported.
  2. The extreme-compression win rests on baselines that have already collapsed, and the evidence base is narrow. At 32x the strongest eviction baseline scores 0.11, 0.15 and 0.26 while Transformer-FSQ scores 0.22, 0.26 and 0.50, which is 31, 34 and 54 percent of the uncompressed model; at 97.5x it scores only 0.04, 0.09 and 0.23. On 8B, TransMLA reaches 0.02 at 8x whereas the same method reaches 0.19 at 0.6B; Palu falls from 0.37 at 10.2x to 0.12 at 15.1x; VecInfer falls from 0.68 at 4.3x to 0.04 at 12.7x. Such cliffs look like baselines pushed outside their working range rather than properties of the methods. All finetuned methods share one learning rate and one token budget, and the sensitivity check is on bits per byte rather than accuracy. Combined with a single 8B seed and RULER measured only at 8,192 tokens, this does not support the abstract's claim that extreme compression is a practical way to extend context. Decode throughput does not compensate: 0.78x the uncompressed speed at 32x and 0.07x at 2x on Qwen3-0.6B.
Questions for authors
  1. Can you provide a configuration that works in bfloat16, for example with quantisation-aware training or boundary noise, and report end-to-end device memory including float32 weights? If 8B requires float32, how should the 16 GiB capacity claims be restated?
  2. TransMLA scores 0.02 at 8x on 8B but 0.19 at 0.6B, and Palu and VecInfer also drop to near zero around 12x. Were these baselines tuned at those ratios, and does the Pareto frontier survive per-method sweeps of learning rate and training tokens measured by accuracy?
  3. All RULER results are at 8,192 tokens while the method targets much longer contexts. Can you add 16k or 32k retrieval results? On 8B narrativeqa, PyramidKV scores 11.5 at 73x while Transformer-FSQ scores 7.2 at 97.5x against 17.4 uncompressed, and on trec Transformer-FSQ drops from 63.0 to 34.5 at 32x. Is the summarisation claim specific to gov_report?
  4. Peak memory in batched evaluation is dominated by the 11.1 GiB prefill transient, and batch 4 runs out of memory. With chunked prefill or FlashAttention, how much of the projected 3.8M-token capacity remains? That figure is currently an extrapolation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer wRXh

Thank you for noting that the compression point is well chosen and the accounting careful and auditable.

With Transformer-FSQ, extreme compression (12x–100x) is usable where others collapse; below 5x others win, as the paper states (RQ1). In brief: FSQ works in bfloat16 (28/32 groups) and on an NF4 8B backbone (0.464 vs 0.493 at 32x); RULER is measured to 128k tokens; eviction had not collapsed and FSQ beats PyramidKV and leads or ties SnapKV at 32x; tuning does not close the gap; from 32x FSQ decodes at 1.38x and 1.34x on 0.6B/1.7B, level on 8B (1.54x at 97.5x), and raises 32k batch 32x.

"The whole model must therefore run in float32."

Every RULER result is now bfloat16, equivalent to float32 in 28 of 32 groups (new, App. D.5), and FSQ finetuned on a 4-bit NF4 8B backbone keeps 0.464 RULER at 32x vs 0.493 in 16-bit (0.92 uncompressed; new, App. E.3). The submission's bf16 pass rounded query and key rotary tables differently; the revision uses the model's one table (checked for this response). The 128 B case scores 0.22 in bf16 (0.72 uncompressed); Fig. 1 is in bf16 (1.39M cached tokens on 16 GiB). On NF4 8B at 32x (App. E.3), FSQ is 0.029 [0.006, 0.054] behind 16-bit. So deployment holds in bf16 and on 4-bit weights.

"... rests on baselines that have already collapsed."

The eviction baseline had not collapsed at 12x or 32x, and FSQ led it by 0.10–0.25 at 32x; against SnapKV and Ada-KV (new, App. G.2), FSQ leads by 0.05–0.21 in 6 of 8 cells across 4x–32x and ties in 2 (0.6B at 32x, 8B at 12x). PyramidKV trails FSQ by 0.12 at 12x and 0.10 at 32x on 0.6B (0.72 uncompressed) and 0.25 at 32x on 8B (0.92 uncompressed; RQ1, bf16). At 32x SnapKV trails FSQ by +0.05 on 1.7B and +0.21 on 8B (Ada-KV within 0.022 of SnapKV ; App. G.2; see our reply to Reviewer Wfe3). At 97.5x (42 B) at 8k, none of our eviction settings fit, and FSQ gets 0.23 on 8B (25% of 0.92, 1 seed). So FSQ beats active eviction at 12x–32x, and at 97.5x none of those fit.

"Were these baselines tuned at those ratios?"

Per-method sweeps of learning rate and tokens at 16x on 0.6B, scored by RULER, leave the best baseline (Palu at 3e-5) at 0.089 vs FSQ's 0.32 (-0.23 [-0.25, -0.22], 3 seeds; 0.72 uncompressed; new, App. D.2, G.4), and on 8B TransMLA fails past ~3k tokens (App. G.8). Over a 10x token range (App. G.4) Palu and TransMLA stay <0.10 while FSQ rises by 0.089 to 0.32 (Table 14). On 8B (App. G.8), TransMLA's loss gap jumps from 0.22 in the first 1k tokens to 2.19 over the last 4k. So tuning by accuracy does not change the comparison.

"Is the summarisation claim specific to gov_report?"

gov_report is the paper's only summarisation task, and we claim no more; on narrativeqa (QA) and trec (classification), eviction leads at high compression on 8B (App. E.1, Table 19). On 8B narrativeqa, PyramidKV gets 11.5 at 73x vs FSQ's 8.5 at 32x and 6.5 at 97.5x (19.7 uncompressed). So the summarisation claim is gov_report-specific.

"Decode throughput does not compensate."

From 32x FSQ decodes at 1.38x and 1.34x native SDPA on 0.6B and 1.7B and level (1.01x [0.98, 1.05], 1.54x at 97.5x) on 8B, vs 0.84x–0.93x at 12x–16x and slower below (new, App. E.8, Table 9; see our reply to Reviewer Wfe3). So at 12x–16x the gain is memory/accuracy, and from 32x speed too.

"How much of the projected 3.8M-token capacity remains?"

32 times the uncompressed batch is measured at 32k tokens on L40S (256 vs 8 at 32x; new, App. G.1, Fig. 19), and storing only codes cuts 32k batch-1 prefill peak memory at 42 B from 4.6 to 1.0 GB on 0.6B at equal TTFT, while 3.8M remains a projection. The 11.1 GiB transient was the reference pipeline's; at 42 B (App. G.1), 2k chunks cut the peak to 0.7 GB and add 25%–33% to prefill time (at 8k, TTFT adds ~10 ms; see our reply to Reviewer nH4H). So the gain is measured at 32k and projected beyond.

"Can you add 16k or 32k retrieval results?"

RULER is now measured to 128k tokens: FSQ keeps 87% of uncompressed (0.517 vs 0.594) at 32k on Qwen3-0.6B at 8x (new, App. E.4), and on Llama-3.2-1B-Instruct at 128k keeps 74% of control (0.35 vs 0.48) at 4x and 31% (0.15) at 32x (App. E.5, Table 7). At 32k on Qwen (8 seeds; baselines run for this response), Palu gets 0.511 at 5.2x, PyramidKV 0.291 and TransMLA 0.000 at FSQ's bytes. On Llama-3.2-1B (Table 7; control 0.72, 0.59, 0.48 at 8k, 32k, 128k), FSQ keeps 86%, 89%, 74% at 4x and 44%, 39%, 31% at 32x. So at 32k FSQ leads the baselines we ran at its bytes.

In bf16 and on NF4 weights, with RULER measured to 128k and baselines tuned: from 32x FSQ leads or ties eviction and decodes at or above native SDPA, holding up to 1.39M cached tokens on 16 GiB (Fig. 1) and 32x the 32k batch (App. G.1, Fig. 19).

Changes made: bf16 (App. D.5); LR (D.2); NF4 (E.3); 128k RULER (E.4–E.5); decode (E.8); serving (G.1); eviction, tuning, TransMLA (G.2, G.4, G.8).

We hope these resolve the soundness concerns and would be grateful if you reconsidered your score.

Thank you for the detailed response and the additional experiments, which have helped clarify several aspects of the paper and increased my review confidence.

However, my concern regarding the sharp collapse of VecInfer at higher compression ratios (from 0.68 to 0.04 around 12.7×) and whether it was adequately tuned—a broader concern about the method's practical usability also raised by another reviewer (Reviewer nH4H)—remains unaddressed.

Therefore, I will maintain my current rating while raising my confidence score.

Thank you for raising these two points. We address VecInfer's configuration and the practical usability regime below.

1. VecInfer configuration at high compression

VecInfer's collapse around twelve-fold compression is a limit of the method at one to one-and-a-quarter bits per coordinate, not an undertuned baseline.

Because VecInfer is training-free, its entire tuning space at a fixed byte budget consists of four hyperparameters—sub-vector group dimension d, codebook bits (b_K, b_V), residual window R, and calibration sample size—and sweeping all of them leaves VecInfer at 0.124 or below, trailing Transformer-FSQ by at least 0.328.

ModelRatio (Bytes)dBits (K/V)Centroids (K/V)Window RCalibVecInferFSQDiff (Vec - FSQ, 95% CI)
0.6B (our paper)12.72× (322 B)88 / 8256 / 2561284k0.0400.383-0.342
0.6B12.72× (322 B)11 / 12 / 21284k0.0190.383-0.364
0.6B12.72× (322 B)22 / 24 / 41284k0.0230.383-0.360
0.6B12.72× (322 B)44 / 416 / 161284k0.0370.383-0.346
0.6B12.72× (322 B)88 / 8256 / 25612816k0.0440.383-0.339
0.6B12.72× (322 B)810 / 61024 / 641284k0.0460.383-0.337 [-0.370, -0.303]
0.6B (VecInfer 1.25-bit)10.61× (386 B)810 / 101024 / 10241284k0.0550.383-0.328 [-0.362, -0.293]
0.6B9.10× (450 B)46 / 664 / 641284k0.0620.556-0.494
1.7B (our paper)12.72× (322 B)88 / 8256 / 2561284k0.0670.431-0.364
1.7B (VecInfer 1.25-bit)10.61× (386 B)810 / 101024 / 10241284k0.0750.431-0.356 [-0.391, -0.320]
1.7B (VecInfer 1.25-bit)10.61× (386 B)812 / 84096 / 2561284k0.0780.431-0.353 [-0.388, -0.317]
8B (our paper)12.72× (322 B)88 / 8256 / 2561284k0.0870.614-0.527
8B12.72× (322 B)810 / 61024 / 641284k0.0760.614-0.538 [-0.572, -0.504]
8B (VecInfer 1.25-bit)10.61× (386 B)810 / 101024 / 10241284k0.0940.614-0.520 [-0.555, -0.484]
8B (VecInfer 1.5-bit)9.10× (450 B)812 / 124096 / 40961284k0.1240.664-0.540 [-0.575, -0.505]

Originally, our budget solver searched all symmetric layouts up to 8-bit (256 centroids, the largest codebook supported by VecInfer's released CUDA kernel) and selected the largest codebook fitting the budget (d=8, b=8 at 322 B). Even after removing the 8-bit and symmetry caps, VecInfer's required 128-token fp16 residual window takes 64 B (66 B with 4 sinks) at 8k context, leaving at most 1.0–1.25 bits/coordinate for the remaining ~8,060 tokens, which is too coarse for multi-key retrieval; the full port (ydyhello/VecInfer, post-RoPE K smoothing and Hadamard rotation) and tests (tests/baselines/test_vecinfer.py) have been in https://github.com/trofsq/paper-code since submission (Table 1, App. D.7).

2. Practical deployability and throughput

Our revised decode throughput measurements alleviate the practical deployability concern: Transformer-FSQ matches or exceeds VecInfer's kernel throughput while operating at higher accuracy and smaller cache footprints.

On Qwen3-8B, Transformer-FSQ at 97.5× (42 B) decodes faster than VecInfer's pooled kernel (1.54× vs. 1.40× native SDPA) while still outscoring VecInfer's 12.7× (322 B) cache (0.232 vs. 0.087 RULER-8k).

ModelMethod (Ratio, Bytes)Decode Speed (vs. Native SDPA)RULER-8k
0.6BVecInfer (pooled; 12.7× / 322 B RULER)1.66× [1.11, 3.03]0.040
0.6BFSQ (32×, 128 B)1.38× [1.02, 1.44]0.22
0.6BFSQ (97.5×, 42 B)2.00× [1.03, 3.12]0.063
1.7BVecInfer (pooled; 12.7× / 322 B RULER)1.79× [1.27, 2.88]0.067
1.7BFSQ (32×, 128 B)1.34× [1.14, 1.42]0.26
1.7BFSQ (97.5×, 42 B)2.17× [1.24, 2.83]0.101
8BVecInfer (pooled; 12.7× / 322 B RULER)1.40× [1.29, 1.78]0.087
8BFSQ (32×, 128 B)1.01× [0.98, 1.05]0.493 (0.464 NF4)
8BFSQ (97.5×, 42 B)1.54×0.232

VecInfer's released kernel achieves high speedup only at larger cache sizes (4.3× / 962 B) or after accuracy has collapsed (12.7× / 322 B), whereas Transformer-FSQ's fused decode kernel turns memory-bandwidth savings into 1.01×–2.17× speedups while preserving usable retrieval at 32×–97.5× (Table 9, App. E.3, E.8).

AuthorsOpenReview ↗

Follow-up to Reviewer wRXh's Response

Thank you for pressing on this. Following it up made the comparison more complete, and VecInfer remains collapsed at 12.7× even under an uncompressed prefill protocol.

Your question led us to check one more difference between our setup and VecInfer's. VecInfer attends to exact keys and values while reading the prompt, and compresses only the cache it keeps for generation (VecInfer, Fig. 4a; released code). We first confirmed that our implementation reproduces VecInfer's published results under that protocol: on its own benchmark (LongBench qasper, Llama-3.1-8B-Instruct, the authors' released codebooks), it scores 40.5 F1 against the 41.2 reported in the VecInfer paper.

We then reran the RULER-8k experiment you asked about with this uncompressed prefill protocol, over all 650 items:

ModelRatio (Bytes)PrefillVecInferFSQ (12.0×, 341 B)
0.6B12.72× (322 B)exact (VecInfer's protocol)0.0550.383

At this ratio VecInfer's codes carry one bit per coordinate once its 128-token full-precision window is paid for. Neither the configuration sweep in our previous reply nor VecInfer's own prefill protocol lifts it.

Your question also made us state this difference in the paper: the revision now says, for each baseline, whether it attends to exact or compressed keys and values while reading the prompt (App. D.7), so readers can see exactly how every method was run.

Following up your VecInfer question also led us to a change that we think makes the paper's practical case considerably stronger.

VecInfer's protocol needs the full-precision cache in memory while reading the prompt. Transformer-FSQ attends to its compressed cache instead, and every accuracy in the paper was measured this way, but our inference code decoded that cache back to full precision before attending. We have now written a prefill that decodes only a small part of the cache at a time. It computes the same attention, and at 97.5× and 128k tokens on Qwen3-0.6B, chunked prefill now needs 261 MB above the weights instead of 2.6 GB, so prefill no longer limits how much context fits on a device (App. G.1).

You wrote that "decode throughput does not compensate" and that the evidence "does not support the abstract's claim that extreme compression is a practical way to extend context". Reviewer nH4H raised the same concern, that "the favorable accuracy–compression and efficiency regimes barely overlap". In our response to their concerns, which you also mentioned, we addressed this by showing that the two regimes now overlap from 32×. We hope that this, together with the changes above, addresses your concerns, and we would be glad to hear if anything remains open.

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript, with new content highlighted in green and modified content in blue.

We summarise the main additions below:

  1. Numerical precision and deployment (R1, R3): We resolved the previously reported FP32 limitation and demonstrate that Transformer-FSQ works in BF16. We additionally evaluate compatibility with 4-bit quantised model weights (NF4). Long-context evaluation (R1, R3): We extend our experiments beyond the original 8k context length, including evaluations at 32k and additional 128k results.
  2. Data efficiency (R2): New ablations demonstrate that finetuning data can be reduced by 10× with no statistically detectable degradation at moderate compression ratios, while quantifying the benefits of additional training at extreme compression.
  3. Additional baselines and ablations: We include SnapKV, Ada-KV and AQUA-KV, further baseline hyperparameter tuning, comparisons with alternative quantisation methods, and reviewer-suggested hybrid architectures.
  4. Kernel and serving efficiency: We have continued optimising our fused Triton kernels, improving decode throughput and adding comprehensive measurements of prefill memory, TTFT, TPOT, and throughput across different batch sizes. At 32× compression, decoding is faster than uncompressed attention on 0.6B and 1.7B models, and comparable on 8B.
  5. Additional experimental analysis: We evaluate alternative optimisers, reasoning performance (MATH-500), and further architectural ablations, including negative results and limitations.

Broader significance. We believe this work demonstrates the potential of extreme KV-cache compression, an increasingly important direction as language models move towards million-token contexts and private, on-device deployment. Our results show that learning compact codebook free discrete latent representations offers a promising alternative to conventional KV-cache compression, particularly in regimes where existing methods suffer substantial accuracy degradation.

Importantly, our current approach uses a simple FSQ bottleneck with uniform quantisation across layers. We have not yet explored more sophisticated strategies, including adaptive layer-wise quantisation budgets (PyramidKV; KVTuner) or alternative lattice-based quantisation geometries (Zhao et al., 2025). These offer promising directions for further improving the accuracy–compression trade-off.

We hope the new experiments and clarifications address the reviewers’ concerns and provide stronger evidence for the practical relevance and future potential of Transformer-FSQ

AuthorsOpenReview ↗

Update: Transformer-FSQ at moderate compression

We thank all three reviewers for their questions on moderate compression. Reviewers Wfe3 and nH4H asked whether Transformer-FSQ could be improved in this regime ("at a moderate compression ratio of 4×, FSQ appears to underperform the baselines"; "the practically useful 5–10 compression regime"), and Reviewer wRXh asked whether its advantage persists where the baselines have not "already collapsed". We are excited to share a result that grew directly out of these questions.

We have added a one-page supplement to the submission titled kv-fitted-init.pdf. Its Figure 1 reproduces Figure 2 of our original submission between 2.6× and 17× compression, with the new runs overlaid for direct comparison with every baseline, and its Table 1 reports each setting with a paired 95% interval.

Initialising Transformer-FSQ's latent from the key and value projections, with a clamp in place of tanh, places it above every baseline curve at 8× compression on Qwen3-1.7B and Qwen3-8B, and level with or above our original submission at 4× on Qwen3-0.6B and Qwen3-1.7B.

On Qwen3-8B at 8× compression, RULER accuracy rises from 66% in our original submission to 84%, averaged over two training seeds.

ModelCompressionOriginal submissionClampClamp + key/value initialisationChange from original submission (95% interval)Uncompressed
Qwen3-0.6B4×56.6%66.3%66.6%+10.0 [7.7, 12.4]71.8%
Qwen3-0.6B8×55.7%65.0%66.5%+10.8 [8.4, 13.2]71.8%
Qwen3-1.7B4×65.9%54.3%66.9%+1.1 [−0.7, 2.8]77.2%
Qwen3-1.7B8×51.8%52.2%64.5%+12.6 [10.3, 15.0]77.2%
Qwen3-8B4×not reported79.8%84.3%not applicable92.2%
Qwen3-8B8×, seed 4266.4%67.5%83.3%+17.0 [14.0, 19.9]92.2%
Qwen3-8B8×, seed 4366.4%66.6%85.1%+18.7 [15.5, 21.9]92.2%

RULER at 8,192 tokens over 650 items. Values from our original submission average three seeds (one for Qwen3-8B), as in its Figure 2; the new runs use one seed, except Qwen3-8B at 8×. Changes are paired over the same items. On Qwen3-0.6B the gain comes from the clamp and on the larger models from the initialisation. Decoding used the general attention path rather than the fused kernel.

This is a substantial gain at the compression ratios of greatest practical interest. At 8×, Transformer-FSQ now improves on the corresponding results in our original submission by 13 to 19 points on Qwen3-1.7B and Qwen3-8B, lies above every baseline curve (no baseline was run at exactly 8×), and on Qwen3-8B comes within about 8 points of the uncompressed model.

Notably, the new Qwen3-8B results required only a tenth of the finetuning data used in our original submission. Reviewer Wfe3 observed that "training the compression modules requires substantial data", citing "2 billion tokens for Qwen3-8B". The new runs were finetuned on roughly 200 million tokens, yet still score far above the corresponding results in our original submission, which further alleviates the concern about training cost. We shall refine the supplement and incorporate it into the camera-ready version, where it will update the paper's figures.

We believe these results substantially address the two remaining concerns about Transformer-FSQ, its accuracy at moderate compression and the cost of training it, and that the paper is considerably stronger for your questions. We would be most grateful if you would take them into account when revisiting your assessments, and we would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.