- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Loce for lossless compression of LLM weights and KV caches. It subtracts a base, stores residuals using fixed-width remainders and bitmap-marked exceptions, and uses the first token of each page as the KV reference. GPU decoding optimizations support inference on compressed data. Experiments report memory savings and inference speedups across several models and numerical precisions.
Critical strengths
The paper addresses a practical problem, as model weights and KV caches consume substantial memory during LLM inference. It is clearly written and easy to follow, with a straightforward explanation of the compression scheme and system design. The figures are well designed and help illustrate the encoding format, decoding process, and inference pipeline.
Critical weaknesses
-
The contribution is incremental. The basic idea is to subtract a base, pack small residuals into fixed-width bits, and use bitmaps to handle exceptions, which is similar to FOR/PFOR compression. Fixed-width exponent encoding and fused GPU decompression are also already used in ZipServ. Loce changes the coding format and extends these ideas to KV caches using the first token of each page as a reference. I see this as a practical adaptation of existing techniques, but not a sufficiently substantial methodological advance to warrant acceptance.
-
The released code does not support the core KV performance claims. In the provided code, the end-to-end path compresses KV tensors only to report storage statistics, discards the compressed payloads, and continues generation with the original uncompressed cache. It therefore does not implement the mechanism needed to demonstrate the claimed KV memory savings and decoding speedups. This is a fundamental gap between the provided code and the paper’s main results, not a minor implementation detail. I cannot consider the core performance claims adequately supported by this implementation.
Questions for authors
- What substantial technical advance does Loce provide beyond adapting existing compression and GPU execution techniques to KV caches?
- Which implementation produced the reported compressed-KV memory and TPOT results, and how does it use compressed KV during decoding?
- What evidence shows that the small exponent residuals come from temporal locality? Could stable per-channel exponent distributions explain the same observation, and how does the first-token reference compare with a simple per-channel statistical reference?
- How much of the reported speedup comes from compression itself? Does the advantage remain when the uncompressed baseline uses comparable kernel optimizations?
- Does Loce reduce peak GPU memory and increase the maximum input context length? Since KV compression happens after prefill, how does it help requests that run out of memory during prefill?
4 comments
Response to Reviewer D6Ty
Thank you for your prompt response and for taking the time to run our code. We reran Qwen3-32B under your specified settings, with a batch size of 4 and the two-stage kernel layout optimization and compression optimizations enabled. The logs are available in reproduce_logs/same_kv, reproduce_logs/Qwen3-32B-rulerqa-64k, and reproduce_logs/Qwen3-32B-weight.
The discrepancies may arise from mismatched code versions and differences in specific experimental settings. We also used AI agents, such as Codex, to reproduce the results in a clean environment. In these runs, we observed that Codex did not enable all Loce optimizations by default, which may also contribute to the discrepancies. To facilitate reproduction, we have provided the execution script script/retest_qwen3_32b_weight_kv_64k.sh.
Follow-up on the reproduction
Thank you for the response and for sharing the updated code and scripts. First, I would like to note that my earlier results were not caused by missing optimizations: your new logs report the same 1.26× (KV) and 1.30× (weights) for the paper's bitmap format that I reported before. I still have your previous release, so I ran both releases with your vLLM benchmark, using the batch size from your reply and the input/output lengths from your request-latency logs (Qwen3-32B, batch size 4, tensor parallel 4, 32K-token inputs, 128 output tokens). I would appreciate clarification on a few points.
| TTFT | TPOT | Request latency | KV ratio | Weight ratio | |
|---|---|---|---|---|---|
| Paper | – | – | – | 1.33× | 1.38× |
| BF16 (no compression) | 37.7 s | 38.6 ms | 42.6 s | – | – |
| Previous release (bitmap) | 38.1 s | 95.9 ms | 50.2 s | 1.26× | 1.30× |
| New release (sparse) | 38.7 s | 1953.5 ms | 286.8 s | 1.34× | 1.38× |
1. Why do the two releases differ so much in both compression ratio and speed? With the previous release, which stores positions as the per-element bitmap described in the paper, I get 1.26× (KV) and 1.30× (weights). The paper's numbers appear only in the new release, which replaces the bitmap with a sparse position layout for weights and a new compact format for KV. However, decoding is about 20× slower per token than in the previous release.
2. Which version produced the results in the paper? As far as I can tell, the new release adds no kernel for the sparse layout. The sparse positions are expanded back into a bitmap on every forward call, and the new KV format is decompressed in full before attention. If the latency results come from the bitmap version and the compression ratios come from the sparse version, they would not describe the same method. Could you clarify which format the paper's compression and latency results are based on? If they are based on the sparse layout, why does the paper describe a per-block bitmap with popcount-based lookup instead? Your new log also labels the 1.38× result as "not the paper fixed-bitmap format".
3. In both releases, the static reference still gives a higher KV ratio than the first-token page reference, so I am still unsure about the benefit of the temporal design.
I will keep my score until these points are fully addressed.
Response to Reviewer D6Ty
Thank you for your timely response and clarification. We are especially grateful that you have taken the time to examine our code and rerun the experiments, while continuing to provide such quick and detailed feedback.
About benefit of temporal design. we agree with the reviewer that a static reference can achieve a higher KV compression ratio than a first-token page reference in some cases. This depends partly on the input data distribution and partly on the additional storage overhead of keeping the first-token reference exponent tensor in each KV page uncompressed. As discussed in the paper, the first token reference tensors can also be further compressed. With this compression, the temporal design can often achieve a higher KV compression ratio than the static reference. The logs are shown in expermental/temporal_locality_compression_qwen3_32b_longbench_64k.log
| Format | Compression ratio |
|---|---|
| Static reference, 3-bit fixed-width residuals | 1.304811× |
| Temporal with first token page compressed | 1.342581× |
Another important advantage of the temporal design is lower encoding overhead. Loce directly uses the first token’s exponents as the reference, avoiding additional statistics collection and reference selection. We compared ZipServ’s static-reference encoder with Loce’s temporal-reference encoder on the same KV cache. Their encoding times were 243 s and 0.36 s, respectively. The script is in expermental/encode_compare.sh and the log is in expermental/encode_compare_qwen3_32b_longbench_64k.log
| Encoding method | Total encoding time |
|---|---|
| ZipServ, static codebook | 243.4005 |
| vLLM Loce, first-token page reference | 0.3615 |
About sparse layout. In fact, Loce’s decoding kernels just remain bitmap-based during inference. Sparse layout only changes the weight storage format, which we consider an implementation detail rather than a fundamental change to the method. As suggested by the reviewer, we will further clarify this in the method section of the paper. Indeed, the new release does not introduce a dedicated kernel for the sparse layout. As you suggested, this remains an opportunity for further optimization, and we fully agree with your recommendation.
About release code version and speedup. We make multiple commits to the code on 16 Nov. Because synchronization between the anonymous and private repositories is not instantaneous, the version you initially downloaded may differ from the final version used in our experiments. All compression-ratio and latency experiments reported in the paper used the same code version available in the latest release, and the compression-ratio results have also been largely verified. We also tested request latency under your specified configuration. The execution log is available at expermental/loce_warp_block_qwen3_32b_longbench_32k.log. At present, we are unsure whether these discrepancies arise from differences in hardware platforms or specific execution procedures. Although the remaining rebuttal period may be insufficient to resolve all your discrepancies, we would be happy to continue verifying the experimental details with you after the decisions are released, and we take full responsibility for all reported experimental results.
| Path | TTFT (s) | TPOT (ms/token) | Full request (s) |
|---|---|---|---|
| BF16 eager | 3.958 | 37.683 | 8.744 |
| Loce + warp-block GEMV, run 1 | 4.223 | 29.529 | 7.972 |
| Loce + warp-block GEMV, run 2 | 4.197 | 28.464 | 7.812 |
Thank you again for reviewing our work so carefully. We sincerely appreciate the time and effort you have devoted to helping us identify and address these issues.
Concerns on reproducibility and authenticity of the results
After examining the released code and running it myself, I have serious concerns about the reproducibility and authenticity of the results in this submission. The reported compression ratios cannot be reproduced with the authors' own code, the core temporal-locality component performs worse than a simple static baseline, and the files provided as experiment logs were not produced by running the code. Unless the authors can provide the code and raw outputs that actually generated the reported results, I will maintain my current score.