acceptodds
Under review as a conference paper at ICLR 2027

Loce: Locality-Aware Lossless Compression for Efficient LLM Inference

Abstract

Lossless compression offers a promising direction for alleviating the memory and bandwidth bottlenecks in LLMs due to inherent floating-point exponent redundancy. We observe that this redundancy primarily arises from extreme value locality and temporal locality, yet existing methods fail to unify these two forms of locality, leaving significant compression potential untapped. To bridge this gap, we propose Loce, a locality-aware lossless compression framework that leverages an algorithm-system co-design. At the algorithmic level, we present a fixed-length Rice-based lossless coding scheme that transforms both value and temporal locality into compact residual representations for lightweight encoding and decoding. At the system level, we introduce a GPU-efficient design with optimized memory layout, decoding kernel, and inference pipeline to accelerate compressed LLM inference. Extensive experiments on various LLMs and floating-point precisions show that, beyond achieving competitive compression ratios, Loce delivers an average 2.3× kernel-level acceleration and 1.5× end-to-end speedup over previous lossless compression methods. Our code is available on Anonymous GitHub.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Inference, Lossless Compression, Rice Coding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_D6TyOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Loce for lossless compression of LLM weights and KV caches. It subtracts a base, stores residuals using fixed-width remainders and bitmap-marked exceptions, and uses the first token of each page as the KV reference. GPU decoding optimizations support inference on compressed data. Experiments report memory savings and inference speedups across several models and numerical precisions.

Critical strengths

The paper addresses a practical problem, as model weights and KV caches consume substantial memory during LLM inference. It is clearly written and easy to follow, with a straightforward explanation of the compression scheme and system design. The figures are well designed and help illustrate the encoding format, decoding process, and inference pipeline.

Critical weaknesses
  1. The contribution is incremental. The basic idea is to subtract a base, pack small residuals into fixed-width bits, and use bitmaps to handle exceptions, which is similar to FOR/PFOR compression. Fixed-width exponent encoding and fused GPU decompression are also already used in ZipServ. Loce changes the coding format and extends these ideas to KV caches using the first token of each page as a reference. I see this as a practical adaptation of existing techniques, but not a sufficiently substantial methodological advance to warrant acceptance.

  2. The released code does not support the core KV performance claims. In the provided code, the end-to-end path compresses KV tensors only to report storage statistics, discards the compressed payloads, and continues generation with the original uncompressed cache. It therefore does not implement the mechanism needed to demonstrate the claimed KV memory savings and decoding speedups. This is a fundamental gap between the provided code and the paper’s main results, not a minor implementation detail. I cannot consider the core performance claims adequately supported by this implementation.

Questions for authors
  1. What substantial technical advance does Loce provide beyond adapting existing compression and GPU execution techniques to KV caches?
  2. Which implementation produced the reported compressed-KV memory and TPOT results, and how does it use compressed KV during decoding?
  3. What evidence shows that the small exponent residuals come from temporal locality? Could stable per-channel exponent distributions explain the same observation, and how does the first-token reference compare with a simple per-channel statistical reference?
  4. How much of the reported speedup comes from compression itself? Does the advantage remain when the uncompressed baseline uses comparable kernel optimizations?
  5. Does Loce reduce peak GPU memory and increase the maximum input context length? Since KV compression happens after prefill, how does it help requests that run out of memory during prefill?
4 comments
Reviewer_D6TyOpenReview ↗

Concerns on reproducibility and authenticity of the results

After examining the released code and running it myself, I have serious concerns about the reproducibility and authenticity of the results in this submission. The reported compression ratios cannot be reproduced with the authors' own code, the core temporal-locality component performs worse than a simple static baseline, and the files provided as experiment logs were not produced by running the code. Unless the authors can provide the code and raw outputs that actually generated the reported results, I will maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer D6Ty

Thank you for your prompt response and for taking the time to run our code. We reran Qwen3-32B under your specified settings, with a batch size of 4 and the two-stage kernel layout optimization and compression optimizations enabled. The logs are available in reproduce_logs/same_kv, reproduce_logs/Qwen3-32B-rulerqa-64k, and reproduce_logs/Qwen3-32B-weight.

The discrepancies may arise from mismatched code versions and differences in specific experimental settings. We also used AI agents, such as Codex, to reproduce the results in a clean environment. In these runs, we observed that Codex did not enable all Loce optimizations by default, which may also contribute to the discrepancies. To facilitate reproduction, we have provided the execution script script/retest_qwen3_32b_weight_kv_64k.sh.

Reviewer_D6TyOpenReview ↗

Follow-up on the reproduction

Thank you for the response and for sharing the updated code and scripts. First, I would like to note that my earlier results were not caused by missing optimizations: your new logs report the same 1.26× (KV) and 1.30× (weights) for the paper's bitmap format that I reported before. I still have your previous release, so I ran both releases with your vLLM benchmark, using the batch size from your reply and the input/output lengths from your request-latency logs (Qwen3-32B, batch size 4, tensor parallel 4, 32K-token inputs, 128 output tokens). I would appreciate clarification on a few points.

TTFTTPOTRequest latencyKV ratioWeight ratio
Paper–––1.33×1.38×
BF16 (no compression)37.7 s38.6 ms42.6 s––
Previous release (bitmap)38.1 s95.9 ms50.2 s1.26×1.30×
New release (sparse)38.7 s1953.5 ms286.8 s1.34×1.38×

1. Why do the two releases differ so much in both compression ratio and speed? With the previous release, which stores positions as the per-element bitmap described in the paper, I get 1.26× (KV) and 1.30× (weights). The paper's numbers appear only in the new release, which replaces the bitmap with a sparse position layout for weights and a new compact format for KV. However, decoding is about 20× slower per token than in the previous release.

2. Which version produced the results in the paper? As far as I can tell, the new release adds no kernel for the sparse layout. The sparse positions are expanded back into a bitmap on every forward call, and the new KV format is decompressed in full before attention. If the latency results come from the bitmap version and the compression ratios come from the sparse version, they would not describe the same method. Could you clarify which format the paper's compression and latency results are based on? If they are based on the sparse layout, why does the paper describe a per-block bitmap with popcount-based lookup instead? Your new log also labels the 1.38× result as "not the paper fixed-bitmap format".

3. In both releases, the static reference still gives a higher KV ratio than the first-token page reference, so I am still unsure about the benefit of the temporal design.

I will keep my score until these points are fully addressed.

AuthorsOpenReview ↗

Response to Reviewer D6Ty

Thank you for your timely response and clarification. We are especially grateful that you have taken the time to examine our code and rerun the experiments, while continuing to provide such quick and detailed feedback.

About benefit of temporal design. we agree with the reviewer that a static reference can achieve a higher KV compression ratio than a first-token page reference in some cases. This depends partly on the input data distribution and partly on the additional storage overhead of keeping the first-token reference exponent tensor in each KV page uncompressed. As discussed in the paper, the first token reference tensors can also be further compressed. With this compression, the temporal design can often achieve a higher KV compression ratio than the static reference. The logs are shown in expermental/temporal_locality_compression_qwen3_32b_longbench_64k.log

FormatCompression ratio
Static reference, 3-bit fixed-width residuals1.304811×
Temporal with first token page compressed1.342581×

Another important advantage of the temporal design is lower encoding overhead. Loce directly uses the first token’s exponents as the reference, avoiding additional statistics collection and reference selection. We compared ZipServ’s static-reference encoder with Loce’s temporal-reference encoder on the same KV cache. Their encoding times were 243 s and 0.36 s, respectively. The script is in expermental/encode_compare.sh and the log is in expermental/encode_compare_qwen3_32b_longbench_64k.log

Encoding methodTotal encoding time
ZipServ, static codebook243.4005
vLLM Loce, first-token page reference0.3615

About sparse layout. In fact, Loce’s decoding kernels just remain bitmap-based during inference. Sparse layout only changes the weight storage format, which we consider an implementation detail rather than a fundamental change to the method. As suggested by the reviewer, we will further clarify this in the method section of the paper. Indeed, the new release does not introduce a dedicated kernel for the sparse layout. As you suggested, this remains an opportunity for further optimization, and we fully agree with your recommendation.

About release code version and speedup. We make multiple commits to the code on 16 Nov. Because synchronization between the anonymous and private repositories is not instantaneous, the version you initially downloaded may differ from the final version used in our experiments. All compression-ratio and latency experiments reported in the paper used the same code version available in the latest release, and the compression-ratio results have also been largely verified. We also tested request latency under your specified configuration. The execution log is available at expermental/loce_warp_block_qwen3_32b_longbench_32k.log. At present, we are unsure whether these discrepancies arise from differences in hardware platforms or specific execution procedures. Although the remaining rebuttal period may be insufficient to resolve all your discrepancies, we would be happy to continue verifying the experimental details with you after the decisions are released, and we take full responsibility for all reported experimental results.

PathTTFT (s)TPOT (ms/token)Full request (s)
BF16 eager3.95837.6838.744
Loce + warp-block GEMV, run 14.22329.5297.972
Loce + warp-block GEMV, run 24.19728.4647.812

Thank you again for reviewing our work so carefully. We sincerely appreciate the time and effort you have devoted to helping us identify and address these issues.

Reviewer_YqjNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Loce is a lossless compression framework for LLM weights and KV caches. It uses the most frequent exponent in each layer as the weight reference and the first token's exponent vector as the reference for each KV page. The resulting residuals are stored using fixed-width remainders, sparse quotients, and bitmaps. The implementation combines this format with GPU decoding and memory-access optimizations. Weights are compressed offline; prefill KV is compressed once, while newly generated KV remains uncompressed. Experiments cover BF16, FP8, and NVFP4 and report memory savings, kernel speedups, and improvements in token generation latency.

Critical strengths
  1. I particularly appreciate the attempt to compress FP8 and NVFP4 further without changing their values. There is less room for compression at these precisions, and metadata and decoding costs can easily outweigh the savings. Table 4 reports weight compression ratios of 1.27–1.28× for FP8 and 1.24–1.26× for NVFP4, with additional savings on KV caches. This is a useful direction, and the reported inference gains make it worth pursuing.

  2. Extending compression to KV caches addresses an important cost in long-context decoding. Using a page-local temporal reference is a concrete addition to weight-oriented approaches such as ZipServ. Compressing the prefill KV once is also a sensible way to avoid repeated encoding on the decode path. The weight-only and weight-plus-KV comparisons provide encouraging evidence that this extension matters in the tested workloads.

Critical weaknesses
  1. The attention implementation and the source of its speedup are not sufficiently explained. Section 3.3 describes the codec, but does not show how compressed prefill KV and uncompressed decode KV are consumed by attention. Figure 4 combines QKV projection and attention, so it does not reveal the attention latency itself. Figures 5 and 7 provide useful comparisons, but leave unclear how much of the gain comes from reduced KV traffic versus changes to the attention kernel or its scheduling. This matters because KV compression is a central part of the contribution.

  2. The evaluation needs a clearer account of the gains over existing approaches. Exponent locality, bitmap decoding, register reconstruction, and overlapping memory access with computation are already present in ZipServ. Loce's case therefore rests more on temporal prediction, its different exception representation, and the low-precision extension. These changes need more direct comparisons. In particular, the ZipServ storage figures need a breakdown, and a same-tensor comparison with SplitZip would help assess the KV codec. The low-precision compute paths and correctness checks also need more detail. An additional H100/H200 or newer datacenter GPU result would strengthen the evidence that the gains extend beyond the current platforms.

Questions for authors
  1. Mixed-format KV attention. How are compressed prefill KV and uncompressed decode KV addressed and consumed during attention? Does one kernel process both regions, or are partial attention results computed separately and merged? Please describe where decompression occurs, how page boundaries are handled, and what index data and temporary buffers are required. A short execution diagram or pseudocode would help.

  2. Performance attribution. Please report QKV projection and attention latency separately for representative cases in Figure 4. Do Loce-W and Loce-W+KV use the same attention backend and scheduling configuration? A matched implementation comparison, or profiling of memory traffic and decoding overhead, would help distinguish compression benefits from other kernel improvements. Please also define the normalization in Figure 5 and explain how the headline average speedups are aggregated.

  3. Baseline accounting and the value of temporal prediction. Could you break down ZipServ's storage cost in Table 4? For example, if Table 5's 96.7% seven-exponent coverage applies to the same Qwen3 weight population, TCA-TBE's payload formula gives about 11.26 bits per weight before indices and padding, whereas Table 4 reports 12.40 bits. Different tensor coverage or accounting may explain this; please specify these choices. Separately, how much does the first-token reference improve compression over a static exponent reference with the same codec? A comparison with SplitZip on the same KV tensors, reporting compression ratio and encode/decode cost, would also be useful. Their different serving targets need not prevent this codec-level comparison.

  4. Low-precision execution and correctness. What quantization and scale configurations are used for FP8 and NVFP4, and what precisions and kernels are used after decompression? Are these matched to the uncompressed baselines? Table 4 states that NVFP4 scaling overhead is included; please also clarify the treatment of other metadata and uncompressed components. Could you provide bitwise round-trip checks for the low-precision tensors and full-request latency results that include prefill and KV encoding? Table 8 currently reports BF16 accuracy results.

  5. Zero-quotient blocks. Tables 5–7 report high fractions of blocks in which all 256 quotients are zero. How are these blocks mapped to tensor dimensions, and is any reordering performed? Please report the element-level nonzero-quotient rate and its density within active blocks, alongside the block-level statistics. This would help explain both the storage cost and the effectiveness of the bypass path. For BF16 with k=3, zero quotients correspond to residuals in [-4, 3], so the reported |delta E| <= 7 coverage does not directly establish this rate.

  6. When does prefill KV compression pay off? Appendix D.1 shows that prefill KV dominates the evaluated workloads. Could you report representative prompt/output length combinations, including the one-time encoding time, TPOT, and total request latency? Short outputs may not amortize encoding, while long outputs increase the uncompressed KV fraction. This would clarify where the current strategy is most useful.

As an additional experiment, I would welcome results on an H100/H200 or a newer datacenter GPU, using an appropriate native baseline for each supported precision. One representative additional platform would already help assess how the balance between memory savings and decompression cost changes with hardware.

Reviewer_SeeZOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Loce proposes lossless compression for LLM weights and prefill-generated KV caches by exploiting locality in floating-point exponents. It subtracts a layer-wise modal exponent for weights and a first-token, per-feature exponent reference within each KV page, maps signed residuals to nonnegative integers, and stores fixed-width remainders with sparse quotient payloads. The codec is combined with an aligned memory layout, warp-cooperative decoding, and overlapped data loading. Experiments cover BF16, FP8, and NVFP4 representations, reporting substantial memory savings and inference speedups across several models and long-context workloads.

Critical strengths

(1) A clear, practically motivated connection between data characteristics and compression design. Sections 1 and 3 connect exponent concentration to a compact residual representation, and Appendix A provides both a Gaussian-model analysis for weights and measurements across layers, precisions, and datasets for KV caches. The motivation goes beyond reducing checkpoint size: it addresses the memory traffic that can limit inference. The distinction between weight value locality and within-page KV exponent similarity makes the design easy to understand.

(2) A coherent algorithm–system design that addresses both weights and KV caches. The fixed-width remainder path, sparse quotient handling, and zero-quotient bypass provide a plausible way to reduce storage while keeping decoding simple. Sections 3.2–3.3 connect these choices to GPU memory layout, warp-level execution, and prefetching. Compressing prefill KV once while leaving generated KV uncompressed is a sensible trade-off for long-input workloads, and its limitation is explicitly acknowledged in Appendix E. The residual mapping and reconstruction equations are reversible, and Table 3's quotient widths cover the full exponent-residual ranges of the stated formats.

Critical weaknesses

(1) The evaluation does not clearly isolate the incremental benefit of temporal prediction over value-based compression of the same KV tensors.

Figures 4–5 compare weight-only with weight+KV compression, and Figure 7(a) attributes portions of the speedup to value and temporal locality. These results support the utility of compressing KV caches, but the description does not establish a matched comparison between using the first token’s exponents as the page-level base and exploiting value locality for KV cache compression. Appendix A.2 verifies that neighboring-token exponent differences are often small, but it does not determine how much this improves compression over the already concentrated marginal exponent distribution.

Questions for authors
  1. Can the authors specify the input construction, actual output lengths, stopping rules, and per-model GPU/tensor-parallel settings for Figures 5–6 and 13, and reconcile the generated-KV fraction with the stated LongBench configuration?

  2. How are the percentages in Figure 7(a) computed? Does the value-locality condition compress the same KV tensors with a non-temporal reference, or does it compress weights only?

  3. Can the authors further evaluate the effectiveness of Loce on LLMs employing hybrid attention architectures, such as Qwen3.5, Qwen3.6, and Qwen3.8?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.