acceptodds
Under review as a conference paper at ICLR 2027

BitLadder: Bit-Level Selective Loading of the KV Cache for Efficient Long-Context LLM Inference

Abstract

Long-context large language models (LLMs) repeatedly load a growing key-value (KV) cache during autoregressive decoding, making attention memory-bandwidth bound. Existing methods reduce this traffic through query-aware sparse attention or KV-cache quantization, but neither aligns loading with current token importance: sparse attention uses a coarse keep-or-skip mask that cannot differentiate retained contributions and may discard the aggregate contribution of long-tail tokens, whereas quantization uses static bit-width assignments that cannot track query-dependent importance. Our analysis shows that token importance is graded and changes across decoding steps. We present BitLadder, an algorithm-system co-designed framework for bit-level selective loading that dynamically adapts loaded KV-cache bit widths to token importance at each decoding step. To support dynamic bit widths without storing multiple quantized copies or performing per-step requantization, we propose truncation-consistent nested quantization (TCNQuant), which encodes the KV-cache once and derives lower-bit representations by truncating the same code. To translate fewer loaded bits into practical speedup, we further design PlaneWeave, a memory layout that stores this nested representation so that a GPU kernel loads only the bits each token needs and supports efficient mixed-precision execution. We evaluate BitLadder on long-context understanding (LongBenchV2, MRCR) and reasoning (GPQA, MATH500, AIME25) benchmarks. At matched accuracy, BitLadder achieves 3.0× and 2.3× speedups over full attention and the state-of-the-art baseline, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KVCache Quantization, Sparse Attention, LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_33SgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BitLadder, a framework that dynamically selects the precision at which each KV-cache block is loaded during decoding, based on an estimate of its importance to the current query. It introduces TCNQuant, which stores each KV entry once in a nested code so lower-bit representations can be obtained by truncating the stored code, and PlaneWeave, a memory layout designed to load only the selected bit planes efficiently. The method estimates block importance from low-bit summaries, assigns blocks to a small set of precision levels, and performs mixed-bit attention. Experiments on two 8B-class models and long-context understanding and reasoning benchmarks report improved accuracy–traffic trade-offs over the evaluated sparse-attention and quantization baselines, with attention speedups of up to 3.0× over full attention.

Critical strengths
  1. The paper combines a query-dependent precision policy with a nested quantizer and a GPU-oriented memory layout. This is a clear algorithm–system contribution: the method addresses not only which precision to use, but also how to avoid loading unused bit planes.
  2. The evaluation includes accuracy, KV-payload traffic, attention latency, and time-per-output-token measurements. It also compares against both sparse-attention and quantization approaches, making the reported trade-off more informative than a traffic-only analysis.
Critical weaknesses
  1. The quality of the allocation depends on low-bit block summaries as proxies for current attention importance. Averaging query-head estimates and assigning a shared bit width to a block may miss head-specific or within-block important tokens. The paper would benefit from a more direct analysis of allocation errors and their effect on the final attention output.
  2. The evaluation covers two 8B-class models. This is limited evidence for a method whose allocation behavior may depend on architecture, attention-head grouping, and long-context training. In addition, the paper should make the distinction between attention-kernel speedups and end-to-end decoding speedups especially clear, since the latter are smaller.
Questions for authors
  1. Can the authors compare the summary-based block allocation with an oracle allocation based on exact attention scores, and report how often the summary selects too few bits for blocks that materially affect the output?
  2. How sensitive are accuracy and latency to the block size, summary precision, and the allocation threshold? Do the conclusions hold when query heads sharing a KV head have substantially different attention patterns?
  3. Can the authors evaluate BitLadder on additional model families or larger models, and report end-to-end throughput at longer contexts alongside the attention-only speedup?
Reviewer_vMs9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BitLadder, which dynamically selects KV-cache precision based on the current query to accelerate long-context inference. Its storage format allows a single KV-cache representation to support multiple precisions at loading time without additional computation. Experimental results show substantial speedups over both full attention and the state-of-the-art baseline at comparable accuracy.

Critical strengths

The method is clearly presented. It effectively combines sparse attention and quantization to address their respective limitations and provides a practical solution.

Critical weaknesses

The experimental evaluation has substantial gaps. The baselines are limited to Quest, INT2, and OSCAR, leaving out other relevant sparse-attention and quantization methods, such as KIVI and Twilight. Moreover, INT2 is less commonly used in practical deployments than INT4, MXFP4, or NVFP4 formats, which should also be considered in the comparison. The efficiency evaluation does not report whether the introduced quantization and summary construction affect prefill performance.

Questions for authors

Could the authors provide additional experiments to address the issues raised in the Weakness section?

Reviewer_dwyTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main idea for this paper is that utilize dynamic bits for different importance of KV-cache. The proposed method provide a balance between pure evince methods (0 bit) and quantization methods ( 2, 4, 8 bits). A training-free truncation-consistent nested quantizer TCNQuant and a memory layout management PlaneWeave are provided. Experiments on accuracy and latency show that the proposed method can speed up LLM inference while with acceptable accuracy decrease.

Critical strengths
  1. The writing and organization are clear and easily to understand the research question and solutions.
  2. Long context inference is an important research topic in the LLM serving.
Critical weaknesses
  • Novelty concern: there are highly similar research [1], and library [2]

  • Experiments:

    • Consider adding NIAH (Needle In A Haystack) results.
    • There are lots of researches about quantization, however the comparison is about INT2, Quest, and OSCAR, which is only a tiny subset. Please consider add more experiments.
    • Since there are extra computation of importance computation and quantization, the time to first token (TTFT) should be provided.
  • [1] Zhang, Junkai, et al. "RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache." arXiv preprint arXiv:2605.08317 (2026).

  • [2] https://github.com/venkateshaddanki287-eng/adaptivekv

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.