acceptodds
Under review as a conference paper at ICLR 2027

Deep Low Rank Projector for Low Rank KV Cache Compression

Abstract

Key-value (KV) cache memory grows linearly with context length and becomes a major bottleneck in long-context language model inference. We propose Deep Low-Rank Projector (DLRP), a framework for compressing the KV cache along the head dimension of pretrained attention models. DLRP first trains full-dimensional Deep Linear Projectors (DLPs) on a frozen backbone using a KL objective and a regularizer that encourages low-rank structure. We establish an upper-bound relation between this regularizer and the nuclear norm of the composed projection matrix, providing a tractable surrogate that avoids direct nuclear-norm computation during training. Layer-wise reduced dimensions are then selected from the learned singular-value spectra, and the trained projectors are reduced and fine-tuned for recovery using only a small amount of data. Finally, the resulting DLRPs are folded into the attention projections, leaving no additional projector modules in the decoding path. Across Qwen3 and Llama-3 models, DLRP preserves downstream performance better than the compared hidden-dimension KV-cache compression methods over a range of compression ratios, remains compatible with token-axis compression, and reduces long-context memory usage while increasing feasible batch size.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deep Linear Network, Low Rank KV Cache Compression, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pj2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Deep Low-Rank Projector (DLRP), a two-stage approach for KV-cache compression. It first trains deep linear projectors on a frozen LLM using KL divergence and a low-rank regularizer, then selects layer-wise compression ranks based on singular-value spectra. The reduced projectors are fine-tuned and folded into the attention weights for inference. Experiments on Qwen3 and Llama-3 models evaluate downstream accuracy, long-context performance, and serving efficiency.

Critical strengths

DLRP achieves better accuracy than the evaluated low-rank compression baselines at matched compression ratios, with relatively small performance degradation. The two-stage training and rank-allocation ablations provide useful evidence for the design choices. Unlike many KV-cache compression papers, the authors also implement a custom Triton kernel and report actual GPU memory usage and serving throughput. The ability to fold projectors into attention weights and combine DLRP with token-axis compression is practically useful.

Critical weaknesses
  1. Unfair baseline comparisons. DLRP uses substantial additional training (100K samples in Stage 1 and 20K in Stage 2), while the compared methods are not given equivalent fine-tuning. Without training-matched comparisons, it is difficult to determine whether the improvements come from the proposed method or additional optimization.

  2. Insufficient evidence for the proposed regularizer. The theoretical result provides an upper bound on the nuclear norm but does not establish that the regularizer improves compression performance. The paper lacks an α = 0 ablation at matched compression ratios, making its actual contribution unclear.

  3. Inconsistent experimental comparisons. Table 5 appears to mix compression ratios for Qwen3-8B. Specifically, the full DLRP score of 66.30 corresponds to 30.1% compression, while several ablation results correspond to 32.4%. This inconsistency undermines the validity of the ablation conclusions.

  4. Limited compression and serving efficiency evaluation. The evaluated compression ratios are relatively modest (approximately 23%–36%), leaving performance under more aggressive compression unexplored. Furthermore, decoding throughput decreases from 14.4 to 11.0 tokens/s at 64K context length, despite reduced KV-cache memory usage.

  5. Limited methodological novelty. The framework primarily combines deep linear parameterization, low-rank regularization, SVD-based rank selection, and recovery fine-tuning. These components are individually well established, and the paper does not sufficiently demonstrate which aspects provide improvements beyond existing low-rank compression techniques.

Questions for authors
  1. Implementation details: Are the projectors shared across attention heads or trained independently? How are Qwen3's Q/K normalization and RoPE handled after dimensionality reduction?

  2. Long-context evaluation: Why does RULER Single-NIAH accuracy increase with context length for all compressed methods in Table 3? Could the authors evaluate more challenging RULER tasks?

  3. Evaluation protocol: Could the authors clarify the dataset splits, evaluation metrics, and number of runs used for the short-context benchmarks? Some reported accuracy values appear inconsistent with standard test-set sizes.

  4. Memory measurements: Could the authors report GPU memory consumption at identical batch sizes? Table 7 currently compares peak memory under different maximum batch sizes, making the reported memory savings difficult to interpret.

Reviewer_Bnp8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DLRP for compressing the head dimension of the KV cache. It first trains full-dimensional, factorized linear projectors with a distillation loss and a low-rank regularizer. It then selects layer-wise dimensions from the learned spectra, reduces the projectors, and performs recovery fine-tuning. The projectors are folded into the attention weights for inference. Experiments cover Qwen3 and Llama models, short- and long-context tasks, and combinations with token-axis compression.

Critical strengths
  1. The method reduces KV-cache dimensions while allowing the trained projectors to be folded into the attention weights. This avoids separate projector modules during decoding and makes the approach useful for memory-constrained inference.

  2. The evaluation covers several model families and both short- and long-context tasks. The ablations on recovery training and layer-wise rank allocation help explain the design choices. The experiments also show that DLRP can be combined with several token-axis compression methods.

Critical weaknesses
  1. The connection between rank selection and the actual reduction procedure is unclear. Equation 9 selects the dimension from the spectrum of the full composed projector, but Equations 13 and 16 reduce only a boundary factor while keeping the other factors unchanged. The important directions of the full product need not match those of an individual factor, so the selected energy threshold does not directly establish what the resulting reduction preserves. Appendix E.4 compares several reduction strategies, but does not resolve this mismatch. This leaves an important part of the proposed compression procedure insufficiently justified.
  2. The training budget for the direct reduced-projector baseline is not clearly specified. Full DLRP uses 100K samples in Stage 1 and another 20K samples for recovery, but it is unclear whether the direct baseline in Tables 5 and 16 receives a comparable total training budget. Without this information, the reported gain does not clearly separate the benefit of the two-stage procedure from the benefit of additional training. A comparison under matched training budgets would help establish whether the extra stage is necessary.
Questions for authors
  1. Why does reducing a boundary factor preserve the important directions identified from the full composed projector? Can you measure the information or approximation error retained by this reduction and compare it with a reduction based directly on the full projector’s singular subspace?
  2. What data volume and number of training steps are used for the direct reduced-projector baseline? Does the advantage of the two-stage method remain when both approaches receive the same total training budget?
1 comment
Reviewer_Bnp8OpenReview ↗

Thanks for detailed rebuttal

The rebuttal has adequately addressed my main concerns. I maintain my positive assessment of the paper and my Weak Accept recommendation.

Reviewer_zhH7OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DLRP, a two-stage framework for compressing KV-caches along the hidden dimension. First, full-dimensional deep linear projectors are trained with a KL-divergence objective and a low-rank regularizer while keeping the pretrained backbone frozen. Layer-wise compression ranks are subsequently determined from the learned singular-value spectra. The reduced projectors are then fine-tuned to recover model accuracy and folded into the attention projection matrices to eliminate additional inference-time projection operations. Experiments across Qwen3 and Llama-3 models demonstrate improved accuracy preservation compared with several existing low-rank KV-cache compression methods.

Critical strengths
  • DLRP outperforms the evaluated low-rank baselines at matched KV-cache compression ratios. For example, on Qwen3-8B at 32.4% compression, DLRP achieves 66.09 average accuracy compared with 59.63 for Palu and 61.07 for Eigen-Attention.

  • The ablations demonstrate benefits from both learned rank allocation and the two-stage training procedure; the recovery fine-tuning is effective.

Critical weaknesses
  • Novelty is insignificant. The core ingredients [trainable low-rank projections, factorized linear transformations, singular-value-based rank selection, and recovery fine-tuning] are individually established already. In particular, MatryoshkaKV already employs trainable projections with adaptive rank selection, while STAR-KV introduces learnable rank control. The distinction introduced by deep linear projectors is not compelling, and not compelling at all after I noticed (from the ablation study) that two-factor projectors perform better than deeper alternatives, raising questions about the necessity and significance of the proposed deep parameterization.

  • Comparison with existing methods is not fair. DLRP uses 100K training samples for projector learning and another 20K for recovery fine-tuning, whereas methods such as Palu are fundamentally post-training decomposition approaches. Although matching compression ratios is appropriate for evaluating final accuracy, the experiments do not isolate how much improvement comes from the proposed projector architecture itself vs. the other optimization effort.

  • Comparisons with stronger recent methods (e.g., STAR-KV) and more aggressive compression settings are missing. The evaluation covers approximately 23-36% KV-cache compression. This is relatively conservative given that existing approaches, including STAR-KV, report substantially more aggressive compression. The omission of STAR-KV is especially concerning because STAR-KV directly addresses adaptive rank selection, a central component of DLRP. It remains unclear whether the reported accuracy advantage would persist at 50-75% compression.

Questions for authors

See "Critical Weaknesses" for my concerns and questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.