acceptodds
Under review as a conference paper at ICLR 2027

ORG-KV: Finding Optimal Rank and Grouping for Low Rank KV-Cache Compression

Abstract

Low-rank KV-cache compression reduces LLM memory usage by storing compact key and value representations, but introduces a fundamental trade-off between accuracy and efficiency: Higher ranks better preserve accuracy but increase both memory and decoding costs and larger decomposition groups improve accuracy at the cost of decoding computation. Existing approaches typically fix the decomposition granularity in advance and subsequently allocate or search for ranks, potentially excluding more favorable granularity and rank configurations. %from costly training or empirical search. We propose ORG-KV (Optimal Rank and Group KV), a low-rank KV-cache compression framework that jointly finds optimal rank for each key-value projection and decomposition granularity, under explicit KV-cache memory and decoding computation resource constraints. %, without requiring additional training. Specifically, we formulate finding joint rank allocation and grouping granularity as a constrained integer optimization that minimizes estimated compression error under KV-cache memory and decoding FLOPs constraints. Experiments across multiple LLM models demonstrate that ORG-KV consistently outperforms existing training-free low-rank KV-cache compression methods under the same compression budgets.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Keywords: Large Language Models, KV Cache Compression, Low-Rank Decomposition, Resource-Constrained Optimization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wQDaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ORG-KV, which jointly selects the group size and total rank for the low-rank decomposition of each key and value projection across layers, subject to KV cache storage and decoding FLOPs constraints. The method evaluates candidates one projection at a time by estimating the KL divergence between the original and compressed output distributions. It reduces the comparison cost using top-64 probabilities plus an aggregated tail, then formulates and solves a multiple-choice multidimensional knapsack problem with an additive error surrogate as the objective. The authors further apply recovery training to the low-rank factors and mixed-precision quantization to the latent cache. Experiments across multiple models show improvements over the evaluated training-free baselines, and the authors report throughput gains in GPU-memory-constrained settings. 42630_ORG_KV_Finding_Optimal_R

Critical strengths
  1. The problem formulation is well motivated, and ablations support the value of joint selection. At the same total rank, group size affects both approximation quality and reconstruction cost, while key and value projections have different computational costs. It is therefore natural to account for both factors in a unified allocation framework. On LongChat at 75% compression, rank optimization with a fixed group size achieves 47.88% accuracy, compared with 56.10% for joint optimization. This supports the value of joint selection beyond simply adding recovery training.
  2. The main results and approximation analyses provide reasonably solid empirical support. At 60% compression, the training-free variant achieves higher average accuracy than PaLU and ReCalKV on all three models, with particularly large gains on LLaMA-3-8B-Instruct. The authors also provide validation of error rankings across joint configurations and of the top-k approximation. They distinguish attention latency at a fixed batch size from full-model throughput at the maximum feasible batch size, and clearly disclose limitations related to metadata accounting and numerical precision settings.
Critical weaknesses
  1. The practical benefits and offline costs of dual-constrained resource optimization are not sufficiently evaluated. The appendix sets relatively high FLOPs ceilings based on PaLU, but does not sufficiently report the actual FLOPs of the selected configurations, budget utilization, accuracy–latency trade-offs under different compute budgets, or the time required for candidate evaluation and integer optimization. The fixed-batch attention speedups and maximum-batch throughput results provide useful evidence, but they do not establish what the FLOPs constraint adds beyond a memory-only constraint, or how well the FLOPs cost model predicts deployment latency.
  2. The attribution and stability analyses for joint configuration quality focus mainly on one model and a small amount of calibration data. Candidate KL estimates use only four 256-token samples. The ranking validation of the additive objective is mainly based on random configurations for LongChat at 60% compression. It does not yet establish whether the objective can reliably distinguish near-optimal configurations or generalize to the GQA setting where the largest gains are observed. The paper also uses K-FAC-weighted decomposition and different recovery strategies. I would therefore like to see cross-model selection ablations under the same decomposition and recovery settings, along with clarification of the exact training, calibration, and evaluation splits. The evidence for long-context accuracy also covers a narrower range of sequence lengths than the latency measurements, with RULER evaluated only at 4K.
Questions for authors
  1. At a fixed KV cache storage budget, how does removing or progressively tightening the FLOPs constraint affect the selected configurations, accuracy, and latency at the same batch size? Please report the actual FLOPs of the selected configurations, budget utilization, the total time required to construct the complete candidate tables, solver runtime, and the MIP gap at termination. These results would directly affect my assessment of the practical value of the dual-constraint design.
  2. Do the benefits of joint selection and the high correlation between estimated and actual error rankings still hold for GQA models such as LLaMA-3-8B-Instruct and at 75% compression? Please compare rank optimization with fixed granularity against joint optimization while keeping the decomposition and recovery pipelines identical, and vary both the number of calibration samples and how they are sampled for candidate KL estimation. Could you examine near-optimal configurations rather than reporting only an overall correlation coefficient across random configurations?
  3. Please specify the candidate configurations for all models and the WikiText-2 splits used for calibration and evaluation, and clarify whether the recovery training and validation sets are strictly deduplicated at the document or token level. Please also explain the discrepancy between 59.15 in Table 1 and 59.20 in Table 4. What causes the non-finite values in FP16 at 16K, and which settings and results are affected? Under the actual settings used for the throughput measurements, are long-context quality, stability, and cache size including metadata consistent with the paper’s claims?
3 comments

We thank Reviewer wQDa for the detailed and constructive feedback. We conducted additional experiments to address the concerns and provide our responses below.


Q1: Role of the FLOPs Constraint and Budget Sensitivity

On LLaMA-2-7B, we removed or progressively tightened the FLOPs constraint while fixing KV-cache compression at 75% and keeping the calibration setting unchanged from the paper. We normalized the PaLU-based FLOPs limit to 1× and evaluated limits of 1×, 0.8×, and 0.6×, denoted as ORG-KV(1×), ORG-KV(0.8×), and ORG-KV(0.6×).

Table 1. FLOPs Budget Sensitivity (75% Compression)

SettingFLOPsFLOPs budgetMemory budgetWiki2C46-task Avg.LatencyK groupsV groupsSolverMIP gap
(G)↓(%)(%)↓↓(%) ↑(ms) ↓(layers)(layers)time (s)(%)
w/o FLOPs const.783.827—1005.5998.56261.73673.6232 (32)32 (32)0.3030.03
ORG-KV(1×)64.23599.691006.0349.73059.1180.001 (18), 2 (11), 4 (3)32 (32)3.0370.11
ORG-KV(0.8×)51.54099.991006.1029.90359.0874.411 (26), 2 (6)16 (1), 32 (31)0.2330.04
ORG-KV(0.6×)38.65799.991006.24510.23258.1771.701 (32)1 (1), 2 (2), 8 (1), 16 (5), 32 (23)3.4720.99

Without the FLOPs constraint, accuracy was higher, but actual FLOPs were approximately 12.20× those of ORG-KV(1×), and latency was up to 4.84× that of the original model and 9.09× that of ORG-KV(1×). Tightening the FLOPs constraint reduced latency across all context lengths and led K projections to select smaller groups, while V projections largely retained larger groups. Constructing the complete candidate table took approximately 21 minutes, and the same table was reused across all settings. Detailed latency results under the same measurement settings as in the paper are provided below.

Table 2. Decoding Latency (ms) Across Context Lengths (75% Compression)

Context lengthOriginalw/o FLOPs const.ORG-KV(1×)ORG-KV(0.8×)ORG-KV(0.6×)
4K42.22136.2224.0922.7421.59
8K78.18298.3342.940.0738.12
16K150.48673.628074.4171.7
32K295.881,411.47155.36144.21141.29
64K587.842,842.24313.45293.6287.98

Q2: Error Ranking Near the Selected Solution and Final Model Performance

We conducted additional analysis on LLaMA-3-8B-Instruct, a GQA model, at 60% and 75% KV-cache compression. To examine ranking reliability near the selected solution, we selected configurations within ±5% of its estimated or actual KL. “Estimated KL ±5%” and “Actual KL ±5%” indicate the criteria used to select each subset. Each entry reports the Spearman correlation between estimated and actual KL rankings within that subset, with the number of configurations in parentheses.

Table 3. Spearman Correlation Between Estimated and Actual KL

CompressionEstimated KL ±5%Actual KL ±5%
60%0.721 (304)0.682 (280)
75%0.822 (405)0.737 (328)

We further examined whether ranking discrepancies affected final performance by comparing the selected solution with the evaluated configuration having the lowest actual KL at 60% compression, where correlations were lower. The differences were small: 0.054 in WikiText-2 PPL, 0.240 in C4 PPL, and 0.27%p in 6-task average accuracy.

Table 4. Performance Comparison with Best Actual KL (60% Compression)

ConfigurationWikiText-2 PPL ↓C4 PPL ↓6-task Avg. (%) ↑
ORG-KV8.55416.91663.29
Best Actual KL8.60817.15663.02

Q3: Rank-Only vs. Joint Optimization

We first compared training-free Rank-only and ORG-KV under the same decomposition pipeline and resource budgets. Rank-only fixes the K/V group size to 4 and optimizes rank, while ORG-KV jointly selects rank and grouping.

On LLaMA-3-8B-Instruct at 60% KV-cache compression, we used one contiguous 1,024-token segment from each of 16 distinct WikiText-2 training documents instead of random sampling. Without recovery training, ORG-KV achieved lower PPL and a 3.94%p higher 6-task average accuracy than Rank-only optimization.

We then applied the same KD recovery pipeline to both configurations. ORG-KV retained lower PPL and a 1.00%p higher average accuracy after recovery.

Table 5. Rank-Only vs. Joint Optimization (60% Compression)

MethodTraining-FreeWiki2 ↓C4 ↓6-task Avg. (%) ↑FLOPs (G)
Rank-only✓9.62120.52959.7923.845
ORG-KV✓8.47116.54463.7324.143
Rank-only + KD7.86113.74965.0823.845
ORG-KV + KD7.58213.48966.0824.143

Q4: Candidate Configurations

For each model, group-size candidates include all positive divisors of the number of KV heads, from head-wise to fully joint decomposition. For each group size, total rank-retention candidates are {0.1, 0.2, ..., 1.0} relative to the original KV projection output dimension.

Table 6. Group-Size Candidates by Model

ModelKV headsGroup sizes
LongChat-7B-v1.5321, 2, 4, 8, 16, 32
LLaMA-2-7B321, 2, 4, 8, 16, 32
LLaMA-3-8B-Instruct81, 2, 4, 8
LLaMA-3.1-8B-Instruct81, 2, 4, 8

Q5: Long-Context Generation Stability

On LLaMA-2-7B with 75% KV-cache compression and KD, we found that repetitive 16K-token inputs caused FP16 overflow in the MLP. With non-repetitive 16K-token inputs, both FP16 KV and mixed-bit KV completed 4,096 decoding steps. We will revise the manuscript to clarify these findings.


Q6: Data Splits, Deduplication, and Evaluation Discrepancy

We used the WikiText-2 training split for calibration and the test split for PPL evaluation. For recovery training, we sampled 10M tokens from the FinePDFs-Edu eng_Latn training split, and for validation, we sampled 20K tokens from the same split using a different seed. We filtered by document ID to ensure that the two sets contained no shared documents.

The scores of 59.15 in Table 1 and 59.20 in Table 4 of the original manuscript were obtained using the same compressed configuration, with evaluation batch sizes of 8 and 1, respectively.


We hope these additional results and clarifications address the reviewer's concerns. We appreciate your time and consideration.

Thank you for the response. The clarification addresses my concerns on this point, and I will raise my score accordingly.

Reviewer_PdpbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ORG-KV, a low-rank KV-cache compression framework that jointly optimizes decomposition group size and rank for each key/value projection. It estimates projection-wise compression errors, then solves a multiple-choice multidimensional knapsack problem under KV-cache memory and decoding-FLOP constraints. The method uses K-FAC-weighted SVD to construct low-rank factors and optionally applies recovery training and mixed-precision quantization. Experiments on several LLaMA-, LongChat-, and Mistral-based models report improvements over selected existing KV-cache compression methods.

Critical strengths
  1. The paper formulates joint group-size and rank selection under both memory and decoding-computation constraints.
  2. The experiments include perplexity, zero-shot accuracy, long-context benchmarks, and inference throughput.
Critical weaknesses

1. Limited methodological novelty and insufficient formal specification: The proposed low-rank decomposition appears to directly reuse the K-FAC-weighted SVD procedure from prior work. The paper does not introduce a new SVD objective, decomposition algorithm, or theoretical formulation. The main contribution seems to be an incremental rank/group allocation framework built on top of an existing SVD method, rather than a substantially new low-rank KV compression technique.

2. Insufficient Evaluation Against Relevant SVD Methods: The experimental comparison is insufficient to establish the benefits of ORG-KV. First, ORG-KV relies on K-FAC-weighted SVD to construct the low-rank factors, but the paper does not include a K-FAC-weighted SVD baseline. Such a comparison is necessary to disentangle the contribution of the proposed granularity/rank allocation from that of the underlying decomposition method. Second, the paper omits recent SVD-based compression methods such as SVD-LLM2, Swift-SVD, and Dobi-SVD, which also perform dynamic rank allocation based on reconstruction error or parameter importance. Although these methods are generally formulated for model compression, they appear relevant when applied specifically to the W_K and W_V projections. The authors should either adapt these methods as KV-cache compression baselines under matched cache-size and decoding-FLOP budgets, or provide a convincing explanation of why such an adaptation is invalid.

3. Limited Evaluation on Recent Model Architectures: The evaluation is limited to models released in 2023 and 2024: LongChat-7B-v1.5 (2023), LLaMA-2-7B (2023), LLaMA-3-8B (2024), LLaMA-3.1-8B (2024), and Mistral-7B (2023). The paper does not evaluate any models released in 2025 or 2026. This is a significant limitation because real-world deployments typically adopt newer model releases, whose attention structures, KV-cache layouts, and overall architectures may differ substantially from those of the models evaluated in this paper. Therefore, the authors should evaluate ORG-KV on more recent open-source models, such as Qwen3.5-4B (https://huggingface.co/Qwen/Qwen3.5-4B), Olmo-Hybrid-7B (https://huggingface.co/allenai/Olmo-Hybrid-7B/tree/main), and Granite-4.0-H-Tiny (https://huggingface.co/ibm-granite/granite-4.0-h-tiny). These models are publicly available, were released at least six months before the ICLR 2027 submission deadline, and have parameter scales comparable to those already evaluated in the paper. The authors should examine whether ORG-KV remains effective across these newer architectures, including models with different attention or hybrid-layer designs. If such experiments are not provided, the authors should discuss in detail why the gains observed on 2023–2024 models are expected to transfer to newer models and why the method remains applicable to their KV-cache structures. Without either experiments on recent models or a convincing transferability analysis, the practical generality of ORG-KV is not sufficiently established.

4. The additive error approximation is weakly validated: The central assumption that jointly compressing all projections produces approximately additive projection-wise KL error is validated only on 200 random allocations for LongChat-7B-v1.5 at one compression rate. The paper does not show whether this approximation remains reliable across models, budgets, compression rates, or calibration datasets.

5. The candidate search space is highly discretized: Group sizes and ranks are restricted to a manually chosen grid. The paper does not study sensitivity to the candidate set or show that the selected grid is sufficient to find good configurations.

Questions for authors

See Weaknesses 1 to 5.

6 comments

We appreciate Reviewer Pdpb's thoughtful comments, particularly regarding the methodological contribution and comparison with existing approaches. We provide additional experimental evidence and clarifications to address these concerns.


Q1: Core Contribution and Distinction from Existing Decomposition Methods

As the reviewer notes, ORG-KV does not introduce a new low-rank decomposition method. Our core contribution is the MMKP formulation for jointly allocating rank and grouping to minimize estimated error under KV-cache memory and decoding FLOPs budgets. To disentangle the contributions of the decomposition method and rank/grouping allocation, we conducted additional experiments on LLaMA-2-7B at 60% KV-cache compression using the same calibration setting as in the paper. Both Uniform and Rank-only use a fixed group size of 4.

Table 1. Decomposition and Allocation Ablation (60% Compression)

DecompositionAllocationWiki2 ↓C4 ↓6-task Avg. (%) ↑FLOPs (G) ↓
Non-K-FACUniform8.07615.11149.24227.412
K-FACUniform6.75212.01754.52227.412
K-FACRank-only5.8999.04359.8595.94
K-FACORG-KV5.4328.02163.1596.492

Under Uniform allocation, K-FAC improves 6-task average accuracy by 5.28%p over Non-K-FAC. With K-FAC fixed, ORG-KV gains 8.63%p over Uniform while reducing decoding FLOPs by 57.6%, and 3.30%p over Rank-only at similar FLOPs.


Q2: Low-Rank Weight Decomposition vs. Low-Rank KV-Cache Compression

Low-rank weight decomposition and low-rank KV-cache compression involve different design choices and optimization goals. SVD-LLM V2[1] and Swift-SVD[2] focus on weight compression, whereas ORG-KV jointly selects the rank and grouping of each key/value projection for KV-cache compression. Even with the same latent cache size, decoding cost can differ depending on the grouping and decomposition structure. ORG-KV considers these differences when selecting rank and grouping under KV-cache memory and decoding FLOPs constraints.

Table 2. Comparison with Low-Rank Weight Decomposition Methods (60% Compression)

MethodWiki2 ↓C4 ↓6-task Avg. (%) ↑FLOPs (G) ↓
SVD-LLM V2*6.0588.70160.031,779.06
Swift-SVD*6.0728.7759.81,779.06
ORG-KV5.4328.02163.1596.492

We applied SVD-LLM V2[1] and Swift-SVD[2] to KV-cache compression on LLaMA-2-7B. ORG-KV improved 6-task average accuracy by 3.12 and 3.35%p, respectively, while reducing decoding FLOPs by approximately 18.44×. (*) Only K/V projections were compressed.

[1] SVD-LLM V2: https://aclanthology.org/2025.naacl-long.217/

[2] Swift-SVD: https://openreview.net/forum?id=nAQ4h8FpdM


Q3: Evaluation on a Recent Hybrid Architecture

We conducted additional experiments on Qwen3.5-4B, which has 8 full-attention blocks and 24 linear-attention blocks. We compressed the KV cache of the full-attention blocks by 75%, using group-size candidates {1, 2, 4} for ORG-KV, which cover all positive divisors of the number of KV heads.

Table 3. KV-Cache Compression on a Hybrid Architecture (75% Compression)

MethodWiki2 ↓C4 ↓6-task Avg. (%) ↑FLOPs (G) ↓
Original8.89213.69966.171.124
PaLU (group 2)11.61217.22662.038.365
ORG-KV9.83115.15563.127.989

ORG-KV does not directly apply to linear-attention blocks, but it can compress the KV cache of full-attention blocks in hybrid models. Compared with PaLU, ORG-KV achieved lower PPL on both datasets and improved 6-task average accuracy by 1.09%p.

Q4: Reliability of the Additive KL Approximation

To evaluate ranking reliability near the selected solution rather than across random allocations, we selected allocations whose estimated KL or actual KL was within ±5% of the corresponding value of the selected solution. “Estimated KL ±5%” and “Actual KL ±5%” indicate the selection criteria for each subset. Each entry reports the Spearman correlation between estimated and actual KL rankings, with the number of allocations in parentheses.

Table 4. Spearman Correlation Between Estimated and Actual KL

ModelComp. (%)Estimated KL ±5% ↑Actual KL ±5% ↑
LLaMA-2-7B600.688 (210)0.587 (338)
LLaMA-2-7B750.572 (302)0.664 (371)
LLaMA-3-8B-Instruct600.721 (304)0.682 (280)
LLaMA-3-8B-Instruct750.822 (405)0.737 (328)

On LLaMA-2-7B at 75% compression, we compared the selected solution with the evaluated configuration having the lowest actual KL. PPL was nearly identical, and 6-task average accuracy differed by only 0.02%p.

Table 5. Comparison with the Lowest Actual KL (75% Compression)

ConfigurationWiki2 ↓C4 ↓6-task Avg. (%) ↑
ORG-KV6.0349.7359.11
Lowest actual KL6.0359.7359.13

We also evaluated correlations across calibration settings by varying one factor at a time: the number of samples {4, 8, 16, 32, 64}, sequence length {256, 512, 1,024, 2,048, 4,096}, or sampling seed (six seeds). Correlations varied slightly across calibration settings, but final performance remained stable. See RgrJ-Q2 for detailed performance results.


Q5: Candidate Search Space and Rank Grid Sensitivity

The group-size candidates already include all positive divisors of the number of KV heads, from head-wise to fully joint decomposition. We set the default rank-retention step to 0.1, as a finer grid showed little change in performance while increasing evaluation cost. These results suggest that performance is relatively insensitive to further refinement of the rank grid.

Table 6. Rank Grid Sensitivity

ModelCompression (%)Rank stepWiki2 ↓C4 ↓6-task Avg. (%) ↑
LLaMA-2-7B60%0.15.4328.02163.15
LLaMA-2-7B60%0.055.4238.00763.04
LLaMA-2-7B75%0.16.0349.7359.11
LLaMA-2-7B75%0.056.0249.7258.95
LLaMA-3-8B-Instruct60%0.18.55416.91663.29
LLaMA-3-8B-Instruct60%0.058.55116.69763.48

We hope these additional results and clarifications address the reviewer's concerns. We appreciate your time and consideration.

Reviewer_PdpbOpenReview ↗

Thank you for your response

Thanks for your response. since some of my concerns are partially solved, I will maintain my rating of 2. First, I still have concerns about novelty, ORG-KV does not introduce a new low rank compression method and primarily focuses on joint rank allocaiton. As this is likely an incremental extension of K-FAC-weighted SVD, I remain concerned if ORG-KV meets the acceptance bar for ICLR. Second, the experiments on Recent Hybrid Architecture lack efficiency comparison/analysis. Low rank compression fundamentally involves an accuracy-efficiency tradeoff, without analysing efficiency metrics such as Time To First Token, Time Per Output Token and Token throughput, it's not clear if ORG-KV can befinite the recent hybrid archeitectures auch as Qwen 3.5 and OLMo. Besides, only one baseline, PaLU is included in experiment on hybrid architecture, making it difficult to analyze how ORG-KV performs against other baselines.

Reviewer_RgrJOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ORG-KV to jointly optimize rank allocation and attention head grouping under memory and decoding FLOPs constraints. Unlike existing methods that fix the grouping size in advance, ORG-KV evaluates different rank and grouping configurations using KL divergence between the original and compressed model outputs. It formulates the allocation as a multiple-choice multidimensional knapsack problem (MMKP) to select configurations that minimize estimated compression error.

Critical strengths
  1. The paper jointly optimizes rank and grouping under memory and FLOPs constraints. The improvement from 47.88% to 56.10% at 75% compression is substantial and shows why grouping matters.

  2. ORG-KV performs better than PaLU and ReCalKV without recovery training across the three evaluated models. After KD, its results are close to STAR-KV. This suggests that the proposed allocation is useful even without additional training.

Critical weaknesses
  1. LEMS (ICML 2026) already uses calibrated error estimates and integer programming for rank allocation. ORG-KV adds joint grouping and rank selection, but it is unclear how much the KL-based MMKP itself contributes. A simple greedy baseline using the same search space would help.

  2. The KL estimates use only four 256-token calibration samples. The reported correlation looks good, but it is unclear whether the selected configurations are stable with different samples, random seeds, or longer contexts.

Questions for authors
  1. Could the authors compare MMKP with a simple greedy method using the same rank/grouping options and resource limits? How much difference does MMKP actually make?

  2. How sensitive are the results to the calibration data? Since only four 256-token samples are used, would different samples, random seeds, or longer contexts lead to different rank and grouping choices?

4 comments

We thank Reviewer RgrJ for the constructive feedback and valuable suggestions. We have conducted additional experiments to address the questions regarding the optimization method and calibration sensitivity, as detailed below.


Q1: Comparison of HiGHS and Simple Greedy

The core contribution of ORG-KV is the MMKP formulation for joint rank and grouping allocation. The formulation can be solved using different methods. We adopted HiGHS to jointly optimize discrete choices under both resource constraints, considering the computational cost of dynamic programming.

On LongChat-7B-v1.5 at a target KV-cache compression rate of 60%, we compared HiGHS with Simple Greedy using the same candidate set and resource constraints. In this setting, Simple Greedy, which evaluates one configuration at a time, exhausted the FLOPs budget while using only 32.35% of the cache budget, resulting in substantially worse PPL and accuracy.

Table 1. HiGHS vs. Simple Greedy (60% Compression)

MethodComp. (%)Cache budget usage (%)FLOPs budget usage (%)Wiki2 ↓C4 ↓6-task Avg. (%) ↑
HiGHS60.00100.0099.917.21811.14759.15
Simple Greedy87.0632.35100.00117.775173.63035.54

Q2: Sensitivity to Calibration Data

We evaluated calibration sensitivity and cost on LongChat-7B-v1.5 at 60% KV-cache compression. Using four 256-token samples from the WikiText-2 training split (seed 42) as the default, we varied one factor at a time: the number of samples {4, 8, 16, 32, 64}, sequence length {256, 512, 1,024, 2,048, 4,096}, or sampling seed (six seeds). Both the sample-count and sequence-length sweeps covered 1,024–16,384 calibration tokens.

Table 2. Calibration Sensitivity (60% Compression)

SettingWiki2 PPL ↓C4 PPL ↓6-task Avg. (%) ↑
Original results7.21811.14759.15
Number of samples7.202 ± 0.01611.098 ± 0.04859.38 ± 0.20
Sequence length7.204 ± 0.00911.128 ± 0.02559.24 ± 0.08
Sampling seed7.200 ± 0.02211.082 ± 0.06359.17 ± 0.16

Performance was stable across calibration settings, with average standard deviations of 0.016 for WikiText-2 PPL, 0.045 for C4 PPL, and 0.15%p for 6-task accuracy across the sample-count, sequence-length, and sampling-seed sweeps.

Candidate table construction took 21 minutes with 4 samples × 256 tokens, compared with 247 minutes for 64 samples and 177 minutes for 4,096-token sequences. We therefore retained 4 samples × 256 tokens as the default.


We hope these additional results and clarifications address the reviewer's concerns. We appreciate your time and consideration.

Thank you for the response for calibration and greedy experiments. However, I agree with Reviewer Pdpb’s novelty concern, my own question about LEMS also indicated the point. The difference to LEMS is still not answered. Therefore, I adjust my rating to 2. If authors show the comparison with LEMS in the same search space and the same budget, I will reconsider.

We thank the reviewer for the follow-up comments and the opportunity to clarify the distinction between ORG-KV and LEMS and further evaluate the benefits of joint optimization.

First, we would like to note that LEMS was originally proposed for weight compression, although its rank allocation framework can also be applied to KV-cache compression. To the best of our knowledge, existing KV-cache compression methods rely on fixed or heuristic grouping strategies, without explicitly optimizing decomposition granularity. The key novelty of ORG-KV is to go beyond rank allocation or grouping optimization alone by jointly optimizing both under KV-cache memory and decoding FLOPs constraints.

To directly address the reviewer's concern and evaluate the effectiveness of joint optimization, we adapt the LEMS rank allocation framework by replacing its weight-memory constraint with KV-cache memory and decoding FLOPs constraints, forming a sequential Rank → Group baseline. Both methods use the same search space and resource budgets, along with the same calibration data, while retaining their respective compression error estimates. We evaluate both methods at 75% KV-cache compression on LLaMA-2-7B and 60% on LLaMA-3-8B-Instruct, using the same FLOPs budget as PaLU for each model. The sequential baseline applies the FLOPs constraint during group allocation.

Table 3. KV-Cache Compression on LLaMA-2-7B and LLaMA-3-8B-Instruct

ModelCompressionMethodWiki2 ↓C4 ↓Acc. (%) ↑GFLOPs ↓
LLaMA-2-7B75%LEMS (Rank → Group)6.3410.2158.7364.38
LLaMA-2-7B75%ORG-KV (Joint)6.049.7359.1364.13
LLaMA-3-8B-Instruct60%LEMS (Rank → Group)9.3620.4961.8224.16
LLaMA-3-8B-Instruct60%ORG-KV (Joint)8.5516.9263.2924.16

ORG-KV achieves lower perplexity and higher accuracy than the sequential LEMS baseline under matched resource budgets on both models, supporting the effectiveness of our joint rank and grouping allocation strategy.


We hope these additional results and clarifications address the reviewer's concerns. We appreciate your time and consideration.

Thank you for your detailed response and the additional experiments. I appreciate your effort in addressing my concerns. I will discuss the remaining points with the other reviewers during the next discussion phase.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated Manuscript

We thank the reviewers for their valuable feedback. We have uploaded a revised manuscript, with changes highlighted in blue. To clarify the benefits of joint rank and grouping allocation, we added decomposition and allocation ablations (Table 4). We also added FLOPs budget sensitivity experiments (Table 6) and updated the decoding throughput results at a 16K input length (Table 7). Additionally, we corrected Table 1, where the results for LLaMA-3-8B-Instruct and Mistral had been mixed up, to clearly distinguish the results for each model.

We thank the reviewers for their constructive feedback. During the rebuttal period, we conducted additional experiments to address concerns regarding methodological novelty, calibration robustness, and inference efficiency. As the main remaining concern appears to be methodological novelty, we summarize the experiments evaluating the contribution of joint rank and grouping optimization.

1. Methodological Novelty

Existing KV-cache compression methods, including PaLU [1], ReCalKV [2], and STAR-KV [3], rely on fixed or heuristic grouping strategies without explicitly optimizing decomposition granularity. In contrast, ORG-KV jointly optimizes grouping granularity and rank under KV-cache memory and decoding FLOPs constraints.

To distinguish this contribution from existing decomposition and allocation methods, we conducted three additional experiments.

Separating Decomposition and Allocation Contributions (Pdpb-Q1, Table 1). With the same K-FAC-weighted SVD decomposition, ORG-KV achieved 3.30 percentage points higher accuracy than rank-only optimization at comparable decoding FLOPs. This result demonstrates the contribution of joint allocation independently of the underlying decomposition method.

Comparison with Extended LEMS (RgrJ Follow-up, Table 3). We extended the LEMS rank allocation framework to KV-cache compression by constructing a Rank → Group baseline. Under the same candidate search space and resource budgets, ORG-KV achieved 0.40 and 1.47 percentage points higher accuracy than the extended LEMS-based baseline on LLaMA-2-7B and LLaMA-3-8B-Instruct, respectively. While the original LEMS addresses rank allocation for weight compression, ORG-KV jointly optimizes rank and grouping granularity under KV-cache memory and decoding FLOPs constraints.

Rank-Only vs. Joint Optimization (wQDa-Q3, Table 5). Under identical decomposition and recovery settings, ORG-KV achieved 3.94 percentage points higher accuracy without recovery training and 1.00 percentage point higher accuracy after KD than rank-only optimization with fixed grouping. These results demonstrate the effectiveness of joint optimization under controlled experimental conditions.

2. Additional Validation of Other Reviewer Concerns

Calibration Robustness (RgrJ-Q2, Table 2). Varying the number of calibration samples, sequence length, and random seed resulted in an average standard deviation of approximately 0.15 percentage points in accuracy, indicating stable final performance across calibration settings.

Recent Hybrid Architecture (Pdpb-Q3, Table 3). At 75% KV-cache compression on the full-attention layers of Qwen3.5-4B, ORG-KV achieved 1.09 percentage points higher accuracy than PaLU, demonstrating its applicability to a recent hybrid architecture.

Effect of the FLOPs Constraint (wQDa-Q1, Tables 1–2). Removing the FLOPs constraint resulted in 12.20× higher decoding FLOPs and 8.42× higher attention latency at a 16K context length compared with ORG-KV (1×). These results demonstrate the importance of considering decoding computation alongside KV-cache memory.

3. Remaining Concerns and Overall Assessment

We understand that the main remaining concern is whether joint rank and grouping optimization constitutes sufficient methodological novelty.

ORG-KV does not introduce a new SVD algorithm. Instead, it explicitly treats grouping granularity, which has not been directly optimized in existing KV-cache compression methods, as an optimization variable alongside rank, and formulates their joint allocation under both KV-cache memory and decoding FLOPs constraints.

The additional experiments distinguish the contribution of ORG-KV from K-FAC decomposition, an extended LEMS-based rank allocation framework, and fixed-group rank-only optimization. We also provided additional evidence regarding calibration robustness, applicability to hybrid architectures, and the practical benefits of FLOPs constraints.

References

[1] PaLU, ICLR 2025.

[2] ReCalKV, 2025.

[3] STAR-KV, ICML 2026.

[4] LEMS, ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.