- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ORG-KV, which jointly selects the group size and total rank for the low-rank decomposition of each key and value projection across layers, subject to KV cache storage and decoding FLOPs constraints. The method evaluates candidates one projection at a time by estimating the KL divergence between the original and compressed output distributions. It reduces the comparison cost using top-64 probabilities plus an aggregated tail, then formulates and solves a multiple-choice multidimensional knapsack problem with an additive error surrogate as the objective. The authors further apply recovery training to the low-rank factors and mixed-precision quantization to the latent cache. Experiments across multiple models show improvements over the evaluated training-free baselines, and the authors report throughput gains in GPU-memory-constrained settings. 42630_ORG_KV_Finding_Optimal_R
Critical strengths
- The problem formulation is well motivated, and ablations support the value of joint selection. At the same total rank, group size affects both approximation quality and reconstruction cost, while key and value projections have different computational costs. It is therefore natural to account for both factors in a unified allocation framework. On LongChat at 75% compression, rank optimization with a fixed group size achieves 47.88% accuracy, compared with 56.10% for joint optimization. This supports the value of joint selection beyond simply adding recovery training.
- The main results and approximation analyses provide reasonably solid empirical support. At 60% compression, the training-free variant achieves higher average accuracy than PaLU and ReCalKV on all three models, with particularly large gains on LLaMA-3-8B-Instruct. The authors also provide validation of error rankings across joint configurations and of the top-k approximation. They distinguish attention latency at a fixed batch size from full-model throughput at the maximum feasible batch size, and clearly disclose limitations related to metadata accounting and numerical precision settings.
Critical weaknesses
- The practical benefits and offline costs of dual-constrained resource optimization are not sufficiently evaluated. The appendix sets relatively high FLOPs ceilings based on PaLU, but does not sufficiently report the actual FLOPs of the selected configurations, budget utilization, accuracy–latency trade-offs under different compute budgets, or the time required for candidate evaluation and integer optimization. The fixed-batch attention speedups and maximum-batch throughput results provide useful evidence, but they do not establish what the FLOPs constraint adds beyond a memory-only constraint, or how well the FLOPs cost model predicts deployment latency.
- The attribution and stability analyses for joint configuration quality focus mainly on one model and a small amount of calibration data. Candidate KL estimates use only four 256-token samples. The ranking validation of the additive objective is mainly based on random configurations for LongChat at 60% compression. It does not yet establish whether the objective can reliably distinguish near-optimal configurations or generalize to the GQA setting where the largest gains are observed. The paper also uses K-FAC-weighted decomposition and different recovery strategies. I would therefore like to see cross-model selection ablations under the same decomposition and recovery settings, along with clarification of the exact training, calibration, and evaluation splits. The evidence for long-context accuracy also covers a narrower range of sequence lengths than the latency measurements, with RULER evaluated only at 4K.
Questions for authors
- At a fixed KV cache storage budget, how does removing or progressively tightening the FLOPs constraint affect the selected configurations, accuracy, and latency at the same batch size? Please report the actual FLOPs of the selected configurations, budget utilization, the total time required to construct the complete candidate tables, solver runtime, and the MIP gap at termination. These results would directly affect my assessment of the practical value of the dual-constraint design.
- Do the benefits of joint selection and the high correlation between estimated and actual error rankings still hold for GQA models such as LLaMA-3-8B-Instruct and at 75% compression? Please compare rank optimization with fixed granularity against joint optimization while keeping the decomposition and recovery pipelines identical, and vary both the number of calibration samples and how they are sampled for candidate KL estimation. Could you examine near-optimal configurations rather than reporting only an overall correlation coefficient across random configurations?
- Please specify the candidate configurations for all models and the WikiText-2 splits used for calibration and evaluation, and clarify whether the recovery training and validation sets are strictly deduplicated at the document or token level. Please also explain the discrepancy between 59.15 in Table 1 and 59.20 in Table 4. What causes the non-finite values in FP16 at 16K, and which settings and results are affected? Under the actual settings used for the throughput measurements, are long-context quality, stability, and cache size including metadata consistent with the paper’s claims?
3 comments
Q4: Candidate Configurations
For each model, group-size candidates include all positive divisors of the number of KV heads, from head-wise to fully joint decomposition. For each group size, total rank-retention candidates are {0.1, 0.2, ..., 1.0} relative to the original KV projection output dimension.
Table 6. Group-Size Candidates by Model
| Model | KV heads | Group sizes |
|---|---|---|
| LongChat-7B-v1.5 | 32 | 1, 2, 4, 8, 16, 32 |
| LLaMA-2-7B | 32 | 1, 2, 4, 8, 16, 32 |
| LLaMA-3-8B-Instruct | 8 | 1, 2, 4, 8 |
| LLaMA-3.1-8B-Instruct | 8 | 1, 2, 4, 8 |
Q5: Long-Context Generation Stability
On LLaMA-2-7B with 75% KV-cache compression and KD, we found that repetitive 16K-token inputs caused FP16 overflow in the MLP. With non-repetitive 16K-token inputs, both FP16 KV and mixed-bit KV completed 4,096 decoding steps. We will revise the manuscript to clarify these findings.
Q6: Data Splits, Deduplication, and Evaluation Discrepancy
We used the WikiText-2 training split for calibration and the test split for PPL evaluation. For recovery training, we sampled 10M tokens from the FinePDFs-Edu eng_Latn training split, and for validation, we sampled 20K tokens from the same split using a different seed. We filtered by document ID to ensure that the two sets contained no shared documents.
The scores of 59.15 in Table 1 and 59.20 in Table 4 of the original manuscript were obtained using the same compressed configuration, with evaluation batch sizes of 8 and 1, respectively.
We hope these additional results and clarifications address the reviewer's concerns. We appreciate your time and consideration.
Thank you for the response. The clarification addresses my concerns on this point, and I will raise my score accordingly.
We thank Reviewer wQDa for the detailed and constructive feedback. We conducted additional experiments to address the concerns and provide our responses below.
Q1: Role of the FLOPs Constraint and Budget Sensitivity
On LLaMA-2-7B, we removed or progressively tightened the FLOPs constraint while fixing KV-cache compression at 75% and keeping the calibration setting unchanged from the paper. We normalized the PaLU-based FLOPs limit to 1× and evaluated limits of 1×, 0.8×, and 0.6×, denoted as ORG-KV(1×), ORG-KV(0.8×), and ORG-KV(0.6×).
Table 1. FLOPs Budget Sensitivity (75% Compression)
Without the FLOPs constraint, accuracy was higher, but actual FLOPs were approximately 12.20× those of ORG-KV(1×), and latency was up to 4.84× that of the original model and 9.09× that of ORG-KV(1×). Tightening the FLOPs constraint reduced latency across all context lengths and led K projections to select smaller groups, while V projections largely retained larger groups. Constructing the complete candidate table took approximately 21 minutes, and the same table was reused across all settings. Detailed latency results under the same measurement settings as in the paper are provided below.
Table 2. Decoding Latency (ms) Across Context Lengths (75% Compression)
Q2: Error Ranking Near the Selected Solution and Final Model Performance
We conducted additional analysis on LLaMA-3-8B-Instruct, a GQA model, at 60% and 75% KV-cache compression. To examine ranking reliability near the selected solution, we selected configurations within ±5% of its estimated or actual KL. “Estimated KL ±5%” and “Actual KL ±5%” indicate the criteria used to select each subset. Each entry reports the Spearman correlation between estimated and actual KL rankings within that subset, with the number of configurations in parentheses.
Table 3. Spearman Correlation Between Estimated and Actual KL
We further examined whether ranking discrepancies affected final performance by comparing the selected solution with the evaluated configuration having the lowest actual KL at 60% compression, where correlations were lower. The differences were small: 0.054 in WikiText-2 PPL, 0.240 in C4 PPL, and 0.27%p in 6-task average accuracy.
Table 4. Performance Comparison with Best Actual KL (60% Compression)
Q3: Rank-Only vs. Joint Optimization
We first compared training-free Rank-only and ORG-KV under the same decomposition pipeline and resource budgets. Rank-only fixes the K/V group size to 4 and optimizes rank, while ORG-KV jointly selects rank and grouping.
On LLaMA-3-8B-Instruct at 60% KV-cache compression, we used one contiguous 1,024-token segment from each of 16 distinct WikiText-2 training documents instead of random sampling. Without recovery training, ORG-KV achieved lower PPL and a 3.94%p higher 6-task average accuracy than Rank-only optimization.
We then applied the same KD recovery pipeline to both configurations. ORG-KV retained lower PPL and a 1.00%p higher average accuracy after recovery.
Table 5. Rank-Only vs. Joint Optimization (60% Compression)