- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OrthoKV replaces layer-wise over-complete dictionaries with per-head complete dictionaries to enable efficient sparse coding of KV cache. This allows smaller coefficient indices and compact sparse codes while orthogonality enables encoding through a single transform followed by topk selection. A fixed coefficient budget is adaptively allocated across heads by retaining the largest-magnitude coefficients, minimizing reconstruction error under that budget. Same dictionaries support multiple compression ratios without retraining by decoupling sparsity-promoting objective from target coefficient budget.
Critical strengths
1] In the line of dictionary-based extreme KV cache compression works represented in Lexico, OrthoKV achieves high efficiency by removing iterative orthogonal matching pursuit, and better compression by using smaller dictionary size. The authors show that replacing OMP with a more efficient orthogonal transform + topk selection, per-head specialization ,and adaptive allocation achieve comparable model accuracy to Lexico.
Critical weaknesses
1] Severe accuracy degradation limits the practical appeal of OrthoKV. KIVI 2bit with similar compression ratio as OrthoKV k=18, shows much higher accuracy retention. While the authors target memory budgets below KIVI-2 configurations, from the rate at which OrthoKV k=6 (7% KV cache size) and k=12 (12% KV cache size) degrades model accuracy, a potentially more appealing alternative would be to use a smaller model, which wouldn't need a custom system backend as well.
2] Latency benefits are limited at batch size 1. While faster than Lexico, it is slower than Full KV and around the same latency as KIVI-2 under similar compression ratio (k=18) despite substantially lower model quality. While throughput improves as batch size increases, the absence of a batched comparison with KIVI-2 makes the advantage less clear. Evaluation of OrthoKV on larger models would have further strengthened the large-batch efficiency benefits, but it is left as a limitation, along with validation on more recent, non-GQA attention architectures.
Questions for authors
1] Can the authors provide a latency breakdown separating KV encoding/compression, cache updates, and attention computation for OrthoKV and Lexico? This will clarify whether the speedup comes from replacing OMP with orthogonal transform and topk selection, or difference in attention kernels and runtime optimizations.
2] With the popularity of "thinking" in LLMs, I think reasoning should be evaluated with workloads such as AIME and MATH in addition to GSM8K. Can the authors use reasoning models like DeepSeek-R1-Distill-Qwen/Llama to validate whether the KV cache compression is effective on long decoding?
3] If KV cache quantization achieves better accuracy at similar compression ratio, why would one use OrthoKV over KIVI?
4] Can the authors compare OrthoKV and KIVI-2 in terms of batched throughput across different input lengths and generation lengths? On top of the provided model accuracy, these measurements would form the quality-memory-throughput tradeoff comparison.
5] Can the authors compare an aggressively compressed 7-8B model using OrthoKV against a smaller model with a less aggressively compressed KV cache, under matched total GPU memory budget and workloads? This would help convince that OrthoKV offers a favorable quality-memory tradeoff. For example a comparison within the Qwen3 family, 8B with OrthoKV versus naive 4B or 1.7B.