- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes LoGIK: exploiting the S≫D geometry of the KV cache, it moves Block-Krylov basis extraction from token space into the channel Gram space (form M=AᵀA once, then perform all Krylov iterations/orthogonalization/projection in the small space), accelerates the Gram with INT4/NVFP4, and adds a per-layer precision-fallback criterion.
Critical strengths
-
Recasting Block-Krylov basis extraction into the channel Gram space (via the Gram correspondence) is well-motivated by the S≫D geometry of KV caches, and the three correspondences are derived cleanly.
-
The paper covers the practical surface beyond the headline result: a two-pass full-SVD lower bound (Appendix B), an O(D²) incremental Gram update for decode (Appendix C), and a Blackwell NVFP4 kernel (Appendix K).
Critical weaknesses
W1: The share of basis extraction / SVD in end-to-end inference is never measured — "accelerating the bottleneck" cannot be assessed
The paper reports only the absolute latency of the optimized component (the numerator), never the full inference latency (the denominator):
- §6.1 (line 980) and Figure 10 (line 4071) both state explicitly: "latency measures KV compression only and excludes shared attention and MLP."
- The abstract's "major bottleneck" (line 66) has no end-to-end data behind it; the "End-to-end validation" contribution and Figure 7's "End to end" are misleading wording (they actually mean "across two pipelines" and "the whole basis-extraction sub-flow," not end-to-end inference).
Basis extraction runs once per request in prefill and is an O(D²) incremental update in decode, with an absolute cost of single-digit milliseconds per layer; whereas in a 32k prefill, attention is O(S²·D) and MLP is O(S·D_model²), likely 1–2 orders of magnitude larger. Without the denominator, the 15.78× cannot be translated into any end-to-end gain (a 1% share → ~+0.94% end-to-end; a 20% share → ~+19% — vastly different value). A paper arguing that it accelerates a bottleneck, yet not reporting that bottleneck's share, leaves its significance claim hanging.
W2: The "exact SVD" baseline implementation is unspecified, and is not the strongest exact baseline — the 10.22× is overstated
- Undefined baseline: the headline "10.22× faster than exact SVD" (exact SVD = 67.96 ms in Table 2) never states the implementation (cuSOLVER gesvd/gesvdj/gesdd? torch.linalg.svd? CPU LAPACK? batched over heads?).
- Wrong baseline choice: the right singular vectors are exactly the eigenvectors of M=AᵀA, and the paper's own Gram+eigh (FP32, CPU) reaches the same 0-excess optimum as exact SVD in 13.68 ms (about 5× faster). Hence LoGIK's speedup over the strongest exact baseline is only ~2× (13.68/6.65; at the 1% target 8.21/4.13≈2×), not 10.22×. The 10.22× is measured against a "slow full decomposition that nobody would actually use."
W3: Limited practical impact
The real-world impact is limited. Low-rank is not a mainstream KV-compression strategy (quantization / eviction are more widely use), and the paper gives no objective analysis of where it wins. Low-rank also carries overhead on both compression and inference — the multiple low-rank GEMMs at decoding are unavoidable — yet the paper optimizes only the prefill/compression stage, i.e. one part of the method's total cost, and its baselines lack the fairness and detail to judge the gain. As an efficiency paper it thus addresses only part of low-rank KV's limitations, whose practical benefit over quantization/eviction is itself limited. The contribution (Gram-space Krylov + low-precision tensor-core Gram) is really a numerical-linear-algebra/kernel result and fits a high-performance computing (HPC) venue better than this conference.
Questions for authors
- Under a real serving stack, what percentage of the full prefill and decode wall-clock latency is occupied by SVD/basis extraction? And what does it become after LoGIK? Please provide end-to-end breakdowns for both baseline and LoGIK; otherwise "major bottleneck" does not hold.
- What exactly is the "exact SVD" implementation (which library/routine, batched over heads)? Why not report the speedup against the strongest exact baseline Gram+eigh (0-excess in 13.68 ms), against which LoGIK is only ~2×?
6 comments
Thank you for the additional information on R1 and R2. However, several core issues remain unresolved:
-
W1 still has not received an answer. My question was: In the actual service engine(like sglang,vllm), what percentage of time is spent on SVD? The supplementary content is still the straw man experiment. Table R1 uses HF 'generate', batch 1, without paged attention, without FlashAttention kernel, without CUDA graph, and without continuous batching. Therefore, the denominator is the unoptimized research script, and the calculated ratio cannot be transferred to the production prefill. Additionally, changing "End-to-end validation" to "Pipeline-level validation" is equivalent to admitting that the original statement lacks support.
-
W2's response is untenable. "The existing pipeline does not use Gram+eigh" cannot be used as a reason to exclude the strongest and most accurate baseline. The slow SVD in GEAR and ShadowKV is a reference implementation for verifying accuracy because they are algorithmic papers; the contribution of this paper is inference performance, 10.22×, "major bottleneck", all are performance claims. The baseline should be the fastest correct method at present, rather than the one just sitting in someone else's repository. Please clearly define your contribution area and provide corresponding convincing experiments. RSVD has not been tested. In real LLM inference, random SVD (
torch.svd_lowrank) will definitely be used (torch.svd_lowrank), so the main competitive baseline in this paper should be RSVD rather than SVD or the several draft implementations provided by the author. This must be tested; it cannot rely solely on discussion. Please add a direct comparison withtorch.svd_lowrankin Table R1 and R2 and provide detailed experimental parameters. -
The experimental parameters are still unclear. Table R2 does not list rank, and Table R1 does not list the bit width of GEAR; the author provides information on how to implement the inverse quantization kernel in the supplementary experiments; these will significantly affect the overall efficiency of the algorithm.
-
The author of W3 responded to another issue. I did not mean "No one presented a low-rank paper at the conference", I meant: The low-rank path incurs considerable delay costs in real inference, and end-to-end calculation is not cost-effective. This is an efficiency judgment, not a judgment on the number of publications. The example of DeepSeek MLA is incorrect. MLA is at the architecture level projecting KV to latent, the low-rank structure is fixed during training, the forward operation is just a low-rank matrix multiplication, no SVD is performed, and there is no online feature extraction. This analogy does not hold.
-
In addition, I further checked the author's other responses and the paper. In the main experiment of the original GEAR paper: The accuracy reduction from 4-bit to 2-bit on the AQuA dataset is approximately 2%. However, in the author's appendix and supplementary experiments, the difference in accuracy caused by different SVD algorithms is as high as approximately 6%. In my opinion, this is abnormal - the low-rank in GEAR only accounts for a very small part (less than 10% of the quantization), according to the figure 3 provided by the author, the error of the low-rank should be below 1%, which is much smaller than the quantization error of 2-bit/4-bit, so the accuracy gap provided by the author is not convincing.
In conclusion, I am not satisfied with the author's response. If the above points cannot provide verifiable data and reasonable responses within the discussion period, I will consider lowering the score.
W1. Share of basis extraction in production serving engines
Thank you for clarifying your concern regarding production serving engines. We agree that the previous HuggingFace profiling alone does not fully characterize the relative overhead of basis extraction in optimized inference systems.
To address this concern, we additionally evaluated the impact of basis-extraction overhead against an optimized vLLM execution baseline. Specifically, we measured prefill and decode latency using vLLM 0.11 (A100, batch 1, 32k prompt, 256 generated tokens), with PagedAttention, FlashAttention, and CUDA graph optimizations enabled where supported. We then combined these measurements with the independently profiled compression times of each method to estimate their relative overheads and potential end-to-end improvements.
[Table R3] 32k end-to-end breakdown on vLLM (vLLM 0.11, batch 1, 256 generated tokens, PagedAttention, CUDA graph, and FlashAttention enabled)
| Model | Method | vLLM baseline TTFT (ms) | Compression time in prefill (ms) | Compression share of prefill | Compression share of decode | TTFT speedup | End-to-end speedup |
|---|---|---|---|---|---|---|---|
| Llama3-8B | GEAR | 3498 | 2016 | 37% | 70% | 1.00x | 1.00x |
| Llama3-8B | GEAR + torch.svd_lowrank | 3498 | 3473 | 50% | 79% | 0.79x | 0.73x |
| Llama3-8B | GEAR + LoGIK | 3498 | 547 | 14% | 42% | 1.36x | 1.72x |
| Mistral-7B | GEAR | 3498 | 2027 | 37% | 71% | 1.00x | 1.00x |
| Mistral-7B | GEAR + torch.svd_lowrank | 3498 | 3520 | 50% | 80% | 0.79x | 0.71x |
| Mistral-7B | GEAR + LoGIK | 3498 | 558 | 14% | 41% | 1.36x | 1.76x |
| Qwen2-7B | GEAR | 2993 | 1413 | 32% | 65% | 1.00x | 1.00x |
| Qwen2-7B | GEAR + torch.svd_lowrank | 2993 | 2264 | 43% | 75% | 0.84x | 0.76x |
| Qwen2-7B | GEAR + LoGIK | 2993 | 318 | 10% | 31% | 1.33x | 1.71x |
The results reveal an important observation: optimizing the underlying inference engine does not eliminate the basis-extraction bottleneck; instead, it can make its relative contribution more significant. When model execution becomes faster, the remaining decomposition overhead accounts for a larger fraction of total latency, assuming its execution cost remains unchanged.
Under this experiment, GEAR's compression accounts for approximately 32–37% of prefill latency and 65–71% of decode latency. Replacing its basis-extraction engine with LoGIK reduces these shares to 10–14% and 31–42%, respectively, yielding estimated end-to-end speedups of 1.71–1.76×.
We emphasize that these estimates quantify the potential benefit under an optimized serving baseline, rather than claiming measured performance under continuous batching or full vLLM integration.
We would also like to clarify the intended scope of our contribution and evaluation. Our primary contribution is a novel basis-extraction algorithm that achieves more accurate and computationally efficient low-rank approximation for the long tensors encountered in KV-cache compression. To evaluate this contribution fairly, we deliberately adopted experimental settings consistent with prior work, including GEAR, SVDq, and ShadowKV, rather than introducing additional serving-system optimizations that could confound direct algorithmic comparisons.
We respectfully disagree that such controlled comparisons constitute a "straw-man experiment." They directly evaluate the component our work aims to improve, under matched conditions with established baselines. While fully integrating the method into production serving engines would provide valuable additional system-level evidence, we believe this is distinct from evaluating the validity and novelty of the proposed algorithm.
Finally, renaming "End-to-end validation" to "Pipeline-level validation" is not a retraction of our findings, but a clarification of the evaluation scope. Section 6.1 already explicitly states that the reported latency covers KV compression and excludes shared attention and MLP operations. We have refined the terminology to avoid ambiguity and added the vLLM-based analysis and its assumptions to the appendix (Appendix L).
W1, Q1. Share of basis extraction in end-to-end inference and full latency breakdown
Thank you for raising this point. We profiled the full GEAR pipeline with HuggingFace generate (A100, batch 1, 32k prompt, 256 generated tokens). As Table R1 shows, KV compression takes roughly 30% of wall-clock time in both prefill and decode.
[Table R1] End-to-end breakdown at 32k (batch 1, 256-token generation)
Compression share = added time over FP16 / Method time. End-to-end = prefill + decode.
Replacing GEAR's power-iteration engine with LoGIK cuts this share to about 10% in both phases, speeding up TTFT and end-to-end generation by about 1.3×. We will add this breakdown to the main text to substantiate the "major bottleneck" claim, and rename "End-to-end validation" to "Pipeline-level validation" and Figure 7's title to "Full basis-extraction time".
W2, Q2. Exact SVD implementation details and speedup over Gram + eigh
Thank you for the precise feedback. Gram + eigh follows naturally from working with the Gram matrix, but existing pipelines (GEAR, SVDq, ShadowKV) do not use it, so our headline speedup is measured against the exact SVD these pipelines actually call. Against FP32 Gram + eigh (CPU eigh, 12.84 ms), LoGIK (loop 5) is 2.0× faster at virtually identical accuracy, and the gap grows to 4.7× at D=512 (DeepSeek-V2-Lite MLA), since eigh scales as O(D3) and LoGIK as O(D2k), where D is the channel dimension and k=rq is the Krylov subspace dimension (rank r times loop count q).
Our exact SVD calls PyTorch's default path (driver=None) in FP32, batched over 8 heads, as in the original SVDq implementation; we will state this in Section 6.1 (Setup). Benchmarking the other cuSOLVER drivers and CPU LAPACK (Table R2) confirms that this default batched path is the fastest exact SVD, and LoGIK is about 10× faster than it.
[Table R2] Latency by exact SVD implementation (Llama-3.1-8B K L15, 32k, 8 heads, FP32, rank 16)
In the SVDq pipeline, LoGIK cuts total compression latency by 5.1× over the original exact SVD while preserving accuracy. To avoid ambiguity, we will revise the headline to name the implementations: "15.78× faster than Block Krylov and 10.22× faster than exact SVD (gesvdj)."
W3. Practical impact of low-rank KV compression and venue fit
Thank you for sharing your perspective on practical impact and venue fit. Low-rank KV projection appears at major venues, e.g., Palu [1] and ShadowKV [2], and in deployed latent-attention architectures such as DeepSeek's MLA [3]. Rather than competing with quantization or token eviction, it serves as a complementary primitive in hybrid pipelines such as GEAR [4] and SVDq [5], on which we evaluate LoGIK. In the input-adaptive variants of these methods, online basis extraction is a major computational cost, and LoGIK is a drop-in replacement that accelerates both prefill and decode; it cuts GEAR's decode re-compression share by more than half (Table R1). We also characterize where LoGIK wins (k≪D) and where direct Gram diagonalization suffices (k≈D). Combining this reformulation with an LLM-specific quantization noise model, a layer-wise precision detector, and pipeline-level validation in real KV compression pipelines, our work addresses a practical bottleneck in long-context LLM inference and fits the ML systems scope of ICLR.
[1] Chang et al. Palu: KV-Cache Compression with Low-Rank Projection. ICLR 2025.
[2] Sun et al. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. ICML 2025.
[3] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.
[4] Kang et al. GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. arXiv:2403.05527.
[5] Hong et al. SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention. arXiv:2502.15304.