acceptodds
Under review as a conference paper at ICLR 2027

LoGIK: Accelerating SVD approximation with Low-precision Gram Iterative Krylov

Abstract

Low-rank KV cache compression reduces memory by projecting each request's cache onto a compact channel subspace. Adapting this subspace to the realized cache is essential for accurate approximation, but requires online basis extraction, which can become a major bottleneck. Exact SVD provides the optimal rank- subspace but is too costly to run for every request, while faster approximations face their own trade-offs: lightweight methods converge slowly, whereas more accurate Krylov methods repeatedly access the full cache, eroding their speed advantage. We propose Low-precision Gram Iterative Krylov (LoGIK), a Gram-space reformulation of Block Krylov that performs refinement in the much smaller channel space. LoGIK forms the channel Gram matrix once, avoiding subsequent cache traversals and tall token-space orthogonalization. We further accelerate Gram construction with low-precision computation, selectively falling back to FP32 when needed to preserve the retained subspace. This design yields substantial accuracy–latency gains: on a 32k-token Llama-3.1-8B K cache at rank 16, LoGIK matches Block Krylov accuracy while reducing basis-extraction latency by . Integrated into existing KV-compression pipelines, it reduces compression latency by up to while preserving downstream quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Low-rank approximation, KV cache compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cfs6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LoGIK: exploiting the S≫D geometry of the KV cache, it moves Block-Krylov basis extraction from token space into the channel Gram space (form M=AᵀA once, then perform all Krylov iterations/orthogonalization/projection in the small space), accelerates the Gram with INT4/NVFP4, and adds a per-layer precision-fallback criterion.

Critical strengths
  • Recasting Block-Krylov basis extraction into the channel Gram space (via the Gram correspondence) is well-motivated by the S≫D geometry of KV caches, and the three correspondences are derived cleanly.

  • The paper covers the practical surface beyond the headline result: a two-pass full-SVD lower bound (Appendix B), an O(D²) incremental Gram update for decode (Appendix C), and a Blackwell NVFP4 kernel (Appendix K).

Critical weaknesses

W1: The share of basis extraction / SVD in end-to-end inference is never measured — "accelerating the bottleneck" cannot be assessed

The paper reports only the absolute latency of the optimized component (the numerator), never the full inference latency (the denominator):

  • §6.1 (line 980) and Figure 10 (line 4071) both state explicitly: "latency measures KV compression only and excludes shared attention and MLP."
  • The abstract's "major bottleneck" (line 66) has no end-to-end data behind it; the "End-to-end validation" contribution and Figure 7's "End to end" are misleading wording (they actually mean "across two pipelines" and "the whole basis-extraction sub-flow," not end-to-end inference).

Basis extraction runs once per request in prefill and is an O(D²) incremental update in decode, with an absolute cost of single-digit milliseconds per layer; whereas in a 32k prefill, attention is O(S²·D) and MLP is O(S·D_model²), likely 1–2 orders of magnitude larger. Without the denominator, the 15.78× cannot be translated into any end-to-end gain (a 1% share → ~+0.94% end-to-end; a 20% share → ~+19% — vastly different value). A paper arguing that it accelerates a bottleneck, yet not reporting that bottleneck's share, leaves its significance claim hanging.

W2: The "exact SVD" baseline implementation is unspecified, and is not the strongest exact baseline — the 10.22× is overstated

  • Undefined baseline: the headline "10.22× faster than exact SVD" (exact SVD = 67.96 ms in Table 2) never states the implementation (cuSOLVER gesvd/gesvdj/gesdd? torch.linalg.svd? CPU LAPACK? batched over heads?).
  • Wrong baseline choice: the right singular vectors are exactly the eigenvectors of M=AᵀA, and the paper's own Gram+eigh (FP32, CPU) reaches the same 0-excess optimum as exact SVD in 13.68 ms (about 5× faster). Hence LoGIK's speedup over the strongest exact baseline is only ~2× (13.68/6.65; at the 1% target 8.21/4.13≈2×), not 10.22×. The 10.22× is measured against a "slow full decomposition that nobody would actually use."

W3: Limited practical impact

The real-world impact is limited. Low-rank is not a mainstream KV-compression strategy (quantization / eviction are more widely use), and the paper gives no objective analysis of where it wins. Low-rank also carries overhead on both compression and inference — the multiple low-rank GEMMs at decoding are unavoidable — yet the paper optimizes only the prefill/compression stage, i.e. one part of the method's total cost, and its baselines lack the fairness and detail to judge the gain. As an efficiency paper it thus addresses only part of low-rank KV's limitations, whose practical benefit over quantization/eviction is itself limited. The contribution (Gram-space Krylov + low-precision tensor-core Gram) is really a numerical-linear-algebra/kernel result and fits a high-performance computing (HPC) venue better than this conference.

Questions for authors
  1. Under a real serving stack, what percentage of the full prefill and decode wall-clock latency is occupied by SVD/basis extraction? And what does it become after LoGIK? Please provide end-to-end breakdowns for both baseline and LoGIK; otherwise "major bottleneck" does not hold.
  2. What exactly is the "exact SVD" implementation (which library/routine, batched over heads)? Why not report the speedup against the strongest exact baseline Gram+eigh (0-excess in 13.68 ms), against which LoGIK is only ~2×?
6 comments

W1, Q1. Share of basis extraction in end-to-end inference and full latency breakdown

Thank you for raising this point. We profiled the full GEAR pipeline with HuggingFace generate (A100, batch 1, 32k prompt, 256 generated tokens). As Table R1 shows, KV compression takes roughly 30% of wall-clock time in both prefill and decode.

[Table R1] End-to-end breakdown at 32k (batch 1, 256-token generation)

ModelMethodCompression share in prefillCompression share in decodeTTFT speedupEnd-to-end speedup
Llama3-8BGEAR32%32%1.00x1.00x
Llama3-8BGEAR + LoGIK11%13%1.31x1.29x
Mistral-7BGEAR33%32%1.00x1.00x
Mistral-7BGEAR + LoGIK12%12%1.32x1.30x
Qwen2-7BGEAR29%32%1.00x1.00x
Qwen2-7BGEAR + LoGIK8%10%1.28x1.31x

Compression share = added time over FP16 / Method time. End-to-end = prefill + decode.

Replacing GEAR's power-iteration engine with LoGIK cuts this share to about 10% in both phases, speeding up TTFT and end-to-end generation by about 1.3×. We will add this breakdown to the main text to substantiate the "major bottleneck" claim, and rename "End-to-end validation" to "Pipeline-level validation" and Figure 7's title to "Full basis-extraction time".


W2, Q2. Exact SVD implementation details and speedup over Gram + eigh

Thank you for the precise feedback. Gram + eigh follows naturally from working with the Gram matrix, but existing pipelines (GEAR, SVDq, ShadowKV) do not use it, so our headline speedup is measured against the exact SVD these pipelines actually call. Against FP32 Gram + eigh (CPU eigh, 12.84 ms), LoGIK (loop 5) is 2.0× faster at virtually identical accuracy, and the gap grows to 4.7× at D=512D=512 (DeepSeek-V2-Lite MLA), since eigh scales as O(D3)O(D^3) and LoGIK as O(D2k)O(D^2 k), where DD is the channel dimension and k=rqk = rq is the Krylov subspace dimension (rank rr times loop count qq).

Our exact SVD calls PyTorch's default path (driver=None) in FP32, batched over 8 heads, as in the original SVDq implementation; we will state this in Section 6.1 (Setup). Benchmarking the other cuSOLVER drivers and CPU LAPACK (Table R2) confirms that this default batched path is the fastest exact SVD, and LoGIK is about 10× faster than it.

[Table R2] Latency by exact SVD implementation (Llama-3.1-8B K L15, 32k, 8 heads, FP32, rank 16)

ImplementationLatency (ms)Excess errorvs. LoGIK loop 5
torch.linalg.svd default (gesvdj), batched67.09010.7x
torch.linalg.svd gesvd, head loop157.87025.1x
CPU LAPACK gesdd (including transfer)1360.210216.7x
LoGIK (INT4) loop 33.840.47%0.6x
LoGIK (INT4) loop 56.280.0091%1.0x

In the SVDq pipeline, LoGIK cuts total compression latency by 5.1× over the original exact SVD while preserving accuracy. To avoid ambiguity, we will revise the headline to name the implementations: "15.78× faster than Block Krylov and 10.22× faster than exact SVD (gesvdj)."


W3. Practical impact of low-rank KV compression and venue fit

Thank you for sharing your perspective on practical impact and venue fit. Low-rank KV projection appears at major venues, e.g., Palu [1] and ShadowKV [2], and in deployed latent-attention architectures such as DeepSeek's MLA [3]. Rather than competing with quantization or token eviction, it serves as a complementary primitive in hybrid pipelines such as GEAR [4] and SVDq [5], on which we evaluate LoGIK. In the input-adaptive variants of these methods, online basis extraction is a major computational cost, and LoGIK is a drop-in replacement that accelerates both prefill and decode; it cuts GEAR's decode re-compression share by more than half (Table R1). We also characterize where LoGIK wins (k≪Dk \ll D) and where direct Gram diagonalization suffices (k≈Dk \approx D). Combining this reformulation with an LLM-specific quantization noise model, a layer-wise precision detector, and pipeline-level validation in real KV compression pipelines, our work addresses a practical bottleneck in long-context LLM inference and fits the ML systems scope of ICLR.

[1] Chang et al. Palu: KV-Cache Compression with Low-Rank Projection. ICLR 2025.

[2] Sun et al. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. ICML 2025.

[3] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.

[4] Kang et al. GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. arXiv:2403.05527.

[5] Hong et al. SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention. arXiv:2502.15304.

Thank you for the additional information on R1 and R2. However, several core issues remain unresolved:

  1. W1 still has not received an answer. My question was: In the actual service engine(like sglang,vllm), what percentage of time is spent on SVD? The supplementary content is still the straw man experiment. Table R1 uses HF 'generate', batch 1, without paged attention, without FlashAttention kernel, without CUDA graph, and without continuous batching. Therefore, the denominator is the unoptimized research script, and the calculated ratio cannot be transferred to the production prefill. Additionally, changing "End-to-end validation" to "Pipeline-level validation" is equivalent to admitting that the original statement lacks support.

  2. W2's response is untenable. "The existing pipeline does not use Gram+eigh" cannot be used as a reason to exclude the strongest and most accurate baseline. The slow SVD in GEAR and ShadowKV is a reference implementation for verifying accuracy because they are algorithmic papers; the contribution of this paper is inference performance, 10.22×, "major bottleneck", all are performance claims. The baseline should be the fastest correct method at present, rather than the one just sitting in someone else's repository. Please clearly define your contribution area and provide corresponding convincing experiments. RSVD has not been tested. In real LLM inference, random SVD (torch.svd_lowrank) will definitely be used (torch.svd_lowrank), so the main competitive baseline in this paper should be RSVD rather than SVD or the several draft implementations provided by the author. This must be tested; it cannot rely solely on discussion. Please add a direct comparison with torch.svd_lowrank in Table R1 and R2 and provide detailed experimental parameters.

  3. The experimental parameters are still unclear. Table R2 does not list rank, and Table R1 does not list the bit width of GEAR; the author provides information on how to implement the inverse quantization kernel in the supplementary experiments; these will significantly affect the overall efficiency of the algorithm.

  4. The author of W3 responded to another issue. I did not mean "No one presented a low-rank paper at the conference", I meant: The low-rank path incurs considerable delay costs in real inference, and end-to-end calculation is not cost-effective. This is an efficiency judgment, not a judgment on the number of publications. The example of DeepSeek MLA is incorrect. MLA is at the architecture level projecting KV to latent, the low-rank structure is fixed during training, the forward operation is just a low-rank matrix multiplication, no SVD is performed, and there is no online feature extraction. This analogy does not hold.

  5. In addition, I further checked the author's other responses and the paper. In the main experiment of the original GEAR paper: The accuracy reduction from 4-bit to 2-bit on the AQuA dataset is approximately 2%. However, in the author's appendix and supplementary experiments, the difference in accuracy caused by different SVD algorithms is as high as approximately 6%. In my opinion, this is abnormal - the low-rank in GEAR only accounts for a very small part (less than 10% of the quantization), according to the figure 3 provided by the author, the error of the low-rank should be below 1%, which is much smaller than the quantization error of 2-bit/4-bit, so the accuracy gap provided by the author is not convincing.

In conclusion, I am not satisfied with the author's response. If the above points cannot provide verifiable data and reasonable responses within the discussion period, I will consider lowering the score.

W1. Share of basis extraction in production serving engines

Thank you for clarifying your concern regarding production serving engines. We agree that the previous HuggingFace profiling alone does not fully characterize the relative overhead of basis extraction in optimized inference systems.

To address this concern, we additionally evaluated the impact of basis-extraction overhead against an optimized vLLM execution baseline. Specifically, we measured prefill and decode latency using vLLM 0.11 (A100, batch 1, 32k prompt, 256 generated tokens), with PagedAttention, FlashAttention, and CUDA graph optimizations enabled where supported. We then combined these measurements with the independently profiled compression times of each method to estimate their relative overheads and potential end-to-end improvements.

[Table R3] 32k end-to-end breakdown on vLLM (vLLM 0.11, batch 1, 256 generated tokens, PagedAttention, CUDA graph, and FlashAttention enabled)

ModelMethodvLLM baseline TTFT (ms)Compression time in prefill (ms)Compression share of prefillCompression share of decodeTTFT speedupEnd-to-end speedup
Llama3-8BGEAR3498201637%70%1.00x1.00x
Llama3-8BGEAR + torch.svd_lowrank3498347350%79%0.79x0.73x
Llama3-8BGEAR + LoGIK349854714%42%1.36x1.72x
Mistral-7BGEAR3498202737%71%1.00x1.00x
Mistral-7BGEAR + torch.svd_lowrank3498352050%80%0.79x0.71x
Mistral-7BGEAR + LoGIK349855814%41%1.36x1.76x
Qwen2-7BGEAR2993141332%65%1.00x1.00x
Qwen2-7BGEAR + torch.svd_lowrank2993226443%75%0.84x0.76x
Qwen2-7BGEAR + LoGIK299331810%31%1.33x1.71x

The results reveal an important observation: optimizing the underlying inference engine does not eliminate the basis-extraction bottleneck; instead, it can make its relative contribution more significant. When model execution becomes faster, the remaining decomposition overhead accounts for a larger fraction of total latency, assuming its execution cost remains unchanged.

Under this experiment, GEAR's compression accounts for approximately 32–37% of prefill latency and 65–71% of decode latency. Replacing its basis-extraction engine with LoGIK reduces these shares to 10–14% and 31–42%, respectively, yielding estimated end-to-end speedups of 1.71–1.76×.

We emphasize that these estimates quantify the potential benefit under an optimized serving baseline, rather than claiming measured performance under continuous batching or full vLLM integration.

We would also like to clarify the intended scope of our contribution and evaluation. Our primary contribution is a novel basis-extraction algorithm that achieves more accurate and computationally efficient low-rank approximation for the long tensors encountered in KV-cache compression. To evaluate this contribution fairly, we deliberately adopted experimental settings consistent with prior work, including GEAR, SVDq, and ShadowKV, rather than introducing additional serving-system optimizations that could confound direct algorithmic comparisons.

We respectfully disagree that such controlled comparisons constitute a "straw-man experiment." They directly evaluate the component our work aims to improve, under matched conditions with established baselines. While fully integrating the method into production serving engines would provide valuable additional system-level evidence, we believe this is distinct from evaluating the validity and novelty of the proposed algorithm.

Finally, renaming "End-to-end validation" to "Pipeline-level validation" is not a retraction of our findings, but a clarification of the evaluation scope. Section 6.1 already explicitly states that the reported latency covers KV compression and excludes shared attention and MLP operations. We have refined the terminology to avoid ambiguity and added the vLLM-based analysis and its assumptions to the appendix (Appendix L).

Reviewer_aKGNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LoGIK, a Gram-space version of Block Krylov meant to speed up the basis extraction step in low-rank KV cache compression. Methods like GEAR and SVDq have to compute the dominant channel subspace of the cache for every request. Block Krylov is accurate here but slow, since it keeps going back to the full S×D cache and does tall QR along the token axis. The main idea is simple: since S is much larger than D, form M=A⊤AM = A^\top A once and do all of the Krylov work (updates, orthogonalization, the final Rayleigh–Ritz step) on the small D×D matrix. After the Gram is built, the cache is never touched again.

That leaves Gram construction as the only part that grows with sequence length, and the authors compute it with INT4 tensor cores (NVFP4 on Blackwell). To decide when this is safe, they model the quantization noise floor of the Gram and get a condition in terms of σ1/σr\sigma_1/\sigma_r. In practice they pick INT4 or FP32 per layer using the Ritz ratio and the measured Gram error, with thresholds set empirically.

On a 32k-token Llama-3.1-8B K cache at rank 16, LoGIK reaches Block Krylov–level accuracy about 15.8× faster. Plugged into existing pipelines, it cuts compression latency by 3.6–4.4× relative to power iteration in GEAR and 5.1× relative to exact SVD in SVDq, with downstream accuracy roughly unchanged.

Critical strengths
  1. The core idea is simple and clearly explained. For long-context KV caches, S is much larger than D, so moving the Block Krylov refinement from token space into channel space fits the problem well, and it looks like the right move once you see it. The engine also drops into existing pipelines (GEAR, SVDq) without changing the compressed representation, which should make it easy to adopt in practice.

  2. The cost analysis shows that once the Gram is formed, the remaining work no longer depends on token length, so the benefit grows as the context gets longer. The authors are also upfront about where this stops working. Forming the Gram is not always cheaper in FLOPs, the advantage over Gram + full EVD only holds when k is much smaller than D, and the Gram itself becomes the remaining bottleneck. I appreciated that the operating regime is laid out this clearly instead of being left for the reader to discover.

  3. The reliability analysis of the low-precision Gram is, to me, the strongest part of the paper. Forming A^T A explicitly is usually discouraged in numerical linear algebra, and the authors take that concern seriously instead of ignoring it. They argue that for a rank-r approximation, what matters is sigma_1/sigma_r rather than the full condition number. From this they derive a noise-floor condition, and they show that the predicted failure boundary matches the measured one across bit widths and channel dimensions (Table 6). Without this analysis the INT4 Gram would look like a hack. With it, it becomes a well-motivated design choice.

  4. The claims are generally stated with appropriate care. The authors flag when numbers come from separate runs, and they explicitly caution against reading too much into small downstream score differences.

Critical weaknesses
  1. I’m not convinced Block Krylov is the right main baseline. When S >> D, token-space Block Krylov is at a structural disadvantage, so a good part of the 15.78× speedup simply reflects that mismatch. The more natural competitor is Gram + eigh, which forms the Gram once and diagonalizes it directly. In Table 2, INT4 Gram + eigh actually gets lower error (0.005% in 8.21 ms) than LoGIK (0.012% in 6.65 ms). Appendix D suggests that LoGIK’s time advantage depends heavily on implementation details. The batched cuSOLVER eigensolver stops batching beyond size 32, so the 128×128 eigh ends up as a CPU round trip of about 7 ms. With a properly optimized batched GPU eigensolver, much of this gap could go away at D=128, and on Mistral at loop 5 the ordering already flips (7.69 vs 7.39 ms). I was also surprised that Gram + eigh is missing from the SVDq setting, where k/D reaches 0.75 and LoGIK’s advantage should be weakest. For the real models tested (D=128), the contribution looks to me more like the INT4 Gram and its reliability analysis than the channel-space Krylov itself.

  2. The downstream results don’t really show that basis quality matters. In GEAR, Block Krylov gives a more accurate basis but scores lower than power iteration (CoT 57.41 vs 58.63, LongBench 38.18 vs 40.69). LoGIK, which approximates the same Krylov subspace, scores the highest. To me this suggests the numbers are dominated by run-to-run noise or pipeline-specific effects, and no seeds or variance are reported. With the current evidence, “preserves quality at lower latency” is a safer claim than “LoGIK leads both.” Latency is also measured only for the compression step, so I can’t tell what share of prefill time or TTFT this step takes at 32k context. Without that, it is hard to judge how much a 3–5× improvement matters in practice.

  3. The evaluation scope feels too narrow to judge generality. All experiments use 7–8B dense models from around 2024 (Llama3/3.1-8B, Mistral-7B, Qwen2/2.5-7B) with head dimension 128. It is unclear whether the speedup and the precision detector would hold on newer models with different KV structures such as MLA, on larger models, or on hybrid (SSM + attention) architectures where attention makes up a smaller share. The method is pitched as a drop-in engine, but downstream evaluation covers only GEAR and SVDq. ShadowKV in particular uses online SVD and fits exactly the scenario LoGIK targets, yet it is only mentioned in the text and never evaluated. The detector thresholds (30, 0.1) were also picked on the same 184 layer-states used for evaluation. The check on other prompts tests whether decisions agree after re-measurement, not whether fixed thresholds generalize. For H100 (Hopper), which lacks INT4/FP4 tensor cores, the paper only mentions falling back to INT8 and gives no latency numbers.

  4. While the claims are stated carefully, the paper itself is not easy to follow. The appendix is dense, and some parts, such as the frontier-point counting in App. D, took me a while to parse. It also isn’t explained why the model versions differ between the basis experiments (Llama-3.1-8B, Qwen2.5-7B) and the downstream experiments (Llama3-8B, Qwen2-7B).

Questions for authors
  1. My main question is about the comparison with Gram + eigh. If Gram + eigh were run with a batched GPU eigensolver tuned for 128×128 symmetric matrices (e.g., MAGMA batched syevd or a custom batched Jacobi kernel), would LoGIK still be clearly faster at the same excess error? If LoGIK keeps a clear time advantage in that comparison, most of my concern in Weakness 1 would be resolved, and I would be willing to raise my rating.
  2. If Gram + eigh (FP32 and INT4) were added as a baseline in the SVDq setting (k/D = 0.75), how would its latency and LongBench score compare?
  3. Could you report the mean and variance of the GEAR results over multiple seeds? I’m also curious why Block Krylov, which gives a more accurate basis, scores lower than power iteration. It would also help in interpreting the results to know whether LoGIK was run with fixed INT4 or with the precision chosen by the detector in the downstream experiments.
  4. At 32k context, what share of prefill time and TTFT does basis extraction (or the full KV compression step) account for?
  5. What happens to latency and accuracy if LoGIK replaces the SVD engine in ShadowKV, which also uses online SVD?
  6. Does the precision decision hold in a leave-one-model-out setting, where the thresholds (30, 0.1) are set on one model and then fixed for the others? Results on a model family not used in the evaluation would also be helpful.
  7. On H100, which lacks INT4/FP4 tensor cores, what is the actual basis extraction latency with an INT8 Gram, and how much faster is it than Block Krylov or Gram + eigh?
  8. Why do the model versions differ between the basis experiments (Llama-3.1-8B, Qwen2.5-7B) and the downstream experiments (Llama3-8B, Qwen2-7B)?
  9. Do you think LoGIK would still be useful for models that already use a latent low-rank KV, such as MLA, or for hybrid (SSM + attention) architectures where attention makes up a smaller share? (This is a future-work question and does not affect my rating.)
3 comments

W1, Q1, Q2. Why use Block Krylov iteration over Gram + eigh: Comparison with batched GPU eigensolvers and in the SVDq setting

Thank you for the concrete suggestions. Following your recommendation, we evaluated 128x128 symmetric eigendecomposition with a batch size of 8 using both MAGMA batched syevd and a custom batched Jacobi kernel (Table R1). While our custom batched Jacobi kernel proved slower than cuSOLVER, MAGMA batched syevd was about 2× faster than cuSOLVER. However, even when pairing Gram + eigh with this faster MAGMA solver, LoGIK remains roughly 1.7× faster at comparable excess error levels across both Llama-3.1-8B and Mistral-7B (Table R2).

[Table R1] 128x128 symmetric eigh, batch size 8, latency (ms)

cuSOLVERMAGMA batchedCustom batched Jacobi
21.5811.2030.33

[Table R2] Gram + eigh vs. LoGIK: Latency (ms) / Excess error, 32k, rank 16

Model, stateGram + eigh (MAGMA batched)LoGIK loop 5
Llama-3.1-8B K13.84 / 0.005%7.95 / 0.012%
Llama-3.1-8B V13.03 / 0.031%7.84 / 0.035%
Mistral-7B K13.09 / 0.005%7.71 / 0.009%
Mistral-7B V13.01 / 0.051%7.85 / 0.055%

This performance gap widens as the channel dimension DD increases because full eigendecomposition scales as O(D3)O(D^3), whereas LoGIK scales as O(D2k)O(D^2 k) with the Krylov subspace dimension k=rqk = rq (rank rr times loop count qq). When applied to the Multi-head Latent Attention (MLA) KV cache of DeepSeek-V2-Lite where D=512D=512, LoGIK achieves a 4.7× speedup over Gram + eigh (2.62 ms vs. 12.2 ms). Conversely, in the SVDq setting where kk reaches 0.75D0.75D, the latency and downstream accuracy of LoGIK and Gram + eigh converge to parity (Table R3), which aligns with our operating-regime analysis in the Appendix. To reflect these findings accurately, we will revise our main text to state that LoGIK provides a clear speed advantage when k≪Dk \ll D, while low-precision Gram + eigh becomes a competitive alternative as kk approaches DD.

[Table R3] SVDq setting (Llama3-8B, bˉ=2\bar{b}=2, D=1024D=1024)

EngineGSM8KLongBench Avg.Compression latency (ms)
Exact SVD78.1741.67151.9
Power iteration78.7741.3554.0
LoGIK78.2441.7527.9
Gram + eigh (FP32)77.9441.5332.9
Gram + eigh (INT4)77.7941.3029.6

W2, Q3, Q4. Basis quality vs. downstream performance, seed variance, and compression share of TTFT at 32k

Thank you for your insightful comments regarding the relationship between basis quality and downstream performance. To evaluate sensitivity to random initialization, we ran the GEAR experiments across three random seeds (Table R4). While LoGIK has the best or tied-best mean accuracy, the seed-to-seed variance is noticeable, so we will tone down our claim to state that LoGIK preserves downstream quality on par with or slightly better than the baselines. At a small target rank of 4, the differences in basis approximation error among methods are minor and largely masked by the subsequent 2-bit residual quantization noise, which explains why Block Krylov scored slightly lower than power iteration in our single-seed main table (Table 3 of the paper). Regarding the precision configuration in these downstream experiments, LoGIK operates in adaptive detector mode rather than fixed INT4, determining the precision on the first compression call for each layer and caching that decision for subsequent calls. We will clarify this protocol in Section 6.1.

[Table R4] GEAR seed {0, 1, 42} mean±std (Llama3-8B, rank 4, loop 3)

EngineGSM8KAQuABBH
Power iteration74.73±1.1844.36±3.3549.37±2.58
Block Krylov73.44±0.7943.31±5.5550.59±0.28
LoGIK74.70±2.4246.19±3.7452.21±1.93

To clarify how compression speedup translates to practical inference gains, we profiled the end-to-end prefill and generation latency at a 32k context length (Table R5). In the original GEAR pipeline, KV compression accounts for roughly 30% of the total prefill time (TTFT). Replacing GEAR's power-iteration engine with LoGIK reduces this compression share to about 10%, which directly accelerates overall TTFT by about 1.3× and boosts end-to-end decoding throughput by roughly 1.3× across all three models. We will integrate these end-to-end measurements into the main text.

[Table R5] End-to-end measurement at 32k (batch 1, 256-token generation)

ModelMethodPrefill (ms)Compression share in prefillTTFT speeduptok/s
Llama3-8BGEAR616932%1.00x8.0
Llama3-8BGEAR + LoGIK471411%1.31x10.3
Mistral-7BGEAR604533%1.00x8.1
Mistral-7BGEAR + LoGIK459212%1.32x10.5
Qwen2-7BGEAR501929%1.00x10.6
Qwen2-7BGEAR + LoGIK39088%1.28x13.8

W3, Q5, Q6, Q7, Q9. Generality verification: Detector thresholds, MLA and hybrid architectures, ShadowKV, and INT8

Thank you for the comprehensive suggestions on evaluating the generality of our method. To verify that our precision detector thresholds are not overfitted to the evaluated models, we first performed a leave-one-model-out cross-validation by calibrating the thresholds on two models and testing on the third. As shown in the top three rows of Table R6, the calibrated thresholds successfully caught 100% of the INT4 failures in every held-out model. Furthermore, when we applied our fixed thresholds (30, 0.1) directly to an unseen model family, Qwen3-8B, the detector caught every INT4 failure and kept the mean excess error well below the 0.1% safety target.

[Table R6] Detector generalization

Calibrated on -> Applied toThresholds (Ritz, Gram error)INT4 detection error rateUnnecessary FP32 fallbacks
Mistral, Qwen2.5 -> Llama(31.6, 0.098)0%0
Llama, Qwen2.5 -> Mistral(31.6, 0.092)0%0
Llama, Mistral -> Qwen2.5(18.4, 0.109)0%3
Three models (fixed) -> Qwen3-8B(30, 0.1)0%2

We also extended our evaluation to broader architectures and pipelines. On the MLA latent KV cache of DeepSeek-V2-Lite (D=512D=512), LoGIK remains highly effective, achieving an 18× speedup over exact SVD and a 4.7× speedup over Gram + eigh. For hybrid SSM-attention architectures, basis extraction applies only to the attention layers; thus, the per-layer compression speedup will remain identical while the end-to-end latency reduction scales proportionally with the fraction of attention layers, a point we will discuss in our future work section. In the online SVD setting of ShadowKV (D=1024D=1024, rank 160), LoGIK runs 6.8× faster than exact SVD and is on par with or slightly faster than Gram + eigh even as kk approaches DD (Table R7).

[Table R7] ShadowKV operating point (Llama3-8B, D=1024D=1024, 8k, rank 160, mean across 5 layers)

MethodLatency (ms)Excess error
exact SVD103.140.0005%
Gram + eigh16.190.0008%
LoGIK (loop 3)15.210.0005%

Measured on channel-centered pre-RoPE K.

Finally, to assess performance when falling back to INT8 on architectures like H100 (which we emulated using A100 INT8 tensor cores as we lack H100 access), we benchmarked an INT8 Gram kernel under the same conditions (Table R8). The INT8 Gram is nearly as fast as INT4 while matching the accuracy of FP32. At loop 5, INT8 LoGIK is 14.6× faster than Block Krylov and 1.5× faster than the CPU eigh-based Gram + eigh. Notably, all Qwen2.5 sink layers that required FP32 fallback under INT4 pass safely under INT8, so we expect that on Hopper GPUs, which lack INT4 tensor cores, INT8 LoGIK can run uniformly across all layers without needing the precision detector. We will include these analyses in the Appendix.

[Table R8] A100 INT8 tensor core (Llama-3.1-8B K L15, 32k)

FP32INT8INT4
Gram (ms)6.0480.4660.401
LoGIK loop 5 (ms / excess)13.00 / 0.0024%7.18 / 0.0024%7.16 / 0.0091%

W4, Q8. Appendix readability and model version discrepancy across experiments

Thank you for pointing out the readability issues. We will streamline the Appendix structure and replace the dense frontier-point counting passage in Appendix D with a concise summary sentence. Regarding the model versions, Llama3-8B-Instruct has a maximum context limit of 8k tokens, which prevented us from extracting 32k-token KV states for our basis-extraction benchmarks. We therefore used the newer Llama-3.1-8B and Qwen2.5-7B models (supporting 128k context) for the 32k basis evaluations, while retaining Llama3-8B and Qwen2-7B in the downstream benchmarks to ensure a faithful, apples-to-apples reproduction of the original GEAR and SVDq pipelines. We will explicitly note this in Section 6.1.

Thank you for the thorough rebuttal and the considerable effort that went into it. Many of the concerns I raised in my initial review have been clarified, so I will keep my score. I appreciate the authors’ hard work.

Reviewer_KmmTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, Low-precision Gram Iterative Krylov (LoGIK) is proposed to solve the bottleneck of KV cache by using Block Krylov in channel space. Moreover, low-precision computation is applied to speed up the Gram construction. The improvement of speed is high and downstream task accuracy is largely preserved.

Critical strengths
  1. This is not simple combination of existing modules, although Gram correspondence, Block Krylov, Cholesky whitening are not new methods but the converting Krylov refinement to channel space really solves the bottleneck of KV cache compression.
  2. The acceleration of speed is impressive. LoGIK reduces KV-compression latency by 3.58–4.39× in GEAR and 5.1× in SVDq while preserving downstream quality. Moreover, the reconstruction error is very competitive.
Critical weaknesses

None.

Questions for authors
  1. The INT4 fallback thresholds are empirically calibrated. Have the authors evaluated whether these thresholds remain reliable across different models or input distributions without recalibration?

  2. Could the authors provide end-to-end LLM inference latency or throughput results to clarify the practical performance gains?

1 comment

Q1. Generalization of INT4 detector thresholds across models and input distributions

Thank you for highlighting threshold generalizability. The Ritz ratio threshold is grounded in our perturbation analysis rather than tuned per model. Group-wise INT4 quantization adds a static diagonal noise floor proportional to λ1\lambda_1, and the retained subspace survives only while λ16\lambda_{16} stays sufficiently above it. This condition depends only on DD and bit-width and predicts a limit of σ1/σ16≈20\sigma_1/\sigma_{16} \approx 20 for D=128D=128 (Equation 13, Appendix E). The threshold 30 lies in the gap between INT4-failing and passing layers and is consistent with this prediction and with the 16-to-32 failure boundary on synthetic spectra (Appendix F). Because the criterion reflects quantization noise geometry rather than model-specific scales, we expected it to hold without recalibration, and verified this in two experiments (Table R1).

[Table R1] Detector generalization

Calibrated on → Applied toThresholds (Ritz, Gram error)INT4 detection error rateUnnecessary FP32 fallbacks
Mistral, Qwen2.5 → Llama(31.6, 0.098)0%0
Llama, Qwen2.5 → Mistral(31.6, 0.092)0%0
Llama, Mistral → Qwen2.5(18.4, 0.109)0%3
Three models (fixed) → Qwen3-8B(30, 0.1)0%2

First, in a leave-one-model-out evaluation, thresholds calibrated on two models caught every INT4 failure on the held-out model with few false positives. Second, we applied the fixed thresholds (30, 0.1) without recalibration to an unseen model family, Qwen3-8B (32k). The detector caught every INT4 failure and kept the mean excess error below 0.1%. For input distributions, with thresholds frozen, KV caches re-dumped from GovReport and RepoBench-P left virtually every one of the 184 layer-state decisions unchanged, and the single change moved safely to FP32 (Table 8 in Appendix F of the paper). We will add the new cross-model results to the revised manuscript.


Q2. End-to-end LLM inference latency and throughput

Thank you for the constructive suggestion. We measured TTFT, total generation latency, and decoding throughput of the full GEAR pipeline with HuggingFace generate on a single A100-80GB GPU, with a 32k context and 256 generated tokens (Table R2).

[Table R2] End-to-end measurement at 32k (batch 1, 256-token generation)

ModelMethodPrefill (ms)Compression share in prefillTTFT speedupEnd-to-end (s)tok/s
Llama3-8BGEAR616932%1.00x31.918.0
Llama3-8BGEAR + LoGIK471411%1.31x24.7410.3
Mistral-7BGEAR604533%1.00x31.678.1
Mistral-7BGEAR + LoGIK459212%1.32x24.3810.5
Qwen2-7BGEAR501929%1.00x24.1610.6
Qwen2-7BGEAR + LoGIK39088%1.28x18.5013.8

In the original GEAR pipeline, KV compression takes about 30% of prefill time. Replacing GEAR's power-iteration engine with LoGIK cuts this share to about 10%, speeding up TTFT by about 1.3×. Because GEAR periodically re-extracts the basis during decoding, decode overhead also drops, raising end-to-end generation throughput by about 30% on all three models. We will add these results to the main text.

Comments on the paper

1 comment

General Response to All Reviewers

We sincerely thank all reviewers (KmmT, aKGN, and cfs6) for their careful reading and constructive feedback. Your comments helped us sharpen the scope of our claims, make the baselines more complete, and clarify the experimental settings. We have uploaded a revised version of the paper, in which all changes are marked in blue. Below we summarize the main revisions by topic and indicate the reviewers whose comments motivated each change.

Summary of Revisions

1. Clarification of the exact SVD baseline (cfs6)

We specify the exact SVD implementation (PyTorch's batched torch.linalg.svd with the default cuSOLVER gesvdj driver) in Section 6.1 and Appendix H, and the speedups in the Abstract, Introduction, and Conclusion now name the implementations they compare against: 15.78× over Block Krylov and 10.22× over exact SVD (gesvdj).

2. LoGED: low-precision Gram + eigh as a formal operating point (cfs6, aKGN)

We formally define the combination of our low-precision Gram with a full eigendecomposition as LoGED (Section 4.2), and state when each method is preferable: LoGIK has a clear speed advantage when the Krylov dimension k ≪ D, while LoGED becomes competitive as k approaches D. We also add LoGED to the basis-extraction results (Table 2), the downstream GEAR and SVDq pipelines (Table 3), the scaling analysis (Section 6.4), and the token-length and rank sweeps (Tables 12 and 13). Since LoGED and LoGIK share the same INT4 Gram, this comparison isolates the benefit of LoGIK's reduced Krylov solve. Appendix D is retitled and summarized accordingly.

3. End-to-end cost in the HuggingFace harness and under vLLM (KmmT, aKGN, cfs6)

We add end-to-end measurements at 32k tokens for three models (Section 6.3, Appendix L, Table 14). LoGIK lowers the compression share of prefill from about 30% to about 10% and speeds up both TTFT and end-to-end generation by about 1.3×. We also estimate the compression share against an optimized vLLM serving baseline (PagedAttention, FlashAttention, CUDA graphs) and describe how the estimate is obtained (Appendix L, Table 15). The corresponding contribution is renamed "Pipeline-level validation".

4. Generalization of the precision detector and an INT8 Gram (KmmT, aKGN)

We add a cross-model evaluation of the detector thresholds (Appendix F, Table 9). In a leave-one-model-out evaluation, the thresholds catch every INT4 failure on the held-out model, and the fixed thresholds transfer without recalibration to an unseen model family, Qwen3-8B. We also describe the adaptive precision protocol used in the downstream pipelines (Section 6.1). Also, We evaluate an INT8 Gram on A100 INT8 tensor cores (Appendix N, Table 18). It is nearly as fast as INT4 and as accurate as FP32, and every layer that needs the FP32 fallback under INT4 passes under INT8.

5. Reproducibility of GEAR and seed sensitivity (cfs6, aKGN)

To verify reproducibility, we evaluate the base LLaMA3-8B used in the original GEAR paper with its official implementation (Appendix M, Table 16), and report the AQuA mean and standard deviation over three seeds for each engine (Table 17), where LoGIK attains the highest mean accuracy. We also clarify how downstream accuracy relates to the reconstruction error.

6. Clarification of experimental settings and presentation (aKGN, cfs6)

We clarify the model versions used in each benchmark, the shared GEAR dequantization, and the compression latency measurement (Section 6.1, Appendix H). We also refine the caption of Table 3, the frontier summary in Appendix D, and the panel title of Figure 7. We also discuss hybrid SSM–attention architectures in the Limitations (Appendix A): the per-layer speedup is unchanged, while the end-to-end gain scales with the fraction of attention layers.

Closing

We are grateful again to all reviewers for the time and effort they devoted to our submission. Your feedback substantially improved the clarity and completeness of the paper, and we would be happy to address any further questions during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.