- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
BasisKV targets long-context tensor-parallel inference, where KV-cache growth and per-GPU attention-output communication are both expensive. The method fits low-rank factors for each tensor-parallel source against the complete attention output, preserves source-specific latents through an AllGather, and reconstructs full-width outputs after communication. For linear Value paths, the same encoder is folded into the Value projection and reused for Value-cache compression. The compressed Value also predicts part of the pre-RoPE Key for page routing, with a low-rank Key residual and exact QK recomputation on selected pages.
Critical strengths
- Using one low-rank Value latent for communication, Value-cache compression, and routing is a coherent and practically motivated system design.
- The joint attention-output objective, alternating-least-squares updates, routing-aligned basis rotation, and Page-Fisher residual fitting are described in useful detail. The evaluation spans multiple model families, retention budgets, long-context benchmarks, and tensor-parallel settings.
- Table 9 provides a useful routing ablation: B16R16 averages 79.48 on RULER, comparable to R32 at 79.52, and improves over R16 with a reported paired interval of [0.80, 5.17]. The Value-derived component therefore has measurable value.
Critical weaknesses
- The efficiency claim is not quality- or latency-matched. The 1.42× speedup uses V64/50% retention on Llama-3.1-8B, where WikiText-2 perplexity rises from 7.22 to 11.50 and average MCQ falls from 0.709 to 0.673. The quality–efficiency trade-off is also model-dependent: at 75% retention, PaLU G-LRD has lower Llama perplexity (7.98 versus 8.61) but much higher communication traffic. A unified accuracy–memory–latency Pareto comparison, including measured PaLU latency, is still needed.
- The scope of the memory claim should be explicit. Table 4 and Eq. (9) give useful logical widths and cache-traffic formulas, and Figure 4 reports measured peak decode memory. However, exact Keys and Values remain GPU-resident in the main sparse-decoding experiments, and the paper lacks a state-level byte breakdown for Table 3, page-metadata overhead, and an actual offload measurement.
- Routing reuse is demonstrated, but its practical advantage needs clearer accounting. Table 9 shows that B16R16 matches R32 while using a 16-dimensional Key residual, so the Value base is useful. However, B16R16 still performs a 32-dimensional full-context routing read, and the aggregate decode results do not isolate whether the saved persistent state translates into routing-specific latency or memory savings.
- The GQA parameterization, uncertainty, and artifacts need strengthening. The paper first fits an encoder per TP source and later states that query heads sharing a physical KV head share a Value encoder while retaining head-specific decoders; the constrained optimization problem and communication-byte formula are unclear. Main tables lack confidence intervals or calibration-seed variance, several margins are small, and no code artifact is linked. The narrow “first” claim should be retained only as an unverified, explicitly scoped statement.
Questions for authors
- What is the exact GQA parameterization, including shared Value encoders, head-specific decoders, and the resulting communication-byte formula?
- Can the authors provide a quality-matched Pareto comparison and measured latency for Dense, PaLU, sparse baselines, and BasisKV?
- Can they report a state-level byte breakdown for the Table 3 setting, page-metadata overhead, actual offload behavior, and a code artifact sufficient to reproduce the timing and memory results?
4 comments
Response to Reviewer V2Q8 — Part 2/2
Q2. Quality–memory–latency comparison
TP8 Pareto comparison. We added a TP8 comparison for Llama-3.1-8B-Instruct on eight L40S GPUs using the Table 1 75% Value-retention checkpoints. For PaLU G-LRD4, we measured both group-latent replication across ranks as PaLU-A and group-latent sharding as PaLU-B, using identical factors and rank allocations. Each request has 4096 input tokens and generates 128 tokens, reporting cohort E2E completion time after one warmup cohort:
| Method | WikiT. / C4 PPL ↓ | MCQ ↑ | B32 E2E s ↓ | B64 E2E s ↓ | B64 logical KV GiB/rank ↓ |
|---|---|---|---|---|---|
| Dense | 7.22 / 9.77 | 0.709 | 11.40 | 20.39 | 4.12 |
| BasisKV | 8.61 / 10.66 | 0.704 | 9.91 | 17.06 | 3.61 |
| PaLU-A: replicated | 7.98 / 10.66 | 0.698 | 12.05 | 22.21 | 8.25 |
| PaLU-B: sharded | 7.98 / 10.66 | 0.698 | 15.20 | 26.71 | 3.61 |
At B64, BasisKV achieves 1.20× E2E speedup over Dense, 1.30× over PaLU-A, and 1.57× over PaLU-B, while reducing logical KV storage by 12.5% relative to Dense, matching PaLU G-LRD4 on C4 perplexity, and improving 7-task MCQ accuracy. PaLU-A doubles logical KV storage over Dense through latent replication, whereas PaLU-B recovers the cache savings at the cost of extra intra-group communication and higher measured latency. Logical KV sizes describe the live cache. We include this comparison in Section 5 and Appendix D.1.
Sparse baselines under TP1 and TP8. We also compare runtime and GPU memory on L40S GPUs for Llama-3.1-8B-Instruct in Appendix D.3 under the dense-Value sparse setting of Appendix C.3. Across 64K and 128K prefills with 128 generated tokens, the results are shown below with Keys offloaded for BasisKV and LRQK and Values offloaded for ShadowKV:
| Setting | Method | Prefill-to-ready s | E2E s | Decode ms/step | GPU resident GiB/rank |
|---|---|---|---|---|---|
| TP1, B1, 64K | BasisKV | 11.47 | 15.22 | 29.44 | 19.72 |
| TP1, B1, 64K | ShadowKV | 18.25 | 22.05 | 29.86 | 16.46 |
| TP1, B1, 64K | LRQK | 15.73 | 54.23 | 302.90 | 26.04 |
| TP1, B1, 128K | BasisKV | 35.12 | 39.32 | 33.20 | 24.23 |
| TP1, B1, 128K | ShadowKV | 46.11 | 50.13 | 31.66 | 17.60 |
| TP1, B1, 128K | LRQK | OOM | OOM | OOM | OOM |
| TP8, B16, 64K | BasisKV | 85.48 | 90.80 | 42.50 | 13.08 |
| TP8, B16, 64K | ShadowKV | 154.11 | 160.32 | 51.64 | 15.33 |
Unlike ShadowKV and LRQK, which rely on online, per-request factorization, BasisKV uses purely offline-calibrated linear projections, avoiding any runtime SVD or factor updates. At TP1, ShadowKV lowers GPU memory by offloading Values, matching BasisKV's decode speed at the cost of extra online prefill SVD, whereas LRQK's online fitting and per-step factor updates slow decoding by over 10× and run out of memory during 128K prefill. In our TP8 implementation of ShadowKV, we preserve its joint Key SVD across all KV heads: at each layer, we gather all ranks' Key shards onto a rotating owner GPU, run the online FP32 SVD per request, broadcast the shared temporal basis to every rank, and scatter the per-head slices. Replicating across all eight GPUs eliminates ShadowKV's single-GPU memory advantage over BasisKV's sharded Values while nearly doubling prefill-to-ready latency.
Q3. State accounting, offload, and reproducibility
State breakdown and Key offload. We make the memory scope explicit in Section 5 and Appendix B.4: the original Table 3 configuration at 128K on one L40S retains exact Keys and Values on GPU. Its 32.98 GiB PyTorch decode allocation comprises 14.96 GiB model parameters, 7.95 GiB each for exact Keys and Values, 0.99 GiB each for the separate Base and Key-residual caches, and 0.14 GiB for routing metadata, RoPE, and workspace buffers. Rerunning this setup alongside matched Base-reuse and Key-offload variants gives:
| Configuration | Decode GPU GiB | Pinned host K GiB | Prefill peak GPU GiB | Decode ms/step |
|---|---|---|---|---|
| Dense | 30.86 | 0 | 38.36 | 47.35 |
| BasisKV, separate Base cache | 32.98 | 0 | 40.48 | 37.40 |
| BasisKV, Base reuse | 31.98 | 0 | 39.48 | 37.83 |
| BasisKV, reuse + Key offload | 24.29 | 7.95 | 31.79 | 39.48 |
Routing reuse, offload, and metadata. Reading Base directly from the rotated Values removes the dedicated Base cache while leaving the 32-dimensional routing scan virtually unchanged at 11.90 versus 11.75 ms, confirming that reuse reduces persistent memory rather than scan latency. Offloading historical Keys to pinned host memory and fetching only missed pages on demand substantially lowers GPU decode residency below Dense with only minor on-demand transfer overhead, while page scores and routing metadata occupy only ~0.05 GiB on GPU.
Code and reproducibility. We provide an anonymous BasisKV repository and benchmark guide covering factor fitting, evaluation, TP serving, sparse decoding, and offload accounting.
Thanks for your comprehensive response, I wouled raise my score to 3.
Thanks for your comprehensive response, I wouled raise my score to 3.
We sincerely thank the reviewer for reconsidering our work and raising the score. We are glad that our response and additional results were helpful. Your detailed feedback helped us strengthen the experimental evaluation and clarify the presentation. Thank you again for your constructive review.
Response to Reviewer V2Q8 — Part 1/2
We sincerely thank the reviewer for the careful reading, thoughtful questions, and constructive suggestions. Below, Part 1/2 covers the GQA formulation, collective-traffic equations, and multi-seed uncertainty analyses, and Part 2/2 covers unified quality–memory–latency comparisons, 128K state and offload accounting, and our anonymous code repository.
Q1. GQA parameterization and communication
GQA is enforced through parameter tying. Let Hq and Hkv be the numbers of query and physical KV heads, and let g(h) map query head h to its KV head. With attention weights Ah, define Zh=AhVg(h). All query heads associated with the same KV head share an encoder Eg∈Rdv×r, while retaining separate decoder blocks Dh∈Rr×dmodel. For the stacked calibration outputs Y, the constrained objective is
\left\\|Y-\sum_{h=1}^{H_q}Z_hE_{g(h)}D_h\right\\|_F^2.$$ The decoder blocks are fitted jointly against the complete output, and updating a shared encoder accounts for all query heads that use it. In the TP-source notation of Equation 1, $E_p$ is therefore block-diagonal over the query heads owned by rank $p$ with blocks tied to $E_{g(h)}$, rather than a single unconstrained matrix per GPU. We cache $\widetilde V_g=V_gE_g$ once per physical KV head. Each query head computes its own attention-output latent $U_h=A_h\widetilde V_{g(h)}$, which remains distinct because the attention weights differ. The ranks AllGather these latents and reconstruct $\widehat Y=\sum_h U_hD_h$. Consequently, the Value cache scales with $H_{\mathrm{kv}}$, whereas the communicated latent width scales with $H_q$. For a layer with uniform rank $r$, $P$ TP ranks evenly partitioning KV groups without replication, and $s$ bytes per element, the compressed Value cache occupies $sH_{\mathrm{kv}}r/P$ bytes per cached token per rank. Including unchanged dense Keys of width $d_k$, the logical KV storage is $$M_{\mathrm{KV}}=\frac{sH_{\mathrm{kv}}}{P}(d_k+r) \quad\text{bytes per cached token per layer per rank}.$$ For $N$ current query tokens across the batch, the ideal ring collective traffic is $$C_{\mathrm{AllGather}}=\frac{P-1}{P}sNH_qr, \qquad C_{\mathrm{AllReduce}}=2\frac{P-1}{P}sNd_{\mathrm{model}}.$$ These are **bytes sent per rank per layer**, counting one direction; receive traffic is the same. The AllReduce expression refers to the dense full-width attention output, giving the relative traffic ratio $C_{\mathrm{AllGather}}/C_{\mathrm{AllReduce}}=H_qr/(2d_{\mathrm{model}})$ reported in Table 1 at average rank $r$. During one decode step, $N$ is the active batch size. We have made the shared-encoder constraint, head-specific decoders, and accounting units explicit in Section 3.2 and Appendix A.1. ### Additional: Calibration-seed variation and confidence intervals We repeated the Table 1 V64 comparison across C4 calibration seeds 42, 43, and 44, resampling calibration windows and refitting both factors and rank schedules per seed, reporting mean ± SD below: |Model|Method|WikiText-2 PPL ↓|C4 PPL ↓|7-task MCQ (%) ↑| |---|---|---:|---:|---:| |Qwen3-8B-Base|BasisKV|**8.220 ± 0.223**|**10.089 ± 0.023**|**67.34 ± 0.59**| |Qwen3-8B-Base|PaLU M-LRD|14.235 ± 0.892|16.257 ± 0.805|60.06 ± 0.64| |Qwen3-8B-Base|PaLU G-LRD4|10.645 ± 0.153|12.956 ± 0.128|64.94 ± 0.21| |Llama-3.1-8B-Instruct|BasisKV|11.096 ± 0.075|**12.529 ± 0.074**|**67.52 ± 0.58**| |Llama-3.1-8B-Instruct|PaLU M-LRD|15.592 ± 0.546|17.495 ± 0.504|59.59 ± 0.67| |Llama-3.1-8B-Instruct|PaLU G-LRD4|**10.506 ± 0.357**|13.040 ± 0.174|65.41 ± 0.38| Calibration variance is small across seeds: BasisKV leads on MCQ and C4 PPL in every seed on both models, while PaLU G-LRD4 retains lower Llama WikiText-2 PPL. Across 10000 task-stratified paired bootstrap resamples, BasisKV's three-seed MCQ gain over PaLU G-LRD4 is **+2.40 pp** with 95% CI 1.79 to 3.02 on Qwen and **+2.12 pp** with 95% CI 1.60 to 2.64 on Llama. We have added these statistics in Section 5 and Appendix C.2. We have also removed the “first” claim in Section 1 to state our cache-communication co-design, joint-fitting, and representation-reuse contributions directly.