- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies KV-cache memory, which grows with both sequence length and network depth. It argues that existing cross-layer approaches either share KV states exactly (CLA, YOCO), leaving no room for layer-specific variation, or compress already-trained caches post hoc (MiniCache, xKV), without end-to-end learnability. It proposes KVPath, which partitions the L layers into groups of m and treats each token's KV states inside a group as a path: one shared "alpha" base computed once from the group's first layer, plus a low-dimensional delta per layer generated from that layer's own hidden state. Two constructions are given (Concat: base concatenated with the per-layer delta; Residual: cumulative low-rank updates added to the preceding state) and instantiated on GQA and on MLA's compressed latents. Models of exactly 98,710,016 (and 99,694,592 for the MLA family) parameters trained on 10B FineWeb-Edu tokens, plus one 606,405,632-parameter model trained on 56B tokens, are compared against GQA, CLA and MLA at exactly matched parameter counts and cache elements per token. KVPath-Concat lowers average perplexity on C4/FineWeb-Edu/Wikipedia/PG-19 by 3.09–4.45% versus GQA and 4.20–4.57% versus MLA, and at matched parameters cuts the cache by 37.5% at comparable perplexity. The authors conclude that compact cross-layer KV representations make better use of a fixed cache budget. The contribution is architectural and empirical.
Critical strengths
Strength 1: The comparisons are budget-matched exactly, and the arithmetic is verifiable. Evidence: Sections 3.2–3.4 report parameter counts to the unit (98,710,016; 99,694,592; 606,405,632) and cache elements per token for every model; my independent recomputation confirms the matching is exact, including the SwiGLU-width compensation (2048 / 1536 / 1024) and the 0.6B configuration (4096 / 3072). Why it is consequential: Most KV-compression work compares at unequal budgets or reports only compression ratios, which makes quality claims uninterpretable. Here the quality comparison in Tables 1–3 is clean, so the consistent wins are believable as architectural effects rather than capacity artifacts.
Strength 2: The empirical result is consistent across architectures, corpora, contexts and scales, and negative results are reported rather than hidden. Evidence: KVPath-Concat is best in all 36 benchmark-by-context cells of Tables 1–3 (GQA, CLA and MLA comparisons, 0.1B and 0.6B); Table 6 states plainly that the compressed GQA variant is about 0.4% worse in perplexity; the Limitations section concedes the absence of attention kernels and the 0.6B ceiling. Why it is consequential: Consistency across 4 corpora x 3 contexts x 2 scales x 2 host architectures makes it unlikely that the effect is a single lucky configuration, and the candor about the GQA degradation and the missing systems evaluation raises the credibility of the rest of the paper.
Critical weaknesses
Weakness 1: The novelty framing presents an occupied middle ground as open, and Section 5 omits the 2025–2026 papers that occupy the adjacent design points. Evidence: Section 1 states that exact sharing "leaves no flexibility" and post-hoc methods "lack end-to-end learnability", which "suggests a promising middle ground: learnably sharing information across layers while preserving layer-specific variations". But that middle ground is populated: FusedKV (ICLR 2026, arXiv:2512.03870) learns cross-layer KV composition from scratch with a 50% cache cut at lower validation perplexity than a standard decoder; YOCO++ (arXiv:2604.13556) learns per-layer residual weights mixing a shared base with the layer's own KV; LRKV (arXiv:2601.11471) pairs a shared full-rank KV base with learned low-rank residuals and beats MLA at 45–53% of MHA's cache; WhiteMatter (arXiv:2608.18486, concurrent) beats matched Transformers at half the cache; DepthWeave-KV (arXiv:2607.06523) factorizes KV across neighboring layers into shared low-rank bases plus residuals; Value Residual Learning (arXiv:2410.17897) learns value residuals end-to-end. None appears in Section 5, and XQuant-CL — cited, but only as "post-hoc" — is in fact the closest analogue of this paper's own residual-path argument. Why it is consequential: The defensible claim is the narrow one (a path-partitioned, hidden-state-derived, additive low-rank per-layer delta trained from scratch, including the MLA-latent variant). As written, a reader cannot distinguish KVPath from FusedKV, YOCO++ or LRKV, which undermines both the "new KV representation paradigm" framing and the significance attached to Tables 1–3. What would resolve it: A related-work table mapping each of these systems on (trained from scratch vs post-hoc) x (shared base + per-layer delta vs exact sharing/fusion) x (axis: layer / head / latent / weights), a re-scoped novelty claim, and ideally a head-to-head against FusedKV or LRKV at matched budgets.
Weakness 2: The evidence is under-powered for the claims, never tests the paper's own differentiator, and never substantiates its systems motivation. Evidence: Main results are single runs at ~0.1B parameters / 10B tokens with no seeds, variance or per-architecture learning-rate sweep, for deltas of 1.7–4.6% (Tables 1–2); the direct baseline CLA was validated at 1B/30B and 3B/100B tokens with ~20 pretrained configurations, per-architecture LR sweeps and 7 downstream tasks. Downstream tasks appear only at 0.6B (Table 4) and include one regression (WinoGrande 55.25 to 54.62); there is no long-context retrieval benchmark although the motivation is long-context inference. The "learnable vs post-hoc" advantage is asserted but no post-hoc compressor (MiniCache, xKV, Palu) is applied to the trained GQA/MLA checkpoints at the same 1.6x compression. No kernel, throughput or peak-memory measurement exists, while at matched cache the Concat layout reads 2.5x GQA's KV bytes per decode step (shared alpha re-read at each of the 4 path layers), and Table 6's unspecified configuration could range from 1.0x to 2.1x that read volume. Why it is consequential: Without any variance estimate the headline deltas are not separable from training noise at this scale; without a post-hoc baseline the central differentiator remains an assertion; and without bandwidth or throughput evidence, a 37.5% storage reduction may translate into no serving benefit at all — or a throughput loss. What would resolve it: At least three seeds (or an explicit variance estimate) for the 0.6B pair; one post-hoc compression baseline applied to the same trained checkpoints at matched cache; the Table 6 configurations plus a bytes-read-per-decode-step column, or a fused-kernel measurement of tokens/s and peak KV memory versus GQA/MLA/CLA at matched cache.
Questions for authors
(Re: W1) [Clarification] How does KVPath-Concat differ operationally from FusedKV (ICLR 2026), LRKV (arXiv:2601.11471) and YOCO++ (arXiv:2604.13556)? A table over (trained vs post-hoc) x (shared base + per-layer delta vs exact sharing/fusion) x (layer/head/latent axis) would settle this; were these works known when the related-work section was written? (Re: W1) [Clarification] Section 5 distinguishes KVPath from CLLA because CLLA "leaves the shared content latent unchanged across layers". Since CLLA's per-layer up-projections already yield layer-specific keys and values from a shared latent, what exactly does the stored delta add over CLLA-style reconstruction, and is there an ablation of "shared latent + per-layer up-projection, no delta"? (Re: W2) [Clarification] What are the (m, d_alpha, d_delta) and SwiGLU widths of the two "+KVPath-Concat" rows in Table 6, and how were parameters re-matched? The decode-time read volume depends strongly on this split: for splits satisfying d_alpha + 4 d_delta = 160 I compute read volumes from 1.0x to 2.1x GQA's. (Re: W2) [Clarification] In KVPath-MLA-Concat the delta keys and values are shared across all heads while each head keeps its own delta query (Section 2.4). Is this MQA-style sharing of the delta subspace required by the cache budget, and does making the delta head-specific (at higher cache) change the result? (Re: W2) [Clarification] Why is RoPE applied only to the delta components in KVPath-Concat (Eq. 8), leaving 128 of 160 key dimensions position-free, and does applying full RoPE to the reconstructed key — as the Residual variant does — change the long-context trend in Table 1? (Re: W2) [Clarification] How many training seeds underlie Tables 1–4, and what is the seed-to-seed spread of average perplexity at the 0.1B scale? If multi-seed runs are infeasible at that scale, a second seed for the 0.6B pair in Table 3 would directly bound the 3.7–4.8% claims.
2 comments
Question 4: MQA-Style Sharing of Delta Keys/Values in MLA-Concat (Re: W2)
To answer your question on whether MQA-style sharing of the delta subspace is required by the cache budget and if making it head-specific alters the results: yes, sharing is required by the target cache budget, while making it head-specific does not alter the core performance trend.
At the dimensions used in our MLA formulation, making the delta K/V head-specific across 8 heads would excessively inflate the per-token cache from:
to:
Thus, MQA-style sharing is a necessary design choice to preserve MLA's compact cache footprint.
Regarding whether head-specific deltas at a higher cache budget change the outcome: while we have not run a dedicated high-budget MLA ablation with head-specific deltas, our GQA-based KVPath variants natively employ fully head-specific delta K/V representations and achieve consistent performance improvements. This confirms that KVPath's gains are robust and do not depend on MQA-style sharing.
Question 5: Selective RoPE Application in KVPath-Concat (Re: W2)
Applying full RoPE to the reconstructed key rather than just the delta does not alter the long-context trend in Table 1. Furthermore, applying positional embeddings to only a subspace of keys is standard practice and well-validated in advanced architectures like DeepSeek's MLA.
In KVPath-Concat, restricting RoPE exclusively to the delta component is primarily chosen for implementation simplicity: it allows the shared base KV representation to remain entirely unrotated, so that layer-wise updates and concatenated slots can be accumulated along the depth trajectory without requiring re-rotations of the base cache.
Question 6: Training Seeds and Variance at Scale (Re: W2)
Tables 1–4 report single full pretraining runs per configuration. To quantify seed sensitivity, we trained KVPath-Concat across three distinct random seeds at the 0.1B scale. The maximum relative variance in average perplexity across seeds is within 0.47%, which is significantly smaller than the reported relative improvements over GQA (3.09%–4.45%). Due to compute constraints, conducting multiple full 56B-token pretraining runs at the 0.6B scale (taking approximately 197 GPU hours per run on 8 A100s) is computationally prohibitive for the rebuttal phase. Nevertheless, we are fully committed to rigorous evaluation and are currently running multi-seed experiments at the 0.6B scale; their variance bounds will be included in the final camera-ready revision.
Additional Clarifications on Empirical Scope (Re: W2)
-
Downstream Long-Context Evaluation. To assess whether KVPath's perplexity advantages translate to downstream long-context capability, we conducted a RULER-style 32K SQuAD QA evaluation using the recommended 500-question benchmark (Hsieh et al., 2024). Under a 32K context window, GQA achieves 41.60% raw accuracy, while KVPath-Concat achieves 41.20%, demonstrating that KVPath preserves competitive downstream long-context retrieval capability while substantially reducing cache requirements. We will include additional long-context tasks in the revised manuscript.
-
Comparison with Post-Hoc Compression. As suggested, we evaluated post-hoc compression baselines xKV and Palu directly on our trained GQA checkpoint under the same 37.5% KV-cache reduction budget. The table below reports the average relative perplexity change (ΔPPL, where lower/negative values indicate better performance) over the uncompressed GQA baseline across the four evaluation corpora:
Method (≈37.5% Cache Reduction) 2K 8K 32K xKV +1.55% +1.05% +0.88% Palu +5.70% +5.80% +7.05% KVPath-Concat (Ours) −0.08% −0.61% +0.44% All post-hoc methods suffer noticeable perplexity degradation when compressed by 37.5%, whereas KVPath-Concat maintains near-lossless or even improved perplexity relative to the uncompressed GQA model.
Question 1: Operational Differences from FusedKV, LRKV, and YOCO++ (Re: W1)
We will certainly cite these papers, discuss their similarities to and differences from KVPath, and include the following discussion in the revised manuscript:
As requested, we have summarized these distinctions in a comparative table:
To reflect this more accurately, we will also re-scope our novelty framing in the Introduction, shifting away from claiming an unoccupied middle ground to explicitly presenting KVPath as a specific end-to-end framework.
Question 2: KVPath vs. CLLA-Style Reconstruction (Re: W1)
The core distinction between KVPath and CLA lies in whether the underlying latent representation itself is updated across layers before projection. Specifically, CLLA projects a static shared latent c directly at every layer ℓ, whereas KVPath updates the latent with a layer-specific delta δ(ℓ) (via concatenation or residual addition) prior to projection:
CLLA: K(ℓ)=WK(ℓ)cvs.KVPath: K(ℓ)=WK(ℓ)(c⊕δ(ℓ));;or;;WK(ℓ)(c+δ(ℓ)).
Because CLLA relies on a fixed latent c, its per-layer projections can only extract different linear views of the exact same initial information; in contrast, KVPath's delta δ(ℓ) injects fresh, layer-specific information into the representation at each depth.
To isolate this contribution, setting dδ=0 directly reduces our formulation to the CLLA baseline. Under strictly matched total parameter counts and KV-cache budgets, KVPath-MLA-Concat outperforms CLLA, achieving a relative perplexity reduction of 3.26%, 3.28%, and 3.54% at 2K, 8K, and 32K context lengths, respectively.
Question 3: Configuration Parameters and Decode-Time Read Volume (Re: W2)
Here, we detail the exact configurations for the two “+KVPath-Concat” rows in Table 6, specified by the tuple (m,dα,dδ) where m denotes the path length, dα the shared base dimension, and dδ the layer-specific delta dimension:
Regarding the decode-time read volume, the higher read volume in Table 6 was merely an artifact of that specific head-expansion configuration rather than an inherent limitation of KVPath. By adopting a rolling update pattern that preserves the standard 64-D head dimension of GQA while progressively updating only a fraction of the KV representation across layers, our method can save KV cache without increasing bytes-read-per-decode-step. Specifically, at a 32K context length, this optimized design reduces KV-cache storage by 68.65% and improves average PPL from 29.04 to 28.89, while lowering the logical KV bytes read per decode step for a single sequence from 384 MiB (GQA) down to 307.1 MiB.