acceptodds
Under review as a conference paper at ICLR 2027

KVPath: Connecting Key-Value Representations Across Transformer Layers

Abstract

Transformers typically cache a separate key-value (KV) representation for each token at every layer. We hypothesize that most of a token's KV representation can be reused across neighboring layers, while the change from one layer's KV state to the next can be represented by a low-dimensional delta. Motivated by this hypothesis, we introduce KVPath, a new KV representation paradigm that connects KV states across network depth during pre-training. KVPath organizes these states as a path, with one component remaining unchanged across layers and a low-dimensional component varying by layer. KVPath supports major attention architectures, including grouped-query attention (GQA) and multi-head latent attention (MLA). Experimental results across multiple model scales and context lengths show that KVPath achieves better performance than GQA and cross-layer attention (CLA) at matched model sizes and KV-cache budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, KV Cache Efficiency, Attention Mechanism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oYiEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache memory, which grows with both sequence length and network depth. It argues that existing cross-layer approaches either share KV states exactly (CLA, YOCO), leaving no room for layer-specific variation, or compress already-trained caches post hoc (MiniCache, xKV), without end-to-end learnability. It proposes KVPath, which partitions the L layers into groups of m and treats each token's KV states inside a group as a path: one shared "alpha" base computed once from the group's first layer, plus a low-dimensional delta per layer generated from that layer's own hidden state. Two constructions are given (Concat: base concatenated with the per-layer delta; Residual: cumulative low-rank updates added to the preceding state) and instantiated on GQA and on MLA's compressed latents. Models of exactly 98,710,016 (and 99,694,592 for the MLA family) parameters trained on 10B FineWeb-Edu tokens, plus one 606,405,632-parameter model trained on 56B tokens, are compared against GQA, CLA and MLA at exactly matched parameter counts and cache elements per token. KVPath-Concat lowers average perplexity on C4/FineWeb-Edu/Wikipedia/PG-19 by 3.09–4.45% versus GQA and 4.20–4.57% versus MLA, and at matched parameters cuts the cache by 37.5% at comparable perplexity. The authors conclude that compact cross-layer KV representations make better use of a fixed cache budget. The contribution is architectural and empirical.

Critical strengths

Strength 1: The comparisons are budget-matched exactly, and the arithmetic is verifiable. Evidence: Sections 3.2–3.4 report parameter counts to the unit (98,710,016; 99,694,592; 606,405,632) and cache elements per token for every model; my independent recomputation confirms the matching is exact, including the SwiGLU-width compensation (2048 / 1536 / 1024) and the 0.6B configuration (4096 / 3072). Why it is consequential: Most KV-compression work compares at unequal budgets or reports only compression ratios, which makes quality claims uninterpretable. Here the quality comparison in Tables 1–3 is clean, so the consistent wins are believable as architectural effects rather than capacity artifacts.

Strength 2: The empirical result is consistent across architectures, corpora, contexts and scales, and negative results are reported rather than hidden. Evidence: KVPath-Concat is best in all 36 benchmark-by-context cells of Tables 1–3 (GQA, CLA and MLA comparisons, 0.1B and 0.6B); Table 6 states plainly that the compressed GQA variant is about 0.4% worse in perplexity; the Limitations section concedes the absence of attention kernels and the 0.6B ceiling. Why it is consequential: Consistency across 4 corpora x 3 contexts x 2 scales x 2 host architectures makes it unlikely that the effect is a single lucky configuration, and the candor about the GQA degradation and the missing systems evaluation raises the credibility of the rest of the paper.

Critical weaknesses

Weakness 1: The novelty framing presents an occupied middle ground as open, and Section 5 omits the 2025–2026 papers that occupy the adjacent design points. Evidence: Section 1 states that exact sharing "leaves no flexibility" and post-hoc methods "lack end-to-end learnability", which "suggests a promising middle ground: learnably sharing information across layers while preserving layer-specific variations". But that middle ground is populated: FusedKV (ICLR 2026, arXiv:2512.03870) learns cross-layer KV composition from scratch with a 50% cache cut at lower validation perplexity than a standard decoder; YOCO++ (arXiv:2604.13556) learns per-layer residual weights mixing a shared base with the layer's own KV; LRKV (arXiv:2601.11471) pairs a shared full-rank KV base with learned low-rank residuals and beats MLA at 45–53% of MHA's cache; WhiteMatter (arXiv:2608.18486, concurrent) beats matched Transformers at half the cache; DepthWeave-KV (arXiv:2607.06523) factorizes KV across neighboring layers into shared low-rank bases plus residuals; Value Residual Learning (arXiv:2410.17897) learns value residuals end-to-end. None appears in Section 5, and XQuant-CL — cited, but only as "post-hoc" — is in fact the closest analogue of this paper's own residual-path argument. Why it is consequential: The defensible claim is the narrow one (a path-partitioned, hidden-state-derived, additive low-rank per-layer delta trained from scratch, including the MLA-latent variant). As written, a reader cannot distinguish KVPath from FusedKV, YOCO++ or LRKV, which undermines both the "new KV representation paradigm" framing and the significance attached to Tables 1–3. What would resolve it: A related-work table mapping each of these systems on (trained from scratch vs post-hoc) x (shared base + per-layer delta vs exact sharing/fusion) x (axis: layer / head / latent / weights), a re-scoped novelty claim, and ideally a head-to-head against FusedKV or LRKV at matched budgets.

Weakness 2: The evidence is under-powered for the claims, never tests the paper's own differentiator, and never substantiates its systems motivation. Evidence: Main results are single runs at ~0.1B parameters / 10B tokens with no seeds, variance or per-architecture learning-rate sweep, for deltas of 1.7–4.6% (Tables 1–2); the direct baseline CLA was validated at 1B/30B and 3B/100B tokens with ~20 pretrained configurations, per-architecture LR sweeps and 7 downstream tasks. Downstream tasks appear only at 0.6B (Table 4) and include one regression (WinoGrande 55.25 to 54.62); there is no long-context retrieval benchmark although the motivation is long-context inference. The "learnable vs post-hoc" advantage is asserted but no post-hoc compressor (MiniCache, xKV, Palu) is applied to the trained GQA/MLA checkpoints at the same 1.6x compression. No kernel, throughput or peak-memory measurement exists, while at matched cache the Concat layout reads 2.5x GQA's KV bytes per decode step (shared alpha re-read at each of the 4 path layers), and Table 6's unspecified configuration could range from 1.0x to 2.1x that read volume. Why it is consequential: Without any variance estimate the headline deltas are not separable from training noise at this scale; without a post-hoc baseline the central differentiator remains an assertion; and without bandwidth or throughput evidence, a 37.5% storage reduction may translate into no serving benefit at all — or a throughput loss. What would resolve it: At least three seeds (or an explicit variance estimate) for the 0.6B pair; one post-hoc compression baseline applied to the same trained checkpoints at matched cache; the Table 6 configurations plus a bytes-read-per-decode-step column, or a fused-kernel measurement of tokens/s and peak KV memory versus GQA/MLA/CLA at matched cache.

Questions for authors

(Re: W1) [Clarification] How does KVPath-Concat differ operationally from FusedKV (ICLR 2026), LRKV (arXiv:2601.11471) and YOCO++ (arXiv:2604.13556)? A table over (trained vs post-hoc) x (shared base + per-layer delta vs exact sharing/fusion) x (layer/head/latent axis) would settle this; were these works known when the related-work section was written? (Re: W1) [Clarification] Section 5 distinguishes KVPath from CLLA because CLLA "leaves the shared content latent unchanged across layers". Since CLLA's per-layer up-projections already yield layer-specific keys and values from a shared latent, what exactly does the stored delta add over CLLA-style reconstruction, and is there an ablation of "shared latent + per-layer up-projection, no delta"? (Re: W2) [Clarification] What are the (m, d_alpha, d_delta) and SwiGLU widths of the two "+KVPath-Concat" rows in Table 6, and how were parameters re-matched? The decode-time read volume depends strongly on this split: for splits satisfying d_alpha + 4 d_delta = 160 I compute read volumes from 1.0x to 2.1x GQA's. (Re: W2) [Clarification] In KVPath-MLA-Concat the delta keys and values are shared across all heads while each head keeps its own delta query (Section 2.4). Is this MQA-style sharing of the delta subspace required by the cache budget, and does making the delta head-specific (at higher cache) change the result? (Re: W2) [Clarification] Why is RoPE applied only to the delta components in KVPath-Concat (Eq. 8), leaving 128 of 160 key dimensions position-free, and does applying full RoPE to the reconstructed key — as the Residual variant does — change the long-context trend in Table 1? (Re: W2) [Clarification] How many training seeds underlie Tables 1–4, and what is the seed-to-seed spread of average perplexity at the 0.1B scale? If multi-seed runs are infeasible at that scale, a second seed for the 0.6B pair in Table 3 would directly bound the 3.7–4.8% claims.

2 comments

Question 1: Operational Differences from FusedKV, LRKV, and YOCO++ (Re: W1)

We will certainly cite these papers, discuss their similarities to and differences from KVPath, and include the following discussion in the revised manuscript:

  • FusedKV reconstructs upper-layer KV states by fusing fixed lower-layer sources.
  • YOCO++ combines bottom-layer KV with each lower-half layer's freshly generated KV.
  • LRKV uses low-rank residuals to specialize parallel heads within each layer. In contrast, KVPath introduces sequential updates across layers.

As requested, we have summarized these distinctions in a comparative table:

MethodJointly TrainedShared BaseDynamic DeltaSpecialization AxisCached Representation
FusedKVYesNoNoLayerFull lower-layer KV
YOCO++YesYesNoLayerLower-half full KV (upper shared)
LRKVYesYesNoHeadShared base + compact head-specific residuals
KVPath-ConcatYesYesYesLayer (sequential)Initial state + compact concatenated slots
KVPath-ResidualYesYesYesLayer (sequential)Initial state + compact cumulative updates

To reflect this more accurately, we will also re-scope our novelty framing in the Introduction, shifting away from claiming an unoccupied middle ground to explicitly presenting KVPath as a specific end-to-end framework.

Question 2: KVPath vs. CLLA-Style Reconstruction (Re: W1)

The core distinction between KVPath and CLA lies in whether the underlying latent representation itself is updated across layers before projection. Specifically, CLLA projects a static shared latent cc directly at every layer ℓ\ell, whereas KVPath updates the latent with a layer-specific delta δ(ℓ)\delta^{(\ell)} (via concatenation or residual addition) prior to projection:

CLLA: K(ℓ)=WK(ℓ)cvs.KVPath: K(ℓ)=WK(ℓ)(c⊕δ(ℓ));;or;;WK(ℓ)(c+δ(ℓ)).\text{CLLA: } K^{(\ell)} = W_K^{(\ell)} c \qquad \text{vs.} \qquad \text{KVPath: } K^{(\ell)} = W_K^{(\ell)} \big(c \oplus \delta^{(\ell)}\big) \\;\\; \text{or} \\;\\; W_K^{(\ell)} \big(c + \delta^{(\ell)}\big).

Because CLLA relies on a fixed latent cc, its per-layer projections can only extract different linear views of the exact same initial information; in contrast, KVPath's delta δ(ℓ)\delta^{(\ell)} injects fresh, layer-specific information into the representation at each depth.

To isolate this contribution, setting dδ=0d_\delta = 0 directly reduces our formulation to the CLLA baseline. Under strictly matched total parameter counts and KV-cache budgets, KVPath-MLA-Concat outperforms CLLA, achieving a relative perplexity reduction of 3.26%, 3.28%, and 3.54% at 2K, 8K, and 32K context lengths, respectively.

Question 3: Configuration Parameters and Decode-Time Read Volume (Re: W2)

Here, we detail the exact configurations for the two “+KVPath-Concat” rows in Table 6, specified by the tuple (m,dα,dδ)(m, d_\alpha, d_\delta) where mm denotes the path length, dαd_\alpha the shared base dimension, and dδd_\delta the layer-specific delta dimension:

  • GQA+KVPath-Concat: (m,dα,dδ)=(4,80,20)(m, d_\alpha, d_\delta) = (4, 80, 20), yielding a 100-D attention head with a SwiGLU width of 1,920.
  • MLA+KVPath-Concat: (m,dα,dδ)=(4,256,18)(m, d_\alpha, d_\delta) = (4, 256, 18) in the compressed latent space. The 256-D shared alpha latent is reconstructed into 62-D content K/V and concatenated with the 18-D layer-specific component to form an 80-D attention head. Its SwiGLU widths are 1,988 for layers 1–11 and 1,987 for layer 12.
  • Parameter re-matching: In both rows, parameter counts were strictly matched by adjusting SwiGLU intermediate widths to offset changes in attention parameters.

Regarding the decode-time read volume, the higher read volume in Table 6 was merely an artifact of that specific head-expansion configuration rather than an inherent limitation of KVPath. By adopting a rolling update pattern that preserves the standard 64-D head dimension of GQA while progressively updating only a fraction of the KV representation across layers, our method can save KV cache without increasing bytes-read-per-decode-step. Specifically, at a 32K context length, this optimized design reduces KV-cache storage by 68.65% and improves average PPL from 29.04 to 28.89, while lowering the logical KV bytes read per decode step for a single sequence from 384 MiB (GQA) down to 307.1 MiB.

Question 4: MQA-Style Sharing of Delta Keys/Values in MLA-Concat (Re: W2)

To answer your question on whether MQA-style sharing of the delta subspace is required by the cache budget and if making it head-specific alters the results: yes, sharing is required by the target cache budget, while making it head-specific does not alter the core performance trend.

At the dimensions used in our MLA formulation, making the delta K/V head-specific across 8 heads would excessively inflate the per-token cache from:

Cache=3×384+12×(32+32)=1,920 elements/token\text{Cache} = 3 \times 384 + 12 \times (32 + 32) = 1{,}920 \text{ elements/token}

to:

Cache=3×384+12×8×(32+32)=7,296 elements/token\text{Cache} = 3 \times 384 + 12 \times 8 \times (32 + 32) = 7{,}296 \text{ elements/token}

Thus, MQA-style sharing is a necessary design choice to preserve MLA's compact cache footprint.

Regarding whether head-specific deltas at a higher cache budget change the outcome: while we have not run a dedicated high-budget MLA ablation with head-specific deltas, our GQA-based KVPath variants natively employ fully head-specific delta K/V representations and achieve consistent performance improvements. This confirms that KVPath's gains are robust and do not depend on MQA-style sharing.

Question 5: Selective RoPE Application in KVPath-Concat (Re: W2)

Applying full RoPE to the reconstructed key rather than just the delta does not alter the long-context trend in Table 1. Furthermore, applying positional embeddings to only a subspace of keys is standard practice and well-validated in advanced architectures like DeepSeek's MLA.

In KVPath-Concat, restricting RoPE exclusively to the delta component is primarily chosen for implementation simplicity: it allows the shared base KV representation to remain entirely unrotated, so that layer-wise updates and concatenated slots can be accumulated along the depth trajectory without requiring re-rotations of the base cache.

Question 6: Training Seeds and Variance at Scale (Re: W2)

Tables 1–4 report single full pretraining runs per configuration. To quantify seed sensitivity, we trained KVPath-Concat across three distinct random seeds at the 0.1B scale. The maximum relative variance in average perplexity across seeds is within 0.47%, which is significantly smaller than the reported relative improvements over GQA (3.09%–4.45%). Due to compute constraints, conducting multiple full 56B-token pretraining runs at the 0.6B scale (taking approximately 197 GPU hours per run on 8 A100s) is computationally prohibitive for the rebuttal phase. Nevertheless, we are fully committed to rigorous evaluation and are currently running multi-seed experiments at the 0.6B scale; their variance bounds will be included in the final camera-ready revision.

Additional Clarifications on Empirical Scope (Re: W2)

  • Downstream Long-Context Evaluation. To assess whether KVPath's perplexity advantages translate to downstream long-context capability, we conducted a RULER-style 32K SQuAD QA evaluation using the recommended 500-question benchmark (Hsieh et al., 2024). Under a 32K context window, GQA achieves 41.60% raw accuracy, while KVPath-Concat achieves 41.20%, demonstrating that KVPath preserves competitive downstream long-context retrieval capability while substantially reducing cache requirements. We will include additional long-context tasks in the revised manuscript.

  • Comparison with Post-Hoc Compression. As suggested, we evaluated post-hoc compression baselines xKV and Palu directly on our trained GQA checkpoint under the same 37.5% KV-cache reduction budget. The table below reports the average relative perplexity change (ΔPPL, where lower/negative values indicate better performance) over the uncompressed GQA baseline across the four evaluation corpora:

    Method (≈37.5% Cache Reduction)2K8K32K
    xKV+1.55%+1.05%+0.88%
    Palu+5.70%+5.80%+7.05%
    KVPath-Concat (Ours)−0.08%−0.61%+0.44%

    All post-hoc methods suffer noticeable perplexity degradation when compressed by 37.5%, whereas KVPath-Concat maintains near-lossless or even improved perplexity relative to the uncompressed GQA model.

Reviewer_nP7KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors hypothesize that KV representations across consecutive layers contain shared information, while layer-specific variation can be captured by low-dimensional vectors. Hence, they propose KVPATH, which reduces Transformer's KV cache complexity by sharing part of the KV representations across a group of consecutive layers, and also allowing for layer-wise expressiveness via a layer-dependent compact vector. They show the effectiveness of KVPATH by augmenting GQA and MLA with it, and show that it outperforms standard GQA and MLA respectively under matched KV-cache and model-parameter budgets.

Critical strengths
  1. The motivation is clear and the idea of sharing part of the KV representation across neighboring layers seems natural to me.
  2. The perplexity performance looks good within the experiments considered in the paper, empirically verifying that it's a simple and effective method.
  3. The manuscript is clear and easy to follow.
Critical weaknesses
  1. Personally, I feel this work would have more impact if KVPATH can be shown to be effective also in finetuning. The uncertainty of doing a costly full pretraining for a new architecture seems to be pretty high, and the experiments in the paper only scale to 0.6B, which makes it hard to predict how good it really is for pretraining. Would it be possible to use KVPATH to finetune a pretrained checkpoint (I guess there will be some workaround given its similarity with LoRA)?
  2. Why downstream tasks performance are only evaluated at 2K context when perplexity is evaluated up to 32K context?
  3. Why is CLA omitted from the 0.6B scaling study?
Questions for authors

See weakness. Happy to adjust my score if point 2 and 3 in weakness are properly addressed in rebuttal.


After reading the authors' rebuttal, I decided to raise my score from 2 to 3.

2 comments

We thank the reviewer for their helpful feedback.

1. Scaling Behavior and Applicability to Pretrained Checkpoints

First, regarding scalability: under strictly matched parameter counts and KV-cache budgets, KVPath-Concat's relative perplexity reduction over GQA consistently widens as both model size and context length grow. Specifically, scaling from the 100M model to the 600M model increases the average relative perplexity reduction from 3.09%, 3.65%, and 4.45% to 3.73%, 4.22%, and 4.81% across 2K, 8K, and 32K context lengths, respectively. While our current pretraining scale is limited by academic compute constraints, KVPath's gains consistently widen at larger model scales and longer context lengths; thus, this clear upward trend gives us confidence in the architecture's favorable behavior at scale.

Second, regarding fine-tuning existing pretrained checkpoints: structurally, KVPath is designed as an architecture trained from scratch to jointly optimize cross-layer shared bases and their sequential delta updates. In a standard pretrained model, KV projections are trained independently per layer without cross-layer alignment. Consequently, retrofitting KVPath onto an already-pretrained checkpoint—such as freezing existing weights and using lightweight adapters like LoRA to bridge independently learned KV spaces—is fundamentally different from end-to-end pretraining. as lightweight fine-tuning is generally insufficient to reorganize deeply entangled cross-layer attention representations.

We thank the reviewer for pointing out this practical angle; exploring low-cost retrofitting or continual pretraining to convert existing pretrained models into KVPath architectures is indeed an interesting direction that we will explicitly discuss in the revised manuscript.

2. Context Length of Downstream Evaluations

The standard zero-shot benchmarks reported in our scaling study (such as MMLU, HellaSwag, and ARC) evaluate reasoning and knowledge retrieval on short-to-medium inputs (typically under 2K tokens), which is why a 2K context limit was originally chosen.

To better assess whether KVPath's long-context perplexity advantages translate to downstream long-context capability, we additionally conducted a RULER-style 32K SQuAD QA evaluation using the recommended evaluation size of 500 questions (Hsieh et al., 2024). Under a 32K context window, GQA achieves 41.60% raw accuracy, while KVPath-Concat achieves 41.20%. This demonstrates that KVPath successfully maintains competitive downstream long-context performance while reducing cache requirements. We will experiment a few more long context tasks and include them in our revision.

3. CLA in the 0.6B Scaling Study

To further validate our findings against hard cross-layer sharing baselines at scale, we fully trained and evaluated a CLA (Cross-Layer Attention) baseline at the 0.6B scale after 56B training tokens.

CLA was strictly matched with GQA and KVPath-Concat in both parameter count and KV-cache size, sharing four consecutive layers. CLA enforces strict, un-updated sharing of the entire KV state. Under identical training protocols and budget constraints, KVPath-Concat achieves lower average perplexity by 4.18% at 2K, 4.67% at 8K, and 5.62% at 32K relative to CLA. This confirms that allocating a fraction of the cache budget to low-dimensional delta vectors is critical for preserving representation capacity compared to hard sharing mechanisms like CLA.

Thanks for the detailed rebuttal and my concerns are mostly addressed. I have raised my score to 3. Please make sure including these additional results in the revision.

Reviewer_Vd45OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVPath, which constructs cross-layer KV representations using a shared base and compact layer-specific deltas. It introduces Concat and Residual, and applies them to GQA and MLA architectures. Experiments under matched model-parameter and KV-cache budgets demonstrate improved performance on multiple language modeling benchmarks, with additional evaluations across different model scales and parameter configurations.

Critical strengths

The paper addresses the practically important problem of KV-cache compression. By combining a shared base representation with compact layer-specific deltas, the paper aims to reduce cross-layer redundancy while preserving layer-specific variations. The experiments cover different context lengths, model scales, and parameter configurations. Under matched model-parameter and KV-cache budgets, the reported results outperform the compared baselines on multiple benchmarks.

Critical weaknesses

(1) The proposed framework constructs cross-layer KV representations by combining a shared base with layer-specific deltas. However, cross-layer sharing and layer-specific reconstruction have already been explored in prior work, such as CLLA. Although the authors explicitly distinguish their approach by introducing compact delta components, the overall method largely extends existing sharing mechanisms. The paper draws inspiration from residual updates in Transformer hidden states and hypothesizes that KV representations in adjacent layers share substantial information. However, the residual structure of hidden states does not directly establish similar properties in KV representations. The paper lacks a direct analysis of cross-layer KV redundancy and low-dimensional variation. (2) For example, page 4 contains the redundant expressions "Eq. equation 7" and "Eq. equation 8", while "both Concat and Residual variant" should use the plural form "variants". The paper lacks direct comparisons with closely related methods such as CLLA, and the existing ablation studies do not adequately isolate the contribution of the proposed delta components from that of existing cross-layer sharing mechanisms.

Questions for authors

see weakness

3 comments

We thank the reviewer for their thoughtful feedback and constructive comments.

1. Direct Evidence for Cross-Layer Redundancy

Our hypothesis is supported by both prior empirical literature and direct statistical analysis:

  • Quantifying Cross-Layer Similarity via CKA. Centered Kernel Alignment (CKA) is a metric ranging from 0 to 1 that quantifies representation similarity across neural network layers, with higher values indicating greater similarity. Prior work such as xKV (Chang et al., 2025, Sections 3.2–3.3 and Figure 2) utilized CKA to demonstrate significant cross-layer singular-vector alignment and low-rank structural properties in multi-layer KV caches for GQA architectures. Building on this analytical framework, we performed linear CKA on both GQA and MLA baselines. The measurements yield adjacent-layer key/value CKA scores of 0.798 / 0.692 for GQA and 0.575 / 0.576 for MLA. These high similarity scores confirm substantial representational redundancy across Transformer layers. However, CKA values substantially below 1 also reveal meaningful inter-layer differences that CLLA fails to capture, which could lead to worse performance.

  • Spectral Concentration of Layer-to-Layer Deltas. To investigate whether cross-layer difference can be captured by low-dimensional representations, we extract adjacent-layer difference matrices ΔK=K(ℓ+1)−K(ℓ)\Delta K = K^{(\ell+1)} - K^{(\ell)} and ΔV=V(ℓ+1)−V(ℓ)\Delta V = V^{(\ell+1)} - V^{(\ell)} from trained GQA and MLA checkpoints and perform Singular Value Decomposition (SVD). Concatenating head features yields total head dimensions of 256256 (4×644 \times 64) for GQA and 512512 (8×648 \times 64) for MLA.

    The table below reports the mean number of principal components (PCs) required to capture 90% of the variance in adjacent-layer updates:

    Baseline ArchitectureTotal Head DimensionPCs for 90% Var (ΔK\Delta K)PCs for 90% Var (ΔV\Delta V)
    GQA256256 (4×644 \times 64)70.9 (27.7% of space)89.9 (35.1% of space)
    MLA512512 (8×648 \times 64)67.5 (13.2% of space)72.1 (14.1% of space)

    These empirical results confirm that layer-to-layer KV update concentrate in a remarkably small subspace, validating our design of modeling updates with low-dimensional delta vectors.

2. CLLA and Additional Ablations

CLLA can be viewed as a zero-delta, shared-only ablation of our design. We directly evaluated CLLA against KVPath-MLA-Concat under strictly matched parameter counts and KV-cache budgets. KVPath-MLA-Concat consistently outperforms CLLA, achieving a relative perplexity reduction of 3.26%, 3.28%, and 3.54% at 2K, 8K, and 32K context lengths, respectively. This suggests that, under a fixed cache capacity, retaining a small update can be critical to model performance.

Thank you for the CKA and SVD analyses and the CLLA comparison, which support the redundancy hypothesis; however, the CLLA comparison covers only perplexity for the MLA-Concat variant, and the contribution beyond existing cross-layer sharing remains incremental, so I will keep my score.

Thank you for acknowledging our analyses. We would like to clarify the conceptual contribution of KVPath. Cross-layer sharing focuses on reusing representations across layers; KVPath focuses on representing layer-specific changes through compact delta updates. The shared base captures common information, while the compact updates preserve variations across layers. Our spectral analysis directly supports this formulation: adjacent-layer KV differences concentrate in a low-dimensional subspace.

We have also extended the CLLA comparison beyond perplexity to downstream benchmarks: HellaSwag, PIQA, ARC-Easy, SciQ, LAMBADA, and OpenBookQA, covering both KVPath-MLA-Concat and KVPath-MLA-Residual. Under zero-shot evaluation, the six-task average scores are 43.42 for MLA, 43.15 for CLLA, 44.87 for KVPath-MLA-Concat, and 44.84 for KVPath-MLA-Residual. All four models have approximately 99.69M parameters and identical KV-cache budgets. Together, these analyses and experiments support the central contribution of KVPath: compactly modeling cross-layer variation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.