acceptodds
Under review as a conference paper at ICLR 2027

Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction

Abstract

The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query attention (GQA) reduces this cost by sharing key–value heads, but still stores both a key and a value at every step. We introduce Grouped Value Attention (GVA), which stores grouped values and reconstructs content keys with a learned linear map. At inference, the map can be absorbed into the query, eliminating the need to materialize content keys in the intended decode path. A small shared decoupled RoPE channel retains positional information through a separately cached positional key. For the configurations studied, this representation reduces persistent cache scalars by approximately 45–47% relative to matched GQA. At the 350M-parameter scale with 30B FineWeb-Edu tokens, the 16-dimensional positional variant reaches 44.35 average accuracy across five tasks, compared with 44.36 for GQA and 43.88 for MLA. These results demonstrate near-GQA benchmark accuracy with a more compact cache representation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache, Efficient Transformers, Memory-Efficient Inference, Large Language Models, RoPE, Autoregressive Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ogLBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduced grouped on value in softmax attention and named it Groupped Value Attention (GVA), instead grouped on queries in GQA. Experiments show that GVA achieves near GQA accuracy on benchmarks with more compact cache representation.

Critical strengths
  • The technique sounds OK, especially the derivation on the decoupled RoPE process.
  • The writing is generally easy to follow.
Critical weaknesses
  • It only compared the cache size to GQA, but does not compared to MLA.
  • It lacks of the number report on efficiency, latency, peak memory, throughput, etc, which makes the whole paper not-throughly verified.
  • It mixed using different quantitive description "the cache is exactly half" (L255)/"roughly halves" (L262)/"approximately 45–47%" (L061). It is better making it more clear.
  • What DRoPE stands for? Decoupled RoPE? Should add citation to the RoPE paper.
  • There are wrong references, "Section 6" in L065, while Section 6 is about limitation, has nothing to do with the "prefill cost".
Questions for authors

see comments on "weakness"

2 comments

I'm sharing you the results of GQA vs GVA vs MLA which has all the metrics requested.

Measured systems performance.

We measured GQA, GVA and MLA on one NVIDIA H100 NVL (94 GB HBM3, 3.9 TB/s peak), bf16, [PyTorch / Triton / FlashAttention / FlashMLA versions].

  • GQA uses FlashAttention-2.
  • GVA uses our Triton prefill and decode kernels.
  • MLA uses FlashAttention-2 for prefill (expanded form, as in vLLM) and FlashMLA (commit b31bfe7) for decode (absorbed form).

Model. Llama-3.2-1B body (16 layers, d_model 2048, MLP 8192, vocab 128k) with 16 query heads in every variant:

  • GQA: 4 KV heads × 128
  • GVA: 4 value groups × 128 + d_r = 16
  • MLA: DeepSeek-V2-Lite (latent 512 + RoPE 64)

KV cache per token per layer: GQA 2048 B, GVA 1056 B (−48.4%), MLA 1152 B (−43.8%). Each request has a 1024-token prompt, then N tokens are decoded. Decode steps run as CUDA graphs. Peak memory is the maximum allocated by PyTorch, including weights, the full KV cache, activations and temporary kernel buffers.

Whole model, batch 32

Context (N)Tokens/s GQA / GVA / MLAGVA vs GQAGVA vs MLAPeak decode memory, GiB (GQA / GVA / MLA)
1k10,773 / 12,473 / 9,7461.16x1.28x4.4 / 3.7 / 3.6
16k4,793 / 7,814 / 6,3701.63x1.23x19.4 / 11.4 / 12.0
32k2,979 / 5,310 / 4,4781.78x1.19x35.4 / 19.7 / 21.0
64k1,680 / 3,241 / 2,7701.93x1.17x67.4 / 36.2 / 39.1
128kOOM / 1,821 / 1,534—1.19xOOM / 69.2 / 75.1

Only Attention Layer: Below are the numbers for only attention layer with 1024 prefill then decoding till context length:

AttentionBatch 32, context NDecode latency (ms)TTFT (ms)Prefill peak mem (MB)Decode ms/tokenDecode tokens/sDecode peak mem (MB)KV cache (MB)TTFT + decode (ms)Speedup (GVA vs GQA)
GQA16k (16384)5468.233.541788.500.33495879.01277.401088.005471.76
GVA16k (16384)3995.642.241183.600.244131214.9751.60561.003997.881.37x
MLA16k (16384)4294.285.002250.500.262122089.7809.00612.004299.54
GQA32k (32768)18527.543.572820.500.56556595.52309.402112.0018531.12
GVA32k (32768)10324.532.261712.600.315101561.61280.601089.0010326.801.79x
MLA32k (32768)12129.405.002831.000.37086449.21390.201188.0012134.57
GQA64k (65536)68313.693.494884.501.04230698.94373.404160.0068317.18
GVA64k (65536)34311.322.252770.600.52461121.32338.602145.0034313.571.99x
MLA64k (65536)40746.055.203991.000.62251468.82550.002340.0040751.26
GQA128k (131072)275064.663.519012.502.09915248.48501.408256.00275068.17
GVA128k (131072)131964.622.274886.601.00731783.64454.604257.00131966.892.08x
MLA128k (131072)158027.605.206311.501.20626541.64870.604644.00158032.83
GQA256k (262144)1092234.243.5217268.504.1677680.216757.4016448.001092237.77
GVA256k (262144)512649.672.279118.601.95616363.28686.608481.00512651.942.13x
MLA256k (262144)637921.995.4010951.002.43413149.09510.009252.00637927.40
GQA512k (524288)4354131.773.5433780.508.3053853.233269.4032832.004354135.32
GVA512k (524288)2026983.062.3017582.603.8668276.917150.6016929.002026985.362.15x
MLA512k (524288)2579519.675.3820233.004.9206504.018792.2018468.002579525.05
GQA1M (1048576)17418708.863.5466804.5016.6121926.366293.4065600.0017418712.40
GVA1M (1048576)8049721.082.3034510.607.6774168.434078.6033825.008049723.392.16x
MLA1M (1048576)————————
MLA at 1M was not run because of compute budget.
  1. DRoPE stands for Decoupled RoPE. Yes, we will add citation to the RoPE paper. Thank you for pointing it out.

  2. Thank you for pointing the wrong references. We will update the paper.

We have also submitted a zip file containing code for Grouped Value Attention that includes our prefill and decode kernels in Triton.

Reviewer_AHbdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the KV-cache cost of autoregressive Transformers through Grouped Value Attention (GVA). GVA retains grouped values as in GQA but reconstructs each query head's content keys from its group's values using a learned linear map, K_h = V_g(h) M_h, rather than independently projecting and caching content keys. At decoding time, the reconstruction map is absorbed into the query side, avoiding materialization of historical content keys. A shared positional key with decoupled RoPE separates positional interactions from the absorbable content path. The intended persistent cache decreases from 2TGd_h scalars for GQA to TGd_h + Td_r, corresponding to approximately 45–47% fewer cache scalars in the reported configurations. The paper also studies reconstruction-map initialization and query normalization. In comparisons at approximately 350M parameters trained on 30B FineWeb-Edu tokens, GVA with DRoPE (d_r = 16) achieves 44.35% average accuracy across five zero-shot tasks, averaged over three runs, compared with 44.36% for GQA. Actual decoding throughput and peak serving memory are not reported.

Critical strengths

The design is simple, the cache accounting and decoding rewrite are clear, and the experiments provide initial evidence of quality retention. Equation (7), Algorithm 1, and Equation (12) explain the absorbed content-key computation and account for the shared positional cache. Table 2 suggests that this compact representation can retain average task performance close to GQA in the small-model setting examined. This provides a concrete starting point for further study of value-sharing cache designs.

Critical weaknesses
  1. The novelty of the method and the need for a separate key-reconstruction map remain unclear. Query-side absorption and decoupled RoPE are already components of MLA as described in the paper; GVA's main change is to use the grouped values themselves as the source of content keys. In the linear content path of the final DRoPE variant, W_Q,h^nope M_h^T can be combined into a single query projection. With the same positional branch and score scaling, directly setting content K = V and using this composite query projection produces the same content attention scores. The authors already use absorption at decoding time; what remains unclear is whether retaining a separate key-reconstruction map during training offers any additional benefit over directly parameterizing the query projection. Both implementations can use only values for the content cache while retaining the shared positional cache.

    The existing Shared KV control uses standard RoPE, whereas the proposed efficient decoding design uses DRoPE. The training interval shown in Figure 3 also does not establish a comparison over the same full training budget. A controlled comparison with matched positional design, initialization, and training conditions, together with explicit parameter and computation budgets, is needed to determine whether the reconstruction map offers practical optimization or quality benefits. The current results do not adequately establish the independent contribution of this parameterization. (Pages 3–7.)

  2. The experimental scale, completeness, and systems validation are insufficient to establish the main practical value. The main GVA comparisons cover only approximately 350M parameters, one 30B-token data setting, and five zero-shot tasks. They do not establish quality or training stability at larger scales, under different compression configurations, or in long-context use. In particular, the narrow shared positional branch is an important part of the proposed decoding path, but the paper lacks retrieval or comprehension tasks at longer contexts that would test its effect on the intended applications. Controlled attention conversion and fine-tuning experiments across multiple pretrained LLM sizes would provide useful evidence of practical applicability.

    More directly, the systems evaluation is incomplete. The reported 45–47% reduction counts scalars in the intended persistent cache; the paper does not report actual KV-cache bytes, total peak GPU memory, prefill latency, per-token decoding latency, throughput, or batch capacity under a fixed memory budget. Whether the smaller cache translates into practical gains depends on implementation layout, the positional branch, read reuse, and computation overhead. Since efficient decoding is the main motivation, the absence of these measurements leaves the central practical advantage unverified.

    In addition, the actual head and group counts, content/positional width allocation, MLA latent width, and complete training configurations are not sufficiently specified. The three runs are summarized by means without measures of variability. These omissions limit assessment of parameter and computation fairness and of how reliably GVA matches GQA. Although the authors acknowledge some of these limitations, the current evidence remains insufficient to establish the quality–efficiency trade-off. (Pages 6–8.)

Questions for authors
  1. Contribution and necessary control: In the final DRoPE architecture, what practical benefit does retaining a separate key-reconstruction map during training offer over directly sharing content K/V and directly parameterizing the query projection? Could the direct-query variant be initialized with the product of GVA's initial query and reconstruction matrices, and then compared under the same positional branch, score scaling, and training budget, with actual parameter counts and computation costs reported? This control would directly affect my assessment of the contribution.

  2. Systems benefits: Could GQA, MLA, and GVA be compared on the same hardware, at the same precision and under comparable optimization conditions, reporting actual cache usage, total peak GPU memory, prefill and decoding latency, throughput, and maximum batch capacity under a fixed memory budget, across context lengths and batch sizes? Quality should be reported for the same configurations to establish whether there is an attractive quality–efficiency trade-off.

  3. Scale and applicability: Are larger-scale GVA results available? A useful extension would replace the relevant attention modules in pretrained LLMs at multiple parameter scales and examine conversion loss, quality recovery, training stability, and systems gains under explicit, comparable fine-tuning budgets. If conversion and fine-tuning are not applicable, please explain why and provide other evidence supporting scalability.

  4. Completeness and long-context evaluation: Could the paper provide complete architectures, parameter counts, training context lengths and optimization settings, DRoPE width allocation, and per-seed results, together with retrieval or comprehension tasks at longer contexts supported by the models? These details are important for assessing comparison fairness and whether the shared positional branch preserves the capabilities needed by the intended applications.

3 comments

I would like to thank the author for the proposed method and their contributions; however, since no rebuttal was provided to address my concerns and the paper has not been further refined, I am maintaining my original score.

Q1. Separate map M vs a direct query projection. We thank the reviewer for this precise observation and agree with it.

In the final DRoPE variant without query normalization, W_Q,h^nope M_hᵀ can be merged into one query projection. Content K = V with this merged query then gives exactly the same attention scores, the same cache (grouped values plus the shared positional key) and the same decode cost. Our decode kernel in fact computes exactly this merged form. At inference, the two models are the same.

Q2. Systems benefits. One H100 NVL (94 GB, 3.9 TB/s), bf16. GQA: FlashAttention-2; GVA: our Triton prefill/decode kernels; MLA: FlashAttention-2 prefill (expanded, as in vLLM) + FlashMLA decode (absorbed). Llama-3.2-1B body (16 layers, d_model 2048, MLP 8192, vocab 128k), 16 query heads; GQA 4 KV heads × 128; GVA 4 groups × 128 + d_r = 16; MLA DeepSeek-V2-Lite (latent 512 + RoPE 64). 1024-token prompt, then N tokens decoded; decode as CUDA graphs. Peak = max memory allocated by PyTorch (weights, full KV cache, activations, kernel buffers).

Whole model, batch 32:

NTokens/s GQA / GVA / MLAGVA vs GQAGVA vs MLAKV cache GiBPeak GiB
1k10773 / 12473 / 97461.16x1.28x2.0 / 1.0 / 1.14.4 / 3.7 / 3.6
16k4793 / 7814 / 63701.63x1.23x17.0 / 8.8 / 9.619.4 / 11.4 / 12.0
32k2979 / 5310 / 44781.78x1.19x33.0 / 17.0 / 18.635.4 / 19.7 / 21.0
64k1680 / 3241 / 27701.93x1.17x65.0 / 33.5 / 36.667.4 / 36.2 / 39.1
128kOOM / 1821 / 1534–1.19x129.0 / 66.5 / 72.6OOM / 69.2 / 75.1
  • Batch capacity: at 128k, GQA cannot serve batch 32 on 94 GB (its KV cache alone is 129 GiB); GVA fits in 69 GiB.
  • Batch 1 (to 512k): GVA is 1.10x (1k) to 2.07x (512k) faster than GQA and 1.25–1.33x faster than MLA.
  • TTFT (batch 32, 1024 prompt): GQA 220–231 ms, GVA 199–212 ms, MLA 252–265 ms.
  • Attention layer alone (batch 32, to 1M): GVA decodes 1.37x (16k) to 2.16x (1M) faster than GQA and 1.07–1.27x faster than MLA (to 512k).
  • KV cache is 48.4% smaller than GQA here (d_h 128); 46.9% at the paper's 350M shape (d_h 64).

Full tables will be in the appendix.

Q3. Scale and conversion. We agree this is the key next step. Larger-scale pre-training and conversion are beyond our compute budget for this revision. Conversion applies naturally: a pretrained GQA layer already has grouped values, and M_h can be initialized by least squares (V_g(h) M_h ≈ K_h) before fine-tuning. We add this as future work and make no claims beyond 350M.

Q4. Completeness and long-context evaluation. We will add a full configuration table for every model to the appendix. GVA + DRoPE (d_r = 16):

SettingValue
ModelLlama-style, 24 layers, hidden 1024, SwiGLU 3144, RMSNorm, tied embeddings, vocab 49,152, [ ]M params
Attention16 query heads, 4 value groups, d_h = 64, per-head M_h, d_n = [ ], Q-norm [on/off]
PositionalDecoupled RoPE, d_r = 16, one shared key, base 1e6
DataFineWeb-Edu, [26.2]B tokens, sequence length 2048
Batch0.92M tokens/step (448 × 2048), 28,610 steps
OptimizerAdamW (0.9, 0.95, ε 1e-7), weight decay 0.1, clip 1.0
LRPeak 5e-4, 1,430 warm-up steps, WSD decay over last 2,861 steps to 1e-5
Otherbf16, z-loss 1e-4, init std 0.006, seed 42

Long context. All models were trained with a 2048-token context, so long-context retrieval is outside what any of the three variants support. Longer-context training is beyond our current budget; we agree it is the right test for the narrow positional branch and list it as a limitation and first future-work item. Our systems results already cover up to 1M tokens, where GVA's speed benefit is largest.

We have also submitted a zip file containing code for Grouped Value Attention that includes our prefill and decode kernels in Triton.

Reviewer_dMAsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors introduce Grouped Value Attention (GVA). It caches only grouped value streams and reconstructs per-head content keys on demand via a learned linear map Kh=Vg(h)MhK_h = V_{g(h)} M_h. At decode time the map is absorbed into the query, so content scores are computed directly against the value cache. Positional information is handled by a small shared decoupled-RoPE channel. Relative to matched GQA this yields an analytical reduction of roughly 45–47 % in persistent KV-cache storage. At the 350 M scale trained on 30 B tokens the dr=16d_r=16 variant reaches 44.35 average accuracy, essentially matching GQA (44.36) and exceeding MLA (43.88).

Critical strengths

Here are strong points of the paper:

  1. Clean mathematical formulation: caching only grouped values, reconstructing keys linearly, and absorbing the reconstruction matrix into the query produces an exact identity that eliminates the content-key cache stream while preserving a shared positional channel. This is consequential because it directly targets the dominant memory and bandwidth cost of autoregressive decoding.

  2. Competitive accuracy at reduced cache size: the method matches GQA within a few hundredths of a point and outperforms MLA on the reported downstream suite, showing that the compression does not immediately destroy quality at the tested scale.

Critical weaknesses
  1. No measured systems performance. All efficiency claims remain analytical; end-to-end latency, throughput, peak memory (including temporary buffers), and batch-capacity numbers are absent. For a paper whose title and abstract emphasize efficient caching and faster inference, this is a decisive gap.

  2. Incomplete controls and limited evidence of novelty. The learned map M appears redundant once a decoupled RoPE slice is introduced (tied K=V plus the same positional channel yields an identical cache); cache-matched baselines (GQA with half the groups, MQA, properly configured MLA) and long-context / retrieval evaluations are missing; results are confined to a single 350 M model and short-context benchmarks whose differences lie inside typical seed noise.

  3. Recent concurrent works such as Kayyam et al. (“Do Transformers Need Three Projections? Systematic Study of QKV Variants”) and Gao & Xu (“Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers”) are not discussed or compared against. A dedicated related-work section in the main text would help situate the proposed multi-anchor QK-coupling approach more clearly within the broader landscape of QKV-efficiency methods.

Questions for authors

Here are some questions for the authors to address:

  1. Does a simple tied-K=VK=V baseline that uses the identical decoupled-RoPE slice (no learned MM) match GVA accuracy at the same cache size?

  2. What are the measured decode latency, throughput, and peak memory of GVA versus GQA and MLA under matched hardware and context lengths?

  3. What accuracy and validation loss do cache-matched GQA (G/2G/2 groups), MQA, and MLA achieve?

  4. Can standard deviations / confidence intervals and final held-out loss be reported for every row in the main results table?

2 comments

Q1. Tied-V baseline with the same decoupled RoPE slice

Tied K = V was our starting point (see Section 5: Results, Fig. 3). With standard RoPE it caches exactly half of GQA, but its training loss never recovered to the GQA baseline: a single vector must serve both for scoring and as retrieved content. Introducing M matrix boosts the accuracy number for GVA.

The decoupled RoPE slice was introduced after M, for efficiency. With standard RoPE the rotation sits between M and the query, so M cannot be absorbed and keys would have to be rebuilt every step; the separate positional channel removes this and lets M be folded into the query at inference.

Note: Shared KV (K=v) does not need Decoupled RoPE.

Q2. Measured decode latency, throughput and peak memory

We ran multiple benchmarks for GQA vs GVA vs MLA and here are the results:

Configuration: Llama-3.2-1B body (16 layers, d_model 2048, MLP 8192, vocab 128k), 16 query heads in every variant. GQA: 4 KV heads × 128 head_dim. GVA: 4 value groups × 128 plus a 16-dim shared RoPE key. MLA: DeepSeek-V2-Lite attention (latent 512 + RoPE 64). Each request has a 1024-token prompt, then N tokens are decoded greedily.

AttentionBatch=32 & Context-LengthTTFTP Peak memD LatencyD ms/tokenD tokens/secKV cacheD peak Mem MBSpeedup
GQA1024 (1k)220.006790.003041.622.9010773.202048.004486.00
GVA200.006095.002627.202.5012472.601056.003791.201.15
MLA252.005993.003362.393.009745.501152.003689.60
GQA3072 (3k)220.008839.0010551.003.409316.804096.006535.00
GVA199.007151.008372.282.7011741.602112.004847.301.26
MLA252.307146.1010693.413.009193.002304.004842.10
GQA16384 (16k)230.0022157.50109383.246.004793.1017408.0019853.50
GVA209.0014016.1067097.424.007813.808976.0011712.101.63
MLA262.0014637.3082310.885.006369.609792.0012333.30
GQA32768 (32k)231.0038549.50352022.2210.002978.7033792.0036245.50
GVA212.0022465.00197479.766.005309.8017424.0020161.101.78
MLA261.0023857.40234148.687.004478.2019008.0021553.40
GQA65536 (64k)230.0071333.501248296.1819.001680.0066560.0069029.50
GVA209.0039363.10647129.609.803240.7034320.0037059.101.92
MLA261.0042297.40757158.4511.002769.8037440.0039993.40
GQA131072 (128k)Out of Memory
GVA209.0073159.102303861.0217.581820.6068112.0070855.10
MLA265.2579177.602735058.4120.001533.5074304.0076873.60

Q3. Cache-matched GQA (G/2), MQA, MLA

We agree these are informative controls. Each requires a full pre-training run at the 350M / 30B-token scale, which is beyond our compute budget for this revision. We will add G/2 GQA and MQA to the limitations and plan them for the camera-ready if compute allows.

We have submitted zip file containing code for Grouped Value Attention that includes our prefill and decode kernels in Triton.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.