- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduced grouped on value in softmax attention and named it Groupped Value Attention (GVA), instead grouped on queries in GQA. Experiments show that GVA achieves near GQA accuracy on benchmarks with more compact cache representation.
Critical strengths
- The technique sounds OK, especially the derivation on the decoupled RoPE process.
- The writing is generally easy to follow.
Critical weaknesses
- It only compared the cache size to GQA, but does not compared to MLA.
- It lacks of the number report on efficiency, latency, peak memory, throughput, etc, which makes the whole paper not-throughly verified.
- It mixed using different quantitive description "the cache is exactly half" (L255)/"roughly halves" (L262)/"approximately 45–47%" (L061). It is better making it more clear.
- What DRoPE stands for? Decoupled RoPE? Should add citation to the RoPE paper.
- There are wrong references, "Section 6" in L065, while Section 6 is about limitation, has nothing to do with the "prefill cost".
Questions for authors
see comments on "weakness"
2 comments
We have also submitted a zip file containing code for Grouped Value Attention that includes our prefill and decode kernels in Triton.
I'm sharing you the results of GQA vs GVA vs MLA which has all the metrics requested.
Measured systems performance.
We measured GQA, GVA and MLA on one NVIDIA H100 NVL (94 GB HBM3, 3.9 TB/s peak), bf16, [PyTorch / Triton / FlashAttention / FlashMLA versions].
Model. Llama-3.2-1B body (16 layers, d_model 2048, MLP 8192, vocab 128k) with 16 query heads in every variant:
Whole model, batch 32
Only Attention Layer: Below are the numbers for only attention layer with 1024 prefill then decoding till context length:
DRoPE stands for Decoupled RoPE. Yes, we will add citation to the RoPE paper. Thank you for pointing it out.
Thank you for pointing the wrong references. We will update the paper.