- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce Grouped Query Experts, a variant of Group Query Attention where each key-value group uses a router to select a set of query heads to use for attention computation, thus reducing the memory costs of the attention computation.
Critical strengths
The proposal is novel and somewhat interesting; there has been suggestions that not all heads are important for predictions thus by constructing an approach to potentially learn to use the correct heads could be a very sensible approach to reduce computation. The motivation is also clear, which is to reduce the amount of computation necessary for attention.
Critical weaknesses
Major
- The presentation of standard attention wasting compute is somewhat contestible, as it's still rather established that this can still achieve accuracy relative to GQA or MQA; while the bottleneck of compute becomes an issue in this setting, framing this as wasting compute is a rather dubious claim and it should be framed rather as the trade-off between precision/quality as opposed to hardware efficiency.
- The compared baselines is limited; the authors only compare against GQA as well as two concatenation variants of the introduced GEQ approach. Furthermore, there is information that is very unclear that compromises understanding of the results. The authors state the setup uses 8 heads and 8 groups for GQA while it is 16 and 8 for GEQ; the actual effect of the number of attention heads can be rather significant at the small model scale as well as because 8 heads is already rather small. Given that the method is an attention variant, it would be extremely relevant to compare with standard multi-headed attention as well as multi-query attention (or even multi-latent attention).
- Beyond this, the use of 16/8 with means that the model is essentially choosing one of two possible query heads at any choice; thus what would be an extremely critical baseline would be to actually just randomly select between the two to verify if any gains are actually coming from the router..
- Evaluation is very limited; while MMLU, PiQA and Arc-E are standard evaluation tasks, they are all rather limited in what is being measured. Beyond just classification tasks, there can be evaluation on other settings such as more retrieval based tasks (ex. question answer or summarization), as well as possibly more difficult benchmarks which one can verify has not been leaked in the training dataset. Actual improvement over GQA at the 250M parameter scale is also limited (only 0.18 percentage points on average) and there is no standard deviation, thus there is the possibility that over multiple seeds, the actual performance gain is insignificant.
- The throughput analysis doesn't show particularly signficant gains until the sequence lengths are extremely long, for example it remains less than 2 times faster for pre-filling throughput even at 1024K tokens. Furthermore, the authors should benchmark generation speed for a better understanding.
Minor
- There is some redundancy in presentation; for example Figure 2 and 3 give the same detail from what I can observe and thus should be merged.
- Section 3.5 appears unnecessary; the computational savings relative to GQA are already fairly evident (use query heads per group instead of the heads/group ratio). The section itself is fairly hand-wavy in its explanations so this should either be explained in further detail or removed; additionally, this doesn't contribute signficantly to the paper's approach so a more fitting place would be the appendix.
Questions for authors
- Some details are confusing;
- What is the "shared attention head" (L193) and how is this selected relative to the number of heads or groups?
- L198 to L200 suggest that the attention softmax probabilites are computed prior to router selection; this implies that the actuall attention is computed over all query heads prior to the router computation, which actually would waste the memory savings of the approach.
- I would also suggest ablations on the actual selection strategy; using Gumbel-Max or Gumbel-Softmax as this might resolve the issue of sum re-weighting in a more natural manner by using a straight-through estimator.
- There should be an actual analysis of what happens with the selection of the query heads; if the task is improperly defined, there is the possiblity that the actual result is the same query head always being chosen within the same group and the setup devolves into multi-head attention except with the number of groups being the practical number of total heads. This would be a relevant comparison to verify if there is actually any gain or benefit from the GEQ setup rather than simply selecting fewer heads.
- What happens when the number of query heads that are associated with each KV group varies; furthermore what happens then the number of activatied query heads varies? There should be signficant ablation studies since this is likely highly sensitive to the choice of hyperparameters for the heads as well as groups.