- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper summary: The paper proposes to improve Grouped Query Experts (GQE) by adapting an approach inspired by HyperMoE: similar to what happens in HyperMoE where a dynamic expert is computed to “replace” the experts that were left out, here a dynamic query head is computed to compensate for the query heads that were not selected. The dynamic query head per token is created from a hypernetwork that takes as input the average embedding over all experts that were not chosen.
Critical strengths
Positive points:
- The transfer of the HyperMoE idea to the GQE setting is reasonable and tries to address an actual drop in performance that occurs when transitioning from Grouped Query Attention (GQA) to GQE caused by selecting the top-K query experts.
Critical weaknesses
Negative points
- The motivation of the approach was to avoid the drop in performance when switching from GQA to GQE. However, there is only a single comparison with GQA in table 1. Furthermore, no comparison with GQA exists in table 2 for the 16 zero-shot tasks. Moreover, it is not clear how statistically significant the 0.69 improvement reported in table 1 is. More generally, the experimental validation is quite limited (a single fine-tuning dataset SST-2 is used and a single model). Therefore, the claim “Exceeding Dense Baselines” (mentioned as one of the contributions) is not sufficiently validated.
- HyperGQE introduces additional active parameters compared to GQE. Therefore, it is unclear if the improvement in performance compared to GQE comes from this or the dynamic expert mechanism. Comparisons with (i) a parameter-matched GQE and (ii) a GQE with an additional static adapter of the same type are needed.
- The zero-shot scores reported in table 2 are very low (ie., close to chance level), so it is not clear how meaningful the gains are
- The overview figure does not help at all in understanding what the proposed approach does (e.g. what do the static layers and dynamic layers correspond to ?)
Questions for authors
The authors should address the issues raised in the critical weaknesses section.