- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies gradient descent learning of the key and value component in a simple softmax attention unit. One of the main observations is that selection direction cannot be learned through correlation of only the labels and key scores. However, by introducing also learnable value scores, learning the selection direction becomes subsequently tractable. Thus, the main contribution of the paper is to characterize a mechanism by which learning first a value direction helps with learning of the selection direction. The authors do this analysis in the context of the sample complexity for the projected gradient descent under certain technical assumptions on the model and initialization and a spectral method using the squared labels. The authors support the derived theory results with numerical experiments.
Critical strengths
I see the following as strengths of this paper.
-
The topic of joint learning seems quite interesting and specifically how you can clearly see numerically that the value direction is learned first and subsequently followed by the learning of the selection direction. What is also nice is that this phenomenon is already exhibited in a quite simple model, which gives clean setting for theoretical analysis.
-
The paper has an interesting theoretical results that are a good step towards understanding the joint learning dynamics of the attention model. In particular I find that Theorem 1 gives a useful insight into where this "value catalyst" phenomenon comes from.
Critical weaknesses
I have found two main weakness points.
-
The main weakness is regarding the proof and implications of the theory contribution of the paper. One of the main results, the optimal sample complexity bound coming from Theorem 6 in appendix, crucially relies on the invariant subspace introduced in Lemma 11 through Theorem 5. Even though this subspace is indeed invariant under population gradient updates, and sample gradients concentrate, the subspace is not necessarily invariant under finite sample updates considered in Theorem 6. Thus, as it currently stands, it is unclear whether the current main convergence result holds. Next, the learnability gap coming from Corollary 1 and Theorem 2, seems a bit misleading. Namely, the gap exists because of the restriction imposed on looking at this as a single-index model by only using key scores correlation, even though it is multi-index and value scores are available. There is also the orthogonality assumption between the selection and value vector. As commented in Remark 1 of the paper, the gap disappears once there is correlation between the vectors value and key vectors. I believe it can be argued that this is not necessarily a generic situation, as even if they would be independently sampled random variables, their overlap would be non-zero. It would be nice to see stronger justification of these assumptions and/or a mechanism explaining hardness of learnability of selection direction. Overall, this weakness raises soundness and informativeness concerns for me.
-
Presentation of the paper, even though understandable, requires quite a bit of effort to follow. Namely, exposition is very dense, important results and assumptions are scattered across the main body and appendix (e.g. Theorem 6, one of the main results of the paper even motivating the title, is located in the appendix and only abruptly referenced in the main body; for the definition of invariant space, crucial for the results in the main body, one needs to look up Lemma 11 in the appendix) and what is more, most of the important quantities used in the introduction are only introduced in subsequent section and understandable after reading that sections first. This makes following arguments and main results more challenging.
Questions for authors
-
Can you please justify the finite sample analysis of Theorem 6, since finite sample updates do not necessarily land in the invariant subspace of Lemma 11 and Theorem 5?
-
Can you please explain how is the gap from Corollary 1 and Theorem 2 related to the hardness of learning the selection direction for the general multi-index setting considered, instead of being an artifact of restricting the analysis to the single-index setting?
-
Can you please justify the orthogonality assumption in more detail? Also, could you expand on the brief discussion from Remark 1 on how the theoretical analysis and possible conclusions would change if instead there was a (small) overlap between the selection and value vectors?