- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study the role of depth in in-context learning via the adaptive inference. For the in-context linear regression, they analyze the adaptive and non-adaptive inference via generalized linear attention transformers under the Gaussian and tree prior. Their analysis predicts that depth provides no advantages under the Gaussian prior, while it can be beneficial under the tree prior. They also empirically support for these predictions.
Critical strengths
The strength of this paper is to interpret the role of MLPs and attention in adaptive inference. MLPs determine which information should be extracted next, while attention extract the information from the context. This interpretation is interesting. The experiments provide good empirical support for the theoretical results.
Critical weaknesses
The theory focuses on in-context linear regression with linear attention. Although the tree prior is considered as a non-Gaussian example, it is still a specific case. Thus, it is not clear if the role of MLPs and attention discussed in this paper can also be obtained in more practical setting, such as nonlinear in-context learning, standard softmax attention, and other non-Gaussian prior.
Questions for authors
Is there any intrinsic difficulty in extending the analysis beyond linear regression, linear attention, and other non-Gaussian prior ? In particular, could the same interpretation of depth and the role of MLPs and attention be extended to these case ?
2 comments
Thank you very much for the clarification. I would like to keep my current score as weak acceptance.
Thank you for your kind comments on our work, we hope that the following has adequately addressed your questions: