- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates context length in global attention modules within transformers. The general consensus has been that the larger the model the longer the context should be to offer better performance. The authors revisit this problem through the perspective of attention sinks. Their insight is that these sinks act as a type of gating and coupled with some observations in the literature they reason that building in short context windows into some of the layers provides the right prior for better model performance.
Critical strengths
The paper is nicely written with a good literature review and a clear explanation of the problem the authors are addressing.
Critical weaknesses
-
The authors don't really explain how building a short sliding window right from the start gives the model the right prior for better performance. In fact, they don't really give detailed insight in how it yields the right prior. They give the impression that it is the right way to build a local prior but why does that actually lead to better performance? In this way real insight into what is going on is lacking in the paper.
-
Given that the paper doesn't offer any real insight as to why their approach can yield better performance I would have expected a few more experiments on different architectures and different modalities.
Questions for authors
-
Can you explain why having a prior by building in short sliding windows from the start leads to better model performance in a more detailed way with some explicit insight?
-
Wouldn't building in such a prior through short sliding windows at the start affect some form of long context tasks like long context needle in the hay stick tasks or more generally long context recall problems?
-
Does this idea hold for general transformer or are you suggesting that it only holds for LLMs?