- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an alternative mechanism to sparse attention layers through a technique inspired by dynamic pointers with the goal to retain O(n) state but reducing the O(n) access of vanilla transformers. Each token keeps a list of pointers down the KV, cache. Initially it starts as a sliding windows and can grow down the KV history through "referral" from previous pointers.
Critical strengths
-Alternatives to sparse attention mechanisms are an important research direction, especially given the scale of retained context in current LLM and the resulting computational cost.
Critical weaknesses
- My main concern with the method is a) it is very hard to properly understand the method due to its presentation and b) it is unclear what the benefits of the proposed method would be. It is argued that it can replace existing SA mechanisms, however, to what end? If I understand it correctly its not about memory reduction as KV is maintained, nor is it faster in training / inference (or is lacking empirical support for it). Further, there is no evidence on a clear improvement of quality (see eval scores), especially if considering it under equal resource budget.
- The description of the mechanism is lackeing: what is its associated computational cost for the different steps and consequences for speed / memory access
- Further, the method is not evaluated in the context where SA method would be actually required, ie. long context retrieval / efficiency or against relevant baseline methods (even a sliding window) -Many of the references are missing further information such as venues, proceedings or a proper reference link to identify them.
Questions for authors
- Can you provide further details on the computational resources used for the experiments, ie. walltime and what tradeoffs your proposed mechanism aims at
- Please include more comparisons for the results, ie. Table 4. As I understand it the performance is near random chance on the evaluated downstream benchmarks?