- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work introduces a new linear-time sequence mixer, FasterWeight-MLP Attention. They evaluate this alternative attention layer on settings such as the NanoGPT benchmark and various commonsense reasoning tasks for 340M & 1.3B models pretrained on fineweb. This work also introduces a view of attention’s query-key interaction which is used to derive the proposed method.
Critical strengths
Critical weaknesses
This work appears to be almost entirely written by LLMs and is very difficult to read. The prose states a lot of buzz words, but does not actually convey clear information. For example, here is a topic sentence of an introductory paragraph: “Cost linear in sequence length constrains how the context may enter a fast MLP: only through statistics that carry forward in a running form.” This sentence structure is confusing and unnatural.
The math is not clear. Variables are not defined before they are used. I am unsure when an object is a function, a trainable parameter, etc., nor its dimensions. The surrounding text does not motivate or explain the point of many equations.
Questions for authors
Can you write the state update rules in a clear way (with dimensions clearly defined) all as a function of the tokens Please denote which terms are trainable parameters vs. activations / online, fast weights.
What does represent and how does this relate to softmax attention? The text of lines 140-143 connects this to Eq. 1, yet the choice of summing over tokens seemingly has no relationship to any of the above equations for softmax attention.