- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explores, in practice, how many key-value associations are able to be recalled in linear-state models. The authors perform experiments on single-head, two-layer deltanet models trained for MQAR. They find that slowly increasing the number of pairs stored in the state during training can help in this setting.
Critical strengths
None
Critical weaknesses
-
I found the language and writing style in this paper to be quite poor, sounding very AI-generated. This made it extremely difficult to follow and lacked actual technical depth.
-
The experiment results are not very rigorous as this is demonstrated on a single-head, two-layer model. The suggested training curriculum is not validated at any sort of real-model scale on real tasks.
Questions for authors
None