- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is an empirical investigation of how different designs for the attention mechanism in a transformer affect the capacity of the model to solve associative recall tasks. The experiments use frozen pretrained embeddings from WikiText-103 and AG-News, fed into a six-layer autoregressive transformer with fixed optimizer settings and varying attention. The task is highly unstable, but the authors find that larger attention heads more reliably learn the task.
Critical strengths
This is an interesting experiment-driven exploration of the capacity of autoregressive transformers to learn associative recall tasks. Since this has been a benchmark task, especially for trying to understand analytically what models can learn, it's important it fully, and this paper contributes to that effort.
Critical weaknesses
I found this paper extremely difficult to read. The abstract is less like a standard scientific abstract than like internal research notes from a team that has been collaborating very closely for a long time. This continues into the paper. From the very start, it is dense with very unusual ways of referring to things. Key terms and experimental concepts are used in the intro but not defined until many pages later. After a while, I could sort of piece things together, but I am not at all confident in my understanding of the paper. Even the core finding was actually sort of difficult for me to extract.
The decision to fix all hyperparameters for all runs strikes me as concerning. These settings seem like exactly the sort of factors that could differentially affect outcomes for different model designs and in turn shape the outcomes. The overall effect for the core finding about width is very small and could be swamped by the variance deriving from proper hyperparameter tuning. (The effects of changing width are larger but don't seem especially pertinent or surprising.)
Questions for authors
Could the authors explain their findings in plain language?
How is performance affected by proper hyperparameter tuning?