- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- paper studies different configurations of attention layer placement in small recurrence-attention hybrids
- the "middle" configuration performs well on both parity and MQAR recall
- removing attention layer retains partiy performance, but destroys recall in the hybrids
Critical strengths
- I find it interesting that removing attention layer retains performance on the parity task
Critical weaknesses
-
there are studies done already ablating the placement of attention layers in models, e.g. https://arxiv.org/abs/2510.04800, https://arxiv.org/abs/2507.06457, https://arxiv.org/abs/2604.21454 and especially https://arxiv.org/abs/2604.21454 -- i am not sure we learn anything new from this paper
-
the tasks are restricted to MQAR and parity -- its unclear whether these findings translate to language modeling
-
the models used in the paper go upto 6 layers only -- its hard to conclude from the experiments whether the claim that one attention layer would be enough when models are 24 layers or 36 layers
Questions for authors
- Would it be possible to try your best configuration on language modeling and test natural language recall and state-tracking? you can compare it attention-only, recurrence-only, and one other common hybrid baseline to showcase that one attention layer is actually enough