acceptodds
Under review as a conference paper at ICLR 2027

Where You Put Attention Matters More Than How Much You Use: Attention Placement in Hybrid Sequence Models

Abstract

Hybrid sequence models use mostly recurrent blocks plus a few attention blocks, and most design studies ask how many attention blocks to use rather than where to put them. We compare six models, each with six blocks and the same number of parameters, so the only difference is where the attention blocks sit. We train each setup with several seeds on a single GPU and test on three tasks: parity state tracking, the A5 task (where order matters), and multi-query associative recall. On parity, putting one attention block at the front or in the middle beats both the recurrent baseline and back placement, while on recall the middle layout works well and the front layout almost fails. To find out why, we ran a paired test that removes attention after training: on parity, accuracy barely changes and the difference is not significant, but on recall it collapses and the difference is highly significant. This means the model needs attention at inference time for recall, while on parity attention mainly shapes what the recurrent blocks learn during training. Middle placement is the only layout that does well on both tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
hybrid sequence models, attention placement, recurrent neural networks, formal languages, state tracking, associative recall

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_UwDrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • paper studies different configurations of attention layer placement in small recurrence-attention hybrids
  • the "middle" configuration performs well on both parity and MQAR recall
  • removing attention layer retains partiy performance, but destroys recall in the hybrids
Critical strengths
  • I find it interesting that removing attention layer retains performance on the parity task
Critical weaknesses
Questions for authors
  • Would it be possible to try your best configuration on language modeling and test natural language recall and state-tracking? you can compare it attention-only, recurrence-only, and one other common hybrid baseline to showcase that one attention layer is actually enough
Reviewer_495rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether the location of a small number of attention blocks within a recurrent-attention hybrid matters for sequence capabilities. The authors compare six parameter-matched, six-block networks: recurrent-only (R6), one attention block at the front/middle/back (A-front/A-mid/A-back), two attention blocks (A2-mid), and attention-only (A6). Experiments use five seeds per configuration on parity, the non-commutative A5 state-tracking task, and multi-query associative recall (MQAR). The paper argues that middle attention placement provides a favorable trade-off between state tracking and recall, and suggests that attention mainly helps learn state tracking during optimization but is used directly for retrieval at inference

Critical strengths

The paper studies an interesting architectural design consideration which might be useful for practical neural network design.

Critical weaknesses

I think this is not ready for publication and might be a very hasty AI generated paper. The writing is clearly not up to the mark. The figures are clearly AI generated. The evaluation is not done properly; on large scale datasets it might not hold.

Questions for authors

Can you try this on large scale data sets? Both from image and NLP domain?

How generalizable are the findings? What happens if we go for large scale networks like with fifty layers or even more?

Please improve the overall writing. It seems the whole thing is AI generated. Even the formatting and the font does not seem to match the ICLR template.

Reviewer_rGNvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper concerns the question of where to place attention blocks in the context of hybrid sequence models. Evaluating on Parity and Associative Recall tasks, the authors find that -- consistent with previous work on other tasks -- the mid-level position of attention provides a good compromise of capabilities and performs best across the two tasks considered.

Critical strengths
  • A nice honest and straightforward evaluation of hybrid RNN models with attention blocks.
  • Limitation section is very well done and honestly describes the limitations of the paper's contributions.
Critical weaknesses
  • Primary weakness is, in some sense, also related to it's strength. On the positive side, this paper is a nice simple study, but on the negative side, the study has very a narrow focus -- asking a narrow question regarding the position of attention blocks in hybrid models. The limitations section gives a fair assessment of this issue, but the reality is that this sort of hyperparameter selection study is likely to have insufficient impact due to the limited scope. In essence, it's not clear what can really be taken from this study that can be confidently applied in other contexts.

  • The presentation is a bit rough. There are formatting issues with the references (regular doubling of refs). Some paragraphs, (eg. Tasks - lines 150-158) are difficult to parse, there might be a formatting issue there as well. Finally the text is quite clearly AI generated in style. I'm not opposed to authors using AI to improve the text, but this is an example where the AI is too present.

Questions for authors

no questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.