acceptodds
Under review as a conference paper at ICLR 2027

Associative-Retrieval Regime Entry Varies with Head Partition Under a Fixed Training Protocol in Transformers

Abstract

Partitioning a fixed attention space across heads can be associated with different associative retrieval outcomes under a shared training protocol. We train six-layer causal transformers with for 20,000 AdamW steps on key–value retrieval tasks using frozen sentence representations from WikiText-103 and AG News. On WikiText, we hold total attention width fixed at while varying the partition from 32 heads of dimension 16 to 4 heads of dimension 128. The fitted 50%-success boundary shifts from to associations, with the four fitted boundary estimates ordered across the tested partitions. A raw load selected on WikiText transfers poorly to AG News, motivating a separate AG News load map. We denote an architecture with heads of dimension by H-D. At the frozen AG News operating point of , H8-D64 enters the prespecified regime in of runs, compared with for H8-D32. An independent fixed-width comparison then matches total attention width, feed-forward width, parameter count, parameter shapes, initialization, data, and training budget. H8-D64 succeeds in of 120 runs, versus for H16-D32, yielding a paired risk difference of (95% CI ). These results show consistent differences in entry into the associative-retrieval regime across the tested head partitions under the shared training protocol.

Primary area:
optimization
Keywords:
transformers, multi-head attention, associative memory, head partition, memory capacity, associative retrieval, optimization dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_FHdAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is an empirical investigation of how different designs for the attention mechanism in a transformer affect the capacity of the model to solve associative recall tasks. The experiments use frozen pretrained embeddings from WikiText-103 and AG-News, fed into a six-layer autoregressive transformer with fixed optimizer settings and varying attention. The task is highly unstable, but the authors find that larger attention heads more reliably learn the task.

Critical strengths

This is an interesting experiment-driven exploration of the capacity of autoregressive transformers to learn associative recall tasks. Since this has been a benchmark task, especially for trying to understand analytically what models can learn, it's important it fully, and this paper contributes to that effort.

Critical weaknesses

I found this paper extremely difficult to read. The abstract is less like a standard scientific abstract than like internal research notes from a team that has been collaborating very closely for a long time. This continues into the paper. From the very start, it is dense with very unusual ways of referring to things. Key terms and experimental concepts are used in the intro but not defined until many pages later. After a while, I could sort of piece things together, but I am not at all confident in my understanding of the paper. Even the core finding was actually sort of difficult for me to extract.

The decision to fix all hyperparameters for all runs strikes me as concerning. These settings seem like exactly the sort of factors that could differentially affect outcomes for different model designs and in turn shape the outcomes. The overall effect for the core finding about width is very small and could be swamped by the variance deriving from proper hyperparameter tuning. (The effects of changing width are larger but don't seem especially pertinent or surprising.)

Questions for authors

Could the authors explain their findings in plain language?

How is performance affected by proper hyperparameter tuning?

Reviewer_MppaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how different attention head partitions affect associative retrieval in Transformers under a fixed training protocol. The authors train small Transformers on key-value retrieval tasks using frozen text representations from WikiText-103 and AG News. They compare different numbers of attention heads and head dimensions, including controlled comparisons with fixed total attention width and matched model parameters. The results show that configurations with fewer but wider attention heads achieve higher retrieval success rates in the tested settings.

Critical strengths

The experimental design is thorough and complex. The authors carefully match model parameters, attention width, initialization, and training settings in the main fixed-width comparison.

Critical weaknesses

-Limited novelty and contribution. Previous studies have already investigated the relationship between attention head dimensions and model capacity [1]. While this paper provides additional empirical results on retrieval success rates, it is unclear how much new understanding these results provide beyond existing work. -The poor writing and structure of this paper makes it particularly difficult to understand what authors are trying to convey. -The experiments attempted to build a relationship between head partition and retrieval performance, but do not fully explain the underlying mechanism.

[1]Amsel, N., Yehudai, G., & Bruna, J. (2025, May). Quality over quantity in attention layers: When adding more heads hurts. In International Conference on Learning Representations (Vol. 2025, pp. 62431-62483).

Questions for authors

Q1. Novelty compared with previous work Previous studies have already shown that attention head dimension can affect model capacity and retrieval performance (Amsel et al., 2025). What is the new insight provided by the capacity boundary and regime-entry analysis beyond existing findings? Q2. Definition of regime entry The paper frequently uses "regime entry" and "regime-entry rate" as central concepts, but does not clearly introduce and explain them as evaluation metrics. Could authors provide a clear definition of this metric?

Reviewer_3crsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper study the setting where we fix the attention space and whether partitioning the attention into more attention heads smaller head dimension is better than fewer heads and larger head dimension. The author conducted experiment on associative retrieval experiments using frozen text embeddings.

Critical strengths

The figure well demonstrated the experiment

Critical weaknesses

This paper has limited novelty. The main conclusion that "larger, fewer attention heads are better than smaller, more attention heads" has been observed by several previous works. While this paper run some more experiments, those experiments are mostly synthetic and do not provide enough mechanistic explanation. The tasks are run with small models and on simple synthetic tasks. On production setting when choosing the size of attention dim it is simple to just ablate on head dimension and check the validation loss and downstream accuracies. If the paper can provide more mechanistic explanation then we can better understand what is happening in the attention mechanism and there are a lot of variants of attention like sparse attention, sliding window attention, etc. However, most experiments are there to verify the claim.

Questions for authors

Check the comments above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.