Associative-Retrieval Regime Entry Varies with Head Partition Under a Fixed Training Protocol in Transformers

Partitioning a fixed attention space across heads can be associated with different associative retrieval outcomes under a shared training protocol. We train six-layer causal transformers with $d_{\mathrm{model}}=512$ for 20,000 AdamW steps on key–value retrieval tasks using frozen sentence representations from WikiText-103 and AG News. On WikiText, we hold total attention width fixed at $d_{\mathrm{attn}}=512$ while varying the partition from 32 heads of dimension 16 to 4 heads of dimension 128. The fitted 50%-success boundary shifts from $F^{\star }=5.50$ to $F^{\star }=7.41$ associations, with the four fitted boundary estimates ordered across the tested partitions. A raw load selected on WikiText transfers poorly to AG News, motivating a separate AG News load map. We denote an architecture with $h$ heads of dimension $d$ by H$h$-D$d$. At the frozen AG News operating point of $F=6$, H8-D64 enters the prespecified regime in $60.0\%$ of runs, compared with $16.7\%$ for H8-D32. An independent fixed-width comparison then matches total attention width, feed-forward width, parameter count, parameter shapes, initialization, data, and training budget. H8-D64 succeeds in $60.8\%$ of 120 runs, versus $45.8\%$ for H16-D32, yielding a paired risk difference of $0.150$ (95% CI $[0.025,0.275]$). These results show consistent differences in entry into the associative-retrieval regime across the tested head partitions under the shared training protocol.