Brain-Aligned Intermediate Transformer States Depend on Modality, Stimulus, and Positional Encoding

Alignment between Transformers and brain activity has mainly been studied at the layer level, but each layer contains computational states with different properties. Previous work identified intermediate states that align more strongly with brain activity than the commonly used block output, yet the generality of this finding and the mechanisms behind the advantage of particular states remain unclear. Here, we examined brain alignment across within-block states in text, audio, music-generation, and vision Transformers. We quantified this alignment by using these states to predict brain responses to speech, music, and images. We also compared models that share the same basic architecture but use different positional-encoding schemes. We found that intermediate states generally yielded better predictions than the conventional block output, but that the strongest correspondences varied with input modality, stimulus, and positional encoding. Text and audio models favored distinct state families even for the same brain responses, and the preferred states changed between speech and music within the same audio model. In vision, positional encoding reshaped the pattern of state-specific alignment. We further examined the computations and representational content underlying these differences, providing a mechanistic interpretation of why particular states align with auditory and visual cortex. These results suggest that the correspondence between brain responses and states is more diverse than previously assumed, and that this diversity is rooted in the computation and the representational content of each state.