Shared Computation, Different Readouts: Depth Compatibility in Looped Transformers

Multi-depth training can improve a looped transformer's accuracy under a shared output head, yet this advantage can reverse when both models receive a small adaptive interface. We study depth compatibility through the excess prediction risk of sharing one readout across depths instead of fitting equally expressive heads separately. In controlled in-context linear regression, we compare fixed-depth and multi-depth training with matched initializations, examples, optimizer updates, and cumulative recurrent computation. Multi-depth training lowers shared-head error but raises the error of a separately fitted head at the reference depth. After we freeze the cores, two shared adaptation directions remove over 90% of the all-depth sharing cost in each of six models. Four directions reverse the training comparison in all three initialization pairs on two separate 4,096-episode evaluations. Depth-specific gain and offset recover substantially less accuracy, even when we jointly optimize the common prediction direction. We characterize capacity-dependent risk under different, possibly singular feature geometries across depths. The task objective separates separately decodable risk from compatibility cost. An exact information-preserving recurrence shows that a large sharing cost can require only two adaptation directions. The comparison reverses with far less flexibility than fully independent heads provide. These results distinguish the prediction accuracy available in recurrent states from their compatibility with a shared output interface.