Readout and Relay: Temporal Functional Geometry in Autoregressive Language Models

Autoregressive transformers use each hidden state both to predict the next token and to support later predictions. We ask how these two roles coexist within a single representation. We identify a compact current-output-sensitive subspace, the readout, and decompose future credit into a component that reuses readout directions and a current-insensitive relay outside this subspace. Three patterns emerge. First, future credit substantially reuses the readout; within these reused directions, current and future credit can align or oppose, giving rise to synergy and conflict. Second, the relay is temporally selective: the component associated with a given future loss mainly affects the corresponding future prediction, with much smaller effects at other positions. For predictions farther in the future, a smaller fraction of future credit lies in the readout, while a larger fraction is carried by the relay. Third, across post-training stages, the balance between readout reuse and temporally selective relay changes little, while current-output sensitivity becomes concentrated in fewer directions. This suggests that post-training sharpens which directions matter for the current prediction without substantially reorganizing how hidden states support present and future predictions. Building on these findings, we introduce Temporal Proximal Credit (TPC), a lightweight training rule that reduces credit from later positions along directions important for the current prediction, without changing inference. TPC improves supervised fine-tuning across two model families, and a matched control shows that both moderating future credit and directing that moderation toward current-sensitive directions contribute to the gains.