Transformers Also Need State: Training-Free Temporal Residual Connections

Residual connections made deep networks trainable, yet standard transformers leave them unused across time. Parallel training optimizes an independent next-token loss at each position, while the model is never asked to generate as a continuous process. The residual stream accumulates layer by layer but is discarded once a token is emitted, surviving only as keys and values for indirect addressing, never as state in the Long Short-Term Memory sense. Multi-step autoregressive generation thus lies outside training. Without a feedback loop, deterministic decoding collapses into repetition; temperature sampling survives ingeniously, but as a compromise. We propose Training-Free Temporal Residual Connections (TF-TRC): the previous step's deep hidden representation is added to the current token embedding at inference, scaled to a small fraction of the embedding norm. With no architectural change and negligible inference overhead, transformers then carry state across time, exploiting the residual stream, although never trained to do so. Across eleven pretrained models from four families and multiple generations, paired bootstrap (over the original eight) shows significant gains on four: filler runs (a degeneration metric) shrink by a third while burst frequency stays unchanged. Same-energy random vectors are harmful, as is injection into Mamba; injection into an AWD-LSTM is inert; transformers and the linear-attention layers of Qwen3.5 benefit. The delimiting counterexample is an xLSTM — it carries state by construction, yet degenerates under sampling, and the same injections turn harmful on it. Only the additive form works; fully replacing the embedding collapses. The gain requires sampling and turns negative under greedy decoding and at low temperature. Its optimum measures how much state-carrying a model is missing.