acceptodds
Under review as a conference paper at ICLR 2027

Transformers Also Need State: Training-Free Temporal Residual Connections

Abstract

Residual connections made deep networks trainable, yet standard transformers leave them unused across time. Parallel training optimizes an independent next-token loss at each position, while the model is never asked to generate as a continuous process. The residual stream accumulates layer by layer but is discarded once a token is emitted, surviving only as keys and values for indirect addressing, never as state in the Long Short-Term Memory sense. Multi-step autoregressive generation thus lies outside training. Without a feedback loop, deterministic decoding collapses into repetition; temperature sampling survives ingeniously, but as a compromise. We propose Training-Free Temporal Residual Connections (TF-TRC): the previous step's deep hidden representation is added to the current token embedding at inference, scaled to a small fraction of the embedding norm. With no architectural change and negligible inference overhead, transformers then carry state across time, exploiting the residual stream, although never trained to do so. Across eleven pretrained models from four families and multiple generations, paired bootstrap (over the original eight) shows significant gains on four: filler runs (a degeneration metric) shrink by a third while burst frequency stays unchanged. Same-energy random vectors are harmful, as is injection into Mamba; injection into an AWD-LSTM is inert; transformers and the linear-attention layers of Qwen3.5 benefit. The delimiting counterexample is an xLSTM — it carries state by construction, yet degenerates under sampling, and the same injections turn harmful on it. Only the additive form works; fully replacing the embedding collapses. The gain requires sampling and turns negative under greedy decoding and at low temperature. Its optimum measures how much state-carrying a model is missing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
residual connections, hidden state, training-free, inference-time intervention, autoregressive generation, decoding

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NQbkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces training-free temporal residual connections: at each token and each layer, it adds a scaled version of the previous token's residual stream at the same level to the current residual stream. The residual connections are claimed to reduce degeneration in the generated text.

Critical strengths

The paper tests a simple intervention idea in a relatively simple inference settings. It compares the effect of its intervention on multiple models against helpful controls (random vectors, rotated or whitened signal). It is explicit about the failure cases, e.g. the collapse of greedy decoding.

Critical weaknesses

The paper is very hard to follow. Its definitions are scattered across the text, there are no results table, and quite a bit of the text reads as LLM-generated, with short sentences that lack context and are hard to interpret.

Questions for authors

Could you evaluate generated text on quality and coherence in addition to testing degenerated cases?

Reviewer_5EZaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a training-free intervention that adds the previous token’s hidden representation to the current residual stream, aiming to reduce degeneration during generation.

Critical strengths

The intervention is simple and lightweight. The random-direction, rotation, and decoding controls provide useful observations about its behavior and limitations, offering a starting point for investigating why the intervention affects generation.

Critical weaknesses
  1. The evaluation does not establish a reliable generation-quality benefit. The main setup uses ten prompts and two seeds, while the injection layer, gain, and signal are selected per model. The separation between configuration selection and testing is unclear. With this limited evaluation, it is difficult to distinguish generalizable improvements from sensitivity to particular prompts or configurations. Early-EOS filtering further complicates interpretation and requires complete accounting.
  2. The evaluation should distinguish correcting degeneration from broadly suppressing repetition. Reduced repetition and increased diversity do not necessarily imply better coherence or task performance. Repetition can also be appropriate, for example when maintaining terminology or following a structured format. I would therefore want to understand both whether the intervention rescues degenerate continuations and whether it preserves the quality of otherwise successful generations. This distinction is important for assessing its usefulness as a general inference-time intervention.
  3. The proposed mechanism remains underdetermined. Changes in previous-position attention and information-decay probes do not uniquely establish the residual-stream carrier explanation; local representation bias or readout reweighting remain plausible alternatives. The random-direction and rotation controls are informative, but it would help to clarify whether they isolate the role of the previous token’s specific content and temporal alignment. A comparison with a temporally mismatched signal of comparable magnitude could help test this distinction, although such a result alone would not establish the full proposed mechanism.
Questions for authors
  1. Can configurations fixed before testing demonstrate consistent benefits on independent prompts, with paired uncertainty estimates and complete accounting of early-EOS outputs?
  2. Relative to standard decoding controls, does the intervention improve semantic quality while preserving successful generations and appropriate repetition?
  3. What evidence distinguishes the contribution of the immediately preceding token’s information from a more general perturbation or readout effect? What prediction of the carrier explanation would distinguish it from these alternatives?
Reviewer_dmSmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I understand, the paper proposes to add a small-scale copy of the previous token's hidden state to the current token's input while sampling from an already trained model, without any training. The scale, the source layer, and the type of signal are set per model by a short sweep. On eleven pretrained models, 10 prompts, and 2 seeds, the paper reports less repetitive text, measured with repetition and diversity counts, with significant gains for four of the eight models where this was tested.

Critical strengths

The method is simple and costs nothing at inference, and the same-energy random-vector control is a sensible check that the direction of the added vector matters.

Critical weaknesses

I fail to see what problem this solves that a decoding setting doesn't already solve. The gain disappears under greedy decoding and below temperature 0.8, and Section 5 says alpha and temperature are "nearly exchangeable" on the reported statistics. The baseline is fixed at temperature 0.9 with top-k 50, while TF-TRC gets a per-model sweep over alpha, layer, and signal on the same 10 prompts used for the significance tests. Without a baseline whose temperature (or top-p) is tuned with the same budget, the reported reduction in repetition may simply be a slightly higher temperature.

Furthermore, the title and abstract claim that the model now "carries state across time", while Section 4 concludes "a readout weighting, not added memory". The closest prior work, which feeds higher-layer states back to the input with training, is missing: Fan et al., 2020, "Addressing Some Limitations of Transformers with Feedback Memory", https://arxiv.org/abs/2002.09402. Several references have wrong author lists, e.g. Universal Transformers.

Questions for authors

Position t can already read the residual stream of position t-1 at every layer through attention, and most models have previous-token heads. How does adding alpha * h_{t-1} to the input differ from raising the attention weight on position t-1 in the first layers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.