acceptodds
Under review as a conference paper at ICLR 2027

The Context-Ready Transformer

Abstract

We introduce the *context-ready transformer*, a decoder architecture that pre-contextualizes each token before it enters the transformer block. During left-to-right generation, a correction network combines the previous position’s final block representation with the current token embedding. The token enters the block as its embedding plus this correction, while ordinary causal attention and the standard key-value (KV) cache remain unchanged. The central computation separates training depth from online inference depth. A context-ready model has a \(D\)-layer block. Parallel training performs \(K\) shared block evaluations over the full sequence, giving effective training depth \(DK\). Streaming inference processes tokens left to right with one block evaluation per new token and online per-token critical path \(D\). A pretrained decoder-only transformer can also be converted by adding a zero-initialized correction feed-forward network (FFN) and fine-tuning. We evaluate validation perplexity (PPL) across widths, depths, block sizes, and two datasets, with comparisons against standard transformers and ablations. In single-run comparisons, a \(D=5\) model obtains lower PPL than a 12-layer transformer while generating \(1.7\times\) faster on an A100. Using \(K=10\) during training, a single-layer model (\(D=1\)) obtains lower PPL than a 6-layer transformer with a \(2.6\times\) inference speedup. Parallel training therefore requires \(K\) block evaluations per iteration, whereas streaming inference requires one per generated token. For the checkpoints in Table 4, sequential inference and parallel \(K=10\) evaluation differ by at most 0.01 PPL. Separately, in a 10-hop pointer-chasing task that requires backpropagation through time (BPTT) across the full correction chain, a \(D=1\) model solves all 11 evaluated query groups, while standard transformers exhibit staircase-like depth dependence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent transformers, parallel training, autoregressive inference, efficient decoding, language modeling, fixed-point iteration, depth efficiency, KV-cache efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZQeMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the context-ready transformer, which adds a correction to the current token's input embedding. The correction is computed by a FFN that takes in previous position's late hidden state and the current token's embedding, effectively breaking the bottleneck of having a single token id after sampling. During inference, it does not add extra cost as the hidden can simply be cached for the next token and discarded. During training, the authors propose to avoid BPTT by running the network K times over the sequence in parallel and using the shifted output of the previous forward (like Jacobi iteration). The experiments use OWT and Wiki and train up to 400M transformers with them. Ablations cover conversion of pretrained transformers via a zero-initialized correction FFN, and a windowed-attention pointer-chasing task under full BPTT.

Critical strengths
  1. The method, implementation-wise, is quite elegant and not invasive to current public SOTA transformer models, maintaining most of the optimized inference stacks like KV cache management, attention backends, etc. The authors empirically showed that the parallel training (Jacobi iteration) mostly matches the full BPTT (under 0.01 PPL) in the config sweep.
  2. The ablation shows that most of the gain comes from the correction step. E.g. include using residual and extra FFN under iso-params giving minimal benefits.
  3. The authors are open about the limitations: single runs, no iso-FLOP baselines, and prefill cost.
Critical weaknesses
  1. The training FLOPs are not well controlled and missing critical baselines. The D-depth model is trained with DK FLOPs essentially. If the argument is more about effective depth, then a strong loop transformer equivalent should be included for comparison. Or the argument should be iso-param mostly, but again loop transformers also fall into this category. The minimal gain is obvious as K=2 gives around 90% of improvement when we go from K=1 to K=5, which has been observed in most of the loop-flavored works.
  2. Most of the theoretical results have already been shown in previous works on loop transformers and similar works. E.g. Theorem 3 is essentially Banach fixed-point theorem and lemma 1 is the standard nilpotency argument for triangular Jacobi iteration. Having these theories is awesome but I wouldn’t consider these as the main contribution for the paper. Efficiency claims are made under batch size = 1, which is usually not the case for most applications.
  3. Given the change is minimal (which itself is a huge advantage of the method), the paper should consider scaling up to larger models (at least going beyond 1B). Moreover, since it is zero-init, it should be able to finetune on existing large checkpoints.
  4. There are many related works that are quite similar to this one. Full bandwidth network (a) is probably the closest one that is not directly compared (or discussed). Others include Coconut (b), Block-Recurrent Transformers (c), and TransformerFAM (d). The FBT for example uses almost the same tricks and additive injection, which is initialized from a pretrained checkpoint (so the zero-init is even more well motivated)
Questions for authors
  1. How does this compare to loop transformers and FBT under a relatively fair setting?
  2. Can you quantify how far the information carried over the last hidden somehow because I feel like this might help explain why K=2 can provide most of the gains already. I remember doing similar ideas like using the past N hiddens and an attention-like probing, which shows that the immediate last token’s hidden has most of the attention weight, suggesting that previous position’s last hidden carries most of the information needed for improvement, and can even better explain why the Jacobi-like iteration method works.
  3. What would happen when the batch size becomes larger? Prefill TTFT, batched throughput etc.
  4. Does the gain become larger or smaller as we scale up?
Reviewer_GoQoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a decoder architecture that conditions each token’s input on the preceding position’s final block representation through a dedicated correction FFN. It retains ordinary causal attention and KV caching. Training uses repeated, weight-shared, sequence-parallel block evaluations to approximate the recurrent computation, while streaming generation evaluates the block once per token. Experiments on OpenWebText and Wikipedia report competitive perplexity with shallower deployed models, faster batch-one decoding, and reduced KV-cache memory. A separate pointer-chasing experiment demonstrates recurrent capacity when trained with full BPTT.

Critical strengths
  1. The architecture provides a simple way to introduce cross-token recurrent computation while retaining standard causal attention and KV caching. The distinction between approximate parallel training and exact streaming evaluation is clearly articulated. The close perplexity agreement between sequential and parallel evaluation in Table 4 provides useful empirical support for the approximation at the tested checkpoints.

  2. The language-modeling results show promising deployment tradeoffs: selected shallow configurations achieve competitive or better perplexity with faster batch-one decoding and smaller KV caches. The correction-network ablations also provide evidence that the particular feedback mechanism contributes beyond simply adding another feed-forward sublayer at inference.

Critical weaknesses
  1. The quality advantage is not sufficiently separated from the additional training computation. The comparisons match token budgets, while context-ready training performs multiple block evaluations per batch. Section 5.1 explicitly states that training-compute-matched baselines are not provided. Spending more training compute to reduce decoding cost is a legitimate objective, but the paper does not establish whether a standard model with a comparable inference budget could obtain similar quality with the additional training budget. Reporting actual training cost and a focused budget-matched comparison would make the architectural benefit substantially clearer.

  2. The practical inference tradeoff is not jointly validated for prompt processing, continuation quality, and decoding. The reported speedups exclude prefill, while exact sequential prefill can be costly and finite-K parallel prefill produces an approximate cache. Table 4 is reassuring about aggregate perplexity on selected checkpoints, but does not establish the quality and latency of finite-K prefill followed by streaming continuation for the headline configurations. This limits the evidence for when the decoding savings translate into a useful end-to-end advantage.

Questions for authors
  1. For the headline D=1 and D=5 configurations, what are the total training FLOPs or GPU-hours relative to their baselines? Is there evidence that the quality advantage persists when a baseline with a comparable inference budget receives a comparable training-compute budget?

  2. For those same configurations, can you report continuation quality and end-to-end latency after finite-K parallel prefill, including time to first token? Please distinguish this setting from fully sequential evaluation and identify the prompt/output-length regimes in which the speed advantage remains.

Reviewer_3RukOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the context-ready transformer, an autoregressive architecture that pre-contextualizes each token embedding using a correction derived from the previous position's final block representation. The architecture retains conventional causal attention and KV caching while introducing a recurrent dependency across generated tokens. During training, the model performs multiple parallel evaluations of a shared transformer block, whereas streaming inference requires only one block evaluation per token. The authors provide theoretical analyses of this formulation and evaluate language-modeling perplexity, inference speed, KV-cache usage, and synthetic pointer-chasing performance. The results suggest that the proposed architecture can reduce online transformer depth while maintaining competitive language-modeling performance.

Critical strengths
  1. An interesting architectural approach to reducing inference depth.

The proposed recurrent correction mechanism provides a different way to allocate contextual computation between training and autoregressive inference. Rather than repeatedly contextualizing a token through a deep transformer stack, the architecture incorporates information from the previous token's final representation before block processing.

  1. A clear formulation of the relationship between parallel training and streaming inference.

The paper explains how the past-only, non-cumulative correction permits parallel iterative training and exact sequential evaluation when the prefix is correctly processed. The theoretical formulation and the observed agreement between sequential evaluation and sufficiently iterated parallel evaluation support this mechanism.

Critical weaknesses
  1. The claimed efficiency advantage is not evaluated under sufficiently comprehensive computational comparisons.

The proposed method requires multiple evaluations of the shared transformer block during training, yet the paper does not provide training-compute-matched comparisons against standard transformers. The language-modeling experiments mainly compare models trained for similar numbers of iterations or tokens, despite their different computational costs. Furthermore, the reported inference speedups are measured in batch-one decoding and exclude prompt-processing latency. Since finite-K parallel prefill also introduces additional computation, it remains unclear whether the architecture provides a favorable end-to-end efficiency tradeoff in realistic serving workloads.

  1. The long-horizon capability experiments do not validate the proposed efficient training procedure.

The pointer-chasing results are interesting, but the successful single-layer context-ready model is trained using full backpropagation through time rather than the fixed-K parallel training procedure used for language modeling. This distinction is important because the claimed architectural advantage depends on avoiding the sequential training cost of BPTT. Therefore, the pointer-chasing experiments demonstrate recurrent representational capacity but do not establish that the proposed efficient training procedure preserves this capability. The limited model scale and context lengths further constrain the conclusions.

Questions for authors
  • Training efficiency. Can the authors provide training-compute-matched comparisons against standard transformers, reporting validation perplexity as a function of total training FLOPs or GPU-hours?

  • End-to-end inference. How do the proposed models compare in time-to-first-token and total request latency when prompt prefill is included? In particular, how does the tradeoff change with prompt length, generation length, and batch size?

  • Long-horizon training. Can the authors demonstrate the pointer-chasing or a comparable long-range dependency task using the proposed fixed-K parallel training procedure rather than full BPTT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.