acceptodds
Under review as a conference paper at ICLR 2027

Latent Test-Time Computation with Energy Transformer Components

Abstract

Looped transformer blocks pass hidden states through shared weights multiple times, offering benefits in model size reduction and latent reasoning. While most work in this area has focused on traditional transformers, a related line of work on energy-based modeling introduced Energy GPT (EGPT) blocks, whose forward pass is explicitly the gradient of an energy, or negative log-likelihood, function. Energy-based models are therefore naturally looped, since inference corresponds to gradient-based optimization of the energy. This raises a natural question: can energy transformers be combined with traditional transformers, and does their gradient structure provide practical benefits? We find that the answer is yes. Hybrid EGPT-GPT models are substantially more robust to reductions in loop steps than purely GPT-based looped models, leading to better test-time compute scaling. We identify two qualitatively different computation modes. Variants with energy attention exhibit distributed refinement, in which predictions improve progressively at each iteration, with correct-token rank improving from 110 to 3 across blocks. In contrast, some GPT variants exhibit concentrated construction, in which useful predictions are deferred until the final iterations, with correct-token rank remaining above 250 until the last block. The practical consequence is that distributed refinement enables compute savings at inference time: a simple convergence-rate criterion saves 19% of measured FLOPs with no quality loss for energy-attention models, while concentrated-construction models cannot be halted early without collapsing. These findings replicate across scales from 200M to 1B parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Energy Based Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7eBVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work investigates whether iterating Energy Transformer components produces qualitatively different computation from repeatedly applying a regular GPT block. The study considers nine ET/GPT architectural variants and ablations, with the main models evaluated at approximately 400M, 800M, and 1B parameters after training from scratch on 126B tokens. The recurrent part of each model occupies the final four blocks, allowing the experiments to vary energy versus standard attention, energy FFNs versus SwiGLU, and parallel versus consecutive update layouts in a controlled setting.

Using per-block and per-iteration LogitLens analyses, the authors identify two computation regimes. In "distributed refinement," energy-attention models progressively improve the prediction across recurrent blocks, making intermediate states useful and prediction-aligned. In "concentrated construction," standard-attention recurrent models perform much of the prediction-forming computation in the final block and iterations. The factorial comparisons further indicate that energy attention is necessary but not sufficient for robustness to fewer inference iterations: pairing it with either an energy FFN or a consecutive update layout prevents the update overshoot observed in the unstable parallel energy-attention/SwiGLU variant.

The paper then uses these findings to motivate inference-time iteration reduction and adaptive halting. A convergence-rate criterion yields up to 19% measured FLOP savings for EGPT-Dual at 400M parameters with only a 0.5% perplexity increase.

Critical strengths
  • The paper identifies an important and interesting distinction in how energy-based and standard looped transformers compute.The distinction between distributed refinement and concentrated construction gives a useful empirical account of when intermediate recurrent states are already predictive and when the model instead relies on late-stage construction. This distinction is consequential because it directly explains why some architectures tolerate reduced test-time iterations while others degrade or collapse.
  1. The findings and the resulting efficiency technique are supported by an extensive, controlled experimental setup. The factorial design varies attention type, FFN type, and update layout across several novel hybrid architectures, while the principal results are checked across approximately 400M, 800M, and 1B parameter scales under a common 126B-token training budget. The paper observes the computation modes, relates them to architectural components, and then converts the observation into measured inference savings. In particular, the reported savings make the analysis practically relevant. This breadth and clarity of validation are central to my positive assessment.
Critical weaknesses

The main experiments compare architectures after a single common training budget of 126B tokens. It would be informative to repeat the central analyses at several matched token counts leading up to 126B. Such a training-trajectory study would show whether distributed refinement, concentrated construction, update overshoot, and robustness to iteration reduction appear early in training, emerge only after sufficient optimization, or change in relative strength as the models consume more data.

Appendix S provides useful partial evidence through additional 400M runs trained on 15.7B and approximately 40B tokens. Those results support the qualitative robustness partition, but they also show considerable variation within the robust group: for example, the 15.7B-token EGPT-Dual runs have a mean iteration-reduction degradation of 14.2% with a 6.7% standard deviation, while the approximately 40B-token EGPT-Dual runs differ noticeably from one another. Because these runs are primarily seed checks, use shorter budgets, and do not trace all main architectures and diagnostics through matched checkpoints, they do not fully reveal how the reported phenomena develop with training. I view this as a limitation of the experimental section and not a soundness problem, but resolving it would strengthen the generality and practical interpretation of the main claims.

Questions for authors
  1. Do you have matched intermediate-checkpoint results, for example at roughly 15B, 40B, and one or more later points up to 126B tokens, for the principal EGPT-Dual and RecGPT comparison? In particular, do the LogitLens rank trajectories, update magnitudes, and perplexity degradation under reduced iterations already separate into distributed refinement and concentrated construction early in training, or does this separation emerge gradually?

  2. How should the shorter-budget results in Appendix S be interpreted relative to the fully trained results? The 15.7B-token EGPT-Dual runs show noticeably larger and more variable degradation under iteration reduction than the main 126B-token model. Is this difference primarily attributable to training maturity, seed variance, or another experimental difference? Evidence distinguishing these explanations would affect how confidently I interpret iteration robustness as an architectural property throughout training.

Reviewer_bgHNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the components of energy transformer affect computation in recurrent transformers. By varying attention mechanism, FFN and update layouts, the authors do extensive study on behavior of recurrent transformers and identify two different tendencies: distributed refinement and concentrated construction. These experiments imply that energy attention allows robustness to reduced test-time iteration. Furthermore, authors propose value tying as a potential cause to explain phenomena.

Critical strengths
  1. Systematic studies on recurrent transformer components.
  • Given this factorial design space, it provides useful insights into how attention, FFN and update layout interact each other. In particular, the results demonstrate that energy attention is insufficient for robustness, emphasizing the importance of interaction with other components.
  1. Extensive experimental evaluation with relevant baselines.
  • The authors provide extensive comparisons with conventional GPT and recurrent GPT under consistent settings. Following experiments cover robustness to reduced iteration and behavior of intermediate states, validated across multiple model scales.
Critical weaknesses
  1. Limited validation of the value-tying mechanism
  • In Section 5.1, authors hypothesize that value-tying property in energy attention can induce prediction-aligned intermediate representations. However, in the table 4, we can check that there are only indirect evidences through weight space perspective.
  • Because energy attention introduces changes in architecture alongside value tying, it remains unclear whether value tying is primary factor responsible for the distributed refinement behavior. Although the authors acknowledge this limitation, a more direct comparison would enhance the proposed explanation.
Questions for authors
  1. Does the authors further isolate the contribution of value tying from other components caused by energy attention?
Reviewer_FmKyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how energy-inspired attention, FFNs, and update layouts affect recurrent transformers. Experiments at 400M–1B scales show that certain energy-attention variants produce more prediction-aligned intermediate states and tolerate fewer inference iterations. The authors attribute this behavior to value tying and report compute savings through early stopping.

Critical strengths
  • The component comparisons reveal that energy attention alone is insufficient: the FFN and update layout also affect robustness.
  • Iteration-reduction experiments across multiple scales provide functional validation beyond the LogitLens observations.
Critical weaknesses
  • The contribution of the energy formulation remains unclear. The studied models primarily introduce value tying, modified FFNs, and different update layouts. The trained dynamics do not minimize energy, and the boundedness argument also applies to standard normalized recurrent blocks. The paper therefore demonstrates interesting architectural effects without clearly establishing what the energy interpretation adds. The targeted ablation isolating value tying is explicitly left to future work.
  • The practical advantage is not sufficiently established. Compute savings are primarily measured against each model’s own full-iteration baseline. At matched compute in Table 26, EGPT-Dual has better perplexity but substantially worse GSM8K accuracy than RecGPT. Simpler variants and reduced RecGPT schedules are also competitive in Tables 27 and 30. These results leave the need for the full Energy Transformer construction uncertain.
Questions for authors
  • What evidence isolates value tying as the cause of robustness, independently of the other architectural changes?
  • What explanatory benefit does the energy formulation provide when the trained dynamics do not minimize energy?
  • At matched absolute compute, when does the full Energy Transformer offer a clear benefit over simpler variants or reduced-iteration RecGPT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.