acceptodds
Withdrawn from ICLR 2027

How to Train Your Energy-Based Transformer: Understanding Stability in EBMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion and autoregressive models generate images, video, and text at scale. Energy-based models (EBMs) cast generation as energy minimization. They learn a scalar energy that scores how well a candidate output matches the data, and energy-based transformers (EBTs) scale this approach by implementing the energy with a transformer. An EBT generates by refining a prediction through a sequence of gradient descent updates on the energy, starting from a noisy or corrupted input. Training differentiates through this refinement sequence, the same computation as backpropagation through time (BPTT) in recurrent networks, and it is often unstable. We analyze this instability. Each backward step multiplies the gradient by a factor set by the step size and by the curvature of the energy with respect to the prediction. We bound the resulting growth and convert the bound into a curvature score computed during training. We compare full BPTT with two lower-cost gradient estimators, truncated BPTT, which differentiates only the last few updates, and Anticipated Reweighted Truncated Backpropagation (ARTBP), which cuts the sequence at random points and reweights the surviving terms to remove the bias introduced by the cut. In image denoising with three model sizes, we find that the best estimator changes with scale. ARTBP gives the highest final training reconstruction quality with the small model, while truncation to the last three updates is best for the base and large models. With the objective and forward trajectory fixed, we find that the truncated gradient changes from closely aligned with the full gradient early in training to pointing in the opposite direction later. The curvature score separates failed from successful completed runs and predicts training stability over a horizon that shrinks with model size.

Primary area:
generative models
Keywords:
energy-based models, energy-based transformers, training stability, backpropagation through time, gradient explosion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qgTjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzed the instability of EBTs by comparing the optimization quality of random truncation across model sizes and the changes in the retained gradient after truncation in existing algorithms. They utilized curvature score (defined using largest positive Hessian eigenvalue and learning factor) calculated during training and used it to monitor the correlation with instability and predict training failure and estimate performance. Their experiments involved using small, base, and large energy-based transformers for denoising.

Critical strengths

It is interesting that the paper consider using a curvature score during training to predict instability and training training failure to estimate performance.

Critical weaknesses

Although the analysis sounds interesting, however, I don't know what is the benefit from doing this. Gradient clipping has been commonly used to avoid instability, the paper did not discuss how clipping will affect their analysis, and I wonder if the maximum eigenvalue provided too generous bound here for the curvature score.

Meanwhile, the paper has many ambiguous statements, and imprecise terminologies. For example: "a gradient component along an eigenvector with curvature x"

Questions for authors

Is the maximum eigenvalue provided too generous bound here for the curvature score?

Reviewer_o47GOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the training instability of image denoising EBTs and the performance of different BPTT algorithms. The authors propose that the high curvature of the energy function and gradient amplification during the backward pass explain the instability. To validate this explanation, the authors define a new metric, the curvature score, and demonstrate that the metric can be used to diagnose and predict training instability in image denoising EBTs. The authors also compare the performance of different BPTT algorithms (full BPTT, TBPTT-3, and ARTBP) and observe that truncated gradients can point in the opposite direction to the full gradients during training.

Critical strengths

This paper analyzes the training instability of image denoising EBTs from the perspective of gradient amplification during BPTT, which is new. The authors propose the curvature score as a diagnostic and predictive tool which can be useful for reliably training EBTs.

Critical weaknesses
  • The paper lacks a clear central idea. Is it about evaluating the performance of different BPTT algorithms, or diagnosing/predicting/mitigating training instability, or the phenomenon of truncated gradients having the opposite direction to full gradients after a certain point in training? How are they connected and what is the main takeaway?
  • The authors argue that the training instability of EBTs stems from the high curvature of the energy landscape and repeated backward steps amplifying the gradient. The main evidence is that the curvature score metric can be used to predict training instability. However, the prediction is not reliable, and the curvature score also does not directly measure gradient amplification, so the claim about the cause of instability is insufficiently substantiated. In addition, the claim is about EBTs in general, but the experiments are limited to image denoising EBTs.
Questions for authors

None

Reviewer_yQETOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to train models that generate outputs by iteratively optimizing predictions while balancing stability, computational cost, and performance. It analyzes gradient amplification and compares different backpropagation horizons. Experiments show that truncation not only changes computational cost but can also substantially alter the direction of learning.

Critical strengths
  1. The research question is practically relevant. The paper links EBT training instability to repeated curvature-dependent factors when differentiating through optimization, providing a testable mechanism.
  2. The gradient comparison is well controlled. Fixing the loss and forward trajectory at each checkpoint helps isolate the effects of truncation from changes in the model state or objective.
Critical weaknesses
  1. I have concerns about the paper’s assumptions. The claim that “no reweighting can recover the lost information” is strong. A negative overall cosine similarity alone does not establish that all reweighting strategies are ineffective.
  2. The experimental scope is also limited. Image denoising with three model sizes supports conclusions about training stability in that setting, but not broad generalizations to text, video, or all EBTs.
Questions for authors
  1. Does the curvature score provide better warnings than simpler indicators? Its loose growth bound may produce high scores even when gradients contract. Are false-positive rates reported, with comparisons against gradient norms, loss changes, and internal step sizes over the same prediction horizon? Does the additional cost of curvature estimation yield earlier or more accurate warnings?

  2. Is the “99% prediction horizon” statistically reliable? How many independent training runs support this result, rather than just how many windows? Given the strong correlation between overlapping windows, are data splits and confidence intervals handled at the run level? Is the horizon selected on validation runs and then tested on independent runs to verify that its success rate remains at least 99%?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All

We sincerely thank the reviewers for their time and their constructive feedback. We are glad that the reviewers find the research question practically relevant and the gradient comparison well controlled (yQET), and that they find the gradient-amplification analysis and the curvature score new and potentially useful for training EBTs reliably (o47G, qgTj).

After careful consideration, we have decided to withdraw the submission. The reviews raise points that we agree with and that require more work than the rebuttal period allows.

  • Central message (o47G). The paper needs one main claim, with a clear connection between the estimator comparison, the curvature score and the change in direction of the truncated gradient.
  • Evidence for the curvature score (yQET, o47G, qgTj). We plan to compare it with simpler indicators such as gradient norms, loss changes and step sizes, to report false-positive rates, to compute the statistics over independent runs, and to study how loose the bound from the largest eigenvalue is.
  • Gradient clipping (qgTj). We plan to study how clipping interacts with our analysis.
  • Scope and precision (yQET, o47G, qgTj). We plan to extend the experiments beyond image denoising, to limit our claims to what the evidence supports, including the statement on reweighting, and to make the terminology precise.

We will continue to work on the paper along these lines and submit it again once it is ready. Thank you again for your suggestions, which will make the next version stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.