acceptodds
Under review as a conference paper at ICLR 2027

Scaling Transformer Energy-Based Models for High Quality Image Synthesis

Abstract

Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes. To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space. On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics. Furthermore, two components make this high-quality at scale: Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model, and advanced sampling strategies that improve sample quality at inference. TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution. We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection.

Primary area:
generative models
Keywords:
Energy-Based Models, Generative Modeling, Transformers, Image Generation, Scalability, Compositional Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nTr4OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TEM scales contrastively trained scalar energy-based models using a latent-space transformer, multiple energy heads, sampler matching, and positive-only representation alignment (PRA). TEM-XL reports FID-50K of 3.82 on ImageNet 256 and 8.24 on ImageNet 512, using 60-step persistent sampling without CFG. Ablations examine the contributions of the loss terms, energy heads, and representation-alignment choices. Additional experiments evaluate compositional generation and OOD detection, illustrate inpainting, and report training and inference costs.

Critical strengths

The ImageNet results and scaling across four model sizes are a substantial empirical advance for explicit contrastive EBMs. The successful integration provides a useful route for further EBM research.

The loss ablations help explain why the recipe works: sampler matching brings large gains, and PRA reduces TEM-B FID from 42.76 to 19.86, versus 36.6 when aligning positives and negatives. Clear loss definitions and sampling descriptions make the method easier to assess. The cost accounting includes input gradients and second-order training work.

Critical weaknesses

The sampling result is specific to persistent evaluation: after 5,000 noise-started chains, later outputs reuse generated 40-step states. The protocol is disclosed, but Figure 5 provides only qualitative trajectories from independent noise starts. Separate quantitative results would clarify how much state reuse contributes to quality and cost.

The efficiency advantage is established against selected unguided diffusion checkpoints, with 250-step sampling in the inference comparison. Guided baselines achieve lower FID. Despite useful cost accounting, the results do not yet establish which advantages persist at comparable quality or with faster diffusion sampling.

Questions for authors

Could you clarify TEM's principal demonstrated or potential advantage over diffusion, whether training cost, inference efficiency, energy-based functionality, or another property, and delimit the conditions supported by the existing cost analysis?

As a focused validation of the sampling claim, could you add independent-noise-restart FID and sampling cost alongside the persistent results? This would quantify the effect of state reuse.

3 comments
AuthorsOpenReview ↗

Response to Review

We thank the reviewer for the clear and constructive review and for the positive assessment. We are encouraged that the reviewer regards the scaling results on ImageNet as a substantial empirical advance for explicit contrastive EBMs, and that the loss ablations and cost accounting were found useful. The requests to quantify the effect of state reuse and to delimit TEM’s advantages were well taken, and we respond to each below.

W1: The sampling result is specific to persistent evaluation: after 5,000 noise-started chains, later outputs reuse generated 40-step states. The protocol is disclosed, but Figure 5 provides only qualitative trajectories from independent noise starts. Separate quantitative results would clarify how much state reuse contributes to quality and cost.

Q2: As a focused validation of the sampling claim, could you add independent-noise-restart FID and sampling cost alongside the persistent results? This would quantify the effect of state reuse.

New Table 2 below reports TEM-XL with every chain initialized independently from N(0, I) without a replay buffer. These samples have high precision as much as samples from replay buffers but low recall. Each sample is high-fidelity, but the set covers few modes, whereas persistent sampling gives much higher recall. The buffer therefore does not improve fidelity but supplies diversity with condition flipping, providing structured seeds far from noise from which chains reach modes that short chains from noise do not. We have added a discussion of this in the new section 4.4 . The per-sample cost is the same under both protocols. Every chain runs exactly T Langevin steps regardless of initialization, so the throughput and FLOPs per sample in Table 9 hold for independent sampling as well.

Table 2 Sampling without a replay buffer.

InitializationStepsFID ↓IS ↑Precision ↑Recall ↑
Noise4014.91236.60.820.18
Noise6012.72303.90.880.15
Noise8013.12320.30.880.13
Noise10013.58311.00.870.12
Noise12014.18292.10.840.12
Replay buffer602.76230.40.800.57

W2 : The efficiency advantage is established against selected unguided diffusion checkpoints, with 250-step sampling in the inference comparison. Guided baselines achieve lower FID. Despite useful cost accounting, the results do not yet establish which advantages persist at comparable quality or with faster diffusion sampling.

We have revised the claims and tables accordingly. Table 7 adds guided DiT/SiT with precision, recall and FLOPs, Table 9 adds accelerated solvers, and the abstract, Section 4.1 and Appendix C state the conditions explicitly. TEM’s inference-cost advantage holds only against the standard 250-step unguided protocol. Guided diffusion reaches a lower FID and accelerated solvers at a lower cost(TEM: FID 2.76 at 14.48TFLOPs vs accelerated diffusion : FID 2.33 at 2.37TFLOPs). Two advantages persist regardless of the sampler. The first is lower training compute (Table 8), and the second is the energy-based functionality that diffusion does not provide, such as composition, inpainting, OOD detection and inference-time scaling from a single conditional model. We have narrowed the efficiency claim to these settings and leave accelerated samplers for EBMs as future work.

Q1. Could you clarify TEM's principal demonstrated or potential advantage over diffusion, whether training cost, inference efficiency, energy-based functionality, or another property, and delimit the conditions supported by the existing cost analysis?

We thank the reviewer for asking us to make this explicit. We now state explicitly where TEM has an advantage and where it does not. TEM’s demonstrated advantages are the energy-based applications (compositional generation, inference-time control, OOD detection and inpainting) and a lower unguided FID than DiT/SiT at lower training compute. However, it does not match guided diffusion in FID or accelerated diffusion in inference cost, and we do not claim this anymore. TEM also has clear future potential on the inference side. Our sampler is plain fixed-step Langevin, and acceleration techniques developed for diffusion, such as distillation and learned few-step samplers, have not yet been applied to EBMs. The revised Limitations paragraph now lists their EBM analogues, including amortized proposal networks, second-order or momentum-based samplers, and distilled few-step samplers, as directions for future work.

Reviewer_nTr4OpenReview ↗

Follow-up on sampling diversity and inference-time scaling

Thank you for the careful response and the additional experiments. The independent-noise sampling results and the expanded diffusion comparisons help clarify the role of the replay buffer and the scope of TEM’s advantages. I remain positive about the central contribution: demonstrating that contrastively trained EBMs can scale to high-resolution image generation. I have two further questions that would help strengthen the discussion.

  1. The substantially lower diversity with independent noise initialization is interesting and somewhat counterintuitive. Section 4.4 attributes this to chains converging to a few dominant modes, while replay-buffer initialization provides structured starting points. Could you explain more precisely why independent noise starts lead to poorer mode coverage, and provide supporting analysis or evidence? In particular, it would be helpful to understand the roles of short-run sampling dynamics, persistence during training, and condition switching in the replay buffer. A few representative cases or matched sampling trajectories in the paper would also help illustrate the mechanism.

  2. Table 7 shows that increasing sampling from 40 to 60 steps improves FID, but further increases reduce recall and worsen FID. For a fixed checkpoint, have you found an effective way to use additional inference-time compute to improve overall generation quality without substantially sacrificing diversity? It would be helpful to clarify whether useful scaling beyond this range has been demonstrated or remains a direction for future work.

I appreciate the authors’ efforts to address the concerns and am maintaining my clear-accept recommendation.

AuthorsOpenReview ↗

Response to follow-up

We thank the reviewer for the positive assessment. We are glad the expanded comparisons were useful. We also find both follow-up questions interesting, as they touch on interesting properties of TEM and that we think deserve a closer look, and we are happy to discuss them here.

  1. Could you explain more precisely why independent noise starts lead to poorer mode coverage?

We first give an overall explanation of the mechanism and then analyze how the image distributions of buffer-start and noise-start samples differ with examples.

Overall Explanation.

Diffusion and flow models are trained on noise-to-data trajectories, so their sampler is a learned transport map that, by construction, carries the full prior onto the data distribution. TEM has no such map. Samples are obtained by running Langevin dynamics on a learned energy.

With long chains Langevin dynamics would visit every mode in proportion to its probability, but a short chain settles into the modes nearest its starting point. This is compounded by training. The energy is shaped only near the manifold, where the buffered negatives lie, so the landscape around N(0,I)N(0,I) is an unconstrained extrapolation that funnels all noise starts toward a few dominant attractors per class, and the remaining modes are not reached within T steps.

The buffer helps with condition switching. A buffered sample of another class can sit close to a far mode of the target class, so re-conditioning it on the target reaches modes that noise starts cannot. Figure 11 in the revised paper illustrates this schematically.

Analysis on noise start vs. replay-buffer start. To quantify the qualitative difference between samples started from Gaussian noise (cluttered, texture-like backgrounds) and samples started from the replay buffer (cleaner backgrounds), we compute four simple statistics on 10K samples per setting and on 10K real ImageNet training images. Let I∈[0,1]HWI \in [0,1]^{HW} be the grayscale image (H=W=256):

  • Flat-region fraction (clean background): split I into K=256 non-overlapping 16x16 patches with pixel standard deviation σk\sigma_k. F=1K∑k1[σk<0.03]F = \frac{1}{K} \sum_{k} \mathbb{1}[\sigma_k < 0.03].

  • Distance from Gaussian noise: with z∈Rdz \in \mathbb{R}^d the SD-VAE latent of a sample (d=4096) and N(0,I)N(0,I), D=Ez,ε∥z−ε∥2/dD = \mathbb{E}_{z,\varepsilon}\lVert z - \varepsilon \rVert_2 / \sqrt{d}, averaged over 200K random (sample, noise) pairs.

  • Edge density (contours / clutter): fraction of pixels marked as edges by the Canny detector, E=1HW∑p1[p∈Canny(I)]E = \frac{1}{HW} \sum_{p} \mathbb{1}[p \in \mathrm{Canny}(I)].

  • PNG bits per pixel (overall visual complexity): B=8∣PNG(x)∣HWB=\frac{8|PNG(x)|}{HW}, the losslessly compressed size of the RGB image x.

Flat fraction FDistance from gaussian in latent space DEdge density EPNG bpp B
Real ImageNet0.2381.300.12014.08
Buffer start (ours, default)0.2661.320.10113.92
Noise start (no buffer)0.0821.270.16915.85

On all four measures the real-image statistics lie between the noise-start and buffer-start samples, so the two initializations are complementary, with noise starts over-represent texture-dominated images and buffer starts slightly cleaner ones. Buffer-start samples are much closer to the real statistics, which explains their closer match to the data distribution in FID and recall.

Examples For the class "lemon" and “snail", chains started from noise produce images (Fig 5 in the paper) filled with many lemons , or have high frequency background like asphalts, whereas chains started from buffered samples of other classes produce a few lemons or cleaner background inheriting the ancestor's layout. A noise start is high-frequency everywhere, so the short chain lands in the texture-dominated modes nearest to noise. However, a buffered seed already has a coherent layout, which the chain keeps while replacing the semantics. Figure 12 in the revised paper shows trajectories for this and two further classes and comparison with Figure 5(starting from noise for lemon and snail) shows this difference.

  1. For a fixed checkpoint, have you found an effective way to use additional inference-time compute to improve overall generation quality without substantially sacrificing diversity?

For a fixed checkpoint, we have not found a way to spend more compute beyond T =60 that improves FID without reducing recall. Possible directions include annealed step-size schedules, momentum-based samplers such as Hamiltonian Monte Carlo, and amortized proposals with neural networks. We think these are interesting directions for future work to move beyond the current trade-off curve by improving mode exploration.

Reviewer_A41UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Transformer Energy-Based Model that aims at improving the scaling of contrastively trained Energy-Based-Models (EBMs). To achieve this, it reduces the dimensionality of the input by operating in the latent space of Stable Diffusion, and uses the following components: (i) multiple energy heads at different layers of the ViT backbone (to capture features at different levels of abstraction) with the total energy being the sum of the energy heads across layers, (ii) REPA style loss applied only to the positive samples, (iii) the sampler matching loss of Du et al. (that aims at allowing for samples of low energy and high diversity) . The most interesting experimental results are the ones related to FID evaluation on ImageNet (256x256 and 512x512) without classifier free guidance.

Critical strengths

Positive points:

  • Strong experimental results for a contrastively trained EBM model
  • It aims at a challenging and interesting problem, which is the scaling of contrastively trained EBMs
  • The utilized components (sampler matching loss, multiple energy heads at different layers and REPA-style loss) appear to be reasonable additions that improve performance as evidenced by the ablations in the paper
Critical weaknesses

Negative points:

  • Still lacking significantly compared to CFG diffusion. Given that the model trains with label dropout, could TEM make use of guidance during inference ? What is the performance in this case ?
  • Inference time scaling can lead to loss of diversity and increased FID
  • During inference most of the samples continue from the replay buffer (that stores states of previous samples). As a result, late samples essentially inherit several earlier refinement updates. What is the sample quality if all samples were generated starting from random noise ?
Questions for authors

Address the issue raised in the paper weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Review

We thank the reviewer for the thoughtful review and for recognizing that scaling contrastively trained EBMs is a challenging and interesting problem, that the experimental results are strong for a contrastively trained EBM, and that the components we introduce are reasonable additions supported by the ablations. We respond to each concern below and have revised the paper accordingly.

W1. Still lacking significantly compared to CFG diffusion. Given that the model trains with label dropout, could TEM make use of guidance during inference ? What is the performance in this case ?

We find that TEM is still behind CFG-guided diffusion. While guidance can be applied to EBMs in the same way as in diffusion, by combining the conditional and unconditional models at sampling time, we tried this with our label-dropout TEM-XL. In practice the two energy landscapes are not calibrated to each other, so guided chains were unstable and had poor FID. Unlike diffusion, where the two score estimates share a noise level, an EBM has no shared scale to anchor the two energies. We believe that finding a stable method is an interesting direction for future work, and we report TEM without guidance. We have revised the limitations to discuss this.

W2. Inference time scaling can lead to loss of diversity and increased FID

More steps or a larger step size raises precision and lowers recall, with FID minimized at T = 60, similar trade-off as the guidance scale in diffusion (Table 7). We have revised the abstract and Section 4.3, which could mislead readers into thinking that more steps monotonically improve FID, to describe steps and step size as inference-time controls that trade diversity for fidelity.

W3. During inference most of the samples continue from the replay buffer (that stores states of previous samples). As a result, late samples essentially inherit several earlier refinement updates. What is the sample quality if all samples were generated starting from random noise ?

New Table 2 below reports TEM-XL with every chain initialized independently from N(0, I) without a replay buffer. These samples have high precision as much as samples from replay buffers but low recall. Each sample is high-fidelity, but the set covers few modes, whereas persistent sampling gives much higher recall. The buffer therefore does not improve fidelity but supplies diversity with condition flipping, providing structured seeds far from noise from which chains reach modes that short chains from noise do not. We have added a discussion of this in the new section 4.4 .

Table 2 Sampling without a replay buffer.

InitializationStepsFID ↓IS ↑Precision ↑Recall ↑
Noise4014.91236.60.820.18
Noise6012.72303.90.880.15
Noise8013.12320.30.880.13
Noise10013.58311.00.870.12
Noise12014.18292.10.840.12
Replay buffer602.76230.40.800.57
Reviewer_TyzYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors introduce a ViT-based energy-based model (EBM) called TEM that is contrastively trained in an SD-VAE latent space. TEM is trained with MCMC negative samples generated by its own short-run Langevin-style sampler and kept in a replay buffer. The model is trained with three objectives: the contrastive loss with L2 energy penalty; the sampler-matching regularizer motivated by the KL implemented with last-step backpropagation and a nearest-neighbor entropy surrogate; and Positive Representation Alignment (PRA) in the spirit of REPA, which consists of patch-wise alignment with DINOv2-L and is applied to data latents only, not to MCMC negative samples.

The model's total energy is the sum of scalar heads attached to the ViT at approximately L/3, 2L/3, and L. At inference time, TEM can trade off sample quality and diversity by tuning the number of steps and step size. The headline results use persistent evaluation with 60 sampling steps, achieving FID-50K scores of 3.82 on ImageNet-256 and 8.24 on ImageNet-512. The paper also evaluates compositional generation and OOD detection and demonstrates qualitative inpainting.

Critical strengths
  • A contrastive, explicit scalar energy model with self-generated MCMC negatives achieves competitive ImageNet-256 results even at the training-time sampling horizon: FID 7.63 and precision/recall of 0.65/0.65 (Table 4) for TEM-XL, comparable to unguided SiT-XL and DiT-XL. Earlier contrastive EBM results were largely concentrated on low-resolution ImageNet or less diverse datasets, including higher-resolution faces and bedrooms, so the results provide important evidence that this training regime can scale to diverse, high-resolution image synthesis. Furthermore, this work provides a potentially useful recipe for training modern contrastive EBMs: a ViT without spectral normalization, latent-space sampling with latent-norm clipping, and PRA.

  • The PRA analysis is neat and insightful. PRA on positive samples only decreases TEM-B FID from 42.8 to 19.9, while aligning both positive and negative samples decreases it only to 36.6. The authors explain that, unlike the noisy-input/clean-target alignment in REPA, TEM and the VFM process the same underlying clean data example in latent and image space, respectively; they motivate excluding negatives because these can be off-manifold, especially early in training. This is the most compelling piece of evidence for the paper at the component level, and a useful lesson for work that uses representation supervision to improve contrastive or EBM training.

Critical weaknesses
  • The claims about sample quality and efficiency need clearer qualification. Increasing TEM’s sampling steps makes its images sharper and more typical of the requested class, but also reduces their diversity. This resembles the trade-off produced by classifier-free guidance in diffusion models. However, the main efficiency comparisons use diffusion models without that guidance. The paper does report guided diffusion results elsewhere, and these achieve better FID at similar precision or recall. Therefore, the comparisons should include guided diffusion and measure computational cost at similar levels of image quality and diversity before drawing broader conclusions about efficiency. The abstract should also clarify that increasing the number of sampling steps improves FID only up to a point: further sampling can sharpen images while worsening the overall score, as the paper already acknowledges in its limitations. The results remain impressive for a contrastively trained EBM, but the claims should clearly state the settings under which TEM has an advantage.

  • The evaluation relies on reusing earlier generated samples. Most new samples start from saved intermediate states of previous generations, rather than independently from fresh noise. This is a valid sampling strategy, and the authors disclose it, but the reported results do not show how well TEM performs without this reuse. Outputs may also depend on one another through their shared sampling history. A comparison in which every sample starts independently from fresh noise would clarify how much the method relies on this strategy and make its comparison with diffusion models easier to interpret. The reported sampling cost is valid for the reuse-based procedure, but does not establish the cost of generating equally good samples independently.

  • The probabilistic interpretation needs clearer qualification. The practical sampler primarily follows the energy gradient, with a small amount of added noise, and the paper acknowledges its bias. Producing convincing images with this procedure does not establish that it samples accurately from the probability distribution defined by the energy function. This distinction matters for the claim that the experiments validate the theoretical advantages of EBMs. Adding energies still formally defines a product of distributions, and the sampler-matching objective remains meaningful, but the experiments demonstrate useful generation and composition rather than accurate sampling from those distributions. The paper should make this distinction clearer when discussing its theoretical claims.

Questions for authors
  1. Please report FID-50K, along with precision and recall, for TEM-XL with every chain initialized independently from N(0, I), no persistence, at T = 40, 60 and 120. App. B.2 specifies 5,000 initially noise-seeded chains; please confirm the evaluation buffer capacity and report how sensitive FID is to this capacity.

  2. Was T = 60 selected using FID-50K against the same reference statistics used for the reported results? What evaluation pipeline and reference set were used? Please compare TEM's step/step-size settings with CFG-guided DiT/SiT at similar precision or recall. For the inference-efficiency claim, how does TEM compare with diffusion baselines using fewer sampling steps or accelerated solvers at similar quality? This would clarify the scope of the claims about outperforming unguided diffusion with less compute.

  3. A main point of the paper is stable training at scale, yet the main ImageNet runs end at 50K training iterations. Does FID continue to improve, stabilize, or diverge with longer ImageNet training? App. B.1 explains the choice of latent-norm clipping thresholds (100/200) and qualitatively discusses their effect; could you quantify their impact on stability and FID, including whether training or sampling remains stable without clipping?

4 comments
AuthorsOpenReview ↗

Response to Review [1/2]

We thank the reviewer for their thoughtful review and for finding the research topic valuable. We appreciate the constructive concerns raised and respond to each below.

W1. Clarify the fidelity–diversity trade-off of inference-time scaling, to compare against CFG-guided diffusion at matched quality and diversity, and to specify the settings in which TEM has an advantage.

The abstract and Section 4.3 are revised to state that more sampling steps trade diversity for fidelity, with FID minimized at an intermediate setting rather than improving monotonically. For the comparison with diffusion, Table 7 (Table 4 in the initial submission) now reports TEM across sampling steps alongside DiT/SiT across guidance scales with precision, recall and FLOPs, and Table 9 adds accelerated solvers with guidance. At matched recall, guided DiT/SiT reach a lower FID than TEM, and accelerated solvers reach it at lower cost. The paper does not claim to outperform either, and all quality claims are relative to unguided baselines. We state the settings explicitly in Section 4.1. Using a single conditional model without guidance, TEM outperforms unguided DiT and SiT under the standard 250-step protocol in both FID and sampling cost, and at lower training compute. Our sampler is plain fixed-step Langevin, so there is substantial room for improvement, and accelerated samplers for EBMs are a natural direction for future work.

W2,Q1 : Results with every chain starting from noise, to clarify the role of the replay buffer at inference and the sensitivity to its capacity, as well as the sampling cost of generating equally good samples without state reuse.

New Table 2 below reports TEM-XL with every chain initialized independently from N(0, I) without a replay buffer. These samples have high precision as much as samples from replay buffers but low recall. Each sample is high-fidelity, but the set covers few modes, whereas persistent sampling gives much higher recall. The buffer therefore does not improve fidelity but supplies diversity with condition flipping, providing structured seeds far from noise from which chains reach modes that short chains from noise do not. We have added a discussion of this in the new section 4.4 .

Table 2 Sampling without a replay buffer

InitializationStepsFID ↓IS ↑Precision ↑Recall ↑
Noise4014.91236.60.820.18
Noise6012.72303.90.880.15
Noise8013.12320.30.880.13
Noise10013.58311.00.870.12
Noise12014.18292.10.840.12
Replay buffer602.76230.40.800.57

Table 3 Sensitivity to evaluation buffer capacity.

Buffer sizeFID ↓IS ↑Precision ↑Recall ↑
10002.77225.30.790.58
20002.68228.70.800.57
30002.69227.10.790.58
40002.71230.40.800.57
50002.76232.10.800.57

New Table 3 above varies the size of replay buffer at inference from 1,000 to 5,000. FID changes by less than 0.1 and precision/recall by less than 0.01, so the result is insensitive to this choice. The per-sample cost is the same under both protocols. Every chain runs exactly T Langevin steps regardless of initialization, so the throughput and FLOPs per sample in Table 9 hold for independent sampling as well.

W3. State that our short-run sampler demonstrates useful generation and composition, not exact sampling from the energy-defined distribution.

Our sampler is a short-run Langevin chain, and its output is a biased approximation of the model distribution. Our experiments show that the learned energy supports high-quality generation and some applications of EBM under this sampler, not that it samples the energy-defined distribution exactly. We have revised the abstract, Sections 4 to state this distinction explicitly.

AuthorsOpenReview ↗

Response to Review [2/2]

Q2. Confirm which evaluation pipeline was used and make comparisons against CFG-guided and accelerated diffusion at matched quality to clarify the scope of the efficiency claim.

We use ADM ImageNet-256 training reference batch and evaluation pipeline, the same evaluation used by DiT, SiT and REPA and number of sampling steps T is selected based on this pipeline. The comparison with classifier-free guidance was already included in the first submission (Table 4), which reported precision and recall for TEM-XL across sampling steps alongside DiT-XL/2 across guidance scales. In the revision this is Table 7, extended with step-size variants, SiT-XL/2 rows and FLOPs per sample. Accelerated-solver baselines [1],[2],[3] are compared on cost in Table 9. At matched recall, TEM is slightly worse in FID than guided DiT/SiT. The paper does not claim to outperform guided diffusion, and all quality claims were stated relative to unguided baselines in the original submission. We note that TEM reaches this operating point with a single conditional model, whereas guidance combines conditional and unconditional evaluations.

[1] Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." Machine Intelligence Research 22.4 (2025): 730-751.

[2] Zhao, Wenliang, et al. "Unipc: A unified predictor-corrector framework for fast sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2023): 49842-49869.

[3] Wang, Shuai, et al. "Differentiable solver search for fast diffusion sampling." arXiv preprint arXiv:2505.21114 (2025).

Q3. Report training beyond current training steps, and for a quantitative account of latent-norm clipping’s effect on stability and FID, including without clipping.

Longer training. Training remains stable beyond 50K iterations and FID continues to improve slightly, reaching 2.76 after 125K with half batch size (1.25X the training compute). FID plateaus after 125K iterations.

Iteration12.5K25K50K75K100K112.5K125K137.5K150K
FID ↓274.421.145.113.383.062.952.762.772.76
IS ↑3.092.0176.1213.6221.8222.0230.4234.7234.2
Precision ↑0.000.590.790.810.800.780.800.800.80
Recall ↑0.000.390.530.550.550.580.570.560.57

Latent-norm clipping. Without clipping, training becomes unstable at a very early stage: before the energy landscape is shaped, the sampler’s chains wander to latents with very large norms far from the data manifold, and training diverges. Clipping keeps the negatives within the range of the latent distribution during this phase and is required for stable training. Once the energy landscape has been shaped even partially, sampling stabilizes, and at inference it is robust to the threshold. New Table 6 below varies the clipping threshold for ImageNet-256, and FID changes by less than 0.2 even with clipping removed entirely.

Table 6 Effect of the latent-norm bound at inference.

BoundFID ↓IS ↑Precision ↑Recall ↑
1002.76230.40.800.57
2002.90227.20.800.57
5002.90227.00.800.57
None2.90227.00.800.57

Thank you for the response. The new experiments answer my questions. Tables 7 and 9 make the comparison with guided and accelerated diffusion transparent. Table 2 shows that noise-initialized chains have low recall, so diversity comes from the persistent buffer.

Two points remain open:

The Introduction (contribution 3, “validate the core theoretical promises”) and the Conclusion still claim to validate EBM theory. I could not find the clarification promised for Section 4. Please align these with the revised abstract. Section 4.4 should say that the results depend on the persistent buffer, rather than that the concern about limited diversity is resolved.

I maintain my recommendation of weak acceptance.

AuthorsOpenReview ↗

Response to comment

Thank you for checking the revision carefully and all your valuable comments. We have revised the paper again to remove the remaining inconsistencies you pointed out.

Introduction and conclusion

Contribution 3 in the Introduction now reads “We further demonstrate the applications of EBMs, inference-time control, compositional generation, out-of-distribution detection and inpainting, at scale,” and the Conclusion states that these applications “can be realized at this resolution.” Neither claims to validate EBM theory, matching the revised abstract.

Section 4

The starting sentence of Section 4.4 clearly states “TEM samples with persistent short-run chains and a replay buffer, so the results in this paper reflect generation under this sampler rather than exact sampling from the model distribution.”

The closing sentences of Section 4.4 now state that TEM’s reported diversity depends on the replay buffer, which we include to compensate for the limited mode coverage of short-run Langevin chains.

Please let us know if you have any other remaining concerns. We will make sure to address and fix them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All [1/2]

We thank all reviewers for their careful and constructive reviews. We are encouraged that the reviewers agree on the significance of the problem and the strength of the results, and we have used their feedback to substantially strengthen the paper. Below we summarize the strengths noted by the reviewers, our responses to the shared concerns, and other changes. Detailed responses follow under each review. Changes in the revised PDF are shown in blue.

Strengths

1. Significance of scaling contrastive EBMs and strong empirical results.

Reviewer A41U notes that the paper “aims at a challenging and interesting problem, which is the scaling of contrastively trained EBMs.” Reviewer TyzY observes that earlier contrastive EBM results “were largely concentrated on low-resolution ImageNet or less diverse datasets,” so that our results “provide important evidence that this training regime can scale to diverse, high-resolution image synthesis,” and highlights that TEM is competitive with unguided DiT/SiT even at the training-time sampling horizon. Reviewer nTr4 describes the scaling result as “a substantial empirical advance for explicit contrastive EBMs” that “provides a useful route for further EBM research.”

2. Useful EBM training strategies with insightful analysis and ablations.

Reviewer TyzY finds that the paper “provides a potentially useful recipe for training modern contrastive EBMs” and calls the PRA analysis “neat and insightful” and “the most compelling piece of evidence for the paper at the component level,” noting it as “a useful lesson for work that uses representation supervision.” Reviewer nTr4 writes that the loss ablations “help explain why the recipe works,” and that the cost accounting “includes input gradients and second-order training work.” Reviewer A41U finds the components “reasonable additions that improve performance as evidenced by the ablations.”

Responses to reviewer’s concern

1. Role of the replay buffer.

All reviewers (TyzY W2/Q1, A41U W3, nTr4 W1/Q2) asked for results without the buffer. New Table 2 below shows that noise-initialized chains still produce high-quality samples, matching persistent sampling in precision, but with much lower recall: TEM generates good images without the buffer, and the buffer’s role is to help the sampler reach more diverse modes (new Section 4.4). New Table 3 below shows FID is insensitive to buffer capacity (TyzY), and per-sample cost is identical under both protocols since every chain runs exactly T steps (nTr4).

Table 2 Sampling without a replay buffer

InitializationStepsFID ↓IS ↑Precision ↑Recall ↑
Noise4014.91236.60.820.18
Noise6012.72303.90.880.15
Noise8013.12320.30.880.13
Noise10013.58311.00.870.12
Noise12014.18292.10.840.12
Replay buffer602.76230.40.800.57

Table 3 Sensitivity to evaluation buffer capacity.

Buffer sizeFID ↓IS ↑Precision ↑Recall ↑
10002.77225.30.790.58
20002.68228.70.800.57
30002.69227.10.790.58
40002.71230.40.800.57
50002.76232.10.800.57

2. Comparison with guided and accelerated diffusion.

Reviewers TyzY (W1/Q2), A41U (W1/W2) and nTr4 (W2/Q1) asked how inference-time scaling compares with guidance and fast solvers. Table 7 now includes precision/recall and FLOPs for TEM alongside guided DiT/SiT, and Table 9 adds accelerated solvers. TEM beats unguided 250-step diffusion at lower cost but does not match guided diffusion or accelerated solvers. We have limited our claims to the unguided, non-accelerated setting and believe that matching these stronger variants is an interesting direction of future work (which we have also revised the conclusion to discuss). Reviewer A41U asked whether TEM can use guidance. It is possible in principle, but the conditional and unconditional energy landscapes are not calibrated to each other, which makes guided generation chains unstable, so we report TEM without guidance.

AuthorsOpenReview ↗

Response to All [2/2]

3. Longer training and latent-norm clipping (TyzY Q3).

Training is stable beyond 50K iterations with slight FID improvement reaching 2.76 after 125K with half batch size (1.25X the training compute).

Iteration12.5K25K50K75K100K112.5K125K
FID ↓274.421.145.113.383.062.952.76
IS ↑3.092.0176.1213.6221.8222.0230.4
Precision ↑0.000.590.790.810.800.780.80
Recall ↑0.000.390.530.550.550.580.57

New Table 6 below shows inference is robust to the clipping threshold, while clipping is needed early in training. Table 6 Effect of the latent-norm bound at inference.

BoundFID ↓IS ↑Precision ↑Recall ↑
1002.76230.40.800.57
2002.90227.20.800.57
5002.90227.00.800.57
None2.90227.00.800.57

4. Scope of claims and probabilistic interpretation (nTr4 Q1, TyzY W3). We now state explicitly where TEM has an advantage and where it does not. TEM’s demonstrated advantages are the energy-based applications (compositional generation, inference-time control, OOD detection and inpainting) and a lower unguided FID than DiT/SiT at lower training compute. However, it does not match guided diffusion in FID or accelerated diffusion in inference cost, and we no longer claim otherwise. We also clarify that our short-run Langevin sampler produces high-quality samples but not exact samples from the energy-defined distribution, so our results demonstrate useful generation and composition under this sampler rather than validating the theoretical properties of EBMs.

Other changes

A second TEM-XL trained with half the batch size (128) for 2.5× the steps (125K), so it sees 1.25× the images at 1.25× the training FLOPs of the submitted model, reaches FID 2.76. It is added to Tables 1 and 8 and used for the new ablations in Tables 2, 3 and 6. Table 1 is also updated with recent baselines. We would be glad to use 2.76 as the headline in the camera-ready if the reviewers consider that appropriate.

We hope these revisions address the reviewers' concerns, and we are happy to provide further clarification during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.