- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TEM scales contrastively trained scalar energy-based models using a latent-space transformer, multiple energy heads, sampler matching, and positive-only representation alignment (PRA). TEM-XL reports FID-50K of 3.82 on ImageNet 256 and 8.24 on ImageNet 512, using 60-step persistent sampling without CFG. Ablations examine the contributions of the loss terms, energy heads, and representation-alignment choices. Additional experiments evaluate compositional generation and OOD detection, illustrate inpainting, and report training and inference costs.
Critical strengths
The ImageNet results and scaling across four model sizes are a substantial empirical advance for explicit contrastive EBMs. The successful integration provides a useful route for further EBM research.
The loss ablations help explain why the recipe works: sampler matching brings large gains, and PRA reduces TEM-B FID from 42.76 to 19.86, versus 36.6 when aligning positives and negatives. Clear loss definitions and sampling descriptions make the method easier to assess. The cost accounting includes input gradients and second-order training work.
Critical weaknesses
The sampling result is specific to persistent evaluation: after 5,000 noise-started chains, later outputs reuse generated 40-step states. The protocol is disclosed, but Figure 5 provides only qualitative trajectories from independent noise starts. Separate quantitative results would clarify how much state reuse contributes to quality and cost.
The efficiency advantage is established against selected unguided diffusion checkpoints, with 250-step sampling in the inference comparison. Guided baselines achieve lower FID. Despite useful cost accounting, the results do not yet establish which advantages persist at comparable quality or with faster diffusion sampling.
Questions for authors
Could you clarify TEM's principal demonstrated or potential advantage over diffusion, whether training cost, inference efficiency, energy-based functionality, or another property, and delimit the conditions supported by the existing cost analysis?
As a focused validation of the sampling claim, could you add independent-noise-restart FID and sampling cost alongside the persistent results? This would quantify the effect of state reuse.
3 comments
Follow-up on sampling diversity and inference-time scaling
Thank you for the careful response and the additional experiments. The independent-noise sampling results and the expanded diffusion comparisons help clarify the role of the replay buffer and the scope of TEM’s advantages. I remain positive about the central contribution: demonstrating that contrastively trained EBMs can scale to high-resolution image generation. I have two further questions that would help strengthen the discussion.
-
The substantially lower diversity with independent noise initialization is interesting and somewhat counterintuitive. Section 4.4 attributes this to chains converging to a few dominant modes, while replay-buffer initialization provides structured starting points. Could you explain more precisely why independent noise starts lead to poorer mode coverage, and provide supporting analysis or evidence? In particular, it would be helpful to understand the roles of short-run sampling dynamics, persistence during training, and condition switching in the replay buffer. A few representative cases or matched sampling trajectories in the paper would also help illustrate the mechanism.
-
Table 7 shows that increasing sampling from 40 to 60 steps improves FID, but further increases reduce recall and worsen FID. For a fixed checkpoint, have you found an effective way to use additional inference-time compute to improve overall generation quality without substantially sacrificing diversity? It would be helpful to clarify whether useful scaling beyond this range has been demonstrated or remains a direction for future work.
I appreciate the authors’ efforts to address the concerns and am maintaining my clear-accept recommendation.
Response to follow-up
We thank the reviewer for the positive assessment. We are glad the expanded comparisons were useful. We also find both follow-up questions interesting, as they touch on interesting properties of TEM and that we think deserve a closer look, and we are happy to discuss them here.
- Could you explain more precisely why independent noise starts lead to poorer mode coverage?
We first give an overall explanation of the mechanism and then analyze how the image distributions of buffer-start and noise-start samples differ with examples.
Overall Explanation.
Diffusion and flow models are trained on noise-to-data trajectories, so their sampler is a learned transport map that, by construction, carries the full prior onto the data distribution. TEM has no such map. Samples are obtained by running Langevin dynamics on a learned energy.
With long chains Langevin dynamics would visit every mode in proportion to its probability, but a short chain settles into the modes nearest its starting point. This is compounded by training. The energy is shaped only near the manifold, where the buffered negatives lie, so the landscape around is an unconstrained extrapolation that funnels all noise starts toward a few dominant attractors per class, and the remaining modes are not reached within T steps.
The buffer helps with condition switching. A buffered sample of another class can sit close to a far mode of the target class, so re-conditioning it on the target reaches modes that noise starts cannot. Figure 11 in the revised paper illustrates this schematically.
Analysis on noise start vs. replay-buffer start. To quantify the qualitative difference between samples started from Gaussian noise (cluttered, texture-like backgrounds) and samples started from the replay buffer (cleaner backgrounds), we compute four simple statistics on 10K samples per setting and on 10K real ImageNet training images. Let be the grayscale image (H=W=256):
-
Flat-region fraction (clean background): split I into K=256 non-overlapping 16x16 patches with pixel standard deviation . .
-
Distance from Gaussian noise: with the SD-VAE latent of a sample (d=4096) and , , averaged over 200K random (sample, noise) pairs.
-
Edge density (contours / clutter): fraction of pixels marked as edges by the Canny detector, .
-
PNG bits per pixel (overall visual complexity): , the losslessly compressed size of the RGB image x.
| Flat fraction F | Distance from gaussian in latent space D | Edge density E | PNG bpp B | |
|---|---|---|---|---|
| Real ImageNet | 0.238 | 1.30 | 0.120 | 14.08 |
| Buffer start (ours, default) | 0.266 | 1.32 | 0.101 | 13.92 |
| Noise start (no buffer) | 0.082 | 1.27 | 0.169 | 15.85 |
On all four measures the real-image statistics lie between the noise-start and buffer-start samples, so the two initializations are complementary, with noise starts over-represent texture-dominated images and buffer starts slightly cleaner ones. Buffer-start samples are much closer to the real statistics, which explains their closer match to the data distribution in FID and recall.
Examples For the class "lemon" and “snail", chains started from noise produce images (Fig 5 in the paper) filled with many lemons , or have high frequency background like asphalts, whereas chains started from buffered samples of other classes produce a few lemons or cleaner background inheriting the ancestor's layout. A noise start is high-frequency everywhere, so the short chain lands in the texture-dominated modes nearest to noise. However, a buffered seed already has a coherent layout, which the chain keeps while replacing the semantics. Figure 12 in the revised paper shows trajectories for this and two further classes and comparison with Figure 5(starting from noise for lemon and snail) shows this difference.
- For a fixed checkpoint, have you found an effective way to use additional inference-time compute to improve overall generation quality without substantially sacrificing diversity?
For a fixed checkpoint, we have not found a way to spend more compute beyond T =60 that improves FID without reducing recall. Possible directions include annealed step-size schedules, momentum-based samplers such as Hamiltonian Monte Carlo, and amortized proposals with neural networks. We think these are interesting directions for future work to move beyond the current trade-off curve by improving mode exploration.
Response to Review
We thank the reviewer for the clear and constructive review and for the positive assessment. We are encouraged that the reviewer regards the scaling results on ImageNet as a substantial empirical advance for explicit contrastive EBMs, and that the loss ablations and cost accounting were found useful. The requests to quantify the effect of state reuse and to delimit TEM’s advantages were well taken, and we respond to each below.
New Table 2 below reports TEM-XL with every chain initialized independently from N(0, I) without a replay buffer. These samples have high precision as much as samples from replay buffers but low recall. Each sample is high-fidelity, but the set covers few modes, whereas persistent sampling gives much higher recall. The buffer therefore does not improve fidelity but supplies diversity with condition flipping, providing structured seeds far from noise from which chains reach modes that short chains from noise do not. We have added a discussion of this in the new section 4.4 . The per-sample cost is the same under both protocols. Every chain runs exactly T Langevin steps regardless of initialization, so the throughput and FLOPs per sample in Table 9 hold for independent sampling as well.
Table 2 Sampling without a replay buffer.
We have revised the claims and tables accordingly. Table 7 adds guided DiT/SiT with precision, recall and FLOPs, Table 9 adds accelerated solvers, and the abstract, Section 4.1 and Appendix C state the conditions explicitly. TEM’s inference-cost advantage holds only against the standard 250-step unguided protocol. Guided diffusion reaches a lower FID and accelerated solvers at a lower cost(TEM: FID 2.76 at 14.48TFLOPs vs accelerated diffusion : FID 2.33 at 2.37TFLOPs). Two advantages persist regardless of the sampler. The first is lower training compute (Table 8), and the second is the energy-based functionality that diffusion does not provide, such as composition, inpainting, OOD detection and inference-time scaling from a single conditional model. We have narrowed the efficiency claim to these settings and leave accelerated samplers for EBMs as future work.
We thank the reviewer for asking us to make this explicit. We now state explicitly where TEM has an advantage and where it does not. TEM’s demonstrated advantages are the energy-based applications (compositional generation, inference-time control, OOD detection and inpainting) and a lower unguided FID than DiT/SiT at lower training compute. However, it does not match guided diffusion in FID or accelerated diffusion in inference cost, and we do not claim this anymore. TEM also has clear future potential on the inference side. Our sampler is plain fixed-step Langevin, and acceleration techniques developed for diffusion, such as distillation and learned few-step samplers, have not yet been applied to EBMs. The revised Limitations paragraph now lists their EBM analogues, including amortized proposal networks, second-order or momentum-based samplers, and distilled few-step samplers, as directions for future work.