- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines the problem of training a class-conditional EBM generative model using short-run sampling from a random initialization. The JEM framework for EBM modelling is used. The key motivation is to enable short-run trajectories to cover a wider distribution of the pixel space than prior work, which typically uses single-shot trajectories from pure noise. The primary method introduced in this work is to run an initial trajectory using a randomly selected class that differs from the target for the first part of the trajectory, followed by a final phase that uses the intended class label. The initial trajectory with a different label is meant to provide a more helpful initial state for the final trajectory than a single-shot short-run from random noise alone provides. The work also explores replacing an i.i.d. Gaussian or uniform initialization with an informative Gaussian initialization that captures statistics of the training data to provide a better initialization. Training and sampling follows the recent DAT work, which scales the energy gradients of positive and negative samples in the loss and replaces Langevin sampling with normalized gradient descent for stability, along with an auxiliary AT loss. Results show strong generative performance for pixel-space ImageNet 256x256, along with OOD and robustness of the learned model.
Critical strengths
- The method presents a straightforward and effective recipe for training EBMs with strong generative performance on complex distributions. To my knowledge, these are the strongest results achieved by a generative EBM on ImageNet 256x256 in the pixel space.
- The model also shows good adversarial robustness and OOD performance.
Critical weaknesses
- The method is currently limited to the class-conditional setting, and relies on several components that seem to heavily require class conditions, including the cross-label warm start and auxiliary training loss. This is a critical weakness. It is not clear how the method could be applied to even conventional simple situations like unconditional CIFAR-10. Additionally, there are only experiments on class-conditional ImageNet. It is worth exploring other datasets like class-conditional CIFAR-10, where evaluation might be more sensitive to more localized errors given the lower resolution.
- There is not an especially high degree of novelty in the current work, which is mostly an initialization-based adjustment. This is somewhat mitigated by the strength of the empirical results.
Questions for authors
- How does the method perform on other (possibly simpler) benchmarks, like CIFAR-10? This would provide a more comprehensive view of the sampler's abilities, in particular for generating fine-grained details.
- Is the AT auxiliary loss essential? Can an ablation be provided?
- Why is the sampler method referred to as PGD? There doesn't seem to be a projection step. Is the gradient normalization essential?
- Can you provide at least rough estimates of the relative costs of training and sampling from this model compared to typical baselines like diffusion models? In particular, the training-time cost of MCMC sampling for EBMs is known to be high.
4 comments
Author Response (Part 2/2)
Auxiliary classification loss
Is the AT auxiliary loss essential? Can an ablation be provided?
We have added an ImageNet64 ablation to test whether classification supervision and explicit gradient regularization can replace the AT auxiliary loss (Appendix A.4, Table 10). We use clean cross-entropy (CE) for classification and an penalty on the input gradients of the conditional energy at real images. We test and , retaining the same AT-pretrained initialization for all variants.
At the FID-selected checkpoints, the replacements give FIDs of 7.10 and 5.09, respectively, versus 2.39 with AT. Both improve clean accuracy, but PGD-6 robust accuracy falls from 25.81% to approximately 5–6%. Their generation quality also deteriorates sharply later in training while clean accuracy remains near 75%. The tested replacements therefore do not preserve the generation quality and training stability obtained with AT, despite maintaining high clean accuracy.
We also tested , but did not observe coherent generation with target-only sampling at any stage of training (the best target-only FID was 92.48, achieved at 1.1M training images). We therefore did not evaluate relay sampling for this run.
Training and sampling cost
Can you provide at least rough estimates of the relative costs of training and sampling from this model compared to typical baselines like diffusion models? In particular, the training-time cost of MCMC sampling for EBMs is known to be high.
We have added training and sampling cost comparisons in Appendix A.8 (Table 14). Our ImageNet256 relay model achieves FID 3.05 with 61 NGD steps from noise. We report 207 A100-hours for EBM training and a generation throughput of 3.82 images/s per A100. For reference, LDM-4-G reports 6,504 V100-equivalent training hours and 0.40 images/s per A100 (Rombach et al., 2022). The training costs exclude AT and autoencoder pretraining, respectively.
Possible extension to unconditional generation
It is not clear how the method could be applied to even conventional simple situations like unconditional CIFAR-10.
For CIFAR-10 without class annotations, one possible extension is to obtain pseudo-labels by clustering image features and use these cluster IDs for both JEM's conditional energies and its auxiliary classification loss (see Liu et al., 2020, for an example of cluster-based conditioning). Relay would construct warm starts under an auxiliary-cluster energy and refine them under a target-cluster energy. For unconditional generation, we could sample target clusters according to their training-set frequencies and auxiliary clusters uniformly, then discard the cluster IDs. The empirical effectiveness of this extension remains to be evaluated.
References:
Liu, S., Wang, T., Bau, D., Zhu, J.-Y., & Torralba, A. (2020). Diverse Image Generation via Self-Conditioned GANs. CVPR.
Thanks to the authors for their comprehensive responses. I am inclined to stick with my score, although I believe there are still some weaknesses. The method works quite well for class-conditional ImageNet but does not perform especially well on class-conditional CIFAR-10 and cannot handle any unconditional distribution without further algorithm adjustments. While the ImageNet results are impressive, it makes me wonder whether this method can server as a foundational sampler or just a specialized sampler for certain situations. As Reviewer Q3CG pointed out, the cross-label sampling method has been in the literature for a long time since the IGEBM paper. DAT and noise initialization are also established in prior work. The methodological contributions are relatively light and the paper rests heavily on its ImageNet empirical result. Overall the paper is reasonable in its current form but it would be nice to see diffusion-level results on unconditional CIFAR-10 for example as evidence of ability to succeed in every situation where other strong methods like GANs and diffusions also do.
Thank you for the careful assessment and for recognizing the ImageNet results. Regarding cross-label persistent replay, relay sampling follows a different design principle: we specifically construct structured warm starts to improve short-run coverage, using intermediate states generated at a prescribed depth from noise. Persistent replay instead continues earlier sampling chains through buffered states, and Du & Mordatch’s test-time sampler similarly continues a long chain across occasional random label changes (see our detailed reply to Reviewer Q3CG). We have revised Related Work, Section 3.1, Table 1, and Appendix B.2 to clarify this distinction and compare the methods on ImageNet64 and in the controlled problem in Appendix B. DAT and uniform-noise initialization are shared by our target-only baseline and relay method, so these comparisons assess the additional benefit of the relay construction.
Our contribution targets short-run coverage in class-conditional EBMs, with the strongest empirical evidence on ImageNet. We acknowledge that the more limited CIFAR-10 results constrain broader empirical claims, and that extending relay to unconditional generation would require further methodological development and validation.
Author Response (Part 1/2)
We thank Reviewer Yjj5 for their helpful feedback. We have revised the sampler terminology, added ablations of gradient normalization and the auxiliary classification loss, and included training and sampling cost comparisons. We also provide preliminary CIFAR-10 results in this response. Below we address your specific concerns.
CIFAR-10 benchmark
In our preliminary CIFAR-10 experiments, relay training has been less stable than on ImageNet, with only limited gains under the configurations evaluated so far. We therefore tested a separate-model extension, using samples generated by the target-only model (M1) as warm starts for an independently trained refinement model (M2). At generation time, M1 constructs an auxiliary-label warm start and M2 refines it under the target label. These results support our core idea of constructing cross-label warm starts for target-conditioned refinement: the separate-model implementation improves both generation quality and coverage over the target-only baseline.
Training setup. We use ConvNeXt-Base with ConvStem (Singh et al., 2023) for all methods, with 87.7M parameters per network. All models use AT initialization and an AT auxiliary loss, normalized gradient descent (NGD) with training step size η=0.3, AutoAugment + Cutout for classification, and random crops and horizontal flips for real images in the generative loss.
Sampler terminology: PGD and NGD
We agree that the PGD name, inherited from DAT (Yin et al., 2026), was imprecise in this setting. Our sampler uses normalized gradient steps followed by clipping to the valid pixel range; unlike the PGD attack (Madry et al., 2018), it does not constrain samples to an ϵ-ball around the initial input. We have therefore renamed the sampler normalized gradient descent (NGD) throughout the paper and clarified the update in Section 4.1 (Sampler).
Effect of gradient normalization
Gradient normalization is not essential for our training and sampling procedure in the evaluated ImageNet64 setting. We have added an ablation replacing NGD with unnormalized GD in both training and generation (Appendix A.5, Table 11). GD achieves FID 2.62 versus 2.40 for NGD, with higher recall and lower precision. Although GD achieves comparable generation quality, we observe less stable training: its proposal AUC shows a transient rise around 0.6–0.8M training images before recovering, whereas NGD’s remains steadier (Figure 4).
References:
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.