acceptodds
Under review as a conference paper at ICLR 2027

Relay Sampling: Cross-Label Warm Starts for Short-Run Joint Energy Models

Abstract

Deep energy-based models commonly rely on short-run samplers from random noise for training and generation, but their finite-step distributions remain dependent on the initialization and may cover only a limited subset of the model distribution. In joint energy models, each label defines a conditional energy surface, yet the label is typically used only as the final generation target. We introduce relay sampling, which also uses label conditions to diversify the path from noise to the target class. The sampler first follows a random auxiliary label to construct a structured intermediate state, which we call a cross-label warm start, and then passes this state to the target-conditioned sampler. This procedure replaces the single target-only path from noise with a mixture of auxiliary-conditioned finite-step paths, and we hypothesize that training with negatives from this mixture yields a model that covers more of each class. On ImageNet 64x64, models trained with relayed negatives reach substantially higher recall and much better FID than models trained with target-only negatives. On ImageNet 256x256, relay training of a ConvNeXt-Large model, in pixel space and from random noise, yields an FID of 3.05, below diffusion and autoregressive baselines that use 1.6 to 3 times as many parameters, while retaining robust classification, OOD detection, and calibration.

Primary area:
generative models
Keywords:
energy-based models, joint energy models, short-run MCMC, generative models, ImageNet

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Yjj5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the problem of training a class-conditional EBM generative model using short-run sampling from a random initialization. The JEM framework for EBM modelling is used. The key motivation is to enable short-run trajectories to cover a wider distribution of the pixel space than prior work, which typically uses single-shot trajectories from pure noise. The primary method introduced in this work is to run an initial trajectory using a randomly selected class that differs from the target for the first part of the trajectory, followed by a final phase that uses the intended class label. The initial trajectory with a different label is meant to provide a more helpful initial state for the final trajectory than a single-shot short-run from random noise alone provides. The work also explores replacing an i.i.d. Gaussian or uniform initialization with an informative Gaussian initialization that captures statistics of the training data to provide a better initialization. Training and sampling follows the recent DAT work, which scales the energy gradients of positive and negative samples in the loss and replaces Langevin sampling with normalized gradient descent for stability, along with an auxiliary AT loss. Results show strong generative performance for pixel-space ImageNet 256x256, along with OOD and robustness of the learned model.

Critical strengths
  • The method presents a straightforward and effective recipe for training EBMs with strong generative performance on complex distributions. To my knowledge, these are the strongest results achieved by a generative EBM on ImageNet 256x256 in the pixel space.
  • The model also shows good adversarial robustness and OOD performance.
Critical weaknesses
  • The method is currently limited to the class-conditional setting, and relies on several components that seem to heavily require class conditions, including the cross-label warm start and auxiliary training loss. This is a critical weakness. It is not clear how the method could be applied to even conventional simple situations like unconditional CIFAR-10. Additionally, there are only experiments on class-conditional ImageNet. It is worth exploring other datasets like class-conditional CIFAR-10, where evaluation might be more sensitive to more localized errors given the lower resolution.
  • There is not an especially high degree of novelty in the current work, which is mostly an initialization-based adjustment. This is somewhat mitigated by the strength of the empirical results.
Questions for authors
  • How does the method perform on other (possibly simpler) benchmarks, like CIFAR-10? This would provide a more comprehensive view of the sampler's abilities, in particular for generating fine-grained details.
  • Is the AT auxiliary loss essential? Can an ablation be provided?
  • Why is the sampler method referred to as PGD? There doesn't seem to be a projection step. Is the gradient normalization essential?
  • Can you provide at least rough estimates of the relative costs of training and sampling from this model compared to typical baselines like diffusion models? In particular, the training-time cost of MCMC sampling for EBMs is known to be high.
4 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank Reviewer Yjj5 for their helpful feedback. We have revised the sampler terminology, added ablations of gradient normalization and the auxiliary classification loss, and included training and sampling cost comparisons. We also provide preliminary CIFAR-10 results in this response. Below we address your specific concerns.

CIFAR-10 benchmark

How does the method perform on other (possibly simpler) benchmarks, like CIFAR-10? This would provide a more comprehensive view of the sampler's abilities, in particular for generating fine-grained details.

In our preliminary CIFAR-10 experiments, relay training has been less stable than on ImageNet, with only limited gains under the configurations evaluated so far. We therefore tested a separate-model extension, using samples generated by the target-only model (M1) as warm starts for an independently trained refinement model (M2). At generation time, M1 constructs an auxiliary-label warm start and M2 refines it under the target label. These results support our core idea of constructing cross-label warm starts for target-conditioned refinement: the separate-model implementation improves both generation quality and coverage over the target-only baseline.

Training setup. We use ConvNeXt-Base with ConvStem (Singh et al., 2023) for all methods, with 87.7M parameters per network. All models use AT initialization and an AT auxiliary loss, normalized gradient descent (NGD) with training step size η=0.3\eta=0.3, AutoAugment + Cutout for classification, and random crops and horizontal flips for real images in the generative loss.

  • Target-only training: Uniform-noise initialization, a training depth T=110T=110, and a clean-CE weight of 1.
  • Relay training: A single model uses (T;T1,T2)=(150;120,60)(T;T_1,T_2)=(150;120,60), ρ=0.5\rho=0.5, and a clean-CE weight of 0. Its AT auxiliary loss is applied to both AA (AutoAugment + Cutout) and original (random crops and horizontal flips) views.
  • Relay training (separate models): We freeze the target-only model to generate 100K warm-start samples, with a sampling budget of 100 steps. We train an independently initialized M2 from this fixed pool, randomly pairing warm-start samples with target labels and using 40 target-conditioned steps. M2 uses a clean-CE weight of 1.
MethodTest sampling steps (η=0.6\eta=0.6)FID50K ↓IS ↑Precision ↑Recall ↑
Target-only training639.189.700.5870.439
Relay training58 + 317.509.820.6070.444
Relay training (separate models)50 + 264.3110.350.6280.528

Sampler terminology: PGD and NGD

Why is the sampler method referred to as PGD? There doesn't seem to be a projection step.

We agree that the PGD name, inherited from DAT (Yin et al., 2026), was imprecise in this setting. Our sampler uses normalized gradient steps followed by clipping to the valid pixel range; unlike the PGD attack (Madry et al., 2018), it does not constrain samples to an ϵ\epsilon-ball around the initial input. We have therefore renamed the sampler normalized gradient descent (NGD) throughout the paper and clarified the update in Section 4.1 (Sampler).

Effect of gradient normalization

Is the gradient normalization essential?

Gradient normalization is not essential for our training and sampling procedure in the evaluated ImageNet64 setting. We have added an ablation replacing NGD with unnormalized GD in both training and generation (Appendix A.5, Table 11). GD achieves FID 2.62 versus 2.40 for NGD, with higher recall and lower precision. Although GD achieves comparable generation quality, we observe less stable training: its proposal AUC shows a transient rise around 0.6–0.8M training images before recovering, whereas NGD’s remains steadier (Figure 4).

References:

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Auxiliary classification loss

Is the AT auxiliary loss essential? Can an ablation be provided?

We have added an ImageNet64 ablation to test whether classification supervision and explicit gradient regularization can replace the AT auxiliary loss (Appendix A.4, Table 10). We use clean cross-entropy (CE) for classification and an R1R_1 penalty on the input gradients of the conditional energy at real images. We test γ=0.01\gamma=0.01 and 0.020.02, retaining the same AT-pretrained initialization for all variants.

At the FID-selected checkpoints, the replacements give FIDs of 7.10 and 5.09, respectively, versus 2.39 with AT. Both improve clean accuracy, but PGD-6 robust accuracy falls from 25.81% to approximately 5–6%. Their generation quality also deteriorates sharply later in training while clean accuracy remains near 75%. The tested replacements therefore do not preserve the generation quality and training stability obtained with AT, despite maintaining high clean accuracy.

We also tested γ=0.03\gamma=0.03, but did not observe coherent generation with target-only sampling at any stage of training (the best target-only FID was 92.48, achieved at 1.1M training images). We therefore did not evaluate relay sampling for this run.

Training and sampling cost

Can you provide at least rough estimates of the relative costs of training and sampling from this model compared to typical baselines like diffusion models? In particular, the training-time cost of MCMC sampling for EBMs is known to be high.

We have added training and sampling cost comparisons in Appendix A.8 (Table 14). Our ImageNet256 relay model achieves FID 3.05 with 61 NGD steps from noise. We report 207 A100-hours for EBM training and a generation throughput of 3.82 images/s per A100. For reference, LDM-4-G reports 6,504 V100-equivalent training hours and 0.40 images/s per A100 (Rombach et al., 2022). The training costs exclude AT and autoencoder pretraining, respectively.

Possible extension to unconditional generation

It is not clear how the method could be applied to even conventional simple situations like unconditional CIFAR-10.

For CIFAR-10 without class annotations, one possible extension is to obtain pseudo-labels by clustering image features and use these cluster IDs for both JEM's conditional energies and its auxiliary classification loss (see Liu et al., 2020, for an example of cluster-based conditioning). Relay would construct warm starts under an auxiliary-cluster energy and refine them under a target-cluster energy. For unconditional generation, we could sample target clusters according to their training-set frequencies and auxiliary clusters uniformly, then discard the cluster IDs. The empirical effectiveness of this extension remains to be evaluated.

References:

Liu, S., Wang, T., Bau, D., Zhu, J.-Y., & Torralba, A. (2020). Diverse Image Generation via Self-Conditioned GANs. CVPR.

Thanks to the authors for their comprehensive responses. I am inclined to stick with my score, although I believe there are still some weaknesses. The method works quite well for class-conditional ImageNet but does not perform especially well on class-conditional CIFAR-10 and cannot handle any unconditional distribution without further algorithm adjustments. While the ImageNet results are impressive, it makes me wonder whether this method can server as a foundational sampler or just a specialized sampler for certain situations. As Reviewer Q3CG pointed out, the cross-label sampling method has been in the literature for a long time since the IGEBM paper. DAT and noise initialization are also established in prior work. The methodological contributions are relatively light and the paper rests heavily on its ImageNet empirical result. Overall the paper is reasonable in its current form but it would be nice to see diffusion-level results on unconditional CIFAR-10 for example as evidence of ability to succeed in every situation where other strong methods like GANs and diffusions also do.

Thank you for the careful assessment and for recognizing the ImageNet results. Regarding cross-label persistent replay, relay sampling follows a different design principle: we specifically construct structured warm starts to improve short-run coverage, using intermediate states generated at a prescribed depth from noise. Persistent replay instead continues earlier sampling chains through buffered states, and Du & Mordatch’s test-time sampler similarly continues a long chain across occasional random label changes (see our detailed reply to Reviewer Q3CG). We have revised Related Work, Section 3.1, Table 1, and Appendix B.2 to clarify this distinction and compare the methods on ImageNet64 and in the controlled problem in Appendix B. DAT and uniform-noise initialization are shared by our target-only baseline and relay method, so these comparisons assess the additional benefit of the relay construction.

Our contribution targets short-run coverage in class-conditional EBMs, with the strongest empirical evidence on ImageNet. We acknowledge that the more limited CIFAR-10 results constrain broader empirical claims, and that extending relay to unconditional generation would require further methodological development and validation.

Reviewer_34RhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the limited mode coverage of short-run MCMC in joint energy models initialized from noise. It proposes relay sampling: a chain runs T1 steps under a random auxiliary label, then T2 steps under the target label, and training uses a mixture of target-only and relayed negatives. A cross-label cache of intermediate states, which target-only runs already produce, amortizes the cost from 174 to 78 steps per negative at 64x64. The contributions are the mechanism, the cache, and an empirical map of the T1/T2 allocation as a coverage-fidelity control. On ImageNet 64x64, FID improves from 4.20 to 2.40 and recall from 0.43 to 0.58; at 256x256 a 198M ConvNeXt-Large reports FID 3.05 from noise, competitive with diffusion and autoregressive models using 1.6 to 3 times as many parameters. Code and six hashed checkpoints are released. Headline numbers are single runs; ablations and one seed duplicate are reported.

Critical strengths
  1. A genuinely new and practical sampling mechanism. The cross-label warm start is a simple idea that, as far as I can tell, has not been proposed: run the chain under a random auxiliary label before the target label, and use the resulting state as a training negative. The paper differentiates it correctly from the nearest prior work (Du and Mordatch's test-time label switching), and the cross-label cache turns it into a cost saving rather than an added expense, reducing the expected steps per negative from 174 to 78 at 64x64. This is consequential because the contribution is a training-time device that can be added on top of an existing recipe, not a leaderboard increment.

  2. A first-of-kind scale result backed by an unusually complete artifact. FID 3.05 at 256x256 from noise, in pixel space and with a 198M-parameter model, places a noise-initialized energy model in the diffusion regime at that resolution, which changes what the subfield treats as achievable. The release includes pinned configs, a config-to-table map, six checkpoints with SHA256 hashes, and a test suite, so the central claim can be verified rather than taken on trust. This is consequential because the paper's main claim is a regime claim, and the artifact is what makes it credible despite the single-run evaluation.

Critical weaknesses
  1. The fine-grained evidence is single-run. Every reported FID, IS, precision, and recall value is a point estimate from one training run. The paper's only duplicate implies run-to-run FID noise of roughly 0.03. That is small relative to the core 64x64 improvement (FID 4.20 to 2.40), which is why I do not treat it as disqualifying, but it is comparable to the 256x256 margins (3.05 versus 3.21 for the Gaussian baseline and 3.29 for DAT) and it exceeds the recall difference the paper emphasizes at 64x64 (0.58 versus 0.56). This is consequential because several comparative claims are stated at a resolution the design cannot support, and reseeding the released checkpoints would settle them at modest cost.

  2. The two controls that would isolate the mechanism are missing. The replay-buffer (persistent-chain) baseline, which is the standard alternative to the cross-label cache, is argued against conceptually but never run, and there is no noise-initialized target-only run at 256x256. Without the latter, the 3.05 cannot be decomposed into the contribution of relay and the contribution of the inherited DAT recipe; without the former, the cache's advantage over persistent chains is asserted rather than measured. This is consequential because the paper's central claim is comparative, and these are the comparisons that would establish it.

Questions for authors

Clarification requests (these could change my recommendation):

  1. The 256x256 comparison rests on single runs: FID 3.05 versus 3.21 for the Gaussian baseline and 3.29 for DAT. Since the checkpoints are released, could you resample them, or train two or three additional seeds, and report the resulting variance? I would like to know whether that ordering survives the noise implied by your seed duplicate (about 0.03 FID).

  2. Can you run the replay-buffer (persistent-chain) JEM baseline at 64x64 under the same budget as the cache? This is the natural alternative to the cross-label cache, and the comparison would establish whether the cache is better or merely cheaper.

  3. Can you add a noise-initialized target-only ConvNeXt-Large run at 256x256? Without it, I cannot separate how much of the 3.05 comes from relay and how much comes from the inherited DAT recipe.

  4. For Table 2, what reference split was used for each baseline's FID, and can you either recompute the key baselines under one protocol or release the reference statistics? The cross-paper protocol heterogeneity is the one part of the comparison I could not check.

  5. The recall difference over the Gaussian baseline at 64x64 (0.58 versus 0.56) appears comparable to the implied run-to-run noise. Do you have evidence that this difference is real, or would you describe the two as tied on recall?

Suggestions for future work (not needed for my assessment):

  • Sweep the auxiliary-label distribution (for example, restricting to similar classes or excluding the target) and the cache size or staleness.
  • Report wall-clock and NFE costs, including the pretraining computation, and ablate the 256x256 controller threshold.
3 comments
AuthorsOpenReview ↗

Author Response

We thank Reviewer 34Rh for their helpful feedback. We have added multi-seed results for ImageNet256, persistent-replay and uniform-noise target-only controls, and clarified the evaluation protocol. Below we address your specific concerns.

Variation across training seeds at 256 × 256

Since the checkpoints are released, could you resample them, or train two or three additional seeds, and report the resulting variance?

We trained two additional seeds for both uniform relay and informative Gaussian and found the same FID ordering across the three runs. FID is 3.01±0.043.01\pm0.04 for relay versus 3.23±0.023.23\pm0.02 for Gaussian (mean ± sample standard deviation; Table 5), with every relay result below every Gaussian result. All relay results are also below the single DAT re-evaluation (FID 3.52). Table 5 reports all four metrics.

Comparison with persistent replay

Can you run the replay-buffer (persistent-chain) JEM baseline at 64x64 under the same budget as the cache?

We have added same-label persistent replay (Grathwohl et al., 2020) and cross-label persistent replay (Du & Mordatch, 2019) within our training framework (Appendix A.2, Table 8). Same-label replay gives FID 6.13 with target-only testing and 4.59 with relay testing; our cache-trained model gives 2.40, with recall 0.58 versus 0.481 and lower precision. These results favor noise-anchored intermediate states over recursive endpoint reuse for FID and coverage.

We match replay's post-curriculum step count (K=120K=120) to the cache construction depth (T1=120T_1=120).

Uniform-noise target-only baseline at 256 × 256

Can you add a noise-initialized target-only ConvNeXt-Large run at 256x256? Without it, I cannot separate how much of the 3.05 comes from relay and how much comes from the inherited DAT recipe.

We have added a uniform-noise target-only ConvNeXt-Large baseline within our training framework (Section 4.5, Table 2). It reaches FID 5.66, compared with 3.05 for relay; recall increases from 0.29 to 0.53, while precision decreases from 0.83 to 0.79. The target-only result shows that the DAT-based training recipe already supports generation from noise at this resolution, while relay further improves FID and coverage.

FID reference and evaluation protocol

For Table 2, what reference split was used for each baseline's FID, and can you either recompute the key baselines under one protocol or release the reference statistics? The cross-paper protocol heterogeneity is the one part of the comparison I could not check.

We evaluate uniform target-only, informative Gaussian, uniform relay, and DAT with 50K generated images per model using the official ADM evaluator (Dhariwal & Nichol, 2021). FID uses the released full-training-set statistics in VIRTUAL_imagenet256_labeled.npz, available through the official evaluation instructions. Precision and recall use the supplied 10K real reference images, pooled across classes.

We re-evaluated DAT under this protocol, obtaining FID 3.52 versus the originally reported 3.29 (Yin et al., 2026), and updated Table 2 accordingly. For the literature-reported baselines:

  • ADM-G / BigGAN-deep: full-training-set reference and ADM evaluation (Dhariwal & Nichol, 2021).
  • VAR: the same released ADM reference file (Tian et al., 2024).
  • DiT: ADM evaluation suite; the reference filename is not specified (Peebles & Xie, 2023).
  • LDM: full-training-set reference, primarily torch-fidelity (Rombach et al., 2022).

Our models and DAT therefore share a common evaluation protocol; the other Table 2 results remain quoted from prior work. We have clarified this distinction in the caption.

Recall relative to the Gaussian baseline

Do you have evidence that this difference is real, or would you describe the two as tied on recall?

We agree that relay (0.58) and the Gaussian baseline (0.56) should be described as having comparable recall at 64×6464\times64; the reported 0.02 difference does not establish a reproducible advantage. We have revised the corresponding statements in Section 4.3 accordingly.

At 256×256256\times256, our evaluation across three training seeds (Table 5) gives recall 0.531±0.0030.531\pm0.003 for relay versus 0.453±0.0160.453\pm0.016 for the Gaussian baseline (mean ± sample standard deviation), showing a consistent recall advantage across the three runs. The Gaussian baseline uses full covariance at 64×6464\times64 and a rank-2048 component plus a diagonal residual at 256×256256\times256 due to computational cost (Appendix A.1.3). We now discuss this cost in Section 4.3. Covariance approximation may contribute to the gap at the higher resolution, but we have not isolated this effect or established whether full covariance would close the gap. Our claim of higher recall at 256×256256\times256 therefore concerns the implemented Gaussian baseline.

Reviewer_34RhOpenReview ↗

Thanks for the improvements but still weak acceptance

The authors addressed my concerns but I maintain my score which is weak acceptance.

One cannot incrementally improve their way to a clear acceptance with more experiments and evaluation when the foundation and the premise of the paper are not particularly remarkable.

A mediocre idea with 100 pages of experiments is still a mediocre idea. That's how it works.

However, I thank the authors for addressing my concerns.

Please note that I mentioned

these could change my recommendation

not "would." So I'm under no obligation to raise the score.

Thank you for your positive recommendation and for acknowledging that our responses addressed your concerns. We added Appendix B to provide controlled analysis and validation of the relay mechanism, including comparisons with persistent replay. Together with the revised related work and Section 3.1, these additions clarify the rationale for our design and its distinctions from closely related approaches.

Reviewer_Q3CGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes relay sampling for short-run joint energy models. Sampling starts from noise under an auxiliary label and then switches to the target label. The model is trained with both target-only and relayed negative samples. A cache reuses intermediate states to reduce repeated computation.

The method aims to improve sample coverage. It builds on the DAT training framework and uses pretrained robust ConvNeXt classifiers. Experiments on ImageNet at 64 × 64 and 256 × 256 show improvements in generation quality and recall.

Critical strengths
  1. The paper addresses a meaningful problem. Limited sample coverage is an important issue for short-run energy models. Using the same model to generate structured starting points is a simple and practical approach. It avoids an additional generator or an external image pool for initialization.

  2. The baseline selection and schedule studies are useful. The informative Gaussian and DAT baselines provide relevant comparisons. The stage-allocation experiments also show a useful trade-off: longer auxiliary stages tend to improve recall, while longer target stages tend to improve precision.

Critical weaknesses
  1. The added value over closely related methods is not clearly established. The released implementation of Du and Mordatch (Implicit Generation and Generalization with Energy Based Models) already permits cross-label reuse during training. A direct comparison with this approach under the same training framework is missing. The informative Gaussian baseline is also competitive in FID, although relay achieves higher recall. Moreover, there is no uniform-noise target-only baseline at 256 × 256. These missing controls make it difficult to establish the specific benefit of the proposed relay procedure.

  2. The central claim is that relay sampling covers more modes of each target class. However, the reported recall is measured across all classes. It provides indirect evidence, but does not directly establish greater diversity within the requested class. The sample grids are helpful, yet quantitative measurements of within-class coverage and target-label accuracy are missing.

Questions for authors
  1. Under the same DAT training framework, does relay sampling improve FID and recall over existing cross-label replay? At 256 × 256, what gain does it provide over uniform-noise target-only training?

  2. How much of the reported gain can be obtained by applying relay sampling only at test time to a target-only-trained model? When the test-time sampler is fixed, does training with relayed negatives provide a further improvement?

  3. Does the recall gain remain when coverage is evaluated separately within each target class? Is target-label accuracy preserved as diversity increases?

4 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank Reviewer Q3CG for their helpful feedback. We have added direct comparisons with cross-label persistent replay and uniform-noise target-only training at 256 × 256, controls examining training and test-time effects, and class-wise coverage and target-label accuracy measurements. Below we address your specific concerns.

Comparison with cross-label persistent replay

Under the same DAT training framework, does relay sampling improve FID and recall over existing cross-label replay?

We have added cross-label persistent replay (Du & Mordatch, 2019) within our training framework (Section 4.2; Appendix A.2, Table 8). With relay test sampling for both models, our method achieves FID 2.40 and recall 0.58, versus 3.81 and 0.523 for persistent replay, at lower precision. Both methods reuse cross-label states, but persistent replay recursively continues previous chains, whereas our cache reuses intermediate states at a fixed depth from noise. These results support the benefit of noise-anchored warm starts beyond cross-label reuse alone.

Uniform-noise target-only baseline at 256 × 256

At 256 × 256, what gain does it provide over uniform-noise target-only training?

We have added a uniform-noise target-only ConvNeXt-Large baseline within our training framework (Section 4.5, Table 2). Relay improves FID from 5.66 to 3.05 and recall from 0.29 to 0.53, while precision decreases from 0.83 to 0.79. The target-only result shows that the DAT-based training recipe already supports generation from noise at this resolution, while relay further improves FID and coverage.

Added value over informative Gaussian initialization

The informative Gaussian baseline is also competitive in FID, although relay achieves higher recall.

We agree that informative Gaussian initialization achieves competitive FID. We have added two training seeds at 256×256256\times256, bringing the total to three (Table 5). Relay achieves recall 0.531±0.0030.531\pm0.003, versus 0.453±0.0160.453\pm0.016 for the Gaussian baseline (mean ± sample standard deviation), showing a consistent recall advantage across the three runs.

Relay also offers practical advantages at high resolution. Estimating and factorizing a dense full covariance becomes increasingly expensive and can be prohibitive as resolution increases. We now discuss this cost in Section 4.3. Relay requires no covariance estimation and provides a training-time control over the coverage–fidelity trade-off through the allocation of auxiliary and target steps. These properties can be useful when covariance fitting is costly or different coverage–fidelity balances are desired.

Finally, a broader motivation is the longstanding goal of competitive high-resolution EBM generation from a fixed, data-independent noise distribution, a starting point familiar from GANs and diffusion models. Informative Gaussian initialization relaxes this requirement by incorporating empirical data statistics into the initial distribution. Our results demonstrate that competitive pixel-space generation on ImageNet 256×256256\times256 is feasible from uniform noise, with the EBM itself constructing the structured warm starts.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Relay sampling applied only at test time

How much of the reported gain can be obtained by applying relay sampling only at test time to a target-only-trained model?

We find only a modest gain from test-time relay alone. Using the target-only model from Table 1 and sweeping the relay configuration as described in Appendix A.1.1, we improve FID from 4.20 to 4.08 and recall from 0.43 to 0.44, at a small cost in precision (Appendix A.3, Table 9). This does not recover the relay-trained model’s FID of 2.40 and recall of 0.58 with its own test schedule. We have summarized this comparison in Section 4.2.

Relay training with a fixed test-time sampler

When the test-time sampler is fixed, does training with relayed negatives provide a further improvement?

We do not observe an additional training benefit under the shared configuration in Table 9. Applying the target-only model’s selected relay sampler unchanged gives FID 6.74 for the relay-trained model versus 4.08 for target-only (rows 2–3). The improvement reported above therefore concerns relay training together with its own test schedule, rather than a gain under this fixed sampler.

Within-class coverage and target-label accuracy

Does the recall gain remain when coverage is evaluated separately within each target class? Is target-label accuracy preserved as diversity increases?

We find higher within-class recall in all ten randomly selected ImageNet64 classes when comparing the uniform-noise target-only and relay models from Table 1. We compute precision and recall separately for each requested class, retaining all generated samples without filtering by predicted labels. Mean recall increases from 0.162 to 0.391, while precision decreases from 0.670 to 0.648 (Appendix A.6, Table 12). We have referenced this evaluation in Section 4.2.

Target-label accuracy is not fully preserved: we measure a decrease in mean top-1 accuracy from 53.2% to 49.1% under an independent ResNet-50, which achieves 45.3% on the real references. These results support improved within-class coverage on the evaluated classes, with a precision and target-label accuracy trade-off.

Thank you for your answer. Your response addresses my questions. However, my main concern is the novelty of the proposed method. The cross-label sampling, DAT and noise initialization are all established in prior works. Thus, I maintain my score.

Thank you for clarifying your remaining concern about novelty. Our relay method differs from cross-label persistent replay in both the role and the construction of its cached states.

  • Role of the cached states. In our method, cross-label intermediate states serve specifically as structured initializations to improve short-run coverage of the requested target class. Our cache supplies these auxiliary starting points for target-label refinement. In persistent replay, buffered states are instead used to continue earlier sampling trajectories across iterations, rather than being specifically constructed as auxiliary initializations to improve coverage.
  • Construction of the cached states. Relay caches intermediate states produced by a prescribed number of auxiliary-label updates from fresh noise. Reusing an intermediate state for target refinement does not recursively extend the cached state’s sampling history. Persistent replay instead returns updated outputs to its buffer, allowing sampling histories to accumulate across uses, with effective depths determined by continuation and resetting. Relay thus explicitly specifies the construction depth of its auxiliary warm starts.
  • Cross-label sampling at generation time. At each round except the final ones, Du & Mordatch’s sampler replaces the current label with a randomly chosen one with a small probability, so its behavior mirrors training-time cross-label persistent replay in allowing the same chain to continue across label changes. Relay starts a short chain from fresh noise under an auxiliary label and switches once, at a prescribed step, to the requested target label for refinement (Section 5).

This construction brings practical benefits.

  • Coverage and controllability. At fixed energies in the 60∘60^\circ setting, fresh relay with gradient descent achieves nearly balanced basin allocation in the first round and in each subsequent round reported in Table 19. Pooled cross-label persistent replay can also attain a similar allocation, but its performance depends on the reset probability and sampling round (Appendix B.2). These results show how the different constructions affect short-run basin coverage. Another benefit of our construction is that we can control the precision–recall trade-off by allocating updates between auxiliary sampling and target refinement (Section 4.2, Figure 2). This explicit training-time control is not available in the persistent replay baseline.
  • Empirical gains. On ImageNet64, our relay configuration achieves considerable gains over cross-label persistent replay in FID and recall (2.40/0.58 vs. 3.81/0.52, Table 1). On ImageNet256, our relay-trained pixel-space EBM generates samples from uniform noise and achieves an FID of 3.01±0.043.01 \pm 0.04 across three training seeds (Table 5). This FID is competitive with those of established GAN and diffusion baselines (Table 2).

Regarding DAT and noise initialization, our target-only baseline and proposed relay method share the same DAT-based training recipe and uniform-noise initialization. We use the uniform-noise target-only model trained with DAT as the direct baseline. In the ImageNet256 comparison reported in Table 2, the relay configuration improves FID/recall from 5.66/0.29 to 3.05/0.53. Our gains over this baseline therefore come from the relay construction rather than from these established components.

We hope these distinctions and comparisons clarify the novelty and practical value of our work.

Reviewer_3GxdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors proposed a novel procedure to train a subclass of energy-based model, the joint EBMs, where data are labelled, and the objective is to treat the label as generation objective. Their novel proposal is focused on how to efficiently build a trade-off between coverage and accuracy when generating the negative samples, which is usually a problematic step in training EBMs. Their idea is based on a relay training: first they build a cache of negative samples with a generic (uniformly sampled) label, and then, with some probability, they complete the relay with a second generation step towards the the real label. In practice, they implement it with an amortized procedure. They claim that this two stage algorithm could avoid mode collapse in generating negative samples, hence help the training phase of a JEMB. They test their idea on real images dataset.

Critical strengths

The paper is well presented; the theory part is essential, but clear. The critical strengths of this paper lies mainly in the simplicity of the idea, and in the experimental validation, particularly in the direction of scalability towards images --- a context where EBMs are known to suffer with respect to other state of the art generative models.

Critical weaknesses

Here I list some weaknesses that makes the paper not ready, in my opinion, for a publication at this stage.

  • Related works. The paper focuses on generation of negative samples, but they adopt SGLD, which is a relatively standard way to generated samples. There are other approaches that have been proposed, such for instance based on non-equilibrium sampling [Carbone et al. NeurIPS 2023], external generators [Cui et al. NeurIPS 2023], also focusing on the "non-converging nature" of the generator procedure [Agoritsas et al. PMLR 2023]. Missing related works on "the missing convergence of the generator of negative samples" is a weakness for a paper on EBMs, especially if focusing on training.

  • Theoretical foundations. The relay procedure introduces a certain number of new hyperparameters, and hypotheses, that appears to be very weakly justified by theoretical results; moreover, their presentation is a bit lost in the flow of the main text. If the algorithm is mainly based on empirical observations and intuitions, it must be clearly stated at the beginning, so that the reader understands that the main idea behind the paper is "we had this intuition, and we propose an algorithm that can be just partially theoretically motivated". In this form, the paper does not contain any theoretical support, neither the appendix does, for the proposed algorithm.

  • Experimental validation. Given the absence of a theoretical clear justification (even in some ideal limit) of the algorithm, the experimental validation which is presented does not suffice. While being interesting to show scalability, if you propose an algorithm based on empirical observation, what should be presented is a very fair comparison with state of the art, not necessarily only on images. First of all, IS and FID are known to not be good metric for images (see for instance [Jayasumana et al. "Rethinking fid: Towards a better evaluation metric for image generation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024] and [Barratt et al., "A note on the inception score." arXiv preprint arXiv:1801.01973 (2018).]. The comparison with state of the art should be based on an ensemble of fair metrics, if the experiments are on images. Metric results should also present error bars on various seeds. The algo should also be tested on some toy model with controlled knobs to "make it more complicate only where needed", in order to visually understand why this algorithm should work better then standard buffer approaches, or other SoTA procedures. This because there are several tricks in image generation whose effects cannot be easily disentangled from the other approximations commonly done in EBMs training.

Questions for authors
  • Could you please expand the related works section, including different approaches on the generation of negative samples? Maybe focusing on the nature of the chain that could be used, on the biased induced by finite chains, on the relation with (non-)equilibrium sampling.

  • Could please elaborate more on the theoretical part? If there is no theoretical justification of the proposed algorithm, it must be clearly stated (it is not necessarily a problem). Moreover, the hypotheses and hyperparameters that are introduced must be clearly highlighted with remarks or "Assuption" blocks. On the contrary, if there is a theoretical justification please add it; clearly, proposing an algorithm with limited theoretical justification would require a more precise experimental assessment (see next).

  • If the theoretical justification of the algorithm is just empirical, the experimental part needs to be substantially revised. More than focusing on scalability, the authors should focus on fairness of comparison with SoTA methods; this because an empirical approach could not be motivated to be better "by chance". On images: could you please compare other SoTA methods with your proposal in terms of fair metrics (not FID or IS) and computational cost, also in situations with strong class imbalance, and put error bars on the results? Could you include other toy and controlled experiments with a focus on the situation in which your algorithm should be better than SoTA?

In general, before running many hours of experiments on images, I would first check on an aforementioned "toy or explainable experiment" if your algorithm can critically beat a SoTA approach, in a fair comparison.

As a final comment, I am not completely sure that a novel algorithm purely based on empirical results could be accepted in ICLR, given the absence of recognised bechmarks on the EBMs.

7 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank Reviewer 3Gxd for their helpful feedback. We have added finite-step analysis and controlled validation in Appendix B (pp. 29–32) and expanded the related-work discussion. We also report persistent-replay comparisons, CMMD, class-wise coverage and target-label accuracy, variation across training seeds, and training and sampling costs. Below we address your specific concerns.

Theoretical support and explicit assumptions

"In this form, the paper does not contain any theoretical support, neither the appendix does, for the proposed algorithm."

We agree that the sampling mechanism needs explicit justification and controlled validation alongside the image experiments. We have added Appendix B, "Theoretical analysis and validation" (pp. 29–32), addressing the theoretical basis, the conditions for success or failure, and the role of stage allocation. In the controlled setting of Appendix B.1, we establish sufficient conditions for cross-label warm starts to improve target-basin coverage within a finite sampling budget. Its central finding is that target-only descent can accurately approach a mode while retaining the unfavorable basin allocation of its initialization. An auxiliary-conditioned path can change that allocation by placing warm starts in a basin that the initialization rarely reaches; subsequent target updates refine those states.

Eqs. (11)–(12) formalize this mechanism and relate its success to the auxiliary geometry and stage length. Appendix B.1 collects the model and sampling parameters; B.2 explicitly states the conditions on mode separation and step size and provides the proof. The guarantee concerns fixed energies and ordinary gradient descent at finite step counts; the effects on learning neural EBMs remain evaluated empirically.

Controlled validation of the sampling mechanism

"The algo should also be tested on some toy model with controlled knobs"

Following this suggestion, we use a high-dimensional model (D=256D=256) with a bimodal target and a unimodal auxiliary, both fixed to known distributions (Appendix B.1). Fixing the energies isolates the sampling mechanism from errors in learning the energy and from heuristics used in image generation. Two target modes are sufficient to expose undercoverage against known reference probabilities, while the unimodal auxiliary isolates the effect of changing the conditional path. Varying its direction lets the same construction test both a predicted improvement and a predicted failure.

Target-only and relay sampling receive the same total budget of 256 gradient evaluations per particle, including relay's auxiliary stage. We evaluate basin mass against known ground truth and separately assess refinement toward target minima. These quantities directly test the proposed mechanism without relying on FID or IS.

Table 17 shows the main result: target-only sampling assigns only 0.025% probability to a basin whose true mass is 50%, whereas relay recovers approximately 50% under the predicted favorable geometry (60∘60^\circ). The gain disappears for the alternative geometry (90∘90^\circ) predicted to keep auxiliary endpoints in the already-covered basin. All three settings in Table 17 produce states close to target minima, showing that accurate refinement alone does not ensure coverage. The analytical predictions agree with measurements pooled over ten sampling seeds, with 95% confidence intervals reported in the table.

Why the stage-length hyperparameters matter

"The relay procedure introduces a certain number of new hyperparameters"

Appendix B.3 examines the switching time under a fixed total budget, measuring both basin coverage and target refinement. Table 18 identifies the allocation trade-off: switching too early leaves the underrepresented basin largely unreached; once coverage saturates, additional auxiliary steps provide no further coverage gain and leave fewer steps for target refinement. The observed transition is consistent with the auxiliary dynamics derived in Eq. (15).

This gives a concrete rationale for allocating computation between the stages: the auxiliary stage needs enough time to establish useful basin coverage, while the target stage needs enough time to refine the warm starts. It also explains why switching before full auxiliary convergence can work. The analysis clarifies the roles of the stage lengths; their numerical values for learned models, as well as the target-only mixing probability and cache size, remain empirical choices.

Comparison with persistent replay

"...why this algorithm should work better then standard buffer approaches, or other SoTA procedures."

Toy experiments test analytical predictions against known basin probabilities. We have added direct comparisons with same-label and cross-label persistent replay under a common ImageNet64 training framework (Section 4.2, Table 1; Appendix A.2, Table 8). Relay obtains lower FID and higher recall than both baselines, at lower precision.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Image-generation metrics beyond FID and IS

"The comparison with state of the art should be based on an ensemble of fair metrics"

We reported precision and recall alongside FID and IS in the original submission (Tables 1 and 2). We have added CMMD (Jayasumana et al., 2024), which evaluates distribution matching in CLIP feature space, for our three models, ADM-G, and BigGAN-deep using a shared reference of 50K ImageNet training images (Appendix A.7, Table 13). Relay achieves lower CMMD than both target-only baselines, but higher CMMD than ADM-G (1.067 vs 0.210), despite its lower FID (3.00 vs 4.59). These results show that better Inception-space distribution matching need not imply better CLIP-space distribution matching.

We include separately attributed CMMD results for five additional baselines from PALATE (Dziarmaga et al., 2025, Table 1, v2).

We also evaluate class-wise coverage and target-label accuracy for the uniform-noise target-only and relay models on ten randomly selected ImageNet64 classes (Appendix A.6, Table 12). Relay increases recall in all ten classes, with mean recall rising from 0.162 to 0.391, at a cost in precision and target-label top-1 accuracy (53.2% to 49.1%).

Computational cost

"fair metrics (not FID or IS) and computational cost"

We have added training and sampling cost comparisons with diffusion, autoregressive, and masked-token baselines in Appendix A.8 (Table 14). Our ImageNet256 relay model achieves FID 3.05 with 61 sampling steps from noise. We report 207 A100-hours for EBM training and a generation throughput of 3.82 images/s per A100. For reference, LDM-4-G reports 6,504 V100-equivalent training hours and 0.40 images/s per A100 (Rombach et al., 2022). The training costs exclude AT and autoencoder pretraining, respectively.

Related work

"Could you please expand the related works section"

We have expanded Section 5 to discuss the bias induced by finite chains, nonequilibrium reweighting (Carbone et al., 2023), learning with non-convergent MCMC (Agoritsas et al., 2023), and generator-initialized MCMC, including dual-MCMC teaching (Cui & Han, 2023). We also clarify that relay changes the finite-step negative distribution using cross-label warm starts from the same EBM, without an auxiliary generator, importance reweighting, or an assumption of equilibrium sampling.

Variation across training seeds

"Metric results should also present error bars on various seeds."

We have added two training runs each for uniform relay and informative Gaussian on ImageNet256. We now report mean ± sample standard deviation across three training seeds for FID, IS, precision, and recall (Table 5). Relay achieves FID 3.01±0.043.01\pm0.04 versus 3.23±0.023.23\pm0.02 for Gaussian, and recall 0.531±0.0030.531\pm0.003 versus 0.453±0.0160.453\pm0.016, at lower precision. The lower FID and higher recall are consistent across the three runs.

Strong class imbalance

"also in situations with strong class imbalance"

In JEM (Grathwohl et al., 2020), the logit for class yy satisfies fθ(x)[y]=log⁡pθ(x∣y)+log⁡pθ(y)+log⁡Zθf_\theta(x)[y]=\log p_\theta(x\mid y)+\log p_\theta(y)+\log Z_\theta, so the learned class prior is already represented in the logits. At generation time, target labels can be drawn according to their training-set frequencies for conditional sampling. For relay, the auxiliary-label distribution π\pi controls warm-start construction, while target labels still follow the sampling of training examples. We have clarified in Section 3 (The relay distribution) that π\pi can follow the training-label frequencies or be uniform without changing the class-frequency information used by JEM to learn the prior.

References:

Dziarmaga, T., Kądziołka, M., Musioł, L., Marszałek, P., Kasymov, A., Kościelniak, P., & Mazur, M. (2025). PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models. arXiv:2503.18462v2, revised 2026.

I thank the authors for the extensive rebuttal and for the additional work. The expanded related work, the explicit statement that the effect on EBM learning remains empirical, the additional training seeds, and the CMMD evaluation are appreciated.

However, the main concerns of my review remain unaddressed, and I maintain my score, towards rejection.

  • Theoretical support. The analysis in Appendix B covers fixed energies and noiseless gradient descent, not SGLD within EBM training. Moreover, the favorable geometry places the auxiliary mode in the underrepresented target basin, which makes the coverage gain expected by construction. This illustrates the intuition but does not justify the algorithm.
  • Controlled comparison. The toy experiment compares only target-only and relay sampling. The comparison with persistent replay, which was the core of my request, is provided only on ImageNet64. In that setting, as noted in my review, the effect of the sampler cannot be disentangled from the other components of training.
  • Empirical evidence. Since the contribution rests on empirical evidence, that evidence must be solid. The new CMMD results is against the main claim: relay obtains a CMMD roughly five times higher than ADM-G (1.067 vs. 0.210) despite a lower FID. This confirms that the FID-based advantage does not carry over to more reliable metrics.
  • Other points. Relay reduces precision and target-label accuracy (53.2% → 49.1%), which matters for a joint EBM. The cost comparison excludes AT pretraining and mixes A100 and V100-equivalent hours. No experiment addresses class imbalance; the rebuttal offers only an argument about the model’s class prior, not about the sampler.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Manuscript Updates

We thank all reviewers for their helpful feedback. We have revised the manuscript in response to their comments and highlighted the changes in blue. The main updates are summarized below.

Theoretical Analysis and Controlled Validation

  • Finite-step analysis on a controlled problem (3Gxd): We have added analysis of a controlled model with fixed, known energies under ordinary gradient descent (Appendix B.1). It shows that target-only descent can refine samples while preserving their initial basin allocation, and gives sufficient conditions for cross-label warm starts to improve target-basin coverage. Experiments at a matched total gradient budget test both successful and ineffective auxiliary paths and clarify the role of stage allocation in basin coverage and target refinement (Tables 17–18).

  • Controlled sampler comparisons (3Gxd): We compare fresh relay with same-label, pooled, and strict-other-label persistent replay under GD and SGLD at matched gradient budgets. Fresh relay with GD reaches nearly balanced basin allocation in its first round; replay can also do so, depending on reset rate and reuse history (Appendix B.2, Table 19).

  • Cache sampling under class imbalance (3Gxd): We adopt label-uniform cache sampling (Section 3, Algorithm 1). A fixed-energy experiment with 10:1 cache and target-request imbalance shows that it restores nearly balanced basin allocation within the majority target class (Appendix B.1.4).

Additional Baselines and Generation Metrics

  • Persistent replay (3Gxd, Q3CG, 34Rh): We have added same-label and cross-label baselines within our ImageNet64 training framework. Relay training achieves lower FID and higher recall at lower precision (Section 4.2; Appendix A.2, Table 8).

  • Uniform-noise target-only training at 256 × 256 (Q3CG, 34Rh): We have added a ConvNeXt-Large baseline, showing that relay further improves FID and recall over the existing training framework, at lower precision (Section 4.5, Table 2).

  • Training and test-time effects (Q3CG): We have added controls showing that test-time relay alone gives limited gains. Relay training outperforms target-only training with its own schedule, but not under the tested shared sampler (Section 4.2; Appendix A.3, Table 9).

  • Class-wise coverage and target-label accuracy (3Gxd, Q3CG): We have added evaluation on ten randomly selected ImageNet64 classes. Relay improves recall in every evaluated class, with lower mean precision and target-label accuracy (Section 4.2; Appendix A.6, Table 12).

  • CMMD evaluation (3Gxd): We have added CLIP-space comparisons using a shared reference. Relay improves CMMD over both target-only baselines, but has higher CMMD than ADM-G despite lower FID (Appendix A.7, Table 13).

Component Ablations

  • Auxiliary classification loss (Yjj5): We have tested clean cross-entropy plus input-gradient regularization as a replacement for AT. The variants retain higher clean accuracy but give worse generation and robustness, with later deterioration or persistent artifacts (Appendix A.4, Table 10).

  • Gradient normalization (Yjj5): We have added unnormalized gradient descent (GD) in training and sampling. Normalization is not essential in this setting, but we observe lower FID and steadier proposal AUC with it (Appendix A.5, Table 11, Figure 4).

Reproducibility and Computational Cost

  • Variation across training seeds (3Gxd, 34Rh): We now report three-seed means and sample standard deviations for relay and informative Gaussian across all four ImageNet256 metrics (Table 5).

  • Evaluation protocol (34Rh): We have clarified reference data and result sources, distinguishing our evaluations from literature results (Table 2 caption).

  • Training and sampling costs (3Gxd, Yjj5): We include estimated clean pretraining, ConvStem adaptation, and AT alongside measured EBM training cost and generation throughput, and express available baseline training costs on an A100 basis (Appendix A.8).

  • Reproduction code (3Gxd): We include code and reference results for the Appendix B experiments in the supplementary package.

Writing and Clarifications

  • Related work (3Gxd, Q3CG): We have expanded the discussion of finite-chain bias, nonequilibrium sampling, non-convergent chains, and generator-initialized MCMC, and clarified relay's distinction from persistent endpoint reuse (Section 5).

  • Scope and terminology (3Gxd, Q3CG, 34Rh, Yjj5): We limit cross-family competitiveness claims to FID (Introduction, Section 4.5, and Conclusion). We also rename PGD to normalized gradient descent (NGD) in Section 4.1 and add a discussion of high-resolution covariance-estimation costs in Section 4.3.

Area_Chair_5C7POpenReview ↗

Author-Reviewer Discussion ends in 1 day, 12 hours

Dear authors and reviewers,

Thank you to the authors for their detailed responses of Nov 15, the updates since then and the revised manuscript. The discussion period closes in about a day and a half, so this is the last chance for a direct exchange before the reviewer-AC discussion starts on Nov 19.

Reviewers Q3CG and 34Rh: I haven’t yet seen a reply from you. Please read the authors’ responses and the revision, which address several of your points directly. Please let us know whether your concerns have been addressed, and update your recommendation if your assessment has changed. Even a short acknowledgment helps.

Reviewers 3Gxd and Yjj5: thank you for engaging. The authors answered 3Gxd on Nov 17. Please let us know whether these settle your remaining points. Yjj5, if you wish to respond to the authors’ CIFAR-10 results and the other additions, please do so.

Authors: please be ready to answer any follow-up questions in the time left, focusing on the points most likely to affect the decision.

Thank you all for keeping the discussion constructive and timely.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.