Relay Sampling: Cross-Label Warm Starts for Short-Run Joint Energy Models

Deep energy-based models commonly rely on short-run samplers from random noise for training and generation, but their finite-step distributions remain dependent on the initialization and may cover only a limited subset of the model distribution. In joint energy models, each label defines a conditional energy surface, yet the label is typically used only as the final generation target. We introduce relay sampling, which also uses label conditions to diversify the path from noise to the target class. The sampler first follows a random auxiliary label to construct a structured intermediate state, which we call a cross-label warm start, and then passes this state to the target-conditioned sampler. This procedure replaces the single target-only path from noise with a mixture of auxiliary-conditioned finite-step paths, and we hypothesize that training with negatives from this mixture yields a model that covers more of each class. On ImageNet 64x64, models trained with relayed negatives reach substantially higher recall and much better FID than models trained with target-only negatives. On ImageNet 256x256, relay training of a ConvNeXt-Large model, in pixel space and from random noise, yields an FID of 3.05, below diffusion and autoregressive baselines that use 1.6 to 3 times as many parameters, while retaining robust classification, OOD detection, and calibration.