JEPA-DIT: FROM-SCRATCH PREDICTIVE WORLD-MODEL PRETRAINING FOR MANIPULATION

World-action models—transformers that jointly denoise future video and actions—represent the world almost exclusively in the latent of a reconstruction VAE inherited from a video-generation prior. That is a liability for control: a re- construction objective must preserve every pixel-level nuisance a decoder has to repaint, whereas a policy needs only the scene’s controllable structure. A policy- free diagnostic makes the cost concrete—under domain randomisation the recon- struction latent moves nearly twice as far from its clean distribution as a predictive (JEPA) latent (ρ= 2.05 vs. 1.16), while a cross-task control shows both register a change of task equally: invariance, not collapse. The field keeps the VAE any- way, because a JEPA latent has no generative prior to build a world model on. We present JEPA-DiT, to our knowledge the first world-action model whose dif- fusion transformer is pretrained from random initialisation in a JEPA latent, car- ried unchanged from egocentric pretraining through teleoperation mid-training to downstream manipulation, and delta-JEPA, which makes the two representations complementary rather than competing by having the predictive stream carry the temporal difference from the present observation. Training from scratch proves to be not the price of a predictive latent but the reason it works: in a width-matched ablation the from-scratch stream develops a language pathway at 25×the floor uniform attention would give while a warm-started run reaches 2.1×, barely above chance, and passes that deficit downstream. Across all 50 RoboTwin 2.0 tasks with randomisation held out of training, JEPA-DiT reaches 84.5% clean and 59.5% out of distribution—the best OOD score and best clean/OOD average among world- action models on this protocol. Each representation alone falls short of the pair: with a mature prior but no predictive stream the model scores 1.9% under the shift, with a predictive stream but no prior 11.9%.