Extra Dimensions, Simpler Flows: Accelerating Flow Matching Training with Pretrained Visual Representations

It is an effective strategy to accelerate the training of diffusion and flow-based generative models by jointly modeling image latent features and semantic representations extracted from pretrained vision encoders. However, stronger encoders do not always improve generation quality and can hinder optimization. We show that a semantic representation reduces velocity ambiguity only when it provides velocity-relevant information that is unavailable from the noisy image state, thereby distinguishing otherwise ambiguous velocity targets. Motivated by this insight, we propose Joint Semantic Flow Matching (JsFlow), which jointly transports the image state and semantic variables from multiple pretrained encoders. JsFlow accelerates semantic transport to provide more informative semantic states throughout the trajectory. Joint image–semantic prediction can still introduce gradient interference through the shared backbone. To address this issue, we predict semantic velocities through a separate branch that takes detached backbone features as input. This preserves semantic guidance during transport while preventing semantic-loss gradients from updating the image backbone. On ImageNet $256\times 256$, JsFlow with LightningDiT-XL/1 achieves an unguided FID of $5.91$ after only $50$K iterations. It outperforms SiT-XL/2 trained for $7$M iterations (FID $8.30$) and LightningDiT-XL/1 trained for $4$M iterations (FID $6.47$), requiring only $1/140$ and $1/80$ of their training iterations, respectively.