Direct Latent Editing for VFM-Based Image Generators

Modern text-to-image generators train a flow model in a compressed latent space defined by an encoder. Recent work replaces the standard VAE encoder with a frozen Vision Foundation Model (VFM) such as DINOv3 to exploit self-supervised semantic features. However, editing for these VFM-based generators has not kept pace with their generation quality. We apply four inference-time editors built for VAE-based generators to VFM-based generators, at their published or tuned settings, and find that they rarely reach the target attribute. We propose Direct Latent Editing (DLE), an inference-time editing method for fully VFM-based generators with a frozen encoder. DLE precomputes one editing direction per concept from flow-model samples. At edit time it adds this direction to the encoded latent and decodes; no flow-model call, inversion or weight update is needed. A single scalar gives continuous, signed control. On three VFM-based generators we trained (CelebA faces, MIMIC-CXR chest X-rays and ADNI-1 brain MRI), DLE reaches targets that the tested baselines do not ($82.0\%$ vs. at most $48.3\%$ on CelebA; $0.927$ vs. at most $0.569$ AUROC on MIMIC-CXR), at $203\times$ lower per-edit flow-model and decoder time than FlowEdit at its official configuration.