Cross-modal drifting

Learning a conditional distribution from paired modalities can be challenging when each source observation has only one observed target. For conditional drifting, using that single paired target as the empirical target specifies a point mass rather than the population-conditional. We introduce Cross-Modal Drifting (CMD), which constructs a conditional target proxy from the paired targets of nearby source observations, weighted by their source-space proximity. A kernel-gradient field matches the generator's feature distribution to this proxy, further smoothed using local target-space covariance. The resulting generator produces each sample in a single network evaluation, with no neighborhood retrieval required at inference. Across cross-modal prediction tasks, CMD achieves the lowest mean Energy Score among the evaluated methods, including multi-step flow-matching baselines whose strongest evaluated setting costs fifty times more at inference. A component ablation supports the benefits of the proposed conditional proxy and, in most settings, of local covariance smoothing.