PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensembles

Synthetic data offers a scalable solution for vision-language pre-training, yet current state-of-the-art methods typically rely on scaling up a single generative backbone, which introduces generator-specific spectral biases and limits feature diversity. We introduce **PolyGen**, a framework grounded in the *Generator-Invariance Hypothesis*: when images are rendered by an ensemble of architecturally distinct generators, generator-specific artifacts vary across sources while semantic content is shared by construction, and a model trained on the union is encouraged to encode the latter. We treat this hypothesis as a training principle with three testable implications, addressing ensemble composition, invariance regularization, and the timing of hard-negative introduction, each examined through dedicated ablations. PolyGen operationalizes it through a multi-positive strategy exposing the model to disjoint visual manifolds rendered from the same caption, together with a Programmatic Hard Negative curriculum enforcing fine-grained compositional understanding. By reallocating the same image budget from unique captions to multi-source variations, PolyGen yields a more robust feature space, outperforming the leading single-source baseline (SynthCLIP) by $+19.0\%$ on aggregate multi-task benchmarks and $+11.3\%$ on compositional reasoning, and, under an identical protocol, exceeding a real-data (CC3M) baseline on cross-modal retrieval while closing two thirds of the zero-shot gap.