Optimizing Synthetic Data with Metagradients

Synthetic data plays an important role in the capabilities, safety, and security of modern AI systems. However, it is unclear how to best generate a synthetic dataset given an objective and a set of data constraints. Current approaches involve significant manual specification and human effort, leaving the limits of synthetic data largely unexplored. To explore the full capabilities of synthetic data, we cast synthetic data generation as an optimization problem: to construct a dataset that induces (via training) a model which maximizes an arbitrary objective. Leveraging recent advances in metagradient computation, we propose a broad family of algorithms for optimizing synthetic data on arbitrary targets. We validate our methods on the challenging task of subliminal transfer: generating a seemingly meaningless dataset that causes models trained on it to exhibit some pre-specified behavior, without any mention of the latter in the dataset. Our methods achieve success rates of up to nearly 100% across several settings, including cases where existing methods show near-zero transfer. Our results illustrate the effectiveness of using synthetic data to shape model behavior, and demonstrate that employing this optimization-based perspective is key to unlocking these capabilities.