Asynchronous Flow Maps for Flow Language Models

The choice of how data elements are ordered and scheduled for modeling has a profound impact on deep generative models. This effect is pronounced in categorical data such as language and code with procedural underlying generative processes. As such, discrete methods, such as autoregressive models and masked diffusion models, can naturally improve by exploiting these orderings during training and inference. However, it is less clear whether this effect can be incorporated into continuous flows for language and categorical data, which have a distinguished advantage of efficient inference through flow map distillation. We introduce asynchronous flows and flow maps, a framework for leveraging token orderings in flow language models by moving each token at a different speed with either prescribed or learned schedules, along with their mathematical formulation and distillation objectives. The resulting asynchronous flow maps are capable of one-step generation while harnessing the benefit of ordered generation by internalizing the procedure into a single forward pass. We train asynchronous flows and flow maps on Sudoku solving, and mathematical coding tasks and achieve state-of-the-art one-step performance.