Hierarchical Text Generation with Next-Concept Modeling (NCM)

Diffusion language models (DLMs) are appealing for their parallel sampling, which can enable faster generation than autoregressive models; this is increasingly important as language model inference has become a dominant computational workload. In practice, however, DLM quality degrades quickly as the number of sampling steps shrinks, every step processes the full sequence, and learning to generate in every possible order makes training expensive. We introduce $\text{Next Concept Modeling (NCM)}$, a two-stage, coarse-to-fine framework in which a multiscale residual VQ-VAE encodes text into a few levels of discrete codes and a transformer generates them one level at a time, predicting all codes within a level in parallel. At the $\sim 100$M-parameter scale, $\text{NCM}$ achieves a better generative perplexity on LM1B than the strongest DLM baseline while using $11\times$ fewer steps and running $21\times$ faster. On OpenWebText, it approaches the generative perplexity of the strongest diffusion baseline while running $14\times$ faster, and on XSum it matches the strongest diffusion baseline in ROUGE-L while sampling $31\times$ faster.