Non-Markovian Uniform-State Diffusion Language Models

Uniform discrete diffusion language models (UDLMs) are typically built on a fixed Markov coupling between noise levels, which jointly determines their reverse dynamics, training objective, sampling behavior, and likelihood evaluation. We introduce NoMUD, a family of non-Markovian UDLMs that makes this coupling explicitly designable through two degrees of freedom while preserving the same marginals. Despite inducing different transition dynamics, all feasible couplings share the same optimal predictor, allowing the coupling to be adapted during inference. We exploit this freedom in three settings: optimizing the coupling tightens the negative evidence lower bound for likelihood evaluation; confidence-aware coupling enables adaptive retention, correction, and exploration during sampling; and a learnable coupling improves the flexibility of few-step distillation. Together, these results demonstrate that explicitly designing the coupling provides a simple and effective way to enhance the flexibility of UDLMs and unlock their broader potential.