Ermine: Neural Differential Scaling Laws for Diffusion Language Models

Scaling laws prescribe how to spend compute; for diffusion language models (DLMs), they should also prescribe *when*. Existing compute-optimal recipes for DLMs inherit the Chinchilla scaling law, which allocates capacity and data uniformly across the diffusion process. We observe, however, that DLM loss varies substantially with the diffusion timestep, leaving compute underutilized under Chinchilla-style allocation. We introduce *Ermine*, a differential scaling law that characterizes the instantaneous loss via time-dependent capacity and data densities. We derive its optimal solution under a fixed compute budget and realize it with quantile-based stratified sampling and a sparse Mixture-of-Experts (MoE) architecture. Ermine accelerates Chinchilla-optimal training by $1.35\times$, improves compute efficiency over dense DLMs by $1.87\times$, and narrows the compute-optimal gap between DLMs and autoregressive models (ARMs) from $17.56\times$ to $9.57\times$. On OpenWebText, our model attains a state-of-the-art validation perplexity of $17.63$ among DLMs, and at $1.1\times 10^{21}$ FLOPs it outperforms existing ARMs and DLMs on five zero-shot commonsense reasoning benchmarks using only $30$-$50$% of their training FLOPs. Our analysis further reveals that Ermine generalizes better by avoiding excessive compute in time regions where scalability diminishes.