acceptodds
Under review as a conference paper at ICLR 2027

Decentralized Diffusion Language Models

Abstract

Discrete diffusion language models (DLMs) generate text by iteratively denoising masked sequences, and their training objective generalizes autoregressive modeling beyond a predetermined generation order. Like other large language models, however, DLMs are trained as monolithic models, whose data-parallel scaling across GPU nodes is limited by cross-node synchronization and by diminishing returns from larger batches. We propose Decentralized Diffusion Language Models (D-DLM), which partition the training corpus into disjoint clusters, train one expert DLM per cluster without cross-expert synchronization, and train a router to predict the cluster membership of a noisy sequence. We prove that the probability velocity of the DLM reverse process decomposes into a mixture of cluster-conditional velocities weighted by the cluster posterior, and that the clean-token posterior admits the same decomposition. Hence, in the realizable limit, D-DLM recovers the same reverse process as centralized DLM sampling, and under masked diffusion, sampling reduces to a router-weighted ensemble of expert predictions. In pre-training at up to 1.2B parameters and 80B tokens, D-DLM improves the average downstream accuracy over monolithic training at every scale, by up to 2.05 points, with no cross-node communication, and it improves training throughput by up to 24% when training is bottlenecked by cross-node communication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, decentralized training, discrete diffusion, expert routing, data clustering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JiBsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a decentralized training and inference framework for diffusion language models. It divides the training corpus into clusters, trains an expert independently on each cluster, and combine experts' predictions using a separately trained router. The method reduces communication between clusters and improves training efficiency.

Critical strengths
  1. The method improves training throughput by up to 24% while improving or maintaining downstream performance over the monolithic baseline.
Critical weaknesses
  1. The evidence for how the training-efficiency benefit scales with model size is limited. Figure 2a reports training throughput improvements only for 250M and 700M models, why is training throughput comparison of 1.2B not shown?
Questions for authors

See weaknesses.

3 comments

W1: Training-efficiency benefit at 1.2B.

We did not include 1.2B in Table 2 because 24 hours on eight single-GPU GH200 nodes is far too short to train a 1.2B model. The models would be so undertrained that their PPL and downstream scores could not reflect the performance of the methods. To show the throughput improvements over monolithic training, we measure the training throughput of 1.2B models in the Table 2 setting. D-DLM improves the training token per second by about 19%.

A better way to show how the benefit scales with model size is the exposed communication share, which is the communication time not overlapped with computation (the net overhead in Fig. 2a). The new Fig. 2a in the revision reports it at all three scales on 8×H100 nodes, with details in the new Appendix H. Against monolithic training, D-DLM lowers the exposed share by 34%, 36%, and 40% at 250M, 700M, and 1.2B. Thus, the relative benefit grows with model size. Table 1 shows the same trend for the cross-node traffic. For monolithic training, it grows from 30.2 TB at 250M to 129.1 TB at 1.2B, but it stays zero for D-DLM.

Thanks for the response, I will keep my positive rating.

We thank the reviewer for keeping the positive assessment of our work.

Reviewer_VNo3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposed a communication efficient procedure for training diffusion language models, where full models are hosted on different devices but are considered different experts. The data is clustered into topics, and each expert trains on their own data cluster. A router also learns to predict the cluster (no added communication, can be on another device). At inference, all experts are used, with the router weighting their outputs. This high level architecture has been used with AR transformers, but the paper shows interesting characteristics when using it for diffusion language models: the posterior mixture can be used in the reverse time dynamics and the router does nto need explicit time conditioning.

Critical strengths
  • Interesting and novel direction, showing that a strong model can be trained with no communication between devices.
  • There are substantial experiments showing that the results can be strong (though not necessarily showing it is better than closely related alternatives).
  • ablations investigate important design choices
  • some exploration into reducing inference cost
Critical weaknesses
  • That this works cleanly from forward process to reverse process is nice, but the maths seems straightforward and some of the results (e.g. the router does not need explicit time conditioning) seem to have been discovered already. It would be good to distinguish what has been known before and what is new.
  • Your results do not take into consideration the extra usage via multiple experts, you should compare to ensembled or model merged options.
  • It would be nice if there was clear exploration of how of the performance is from capacity and how much is from specialization.
  • Is the trained router cost high? This is not factored in the comparison.
Questions for authors

The trade-off is worse for inference, it seems like there is a lot that can be done there to compress or combine the experts?

9 comments

W1: Novel contributions of our work.

To the best of our knowledge, contributions listed in our paper are all new. They include the decentralized form of the discrete reverse process, the time-independent cluster posterior, and the D-DLM framework that we build on them. We understand that the method can look straightforward from a bottom-up view ("train experts, then ensemble them"). But we derive it top-down from the principles of discrete diffusion. Thus, both the expert training and the combination at inference follow from the formulation findings. Below, we clarify what is known and what is new for each point.

  1. Decomposition of the discrete probability velocity (Prop. 3.1). The reviewer may know: AR expert methods, such as Li et al. (2022), Gururangan et al. (2023), and Shi et al. (2025), combine next-token distributions, with domain or knowledge experts as their motivation, which look similar to our approach. Decentralized diffusion models, such as McAllister et al. (2025) and Jiang et al. (2025), combine continuous scores on images. However, our work is essentially different: Text tokens are discrete, and DLM denoising is a CTMC, as in Campbell et al. (2024) and Gat et al. (2024). Sec. 3.1 shows that the probability velocity of this CTMC decomposes into cluster-conditional velocities, weighted by the cluster posterior. Thus, the experts train without cross-expert synchronization and still recover the centralized reverse process in the realizable limit. No prior work gives this decomposition for a categorical reverse process. The decomposition applies to both masked and uniform-state diffusion. We use MDLM because it is the most mature choice for language. With MDLM, the router-weighted combination of the experts' clean-token posteriors comes from this decomposition, not from a heuristic ensemble. It decomposes each denoising step, not the sequence likelihood. The AR likelihood mixture is only the special case with block size 1 (see W1 of our response to Reviewer d6wx).

  2. A time-free router must be proved. AR expert methods route by knowledge domain, not by denoising step. Thus, they do not need to ask if the router depends on time. When we route at each denoising step, this question is necessary. In the DLM literature, time-independence is known for the expert LM: Gat et al. (2024) and Zheng et al. (2024) show that the token denoiser of masked diffusion does not depend on t, and we use this result for our experts (Section 3.2). For the router, the weight pt(y∣z)p_t(y|z) in Prop. 3.1 depends on t in general, and the closest prior work, McAllister et al. (2025), conditions its router on t. Our Prop. C.1 (Appendix C) proves that pt(y∣z)p_t(y|z) also does not depend on t under masked diffusion. No prior work gives this result. Without it, the time-free router in Section 3.2 has no justification. The property is not general. Under uniform-state diffusion, the router needs t.

  3. This is not the known time-independence of masked denoisers. The known result is about the token denoiser, which is a language-modeling quantity. Prop. C.1 is a routing result about a different quantity under a different formulation: the posterior of a sequence-level cluster label. An arbitrary data partition defines this label, and a classifier learns it. Thus, the known result does not cover our router, and no prior DLM work has such a posterior to analyze.

Beyond the new formulation results, we show that k-means clusters give DLM experts lower losses and a learnable router, but random partitions do not (Sec. 4.3 and Appendix D).

We hope that this makes clear that the simple final recipe is a result of our formulation and training practice, not a sign that our method is straightforward.

  • Li et al. (2022): Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
  • Gururangan et al. (2023): Scaling Expert Language Models with Unsupervised Domain Discovery
  • Shi et al. (2025): FlexOlmo: Open Language Models for Flexible Data Use
  • McAllister et al. (2025): Decentralized Diffusion Models
  • Jiang et al. (2025): Paris: A Decentralized Trained Open-Weight Diffusion Model
  • Campbell et al. (2024): Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design
  • Gat et al. (2024): Discrete Flow Matching
  • Zheng et al. (2024): Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

W2: Extra usage of multiple experts, ensembles, and model merging.

We already account for the extra usage in both training and inference. For training, the monolithic model and the four D-DLM experts in Table 1 use the same four nodes, the same per-node batch, and the same training steps. Thus, the experts together use the same compute as the monolithic model. The router cost is small, and we discuss it in W4. For inference, Fig. 4 gives the FLOPs of each routing scheme, router included. Top-1 with a 75M router costs about 1.1x the monolithic model (370 vs. 327 TFLOPs) and lowers PPL from 50.29 to 47.13. We also add a new result at 1.2B: with nearly matched training and inference compute, Top-1 with a 250M router lowers WikiText-2 PPL from 33.73 to 31.70.

Ensemble. D-DLM is itself an ensemble. But its weights are the cluster posterior from Prop. 3.1, not a heuristic. A fair ensemble baseline must also match the training compute, so each member can train only on a part of the data. An ensemble of four full-data monolithic models costs 4x the training compute. The plain ensemble of our experts ("Avg.", a uniform average of their predictions) is exactly the fair baseline. In Fig. 4, it gets 48.00 PPL at 4x the monolithic inference cost. This is worse than our full routing (44.34), and also worse than Top-1 with a 75M router (47.13) at about 1.1x the cost. Another new result at 1.2B: the plain ensemble gets 34.00 WikiText-2 PPL. This is even worse than the monolithic model (33.73), while full routing lowers the PPL to 30.57 (Table 1).

Model merging. Merging the experts into a single model is a natural next step of our decentralized training. We try a naive merging baseline: the uniform weight average of the four 1.2B experts gives a mean held-out loss of 7.22 over five test sets, against 3.78 for the monolithic model. Thus, merging independently trained experts needs more advanced methods. A straightforward way is to distill the experts into one model, as in Wan et al. (2024), with the router-weighted mixture of their predictions as the target distribution. This direction needs its own study, and we leave it for future work. DiLoCo, a compared method in Table 1, can be seen as a strong baseline with an online merge operation: its outer loop gradually merges the weights of the workers during training. As Table 1 shows, D-DLM performs comparably to DiLoCo without any synchronization and merging step.

  • Wan et al. (2024): Knowledge Fusion of Large Language Models

W3: Capacity vs. specialization.

The new Appendix I in the revision gives the clearest evidence. Its Fig. 7 reports the PPL of each expert, the monolithic model, and D-DLM on each data cluster and on the full set. Each expert has the same capacity as the monolithic model. On its own cluster, every expert beats the monolithic model, at both 250M and 1.2B (1.2B: 22.67 vs. 24.30, 26.12 vs. 27.67, 24.32 vs. 25.68, and 17.44 vs. 19.33). On the other clusters and on the full set, every expert is much worse (1.2B full set: 31.16 to 34.38 vs. 25.04). Thus, the gains of the experts come from specialization, not from capacity. D-DLM is the best on every cluster and on the full set (1.2B: 23.54), so the router combines these specializations.

Three more results agree on specialization. First, Top-1 routing adds no LM capacity, but it still lowers PPL from 50.29 to 47.13 at 250M (Fig. 4) and from 33.73 to 31.70 at 1.2B (new result presented in W2). Second, the plain ensemble uses all four experts with uniform weights, and it is worse than routing (48.00 vs. 44.34 at 250M in Fig. 4, and 34.00 vs. 30.57 at 1.2B as a new result presented in W2). Third, with random partitions (Sec. 4.3), the capacity stays the same but the specialization disappears. The experts then converge to higher losses than the monolithic model (Fig. 3).

W4: Router cost.

The router cost is small, and our comparisons include it. In Table 1, we train a router of the same size as the experts on a separate node. But the router does not need this budget. Its training converges fast (Fig. 3c), and a much smaller router gives nearly the same results. At 1.2B, a 250M router costs less than 3% of the expert training compute. In our new experiments, 1.2B experts with a 250M router give 30.71 WikiText-2 PPL with full routing, close to 30.57 with the 1.2B router. The router also trains on its own, without any communication with the experts. For inference, the FLOPs in Fig. 4 include the router, and a 75M router is enough (44.18 vs. 44.34 PPL for a 250M router with full routing). In parallel serving, the router runs on its own GPU at the same time as the experts (Fig. 2c).

Q: Inference trade-off and compressing or combining the experts.

We agree that this direction has much room, and the paper already gives three ways to reduce the inference cost. First, Top-k routing with a small router runs only one or two experts per step. Top-1 with a 75M router costs about 1.1x the monolithic model and still has lower PPL (Fig. 4, W2). Second, the router and the experts run in parallel on separate GPUs, so the latency stays within 1.01x of the monolithic model at every scale (Fig. 2c). Third, the experts can be combined into one model by distillation, with the router-weighted mixture as the target (W2). This needs its own study, and we leave it for future work.

Reviewer_d6wxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cross-node synchronization costs and diminishing returns from larger global batches in DLM training. It proposes clustering the corpus, training one DLM expert per cluster without cross-expert synchronization, and training a router to predict cluster membership from noisy sequences. At inference time, expert predictions are combined using router weights. Experiments cover models with up to 1.2B backbone parameters and 80B training tokens, including comparisons with monolithic training and DiLoCo.

Critical strengths

The paper provides a principled formulation for combining independently trained DLM experts and evaluates the approach under both large-batch and communication-constrained training settings. However, I find the empirical evidence for a compelling practical advantage limited.

Critical weaknesses
  1. Reducing synchronization is an interesting research direction, but its connection to DLMs needs stronger motivation. The underlying bottleneck also applies to autoregressive LLMs, and the paper acknowledges prior approaches using independently trained AR experts. Although the reverse-process decomposition provides a DLM-specific justification, the conceptual advance over existing expert-composition approaches remains unclear to me.

  2. Absolute performance remains low on several benchmarks: Table 1 reports ARC-C scores of 20.7–23.6%, HellaSwag scores of 27.2–33.9%, and OpenBookQA scores of 24.8–28.6%. This limits how convincing the experiments are as evidence for scaling stronger DLMs. Uncertainty estimates would also help assess the robustness of the modest improvements.

  3. The perplexity results are mixed. At 700M in Table 2, D-DLM has worse reported perplexity bounds on three of four corpora. DiLoCo also achieves better reported perplexity bounds at both 250M and 700M in Table 1.

  4. Figure 1a does not clearly distinguish time spent executing NCCL collectives from communication overhead that actually delays training. Collectives may overlap with computation, and the measurement includes intra-node communication. Clarifying overlap accounting and isolating the exposed cross-node overhead would strengthen the motivation.

Questions for authors
  1. Could the authors provide a better-tuned monolithic baseline at 1.2B/80B, particularly through batch-size and optimization-schedule ablations? The 1.2B baseline performs worse than the 700M baseline, while its per-node batch size doubles, resulting in fewer updates under the same token budget.

  2. Could the authors provide comparisons with both training and inference budgets matched, especially at 1.2B/80B? Figure 4 provides a nearly inference-compute-matched comparison at 250M, but each of its four experts receives the same batch size and number of training steps as the monolithic baseline, substantially increasing aggregate training compute.

2 comments

Q1: Updated Table 1 results.

We have updated the Table 1 setting (highlighted in blue in the revision): all methods at all three scales are better tuned and now trained for the same number of training steps (2,500) instead of the same budget of 80B training tokens. The 1.2B models no longer receive fewer updates than the 700M models. For 1.2B DiLoCo, we use H=25 and an outer learning rate of 0.2 (best out of 0.3, 0.5, 0.6, and 0.7), while a larger outer learning rate (e.g., 0.7 in the initial submission version) diverges at this scale. In the updated Table 1, the 1.2B monolithic baseline outperforms the 700M one, and D-DLM improves both the perplexity upper bound and the average downstream score over monolithic training at every scale with zero cross-node traffic. D-DLM performs comparably to DiLoCo, with the best average downstream score at every scale and the lowest perplexity upper bound at 1.2B.

Q2: Comparison with both training and inference budgets matched.

The training budget is already matched in Table 1: the monolithic model and the four D-DLM experts use the same four nodes, the same per-node batch, and the same 2,500 training steps. Each expert thus trains with 1/4 of the monolithic (global) batch, so the four experts together process the same number of tokens as the monolithic model. Only the ablations in Table 3 and Fig. 4 give each expert the full monolithic batch, to isolate the effect of the data partition and the routing scheme. The router adds little to this budget: its training converges very fast (Fig. 3c), and a smaller router gives comparable performance (Fig. 4). Thus the router training cost is negligible in large-scale training.

For inference, D-DLM can use Top-1 routing (Fig. 4), which runs a single expert per denoising step. Each expert has the same size as the monolithic model, so the language model that predicts the tokens costs exactly the same as the monolithic model. To reduce the inference costs from router model, we choose a relatively small 250M router model. With the 1.2B checkpoints, the perplexity upper bound on WikiText-2 (lower is better) is:

1.2B modelWikiText-2 PPL
Monolithic33.73
D-DLM, Top-131.70

Thus, with the nearly matched inference compute per step as the monolithic model, D-DLM can still achieve a lower perplexity, leading to similar conclusion from Fig. 4.

W1: Connection to DLMs and advance over AR expert composition.

Our method has clear concept advances in three ways:

  1. Where the decomposition comes from. The decentralized form follows from the diffusion denoising formulation itself, which removes gradient synchronization from LM training. AR expert-combination methods instead rely on the exact next-token likelihood decomposition and on the intuition of knowledge experts. We decompose the discrete diffusion denoising step which is neither simple combination of likelihood nor intuition-driven. The decomposition also applies to block and uniform-state diffusion. We use a masked diffusion backbone in experiments because it is the most mature choice for language.

  2. Routing during decoding. Our router reads a partially masked sequence with masks at arbitrary positions, weighs the experts' predictions for all masked tokens jointly, and is re-evaluated at every denoising step. A fully masked sequence carries no routing information, so the router starts near uniform. LLaDA sampling decodes high-confidence tokens first, which gradually reveals the key information for routing, so the router weights sharpen as masks are revealed. Fig. 6 in Appendix G (now referenced in Section 3.2) shows the router entropy falling from near-uniform to a confident region.

  3. AR composition is a special case. AR can be written as block masked diffusion with block size 1, causal attention, and logit shift. Independently trained AR experts combined afterwards are thus a special case of our method, where each denoising step covers a single token. Only in this case is the exact likelihood decomposition valid, and there is no entropy-drop phase because each block has a single decoding step.

W2: Absolute performance and uncertainty estimates.

The absolute scores are normal for DLMs at our model scale and training budget, so our training behaves as expected. For reference, Nie et al. (2024) report 39.0 / 44.4 on ARC-Easy for 220M / 1.1B MDMs, and 26.2 on OpenBookQA for a 220M AR model (with different data and evaluation protocols from us), while D-DLM obtains 42.1-54.8 on ARC-Easy and 27.0-27.6 on OpenBookQA. Our goal is a controlled comparison of training methods, not state-of-the-art accuracy: all methods share the same data, training steps, and evaluation, so their relative gains remain valid evidence at this scale. For uncertainty, we computed standard errors (std. err.) for the Table 1 comparisons. Against monolithic training, D-DLM lowers the held-out loss on WikiText-2 and LAMBADA by more than 4.0× the std. err. at every scale, and raises the average accuracy by 1.02, 1.65, and 1.29 points (2.6×, 4.2×, and 3.1× the std. err.) at 250M / 700M / 1.2B. The larger this multiple is, the less likely the gain comes from randomness. When it's above 2×, a gain is significant at the 95% level.

Nie et al. (2024): Scaling up Masked Diffusion Models on Text

W3: The perplexity results are mixed.

Under matched compute, decentralized training usually trades performance for efficiency and low coupling. In Table 1, each expert trains with 1/4 of the monolithic batch and no gradient synchronization, whereas a 4x larger synchronized batch is normally expected to perform better at the same number of training steps, even in the batch-saturation region. With k-means data partitioning, D-DLM still matches or exceeds full DDP, so scaling across nodes with decentralized experts instead of more data parallelism gives low communication cost and low coupling nearly for free. Likewise, in Table 2, D-DLM achieves perplexity and downstream scores comparable to monolithic training, while training 21-24% faster within the same 24-hour budget. DiLoCo does not fully drop gradient synchronization and adds an outer optimizer. D-DLM performs comparably to it with zero gradient synchronization. In the submitted version, 1.2B DiLoCo underperformed because its outer optimization diverged, which shows that DiLoCo's outer-optimizer hyperparameters can add tuning cost. As requested by the reviewer, we tuned 1.2B DiLoCo to avoid this divergence, and found D-DLM still outperforms the new 1.2B DiLoCo.

W4: Communication share without overlap with computation.

We agree that the NCCL time in Fig. 1a includes communication overlapped with computation and intra-node traffic. To show the overhead that actually delays training, we measure the net (exposed) overhead, i.e., the NCCL time not overlapped with computation (see the new Appendix H for details), in a practical scaled-out training setting of 4x nodes. The table below gives the communication share and the net overhead, in % of the step time:

Model sizeMonolithic (share / net)DiLoCo (share / net)D-DLM (share / net)
250M53.2 / 43.647.3 / 38.638.7 / 28.8
700M48.6 / 28.946.8 / 26.639.8 / 18.4
1.2B49.1 / 19.739.7 / 18.034.7 / 11.9

D-DLM has the lowest communication share and net overhead at every scale, reducing the net overhead by 34-40% relative to monolithic training and by 25-34% relative to DiLoCo. We also added Fig. 2a in the revision to visualize these results. To isolate cross-node communication, Table 1 reports the gradient traffic between the four training nodes, measured at their RDMA ports: 30-129 TB for monolithic training, 0.62-4.86 TB for DiLoCo, and zero for D-DLM, whose experts synchronize only within their own node. Hence, D-DLM's net overhead above is purely intra-node, and the extra net overhead of the baselines mainly comes from cross-node synchronization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.