- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a decentralized training and inference framework for diffusion language models. It divides the training corpus into clusters, trains an expert independently on each cluster, and combine experts' predictions using a separately trained router. The method reduces communication between clusters and improves training efficiency.
Critical strengths
- The method improves training throughput by up to 24% while improving or maintaining downstream performance over the monolithic baseline.
Critical weaknesses
- The evidence for how the training-efficiency benefit scales with model size is limited. Figure 2a reports training throughput improvements only for 250M and 700M models, why is training throughput comparison of 1.2B not shown?
Questions for authors
See weaknesses.
3 comments
Thanks for the response, I will keep my positive rating.
We thank the reviewer for keeping the positive assessment of our work.
W1: Training-efficiency benefit at 1.2B.
We did not include 1.2B in Table 2 because 24 hours on eight single-GPU GH200 nodes is far too short to train a 1.2B model. The models would be so undertrained that their PPL and downstream scores could not reflect the performance of the methods. To show the throughput improvements over monolithic training, we measure the training throughput of 1.2B models in the Table 2 setting. D-DLM improves the training token per second by about 19%.
A better way to show how the benefit scales with model size is the exposed communication share, which is the communication time not overlapped with computation (the net overhead in Fig. 2a). The new Fig. 2a in the revision reports it at all three scales on 8×H100 nodes, with details in the new Appendix H. Against monolithic training, D-DLM lowers the exposed share by 34%, 36%, and 40% at 250M, 700M, and 1.2B. Thus, the relative benefit grows with model size. Table 1 shows the same trend for the cross-node traffic. For monolithic training, it grows from 30.2 TB at 250M to 129.1 TB at 1.2B, but it stays zero for D-DLM.