- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Shows a connection between SEDD, MDLM/GIDD, M2S, and Neural CTMC, and shows that those methods loss function are actually equivalent, in a unified mathematical formulation.
Under the unified formulation, the actual differences boil down to parameterization and training details like gradient clipping.
They use a transfer learning recipe transform an AR model into a uniform-noise diffusion model.
Critical strengths
Nice unification of the different models and losses, and working out the actual differences (parameterization).
Critical weaknesses
Comparison of parameterizations is incomplete. Figure 1 only compares GIDD/MDLM and M2S, but not SEDD, neural CTMC. There is no systematic direction comparisons of all parameterizations.
Fig 4, table 2, table 4 comparisons unfair: Unifusion starts from GPT2 (124M), pretrained on massive data, and then finetunes on FineWeb Baselines (SEDD, GIDD, M2S, Neural CTMC) use DiT architecture (170M) trained from scratch on OpenWebText.
Unifusion is presented as a new transfer learning recipe. But in fig 4, table 2, table 4, elsewhere, unifusion is used to refer to a model. This is misleading. You show that it is basically always the same model (maybe different parameterization), same loss. So the actual difference is the training recipe, that unifusion is taking an existing pretrained model and adapts it. Obviously that should be better when the base model was trained on much more data.
The more interesting comparison would be to compare it to other transfer learning methods, always taking the same base model (GPT2).
The mathematical unification and the actual transfer learning recipe are largely orthogonal, independent.
The mathematical unification, while nice to have, does not contribute to any new simpler loss or better parameterization. It is not shown that there is practical use for it, despite the better understanding of the relation of the models and losses.
No real ablations on the transfer learning recipe. Even simple baselines like random vs pretrained initialization (on consistent architecture, consistent data) is missing. Or varying the base model, showing that the recipe generalizes. Or training on larger models. Studying scaling laws.
Questions for authors
GPT2 checkpoint as base / "Through systematic evaluation of 124M- and 355M-parameter models" - Isn't this too small? Wouldn't it be good to also see other examples despite GPT2 as base?
Fig 3, AR -> uniform -> mask results in much worse GenPPL. Why?
"Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation" (https://arxiv.org/pdf/2605.22765) has a similar conversion formula. Can you point out the differences?
How much does the noise schedule matter?
How novel is transfer learning recipe to transform an AR model into a uniform-noise diffusion model really? E.g. DiffusionGemma )(https://arxiv.org/abs/2608.00146), don't they do the same?
More in general: How relevant is the transfer learning from AR? Isn't this somewhat only relevant for the temporary situation that we are in right now, where there are some existing autoregressive models? Given enough compute, doesn't it make more sense to train the diff LM from scratch?
12 comments
Reviewer comment. Fig 3, AR -> uniform -> mask results in much worse GenPPL. Why?
Response. Thank you for your question. Figure 3 shows different behavior in the two transfer directions. In AR→mask→uniform, GenPPL continues to decrease after switching to uniform diffusion, indicating that the masked model's weights provide an effective initialization for uniform training. In AR→uniform→mask, GenPPL decreases during the uniform stage, rises substantially after switching to masked diffusion, and then decreases again as masked training proceeds. This indicates that the generation performance achieved during uniform training does not carry over directly to the masked setting, requiring further adaptation after the switch.
Although both models use the same prediction form, their corrupted training inputs and reverse generation processes differ. This shared prediction form enables weight reuse but does not guarantee that generation quality is preserved when switching corruption kernels. Under our experimental settings, the mask→uniform transition is relatively smooth, whereas the uniform→mask transition incurs a substantial performance drop. Further training partially recovers performance, but within the given budget, it does not reach the level of the other paths.
Reviewer comment. "Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation" (https://arxiv.org/pdf/2605.22765) has a similar conversion formula. Can you point out the differences?
Response. Thank you for pointing out this connection. Under uniform corruption, the conversion from the leave-one-out prediction to the denoising posterior in Eq. (11) of that work has the same mathematical form as our conversion. Its score conversion in Eq. (12) also corresponds to our conversion. We will explicitly state these correspondences alongside our conversion formulas.
Our paper further provides conversions from the same prediction to Neural CTMC's exit rate and jump distribution. We also prove that, under the same forward process and noise schedule, the losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC correspond to a common reverse-rate objective when they induce the same reverse rates. For model adaptation, we use these conversions to retain the pretrained AR model's output head, treat its shifted output as an prediction, and construct the predictions required by each method.
Reviewer comment. How much does the noise schedule matter?
Response. Thank you for this question. We conducted a controlled ablation of the noise schedule. For the GPT2-small model, we kept the AR initialization, FineWeb data, sequence length of 512, 240,000 training steps, GIDD objective, and other optimization settings fixed. We varied only the forward noise schedule and used the corresponding reverse sampling process. We compared Linear , Quadratic , and Cosine-squared schedules.
| Noise schedule | 16 steps | 32 steps | 64 steps | 128 steps | 256 steps |
|---|---|---|---|---|---|
| Linear | 143.944 ± 0.872 | 114.571 ± 0.850 | 103.441 ± 1.394 | 99.199 ± 0.930 | 97.783 ± 1.901 |
| Cosine-squared | 145.1152 ± 0.8814 | 114.8905 ± 1.3542 | 105.1445 ± 1.6455 | 100.6836 ± 1.7380 | 98.2249 ± 0.9824 |
| Quadratic | 165.4526 ± 0.8005 | 131.3244 ± 0.7807 | 121.1255 ± 1.0784 | 119.1245 ± 0.7741 | 118.6494 ± 1.8356 |
The results show that the noise schedule has a clear effect on generation quality. Across all five sampling budgets, Linear achieves lower GenPPL than Quadratic and slightly lower GenPPL than Cosine-squared. At 256 sampling steps, the GenPPL values for Linear, Cosine-squared, and Quadratic are , , and , respectively. Cosine-squared performs close to Linear, whereas Quadratic yields substantially higher GenPPL.
Reviewer comment. How novel is transfer learning recipe to transform an AR model into a uniform-noise diffusion model really? E.g. DiffusionGemma (https://arxiv.org/abs/2608.00146), don't they do the same?
Response. Thank you for pointing out this related work. DiffusionGemma also starts from a pretrained AR model and adapts it to uniform-noise diffusion, so our work shares this goal. According to the initial arXiv submission records, a related preprint was submitted on July 27, 2026, and DiffusionGemma on July 31, 2026.
The two methods differ in their training objectives and procedures. DiffusionGemma uses block-wise generation with KV caching and self-conditioning, denoising 256 tokens at a time. Its initial adaptation stage uses a cross-entropy loss for clean-token denoising (Eq. (11) of DiffusionGemma):
=-\sum_{i=1}^{C}\log p_\theta(x_0^i\mid x_t,z_t,H), \qquad C=256,$$ where $z_t$ is the self-conditioning signal and $H$ is the KV cache of the clean context. This stage is followed by joint reinforcement learning and sampler distillation to improve generation quality and inference efficiency. We retain the AR model's output head and use a one-position output shift to obtain an $x_0$ prediction. Together with causal-to-bidirectional attention annealing, we use the GIDD loss to continue pretraining on full sequences under uniform corruption. Its token-level form is $$\mathcal{L}_{\mathrm{GIDD}}(\theta)=\mathbb{E}_{t,x_t}\Bigg[w_t(x_t,x_0)\Bigg(\operatorname{KL}\left(q_t(\cdot\mid x_0) \Vert q_t(\cdot\mid x_\theta)\right)+\frac{q_t(x_t\mid x_0)}{q_t(x_t\mid x_\theta)}-\log\frac{q_t(x_t\mid x_0)}{q_t(x_t\mid x_\theta)}-1\Bigg)\Bigg].$$ Here the expectation is over $t\sim\mathrm{Unif}(0,1)$ and $x_t\sim q_t(\cdot\mid x_0)$, and the loss is summed over sequence positions. Our paper also examines how prediction parameterizations and training objectives support this adaptation. We prove that, under the same forward process and noise schedule, the losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC correspond to a common reverse-rate objective when they induce the same reverse rates, and derive conversions from an $x_0$ prediction to each parameterization. **Reviewer comment.** **A comparison between random and pretrained initialization using the same architecture and training data is missing.** **Response.** Thank you for your suggestion. We have now conducted an initialization ablation at both GPT2-small (124M) and GPT2-medium (355M) scales. At each scale, we keep the model architecture, FineWeb training data, cumulative number of diffusion-training tokens, all other training settings, and evaluation procedure fixed, varying only whether the model starts from randomly initialized weights or pretrained AR weights. The table below reports GenPPL at all five sampling budgets, evaluated using GPT2-large. All values are the mean ± sample standard deviation across five generation seeds, with 1,024 samples per seed. | Model scale | Initialization | 16 steps | 32 steps | 64 steps | 128 steps | 256 steps | |---|---|---:|---:|---:|---:|---:| | GPT2-small | Random | 148.781 ± 0.885 | 118.967 ± 0.943 | 106.583 ± 1.095 | 102.377 ± 0.729 | 101.324 ± 1.247 | | GPT2-small | AR pretrained | 143.944 ± 0.872 | 114.571 ± 0.850 | 103.441 ± 1.394 | 99.199 ± 0.930 | 97.783 ± 1.901 | | GPT2-medium | Random | 140.411 ± 1.929 | 97.773 ± 1.780 | 83.417 ± 1.086 | 77.448 ± 1.313 | 75.481 ± 0.834 | | GPT2-medium | AR pretrained | 128.832 ± 1.968 | 92.007 ± 1.547 | 79.207 ± 1.215 | 74.162 ± 0.901 | 71.516 ± 1.165 | At both model scales and all five sampling budgets, AR pretrained initialization yields lower mean GenPPL. At 256 sampling steps, it reduces GenPPL by approximately 3.5% for GPT2-small and 5.3% for GPT2-medium relative to random initialization. These results show that reusing pretrained AR weights improves the GenPPL of generated text under the matched diffusion-training budgets evaluated here.
Response to Reviewer : we thank the reviewer for the careful and constructive comments.
Reviewer comment. Comparison of parameterizations is incomplete. Figure 1 only compares GIDD/MDLM and M2S, but not SEDD, neural CTMC. There is no systematic direction comparisons of all parameterizations.
Response. Thank you for your suggestion. Figure 1 tests how to use the pretrained AR model's output. After a one-position shift, the AR output is a probability distribution over the vocabulary and can be used either as the x0 prediction in MDLM/GIDD or as the μ prediction in M2S. We therefore compare these two uses under the same AR initialization and the same M2S loss. Using the AR output as an x0 prediction and then converting it to μ yields 48.6% lower average loss over the first 2,000 updates, so we adopt this approach.
SEDD predicts probability ratios, while Neural CTMC predicts an exit rate and a jump distribution. These quantities have different meanings from the token probabilities produced by an AR model. We construct the corresponding predictions using the conversion formulas in our paper. Figure 2 and Table 1 compare the training behavior of all four objectives using the same AR initialization, training data, and training budget.
Reviewer comment. Fig 4, table 2, table 4 comparisons unfair: Unifusion starts from GPT2 (124M), pretrained on massive data, and then finetunes on FineWeb Baselines (SEDD, GIDD, M2S, Neural CTMC) use DiT architecture (170M) trained from scratch on OpenWebText. The more interesting comparison would be to compare it to other transfer learning methods, always taking the same base model (GPT2).
Response. Thank you for your suggestion. This paper studies whether pretrained AR weights can be reused to obtain a uniform-noise diffusion model through continued training. Our main comparison is therefore with DiffuGPT, which also adapts pretrained AR models. At both model scales, the two methods start from the corresponding GPT2 pretrained weights and continue training on 30B tokens of FineWeb data. DiffuGPT uses mask corruption, while our method uses uniform corruption. We use its publicly released checkpoints and evaluate both methods with the same evaluation procedure. The cumulative training token counts for both methods are reported in Appendix C.3.
The other diffusion baselines retain their original architectures and training data. We use publicly released checkpoints where available and otherwise train the models following their original settings. We agree that retraining these baselines on the same training data and with the same budget would allow a more direct comparison of training methods. Due to limited compute, we have not completed such matched retraining, and we will explicitly state this limitation in the paper.
Reviewer comment. Unifusion is presented as a new transfer learning recipe. But in fig 4, table 2, table 4, elsewhere, unifusion is used to refer to a model. This is misleading. You show that it is basically always the same model (maybe different parameterization), same loss. So the actual difference is the training recipe, that unifusion is taking an existing pretrained model and adapts it. Obviously that should be better when the base model was trained on much more data.
Response. Thank you for your suggestion. Unifusion refers to the training method for adapting pretrained AR models to uniform-noise diffusion, and UnifusionGPT refers to the models obtained through this method. UnifusionGPT retains the architecture of the corresponding GPT2 model and uses GIDD's training objective. We will use these model names consistently in the main text, figures, and tables.
We agree that pretrained weights themselves can improve performance. Our main comparison is therefore with DiffuGPT, which also reuses AR weights. At both GPT2 scales, the two methods start from the corresponding pretrained weights and continue training on a 30B-token FineWeb corpus. DiffuGPT adapts these checkpoints to masked diffusion, while Unifusion adapts them to uniform diffusion. Both methods therefore benefit from the knowledge acquired during AR pretraining. At 256 sampling steps, compared with DiffuGPT, UnifusionGPT-S reduces GenPPL by 16.7% with similar, slightly lower unigram entropy, while UnifusionGPT-M reduces GenPPL by 21.1% with higher unigram entropy. Moreover, UnifusionGPT-M at just 64 steps achieves lower GenPPL and higher unigram entropy than DiffuGPT-M at 256 steps.