acceptodds
Under review as a conference paper at ICLR 2027

Unifusion: adapting autoregressive language models into discrete diffusion under a unified reverse-rate objective

Abstract

Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling. However, adapting AR checkpoints across corruption kernels remains challenging because existing DLMs use different objectives and prediction parameterizations. We establish connections among SEDD, MDLM/GIDD, M2S, and Neural CTMC by expressing their conditional losses as a single generalized Kullback–Leibler objective over model reverse rates. We further derive conversions from clean-token predictions to concrete-score, posterior-mean, and exit-rate/jump parameterizations, yielding a shared \(x_0\) interface that supports switching between mask and uniform kernels. Building on these connections, we propose , a simple continual pre-training approach for directly adapting pretrained GPT2 checkpoints to uniform-noise diffusion. Through systematic evaluation of 124M- and 355M-parameter models, we show that steadily improves the trade-off between generative perplexity (GenPPL) and unigram entropy as the sampling budget increases from 16 to 256 steps. At 256 steps, -S and -M achieve GenPPL/entropy pairs of \(97.783/5.2626\) and \(71.516/5.6669\), respectively; no evaluated model at the same scale simultaneously outperforms on both metrics. At both scales, also achieves the highest WinoGrande, SIQA, and BBH accuracy among the compared diffusion models.

Primary area:
generative models
Keywords:
discrete diffusion models, large language model, autoregressive language models, model adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_trLeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Shows a connection between SEDD, MDLM/GIDD, M2S, and Neural CTMC, and shows that those methods loss function are actually equivalent, in a unified mathematical formulation.

Under the unified formulation, the actual differences boil down to parameterization and training details like gradient clipping.

They use a transfer learning recipe transform an AR model into a uniform-noise diffusion model.

Critical strengths

Nice unification of the different models and losses, and working out the actual differences (parameterization).

Critical weaknesses

Comparison of parameterizations is incomplete. Figure 1 only compares GIDD/MDLM and M2S, but not SEDD, neural CTMC. There is no systematic direction comparisons of all parameterizations.

Fig 4, table 2, table 4 comparisons unfair: Unifusion starts from GPT2 (124M), pretrained on massive data, and then finetunes on FineWeb Baselines (SEDD, GIDD, M2S, Neural CTMC) use DiT architecture (170M) trained from scratch on OpenWebText.

Unifusion is presented as a new transfer learning recipe. But in fig 4, table 2, table 4, elsewhere, unifusion is used to refer to a model. This is misleading. You show that it is basically always the same model (maybe different parameterization), same loss. So the actual difference is the training recipe, that unifusion is taking an existing pretrained model and adapts it. Obviously that should be better when the base model was trained on much more data.

The more interesting comparison would be to compare it to other transfer learning methods, always taking the same base model (GPT2).

The mathematical unification and the actual transfer learning recipe are largely orthogonal, independent.

The mathematical unification, while nice to have, does not contribute to any new simpler loss or better parameterization. It is not shown that there is practical use for it, despite the better understanding of the relation of the models and losses.

No real ablations on the transfer learning recipe. Even simple baselines like random vs pretrained initialization (on consistent architecture, consistent data) is missing. Or varying the base model, showing that the recipe generalizes. Or training on larger models. Studying scaling laws.

Questions for authors

GPT2 checkpoint as base / "Through systematic evaluation of 124M- and 355M-parameter models" - Isn't this too small? Wouldn't it be good to also see other examples despite GPT2 as base?

Fig 3, AR -> uniform -> mask results in much worse GenPPL. Why?

"Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation" (https://arxiv.org/pdf/2605.22765) has a similar conversion formula. Can you point out the differences?

How much does the noise schedule matter?

How novel is transfer learning recipe to transform an AR model into a uniform-noise diffusion model really? E.g. DiffusionGemma )(https://arxiv.org/abs/2608.00146), don't they do the same?

More in general: How relevant is the transfer learning from AR? Isn't this somewhat only relevant for the temporary situation that we are in right now, where there are some existing autoregressive models? Given enough compute, doesn't it make more sense to train the diff LM from scratch?

12 comments
AuthorsOpenReview ↗

Response to Reviewer : we thank the reviewer for the careful and constructive comments.

Reviewer comment. Comparison of parameterizations is incomplete. Figure 1 only compares GIDD/MDLM and M2S, but not SEDD, neural CTMC. There is no systematic direction comparisons of all parameterizations.

Response. Thank you for your suggestion. Figure 1 tests how to use the pretrained AR model's output. After a one-position shift, the AR output is a probability distribution over the vocabulary and can be used either as the x0x_0 prediction in MDLM/GIDD or as the μ\mu prediction in M2S. We therefore compare these two uses under the same AR initialization and the same M2S loss. Using the AR output as an x0x_0 prediction and then converting it to μ\mu yields 48.6% lower average loss over the first 2,000 updates, so we adopt this approach.

SEDD predicts probability ratios, while Neural CTMC predicts an exit rate and a jump distribution. These quantities have different meanings from the token probabilities produced by an AR model. We construct the corresponding predictions using the conversion formulas in our paper. Figure 2 and Table 1 compare the training behavior of all four objectives using the same AR initialization, training data, and training budget.

Reviewer comment. Fig 4, table 2, table 4 comparisons unfair: Unifusion starts from GPT2 (124M), pretrained on massive data, and then finetunes on FineWeb Baselines (SEDD, GIDD, M2S, Neural CTMC) use DiT architecture (170M) trained from scratch on OpenWebText. The more interesting comparison would be to compare it to other transfer learning methods, always taking the same base model (GPT2).

Response. Thank you for your suggestion. This paper studies whether pretrained AR weights can be reused to obtain a uniform-noise diffusion model through continued training. Our main comparison is therefore with DiffuGPT, which also adapts pretrained AR models. At both model scales, the two methods start from the corresponding GPT2 pretrained weights and continue training on 30B tokens of FineWeb data. DiffuGPT uses mask corruption, while our method uses uniform corruption. We use its publicly released checkpoints and evaluate both methods with the same evaluation procedure. The cumulative training token counts for both methods are reported in Appendix C.3.

The other diffusion baselines retain their original architectures and training data. We use publicly released checkpoints where available and otherwise train the models following their original settings. We agree that retraining these baselines on the same training data and with the same budget would allow a more direct comparison of training methods. Due to limited compute, we have not completed such matched retraining, and we will explicitly state this limitation in the paper.

Reviewer comment. Unifusion is presented as a new transfer learning recipe. But in fig 4, table 2, table 4, elsewhere, unifusion is used to refer to a model. This is misleading. You show that it is basically always the same model (maybe different parameterization), same loss. So the actual difference is the training recipe, that unifusion is taking an existing pretrained model and adapts it. Obviously that should be better when the base model was trained on much more data.

Response. Thank you for your suggestion. Unifusion refers to the training method for adapting pretrained AR models to uniform-noise diffusion, and UnifusionGPT refers to the models obtained through this method. UnifusionGPT retains the architecture of the corresponding GPT2 model and uses GIDD's training objective. We will use these model names consistently in the main text, figures, and tables.

We agree that pretrained weights themselves can improve performance. Our main comparison is therefore with DiffuGPT, which also reuses AR weights. At both GPT2 scales, the two methods start from the corresponding pretrained weights and continue training on a 30B-token FineWeb corpus. DiffuGPT adapts these checkpoints to masked diffusion, while Unifusion adapts them to uniform diffusion. Both methods therefore benefit from the knowledge acquired during AR pretraining. At 256 sampling steps, compared with DiffuGPT, UnifusionGPT-S reduces GenPPL by 16.7% with similar, slightly lower unigram entropy, while UnifusionGPT-M reduces GenPPL by 21.1% with higher unigram entropy. Moreover, UnifusionGPT-M at just 64 steps achieves lower GenPPL and higher unigram entropy than DiffuGPT-M at 256 steps.

Reviewer comment. Fig 3, AR -> uniform -> mask results in much worse GenPPL. Why?

Response. Thank you for your question. Figure 3 shows different behavior in the two transfer directions. In AR→mask→uniform, GenPPL continues to decrease after switching to uniform diffusion, indicating that the masked model's weights provide an effective initialization for uniform training. In AR→uniform→mask, GenPPL decreases during the uniform stage, rises substantially after switching to masked diffusion, and then decreases again as masked training proceeds. This indicates that the generation performance achieved during uniform training does not carry over directly to the masked setting, requiring further adaptation after the switch.

Although both models use the same x0x_0 prediction form, their corrupted training inputs and reverse generation processes differ. This shared prediction form enables weight reuse but does not guarantee that generation quality is preserved when switching corruption kernels. Under our experimental settings, the mask→uniform transition is relatively smooth, whereas the uniform→mask transition incurs a substantial performance drop. Further training partially recovers performance, but within the given budget, it does not reach the level of the other paths.

Reviewer comment. "Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation" (https://arxiv.org/pdf/2605.22765) has a similar conversion formula. Can you point out the differences?

Response. Thank you for pointing out this connection. Under uniform corruption, the conversion from the leave-one-out prediction to the denoising posterior in Eq. (11) of that work has the same mathematical form as our xθ→μθx_\theta\to\mu_\theta conversion. Its score conversion in Eq. (12) also corresponds to our xθ→sθx_\theta\to s_\theta conversion. We will explicitly state these correspondences alongside our conversion formulas.

Our paper further provides conversions from the same x0x_0 prediction to Neural CTMC's exit rate and jump distribution. We also prove that, under the same forward process and noise schedule, the losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC correspond to a common reverse-rate objective when they induce the same reverse rates. For model adaptation, we use these conversions to retain the pretrained AR model's output head, treat its shifted output as an x0x_0 prediction, and construct the predictions required by each method.

Reviewer comment. How much does the noise schedule matter?

Response. Thank you for this question. We conducted a controlled ablation of the noise schedule. For the GPT2-small model, we kept the AR initialization, FineWeb data, sequence length of 512, 240,000 training steps, GIDD objective, and other optimization settings fixed. We varied only the forward noise schedule and used the corresponding reverse sampling process. We compared Linear (αt=1−t,βt=t)(\alpha_t=1-t,\beta_t=t), Quadratic (αt=1−t2,βt=t2)(\alpha_t=1-t^2,\beta_t=t^2), and Cosine-squared (αt=cos⁡2(πt/2),βt=sin⁡2(πt/2))(\alpha_t=\cos^2(\pi t/2),\beta_t=\sin^2(\pi t/2)) schedules.

Noise schedule16 steps32 steps64 steps128 steps256 steps
Linear143.944 ± 0.872114.571 ± 0.850103.441 ± 1.39499.199 ± 0.93097.783 ± 1.901
Cosine-squared145.1152 ± 0.8814114.8905 ± 1.3542105.1445 ± 1.6455100.6836 ± 1.738098.2249 ± 0.9824
Quadratic165.4526 ± 0.8005131.3244 ± 0.7807121.1255 ± 1.0784119.1245 ± 0.7741118.6494 ± 1.8356

The results show that the noise schedule has a clear effect on generation quality. Across all five sampling budgets, Linear achieves lower GenPPL than Quadratic and slightly lower GenPPL than Cosine-squared. At 256 sampling steps, the GenPPL values for Linear, Cosine-squared, and Quadratic are 97.783±1.90197.783\pm1.901, 98.2249±0.982498.2249\pm0.9824, and 118.6494±1.8356118.6494\pm1.8356, respectively. Cosine-squared performs close to Linear, whereas Quadratic yields substantially higher GenPPL.

Reviewer comment. How novel is transfer learning recipe to transform an AR model into a uniform-noise diffusion model really? E.g. DiffusionGemma (https://arxiv.org/abs/2608.00146), don't they do the same?

Response. Thank you for pointing out this related work. DiffusionGemma also starts from a pretrained AR model and adapts it to uniform-noise diffusion, so our work shares this goal. According to the initial arXiv submission records, a related preprint was submitted on July 27, 2026, and DiffusionGemma on July 31, 2026.

The two methods differ in their training objectives and procedures. DiffusionGemma uses block-wise generation with KV caching and self-conditioning, denoising 256 tokens at a time. Its initial adaptation stage uses a cross-entropy loss for clean-token denoising (Eq. (11) of DiffusionGemma):

=-\sum_{i=1}^{C}\log p_\theta(x_0^i\mid x_t,z_t,H), \qquad C=256,$$ where $z_t$ is the self-conditioning signal and $H$ is the KV cache of the clean context. This stage is followed by joint reinforcement learning and sampler distillation to improve generation quality and inference efficiency. We retain the AR model's output head and use a one-position output shift to obtain an $x_0$ prediction. Together with causal-to-bidirectional attention annealing, we use the GIDD loss to continue pretraining on full sequences under uniform corruption. Its token-level form is $$\mathcal{L}_{\mathrm{GIDD}}(\theta)=\mathbb{E}_{t,x_t}\Bigg[w_t(x_t,x_0)\Bigg(\operatorname{KL}\left(q_t(\cdot\mid x_0) \Vert q_t(\cdot\mid x_\theta)\right)+\frac{q_t(x_t\mid x_0)}{q_t(x_t\mid x_\theta)}-\log\frac{q_t(x_t\mid x_0)}{q_t(x_t\mid x_\theta)}-1\Bigg)\Bigg].$$ Here the expectation is over $t\sim\mathrm{Unif}(0,1)$ and $x_t\sim q_t(\cdot\mid x_0)$, and the loss is summed over sequence positions. Our paper also examines how prediction parameterizations and training objectives support this adaptation. We prove that, under the same forward process and noise schedule, the losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC correspond to a common reverse-rate objective when they induce the same reverse rates, and derive conversions from an $x_0$ prediction to each parameterization. **Reviewer comment.** **A comparison between random and pretrained initialization using the same architecture and training data is missing.** **Response.** Thank you for your suggestion. We have now conducted an initialization ablation at both GPT2-small (124M) and GPT2-medium (355M) scales. At each scale, we keep the model architecture, FineWeb training data, cumulative number of diffusion-training tokens, all other training settings, and evaluation procedure fixed, varying only whether the model starts from randomly initialized weights or pretrained AR weights. The table below reports GenPPL at all five sampling budgets, evaluated using GPT2-large. All values are the mean ± sample standard deviation across five generation seeds, with 1,024 samples per seed. | Model scale | Initialization | 16 steps | 32 steps | 64 steps | 128 steps | 256 steps | |---|---|---:|---:|---:|---:|---:| | GPT2-small | Random | 148.781 ± 0.885 | 118.967 ± 0.943 | 106.583 ± 1.095 | 102.377 ± 0.729 | 101.324 ± 1.247 | | GPT2-small | AR pretrained | 143.944 ± 0.872 | 114.571 ± 0.850 | 103.441 ± 1.394 | 99.199 ± 0.930 | 97.783 ± 1.901 | | GPT2-medium | Random | 140.411 ± 1.929 | 97.773 ± 1.780 | 83.417 ± 1.086 | 77.448 ± 1.313 | 75.481 ± 0.834 | | GPT2-medium | AR pretrained | 128.832 ± 1.968 | 92.007 ± 1.547 | 79.207 ± 1.215 | 74.162 ± 0.901 | 71.516 ± 1.165 | At both model scales and all five sampling budgets, AR pretrained initialization yields lower mean GenPPL. At 256 sampling steps, it reduces GenPPL by approximately 3.5% for GPT2-small and 5.3% for GPT2-medium relative to random initialization. These results show that reusing pretrained AR weights improves the GenPPL of generated text under the matched diffusion-training budgets evaluated here.
Reviewer_eEGbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript seeks to adapt pretrained autoregressive (AR) language models as uniform-noise diffusion language models. It first (section 3.1) establish a connection between different conditional losses as a generalized KL objective over model reverse rates. Different models parametrize the learning problem differently, and the paper provides conversions from clean-token predictions.

Then section 3.2 introduces Unifusion, a method adapting a pretrained AR model to uniform-noise diffusion. It uses the standard (for mask diffusion) output shift and causal-to-bidirectional mask annealing, adapted with GIDD's sampler and loss.

Experiments use Unifusion under different training objective, evaluating each of them with their native samplers. Although all should eventually end up agreeing, GIDD performs best in practice. A similar experiment determines that direct AR-to-uniform is to be used.

Figure 4 shows the generative perplexity vs unigram entropy pareto frontier. Unifusion compares favorably, with a 16-to-256 step path always going in the "right" direction. Zero-shot downstream accuracy is also evaluated, also favorable to Unifusion (at two scales, GPT-S and GPT-M).

No block diffusion nor remasking inference strategies is considered for the baselines.

Critical strengths

To my knowledge, this is the first algorithm converting AR models to uniform-noise diffusion ones. That it does so unifying multiple existing paradigms is a nice touch.

Critical weaknesses

The mask diffusion baselines are not given the opportunity to leverage block diffusion, which has now become quite standard. My Q1 covers this topic.

Despite the authors' claim that "a mask model can update only masked positions, and once revealed, a token is fixed for the remaining denoising steps", this is not true in practice. Indeed, there is now a full literature of training-free and training-based remasking inference methods that improve a MDM's performances. My Q1 also covers this topic.

On reproducibility: there is no mention as to whether the code/weights will be released. This is my Q2.

The text in Figures 3 and 4 (and to a lesser extent Figure 2) is illegible. Figures should be readable on a bad screen when the zoom level is such that the page is 8.5 inches wide.

Questions for authors

Q1. Does Unifusion's uniform strategy still compare well to mask diffusion under standard practices?

Standard practice for mask diffusion is to proceed in a block-wise manner, see "Block diffusion: Interpolating between autoregressive and diffusion language models." (Arriola et al, 2025.)

It is also standard practice to use some remasking strategy, a common one being Fast-dLLM's confidence-aware parallel decoding (Wu et al., 2025).

Q2. Will code/weights be released? If yes, under what license?

This is crucial for reproducibility.

3 comments
AuthorsOpenReview ↗

Response to Reviewer : we thank the reviewer for the careful and constructive comments.

Reviewer comment. Does Unifusion's uniform strategy still compare well to mask diffusion under standard practices? The baselines do not use block-wise generation (Arriola et al., 2025) or Fast-dLLM's confidence-aware parallel decoding (Wu et al., 2025).

Response. Thank you for highlighting block-wise generation and confidence-aware decoding. We evaluated three official BD3-LM checkpoints with their official sampler, generating sequences of 512 tokens and using the same GenPPL–entropy scoring procedure as in our paper. For each model, we used five seeds and generated 1024 samples per seed. The table reports means and standard deviations across seeds.

BD3-LM checkpointParametersGenPPLEntropy
Block size 4169.6M26.363 ± 0.9975.006 ± 0.012
Block size 8169.6M31.959 ± 1.0645.002 ± 0.009
Block size 16169.6M34.581 ± 0.7005.003 ± 0.011

BD3-LM achieves substantially lower GenPPL than the DiffuGPT and Unifusion models in our paper, with lower unigram entropy. It differs from our models in training procedure, training data, and parameter count. During training, it denoises small blocks conditioned on the preceding clean text, reducing the number of tokens to recover at a time. This may help improve prediction. We appreciate this suggestion and plan to explore block-wise training for Unifusion in future work to test whether it improves generation.

Our GenPPL–entropy comparison uses each checkpoint's native sampler and retains its original generation settings. Fast-dLLM's published experiments use LLaDA-8B and Dream-7B, which are larger than our 124M and 355M models. We therefore adapted its confidence-aware decoding to DiffuGPT-S/M to evaluate it at the model scales studied in our paper.

We treated the full 512-token sequence for DiffuGPT-S and the full 1024-token sequence for DiffuGPT-M as a single block, set the confidence threshold to 0.9, and recomputed predictions over the entire sequence at each step. Both models produced highly repetitive text: GenPPL decreased to 1.299 and 1.407, respectively, while entropy fell from 5.279 and 5.647 to 0.966 and 2.034. Decoding used an adaptive number of steps, averaging approximately 465 and 794 forward passes, respectively. Under this configuration, the lower GenPPL was accompanied by a substantial loss of lexical diversity and did not improve the GenPPL–entropy trade-off.

Reviewer comment. Will code/weights be released? If yes, under what license? This is crucial for reproducibility.

Response. Yes. We will organize and publicly release the sampling code and model checkpoints after the paper is accepted. The sampling code and model checkpoints will be released under the MIT License.

Thank you for the reply.

BD3-LM achieves substantially lower GenPPL than the DiffuGPT and Unifusion models in our paper

Indeed.

We treated the full 512-token sequence for DiffuGPT-S and the full 1024-token sequence for DiffuGPT-M as a single block

Why?! I wouldn't dare go above block size 32.

The sampling code and model checkpoints will be released under the MIT License.

This is good, thanks.

I do not have further questions to ask. My overall opinion of the manuscript has not improved, and may have deteriorated.

AuthorsOpenReview ↗

Response to Reviewer : we thank the reviewer for the careful and constructive comments.

Thank you for pointing this out. We used a single full-sequence block to retain DiffuGPT’s original full-sequence generation setup while examining the effect of confidence-aware decoding, recomputing the full bidirectional context at every step without reusing the KV cache. However, this experiment alone does not adequately address your concern about small-block decoding. We had also evaluated a block size of 32, but omitted these more relevant results from our previous response. We apologize for this omission.

Specifically, we adapted Fast-dLLM’s confidence-aware decoding with PrefixCache to the same DiffuGPT-S/M checkpoints, using a block size of 32, a confidence threshold of 0.9, and a temperature of 1.0. The total sequence lengths remained 512 and 1024, respectively. For each model, we used five seeds and generated 1,024 samples per seed. The table below reports means and standard deviations across seeds.

ModelGenPPL ↓Unigram entropy
DiffuGPT-S, block size 322.085 ± 0.0182.424 ± 0.030
DiffuGPT-M, block size 321.730 ± 0.0112.423 ± 0.015

For reference, the corresponding entropies with the original sampler at 256 steps were 5.279 and 5.647. Thus, even with a block size of 32, lower GenPPL was accompanied by a substantial reduction in lexical diversity. Inspection of the generated samples revealed extensive repetition of words and text fragments, resulting in monotonous, low-quality text. In this case, the lower GenPPL did not correspond to better text quality and was instead accompanied by pronounced repetition. This observation applies only to the checkpoints and configuration evaluated here; it does not imply that confidence-aware or block-wise decoding is generally ineffective.

This is also why Section 4.3 evaluates GenPPL together with unigram entropy: lower GenPPL does not necessarily indicate better generation quality, since repetitive, high-probability text can also receive very low GenPPL. Conversely, high entropy can arise from noise or random symbols. Both metrics, together with the generated samples, are therefore needed to assess the balance between fluency(GenPPL) and lexical diversity.

To clarify the specific advantages observed in our comparison, we provide the results for UniFusion and DiffuGPT using their respective native samplers across different sampling budgets. Each entry reports GenPPL / unigram entropy, averaged over five sampling seeds.

Model16 steps32 steps64 steps128 steps256 steps
DiffuGPT-S494.49 / 5.5106271.21 / 5.4427185.12 / 5.3821144.98 / 5.3333117.33 / 5.2792
UnifusionGPT-S143.94 / 5.2461114.57 / 5.2530103.44 / 5.256399.20 / 5.261397.78 / 5.2626
DiffuGPT-M535.98 / 5.9229229.41 / 5.8319147.87 / 5.7685111.60 / 5.709990.65 / 5.6472
UnifusionGPT-M128.83 / 5.638692.01 / 5.651579.21 / 5.660974.16 / 5.665771.52 / 5.6669

UnifusionGPT-M at only 64 steps achieves both lower GenPPL (79.21 vs. 90.65) and higher unigram entropy (5.6609 vs. 5.6472) than DiffuGPT-M at 256 steps, improving both metrics with one-quarter of the sampling steps. At the small scale, UnifusionGPT-S at 64 steps also achieves lower GenPPL (103.44 vs. 117.33), with comparable but slightly lower entropy (5.2563 vs. 5.2792). These results support our finding that, in this comparison using native samplers, UniFusion achieves lower GenPPL with fewer sampling steps while maintaining comparable or higher lexical diversity.

We acknowledge BD3-LM’s better GenPPL results and appreciate your suggestion to include block-wise decoding in the comparison.

Reviewer_YXSBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates a theoretically-grounded way of adapting an autoregressive LLM into a uniform-noise diffusion LLM (DLM). The authors ambitiously do not restrict themselves to a single DLM loss, but first establish a generalization of four commonly used DLM losses.

Critical strengths
  • Adapting an autoregressive model into a uniform-noise diffusion model has not been achieved before, it's undoubtedly a novel contribution of this paper.
  • As far as I can tell (I'm not an expert here), the theoretical contributions -- unifying different DLM losses -- is a clear strength of this paper. The experimental part of this paper is however lacking, as discussed below.
Critical weaknesses
  • I found this paper to be very confusing and hard to follow. In some sense, the paper reads like a direct output from some LLM, it is full of very specific details and different tangents, but without a clear focus. I believe that the paper would be much stronger if the authors either focus on 1) generalization of multiple DLM losses, or 2) adapting AR models for uniform-noise diffusion. Fitting both into a single 9-page paper doesn't work.
  • Appendix C.4 adapts the 4.6B Gemma4 model with UniFussion, getting substantially worse downstream performance than the original autoregressive model (I believe, the paper doesn't report the AR results, but the reported UniFussion scores are at least much worse than Gemma3 scores). I find it very strange that this issue is not discussed in the main text. The main text only evaluates UniFussion on adaptation of small and weak GPT2 models that achieve near-chance performance on many downstream tasks (WinoGrande, SIQA).
Questions for authors

Is your method actually beating the (much more straightforward) adaptation to a masked DLM?

1 comment
AuthorsOpenReview ↗

Response to Reviewer : we thank the reviewer for the careful and constructive comments.

Reviewer comment. The paper is hard to follow. The loss unification and AR-to-uniform adaptation would be clearer as separate contributions.

Response. Thank you for your suggestion. The main text focuses on how to adapt an AR model directly to uniform diffusion: Section 3.1 explains how to use the AR output, Section 3.2 presents the adaptation steps, and Section 4 evaluates training and generation.

The connection between the two parts is that AR models output token probabilities, whereas diffusion generation requires reverse jump rates. Our theory expresses the four methods' losses as the same rate objective and provides conversions from token probabilities to these rates. Using these conversions, the adaptation method retains the AR output head, aligns the prediction positions, and continues training under uniform noise.

Figure 1 tests how to use the AR output: with the same AR initialization and M2S loss, treating the output as an x0x_0 prediction and then converting it yields 48.6% lower average loss over the first 2,000 updates than treating it directly as the M2S posterior prediction. This experiment shows how the theoretical conversion formulas guide the use of pretrained outputs during adaptation.

Reviewer comment. The Gemma adaptation appears much worse than its AR base, but the AR results and a main-text discussion are missing. GPT2 is small and near chance on several tasks.

Response. Thank you for your suggestion. Training of Gemma 4 E2B in Appendix C.4 stopped before convergence after 16,400 updates and approximately 9.67B tokens because of limited compute. Figure 5 shows an overall downward trend in its training loss, with lower values than UnifusionGPT-M and UnifusionGPT-S at the same early training steps. This motivates us to continue training when more compute becomes available and test whether lower training loss leads to better downstream performance. We will explain this training limitation and add the following comparison with the original Gemma AR checkpoint to the appendix.

DatasetGemma 4 E2BUnifusionGemma
HellaSwag72.33%46.4%
WinoGrande62.43%54.1%
SIQA49.03%40.0%
PIQA79.22%62.2%
BBH43.94%35.9%

In Table 2, the scores are generally close. Unifusion has a small advantage over DiffuGPT on WinoGrande, SIQA, and BBH: 2.2, 0.8, and 0.8 percentage points at 124M, and 1.1, 0.7, and 1.4 points at 355M, respectively. The improvement is clearer in the GenPPL–entropy evaluation: at 256 sampling steps, compared with DiffuGPT, the small model reduces GenPPL by 16.7% with slightly lower entropy, while the medium model reduces GenPPL by 21.1% with higher entropy.

Reviewer comment. Is your method actually beating the more straightforward adaptation to a masked DLM?

Response. Thank you for your question. We compare with the publicly released checkpoints of DiffuGPT, a masked DLM. At each model scale, both methods start from the same GPT2 AR weights and are trained on the same 30B-token FineWeb subset.

The table below reports GenPPL / unigram entropy at different sampling steps, averaged over five sampling seeds. Lower GenPPL is better, and unigram entropy measures lexical diversity.

Model16 steps32 steps64 steps128 steps256 steps
DiffuGPT-S494.49 / 5.5106271.21 / 5.4427185.12 / 5.3821144.98 / 5.3333117.33 / 5.2792
UnifusionGPT-S143.94 / 5.2461114.57 / 5.2530103.44 / 5.256399.20 / 5.261397.78 / 5.2626
DiffuGPT-M535.98 / 5.9229229.41 / 5.8319147.87 / 5.7685111.60 / 5.709990.65 / 5.6472
UnifusionGPT-M128.83 / 5.638692.01 / 5.651579.21 / 5.660974.16 / 5.665771.52 / 5.6669

Unifusion achieves lower GenPPL at every tested sampling budget. At 256 steps, the small model has 16.7% lower GenPPL with similar, slightly lower entropy; the medium model has 21.1% lower GenPPL with higher entropy. Moreover, UnifusionGPT-M at just 64 steps achieves lower GenPPL and higher entropy than DiffuGPT-M at 256 steps. These results show that, compared with the publicly released DiffuGPT checkpoints, we can achieve lower GenPPL while maintaining comparable or higher lexical diversity.

Furthermore, Section 4.2 demonstrates that uniform-noise diffusion models can also be obtained by continued training from masked-diffusion checkpoints, suggesting a direction for future work: extending our adaptation approach to stronger pretrained masked diffusion models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript with Revisions Highlighted

Dear Area Chair and Reviewers,

We have uploaded an updated version of our manuscript. For ease of review, all newly added content is highlighted in red. The main additions can be found in the main text and Appendix C.7.

We hope these updates help clarify the manuscript and facilitate the evaluation of our revisions. Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.