acceptodds
Under review as a conference paper at ICLR 2027

BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion

Abstract

Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion. Without architecture-specific tuning, BYOD induces diffusion generation in Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B; to our knowledge, this is the first reported AR-to-DLM conversion of a Mistral model. The adapters comprise 4.1-5.8% of final parameters while every base-model parameter remains frozen; each 25,000-iteration conversion needs at most 102.4M tokens, processed in 3.1-4.0 hours on one Google Colab GPU. All four models generate from fully masked sequences, use bidirectional context in a controlled attention test, and retain substantial parent-model knowledge. BYOD-Gemma is the strongest BYOD conversion on our nine-task evaluation, reaching 50.6% average accuracy compared with 51.9% for LLaDA 8B Instruct in the same harness. Open-generation quality at low network function evaluation (NFE) budgets remains the principal limitation: BYOD perplexity is 8.3-11.1 versus 4.1 for LLaDA and 2.2-3.0 for the AR parents when generating 128 tokens in 64 steps. To enable further experimentation and inspection, we release our training and evaluation code, converted models, and accessible interactive demos upon acceptance. BYOD thus provides a practical route for extending diffusion generation to other existing model families while identifying the gap that efficient conversion must next close.

Primary area:
generative models
Keywords:
diffusion language models, parameter-efficient fine-tuning, LoRA, autoregressive-to-diffusion conversion, masked denoising

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fC22OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces the method BYOD which converts instruction-tuned AR LMs into masked diffusion LMs with bidirectional attention mask. The method uses rank-1024 LoRA on Q/V with frozen base weights. It is trained on a mixture of instruction, multiple-choice, math and code data. Each conversion uses $\le$102.4M tokens and takes roughly 4 h on one GPU. It is applied to Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B. The converted models are compared with their frozen AR parents and LLaDA 8B Instruct on nine benchmarks on Phi-4 perplexity and Distinct-n of open-ended generations, and on a right-context copy test. The best model, BYOD-Gemma, averages 50.6% vs 51.9% for LLaDA and 65.7% for its parent.

Critical strengths
  1. A cheap, uniform, reproducible conversion recipe with candid reporting. One fixed configuration converts four different model families in single-GPU hours. The best model comes within 1.3 points of LLaDA on the paper's harness (Table 3). The limitations are stated openly (Sec. 7).
Critical weaknesses
  1. The evaluation cannot support the comparative claims.

    • All benchmarks use 125-example subsets (App. D), giving about ±9\pm 9 points of uncertainty per task, but no intervals are reported. Yet the text interprets differences of 0.5-1.3 points (BYOD-Gemma vs LLaDA; 25k vs 50k updates).
    • Four AR-parent scores cannot come from 125 examples: HumanEval .740/.890/.940 and Gemma MBPP .740 (Table 3).
    • The training mixture includes the MMLU, HellaSwag, GSM8K and MBPP training splits (Sec. 3.3), while the AR parents get no comparable fine-tuning. So the claimed knowledge retention and the "conversion gap" are confounded with task-format exposure.
  2. Limited novelty and no demonstrated benefit of the converted models.

    • The recipe combines known components; the paper itself credits LAD with first showing that LoRA induces diffusion generation (Sec. 2). Its own choices (rank, Q/V-only targets, label shift, loss weights) are not ablated.
    • All benchmarks decode one token per step (Table 8), so parallel decoding is never tested. Where NFE is reduced, quality degrades sharply: BYOD-Gemma perplexity is 14.56 at 32 NFE vs 3.01 for its parent (Table 12).
    • With no latency results and large math/code drops (e.g. Qwen MATH-500 .352 vs .784), the contribution rests on conversion cost alone.
Questions for authors

Questions For Authors

  1. How were the four non-k/125k/125 AR scores in Table 3 obtained? Can you report full-benchmark results with confidence intervals?
  2. What are benchmark accuracies at reduced NFE (e.g. 2-4 tokens per step), and the wall-clock latency vs the AR parents?
  3. Training sequences are capped at 256 tokens (Table 7), yet GSM8K/MATH/HumanEval decode 256 tokens plus the prompt (Table 8). How much of the math/code gap does this length mismatch explain?
2 comments

Thank you for your careful review. We revised the manuscript and expanded the experiments and data. We address your comments below and refer to our responses to other reviewers where questions overlap to save space.

Weakness 1.1

125-example subsets ...

Please see our response to Reviewer mNeh, Weakness 2 and Question 2, for the expanded evaluation, confidence intervals and revised Gemma–LLaDA comparison. We also withdrew the longer-training accuracy-gain claim: Llama averages 47.80% at 50k updates versus 47.93% at 25k.

Weakness 1.2

Four AR-parent scores ...

Thank you for catching this. We corrected intermediate scores and incorrect counts using verified results (Appendix D).

Weakness 1.3

The training mixture includes ...

Although we agree that exposing the models to the training sets of their benchmarks might have improved results, using this data is not actually uncommon. Qwen2.5-Math explicitly uses GSM8K and MATH training examples for SFT (Section 3.1), and d1 trains on GSM8K/MATH training splits and evaluates on held-out questions (Appendix B.1.1). However, most models, like LLaDA for instance, do not release their training data at all.

We agree that additional task exposure confounds a direct causal interpretation of the parent-to-conversion difference. However, our main question is whether an inexpensive conversion works across different models. The parents provide a reference, rather than a target we claim to outperform. Broad benchmark capabilities after only 400,000 training samples are indicative of substantial knowledge retention: learning this much knowledge from such a small additional training set is unlikely.

Weakness 2.1

The recipe combines ...

Please see our response to Reviewer dPsL, Weakness 1 and Question 1, for the LAD comparison, rank ablation and distinction between a shared working recipe and optimal settings. Removing the AR label shift implements same-position reconstruction. We have not separately ablated label alignment or loss weights.

Weakness 2.2

One token per step ...

We have indeed not evaluated the full benchmark suite at reduced NFE. Parallel denoising is instead tested in our open-generation experiments: Perplexity- and diversity metrics are an established way to study this trade-off, as in MDLM, SEDD and Time-Agnostic Masked Models. Other studies do report reduced-step benchmark performance, but often only on a few tasks: LLaDA tests only mathematics/coding tasks and Dream uses a one task (Countdown). Showing (full) benchmark performance with reduced NFE is thus not standard practice.

We corrected the perplexity claim: the 2.7–5.0 times AR range applies only at 64 NFE. At 128 NFE, BYOD-Gemma scores 4.33 versus its parent's 3.01. We do not claim faster inference at matched quality.

Weakness 2.3

No latency results ...

Please see our response to Reviewer dPsL, Weakness 3, for the mathematics and code results, remaining perplexity gap and conversion-cost comparison. These support useful diffusion capabilities after inexpensive conversion, rather than equivalence to the AR parents or LLaDA. Wall-clock latency remains unmeasured.

Question 1

How were ... scores ... obtained ...

Thank you for catching this. Intermediate scores remained in the submitted table; we replaced them with verified results. Please see our response to Reviewer mNeh, Weakness 2 and Question 2, for expanded coverage and uncertainty estimates.

Question 2

What are benchmark accuracies at reduced NFE ...

As discussed in W2.2, we report perplexity- and diversity metrics at 32 and 64 NFE for 128-token outputs, but have not measured benchmark accuracies at those budgets. A reduced-NFE benchmark accuracy sweep remains outside this study's scope.

Wall-clock profiling also remains future work. However, merging folds the LoRA weights into the parent projections and removes the separate adapter branches, so the adapters need not add significant inference overhead compared to other diffusion models.

Question 3

Training sequences are capped ...

Handling variable-length sequences is part of our training setup: examples have different prompt and answer lengths, and each batch is padded to its longest sequence. However, the reviewer is correct that the limits differ: 256 tokens covers the complete training sequence, whereas evaluation allows up to 256 answer tokens plus the prompt.

How length affects quality thus remains an interesting question. Simply comparing shorter and longer examples would mix length effects with question difficulty. A controlled comparison with a model trained on longer sequences would more directly test this, but is beyond the scope of this study.

I sincerely thank the authors for their clarifications and revision of manuscript. I will raise my score to 2.

Reviewer_dPsLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BYOD converts four pretrained autoregressive model families into masked diffusion models using attention-only LoRA and a shared training recipe. Conversion takes a few single-GPU hours, although generation quality remains below the AR parents and often below LLaDA.

Critical strengths
  • The conversion recipe is simple and accessible to groups with limited training hardware.
  • Testing four model families with their AR parents gives useful practical evidence.
  • The paper clearly reports quality failures, low-NFE limitations, and the difference between training cost and inference speed.
Critical weaknesses
  • The recipe is not compared with a matched-budget LoRA conversion baseline such as LAD. Without rank and trainable-module ablations, the benefit of the specific BYOD choices is unclear.
  • Each task uses only 125 fixed examples, without uncertainty estimates. The 1.3-point mean gap to LLaDA is therefore hard to interpret, and MMLU-Pro gives AR 2,048 output tokens but diffusion only 128.
  • Generation quality remains a large limitation. Math and code lose substantial accuracy versus the AR parents, and even 128-step generation has worse perplexity than LLaDA.
Questions for authors
  • How does BYOD compare with LAD or a simple LoRA conversion control at the same token and compute budget, and how sensitive is it to rank and target modules?
  • Do the main comparisons hold on full benchmarks or with paired confidence intervals, using matched output limits for MMLU-Pro?
  • Can you show an infilling or constrained-generation task where the conversion provides a useful capability at an acceptable quality and latency?
1 comment

Thank you for your careful review. We expanded evaluation and added intervals and a rank study.

Weakness 1

The recipe is not compared ...

Response:

BYOD builds on LAD's LoRA and bidirectional attention. LAD uses masking plus structured corruption within Llama; BYOD tests mask-only MDLM across Gemma, Llama, Qwen and Ministral. Although it was not feasible to run a LAD control with matched data and budget, we have added Appendix B (Table 5), which compares our results with the published LAD 8B results.

Furthermore, we added rank ablation testing in Appendix C (Figure 5, Table 6). We test ranks 128, 512, 1024 and 2048 on Llama 3.1 8B under fixed settings. Rank 1024 was originally chosen as it allowed batch size 16 on an 80GB A100 or 8 on a 40GB A100. Rank 2048 still lowers validation loss but doubles adapter parameters and takes longer per update. Also, its loss gain per added rank unit is smaller. We did perform earlier, experimental module/learning-rate tests to inform our Q/V and rank choices, but these were not systematic, instead prioritising cross-family testing.

Weakness 2

Each task uses only 125 fixed examples ...

Response:

We now use 250 shared questions on eight tasks and all 164 HumanEval questions across ten model configurations. We also rescored some results after we found some discrepancies; Appendix D documents these changes. We also included confidence intervals, with an updated Table 3 showing averages and Appendix D gives per-task scores and intervals.

Although we agree that reporting confidence intervals strengthens results, we'd like to note that adding confidence intervals is not often done in these studies, which why it was originally omitted; we inspected ten prominent diffusion papers which all also omit confidence intervals: LLaDA, Dream, DiffuLLaMA, Scaling up Masked Diffusion Models, MDLM, SEDD, Time-Agnostic Masked Models, DiffuCoder, LLaDA2.0 and Mercury.

After expanding the benchmarks BYOD-Gemma averages 52.43% versus LLaDA's 49.55%. However, this does not prove superiority, as the difference is not significant after compensating for multiple testing. Moreover, Gemma is larger and has a different training history.

Finally, the reason for the larger AR budget for MMLU-Pro: On MMLU-Pro, AR models often reasoned before answering and needed longer budgets. The 2,048-token limit is a maximum; answers often ended earlier. Diffusion answers were usually short and direct, so a shorter canvas reduced cost. We found none cut short by the budget.

Weakness 3

Generation quality remains a large limitation ...

Response:

We agree that the BYOD models underperform in math/code tasks, but this is a feature of many diffusion models, not just BYOD. BYOD-Gemma scores similar to LLaDA on these tasks, with 24.4/26.0% on MATH-500, 43.6/42.8% on MBPP, 68.8/59.6% on GSM8K and 29.9/34.8% on HumanEval (Table 3). These mixed results support useful capabilities approaching a pretrained DLM, not equivalence or superiority to AR parents.

At 128 NFE, BYOD perplexity is 4.33–6.21 versus LLaDA's 2.79, showing that perplexity is indeed the largest gap between LlaDa and the BYOD models, and we explicitly cite this as the gap we still would like to close. We therefore propose in our manuscript that our recipe can be used as a starting point for for full diffusion fine-tuning from merged models to improve quality while preserving capabilities.

Question 1

How does BYOD compare with LAD ...

Response:

Please see our response at Weakness 1.

Question 2

Do the main comparisons hold ...

Response:

Please see our response at Weakness 2.

Question 3

Can you show an infilling ...

Response:

We agree that infilling and constrained generation are interesting research directions. However, our study focuses text generation capabilities of diffusion models, and more specifically on whether a low-cost LoRA recipe can convert AR models into functional diffusion models, and which capabilities they retain after conversion.

We have therefore not expanded the study to evaluate a realistic infilling or constrained-generation benchmark. We did already include our right-context copy test (Figure 2 and Appendix J) which confirms access to later context, but is a diagnostic rather than evidence of practical infilling. A dedicated evaluation of task quality, constraint satisfaction and wall-clock latency against suitable baselines would be a valuable extension, but is outside the present study's scope.

Reviewer_mNehOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BYOD, which is a LoRA-based method converting frozen autoregressive models into diffusion language models. The method is applied to four model families, which requires 3.1 to 4 hours on a single GPU. The authors evaluated the performance, generation quality, and bidirectional attention. For the main result, BYOD-Gemma reaches 50.6% average accuracy compared to 51.9% for LLaDA, although the performance and the quality remains below the autoregressive baselines.

Critical strengths
  • The one strength of BYOD that I want to point out is that the conversion provides a practical way to convert an autoregressive model to a diffusion language model with a limited compute. Furthermore, the evaluation on four different model families adds generalizability to the method.
  • The comparison to the original autoregressive model was meaningful for understanding what features are preserved after the conversion and where the performance drops. This is especially important for future research since we can have a better view of what needs to be improved.
Critical weaknesses

The most critical weakness that I found is the lack of experimental support on the existing LoRA-based conversion methods. LAD already claims a conversion using LoRA and bidirectional attention, while the paper lacks the comparison against it. The ablation on different hyperparameters that are related to LoRA is lacking and also, it is not clear whether the additional parameter provides a meaningful tradeoff between the conversion cost and performance.

Furthermore, the benchmark evaluation uses only 125 examples per task and does not report uncertainty estimates. This makes it difficult to judge how reliable the small average gap against LLaDA is. Since competitive performance at a low conversion cost is a major reason to use the method, I would want stronger evidence for this comparison.

Reference: Kuiper, R. J. A., de Groot, L., van Es, B., van Smeden, M., & Bagheri, A. (2025). LAD: LoRA-Adapted Diffusion. Proceedings of EMNLP 2025: System Demonstrations, pp. 97–110.

Questions for authors
  • Can the authors compare BYOD against LAD using the same training data and budget? How much does performance change with a smaller LoRA rank?
  • Does the reported proximity to LLaDA hold on larger benchmark subsets, with confidence intervals for the paired differences?
1 comment

Thank you for your careful review and constructive feedback. We have revised the manuscript, expanded the benchmark evaluation, added uncertainty estimates and a LoRA-rank ablation, and clarified our comparisons and claims. We address your comments individually below.

Weakness 1

The most critical weakness that I found ...

Response:

We understand this concern. However, LAD has a significantly different generation objective; it combines masking and structured corruption, resulting in a different inference paradigm. BYOD instead uses mask-only MDLM, and focusses on whether a reproducible recipe works across model families, whereas LAD tested on only one model family.

We do agree that a more in-depth comparison is granted, and have added LAD's published 8B benchmark results alongside BYOD-Llama in Appendix B (Table 5). However, these were not obtained in the same harness and thus not a controlled comparison.

To address the hyperparameter question, we have added a rank ablation on Llama 3.1 8B in Appendix C (Figure 5 and Table 6), testing rank 128, 512, 1024 and 2048. Rank 1024 was originally chosen to balance validation quality with single-GPU training, which was a major target of this study: it allowed batch size 16 on an 80GB A100 or batch size 8 on a 40GB A100. Rank 2048 lowers validation loss further, but doubles the adapter parameters and takes longer per update. Its loss reduction per added rank unit is smaller: going from 1024 to 2048 adds twice as many rank units as going from 512 to 1024. Earlier exploratory tests of other attention projections, MLP targets and learning rates informed the recipe; a full sweep of these interacting choices was outside this study's scope.

Weakness 2

Furthermore, the benchmark evaluation uses only 125 examples per task ...

Response:

We appreciate this point and agree that the comparison needs stronger evidence. We have now doubled the target from 125 to 250 questions on eight tasks and all 164 HumanEval questions across all ten model configurations. We also revised some benchmarks sligthly due to some discrepancies in scoring, which are now solved. These changes are documented in Appendix D.

We also appreciate the need for confidence intervals, which we have now included in our results. For context, the main comparisons we inspected in ten prominent diffusion language model papers present point estimates without accompanying confidence intervals, and often omit per-task sample counts beside the scores: LLaDA, Dream, DiffuLLaMA, Scaling up Masked Diffusion Models, MDLM, SEDD, Time-Agnostic Masked Models, DiffuCoder, LLaDA2.0 and Mercury.

We nevertheless agree that providing both counts and uncertainty estimates strengthens our evaluation and have therefore included confidence intervals for all benchmark results. Table 3 gives the updated scores using the 250 samples, and 95% confidence intervals for the nine-task averages. Appendix D gives the full per-task table with 95% confidence intervals, the actual sample counts and the calculation details.

BYOD-Gemma now averages 52.43%, compared with 49.55% for LLaDA. This does not prove that the BYOD-Gemma architecture itself is superior, as the difference is still not signicant when accounting for multiple testing. Moreover, BYOD-Gemma's 9B parent is larger than 8B LLaDA, and their training histories differ. The comparison rather shows that LoRA conversion can produce useful diffusion capabilities at low training cost.

Question 1

Can the authors compare BYOD against LAD ...

Response:

A LAD control using the same data and training budget would most directly test our mask-only formulation, but was outside of the scope of this study. However, Appendix B (Table 5) now compares published LAD results with BYOD under different protocols. Appendix C examines rank under otherwise fixed settings. Smaller ranks give higher validation loss, while rank 2048 improves it further at greater training cost. Rank 1024 was chosen as a practical balance within a single-A100 training budget; the loss reduction per added rank unit decreases at higher ranks.

Question 2

Does the reported proximity to LLaDA hold ...

Response:

Yes. As described in W2, we now use 250 questions per task. BYOD-Gemma averages 52.43%, versus 49.55% for LLaDA, with a difference of +2.88 percentage points and a nominal 95% interval of [+0.74, +4.98]. This interval is unadjusted for multiple comparisons. We make no superiority claim; the models also differ in size and training history. Table 3 and Appendix D give the updated results and uncertainty estimates.

Comments on the paper

1 comment

Final summary of revisions

We would like to thank all three reviewers for their constructive feedback and engagement during the discussion period.

In response to the reviews, we have substantially revised the manuscript and addressed several of the main concerns:

  • Expanded evaluation: Doubled the benchmark sample size to 250 per task (164 for HumanEval), corrected scoring discrepancies, and added 95% confidence intervals throughout.
  • Additional experiments: Added a LoRA-rank ablation (128–2048) and a comparison with published LAD results, while acknowledging the limitations of an unmatched comparison.
  • Clarified contributions: Better distinguished BYOD from LAD, emphasizing our different training objectives and inference approaches, and our main contribution: a reproducible, low-cost conversion recipe demonstrated across four model families. We also prioritized a direct comparison with LLaDA, a more closely matched and well-established diffusion model.
  • More conservative claims: Explicitly acknowledged limitations concerning training-data exposure, generation quality, and comparisons with autoregressive models and LLaDA.

We appreciate the reviewers who have already revisited our manuscript and provided further feedback. We would also welcome any remaining feedback from the other reviewer before the discussion period closes. We hope that these substantial revisions and additional experimental evidence will be considered in the final assessments.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.