- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces the method BYOD which converts instruction-tuned AR LMs into masked diffusion LMs with bidirectional attention mask. The method uses rank-1024 LoRA on Q/V with frozen base weights. It is trained on a mixture of instruction, multiple-choice, math and code data. Each conversion uses $\le$102.4M tokens and takes roughly 4 h on one GPU. It is applied to Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B. The converted models are compared with their frozen AR parents and LLaDA 8B Instruct on nine benchmarks on Phi-4 perplexity and Distinct-n of open-ended generations, and on a right-context copy test. The best model, BYOD-Gemma, averages 50.6% vs 51.9% for LLaDA and 65.7% for its parent.
Critical strengths
- A cheap, uniform, reproducible conversion recipe with candid reporting. One fixed configuration converts four different model families in single-GPU hours. The best model comes within 1.3 points of LLaDA on the paper's harness (Table 3). The limitations are stated openly (Sec. 7).
Critical weaknesses
-
The evaluation cannot support the comparative claims.
- All benchmarks use 125-example subsets (App. D), giving about points of uncertainty per task, but no intervals are reported. Yet the text interprets differences of 0.5-1.3 points (BYOD-Gemma vs LLaDA; 25k vs 50k updates).
- Four AR-parent scores cannot come from 125 examples: HumanEval .740/.890/.940 and Gemma MBPP .740 (Table 3).
- The training mixture includes the MMLU, HellaSwag, GSM8K and MBPP training splits (Sec. 3.3), while the AR parents get no comparable fine-tuning. So the claimed knowledge retention and the "conversion gap" are confounded with task-format exposure.
-
Limited novelty and no demonstrated benefit of the converted models.
- The recipe combines known components; the paper itself credits LAD with first showing that LoRA induces diffusion generation (Sec. 2). Its own choices (rank, Q/V-only targets, label shift, loss weights) are not ablated.
- All benchmarks decode one token per step (Table 8), so parallel decoding is never tested. Where NFE is reduced, quality degrades sharply: BYOD-Gemma perplexity is 14.56 at 32 NFE vs 3.01 for its parent (Table 12).
- With no latency results and large math/code drops (e.g. Qwen MATH-500 .352 vs .784), the contribution rests on conversion cost alone.
Questions for authors
Questions For Authors
- How were the four non- AR scores in Table 3 obtained? Can you report full-benchmark results with confidence intervals?
- What are benchmark accuracies at reduced NFE (e.g. 2-4 tokens per step), and the wall-clock latency vs the AR parents?
- Training sequences are capped at 256 tokens (Table 7), yet GSM8K/MATH/HumanEval decode 256 tokens plus the prompt (Table 8). How much of the math/code gap does this length mismatch explain?
2 comments
I sincerely thank the authors for their clarifications and revision of manuscript. I will raise my score to 2.
Thank you for your careful review. We revised the manuscript and expanded the experiments and data. We address your comments below and refer to our responses to other reviewers where questions overlap to save space.
Weakness 1.1
Please see our response to Reviewer mNeh, Weakness 2 and Question 2, for the expanded evaluation, confidence intervals and revised Gemma–LLaDA comparison. We also withdrew the longer-training accuracy-gain claim: Llama averages 47.80% at 50k updates versus 47.93% at 25k.
Weakness 1.2
Thank you for catching this. We corrected intermediate scores and incorrect counts using verified results (Appendix D).
Weakness 1.3
Although we agree that exposing the models to the training sets of their benchmarks might have improved results, using this data is not actually uncommon. Qwen2.5-Math explicitly uses GSM8K and MATH training examples for SFT (Section 3.1), and d1 trains on GSM8K/MATH training splits and evaluates on held-out questions (Appendix B.1.1). However, most models, like LLaDA for instance, do not release their training data at all.
We agree that additional task exposure confounds a direct causal interpretation of the parent-to-conversion difference. However, our main question is whether an inexpensive conversion works across different models. The parents provide a reference, rather than a target we claim to outperform. Broad benchmark capabilities after only 400,000 training samples are indicative of substantial knowledge retention: learning this much knowledge from such a small additional training set is unlikely.
Weakness 2.1
Please see our response to Reviewer dPsL, Weakness 1 and Question 1, for the LAD comparison, rank ablation and distinction between a shared working recipe and optimal settings. Removing the AR label shift implements same-position reconstruction. We have not separately ablated label alignment or loss weights.
Weakness 2.2
We have indeed not evaluated the full benchmark suite at reduced NFE. Parallel denoising is instead tested in our open-generation experiments: Perplexity- and diversity metrics are an established way to study this trade-off, as in MDLM, SEDD and Time-Agnostic Masked Models. Other studies do report reduced-step benchmark performance, but often only on a few tasks: LLaDA tests only mathematics/coding tasks and Dream uses a one task (Countdown). Showing (full) benchmark performance with reduced NFE is thus not standard practice.
We corrected the perplexity claim: the 2.7–5.0 times AR range applies only at 64 NFE. At 128 NFE, BYOD-Gemma scores 4.33 versus its parent's 3.01. We do not claim faster inference at matched quality.
Weakness 2.3
Please see our response to Reviewer dPsL, Weakness 3, for the mathematics and code results, remaining perplexity gap and conversion-cost comparison. These support useful diffusion capabilities after inexpensive conversion, rather than equivalence to the AR parents or LLaDA. Wall-clock latency remains unmeasured.
Question 1
Thank you for catching this. Intermediate scores remained in the submitted table; we replaced them with verified results. Please see our response to Reviewer mNeh, Weakness 2 and Question 2, for expanded coverage and uncertainty estimates.
Question 2
As discussed in W2.2, we report perplexity- and diversity metrics at 32 and 64 NFE for 128-token outputs, but have not measured benchmark accuracies at those budgets. A reduced-NFE benchmark accuracy sweep remains outside this study's scope.
Wall-clock profiling also remains future work. However, merging folds the LoRA weights into the parent projections and removes the separate adapter branches, so the adapters need not add significant inference overhead compared to other diffusion models.
Question 3
Handling variable-length sequences is part of our training setup: examples have different prompt and answer lengths, and each batch is padded to its longest sequence. However, the reviewer is correct that the limits differ: 256 tokens covers the complete training sequence, whereas evaluation allows up to 256 answer tokens plus the prompt.
How length affects quality thus remains an interesting question. Simply comparing shorter and longer examples would mix length effects with question difficulty. A controlled comparison with a model trained on longer sequences would more directly test this, but is beyond the scope of this study.