acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning Discrete Diffusion LLMs by Distilling from Autoregressive Teachers

Abstract

Discrete diffusion language models (dLLMs) have recently emerged as an alternative paradigm for text generation, with the potential to achieve more efficient inference than autoregressive (AR) models. However, AR models still continue to achieve stronger generation quality than dLLMs. The AR models provide strong token-level predictive distributions and benefit from more established training methods. We explore how to leverage AR models during training while preserving the parallel generation capability of dLLMs at inference time. However, directly distilling AR knowledge into dLLMs is challenging because AR models and dLLMs condition on different contexts, leading to mismatched predictive distributions. To address this challenge, we propose Weight-annEaled Autoregressive distillatioN for dLLMs (WEAN), the first AR-to-dLLM distillation method designed for downstream fine-tuning. WEAN controls where and when AR supervision is applied: it selectively transfers AR knowledge to reliable token positions and dynamically adjusts the distillation strength according to denoising steps and training stages. Specifically, WEAN focuses distillation on structural tokens, where AR and dLLM predictions are more compatible, and gradually reduces AR guidance as the dLLM adapts to the target distribution. Experimental results show that WEAN consistently improves both generation quality and downstream utility compared with the strongest baselines by 17.0% and 2.03% on studied text datasets. WEAN generalizes to code and mathematics domains, showing its broad applicability. We release the replication package on the anonymous link.

Primary area:
generative models
Keywords:
Discrete Diffusion LLMs, Model Distillation, Autoregressive Model, Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pJKuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WEAN uses an autoregressive teacher to improve downstream fine-tuning of pretrained discrete diffusion language models. It distills structural tokens, weights supervision by the noise level, and removes teacher guidance later in training. Cross-tokenizer alignment and cached teacher distributions enable transfer across model families. Experiments on two dLLM backbones and six datasets evaluate generation quality and the downstream utility of synthetic data.

Critical strengths
  1. The method addresses a practical mismatch between AR and diffusion conditioning. Selective, temporary teacher supervision offers a sensible way to use a stronger teacher while allowing the student to adapt to task data. The trained model retains its original inference procedure.

  2. Improvements are consistent across backbones and datasets. In Table 3, structural-token distillation outperforms semantic-only and all-token distillation on all ten metrics, directly supporting the token-selection design. The code and mathematics experiments also show gains in synthetic-data utility.

Critical weaknesses
  1. The explanation for the advantage of structural tokens remains indirect. Figure 2 compares AR and dLLM predictions only on structural tokens; its other panels measure changes within the dLLM after SFT. Neither directly compares AR–dLLM compatibility across structural and semantic tokens. Table 3 supports the selection rule, but does not establish that reduced conditional mismatch explains its gains.

  2. The training-schedule experiment confounds timing with cumulative distillation weight. The best setting in Figure 3, ρ = 0.2, uses distillation for the first 20% of training and then switches it off. Changing ρ changes both when supervision is applied and its total weight, leaving unclear whether early guidance or simply weaker overall distillation drives the improvement.

Minor comment: On p. 6, remove the leading minus sign in the KD loss to match Algorithm 1 and the implementation.

Questions for authors
  1. Under matched samples and masking conditions, how do AR–dLLM KL divergence and probability overlap compare between structural and semantic tokens?

  2. With the same teacher and noise weighting, does early distillation outperform weaker supervision throughout training when cumulative distillation weights are matched?

4 comments

A1 (W1 & Q1): Direct AR–dLLM compatibility on structural vs. semantic tokens. Thank you for this suggestion. We agree that Figure 2 does not directly compare the two token types, and we provide this comparison below.

(a) Direct comparison (Q1). Using the same held-out samples, masking patterns, and projection settings (M = 128, K = 32), we compare the task-adapted Qwen3-32B teacher with the vanilla-SFT LLaDA-8B-Instruct student on structural and semantic positions. Both models are fixed, and the student has not undergone WEAN distillation, so the comparison reflects the intrinsic compatibility of the two models rather than an effect of our training.

Token typeForward KL ↓ (temperature 4)Probability overlap ↑ (temperature 1)
Structural6.300.310
Semantic6.580.220

Structural tokens show higher probability overlap (0.310 vs. 0.220, a 41% relative increase) and lower forward KL divergence (6.30 vs. 6.58) than semantic tokens, directly confirming that AR and dLLM predictions are more compatible at structural positions.

(b) Link to the selection gains. The downstream results follow the same ordering. In Table 3, structural-token distillation outperforms all-token and semantic-only distillation on all ten metrics (e.g., Yelp Category 73.8% vs. 70.7% and 64.9%), and semantic-only distillation, which supervises the least compatible positions, performs worst. Moreover, replacing the teacher's soft targets at structural positions with one-hot labels under the same weighting removes 54–70% of WEAN's gain over SFT (see our response to Reviewer A), which rules out the alternative explanation that structural selection helps merely by upweighting these positions. Together, these results establish a consistent association between AR–dLLM compatibility and distillation gains, although not a complete causal decomposition.

This selectivity reflects the goal of WEAN. The AR model serves as an auxiliary teacher during fine-tuning, not as a target for the dLLM to replicate or surpass. WEAN therefore transfers AR knowledge only where it is compatible with the dLLM's conditioning, while the final model remains a dLLM that retains its parallel generation and faster inference.

A2 (W2 & Q2): Timing at matched cumulative weight. Thank you for pointing out this confound. With ρ = 0.2, Eq. (2) keeps λ_s(s) = 1 for the first 20% of training and then sets it to zero. The ρ sweep in Figure 3 changes both when supervision is applied and its total weight; here we isolate the effect of timing. On OpenReview, using the same task-fine-tuned Qwen3-32B teacher, LLaDA-8B-Instruct student, and noise weighting λ_t(t) = α(1 − t), we compare four progress-weight profiles with equal integrated weight and matched cumulative KD weight.

Training scheduleProgress-weight profile λ_sArea↑Rating↑MAUVE↑
Early abrupt cutoff (WEAN, ρ = 0.2)1 during the first 20%; 0 afterward48.441.90.531
Early gradual decayLinear decay from 2 to 0 during the first 20%; 0 afterward47.941.60.523
Weaker full-run guidance0.2 throughout training46.740.60.463
Late guidance1 during the final 20%; 0 beforehand46.040.60.454

Early guidance outperforms both weaker full-run guidance (+1.7 Area) and late guidance (+2.4 Area). Since all profiles share the same cumulative weight, the improvement stems from applying guidance early rather than from weaker overall distillation (Q2). Given an early window, the decay shape has only a minor effect: the abrupt cutoff performs comparably to, and slightly better than, gradual decay (+0.5 Area). Accordingly, the training-progress annealing in WEAN refers to reducing the teacher weight to zero early in training, which, under ρ = 0.2, takes the form of an abrupt cutoff.

A3 (Minor comment): KL sign. Thank you for catching this. The implemented objective adds a positive forward KL to the denoising loss, as in Algorithm 1; the minus sign in Sec. 3.3 is a typographical error. In addition, L_KD is averaged rather than summed over the masked structural positions (see our response A3 to Reviewer B). Neither correction changes the implemented objective or requires rerunning experiments.

Dear Reviewer pJKu,

Thank you again for your time and constructive feedback on our submission.

As the rebuttal period is coming to a close today (November 18, AoE), we would like to gently follow up to see if our responses and additional experiments have addressed your concerns or if there are any remaining questions we can clarify before the deadline.

We truly appreciate your time and effort in evaluating our work.

I thank the authors for the detailed response and additional experiments, which address most of my concerns. Based on my overall assessment of the paper's contributions, I will maintain my original score.

Dear Reviewer pJKu,

Thank you for your thoughtful review and for engaging with our response. We are glad that we addressed most of your concerns, and we appreciate your positive assessment of our work.

Reviewer_BcM1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WEAN studies how an autoregressive teacher can assist downstream fine-tuning of an already pretrained diffusion language model. It first fine-tunes the AR teacher on task data and caches projected teacher distributions. Distillation is restricted to masked structural-token positions, weighted more strongly at lower masking levels, and removed according to a training-progress schedule. To support different tokenizers, the method maps AR vocabulary entries that re-tokenize into a single student token, aggregates their probabilities, and aligns response positions using character offsets. Experiments use LLaDA-8B-Instruct and Dream-7B students with a Qwen3-32B teacher. Evaluation focuses on synthetic-data distributional similarity and the utility of generated data for downstream text, code, and mathematics tasks.

Critical strengths

S1. The paper addresses a relevant conditional-distribution mismatch. An AR teacher and a masked diffusion student do not generally observe equivalent contexts. Treating AR predictions as selective, temporary auxiliary supervision rather than a persistent imitation target is a reasonable design. The method preserves the student’s inference architecture and avoids concurrent teacher execution during student fine-tuning. S2. The reported evaluation spans multiple domains and includes downstream utility. The experiments cover two diffusion backbones and text, code, and mathematics datasets. Measuring downstream models trained on generated samples adds a useful perspective beyond text-distribution metrics alone. Structural-only distillation also outperforms semantic-only and all-token variants in the reported ablation, providing preliminary support for selective supervision.

Critical weaknesses

W1. The cross-tokenizer distillation target is not sufficiently validated or fully specified. Vocabulary projection discards candidates that do not map to exactly one student token and renormalizes the remaining mass. Position alignment is described through character offsets, but the rule for mismatched token boundaries is not specified. If a student position begins inside an AR token, there may be no equivalent teacher next-token prediction at that boundary. The paper does not quantify aligned-position coverage, retained probability mass, or sensitivity to candidate/cache sizes. Consequently, it remains unclear how faithfully the cached targets represent the intended teacher supervision, particularly across tokenizer families and domains. W2. The experiments do not adequately isolate the proposed scheduling mechanisms or establish the reliability of the reported gains. The main baselines do not receive the same task-adapted AR teacher. The pretrained-AR comparison therefore changes teacher adaptation as well as the supervision scheme. Structural-token ablations are useful, but there is no clean factorial comparison separating task-adapted teacher supervision, token selection, mask-dependent weighting, and training-progress annealing. Results also lack repeated-seed uncertainty despite involving generator fine-tuning, synthetic sampling, and downstream-model training. This matters for interpreting small utility gains and the substantial degradation of some baselines.

Questions for authors

Q1. What exact position-alignment rule is used when AR and student token boundaries do not coincide? Please report the fraction of structural positions receiving supervision and the teacher probability mass retained before renormalization for each teacher–student pair and domain. Q2. Can you provide controlled ablations using the same task-fine-tuned AR teacher throughout: constant-weight distillation, mask-dependent weighting only, training-progress annealing only, and their combination? Token-selection comparisons should also control the effective distillation strength or supervised-token count. Q3. Please reconcile the negative KL definition in Section 3.3 with the positive KL in Algorithm 1. Which objective is implemented? Please also provide the actual MM, KK, optimizer settings, training seeds, decoding configuration, and POS-tagging/token-span rules. Q4. How stable are the results across generator, synthetic-data, and downstream-model seeds? In particular, what explains the Dream CodeNet accuracy drops to 10.0 and 19.3 for GIFT and LIFT, compared with 93.9 for ordinary SFT? Q5. Since synthetic-data utility is the central evaluation, how does WEAN compare with generating the same amount of training data directly from the task-adapted AR teacher? Please report both utility and total preparation/generation cost.

6 comments

A1 (W1 & Q1): Cross-tokenizer alignment and target fidelity. Thank you for these detailed questions. Below we specify the alignment rule and quantify coverage, retained probability mass, and sensitivity to M and K.

(a) Position-alignment rule (Q1). For each student token, we locate the AR token whose character span contains the student token's starting character, and use the AR next-token distribution immediately preceding that AR token; this defines π(i) in Eq. (5). (i) If both tokens start at the same character (exact-start match), the teacher and student predictions refer to the same boundary. (ii) If the student token starts inside an AR token, we reuse the prediction at the start of the containing AR token, which is an approximation. (iii) If the starting character falls between AR token spans, we use the preceding AR token. Positions without an AR counterpart receive no target and are excluded from L_KD. Exact-start matches account for 94.9–99.9% of the audited structural positions, so the approximations in (ii) and (iii) affect at most 5.1% of supervised positions.

(b) Coverage and retained mass per domain (Q1). We audit 500 randomly sampled training responses per benchmark, pairing the Qwen3-32B teacher with the LLaDA-8B-Instruct tokenizer (the cross-tokenizer case), with M = 128 and K = 32. Coverage is the fraction of all trainable structural positions that receive a non-empty target. Retained mass is the original teacher probability of mapped candidates before renormalization, averaged over structural positions with usable AR predictions and counting empty projections as zero.

BenchmarkAudited positionsCoverage (%)Retained teacher mass (%)
Yelp24,68496.596.7
OpenReview92,20599.195.0
PubMed53,64898.798.0
CodeNet37,381100.071.6
PrimeVul59,802100.067.4
DM-Math9,216100.098.2

Coverage is 96.5–100.0% across all domains. Retained mass is 95.0–98.2% on text and mathematics but 67.4–71.6% on code. Since enlarging M recovers at most 1.8 points of mass (see (c)), the loss on code stems mainly from AR candidates without a single-token counterpart in the LLaDA vocabulary rather than from top-M truncation. Renormalization preserves the relative probabilities among the retained candidates, so the target is the teacher distribution restricted to tokens that the student can represent as a single prediction. Even with this lower retention, WEAN achieves the best results on both code benchmarks with both backbones (Table 4). For efficiency, this audit uses the pretrained Qwen3-32B in 8-bit precision.

(c) Sensitivity to M and K. Against a reference target constructed with the same projection rule over the full AR vocabulary and without top-K truncation, the mean total-variation distance of the cached target decreases from 0.024 at K = 32 to 0.014 at K = 64 (temperature 1), and increasing M from 128 to 1,024 recovers at most 1.8 points of probability mass. The default M = 128 and K = 32 therefore closely approximate the full projected target, with little additional error from truncation.

A2 (W2 & Q2): Controlled ablations with the same teacher. Thank you for this comment. All ablations below use LLaDA-8B-Instruct and the same task-fine-tuned Qwen3-32B teacher, and vary one factor at a time while holding the others at WEAN's setting. Together, they separate teacher supervision, mask-dependent weighting, training-progress annealing, and token selection. Seed variability is addressed in A4.

(a) Teacher supervision vs. weighting. SFT-DLLM-Struc is identical to WEAN except that the cached teacher distribution at each masked structural position is replaced with the one-hot ground-truth token. It uses the same positions, α, and schedules, so structural positions receive the same total weight in both methods, which differ only in the target distribution.

MethodYelp Cate.↑Yelp Rating↑Yelp C2ST-S.↑Yelp MAUVE↑OR Area↑OR Rating↑OR C2ST-S.↑OR MAUVE↑
SFT-DLLM71.867.90.1900.12045.640.20.2560.342
SFT-DLLM-Struc72.568.90.2120.16946.540.80.3100.418
WEAN73.870.10.2380.28448.441.90.3880.531

Teacher soft targets account for 54–70% of WEAN's gain over SFT-DLLM across all eight metrics, beyond the effect of upweighting structural positions.

(b) Mask-dependent weighting and training-progress annealing (Q2). On OpenReview, a disabled factor (✗) is replaced by a constant, and the coefficients are rescaled so that the cumulative KD weight is matched across variants.

Mask-dependent weighting λ_t(t)Training-progress annealing λ_s(s)Area↑Rating↑MAUVE↑
✗✗44.839.80.405
✓✗46.740.60.463
✗✓47.141.00.490
✓✓48.441.90.531

Each factor improves Area accuracy over constant weighting (+1.9 and +2.3 points), and their combination performs best (+3.6 points; +1.7/+1.3 over either alone). Under the same total KD weight, concentrating guidance early is also more effective than spreading it over training or applying it late (Area 48.4 vs. 46.7 and 46.0), while a gradual decay within the early window performs similarly (47.9); see A1(c) to Reviewer A.

(c) Token selection (Q2). In our implementation, L_KD is averaged over the participating masked positions; the summation in Sec. 3.3 and Algorithm 1 is a notational error (see A3). All selection variants use identical λ(s, t), α, temperature, and schedules. The total KD weight per step is therefore matched across variants, so the effective distillation strength is controlled, while the number of supervised tokens differs by design. Under this control, structural selection outperforms semantic-only and all-token selection on all ten metrics (Table 3).

(d) Baselines without a teacher. GIFT, LIFT, and NaRA are existing fine-tuning methods without distillation and are therefore compared as proposed. The teacher-controlled comparisons in (a)–(c) isolate the contribution of each WEAN component under the same task-fine-tuned teacher.

A3 (Q3): Implemented objective and settings. Thank you for catching these inconsistencies.

(a) Implemented objective. We minimize L_SFT + λ(s, t)·L_KD with a positive forward KL, consistent with Algorithm 1. The minus sign in Sec. 3.3 is a typographical error, and L_KD is averaged over the masked structural positions in each batch rather than summed. With temperature τ = 4, the cached teacher log-probabilities are divided by τ and renormalized over their top-K support, the student distribution is softmax(logits/τ) over the full vocabulary, and the KL is scaled by τ² (Hinton et al., 2015):

LKD=τ2N∑i∈I∑k=1Kqiτ(k)[log⁡qiτ(k)−log⁡pθτ(k∣xt,t)],\mathcal{L}_{\mathrm{KD}} = \frac{\tau^2}{N} \sum_{i \in \mathcal{I}} \sum_{k=1}^{K} q_i^{\tau}(k) \left[ \log q_i^{\tau}(k) - \log p_\theta^{\tau}(k \mid x_t, t) \right],

where I is the set of masked structural positions in the batch and N = |I|. The weight is λ(s, t) = α(1 − t)·λ_s(s) with α = 0.3. L_SFT is the standard masked-diffusion objective: t ~ U[10⁻³, 1), each response token is masked with probability t, and the cross-entropy at masked positions is weighted by 1/t and normalized by the number of response tokens. Prompt tokens are never masked.

(b) Settings.

ComponentSetting
StudentLLaDA-8B-Instruct with LoRA (r = 32, scaling 64, dropout 0.05, all linear layers), bf16
OptimizerAdamW, learning rate 2×10⁻⁵, cosine schedule with 10% warmup, gradient clipping at 1.0, no weight decay
Batch and length4 per GPU × 256 accumulation steps × 4 GPUs = 4,096 sequences; maximum length 512 (Yelp) or 1,024 (OpenReview, PubMed), right truncation
Epochs20 for WEAN and its KD variants
TeacherLoRA SFT (r = 16, scaling 32) on the same training split for 20 epochs, learning rate 2×10⁻⁵, DeepSpeed ZeRO-2, bf16; queried once offline in bf16
CacheM = 128 AR candidates, K = 32 projected targets; single-token decode–re-encode vocabulary map
Decoding5,000 samples, up to 512 new tokens, 128 denoising steps over 4 blocks of 128 tokens, low-confidence remasking, Gumbel temperature 1.0; label pairs drawn i.i.d. from the training label distribution
Seeds42 for fine-tuning and generation; downstream classifiers trained with seeds 0, 1, and 2
Downstream (text)RoBERTa-base, 10 epochs, learning rate 3×10⁻⁵, batch size 32, maximum length 512

(c) POS-to-subtoken assignment. spaCy en_core_web_sm (NER, parser, and lemmatizer disabled) tags the decoded response text. Each spaCy word is re-tokenized in isolation with the student tokenizer, its POS tag is assigned to all resulting subwords, and the concatenated tags are aligned to the response tokens by position; any remaining positions receive a non-structural tag. A position is structural if its tag is DET, AUX, ADP, PUNCT, CCONJ, SCONJ, or PART. Whitespace tokens, prompt positions, and special tokens are always non-structural. Position alignment with the AR teacher is specified in A1(a). The corresponding scripts are scripts/run_experiment.sh and scripts/evaluate.sh in the anonymous repository.

Reviewer_MrWJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WEAN, a method for fine-tuning pretrained discrete diffusion language models using autoregressive teacher supervision. To address the mismatch between AR and diffusion conditioning contexts, WEAN selectively distills predictions at structural-token positions and adjusts the distillation weight according to the diffusion timestep and training progress. Teacher distributions are cached offline, preserving the student's original inference architecture. Experiments on LLaDA-8B and Dream-7B across several text, code, and mathematics datasets demonstrate improvements in synthetic-data quality and downstream utility.

Critical strengths
  1. The paper is well written and logically organized. The mismatch between AR and dLLM conditional distributions is a concrete technical challenge, and the proposed selective-distillation approach is intuitive and well motivated. The analysis in Figure 2 provides useful empirical support for the method's design choices.

  2. WEAN achieves strong results across two dLLM backbones and multiple datasets, outperforming the baselines on 19 of 20 text-generation metric/model combinations. The gains in distributional similarity are also encouraging, such as the improvement in Yelp MAUVE from 0.176 to 0.284 on LLaDA-8B. The ablation comparing structural-, semantic-, and all-token distillation also supports the importance of selecting appropriate supervision targets.

  3. WEAN does not require architectural modifications to the pretrained dLLM, and its offline teacher caching avoids additional inference-time overhead. The approach is relatively straightforward to incorporate into existing fine-tuning pipelines.

Critical weaknesses
  1. While the overall method performs well, the ablations do not fully establish which components account for the improvements. In particular, applying an additional KL loss at structural-token positions effectively combines teacher-based target smoothing with increased supervision weight on those positions. A baseline applying the same structural-token weighting without AR soft targets would help determine how much improvement comes from teacher knowledge itself. Similarly, the timestep-dependent and training-progress weighting strategies are not independently ablated.

  2. Although six datasets are considered, the experiments involve only two similarly sized dLLMs and a single AR teacher. Moreover, downstream utility is evaluated primarily through classifiers or solvers trained on synthetic data, rather than through direct assessments of the fine-tuned dLLMs' generation or reasoning capabilities. This is a meaningful evaluation setting but supports a narrower conclusion than general improvements in dLLM fine-tuning. Several absolute results on code and mathematics also remain low.

  3. While the offline caching strategy preserves inference efficiency, adapting the AR teacher and constructing the supervision cache introduce substantial additional computation. The reported representative training cost is considerably higher than standard dLLM SFT. A comparison against direct synthetic-data generation using the adapted AR teacher, or a cost-matched fine-tuning baseline, would better display the practical trade-off.

  4. The cross-tokenizer projection discards candidates that cannot be mapped to a single student token and truncates the projected distribution. It is also unclear how reliably the POS-based structural-token selection transfers to code and mathematical expressions.

Overall, this is a well-presented paper addressing a relevant problem in dLLM fine-tuning. The proposed method is intuitive, easy to integrate, and supported by reasonably consistent empirical improvements. My main concerns are the incomplete attribution of gains to the proposed mechanisms and the limited scope of the evaluation. Nevertheless, the paper presents a useful fine-tuning approach with promising results. I would be willing to increase my score if the concerns are properly addressed.

Questions for authors
  1. How much of WEAN's improvement comes from the AR teacher's soft predictions, compared with simply upweighting structural-token reconstruction? Could the authors evaluate a matched structural-token weighting baseline without distillation?

  2. Can the authors separately ablate the diffusion-timestep weighting and training-progress annealing? In particular, how does gradual annealing compare with a fixed early distillation phase followed by an abrupt cutoff?

  3. How robust are the results to different AR teacher sizes, and what is the variability across fine-tuning and generation seeds? Additional uncertainty estimates would be particularly helpful for the smaller performance improvements.

  4. What fraction of teacher probability mass survives tokenizer projection and top-K truncation? How are structural tokens identified for code and mathematics, where standard linguistic POS categories may not apply directly?

6 comments

A1 (W1, Q1 & Q2): Ablation study. Thank you for this insightful comment. We conducted controlled ablations that separately isolate (i) the contribution of teacher soft targets from that of structural-token upweighting, and (ii) the contribution of each component of the distillation weight λ(s, t) = λ_s(s)λ_t(t) in Eq. (3), including a comparison between gradual annealing and an early abrupt cutoff. These results will be added to the revised paper.

(a) Teacher soft targets vs. structural upweighting (Q1). Using LLaDA-8B-Instruct, we add a baseline, SFT-DLLM-Struc, which is identical to WEAN except that the cached teacher distribution at each masked structural position is replaced with the one-hot ground-truth token, so that L_KD becomes a cross-entropy loss on the same positions I_t = M_t ∩ S. It uses the same α, timestep-dependent weighting λ_t(t) = α(1 − t), and training-progress schedule λ_s(s) (Eq. (2)) as WEAN. Structural positions therefore receive the same total weight, 1 + λ(s, t), in both methods, which differ only in the target distribution.

MethodYelp Cate.↑Yelp Rating↑Yelp C2ST-S.↑Yelp MAUVE↑OR Area↑OR Rating↑OR C2ST-S.↑OR MAUVE↑
SFT-DLLM71.867.90.1900.12045.640.20.2560.342
SFT-DLLM-Struc72.568.90.2120.16946.540.80.3100.418
WEAN73.870.10.2380.28448.441.90.3880.531

Structural upweighting alone improves Category/Area accuracy over SFT-DLLM by 0.7/0.9 points, whereas replacing the hard labels with teacher soft targets yields a further 1.3/1.9 points. Across all eight metrics, teacher soft targets account for 54–70% of WEAN's total gain over SFT-DLLM. This indicates that the improvement stems primarily from the knowledge in the teacher's predictive distribution rather than from increased supervision weight on structural positions.

(b) Separate ablation of the two schedules (Q2). On OpenReview, we independently ablate the two factors of λ(s, t): the timestep-dependent weighting λ_t(t) = α(1 − t) and the training-progress schedule λ_s(s). When a factor is disabled (✗), it is replaced by a constant, and the coefficients are rescaled so that the cumulative KD weight over training is matched across variants. All variants use LLaDA-8B-Instruct and the same task-adapted Qwen3-32B teacher.

Timestep weighting λ_t(t)Training-progress schedule λ_s(s)Area↑Rating↑MAUVE↑
✗✗44.839.80.405
✓✗46.740.60.463
✗✓47.141.00.490
✓✓48.441.90.531

Each schedule individually improves Area accuracy over constant weighting (+1.9 and +2.3 points), and combining them yields the best result (+3.6 points over constant weighting; +1.7/+1.3 points over either schedule alone), indicating that the two schedules are complementary. Because the teacher and the total KD weight are held fixed, these gains reflect how teacher supervision is distributed across denoising timesteps and training stages, rather than how much supervision is applied. Notably, constant-weight KD improves MAUVE over SFT-DLLM (0.405 vs. 0.342) but slightly lowers Area accuracy (44.8 vs. 45.6), suggesting that uniformly applied teacher supervision is not sufficient by itself; the two schedules are essential for translating teacher supervision into consistent gains.

(c) Gradual annealing vs. early abrupt cutoff (Q2). With our default ρ = 0.2, Eq. (2) keeps λ_s(s) = 1 for the first 20% of training and then sets it to zero, i.e., it already implements a fixed early distillation phase followed by an abrupt cutoff (for ρ > 0.2, λ_s(s) instead decays linearly after the first 20% of training). The ρ sweep in Figure 3 changes both the shape of the schedule and its total weight; here we isolate the effect of the shape. We compare the default with gradual decay within the same early window, weaker full-run guidance, and late guidance, using OpenReview, LLaDA-8B-Instruct, the same task-adapted Qwen3-32B teacher, and the same λ_t(t). The profiles have equal integrated progress weight, and the cumulative KD weight is matched as in (b); the weaker full-run profile therefore coincides with the constant-progress variant in (b).

Training scheduleProgress-weight profile λ_sArea↑Rating↑MAUVE↑
Early abrupt cutoff (WEAN, ρ = 0.2)1 during the first 20%; 0 afterward48.441.90.531
Early gradual decayLinear decay from 2 to 0 during the first 20%; 0 afterward47.941.60.523
Weaker full-run guidance0.2 throughout training46.740.60.463
Late guidance1 during the final 20%; 0 beforehand46.040.60.454

Early guidance outperforms both weaker full-run guidance (+1.7 Area) and late guidance (+2.4 Area), with late guidance yielding the lowest Area accuracy and MAUVE. Thus, when teacher guidance is applied matters beyond its total weight: the key factor is concentrating guidance early in training and then removing it, so that the student can adapt to the target data through L_SFT alone. Given an early window, the decay shape has only a minor effect: the abrupt cutoff performs comparably to, and slightly better than, gradual decay (+0.5 Area). In the revision, we will clarify that the training-progress annealing in WEAN refers to reducing the teacher weight to zero early in training, which, under the default ρ = 0.2, takes the form of an abrupt cutoff.

Summary. Together, these results show that WEAN's improvements cannot be explained by increased supervision weight alone: teacher soft targets account for the majority of the gain, the two schedules contribute complementary improvements under a fixed total KD budget, and early guidance is more effective than full-run or late guidance with the same total weight.

A2 (W2 & Q3): Evaluation scope and robustness. Thank you for these comments.

(a) Scope of the claims. We agree that our evidence supports a narrower conclusion than general improvements in dLLM fine-tuning. WEAN targets task-specific adaptation of pretrained dLLMs as data generators, and we will revise the abstract, introduction, and conclusion to scope our claims accordingly. Within this scope, our evaluation already includes direct assessments of the fine-tuned dLLMs' outputs rather than only downstream utility: on the text benchmarks, C2ST-Sim and MAUVE directly measure how close the generations of the fine-tuned dLLM are to real data (Appendix B.3), and this is where WEAN's gains are largest (e.g., MAUVE improves from 0.176 to 0.284 over the strongest baseline on Yelp with LLaDA-8B-Instruct). For code and mathematics, where these text-based metrics do not apply, we assess the generated data through downstream utility.

(b) Low absolute values on code and mathematics. The low scores occur on PrimeVul and DM-Math, whereas CodeNet accuracy reaches 96.5–98.7% with WEAN. These scores are not the dLLM's own coding or reasoning accuracy: fixed CodeBERT classifiers and a T5-base solver are trained only on 5,000 synthetic examples and evaluated on real held-out data, using positive-class F1 for PrimeVul and normalized answer EM for DM-Math. Both tasks are inherently difficult under this protocol. PrimeVul was designed as a realistic vulnerability-detection benchmark on which even fine-tuned code LMs achieve low F1 (e.g., a state-of-the-art 7B model scores 3.09% F1 on PrimeVul versus 68.26% on BigVul; Ding et al., 2024), and DM-Math requires a small solver to learn exact-answer computation from only 5,000 synthetic question–answer pairs. The absolute values should therefore be read within this synthetic-only protocol, while the matched comparisons, which share the same generation budget, downstream model, and test set, establish WEAN's relative gains. Notably, WEAN is the only method that improves over Pre-DLLM on all three datasets with both backbones (Table 4). We will additionally evaluate a data-augmentation setting in which each method's synthetic data is combined with the same real-training subset, while retaining the original test sets and metrics.

(c) Coverage of dLLMs and teachers (Q3). Although the two dLLMs are of similar size, they follow different construction paradigms: LLaDA-8B-Instruct is trained from scratch as a dLLM, whereas Dream-v0-Instruct-7B is initialized from the Qwen2.5 AR model (Appendix A). WEAN improves both backbones (Tables 2 and 4). For the teacher, on Yelp with LLaDA-8B-Instruct, replacing Qwen3-32B with Qwen3.8-27B, a teacher from a newer model generation, also yields consistent improvements over ordinary SFT, with slightly larger gains.

StudentTeacherCate.↑Rating↑MAUVE↑
LLaDA-8B-InstructNone (ordinary SFT)71.867.90.120
LLaDA-8B-InstructQwen3-32B73.870.10.284
LLaDA-8B-InstructQwen3.8-27B74.670.60.302

These results suggest that WEAN is not tied to a specific teacher and benefits from a stronger teacher.

(d) Variability across seeds (Q3). We will supplement the comparisons with repeated runs varying the generator fine-tuning, synthetic sampling, and downstream-training seeds, and report mean ± standard deviation together with paired improvements under matched seeds. This analysis will focus on the settings with smaller margins, including the ablations in A1 and the DM-Math and PrimeVul results, to directly assess whether the reported improvements persist across the stochastic stages of the evaluation.

A3 (W3): Computational cost and practical trade-off. Thank you for this suggestion. We agree that WEAN incurs a substantially higher one-time preparation cost than standard SFT. We clarify the trade-off below using the representative setting in Appendix D (Table 5; LLaDA-8B-Instruct, four A100-80GB GPUs, 5,000 generated examples).

(a) Stage-wise cost.

MethodAR-teacher fine-tuningCache constructionStudent fine-tuningGeneration (5k examples)Total
SFT-DLLM––5.2 h7.0 h12.2 h
WEAN18.0 h3.5 h5.8 h7.0 h34.3 h
Direct AR generation (task-adapted Qwen3-32B)18.0 h––14.7 h32.7 h

WEAN's preparation takes 27.3 h versus 5.2 h for SFT, almost entirely due to the offline teacher stages. Because the teacher is queried offline, the teacher and student never reside in GPU memory together, so the student stage remains close to SFT (5.8 h and 45.0 GB vs. 5.2 h and 43.0 GB peak memory per GPU). Moreover, the offline costs are reusable: the task-adapted teacher does not depend on the student and can be shared across student models, and the cache is reused across all runs with the same student tokenizer, e.g., during hyperparameter tuning. Each additional WEAN run therefore costs 5.8 h versus 5.2 h for SFT.

(b) Comparison with direct AR generation. WEAN is a distillation method in which an 8B dLLM student learns from a 4× larger 32B AR teacher, so the student is expected to remain below its teacher in generation quality. The relevant question is therefore how much of the teacher's quality the student recovers, and at what generation cost. We compare against direct generation from the task-adapted Qwen3-32B teacher on Yelp under matched conditioning inputs, a 5,000-example budget, and identical evaluation.

GeneratorCate.↑Rating↑C2ST-S.↑MAUVE↑
SFT-DLLM (LLaDA-8B-Instruct)71.867.90.1900.120
WEAN (LLaDA-8B-Instruct)73.870.10.2380.284
Task-adapted Qwen3-32B (direct generation)74.571.00.2500.301

WEAN recovers 71–91% of the teacher's quality advantage over SFT-DLLM across all four metrics (Category 74%, Rating 71%, C2ST 80%, MAUVE 91%), while the resulting dLLM generates about 2.1× faster than the AR teacher (40.0 vs. 19.0 tokens/s) with lower peak memory (24.5 GB vs. 69.0 GB per GPU). WEAN thus combines most of the AR teacher's generation quality with the faster parallel generation of dLLMs. At the 5,000-example budget, direct AR generation is slightly cheaper in total (32.7 h vs. 34.3 h) because WEAN's one-time preparation dominates. This overhead (9.3 h) is amortized by faster generation (saving 7.7 h per 5,000 examples), so WEAN becomes cheaper beyond roughly 6,000 examples; for example, generating 50,000 examples takes 97.3 h with WEAN versus 165.0 h with direct AR generation (41% less).

(c) Practical trade-off. Direct AR generation remains a reasonable choice for small, one-off budgets, where it is slightly cheaper and serves as the quality reference. WEAN is preferable when a task-adapted dLLM is itself the goal, or when synthetic data are generated at scale or repeatedly: it retains most of the teacher's quality, while the faster and more memory-efficient dLLM amortizes the one-time teacher cost. We will add this analysis to Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.