- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WEAN uses an autoregressive teacher to improve downstream fine-tuning of pretrained discrete diffusion language models. It distills structural tokens, weights supervision by the noise level, and removes teacher guidance later in training. Cross-tokenizer alignment and cached teacher distributions enable transfer across model families. Experiments on two dLLM backbones and six datasets evaluate generation quality and the downstream utility of synthetic data.
Critical strengths
-
The method addresses a practical mismatch between AR and diffusion conditioning. Selective, temporary teacher supervision offers a sensible way to use a stronger teacher while allowing the student to adapt to task data. The trained model retains its original inference procedure.
-
Improvements are consistent across backbones and datasets. In Table 3, structural-token distillation outperforms semantic-only and all-token distillation on all ten metrics, directly supporting the token-selection design. The code and mathematics experiments also show gains in synthetic-data utility.
Critical weaknesses
-
The explanation for the advantage of structural tokens remains indirect. Figure 2 compares AR and dLLM predictions only on structural tokens; its other panels measure changes within the dLLM after SFT. Neither directly compares AR–dLLM compatibility across structural and semantic tokens. Table 3 supports the selection rule, but does not establish that reduced conditional mismatch explains its gains.
-
The training-schedule experiment confounds timing with cumulative distillation weight. The best setting in Figure 3, ρ = 0.2, uses distillation for the first 20% of training and then switches it off. Changing ρ changes both when supervision is applied and its total weight, leaving unclear whether early guidance or simply weaker overall distillation drives the improvement.
Minor comment: On p. 6, remove the leading minus sign in the KD loss to match Algorithm 1 and the implementation.
Questions for authors
-
Under matched samples and masking conditions, how do AR–dLLM KL divergence and probability overlap compare between structural and semantic tokens?
-
With the same teacher and noise weighting, does early distillation outperform weaker supervision throughout training when cumulative distillation weights are matched?
4 comments
Dear Reviewer pJKu,
Thank you again for your time and constructive feedback on our submission.
As the rebuttal period is coming to a close today (November 18, AoE), we would like to gently follow up to see if our responses and additional experiments have addressed your concerns or if there are any remaining questions we can clarify before the deadline.
We truly appreciate your time and effort in evaluating our work.
I thank the authors for the detailed response and additional experiments, which address most of my concerns. Based on my overall assessment of the paper's contributions, I will maintain my original score.
Dear Reviewer pJKu,
Thank you for your thoughtful review and for engaging with our response. We are glad that we addressed most of your concerns, and we appreciate your positive assessment of our work.
A1 (W1 & Q1): Direct AR–dLLM compatibility on structural vs. semantic tokens. Thank you for this suggestion. We agree that Figure 2 does not directly compare the two token types, and we provide this comparison below.
(a) Direct comparison (Q1). Using the same held-out samples, masking patterns, and projection settings (M = 128, K = 32), we compare the task-adapted Qwen3-32B teacher with the vanilla-SFT LLaDA-8B-Instruct student on structural and semantic positions. Both models are fixed, and the student has not undergone WEAN distillation, so the comparison reflects the intrinsic compatibility of the two models rather than an effect of our training.
Structural tokens show higher probability overlap (0.310 vs. 0.220, a 41% relative increase) and lower forward KL divergence (6.30 vs. 6.58) than semantic tokens, directly confirming that AR and dLLM predictions are more compatible at structural positions.
(b) Link to the selection gains. The downstream results follow the same ordering. In Table 3, structural-token distillation outperforms all-token and semantic-only distillation on all ten metrics (e.g., Yelp Category 73.8% vs. 70.7% and 64.9%), and semantic-only distillation, which supervises the least compatible positions, performs worst. Moreover, replacing the teacher's soft targets at structural positions with one-hot labels under the same weighting removes 54–70% of WEAN's gain over SFT (see our response to Reviewer A), which rules out the alternative explanation that structural selection helps merely by upweighting these positions. Together, these results establish a consistent association between AR–dLLM compatibility and distillation gains, although not a complete causal decomposition.
This selectivity reflects the goal of WEAN. The AR model serves as an auxiliary teacher during fine-tuning, not as a target for the dLLM to replicate or surpass. WEAN therefore transfers AR knowledge only where it is compatible with the dLLM's conditioning, while the final model remains a dLLM that retains its parallel generation and faster inference.
A2 (W2 & Q2): Timing at matched cumulative weight. Thank you for pointing out this confound. With ρ = 0.2, Eq. (2) keeps λ_s(s) = 1 for the first 20% of training and then sets it to zero. The ρ sweep in Figure 3 changes both when supervision is applied and its total weight; here we isolate the effect of timing. On OpenReview, using the same task-fine-tuned Qwen3-32B teacher, LLaDA-8B-Instruct student, and noise weighting λ_t(t) = α(1 − t), we compare four progress-weight profiles with equal integrated weight and matched cumulative KD weight.
Early guidance outperforms both weaker full-run guidance (+1.7 Area) and late guidance (+2.4 Area). Since all profiles share the same cumulative weight, the improvement stems from applying guidance early rather than from weaker overall distillation (Q2). Given an early window, the decay shape has only a minor effect: the abrupt cutoff performs comparably to, and slightly better than, gradual decay (+0.5 Area). Accordingly, the training-progress annealing in WEAN refers to reducing the teacher weight to zero early in training, which, under ρ = 0.2, takes the form of an abrupt cutoff.
A3 (Minor comment): KL sign. Thank you for catching this. The implemented objective adds a positive forward KL to the denoising loss, as in Algorithm 1; the minus sign in Sec. 3.3 is a typographical error. In addition, L_KD is averaged rather than summed over the masked structural positions (see our response A3 to Reviewer B). Neither correction changes the implemented objective or requires rerunning experiments.