- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes CrossDLM, a framework for distilling autoregressive language models into diffusion language models using supervision aligned with partially observed text. It combines approximate suffix scoring with importance-weighted aggregation over student-generated completions and improves performance over causal-prefix distillation across multiple benchmarks and model families.
Critical strengths
-
Clear motivation. The paper identifies a concrete supervision mismatch: the teacher predicts from a left prefix, whereas the student reconstructs missing tokens using context on both sides. It also distinguishes observed evidence from unresolved content that has only been provisionally completed, making the intended improvement precise.
-
A clear connection between theory and implementation. The formulation establishes conditional teacher marginals as the desired supervision targets, and the practical algorithm provides a principled approximation through local scoring and completion aggregation.
-
Meaningful empirical improvements. The method achieves substantial gains on several tasks, particularly mathematical reasoning, with benefits observed across multiple model families. The matched backbones and training configurations used for OPDLM and CrossDLM reduce confounding from differences in model size, training data, and training budget.
Critical weaknesses
-
Limited validation at larger block sizes. The main experiments use four-token blocks, leaving it unclear whether the method remains effective when more tokens are generated in parallel. Larger blocks introduce stronger dependencies among unresolved positions and may make the proposed marginal estimation less reliable.
-
Missing comparisons with the original autoregressive models. The paper does not provide a comprehensive comparison against the original AR models in terms of both task performance and generation speed. Consequently, it remains unclear how much capability is retained during conversion and whether the resulting models offer a favorable quality–speed trade-off.
Questions for authors
Evaluate several larger block sizes and report task accuracy, generation latency, and throughput alongside the original AR models and OPDLM to establish the practical quality–speed trade-off.
3 comments
Thank you to the authors for the detailed response and clarifications. In light of the rebuttal, I have raised my Soundness score, while keeping my Rating unchanged.
Thank you for your thoughtful consideration and for updating your assessment! We appreciate your feedback.
We thank the reviewer for highlighting the practical quality–speed trade-off at larger block sizes. Below, we report the corresponding scaling and source-model comparisons.
W1: larger blocks and the quality–speed trade-off
To test scaling directly, we trained two matched B16 OPDLM/CrossDLM pairs rather than decoding B4 checkpoints with larger blocks:
Both seeds show consistent gains on both reasoning tasks. To isolate the quality–parallelism trade-off within diffusion decoding, we then changed only the denoising schedule:
d=16uses 16 forwards per 16-token block (about one token per forward), whereasd=8uses eight (about two tokens per forward). Across 3,638 responses, wall time falls from 984 to 603 s (1.63x aggregate throughput), while CrossDLM remains +5.76/+7.40 points above OPDLM on GSM8K/MATH-500. This is a schedule-level speedup, not a CrossDLM-over-OPDLM or CrossDLM-over-AR latency claim.W2: comparison with the source AR model
We appreciate the request for a direct source-AR reference. On terminal Qwen3.5-4B, cap-aware Source AR / CrossDLM is 92.42 / 91.05 on GSM8K, 79.40 / 74.20 on MATH-500, and 74.11 / 76.31 on MMLU. Thus conversion is not universally lossless, but CrossDLM preserves most source quality while enabling blockwise decoding. In the Qwen3-4B B16 control, it also recovers approximately 43% of OPDLM's conversion loss on both GSM8K and MATH-500.
For speed, an implementation-normalized Qwen3-4B H100 benchmark fixes both prompt and output to 128 tokens on the same PyTorch/SDPA/BF16 stack. B16/d8 CrossDLM needs 64 forwards and 2.991 s, whereas raw AR needs 128 decode steps and 4.878 s with the standard Hugging Face KV cache (5.571 s without it), giving 1.63x/1.86x speedups. This measures the benefit of lower sequential depth. On the separately optimized Qwen3.5 serving stack, raw AR is currently faster (4.240 versus 5.863 s), consistent with the greater maturity of AR infrastructure such as vLLM/SGLang. We therefore frame the contribution as improved adaptation with algorithmic parallelism, while optimized DLM serving remains a future direction.