- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a post-training procedure for DLMs, with the goal of improving performance vs. number of steps tradeoff (improve performance with fewer generation steps) which requires teaching the model to decode more tokens per forward pass. The main motivation is to simplify DLM postraining as compared to two families of methods discriber in ll.031-037 (distillation with teacher-derived targets) while matching them on quality at given step budget.
What proposed method does roughly is:
- does sample positions S and then sample completions at these positions by the trainable model, G*B times for some context c
- then get representations of these positions with a pre-trained frozen MDM
- also get representations for these positions for the ground truth for that context c
- then compute MMD loss using RBF as a similarity
- training with MMD as reward (per group, paper uses REINFORCE since to calculate gradients w.r.t. params) loss encourages sampling examples that tend to be more similar in their representations to ground truth representations and less similar within themselves (within B sampled examples), these are also examples that are more likely under the data distribution. Intuitively, given the current partially-masked input, choose per-position distributions such that filling all masked positions at once, independently, tends to produce sequences a pret-trained DLM would recognize as real text.
The continuous variant differentiates through generated representations instead of REINFORCE.
Critical strengths
The soundness of the proposed method constitutes the main strength. The method is carefully designed, main design choices are ablated, the idea of using MMD in the frozen feature space of a pre-trained DLM is novel to my knowledge, simplifies the post-raining design and is shown to work in the experiments.
Critical weaknesses
None
Questions for authors
- how does the quality of a pre-trained DLM in who's feature space we perform MMD effects the method performance? Does it need to be the same checkpoint as the trained model?
- in the current version only representations from a single layer of the frozen reference DLM are used, how are the layer(s) selected? Did authors observe layer choice impacting the learning quality?
- how are the hyperparameters like \sigma and the layer index chosen?
- the main motivation is to simplify DLM post-training as compared to two families of methods described in ll. 032-037. However, if I am not mistaken, for the discrete DLM experiments only baselines from the latter family are used (, i.e. all baselines IDLM, IDLM-REINFORCE, DiDi-Instruc all belong to the second family). I think adding baselines from the first family (SDTT , CDLM) would make the comparison complete.
- Why no other baselines are included in Table 1 (IDLM or DiDi-Instruct style methods)?
4 comments
Rebuttal | Part 2
Q4. Baselines from the first family of methods: SDTT and CDLM
I think adding baselines from the first family (SDTT, CDLM) would make the comparison complete.
We thank the reviewer for the suggestion and have added both baselines.
SDTT (OWT, unconditional). At matched entropy, SDTT has the highest generative perplexity among all methods at 8, 16 and 32 steps, please see the figure: anonymous link
CDLM (TinyGSM). We adapted the CDLM implementation to our setup:
| Model | 4 steps | 8 steps | 16 steps | 32 steps | 48 steps |
|---|---|---|---|---|---|
| MDLM | 2.4 | 25.3 | 35.2 | 36.7 | 35.5 |
| CDLM | 9.0 | 28.3 | 35.3 | 38.0 | 38.1 |
| MDLM-MMD | 5.1 | 27.0 | 40.3 | 48.2 | 52.3 |
| MDLM-MMD+CDLM | 10.3 | 37.8 | 49.9 | 53.6 | 55.1 |
CDLM is more accurate at 4–8 steps, while MDLM-MMD is 5–14 points better at 16–48 steps.
As a side note, our conclusion mentions combining MMD with distillation losses as a promising direction. Interestingly, when we applied CDLM on top of MDLM-MMD, it significantly improved accuracy at every budget. At 8–48 steps, the gain is 2.8–10.8 points, while CDLM on the base MDLM gains at most 3 points. These results encourage further work on such combinations.
CDLM results for the 16B DMax setup are reported under Q5 below.
Q5. Additional baselines in Table 1
Why no other baselines are included in Table 1 (IDLM or DiDi-Instruct style methods)?
We initially could not obtain improvements with IDLM at the 16B scale and therefore did not include it. During the rebuttal, we found a working configuration and report it below, together with CDLM. We focus on IDLM rather than DiDi-Instruct, since IDLM is a more recent approach that shows superior results on other benchmarks.
| Method | GSM8K | MATH500 | Minerva-Algebra | ASDIV | HumanEval-Instruct | MBPP-Instruct |
|---|---|---|---|---|---|---|
| DMax | 92.1 / 5.48 | 75.4 / 5.94 | 91.5 / 7.03 | 92.5 / 5.62 | 83.5 / 7.36 | 79.2 / 5.86 |
| DMax-CDLM | 92.0 / 5.66 | 75.3 / 6.24 | 91.4 / 7.36 | 92.9 / 5.74 | 83.9 / 7.60 | 81.0 / 5.86 |
| DMax-IDLM | 92.1 / 6.16 | 76.3 / 6.86 | 91.1 / 8.14 | 92.7 / 6.24 | 86.3 / 8.20 | 81.6 / 6.13 |
| DMax-MMD | 92.1 / 6.15 | 76.0 / 6.84 | 92.1 / 8.19 | 92.9 / 6.20 | 85.9 / 8.07 | 83.0 / 6.10 |
DMax-IDLM is close to DMax-MMD in TPF and in accuracy on four of the six benchmarks, with more noticeable DMax-MMD gains on Minerva-Algebra (+1.0) and MBPP (+1.4). Also, DMax-IDLM takes about 50% more training time (~20 min vs ~13 min for DMax-MMD) and requires training an additional model.
DMax-CDLM improves TPF only slightly over DMax (by at most 0.33). Compared to DMax-MMD, it has lower TPF on every benchmark and lower accuracy on five of the six.
Rebuttal Follow-up
Dear Reviewer 3QLA,
Thank you for your positive assessment and helpful suggestions. Since the discussion ends soon, we wanted to follow up on our response.
In our rebuttal, we answered your questions about the choice of the feature extractor, its layer and the kernel bandwidth with additional experiments, and added the baselines you asked about: SDTT on OpenWebText, CDLM on TinyGSM, and IDLM and CDLM for the 16B DMax models in Table 1.
We have added all these results to the revised paper. Please let us know if you have any remaining questions.
I thank the authors for their extensive responses. I will keep my positive score for this paper.
Rebuttal | Part 1
We thank the reviewer for their positive and constructive feedback. We address each question below and will provide a revised manuscript and a general response to the reviews in the coming days.
Q1. Feature extractor quality and choice of checkpoint
We thank the reviewer for the interesting question. To explore it, we repeat MMD post-training on TinyGSM with features from a different pretrained DLM.
ELF. We use ELF-B and the larger, more accurate ELF-M (Appendix C.6) as feature extractors for both models:
The stronger ELF-M features do not improve ELF-B, and the weaker ELF-B features work almost as well for ELF-M as its own features.
MDLM. We could not find a discrete DLM trained on TinyGSM with noticeably better accuracy. Therefore, we use MDLM 1.7B [1], a much larger general-purpose model that was not trained on TinyGSM:
With MDLM 1.7B features, the accuracy is slightly lower than with the model's own features, which we attribute to MDLM 1.7B not being fitted to the TinyGSM format. MMD post-training still clearly improves over the base MDLM at 16–48 steps.
Overall, the feature extractor does not need to be the same checkpoint as the trained model: MMD post-training also works with features from a different model, even one not trained on the target dataset. However, a larger feature extractor does not necessarily bring additional gains, so we suggest using the pretrained model itself by default, since it also requires no additional model.
[1] Sahoo et al. (2026), “Scaling Beyond Masked Diffusion Language Models”
Q2. Selection of the feature extraction layer
By default, we use a single middle layer for simplicity. The only exception is the unconditional MDLM setup, where an earlier layer gave slightly better results.
We also tried other layers and did not find the particular choice to be critical. Our rule of thumb is to take a layer that is deep enough to provide a semantically structured feature space. Below, we compare different layers for MDLM and ELF on TinyGSM:
Overall, the layer choice does not significantly impact the learning quality. The middle layer works best at smaller step budgets in both setups, and no other layer consistently outperforms it, so we suggest using it by default.
Q3. Selection of σ and the layer index
σ selection. We first find a reasonable range of values with the median heuristic from prior work [2, 3]: we extract token features from a batch of real training samples and compute the median pairwise distance between them. We then try a few values below and above this estimate and choose the one that performs best on a held-out set. This set contains 512 examples from the GSM8K training split and does not overlap with the training or test data.
We did not perform any extensive σ tuning, and the results are not too sensitive to the particular value within a reasonable range. The σ ablation is provided in our response to Reviewer j8mG (Q4).
Layer selection. As discussed in Q2, we use the middle layer by default and do not tune this choice in most cases.
[2] Gretton et al. (2012), “A Kernel Two-Sample Test”
[3] Garreau et al. (2017), “Large Sample Analysis of the Median Heuristic”