- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies learned generation orders for compute-efficient training of masked diffusion language models. SOLDIFF applies the latent-order formulation of LO-ARM within a block diffusion model, using a clean sequence posterior to sample training orders and a masked state prior to guide generation. The paper introduces rank-weighted reconstruction and order prediction losses for parallel decoding. Experiments include continued pretraining at 1.7B, 8B and 14 scales, comparisons under matched training FLOPs and controlled tasks with known ordering dependencies.
Critical strengths
- The paper demonstrates learned-order training for masked diffusion model. with the same initialization,data, and training FLOPs,SOLDIFF LO improves 12 benchmarks average score from 47.44 to 52.75 over standard block diffusion. Table 7 also shows gains when both methods start from the same MDM checkpoint.
- The auxiliary objectives improve parallel decoding. Figures 4 and 10 show that the learned-order objective alone loses accuracy rapidly as decoding parallelism increases, while supervision over the remaining masked positions largely prevents this degradation.
Critical weaknesses
- Limited methodological novelty. The latent permutation model, variational objective, Plackett-Luce posterior and two-sample RLOO estimator follow LO-ARM. The block factorization is also established in block diffusion. The methodological additions are mainly the rank-weighted reconstruction and suffix-order losses; much of the remaining contribution consists of evaluating existing design choices at larger scales.
- The experiments don't sufficiently isolate the benefit of data dependent order learning. Table 6 compares the learned posterior with deterministic LTR, changing both dependence on the input and varication in the sampled orders. It does not establish whether the gains require example specific orders or can be obtained by sampling small deviations from LTR. The average gains are 0.47-1.75, with the improvement concentrated in HumanEval. Moreover, the fixed LTR variant retains two token-predictor evaluations per example despite having a deterministic poesterior. The absence of parallel decoding results for fixed LTR leaves the question unresolved for the auxiliary objectives.
Questions for authors
- In the fixed LTR experiment, do the two token-predictor evaluations receive identical masked inputs under the shared step sampling? If so, why retain both evaluations? How much of SOLDIFF’s training-compute advantage remains against a fixed LTR baseline that removes this duplicate computation and uses the saved compute for additional training, while retaining the same auxiliary losses?
- Does conditioning the order distribution on each example provide a measurable benefit beyond a stochastic preference for LTR? A direct control would use a Plackett–Luce posterior with learned position scores that are shared across examples and independent of their content. With the same auxiliary objectives and matched training FLOPs, how does this control compare with SOLDIFF when generating one token or multiple tokens per step? Table 6 does not separate these two effects.
- How much of the final model’s performance depends on the learned prior at inference? Keeping the trained token predictor fixed, what happens when the prior’s ranking is replaced by LTR or token-confidence ranking at the same number of tokens per decoding step? The prior ablations in Table 3 retrain the model and therefore do not separate the benefit of the training procedure from the benefit of the inference-time order.