acceptodds
Under review as a conference paper at ICLR 2027

Scaling Order Learning for Diffusion LLMs

16 views

Abstract

The standard masked diffusion model (MDM) objective treats all unmasking orders uniformly, potentially allocating substantial training compute to orders poorly suited to individual examples. The resulting training inefficiency makes high-performing MDMs costly to train even when initialized from strong autoregressive models (ARMs). We introduce SolDiff, a scalable variational framework that learns data-dependent order distributions to train stronger MDMs under limited compute budgets. We model orders as latent permutations, with a learned posterior focusing token-prediction training on mask patterns induced by preferred orders and a learned prior reproducing these preferences at inference. To scale this formulation to MDM pretraining, we systematically identify key design choices for stable and effective optimization and introduce rank-weighted auxiliary objectives to support parallel decoding. Across 12 benchmarks, SolDiff-8B outperforms all other evaluated ARMs and MDMs of comparable size on average. We highlight that this result requires only 100B tokens of continued pretraining, less than 0.3% of the base model's pretraining tokens and substantially fewer than those used in existing ARM-to-MDM adaptations. In controlled comparisons, SolDiff outperforms standard block diffusion trained with as much compute. We further observe consistent gains across model sizes from 1.7B to 14B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion, scaling, large language model, variational inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YzZqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learned generation orders for compute-efficient training of masked diffusion language models. SOLDIFF applies the latent-order formulation of LO-ARM within a block diffusion model, using a clean sequence posterior to sample training orders and a masked state prior to guide generation. The paper introduces rank-weighted reconstruction and order prediction losses for parallel decoding. Experiments include continued pretraining at 1.7B, 8B and 14 scales, comparisons under matched training FLOPs and controlled tasks with known ordering dependencies.

Critical strengths
  1. The paper demonstrates learned-order training for masked diffusion model. with the same initialization,data, and training FLOPs,SOLDIFF LO improves 12 benchmarks average score from 47.44 to 52.75 over standard block diffusion. Table 7 also shows gains when both methods start from the same MDM checkpoint.
  2. The auxiliary objectives improve parallel decoding. Figures 4 and 10 show that the learned-order objective alone loses accuracy rapidly as decoding parallelism increases, while supervision over the remaining masked positions largely prevents this degradation.
Critical weaknesses
  1. Limited methodological novelty. The latent permutation model, variational objective, Plackett-Luce posterior and two-sample RLOO estimator follow LO-ARM. The block factorization is also established in block diffusion. The methodological additions are mainly the rank-weighted reconstruction and suffix-order losses; much of the remaining contribution consists of evaluating existing design choices at larger scales.
  2. The experiments don't sufficiently isolate the benefit of data dependent order learning. Table 6 compares the learned posterior with deterministic LTR, changing both dependence on the input and varication in the sampled orders. It does not establish whether the gains require example specific orders or can be obtained by sampling small deviations from LTR. The average gains are 0.47-1.75, with the improvement concentrated in HumanEval. Moreover, the fixed LTR variant retains two token-predictor evaluations per example despite having a deterministic poesterior. The absence of parallel decoding results for fixed LTR leaves the question unresolved for the auxiliary objectives.
Questions for authors
  1. In the fixed LTR experiment, do the two token-predictor evaluations receive identical masked inputs under the shared step sampling? If so, why retain both evaluations? How much of SOLDIFF’s training-compute advantage remains against a fixed LTR baseline that removes this duplicate computation and uses the saved compute for additional training, while retaining the same auxiliary losses?
  2. Does conditioning the order distribution on each example provide a measurable benefit beyond a stochastic preference for LTR? A direct control would use a Plackett–Luce posterior with learned position scores that are shared across examples and independent of their content. With the same auxiliary objectives and matched training FLOPs, how does this control compare with SOLDIFF when generating one token or multiple tokens per step? Table 6 does not separate these two effects.
  3. How much of the final model’s performance depends on the learned prior at inference? Keeping the trained token predictor fixed, what happens when the prior’s ranking is replaced by LTR or token-confidence ranking at the same number of tokens per decoding step? The prior ablations in Table 3 retrain the model and therefore do not separate the benefit of the training procedure from the benefit of the inference-time order.
Reviewer_VehkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SoLDiff learns which tokens to generate first while adapting an autoregressive model to masked diffusion. A training-only posterior sees the complete text, while a learned prior chooses positions from partially masked text at inference. The method combines a variational objective, a two-sample gradient estimator, and auxiliary losses for parallel decoding. Experiments cover 1.7B–14B models, compute-matched baselines, design ablations, and synthetic tasks with known dependency structure.

Critical strengths
  1. The controlled comparisons make a convincing case for learned-order training. With the same initialization and data, SoLDiff improves on several masking baselines at matched training FLOPs. The authors also compare against a fixed left-to-right order and test continued training from an existing diffusion checkpoint. These controls help rule out simpler explanations for the gains.

  2. The paper turns an existing latent-order framework into a useful large-model training recipe. The ablations explain why posterior capacity, gradient flow, and variance reduction matter. The added losses improve parallel decoding, where the learned-order objective alone struggles. The synthetic tasks also show that the model can learn dependencies.

Critical weaknesses
  1. The paper lists broad training data categories but does not give dataset sources, their proportions, or checks for overlap with evaluation benchmarks. These details matter for reproducing the results and interpreting the small lead over external 8B baselines. The comparisons using the same data support the method. They do not establish that order learning explains the gains over external models, which use different training data.

  2. The paper does not show how much time the method saves. It compares training using estimated FLOPs and parallel decoding using tokens per forward pass. The paper gives GPU-hours for SoLDiff, but no matched runtime comparison. A diffusion pass and an autoregressive decoding step can take different amounts of time. The plots therefore do not establish the actual speedup.

Questions for authors
  1. Which datasets and proportions did you use for continued pretraining and instruction tuning? How did you check for overlap with evaluation benchmarks? Did all controlled baselines use the same data distribution?

  2. Do you have training and decoding time measurements under matched conditions? Please specify hardware, batch size, sequence length, and caching. If not, please make clear that the efficiency results measure FLOPs and decoding steps, rather than elapsed time.

  3. How does SoLDiff differ from [1], which learns an unmasking policy and also studies fine-tuning along learned trajectories? Please discuss the differences in objectives and which model components are trained.

[1] Xu, et al., Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models. ICML, 2026. arxiv.org/abs/2606.23567

Reviewer_Hw3gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SOLDIFF, a framework aimed at learning data-dependent token generation orders for masked diffusion models. It conceptualizes these generation orders as latent permutations and utilizes a variational objective to train an order posterior and a prior that can effectively guide the inference process. Additionally, the paper introduces a block-wise formulation designed to enhance the practicality of learned-order training at scale. Experiments conducted across various model sizes and multiple benchmarks demonstrate notable improvements over traditional masked diffusion training and selected masking strategies, all while adhering to matched compute budgets.

Critical strengths

The paper addresses an important question for masked diffusion models: whether learning the generation order can improve the use of training compute and the way tokens are generated. The block-wise formulation is motivated by the difficulty of learning full-sequence orders, and the paper includes controlled comparisons that match training data, initialization, and compute for the 1.7B setting. The evaluation also includes ablations and scaling experiments, providing useful evidence on the proposed framework’s behavior.

Critical weaknesses

The PUMA comparison is limited to PUMA-1S, which the paper describes as a one-shot variant. Since the original PUMA uses progressive unmasking, comparison only with this adaptation may not represent the original method’s performance. The paper should explain why the original method is not included or otherwise substantiate the choice of variant.

In "Related Work," LoMDM, Fox et al., PAPL, and AOP are mentioned. These methods also study learned or planner-based order preferences and their interaction with the denoising objective. These methods differ in formulation and experimental setup, so not all may be directly comparable at the same scale. Still, their absence makes it difficult to assess how much SOLDIFF advances beyond existing learned-order approaches rather than beyond heuristic schedules.

Questions for authors

Could you compare SOLDIFF with LoMDM and Fox et al.’s variational generation-order method, or explain why a controlled comparison is not feasible? Could you clarify why the controlled 1.7B experiments exclude other learned or planner-based order methods, such as PAPL and AOP? If these methods are not directly comparable, please explain the differences in setup and scope. Why is the PUMA baseline limited to PUMA-1S rather than the original progressive-unmasking method? Could you provide a comparison with the original method or clarify the limitation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.