acceptodds
Under review as a conference paper at ICLR 2027

On Trajectory-Aware Training for Masked Diffusion Language Models

Abstract

Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly masked sequences, whereas inference follows a trajectory shaped by the model’s own predictions. Additionally, each step has no access to what the previous one computed. Recent methods narrow these limitations from separate angles, leaving open how these choices interact. We introduce PUMBA, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time. A controlled study of this design space shows that i) exact train–inference alignment fails due to local overfitting, whereas a looser alignment still brings training masks closer to those seen at inference; ii) passing continuous information outperforms discrete gradient estimators through the commitment at each step; and iii) performance improves as backpropagation through time spans more steps, which we support theoretically. Combined, these components match the best checkpoint of a same-size autoregressive model. Building on these findings, we scale PUMBA to supervised fine-tuning of LLaDA-8B, where it improves the trade-off between performance and number of function evaluations (NFEs) in both full-canvas and block diffusion generation. At matched performance, it needs up to 22% fewer NFEs than standard fine-tuning with twice the budget in full-canvas generation, and up to 26% fewer than standard fine-tuning for the same number of steps in block diffusion.

Primary area:
generative models
Keywords:
Masked Discrete Diffusion, Training Algorithm

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_T4CkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Broadly speaking, the paper addresses the problem of discrepancy between the training- and inference-time trajectories of masked diffusion models, where the masking configurations seen at training times are typically random, while the masking configurations seen at inference time are generated by the model’s own policy. To address this problem, the authors introduce PUMBA: a method which combined three existing solutions (progressive unmasking, latent representations passed through steps, and backpropagation through time). Through careful analysis and ablations, the authors show how these choices interact and complement each other in different settings and under different decoding strategies, developing guidelines for more holistic and efficient design of trajectory-aware MDMs.

Critical strengths
  • The paper carefully ablates different aspects and design choices around trajectory-aware training, showing that the different methods proposed in this context complement each other effectively.
  • I appreciate that in the experiments the authors consider different decoding paradigms (e.g. block-wise decoding vs full-canvas decoding) and training regimes (training from scratch on TinyGSM vs finetuning LLaDA).
  • I also appreciate that the authors attempt not only to show what works best, but also explain why (e.g. section 3.1 and appendix D)
Critical weaknesses
  1. As the focus of the paper is quite diffused, spanning three different aspects of the trajectory-aware training, at points I found the paper difficult to follow, with not sufficient amount of details provided in the main text, and the results in the appendix also being quite densely presented. For example:
    1. the completion loss in section 3.1 (as well as the results in Figure 2) is quite non-standard, and as such I believe it could be described and justified in more details. Why are the experiments tracking only select 16 sequences? What is the final performance of the model trained with the different uu values, evaluated e.g. on GSM8k? If the completion loss returns to the value of 1.0 after the example exits the batch, is this effect even detrimental?
    2. I also believe that the results in Figure 4 could be described in more details. In the plots, what do the dots correspond to? How are the NFEs measured, and why are NFEs of SFT variants and MDM different? If the Post-SFT runs start from the SFT checkpoints marked in bold, why are some of the performance values below the performance of the best performing SFT-checkpoint. These ambiguities make it difficult to interpret claims such as “Post-SFT MDM stays close to its starting checkpoint” (l.367). While I appreciate that the authors wanted to evaluate both the in-distribution (IFEval) and out-of-distribution performance (GSM8k, MBPP) performance of the training methods, I think it might be more clear to plot these separately, rather than averaging the scores. Finally, it would also be great to see some error bars.
  2. Secondly, although the authors provide a lot of different insights into the workings and failure modes of the different trajectory-aware training methods, I think the paper would have greater value if some of these insights could be turned into potential improvements. For example, a significant part of the paper is spent analysing the effect of uu used during training. While the authors identify that using too small of a uu could be detrimental, even though it aligns the training and inference more, they do not attempt to find solutions which would allow to obtain benefits of small uu values during training while avoiding the negative effects. For example, Figure 3 suggests that the discrepancy between train and inference distribution is particularly high for t≈0.5t \approx 0.5 and I would be curious to see whether this insight could be used to propose more principled methods for deciding which parts of the unmasking trajectory should be included in the training process. Overall, I think the value of the paper could be further improved if some of the insights were more actionable.
Questions for authors
  • It seems to me that progressive unmasking can be might not be particularly beneficial early on in the training process, where the unmasking policy used by the model is not well-calibrated. Have the authors considered tracking how the discrepancy between the training and inference unmasking schedules differs at different stages of training, or using the PU only from a certain training step onwards?
8 comments

We thank the reviewer for the positive feedback, and for appreciating the breadth of our empirical validation and our effort to explain why each design choice works. We address the reviewer's points below. All changes in the revised manuscript are highlighted in green.

W1

The paper indeed spans three aspects of trajectory-aware training, namely the trajectory policy, the carry, and the gradient window. We see this breadth as necessary, since these aspects are the axes of a single design space for aligning training with inference, and covering them together is what lets us characterize this space theoretically and empirically and derive a recipe that scales to an 8B model.

To address the reviewer's concern and make the paper easier to follow, we restructured parts of it and added clarifications. In particular, we clarified the local overfitting analysis, including the completion loss, in Sec. 3.1 and Fig. 2, and restructured App. D around the runs with and without the training-time threshold, now including the GSM8K accuracy of full training runs. In Sec. 4, we also added more information to facilitate the interpretation of Fig. 4 (thresholds, NFEs, and baselines) and of the claims drawn from it. Additionally, as requested by the reviewer, we added an analysis of calibration and alignment over training (App. C.5) and error bars (App. F.5). We also extended the Limitations and Future Work section with the selection of training segments by mask discrepancy and a delayed start of PU.

W1.1

Completion loss. The completion loss reveals the prompt and the first half of the answer, and measures the cross-entropy on the rest of the answer (App. D.1). We chose it for three reasons. First, its mask is fixed and shared across methods, so any change reflects the model alone, and PU and MDM can be compared directly, unlike their training losses, which are computed on different masks. Second, it requires a single forward pass, so we can evaluate each tracked sequence after every update. Third, it is continuous, whereas generation-based accuracy (e.g., GSM8K accuracy) is binary per sequence and uninformative in short runs from scratch (see, e.g., Fig. 13(d)).

Tracked sequences. We track 12 held-out sequences, inserted into the batch in groups of four at updates 750, 1,500 and 2,250 (App. D.1). We corrected this number from 16, since the other four were never inserted and are not used in any result. Each inserted sequence starts at the beginning of a trajectory, like any other sequence entering the pool, so we can observe it along its entire training trajectory. We keep each group small relative to the batch of 256 sequences, so that the insertions barely perturb the training dynamics, while each group still reliably captures the average behavior. We insert the groups at different points of training to check whether the effect depends on when a sequence enters the batch. Since the three groups respond similarly (Fig. 18, left), Fig. 2 now shows only the group inserted at update 1,500, which simplifies interpretation and fits the flow of Sec. 3.1.

Final performance. In these runs, the relevant metric is the validation completion loss on 256 held-out sequences. For clarity, Fig. 2 now shows runs with the confidence threshold (τ=0.9\tau=0.9), as in the PUMA recipe used in the rest of Sec. 3. Local overfitting is weaker than without the threshold, but persists at u=2u=2, where PU still trails MDM (App. D.2). The runs without the threshold, in which each sequence stays in the batch for exactly JJ updates, are now in App. D.3, together with the sweep over uu (Figs. 16 and 17, Table 13). Since GSM8K accuracy is still close to zero at the end of such short runs (see, e.g., Fig. 13(d)), we additionally report the best GSM8K accuracy (%) of full training runs (~1M steps) with the threshold across training uu (Table 12).

Training uu1211193858115MDM
Inference u=2u=26.721.736.838.842.039.641.034.8
Inference u=3u=35.820.537.540.844.041.741.936.6

As with the validation completion loss in the short runs (see Table 13), PU at small training uu trails MDM by a wide margin, and it outperforms MDM at larger values of training uu.

Completion loss returns to 1. Local overfitting shows up in the drop of the relative completion loss (relative to the validation set), not in its return to 1. While a sequence is in the batch, this loss falls well below 1 for small training uu (e.g., 0.54 at u=2u=2, Table 13), so the model fits that sequence much better than unseen ones. A few hundred updates after the sequence leaves, the loss returns to 1, so it is again predicted only as well as an unseen validation sequence. The JJ updates spent on it thus bring no lasting gain, while the run visits far fewer distinct sequences (3.6k at u=2u=2 vs 768k for MDM). In line with this, the validation completion loss is higher at small uu (Table 13).

In response to the questions above, we updated Sec. 3.1 and Fig. 2, and restructured App. D (D.1–D.3) accordingly.

W1.2

Dots in the plots. Each dot corresponds to one Fast-dLLM decoding threshold τ∈{0.7,0.8,0.9,1}\tau \in \lbrace 0.7, 0.8, 0.9, 1 \rbrace, connected in order of τ\tau along each curve. A lower τ\tau commits more tokens per step, so τ=1\tau=1 (one token per step) is the rightmost point of each curve. We now state this in the caption of Fig. 4.

NFEs. NFEs count the denoiser calls needed to complete a sequence up to its first EOS, averaged over prompts. They differ across models because Fast-dLLM is adaptive. At each step, it commits every position whose confidence exceeds τ\tau, so the number of steps depends on how confident the model is throughout the generation, given each prompt.

Variants. "SFT, 1.5/3 epochs" (full canvas) and "SFT, 3/6 epochs" (block diffusion) are the final checkpoints of separate MDM SFT runs. All other curves are Post-SFT runs of 4000 steps from the checkpoint in bold, i.e., 1.5 epochs in full canvas and 6 epochs in block diffusion. "MDM" denotes the Post-SFT MDM control. The caption of Fig. 4 now states this as well.

Points below the best SFT checkpoint. Each curve spans the four decoding thresholds, so the comparison should be made in terms of the Pareto frontier of score (higher is better) against NFEs (lower is better). While a few points of some PUMBA curves lie below the curves of the baselines (the SFT checkpoints and Post-SFT MDM), the PUMBA curves typically Pareto-dominate them over most of their τ\tau range (Fig. 4). Finally, by "Post-SFT MDM stays close to its starting checkpoint", we refer to the best score (61.2 vs 60.9), which we now state explicitly in Sec. 4.1.

Individual benchmarks. Per-benchmark results for the runs of Fig. 4 are in App. F, in Fig. 22 for full canvas and Fig. 27 for block diffusion. Individual benchmarks vary more across thresholds, so the main text reports, at each τ\tau, their average (the score), which reduces this variance.

Error bars. We added error bars in Fig. 28 (App. F.5), as 95% confidence intervals on the score difference between each PUMBA variant and the baselines at matched NFE. We keep them out of Fig. 4, which is already dense, to preserve readability.

We integrated these clarifications into Sec. 4, and added App. F.5.

Reviewer_RodCOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies trajectory-aware training for masked diffusion LMs and combines three existing ideas into one recipe, PUMBA:

  1. Progressive unmasking: training on the masking patterns produced by the inference policy. Commits are teacher-forced, so only the positions are aligned, not the content.
  2. A continuous carry: the hidden state is passed from step to step.
  3. BPTT: gradients run across a window of W denoising steps.

In a controlled 125M study, the authors show that exact train–inference alignment fails because of local overfitting, that the carry beats discrete gradient estimators, and that accuracy grows with W, reaching a same-size AR model. A proposition bounds the gap to the best sampler by a term that shrinks with W and vanishes when W = J, the number of decoding steps, i.e., when the window covers the whole trajectory. The theory assumes matched training and inference rules, which the large-scale recipe relaxes (the 0.9-confidence commits in training, Fast-dLLM decoding at inference).

As a short post-SFT stage for LLaDA-8B, PUMBA gains about 2–3 points and needs up to 22–26% fewer forward passes at matched score.

Critical strengths
  1. A careful, step-by-step design study.

Each component is tested on its own before they’re combined, with compute-matched controls. Several results are valuable:

  • accuracy peaks at a small number of tokens per step (u = 2–4).
  • local overfitting explains why training at that small u fails, which gives a clear alignment–diversity trade-off.
  • progressive unmasking keeps training masks close to inference masks (the mask-discrepancy metric).
  • the student-forcing analysis (C.3) shows why naive scheduled sampling fails.
  1. A simple recipe that holds at scale, with honest theory.

The carry is easy to add and clearly beats discrete gradient estimators. The gains carry over from 125M to LLaDA-8B in both full-canvas and block decoding. The theory is sound under its assumptions.

Critical weaknesses
  1. The gap between theory and practice is not stated in the main text.

The theory assumes training and inference use the same reveal rule, but the large-scale recipe departs from it: training adds extra commits above confidence 0.9, and inference uses the Fast-dLLM threshold over several values of τ\tau rather than the training policy. Appendix E discusses the scope of the bound, but a reader of section 4 does not see these deviations.

  1. Evaluated samplers are narrower than they could be.
  • The intro and related work do not position PUMBA against dLLMs that update or correct their own decisions using the model’s own predictions, such as ProSeCo, alongside DiffusionGemma and RCD. This line of work tackles the content side of the train–inference gap. In addition, since the carry is updated at every step, a student-forcing variant in that spirit seems like a natural ablation or future direction.

  • At inference, the trained models are evaluated only with top-uu and the Fast-dLLM confidence threshold. Since training is aligned to a specific reveal policy, other samplers could be tested on the trained models, such as EB-Sampler, DUS, PUNT, KLASS, and DAPD to show whether the gains hold beyond the policies used in the paper.

Questions for authors

Questions

  1. Theory vs. practice. Can you state in the main text (sec.4) which choices deviate from the theory, such as the 0.9 commits and Fast-dLLM decoding? If possible, test their effect, e.g., train with the Fast-dLLM policy.

  2. Other samplers. Do the gains hold with EB-Sampler, DUS, PUNT, KLASS or DAPD?

  3. Memory and time. How do training memory and wall-clock time grow with WW? Numbers per Post-SFT configuration, against the extra SFT epochs, would help.

Suggestions

  • Position PUMBA against self-correcting dLLMs (ProSeCo, DiffusionGemma, RCD), and consider a student-forcing ablation.
  • Move Algorithm 1 to the main text.
  • Simplify Fig. 4: one WW in the main figure, with the WW ablation later or in the appendix.
  • Add a short appendix section comparing the discrete-bridge estimators (REINFORCE, Gumbel-softmax and straight-through), including how each is implemented, to make the comparison with the carry (Table 1) clearer.
5 comments

We thank the reviewer for the positive review of our work, and for highlighting the careful design of our study and its validation at scale. We address the reviewer's comments and questions below. All changes in the revised manuscript are highlighted in green.

W1 and Q1

We now list these deviations in a new paragraph at the end of Sec. 4.3. Our training rule is the Fast-dLLM policy at τ=0.9\tau=0.9, except that each step unmasks at least uu tokens, regardless of their confidence. The recipe therefore deviates from Prop. 1 in three ways: we train with u>1u>1, decode with thresholds other than τ=0.9\tau=0.9, and, in full canvas, decode in blocks of 32 tokens that training does not use (for block diffusion, there is no mismatch, as the block of 32 is present in both training and inference). Training with the Fast-dLLM policy, as the reviewer suggests, amounts to u=1u=1. In block diffusion, this lags behind larger uu (App. F.4, Fig. 26), and, in full canvas, it collapses because of local overfitting (Fig. 34). Even so, BPTT improves performance at 8B (Sec. 4.1 and 4.2), which suggests that the prediction of Prop. 1 holds beyond its exactly aligned setting.

W2.1 and S1

Self-correcting models address the “token content” side of the train–inference gap, by revising committed tokens either through corrective steps in MDMs (ProSeCo) or through uniform-noise diffusion (DiffusionGemma), whereas our recipe aligns which states training visits. PUMBA and self-correcting models are thus complementary. We added them to Secs. 5 and 6, and note that extending our training to their trajectories is a natural next step. Additionally, DiffusionGemma is already discussed in App. G, as it employs a carry mechanism close to ours. Regarding RCD, it does not revise committed tokens, so we see it as state propagation rather than self-correction. It passes the probability-weighted embedding of its predictions, which during training comes from a frozen reference model instead of from the model itself. It is thus another carry configuration within our framework, and training it with PUMBA is a natural extension. We discuss it accordingly in Sec. 5 and App. G.

Regarding student forcing, App. C.3 tests scheduled sampling on TinyGSM. However, no schedule improves on teacher forcing, as a wrong sampled token cannot be revised and leaves the context inconsistent with the targets. Self-correcting samplers could remove this obstacle, which makes them a natural setting for a student-forcing variant of our recipe. We added this perspective to our Future Work section.

W2.2 and Q2

To further test whether the gains extend beyond the training policy, we decode three checkpoints per setting with the suggested samplers, without retraining: the SFT checkpoint from which Post-SFT starts, Post-SFT MDM, and the PUMBA configuration with the highest score (u=32u=32, W=8W=8 in full canvas; u=16u=16, W=2W=2 in block diffusion). For each sampler, we only sweep the hyperparameter that sets how many positions are committed per step. We keep the others at the values used for LLaDA in the corresponding paper or official code, without tuning them on our models (App. F.10, Table 18). In full canvas, PUMBA reaches the highest score with every sampler (Fig. 37). In block diffusion, it is clearly above both baselines with Fast-dLLM, EB-Sampler, and KLASS, slightly above or on par with them with DUS and PUNT, and on par with DAPD. Therefore, although PUMBA is designed to align training with a specific policy, these results suggest that its gains are not limited to that policy.

(Two data points, for Post-SFT MDM and PUMBA with PUNT in full canvas, are still running; we will add them to the manuscript and notify the reviewer once they finish.)

Additionally, one could expect that training on the trajectories of a single policy narrows the learned distribution. Interestingly, we observe the opposite on TinyGSM: PU, the carry, and BPTT all improve pass@kk (App. C.1, Fig. 9), a metric used as a proxy for the diversity of the solutions a model can produce [1].

Q3

We added these measurements for every Post-SFT configuration and SFT control to App. B.5 (Tables 9–11). Importantly, only the window WW adds training cost, as PU and the carry without BPTT train in about the same time and with about the same peak memory as MDM. Specifically, each doubling of WW multiplies the wall-clock time by about 1.8–1.9 (Table 9). With activation checkpointing, the peak memory relative to MDM grows from 1.5×1.5\times at W=2W=2 to 2.3×2.3\times at W=8W=8 in full canvas, and from 1.6×1.6\times at W=2W=2 to 2.1×2.1\times at W=4W=4 in block diffusion (Table 10). Notably, PUMBA also compares favorably with additional SFT epochs. In full canvas, W=4W=4 improves three times as much as 1.5 additional epochs (2.1 vs. 0.7 points), while requiring less time (7.7 vs. 9.6 h). Similarly, in block diffusion, W=2W=2 gains more than three additional epochs (2.2 vs. 1.4 points) in about a quarter of the time (5.6 vs. 21.4 h).

Finally, at inference, the carry adds at most 0.3% to the time of a forward pass and 0.91% to the peak memory (Table 11).

S2 and S3

We thank the reviewer for these suggestions. Since other reviewers' discussions may also need main-text space, we will settle the main-text layout once the discussion has converged.

S4

We added a paragraph at the start of App. C.2 that describes how the discrete bridges are implemented and how the gradients flow through them, in contrast with the carry.

We thank the reviewer again for their constructive feedback, which helped us clarify the scope of our theory and the positioning of PUMBA. We remain available to address any further questions.


[1] - The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models, Ni et al., ICML 2026

W2.2 and Q2 (cont.)

The two remaining PUNT data points in full canvas (Post-SFT MDM and PUMBA) have finished, and we updated Fig. 37 (App. F.10) accordingly. The conclusions are unchanged (PUMBA reaches the highest score with every sampler).

We thank the reviewer again and we remain available for any further questions.

Reviewer_hLPdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how masked diffusion language models can be trained to better reflect their iterative generation process. The proposed method, PUMBA, combines policy-guided progressive unmasking, hidden-state propagation between denoising steps, and joint optimization through BPTT. A particularly interesting finding is that matching the inference trajectory too closely can cause local overfitting, as small unmasking counts lead to repeated training on the same samples. The authors address this by using larger unmasking counts during training while retaining some alignment with inference. Controlled experiments on a 125M model examine the contribution of each component, showing consistent gains with longer BPTT windows and eventually reaching accuracy comparable to an autoregressive model. In the 8B scale experiments, the proposed method improves the performance vs. NFE trade-off in both full and block-wise generation.

Critical strengths

S1. Trade-off between alignment and learnability

The paper shows that closer alignment between training and inference does not necessarily improve performance. Through an analysis of training dynamics, the authors identify local overfitting caused by repeatedly training on the same samples when the unmasking count is small. They also show that progressive unmasking reduces the discrepancy between training and inference mask distributions, even with a much larger unmasking count during training.

S2. Systematic ablations and scaled validation

The paper carefully examines the contributions of PU, carry, and BPTT through controlled experiments. On the 125M model, comparisons matched for the number of denoiser forward and backward passes show that longer BPTT windows improve accuracy,. The method also improves the performance vs. NFE trade-off in 8B model scale, for both full-canvas and block-wise generation.

Critical weaknesses

W1. Compute-matched comparisons

PUMBA and MDM are compared using the same 4k optimizer updates, in 8B experiments, despite substantial differences in their actual training costs. PUMBA constructs training trajectories sequentially based on the model's predictions and jointly optimizes multiple denoising steps through BPTT, increasing both computation and memory requirements as the window size grows. Although compute-matched comparisons are provided for the smaller model, no corresponding comparison is reported for LLaDA-8B. It is therefore unclear whether the reported performance gains justify the additional training cost. Moreover, inference efficiency is evaluated primarily in terms of NFE, which may be affected by the overhead of propagating hidden states between steps. A comparison under matched training compute, along with measured inference latency, would be necessary to establish the practical advantages of the method.

W2. Novelty relative to existing methods

The central components of the proposed method closely overlap with the existing approaches, including PUMA, Loopholing, and Relay. The underlying training procedures therefore appear to be largely similar. Instesd, the main contributions seem to lie in its systematic analysis of train–inference alignment, its theoretical and empirical investigation of the window size, and its extension to an 8B model. While these are useful contributions, the algorithmic novelty as a new training framework appears limited; the authors should more clearly distinguish its original methodological contributions from the existing approaches, especially Relay.

Questions for authors
  • Related work section discusses MDPO but omits wd1++ [1], DiSPO [2], and SAPO [3], which also incorporate intermediate denoising steps into optimization in diffusion language models. Could the authors clarify how PUMBA differs from these approaches?

  • Training still differs from inference due to teacher-forced commitments and different unmasking counts. How much do these discrepancies affect performance, and how should the BPTT theoretical analysis be interpreted under this mismatch?

[1] Tang et al., "wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models"

[2] Oba et al., "Diffusion-State Policy Optimization for Masked Diffusion Language Models".

[3] Xie et al., "Step-Aware Policy Optimization for Reasoning in Diffusion Large Language Models".

7 comments

We thank the reviewer for their review and for the positive feedback on the analysis of the trade-off between training-inference alignment and learnability, and the breadth of our experimental validation. We address the reviewer’s concerns below. All changes in the revised manuscript are highlighted in green.

W1

Compute-matched comparison at 8B scale. We added this experiment (App. F.11). Since each update with W=8W=8 costs about 8×8 \times an MDM update (Table 9), we continued Post-SFT MDM for 32k steps instead of 4k. It reaches a best score of 61.0, no higher than with 4k steps (61.2), consistent with MDM training having nearly saturated our SFT data (Sec. 4.1). In contrast, every PU-based run reaches a higher best score, including PU alone, whose updates cost the same as those of MDM (Table 9).

Importantly, from a practical perspective, the training overhead of PUMBA is paid once, whereas its NFE savings apply to every generated sequence, a trade-off that favors extra training compute when inference demand is large [1].

Inference latency. The window WW only affects training: at inference, the carry is passed from step to step in the same way for every WW, so all PUMBA models have the same inference cost. The carry adds no forward passes, only a single L×dL \times d tensor per sequence. Measured on one H100, it adds at most 0.3% to the time of a forward pass and 0.91% to the peak memory (Table 11). Consequently, the NFE savings at matched score (up to 22% in full canvas and 26% in block diffusion) translate almost one-to-one into wall-clock speedups.

W2

PUMBA is not a combination of PUMA, Loopholing, and Relay. Our contribution is to formalize a broader design space for aligning training with inference (trajectory policy, carry, gradient window), characterize it theoretically and empirically, and show how to choose within it. PUMA, Loopholing, and Relay emerge as particular configurations within this broader space, but leave important parts of it unexplored, such as windows longer than two steps and the choice of the training uu under local overfitting. We detail these differences below.

Framework. The three axes follow from two gaps between training and inference (Sec. 1, App. G). The trajectory policy determines which masked states the denoiser sees, while the carry and the BPTT window WW determine what information it receives from the previous step and how far gradients flow back. PUMA follows the first axis, but trains each step independently of the others. Loopholing passes a carry between steps, but at training time obtains it from an extra model forward on the same partially masked sequence rather than from the preceding step. This extra pass receives no gradient. In PUMBA, the carry comes from the preceding step of the training trajectory, and every pass receives gradients, both from its own loss and, within the BPTT window, through the carry from the steps that follow. Consequently, the carry is trained along trajectories closer to those encountered at inference.

Theory. Our analysis shows why the carry and the BPTT window help (Prop. 1, App. E). Under teacher forcing, the actual gradient through the committed tokens is zero, so discrete gradient estimators can only replace it with a surrogate. In contrast, the carry provides the actual gradient, reaching at most W−1W-1 earlier steps. We demonstrate empirically that the carry outperforms all discrete gradient estimators (Table 1). We further show that the gap to the best achievable sampler shrinks as the window grows, and vanishes when the window covers the whole trajectory, motivating our approach of increasing the BPTT window in practice. To the best of our knowledge, no prior work on masked diffusion models provides such results.

Local overfitting. Importantly, we show that training cannot be aligned with inference at the small uu used for decoding, because of the training dynamics it induces. In particular, at small training uu, each sequence stays in the batch for many consecutive steps, so the model overfits to it locally and loses this gain once the sequence leaves (Sec. 3.1, Fig. 2, App. D). PUMA motivates starting training at a large reveal count by the noisy policy early in training and by the redundant signal of long chains, treating small counts as sample-inefficient. We show that they are in fact harmful, as the model overfits each sequence locally, and this persists at 8B scale (Sec. 4.3). Nevertheless, we also demonstrate why progressive unmasking can still be effective: our mask discrepancy metric shows that training at a larger uu still brings the training masks closer to those seen at inference than MDM does (Fig. 3). We can therefore train at a larger uu than that we decode with, while retaining part of the alignment.

Scale and scope. Relay corresponds to a single configuration of our design space, a two-step window in block diffusion. It is used in a short, 200-step adaptation of a 1.5B model to domain-specific data (code/math). Our analysis instead shows how to choose each component: local overfitting determines the training uu, and both the theory and our smaller-scale results (TinyGSM, Table 2) motivate longer windows. This is what allows us to design a general recipe for instruction post-training of an 8B model, combining progressive unmasking, the carry, and BPTT over windows of up to W=8W=8. The recipe handles responses of variable length and applies to both full-canvas and block diffusion (Sec. 4, App. B.2).

We revised Sec. 5 and App. G to make these distinctions more explicit.

Q1

Like MDPO, these methods use task rewards and incorporate intermediate denoising states into policy optimization. PUMBA differs from them in two respects. First, it does not require a reward: it is a supervised fine-tuning method that maximizes the likelihood of ground-truth tokens along trajectories generated by the inference policy. Second, PUMBA propagates credit to earlier denoising steps through the differentiable carry and BPTT, whereas these methods use rewards obtained from completions associated with intermediate states to guide optimization. The evaluation settings also differ: these methods are evaluated on tasks with verifiable answers, such as math and planning, while PUMBA targets general instruction post-training. Finally, these approaches are complementary, as a PUMBA checkpoint could serve as a starting point for reward-based optimization.

We thank the reviewer for pointing us to these works. We added them to the updated manuscript and clarified the distinction in Sec. 5.

Q2

Teacher-forced commitments. To quantify the effect of teacher forcing, we replace it with scheduled sampling (partial student forcing), which progressively feeds the model's own predictions at the revealed positions (App. C.3, Fig. 12). None of the schedules we tried improves on teacher forcing. In fact, accuracy decreases as the share of predicted tokens grows, and falls below MDM once about 10% of the revealed tokens are predicted. Intuitively, since committed positions are never revisited, a wrong prediction remains in the context, while the loss still targets the ground truth at the remaining positions. Interestingly, the carry avoids this failure while still exposing each step to the model's own computation, which makes it a weak form of student forcing (Sec. 3.2).

Different unmasking counts. Ideally, we would train at the uu used at inference, which in practice is small. However, we show that a small training uu harms the training dynamics and triggers local overfitting, which degrades performance (Sec. 3.1 and 4.3). At the other extreme, a large training uu reduces trajectories to a few coarse steps. Therefore, performance peaks in between (see inverted-U shapes in Fig. 34). Importantly, even with a training uu larger than the inference uu, training masks stay closer to inference ones than under MDM (Fig. 3). This is consistent with PU improving over MDM when its training uu is appropriately chosen (Table 2).

Interpreting the theory. Prop. 1 has two parts (App. E, "Scope of the bound"). Part (i), that a step's loss reaches earlier steps only through the carry and at most W−1W-1 of them, holds along any teacher-forced trajectory and does not depend on the training uu, so it is not compromised by any of the mentioned mismatches. Part (ii) bounds the sampler that decodes with the training reveal rule (training uu = inference uu). We read it as a guarantee for an idealized, aligned setting, and its prediction that the gap shrinks as WW grows as a hypothesis for ours, which the experiments support: e.g., accuracy at u=2u=2 increases with WW (44.4, 48.7, 52.6, 55.6% for W=1,2,4,8W=1,2,4,8; Table 2), and the ordering holds at every decoding budget (Fig. 7).

We thank the reviewer again for their feedback and hope that our responses have addressed their concerns. We remain available for any further clarification.


[1] - Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws, Sardana et al., ICML 2024.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We are grateful for the detailed feedback and questions. Reviewers found the analysis of the trade-off between train–inference alignment and learnability insightful (hLPd, RodC), the ablations careful and systematic (hLPd, RodC, T4Ck), and the recipe simple and effective at the 8B scale (hLPd, RodC).

Below, we summarize the main updates, which are highlighted in green in the revised PDF.

  • Compute-matched comparison at 8B scale (hLPd). We demonstrate that continuing Post-SFT MDM for 8$\times$ more steps does not improve its best score, whereas every PUMBA run scores higher (App. F.11).
  • Training and inference cost (hLPd, RodC). Only the BPTT window adds noticeable training cost, and this compute is better spent on BPTT than on longer MDM training (Table 2 for TinyGSM; see above for 8B). At inference, we measured that the carry adds at most 0.3% to the time of a forward pass, so the NFE savings at matched score (up to 22% in full canvas and 26% in block diffusion) translate almost directly into wall-clock speedups (App. B.5).
  • Other samplers (RodC). We decoded with five additional samplers (EB-Sampler, DUS, PUNT, KLASS and DAPD). In full canvas, PUMBA reaches the highest score with every sampler, and in block diffusion it is above or on par with the baselines (App. F.10).
  • Positioning (hLPd, RodC). We clarified that PUMA, Loopholing and Relay are particular configurations of the design space we formalize, which leave out, for example, longer BPTT windows, their theoretical motivation, and local overfitting, all of which we analyze. We also positioned PUMBA with respect to reward-based (wd1++, DiSPO, SAPO) and self-correcting (ProSeCo, DiffusionGemma, RCD) methods (Sec. 5, App. G).
  • Theory and practice (hLPd, RodC). We clarified where the 8B recipe departs from the assumptions of our theoretical results (Sec. 4.3). Its prediction still holds empirically, since accuracy generally grows with the window (Table 2 and Fig. 4). Additionally, replacing teacher forcing with student forcing (scheduled sampling) does not improve accuracy (App. C.3).
  • Clarity (RodC, T4Ck). We further clarified the local overfitting analysis (Sec. 3.1, App. D) and the interpretation of the results at the 8B scale (Sec. 4). We also added a more detailed description of how the discrete-bridge estimators are implemented (App. C.2).
  • Additional analyses (T4Ck). We added error bars (App. F.5) and an analysis of calibration over training (App. C.5), which shows that the denoiser trained with PU is miscalibrated only during the first 0.1% of training. We also highlight our mitigation strategies for small training uu at 8B scale, which partially recover performance (App. F.7.1).
  • Future work (RodC, T4Ck). Following suggestions of the reviewers, we added a discussion of student forcing combined with self-correcting samplers, the selection of training segments by mask discrepancy, and a delayed start of progressive unmasking.

We thank the reviewers for these suggestions, which helped improve the paper. We would be very grateful if the reviewers could share any remaining questions or concerns, and we would be happy to discuss them before the discussion period ends.

Best,

Authors

Dear Area Chair and Reviewers,

To further illustrate the capabilities of PUMBA, we have added an additional experiment.

Dream-7B. We repeated the full-canvas experiment on Dream-7B. PUMBA outperforms MDM trained with matched compute at every threshold (App. F.12, Fig. 40).

Additionally, we extended the compute-matched comparison to block diffusion. PUMBA shows a similar dominance to the full-canvas setting (App. F.11, Fig. 39).

(Both additions are highlighted in green in the revised PDF.)

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.