- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Broadly speaking, the paper addresses the problem of discrepancy between the training- and inference-time trajectories of masked diffusion models, where the masking configurations seen at training times are typically random, while the masking configurations seen at inference time are generated by the model’s own policy. To address this problem, the authors introduce PUMBA: a method which combined three existing solutions (progressive unmasking, latent representations passed through steps, and backpropagation through time). Through careful analysis and ablations, the authors show how these choices interact and complement each other in different settings and under different decoding strategies, developing guidelines for more holistic and efficient design of trajectory-aware MDMs.
Critical strengths
- The paper carefully ablates different aspects and design choices around trajectory-aware training, showing that the different methods proposed in this context complement each other effectively.
- I appreciate that in the experiments the authors consider different decoding paradigms (e.g. block-wise decoding vs full-canvas decoding) and training regimes (training from scratch on TinyGSM vs finetuning LLaDA).
- I also appreciate that the authors attempt not only to show what works best, but also explain why (e.g. section 3.1 and appendix D)
Critical weaknesses
- As the focus of the paper is quite diffused, spanning three different aspects of the trajectory-aware training, at points I found the paper difficult to follow, with not sufficient amount of details provided in the main text, and the results in the appendix also being quite densely presented. For example:
- the completion loss in section 3.1 (as well as the results in Figure 2) is quite non-standard, and as such I believe it could be described and justified in more details. Why are the experiments tracking only select 16 sequences? What is the final performance of the model trained with the different values, evaluated e.g. on GSM8k? If the completion loss returns to the value of 1.0 after the example exits the batch, is this effect even detrimental?
- I also believe that the results in Figure 4 could be described in more details. In the plots, what do the dots correspond to? How are the NFEs measured, and why are NFEs of SFT variants and MDM different? If the Post-SFT runs start from the SFT checkpoints marked in bold, why are some of the performance values below the performance of the best performing SFT-checkpoint. These ambiguities make it difficult to interpret claims such as “Post-SFT MDM stays close to its starting checkpoint” (l.367). While I appreciate that the authors wanted to evaluate both the in-distribution (IFEval) and out-of-distribution performance (GSM8k, MBPP) performance of the training methods, I think it might be more clear to plot these separately, rather than averaging the scores. Finally, it would also be great to see some error bars.
- Secondly, although the authors provide a lot of different insights into the workings and failure modes of the different trajectory-aware training methods, I think the paper would have greater value if some of these insights could be turned into potential improvements. For example, a significant part of the paper is spent analysing the effect of used during training. While the authors identify that using too small of a could be detrimental, even though it aligns the training and inference more, they do not attempt to find solutions which would allow to obtain benefits of small values during training while avoiding the negative effects. For example, Figure 3 suggests that the discrepancy between train and inference distribution is particularly high for and I would be curious to see whether this insight could be used to propose more principled methods for deciding which parts of the unmasking trajectory should be included in the training process. Overall, I think the value of the paper could be further improved if some of the insights were more actionable.
Questions for authors
- It seems to me that progressive unmasking can be might not be particularly beneficial early on in the training process, where the unmasking policy used by the model is not well-calibrated. Have the authors considered tracking how the discrepancy between the training and inference unmasking schedules differs at different stages of training, or using the PU only from a certain training step onwards?
8 comments
W1.1
Completion loss. The completion loss reveals the prompt and the first half of the answer, and measures the cross-entropy on the rest of the answer (App. D.1). We chose it for three reasons. First, its mask is fixed and shared across methods, so any change reflects the model alone, and PU and MDM can be compared directly, unlike their training losses, which are computed on different masks. Second, it requires a single forward pass, so we can evaluate each tracked sequence after every update. Third, it is continuous, whereas generation-based accuracy (e.g., GSM8K accuracy) is binary per sequence and uninformative in short runs from scratch (see, e.g., Fig. 13(d)).
Tracked sequences. We track 12 held-out sequences, inserted into the batch in groups of four at updates 750, 1,500 and 2,250 (App. D.1). We corrected this number from 16, since the other four were never inserted and are not used in any result. Each inserted sequence starts at the beginning of a trajectory, like any other sequence entering the pool, so we can observe it along its entire training trajectory. We keep each group small relative to the batch of 256 sequences, so that the insertions barely perturb the training dynamics, while each group still reliably captures the average behavior. We insert the groups at different points of training to check whether the effect depends on when a sequence enters the batch. Since the three groups respond similarly (Fig. 18, left), Fig. 2 now shows only the group inserted at update 1,500, which simplifies interpretation and fits the flow of Sec. 3.1.
Final performance. In these runs, the relevant metric is the validation completion loss on 256 held-out sequences. For clarity, Fig. 2 now shows runs with the confidence threshold (), as in the PUMA recipe used in the rest of Sec. 3. Local overfitting is weaker than without the threshold, but persists at , where PU still trails MDM (App. D.2). The runs without the threshold, in which each sequence stays in the batch for exactly updates, are now in App. D.3, together with the sweep over (Figs. 16 and 17, Table 13). Since GSM8K accuracy is still close to zero at the end of such short runs (see, e.g., Fig. 13(d)), we additionally report the best GSM8K accuracy (%) of full training runs (~1M steps) with the threshold across training (Table 12).
| Training | 1 | 2 | 11 | 19 | 38 | 58 | 115 | MDM |
|---|---|---|---|---|---|---|---|---|
| Inference | 6.7 | 21.7 | 36.8 | 38.8 | 42.0 | 39.6 | 41.0 | 34.8 |
| Inference | 5.8 | 20.5 | 37.5 | 40.8 | 44.0 | 41.7 | 41.9 | 36.6 |
As with the validation completion loss in the short runs (see Table 13), PU at small training trails MDM by a wide margin, and it outperforms MDM at larger values of training .
Completion loss returns to 1. Local overfitting shows up in the drop of the relative completion loss (relative to the validation set), not in its return to 1. While a sequence is in the batch, this loss falls well below 1 for small training (e.g., 0.54 at , Table 13), so the model fits that sequence much better than unseen ones. A few hundred updates after the sequence leaves, the loss returns to 1, so it is again predicted only as well as an unseen validation sequence. The updates spent on it thus bring no lasting gain, while the run visits far fewer distinct sequences (3.6k at vs 768k for MDM). In line with this, the validation completion loss is higher at small (Table 13).
In response to the questions above, we updated Sec. 3.1 and Fig. 2, and restructured App. D (D.1–D.3) accordingly.
W1.2
Dots in the plots. Each dot corresponds to one Fast-dLLM decoding threshold , connected in order of along each curve. A lower commits more tokens per step, so (one token per step) is the rightmost point of each curve. We now state this in the caption of Fig. 4.
NFEs. NFEs count the denoiser calls needed to complete a sequence up to its first EOS, averaged over prompts. They differ across models because Fast-dLLM is adaptive. At each step, it commits every position whose confidence exceeds , so the number of steps depends on how confident the model is throughout the generation, given each prompt.
Variants. "SFT, 1.5/3 epochs" (full canvas) and "SFT, 3/6 epochs" (block diffusion) are the final checkpoints of separate MDM SFT runs. All other curves are Post-SFT runs of 4000 steps from the checkpoint in bold, i.e., 1.5 epochs in full canvas and 6 epochs in block diffusion. "MDM" denotes the Post-SFT MDM control. The caption of Fig. 4 now states this as well.
Points below the best SFT checkpoint. Each curve spans the four decoding thresholds, so the comparison should be made in terms of the Pareto frontier of score (higher is better) against NFEs (lower is better). While a few points of some PUMBA curves lie below the curves of the baselines (the SFT checkpoints and Post-SFT MDM), the PUMBA curves typically Pareto-dominate them over most of their range (Fig. 4). Finally, by "Post-SFT MDM stays close to its starting checkpoint", we refer to the best score (61.2 vs 60.9), which we now state explicitly in Sec. 4.1.
Individual benchmarks. Per-benchmark results for the runs of Fig. 4 are in App. F, in Fig. 22 for full canvas and Fig. 27 for block diffusion. Individual benchmarks vary more across thresholds, so the main text reports, at each , their average (the score), which reduces this variance.
Error bars. We added error bars in Fig. 28 (App. F.5), as 95% confidence intervals on the score difference between each PUMBA variant and the baselines at matched NFE. We keep them out of Fig. 4, which is already dense, to preserve readability.
We integrated these clarifications into Sec. 4, and added App. F.5.
We thank the reviewer for the positive feedback, and for appreciating the breadth of our empirical validation and our effort to explain why each design choice works. We address the reviewer's points below. All changes in the revised manuscript are highlighted in green.
W1
The paper indeed spans three aspects of trajectory-aware training, namely the trajectory policy, the carry, and the gradient window. We see this breadth as necessary, since these aspects are the axes of a single design space for aligning training with inference, and covering them together is what lets us characterize this space theoretically and empirically and derive a recipe that scales to an 8B model.
To address the reviewer's concern and make the paper easier to follow, we restructured parts of it and added clarifications. In particular, we clarified the local overfitting analysis, including the completion loss, in Sec. 3.1 and Fig. 2, and restructured App. D around the runs with and without the training-time threshold, now including the GSM8K accuracy of full training runs. In Sec. 4, we also added more information to facilitate the interpretation of Fig. 4 (thresholds, NFEs, and baselines) and of the claims drawn from it. Additionally, as requested by the reviewer, we added an analysis of calibration and alignment over training (App. C.5) and error bars (App. F.5). We also extended the Limitations and Future Work section with the selection of training segments by mask discrepancy and a delayed start of PU.