- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of decoding masked diffusion language models, aiming to find a better balance between premature stopping that leads to immature answers and excessive decoding that may degrade performance. The authors propose a decoding method called Trajectory-MBR, which combines a sliding window with trajectory-likelihood-weighted minimum Bayes risk. The theoretical analysis centers on trajectory score approximation error and conditional regret bounds for response selection. Experiments on language and multimodal benchmarks provide empirical support for the method, showing improvements in performance and efficiency.
Critical strengths
The algorithm brings clear improvements in computational efficiency and provides additional design flexibility for future work by allowing the output to be selected from historical checkpoints rather than always returning the latest one before early stopping. In the reported settings, the method also achieves competitive answer quality, though its advantage over baselines such as Sparse-dLLM appears relatively small in some settings.
Critical weaknesses
One major weakness of the paper is that it does not provide sufficient insights into the underlying mechanism. The theory part helps little here. Theorem 1 mainly characterizes the approximation error of the trajectory score. While Theorem 2 and Proposition 1 rely only on general assumptions about the utility function, their bounds do not explain what kind of utility or weighting plays a key role in the algorithm. Although the examples in Section 6.2 illustrate when temporal agreement helps, this phenomenon is already well established rather than being the main contribution of this paper. This may leave readers unclear about why this particular design works well, especially since the trajectory scoring scheme is, in my opinion, one of the unique contributions of the paper. Also, the experiments do not clearly separate the gains from different components of the algorithm. I think a comparison with simpler weighting schemes under the same stopping framework would help clarify the contribution of the trajectory score.
Questions for authors
-
Could the authors provide more insights into the mechanism behind the algorithm? In particular, why should the proposed trajectory-likelihood weighting work better than simpler weighting schemes? I think this would help clarify the unique contribution of the method.
-
Could the authors compare the proposed method with a fixed-budget early-exit baseline under the same inference budget? For example, stopping the base decoder after a fixed number of steps and returning its current prediction. I think this would help clarify how much of the efficiency and quality gains come from adaptive stopping rather than simply reducing the number of decoding steps.
-
How often does the method select a response different from the final response of full decoding? Under what circumstances does this selection improve or degrade answer quality? I think such an analysis would help readers better understand the situations where the algorithm is most effective and its limitations.
10 comments
Comparison under the same stopping framework
In Table 4 and Table 5 in Section 6.2 of the revised manuscript, we fix the UltraLLaDA settings of Table 1 and change only the weighting and the selection rule. The evaluation uses 200 LSHT examples and 500 LCC examples. Default uses the same , utility, and , and Tuned adjusts only so that the speedup approaches that of Trajectory-MBR.
| Method | LSHT Score | LSHT Speedup | LCC Score | LCC Speedup |
|---|---|---|---|---|
| Trajectory-MBR | 42.50 | 12.94× | 68.30 | 1.21× |
| Uniform weights, Default | 41.50 | 5.13× | 68.07 | 1.15× |
| Uniform weights, Tuned | 42.50 | 9.57× | 68.03 | 1.21× |
| Completion confidence, Default | 41.50 | 5.13× | 68.07 | 1.15× |
| Completion confidence, Tuned | 41.00 | 12.80× | 68.03 | 1.21× |
| Current candidate, same stopping point | 42.00 | 12.94× | 68.18 | 1.21× |
| Deduplicated Trajectory-MBR | 42.50 | 12.92× | 68.28 | 1.21× |
| Best simple rule at the same speedup, Table 5 | 41.50 | 12.94× | 68.08 | 1.21× |
Trajectory weights reach the same quality much faster. Under the same settings, switching to a simpler weighting reduces the LSHT speedup from 12.94× to 5.13×, so Trajectory-MBR is 2.5 times faster. Trajectory-MBR also scores higher on both tasks. These results appear in the Uniform weights and Completion confidence columns of Table 4.
Simpler weightings do not catch up even at a matched speed. When is adjusted to match the speedup, both simpler weightings reach 68.03 on LCC, which is 0.27 points below Trajectory-MBR, and completion confidence is 1.50 points lower on LSHT. Uniform weights need 35% more denoiser evaluations to reach the same 42.50 on LSHT.
Selecting earlier candidates raises the score at no extra cost. Stopping at the same point and returning the current candidate gives 42.00 and 68.18, whereas also selecting among the earlier candidates in the window with trajectory weights gives 42.50 and 68.30 with the same computation. This is the Current candidate column of Table 4.
The effect does not come from repeated support for identical responses. Keeping only one copy of each identical string and recomputing changes the scores by at most 0.02 points and leaves the speed essentially unchanged. This is the Deduplicated column of Table 4.
No simple rule at the same speed matches Trajectory-MBR. In Table 5, confidence stopping, majority voting, and confidence argmax stay at or below 41.50 on LSHT and 68.08 on LCC even when the speedup is matched.
In summary, Theorem 1 supports that the trajectory score approximates the likelihood, and Table 4 and Table 5 show that this weighting in practice stops earlier while selecting better candidates.
Q2. Fixed-budget early exit at the same inference budget
Following the reviewer's suggestion, we compared against a baseline that stops the base sampler at a fixed budget and returns the current candidate at that point. We report both a comparison with the budget matched exactly to Trajectory-MBR and a comparison across all 13 tasks at several budgets.
Comparison at the same budget. The Matched budget columns of Table 5 in Section 6.2 of the revised manuscript stop after 5 evaluations on LSHT and 53 on LCC, matching the average cost of Trajectory-MBR. Returning the current candidate at the same budget gives 41.00 on LSHT and 68.05 on LCC, and selecting by majority voting or confidence argmax does not exceed 41.00 and 68.05. Trajectory-MBR obtains 42.50 and 68.30 at the same budget.
Comparison across all 13 tasks. The table below compares fixed budgets that correspond to 2.00×, 2.92×, and 4.00× speedups on the 13 LongBench tasks for both backbones.
Table R4. Comparison of fixed-budget early exit and Trajectory-MBR on UltraLLaDA and Dream-7B-Instruct. † denotes results reported by Long et al. (2026). Fixed-budget early exit has a lower average than Trajectory-MBR even with similar or more computation, which shows that the gain does not come from simply reducing the number of steps.
| UltraLLaDA | Qasper | MF-en | HotpotQA | 2WikiMQA | MuSiQue | GovReport | QMSum | TREC | TriviaQA | LSHT | PRe | LCC | RB-P | Avg. Score | Avg. Speedup |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full decoding† | 19.14 | 25.87 | 16.27 | 18.00 | 12.08 | 32.83 | 22.48 | 80.00 | 91.58 | 41.00 | 96.75 | 68.23 | 59.50 | 44.90 | 1.00× |
| Fixed budget, 2.00× | 20.86 | 30.07 | 16.82 | 17.71 | 12.50 | 29.98 | 22.31 | 80.00 | 91.58 | 41.00 | 93.28 | 68.02 | 59.23 | 44.87 | 2.00× |
| Fixed budget, 2.92× | 23.37 | 34.28 | 18.65 | 19.69 | 14.14 | 27.37 | 22.53 | 80.00 | 91.58 | 41.00 | 93.16 | 67.98 | 58.92 | 45.59 | 2.92× |
| Fixed budget, 4.00× | 24.58 | 36.26 | 25.58 | 25.16 | 18.69 | 25.39 | 22.65 | 80.00 | 90.93 | 41.00 | 85.12 | 67.80 | 58.35 | 46.27 | 4.00× |
| Trajectory-MBR | 23.41 | 34.25 | 28.03 | 27.76 | 20.30 | 32.88 | 22.91 | 80.00 | 91.58 | 42.50 | 97.33 | 68.30 | 59.50 | 48.37 | 3.76× |
| Dream-7B-Instruct | Qasper | MF-en | HotpotQA | 2WikiMQA | MuSiQue | GovReport | QMSum | TREC | TriviaQA | LSHT | PRe | LCC | RB-P | Avg. Score | Avg. Speedup |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full decoding† | 35.58 | 40.49 | 41.59 | 42.10 | 23.36 | 23.51 | 19.66 | 71.50 | 87.34 | 15.75 | 32.50 | 63.79 | 62.23 | 43.03 | 1.00× |
| Fixed budget, 2.00× | 34.67 | 38.98 | 34.85 | 33.84 | 15.06 | 19.92 | 19.79 | 72.66 | 86.76 | 19.44 | 28.76 | 63.31 | 50.65 | 39.90 | 2.00× |
| Fixed budget, 2.92× | 33.68 | 38.54 | 33.22 | 32.76 | 15.35 | 19.16 | 19.66 | 72.66 | 86.80 | 19.44 | 29.65 | 62.84 | 49.70 | 39.50 | 2.92× |
| Fixed budget, 4.00× | 32.16 | 37.73 | 27.75 | 30.59 | 13.56 | 18.47 | 19.55 | 72.66 | 85.27 | 19.44 | 29.62 | 62.29 | 47.58 | 38.21 | 4.00× |
| Trajectory-MBR | 38.33 | 43.36 | 41.82 | 42.46 | 23.90 | 21.87 | 20.05 | 72.50 | 89.31 | 18.00 | 40.50 | 63.86 | 63.38 | 44.56 | 1.94× |
Adaptive stopping scores higher at a similar amount of computation. On Dream-7B-Instruct, fixed-budget early exit averages 39.90 at 2.00×, which is 3.13 points below full decoding. Trajectory-MBR reaches 44.56 at a nearly identical 1.94×, which is 1.53 points above full decoding. On UltraLLaDA, fixed-budget early exit reaches only 45.59 even at 2.92×, where it uses more computation than Trajectory-MBR, and this is below the 48.37 of Trajectory-MBR at 3.76×.
A fixed budget collapses on certain tasks, whereas Trajectory-MBR does not. Fixed-budget early exit drops from 96.75 to 85.12 on PassageRetrieval-en with UltraLLaDA and from 62.23 to 50.65 on RepoBench-P with Dream-7B-Instruct, because different examples need different numbers of steps. In contrast, Trajectory-MBR matches or exceeds full decoding on all 13 tasks with UltraLLaDA and on 12 of the 13 tasks with Dream-7B-Instruct. Because it stops only when the candidates agree sufficiently, it saves computation on examples where early stopping is safe and continues decoding on the others.
Exploiting early responses requires deciding the stopping point for each example. On UltraLLaDA, fixed-budget early exit alone already raises the QA scores. On this backbone, responses at early checkpoints are often better than the terminal response, which is consistent with the per-example analysis in Q3. However, a fixed budget falls short of Trajectory-MBR on average because of the losses on other tasks. This benefit is obtained without such losses only when the stopping point is decided for each example.
The same-budget comparison was added to Table 5 in Section 6.2 of the revised manuscript, and the 13-task comparison was added to Table 11 in Appendix C.
We sincerely thank the reviewer for the careful reading and the constructive feedback. We address each point below.
We appreciate this comment. Masked diffusion language models can overwrite a good response that was already generated earlier in decoding, and work on temporal voting also studies this phenomenon. The distinguishing point of our method is that it performs this recovery without paying the cost of full decoding. Existing methods obtain only one of the two benefits. Early-commitment methods reduce computation but cannot recover an earlier response, while temporal voting recovers earlier responses but still pays the full decoding cost.
Trajectory-MBR jointly decides when to stop and which response to return, so it obtains both benefits at once. When the terminal state is poor, it recovers a better earlier response and improves quality. When the terminal state is already good, it preserves quality while reducing computation through early stopping. This is consistent with the experiments and the theoretical analysis in the paper.
We agree that the existing theoretical analysis alone did not sufficiently explain the role of the weights. We summarize the rationale for the weighting design in three points, and we added ablations that compare against simpler weightings under the same stopping framework to Table 4 and Table 5 in Section 6.2 of the revised manuscript.
Rationale for the weighting design
First, the trajectory score approximates the likelihood of a candidate. Theorem 1 and Equation 9 in Section 4.2 show that the window weights approximate the likelihood weights under the stochastic ancestral counterpart. In the controlled experiment of Figure 2, the mean TV between the two sets of weights was below 0.10, and replacing the trajectory-score weights with exact likelihood weights selected the same response in 81.6% to 90.0% of the windows that contain distinct responses. The weighted risk of Equation 5 therefore places more weight on candidates that the model considers more plausible, which is the same principle by which MBMBR (Jinnai et al., 2024) improves over uniform MBR.
Second, it compares candidates from different checkpoints on a common scale. The trajectory score of Equation 2 multiplies the token probabilities at all L positions for every candidate. If only the probabilities of the current completion were multiplied, later candidates with fewer masked positions would have fewer factors and would be favored. Completion confidence avoids this by averaging probabilities, but it discards the history of the tokens that are already revealed, and uniform weights do not reflect probability differences at all.
Third, the weights also determine when to stop. The stopping condition of Equation 7 uses the weighted risk of Equation 5 directly. With exact match and K=4, as on LSHT, the uniform risk changes only in steps of 0.25, so satisfying ϵ=0.05 requires all four candidates to be identical. With trajectory weights, a single low-scoring candidate that gives a different answer raises the risk only slightly, so stopping is barely delayed.