- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DUEL, a method that computes the exact likelihood of a masked diffusion model with deterministic unmasking. Authors observe that such model defines a proper probability distribution. Since the unmasking order is fixed, the exact likelihood can be computed just by running the same rule on the true sequence. The authors compare DUEL with the ELBO, which is only a bound and assumes a uniformly random order. With open models DUEL makes the perplexity gap between MDMs and AR models smaller. DUEL also reduces the compute and also improves LLaDA-8B's multi choice accuracy. Also the authors use DUEL perplexity to rank samplers and find the ranking align with LLM judge.
Critical strengths
The core observation is correct and practical.
The method requires no training and gives exact normalized likelihood.
And also the method reflects the actual sampler whereas the ELBO doesn't.
The ablation in Table 16 of the appendix clearly mentiond about the limits of the method.
Critical weaknesses
The framing is stronger than the evidence. DUEL and the ELBO measure different distributions. With a left-to-right order, DUEL is basically the likelihood of an AR-style decoder from MDM. Table 16 shows that the exact uniform order likelihood already beats the AR model, while DUEL is worse than both. So the reduced compute gap depends on the chosen policy
The sampler-selection evidence is weak. The perfect agreement only counts pairs with a clear judge preference. These pairs come from just a few rules, and half use small models that produces bad meaningless text. There is only one LLM judge and no human check. So more concrete evaluatino is needed. DUEL also ignores the nucleus sampling used for generation. On GSM8K, it even prefers margin over greedy, though greedy is more accurate.
Questions for authors
-
This is my understanding of Table 16 : The ELBO seems to overstate the gap to AR mainly because of the Jensen gap, and DUEL looks like a separate decoder that is worse than the uniform-order model. Can you provide explanation on this.
-
How much does the Table 2 compute gap change with the change in policy, for example left-to-right vs greedy vs margin.
-
Can you add a human or other judge check on some of the pairwise labels, plus a simple baseline like entropy or gen-ppl?
2 comments
Thank you for the detailed rebuttal and the additional experiments, which address my questions. I maintain my score of 3.
Response to Reviewer JQwo (R3)
We thank the reviewer for the careful reading. We agree with most of the points raised and answer each question with new results. If this addresses the reviewer's concerns, we would be grateful if the reviewer would consider raising their score.
W1 / Q1. DUEL and the ELBO measure different distributions.
"DUEL and the ELBO measure different distributions." We agree. Of the two, only DUEL measures the distribution we actually sample from at inference time. The ELBO bounds the likelihood of a uniform random-order sampler, which no current MDM generates with.
"With a left-to-right order, DUEL is basically the likelihood of an AR-style decoder." We agree with this too. The reason it needs an argument is that an MDM is trained on every order, so its likelihood is a sum over all of them, p(x)=∑σp(x,σ), which is intractable in general. Our paper shows that under a deterministic rule this sum collapses to a single term, so the likelihood becomes exact and computable. Left-to-right is the simplest such rule. The same result covers confidence and margin rules, which are not AR-style and whose order depends on x.
Table 16. Your reading is right: the ELBO overstates the gap to AR mainly through the Jensen gap, and the uniform-order sampler is a better density than DUEL. We have now confirmed this exactly on in-domain data for two models (Table R2.1 in our response to Reviewer Lgnp). But the uniform-order sampler is not usable in practice: its exact likelihood is intractable beyond short blocks, and it generates poorly (37.8% on GSM8K, against 64.0% with greedy confidence; Table 18).
Q2. How much does the Table 2 compute gap change with the policy?
We agree that the reduction depends on the policy.
Table R3.1: Compute-optimal loss (NLL, nats/token) and, in parentheses, the MDM–AR compute gap. Table 2 protocol, same SMDM checkpoints.
Every rule is below the ELBO on every checkpoint. We chose left-to-right for Table 2 because it is the most comparable to the AR baseline: both models are then scored under the same generative procedure, one token at a time from left to right. Under greedy confidence the compute gap still falls from ×15–16 to about ×11 (and to ×12–13 under margin), though these rules score a different generative model than left-to-right. We will add greedy confidence to the scaling-law results.
Q3 / W2. A second judge and a simple baseline.
We re-ran the pairwise study with a judge from a different model family (OpenAI GPT-6.1 Sol): same prompts, blinding and both presentation orders, on the same 200 sample pairs for each of the 24 pairs (9,600 calls, none missing). We also added token entropy as a baseline.
Table R3.2: Agreement with each judge on the pairs that judge decides.
On the remaining points: