- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DJRS, a training free sampler for masked diffusion language models. It scores each masked token with confidence, margin, and JSD, and top-1 change flag. It then commits extra low risk tokens so the blocks finish early. The paper evaluates the model on several models and benchmarks.
Critical strengths
NFE savings at 128 steps are fairly large for a training free method, which shows the effectiveness of the method.
Also the authors are upfront that the method is a heuristic and also show the unfavorable results such as 128-step MATH-500, LATCH being better at 128 steps, the GSM8K ablation.
Critical weaknesses
-
The main claim doesn't hold up against the paper's ablations. The performance increase is very small or worse - on GSM8K, confidence only is 65.0% with 39.0 NFE, which beats full DJRS (63.8% at 45.6) on both accuracy and NFE.
-
On MATH-500 30.6% at 45.3 NFE / 31.4% at 49.7, small gap under standard error, and adding JSD drops it to 29.0%. So the savings likely come from extra confident tokens and early blocks finish. Maybe not from margin / JSD / prediction change.
-
Also the paper does not explain about weights and thresholds.
-
Also more empirical evidence is needed. Everything is a single greedy run with no CIs.
Questions for authors
-
Table 6 shows confidence only results are better than the full DJRS. Does margin, JSD or prediction change actually help, and can you give confidence only numbers for MATH-500 128 steps / GSM8K 128 steps and HumanEval?
-
How does DJRS compare at matched NFE with Default using fewer steps and with Fast-dLLM's confidence-threshold decoding?
-
Can you add results with multiple seeds, and also provide details about the choice of weights and thresholds.?
2 comments
Thank you for the rebuttal. My main concerns are not fully addressed so I keep my score.
Does the evidence establish that all four signals help?
The evidence supports adaptive commitment more strongly than it supports the necessity of all four signals. The confidence-only variant uses the same cumulative-budget selection and block-completion mechanism, and improves both accuracy and NFE on GSM8K relative to the full score. We therefore distinguish two conclusions: the explicit adaptive construction achieves useful measured trade-offs against the fixed schedule, while the full four-signal operating point is not consistently preferable to simpler settings. The MATH-500 differences do not resolve that second question because accuracy and compute change together.
The completed record supplies a GSM8K confidence-only result at nominal budget 128: 68.61% at 45.955 mean NFE. The MATH-500 confidence-only near-compute comparison was run at nominal budget 80, reaching 33.0% at 51.784 NFE; it is not the requested 128-step MATH-500 result. Confidence-only HumanEval results are unavailable. We state those coverage limits explicitly rather than presenting another configuration as the requested experiment.
Default using fewer steps and confidence-threshold decoding
The completed near-compute comparison directly addresses the fixed-schedule question. On GSM8K, full DJRS at nominal budget 128 reaches 68.54% at 57.021 mean NFE; Default at 56 measured NFE reaches 60.05%. The paired difference is +8.49 points, with 95% bootstrap CI +5.91 to +11.07 over the same 1,319 examples. With eight blocks, the scheduler's fixed-work settings occur in multiples of eight; 56 is the nearest legal point, so this is a near-match rather than exact compute equality.
On MATH-500, DJRS reaches 33.6% at 69.572 NFE versus Default at 30.8% and 72 NFE. The difference is +2.8 points, with 95% CI −1.2 to +6.8; this is inconclusive. Default has 2.428 more calls per example, which should remain visible in interpreting the comparison.
These results support DJRS relative to that fixed Default schedule on GSM8K. They do not establish superiority over simpler adaptive confidence policies: confidence-only and the completed local threshold sweep give strong counterexamples, as reported in our response to Reviewer nMBX. We report both the favorable fixed-schedule comparison and the unfavorable simpler-policy comparisons. No latency advantage follows solely from these NFE measurements.
Confidence intervals and repeated seeds
The completed paired analysis uses the saved deterministic outputs, pairs the same problems, and records matching document, prompt, and target hashes. Bootstrap intervals use 20,000 problem-level resamples with seed zero. For the near-compute GSM8K comparison, DJRS is correct on 904 examples and Default on 792, with 212 DJRS-only wins and 100 Default-only wins. This yields the +8.49-point difference and interval above. For MATH-500, the corresponding counts are 168 versus 154 correct and 60 versus 46 discordant wins; the interval includes zero.
Hyperparameter and threshold selection
The exact numerical defaults are specified in our response to Reviewer nMBX. The available record does not establish an original development split, search space, or selection criterion. We therefore make no held-out-selection or robustness claim for those defaults. Weight sensitivity, budget-shape sensitivity, and independent filter/union ablations are not available. The completed threshold sweep is explicitly a test-set diagnostic curve, not a validation-selected operating point. These are limits on how broadly the existing results can be interpreted.