- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the train–inference mismatch in discrete diffusion: the denoiser is trained on forward-corrupted states but at inference sees states produced by its own sampler. The mismatch is defined as a divergence between the forward and sampler marginals at each time (the "Mirror Gap") and measured in the denoiser's hidden space with three proxies: a forward-vs-reverse probe, a standardized centroid shift and a quantized-cell JS divergence. Two training-free interventions follow: Recenter, which subtracts a calibrated rank-one hidden-state shift at one block, and MirrorGate, which rejects transitions flagged by per-layer classifiers. Experiments on OpenWebText (masked and uniform-state models, several samplers), CIFAR-10 and a 3B model report large MAUVE and FID gains at small step budgets.
Critical strengths
Recenter is simple, free at inference and consistently effective: one vector addition per step improves MAUVE several-fold across two noise families, four samplers, two modalities and three model scales (Tables 1 and 7, Fig. 1), with reasonable controls (random and opposite directions, temperature, an LLM judge). The effect is large and replicated widely enough that I expect it to survive better-tuned baselines, which is why I lean toward acceptance despite the concerns below. The probe staying saturated even at T = L (Fig. 2) is a useful indication that parallel token draws are not the whole source of the mismatch.
Critical weaknesses
My main concern is that the methodological contribution is considerably simpler than the presentation suggests. The paper acknowledges that the Mirror Gap is the mismatch studied by Ning et al. and Park et al. and that Recenter is activation steering, yet the framing ("formalise", "fundamental yet overlooked"), the pushforward notation, the three lenses and Theorem 1 suggest more. No quantity measured in Sections 4–5 is a KL, so Section 3 and Theorem 1 are decoupled from what is measured and intervened on. The transfer results also undercut the measurement framework: the MDLM correction transplanted to ReMDM matches destination-calibrated Recenter (Table 3), and the frozen T = 32 correction applied at T = 64 and 128 beats it (0.119/0.222 vs 0.078/0.182, Table 4 vs Table 1). A single generic direction seems to do the work, so per-configuration calibration and the lens apparatus are not what produces the gain. I would prefer Recenter presented plainly as inference-time steering toward the forward distribution, with the analysis as motivation.
Second, the headline comparisons are not matched. γ is selected from 61 values on 5k validation generations per configuration, with budget-specific overrides, while the baselines keep their released η and temperature at every T. The one matched control, temperature (Fig. 15), reproduces the GenPPL gain but at lower MAUVE and entropy, which supports the paper but is not a tuned baseline. Table 1 reports no uncertainty for 5k-sample MAUVE, and its T = 128 MDLM and ReMDM rows coincide with Wang et al. (2025) while the Recentered rows are new runs (Table 4's own native MDLM at T = 128 is 0.017, not 0.015), which should be stated. The text 8× and image 8×/16× efficiency factors compare against numbers quoted from other papers at 1024 and 4096 NFE rather than baselines run in this pipeline, and "zero extra NFE" omits the offline calibration and γ-sweep cost per configuration. Minor: the "11× MAUVE improvement" (p. 8) matches no summary of Table 1, where ratios range from 2× to 15× over baselines near zero, and MirrorGate's gains are an order of magnitude smaller than Recenter's.
Questions for authors
- Does a single global forward-vs-reverse direction with constant amplitude and no time schedule match the gains in Table 1? If so, what do the three lenses and per-(sampler, T) calibration contribute?
- Were the ReMDM and Ψ-sampler η and the temperature tuned on validation MAUVE at each T, as γ was? What is the margin over baselines tuned this way?
- Please run native ReMDM and Ψ-sampler to 1024 and 4096 NFE in your pipeline, mark which Table 1 cells are quoted, and add generation-seed uncertainty.
- Does the denoiser's per-token NLL on reverse states actually improve under Recenter? This is the mechanism the motivation asserts, and the AR result in Fig. 30 suggests a generic quality-steering effect instead.
4 comments
Q3. We appreciate and share the reviewer’s concern for statistical rigor and reproducibility. We built our methods directly on the codebases/pipelines released by the authors of [2] and [3], we used these pipelines to rerun the native baselines underlying our sampling-efficiency comparisons, so our comparisons are indeed matched in implementation and have all been reproduced across seeds. With respect to the transfer ablation in Table 4 highlighted by the reviewer: at , we obtain 0.0174 MAUVE versus 0.015 in [2]. Across our 10 generation seeds, the Recenter gain was +0.2045 ± 0.0186 MAUVE (mean ± SD), so the 0.0024 discrepancy is only 1.2% of the mean gain and is consistent with generation-seed variability of MAUVE at values close to 0.
Q4. Our motivation is that the measured discrepancy between forward and reverse states can guide the choice of a steering direction that improves final sample quality. An unconditional reverse trajectory has no ground-truth target for its missing tokens.
One fixed direction and unit gain substantially improved MAUVE across both Dream‑7B samplers at every budget tested, without gain, temperature, direction tuning. We hope these gains and the paired-seed results address the reviewer's main concerns and give them enough reason to consider raising their recommendation. If there are any further questions held by the reviewer, we remain eager to engage further.
[1] Fred Zhangzhi Peng, Zachary Bezemek, Jarrid Rector-Brooks, Shuibai Zhang, Anru R. Zhang, Michael Bronstein, Alexander Tong, and Avishek Joey Bose. Planner aware path learning in diffusion language models training. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.23405. Oral.
[2] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2503.00307.
[3] Justin Deschenaux, Caglar Gulcehre, and Subham Sekhar Sahoo. The diffusion duality, chapter II: Ψ-samplers and efficient curriculum. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2602.21185.
[4] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7B: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. URL https://arxiv.org/abs/2508.15487.
[5] Luca Soldaini et al. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. URL https://aclanthology.org/2024.acl-long.840/.
Thanks, the Dream-7B result is a nice addition. The comparison with properly tuned baselines is still missing, so I keep my score.
We thank the reviewer for their follow-up. We have now completed the requested comparison against extensively tuned baselines:
We swept η over the dense grid {0, 0.0025, …, 0.15}, comprising 61 configurations per sampler and per budget, for both ReMDM and Duo++ with the Ψ-sampler at T ∈ {32, 64, 128}: 366 sampler–budget–η combinations in total. As for γ, we selected η separately at each budget using validation MAUVE on 5,000 generations. (Throughout the main text, temperature is fixed at 1 for all models and samplers and is not swept in any comparison.)
For ReMDM, none of the swept configurations exceeded the MAUVE of the η = 0.04 setting selected through Wang et al.’s [2] own sweep and used in Table 1. We therefore retain this setting. For the Ψ-sampler, the selected η values were 0.0975, 0.125, and 0.1025 at T = 32, 64, and 128, respectively.
We evaluated each setting using 10 fresh generation seeds, each producing 5,000 samples. The table below reports mean MAUVE (averaged over 20 reference-sampling seeds and 10 clustering seeds per generation replicate).
| Sampler | T | η | Baseline MAUVE | Recenter MAUVE | Absolute margin |
|---|---|---|---|---|---|
| ReMDM | 32 | 0.0400 | 0.007 | 0.055 | +0.048 |
| ReMDM | 64 | 0.0400 | 0.016 | 0.211 | +0.195 |
| ReMDM | 128 | 0.0400 | 0.057 | 0.405 | +0.348 |
| Duo++ + Ψ | 32 | 0.0975 | 0.032 | 0.368 | +0.336 |
| Duo++ + Ψ | 64 | 0.1250 | 0.105 | 0.448 | +0.343 |
| Duo++ + Ψ | 128 | 0.1025 | 0.221 | 0.489 | +0.268 |
Across all baseline settings, the generation-seed standard deviation was at most 7.7% of the corresponding Recenter margin shown above, indicating that baseline generation-seed variability is small relative to the observed improvements.
This extensive baseline-tuning sweep confirms the large gains from Recenter: for instance, 13.2× the MAUVE of ReMDM at T = 64 and 11.5× for the Ψ-sampler at T = 32.
We thank the reviewer for their patience, as this was a long experiment to run. We hope this now addresses the reviewer’s last reservation and gives them enough reason to consider raising their score.
We thank the reviewer for their constructive feedback and we are glad they appreciated the fact that our Recenter intervention is simple, free at inference yet consistently very effective. We are also glad that the reviewer values the empirical breadth of our study and highlights that the effects we find are large, with several-fold improvements and are widely replicated.
We agree that the methodological contribution of our Recenter intervention is simple. We hope that its simplicity and effectiveness across settings will motivate others to adopt our method, similarly to simple inference-time methods like top-p nucleus sampling. Beyond that, we do think the lenses and Theorem 1 provide an interesting point of view and new tools to observe the mismatch that we haven't seen explored before. Notably, the centroid lens is what enabled us to discover our effective intervention. It revealed the low-rank structure of the shift between forward and reverse mean activations, without which a steering direction would have been hard to find. We value communicating to our reader the process through which we uncovered these interventions, in the hope that they may build upon it and improve it further.
We thank the reviewer for pointing out the gap between the KL analysis in Section 3 and the empirical measurements in Sections 4–5, which use JS divergence. We have added corresponding figures that measure the gap using KL on the Voronoi cells, making the connection to Section 3 more explicit.
W2. We address this in Q1,2,3,4 below.
Q1. The correction indeed transfers very well within noises for different samplers. Thus, a direction does work well in practice within a noise family across budgets and samplers, which we see as a strength. We additionally provide the analysis in the appendix to show its effectiveness across a wide range of configurations, as well as to detail when it works. We encourage the reader to apply the simplest setting, which we detail in the main text.
Q2. We use the sampler settings studied in [2, 3] which have been extensively swept and carefully tuned. For ReMDM, [2, Appendix E.2.4, Figs. 6–8] sweeps 14 settings at each of three fast-sampling budgets (T=128,256,512), for 42 setting–budget combinations. At T=128, even taking the best MAUVE across all 14 settings (approximately 0.065), Recenter reaches 0.405, an absolute gain of 0.340. For Ψ, [3, Appendix D.1.2, Table 13] evaluates 21 configurations per budget across eight budgets, or 168 setting–budget combinations for Duo alone (336 including MDLM). We use the same sampling and evaluation pipeline as [3] and adopt the strongest Ψ setting the authors recommend based on the sweep.
Q1–Q2 Additional Experiments. To test the reviewer’s suggestion directly, we use a single direction with constant amplitude on a new larger checkpoint, Dream‑7B‑Base [4]. We estimate the direction at block 25 from just 512 trajectories of the original sampler and apply the same vector at every step. We set the gain to 1 throughout, with no per-(sampler,T) gain tuning and reuse it across all budgets and both the original sampler and ReMDM (remasking), with top‑p = 0.9, temperature = 1, and ReMDM η = 0.04. The table reports MAUVE (higher is better) with Dolma [5] as the reference corpus (used to pretrain Dream‑7B [4]) on 5,000 samples of length 256 per point. Parentheses give the absolute gain (margin) over the corresponding sampler.
A single direction therefore strongly improves MAUVE in all comparisons, with no per(sampler,T) tuning.
[continued in next comment]