- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that tracks the class posterior online during generation in discrete diffusion models. The posterior is updated by accumulating the log-difference between conditional and unconditional predictions (Eq. 10), and taking this as weights, logical operators AND/OR/NOT are derived at the level of discrete rate functions. The core application is a dynamic NOT gate, where repulsive guidance increases only when there are signs that the class to be avoided is appearing. The authors show 1. tracking accuracy on synthetic data where analytical ground truth exists, and 2. interpretability and NOT gate performance across four modalities: MNIST, text, and antibody sequences. Furthermore, they point out that discrete CFG and continuous CFG are different operations that multiplicatively mix data distribution and marginals, respectively.
Critical strengths
-
Unlike autoregressive models, it was difficult in Diffusion LMs to see "at what point what is decided," but this paper tracks the fluctuation of the posterior in units of token flips, providing direct signals such as "surging at wine/variety/red, no reaction at a/the" (Table 1). This can be reused not only for generation control but also as a tool to analyze the internal dynamics of dLMs, and opens a genuinely new analysis axis to this community in that it has no counterpart in the continuous domain. (It is considered the longest-lasting contribution of this paper.)
-
Under extreme class imbalance in antibody generation(IGHV3 44% vs IGHV7 0.5%), it safely removes both majority/minority with a single hyperparameter. This contrasts with static CFG requiring recalibration per regime. In image and protein modalities where concepts cannot be avoided by prompting, it enables "avoiding specific classes," so there is room for it to be cited importantly in academia such as biological sequence design.
Critical weaknesses
-
The point that discrete CFG multiplicatively mixes data distributions is conceptually interesting, but there is no answer to what measurable consequence this difference produces in generation quality, or what practitioners should change. Fig 5 is merely a toy illustration; there is no prescription like "in this case, this mixture is correct," nor an experiment isolating and showing the quality difference between the two operations. For this observation to carry real weight, it needs to guide actual modeling choices; right now, it reads more like a side remark. This itself limits the paper's potential impact.
-
Posterior tracking (Eq. 10) directly follows Koulischer's formula, and the authors acknowledge that the NOT gate's "only difference from continuous is changing score to rate." Moreover, the authors directly state that the OR gate is "not strictly necessary for class-conditional generation," and text NOT is "rather silly since it can be done by prompting" (Appendix E.3). As a result, the new practical value converges to the NOT gate of image/antibody. The benchmarks are also weak to support this: MNIST is insufficient to prove controllability based on 2027 standards, and the near-perfect performance in antibodies is explained by the authors themselves as "because the classes are so well separated that the task is easy"(Appendix F.4). The baseline also stops at a single comparison with static negative CFG, lacking a contrast with more sophisticated discrete guidance in this field (such as time-dependent schedules of Rojas 2026). That is, the content regarding "why this was non-trivially difficult in discrete and what we broke through" is empty.
Questions for authors
-
Can you theoretically characterize the phenomenon where the tracked posterior deviates from the ground-truth when using the DFM sampler used in practice (Fig 6b)? Does this gap place an upper bound on the reliability of the NOT gate?
-
Can you show measurable differences that the CFG mismatch produces in generation results? For example, are there results comparing quality metrics by isolating data-level mixing and marginal-level mixing in the same setting?
-
How much does the superiority of the dynamic NOT gate (Fig 4, 17) depend on the hyperparameter search budget? Is the Pareto superiority maintained even when matched to the same search budget as the static baseline?
-
How does the AUC of posterior tracking change in more difficult discrete image/text settings beyond MNIST? How does saturation shift depending on task difficulty?
3 comments
I appreciate for thorough rebuttal and newly added experiments. I think this paper makes a meaningful contribution to the community. However, some concerns remain. First, for Q1, I asked whether the gap between the DFM-tracked and the true posterior places upper bound on the reliability of NOT gate. The response bounds the average posterior error but does not connect that bound to NOT-gate performance. For Q4, CIFAR experiments cover only posterior tracking. NOT gate is still evaluated only on MNIST, so the controllability concern remains. For Q3, a broad region of strong performance does not show that the Pareto advantage holds under a matched search budget. For it, you should show either 1. the Pareto curve with a randomly chosen or worst-case τ, or 2. results with τ selected on a validation set and evaluated on a held-out test set. Lastly, it is still unclear how easily practitioners can adopt this method in real-world tasks, given the extra hyperparameters and the authors' own recommendation of negative CFG for text. (If the concerns in Q1 and Q3 are adequately addressed, I am willing to raise my score)
Reply to remaining concerns
General We thank the reviewer for acknowledging the meaningful contribution of our manuscript, and hope that the clarifications below address the remaining concerns.
Further discussion on Q1 and Q3:
-
Q1 (Effect of posterior error on the NOT gate): We added Appendix A.3 on this. With an approximate posterior , the gate rate becomes with (see Fig. 7). Underestimating the posterior moves it towards the unconditional rate, leaking parts of class . Overestimating the posterior moves the rate past the exact NOT rate, overshooting the suppression and possibly producing negative rates, which are clamped to zero. We also noticed while addressing this question, that this clamping was not stated explicitly in the original manuscript; it is now specified in the main text. We thank the reviewer for their comment, which helped us address a small but meaningful gap in the analysis.
-
Q3 (Sensitivity of NOT gate on ): In response to the remaining concern, we added the experiment shown in Fig. 21 (see App. C.4): it shows how the pareto curves vary for the values of swept in the grid search for the NOT gate on MNIST. The 5 ablated curves outperform the static CFG baseline. We believe that this much better illustrates our method's stability in on this modality and thank the reviewer for suggesting the addition of this figure to the document.
We hope that these additional results help the reviewer to overcome their remaining hesitation.
Reply to review
General We thank the reviewer for their careful review and helpful feedback. We are glad they recognize the potential of online posterior estimation for analyzing denoising dynamics in discrete diffusion and the advantages of dynamic over static guidance, particularly for highly unbalanced distributions.
Reply to Ws and Qs Below we will adress all weaknesses and questions brought up by the reviewer. We split the second weakness in three subtopics for more readability.
Image experiments: We evaluate posterior tracking on a 120M-parameter CIFAR10 uniform diffusion model, with corresponding posterior-tracking AUCs in Fig. 29. Despite our best efforts, obtaining a high quality unconditional CIFAR model proved difficult. We have added Appendix C.5 on this topic, with for instance comparison to smaller 35M models on the same modality. The AUC curves show the same qualitative behavior as the MNIST curves with as difference that the final obtained AUC is lower.
Text experiments: We evaluate the posterior using LLaDA on a more realistic story-generation task (App. D). The posterior stabilizes at around 86% AUC (Fig. 37), despite greater diversity of possible generations. Together with our antibody experiments, these results demonstrate posterior tracking's applicability to open-ended generation across modalities.
We acknowledge that a systematic investigation of how tracking accuracy scale with task complexity remains an interesting direction for future work, and have added this to the limitations.
[1] Measuring Mathematical Problem Solving With the MATH Dataset, Hendrycks et al., NeurIPS21
[2] Improving Classifier-Free Guidance in Masked Diffusion: Low-Dim Theoretical Insights with High-Dim Impact, Rojas et al., ICLR26
[3] Dynamic Negative Guidance of Diffusion Models, Koulischer et al., ICLR25