- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work shows that there is a missing-marginal problem in diffusion post training. It shows that given that reinforcement learning or on-policy distillation (OPD) approaches only update the conditional branch of the model and don’t use the unconditional branch, they degrade performance when we use CFG at inference time. They then formalize a property called marginal closure that, under a shared-target MSE objective, the optimal unconditional prediction is equal to the posterior average of conditional procedure. Given that, they propose Marginal Closure Regularization (MCReg) that jointly supervises conditional and unconditional brach using same target in post-training and use it in both DiffusionNFT and OPD. For DiffusionNFT, they derive a reward-dependent regression target from the original objective and for OPD they reuse conditional teacher prediction without unconditional teacher queries. Experiments on SD3.5-medium and FLUX.2 Klein show with MCReg most evaluated metrics when CFG used gets better and marginal closure error is reduced.
Critical strengths
- The work has an interesting finding that after RL finetuning when CFG is used the performance gets worse, and it is an important concern given that CFG is widely used in different methods.
- The proposed MCReg is model-agnostic and can be added in different post-training frameworks and has been evaluated in both RL and diffusion distillation.
- The theoretical analysis in section 3.1 formalizes the relationship between conditional and unconditional MSE optima and shows how the errors we have in the uncondtional branch can be amplified as we increase the CFG scales.
- The experiment section evaluates multiple methods like DiffusionNFT, Flow-GRPO-Fast, DiffusionOPD, and DanceOPD and shows the gain of using MCReg in several genreation metrics.
Critical weaknesses
- The proposed shared-target supervision is similar to how conditioning dropout works. Assuming the regression target is the same, MCReg seems to be equivalent of standard conditioning dropout with probability . Even though the work derives post-training-specific targets that are valuable for the future research, it is not clear what is its distinction from conditionsng-dropout that is widely used in pretraining.
- In DiffusionNFT, the proposed MCReg replaces the original per-sample loss weights with a minibatch-averaged weight (Eq. 15 in sec. 3.3). Therefore it remains unclear how much of the claimed improvement is because of the unconditional supervision vs the new weighting scheme.
Questions for authors
- What is the methodological distinction between MCReg and conditional dropout if applied during post-training?
- What is the gain of minibatch average in eq. 15? an ablation isolating this from additional unconditional supervision will be helpful.
3 comments
Thank you to the authors for the clear rebuttal and additional experiments.
With regard to my first concern, the distinction of the relationship of MCReg and conditioning dropout is appreciated. The mathematical equivalence of MCReg and conditioning dropout under the assumption of shared-target regression and the definition of the contribution as post-training specific target construction and marginal closure analysis are particularly useful clarifications.
With regard to my second concern, the ablation study presented in Table 4 clearly isolates the effects of minibatch averaging and unconditional supervision. The continuous improvement when applying the former technique while maintaining the latter shows that the gain in performance comes mainly from the proposed supervision technique.
Thus, I find that both my concerns are well-addressed by the authors, although I still think that there is not enough methodological novelty due to the relation to conditioning dropout. However, the theoretical analysis and empirical improvements under various post-training techniques make the contribution valid.
Therefore, I am maintaining my 3 (Weak Acceptance) score for this paper.
Thank you for your constructive feedback and support
We sincerely thank you for your prompt follow-up and positive assessment of our work. We appreciate your careful consideration of our rebuttal and additional experimental results, as well as your constructive feedback and continued support throughout the review process.
Response to Reviewer dxka
Thank you for highlighting the conditioning-dropout connection and the role of minibatch averaging. We have made the theoretical connection explicit and added controls that isolate unconditional supervision. Revisions are marked in blue.
W1/Q1. Conditioning dropout as an interpretation of MCReg.
MCReg applies the conditioning-dropout principle used in supervised fine-tuning [1, 2] to post-training-specific regression targets. The connection helps explain both the objective and what its optimization controls.
The post-training target determines the unconditional quantity learned by this regression. In DiffusionNFT, we recover a reward-dependent target, including the reference penalty, from the conditional objective. In OPD, both predictions reuse conditional teacher targets under the student-rollout law; the teacher's own unconditional prediction need not be the corresponding marginal under that law. These constructions make shared-target supervision applicable to the post-training objectives studied here.
At a fixed update, let ℓC=∥C(Z,c)−Y∥2 and ℓN=∥N(Z)−Y∥2 use the same sampled state and detached target. Conditioning dropout with probability p=ρ/(1+ρ) has expected loss
=\frac{\ell_C+\rho\ell_N}{1+\rho}.$$ The identity also holds with common detached weights. Our implementation evaluates both terms on each sampled state and target; its normalized risk equals the expected conditioning-dropout regression risk. We now state this connection in Section 3.2. Theorem 1 then gives a quantitative interpretation of the connection. Under its fixed sampling law, the normalized excess risk decomposes into nonnegative prompt-dependent and common-marginal fitting errors plus $p(1-p)R_{\rm abs}$. Consequently, $$R_{\rm abs}\leq\frac{\Delta R}{p(1-p)},$$ where $\Delta R$ is measured relative to the unrestricted regression optimum. For fixed $0<p<1$, approaching this optimum also drives closure error to zero. The same interpretation holds under a normalized common weighted law. Thus the contribution lies in the post-training target construction, the induced unconditional optimum, and the analysis of what joint supervision controls. Section 3.2 presents the risk decomposition and bound; Appendix A.3, Eqs. (26)–(27), gives the dropout-probability form. **W2/Q2. Minibatch averaging versus unconditional supervision.** Batch averaging gives equal regression coefficients within a minibatch, with a common scale set by the mean recovered weight. The theory also permits shared per-sample weights under the corresponding weighted law; batch-mean conditional weighting is an implementation choice. To separate the two effects, Table 4 compares native and batch-mean conditional weighting, each with and without the same unconditional supervision term. Each addition preserves its conditional weighting rule and coefficient. Table 4 reports mean ± SD over three runs on SD3.5-Medium at $w=3$ (PaP: Pick-a-Pic; DB: DrawBench). Both $+N$ controls use batch-mean $N$ weights. |$C$ weights|$+N$|PaP PickScore|PaP CLIP|PaP HPSv2.1|DB PickScore|DB CLIP|DB HPSv2.1|DB ImgRwd|DB UniReward|DB DeQA| |---|:---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| |Native|—|22.771 ± 0.107|30.813 ± 0.219|34.352 ± 0.233|23.305 ± 0.091|29.912 ± 0.039|33.255 ± 0.101|1.486 ± 0.018|0.672 ± 0.001|4.241 ± 0.023| |Batch-mean|—|22.360 ± 0.108|30.899 ± 0.295|33.383 ± 0.694|22.975 ± 0.109|29.983 ± 0.033|32.230 ± 0.572|1.439 ± 0.028|0.659 ± 0.006|3.988 ± 0.103| |Native|✓|**23.411** ± 0.077|**31.968** ± 0.222|**36.797** ± 0.097|**23.801** ± 0.059|**30.331** ± 0.023|**34.996** ± 0.006|1.558 ± 0.007|0.691 ± 0.004|4.329 ± 0.019| |Batch-mean|✓|23.326 ± 0.023|31.721 ± 0.028|36.484 ± 0.104|23.690 ± 0.042|30.191 ± 0.180|34.676 ± 0.174|**1.564** ± 0.016|**0.691** ± 0.003|**4.339** ± 0.020| Across three SD3.5-Medium training runs at $w=3$, batch-mean weighting alone raises mean CLIPScore but lowers PickScore and HPSv2.1 on both benchmarks. Adding unconditional supervision improves **all nine metric means under either conditional weighting rule**. With the native conditional loss retained, all nine metrics improve in each of the three paired runs, and this variant attains the highest PickScore and HPSv2.1 means on both benchmarks. These controls support **unconditional supervision as the key component**. **Batch-mean conditional weighting is an implementation choice**: changing conditional weights alone has mixed effects, while the gains from unconditional supervision persist with the native conditional objective unchanged. Section 3.3 clarifies the weighting choice, Section 4.4 discusses the results, and Appendix E.2 specifies the controls. **References** [1] Jonathan Ho and Tim Salimans. *Classifier-Free Diffusion Guidance*. NeurIPS Workshop, 2021. [2] Jincheng Zhong et al. *Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model*. ICLR, 2025.