acceptodds
Under review as a conference paper at ICLR 2027

The Missing Marginal: Marginal Closure for Classifier-Free Guidance in Diffusion Post-Training

Abstract

Classifier-Free Guidance (CFG) combines conditional and unconditional predictions to improve diffusion sampling, yet we observe that it can degrade performance after online reinforcement learning (RL). Examining existing post-training objectives, we find that optimizing either the conditional branch alone or a single fixed-scale CFG output leaves the unconditional branch underdetermined. We term this identification gap the missing-marginal problem. Under shared-target squared-error regression, the unconditional optimum equals the posterior average of the conditional optimum given the noised state, and is determined by the regression target and sampling distribution. We formalize this relation between learned predictions as marginal closure and analyze how closure errors affect CFG. Based on this relation, we introduce Marginal Closure Regularization (MCReg) to train both predictions on a shared target. We derive reward-dependent targets for DiffusionNFT. In on-policy diffusion distillation (OPD), we identify the student's unconditional counterpart and train both branches on conditional teacher targets. Experiments across online RL and multitask OPD show that MCReg improves most evaluated generation metrics under CFG.

Primary area:
generative models
Keywords:
Diffusion Post-Training, Classifier-Free Guidance, Reinforcement Learning, On-Policy Diffusion Distillation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_dxkaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work shows that there is a missing-marginal problem in diffusion post training. It shows that given that reinforcement learning or on-policy distillation (OPD) approaches only update the conditional branch of the model and don’t use the unconditional branch, they degrade performance when we use CFG at inference time. They then formalize a property called marginal closure that, under a shared-target MSE objective, the optimal unconditional prediction is equal to the posterior average of conditional procedure. Given that, they propose Marginal Closure Regularization (MCReg) that jointly supervises conditional and unconditional brach using same target in post-training and use it in both DiffusionNFT and OPD. For DiffusionNFT, they derive a reward-dependent regression target from the original objective and for OPD they reuse conditional teacher prediction without unconditional teacher queries. Experiments on SD3.5-medium and FLUX.2 Klein show with MCReg most evaluated metrics when CFG used gets better and marginal closure error is reduced.

Critical strengths
  • The work has an interesting finding that after RL finetuning when CFG is used the performance gets worse, and it is an important concern given that CFG is widely used in different methods.
  • The proposed MCReg is model-agnostic and can be added in different post-training frameworks and has been evaluated in both RL and diffusion distillation.
  • The theoretical analysis in section 3.1 formalizes the relationship between conditional and unconditional MSE optima and shows how the errors we have in the uncondtional branch can be amplified as we increase the CFG scales.
  • The experiment section evaluates multiple methods like DiffusionNFT, Flow-GRPO-Fast, DiffusionOPD, and DanceOPD and shows the gain of using MCReg in several genreation metrics.
Critical weaknesses
  • The proposed shared-target supervision is similar to how conditioning dropout works. Assuming the regression target is the same, MCReg seems to be equivalent of standard conditioning dropout with probability p=ρ1+ρp = \frac{\rho}{1 + \rho}. Even though the work derives post-training-specific targets that are valuable for the future research, it is not clear what is its distinction from conditionsng-dropout that is widely used in pretraining.
  • In DiffusionNFT, the proposed MCReg replaces the original per-sample loss weights with a minibatch-averaged weight (Eq. 15 in sec. 3.3). Therefore it remains unclear how much of the claimed improvement is because of the unconditional supervision vs the new weighting scheme.
Questions for authors
  • What is the methodological distinction between MCReg and conditional dropout if applied during post-training?
  • What is the gain of minibatch average in eq. 15? an ablation isolating this from additional unconditional supervision will be helpful.
3 comments
AuthorsOpenReview ↗

Response to Reviewer dxka

Thank you for highlighting the conditioning-dropout connection and the role of minibatch averaging. We have made the theoretical connection explicit and added controls that isolate unconditional supervision. Revisions are marked in blue.

W1/Q1. Conditioning dropout as an interpretation of MCReg.

MCReg applies the conditioning-dropout principle used in supervised fine-tuning [1, 2] to post-training-specific regression targets. The connection helps explain both the objective and what its optimization controls.

The post-training target determines the unconditional quantity learned by this regression. In DiffusionNFT, we recover a reward-dependent target, including the reference penalty, from the conditional objective. In OPD, both predictions reuse conditional teacher targets under the student-rollout law; the teacher's own unconditional prediction need not be the corresponding marginal under that law. These constructions make shared-target supervision applicable to the post-training objectives studied here.

At a fixed update, let ℓC=∥C(Z,c)−Y∥2\ell_C=\Vert C(Z,c)-Y\Vert ^2 and ℓN=∥N(Z)−Y∥2\ell_N=\Vert N(Z)-Y\Vert ^2 use the same sampled state and detached target. Conditioning dropout with probability p=ρ/(1+ρ)p=\rho/(1+\rho) has expected loss

=\frac{\ell_C+\rho\ell_N}{1+\rho}.$$ The identity also holds with common detached weights. Our implementation evaluates both terms on each sampled state and target; its normalized risk equals the expected conditioning-dropout regression risk. We now state this connection in Section 3.2. Theorem 1 then gives a quantitative interpretation of the connection. Under its fixed sampling law, the normalized excess risk decomposes into nonnegative prompt-dependent and common-marginal fitting errors plus $p(1-p)R_{\rm abs}$. Consequently, $$R_{\rm abs}\leq\frac{\Delta R}{p(1-p)},$$ where $\Delta R$ is measured relative to the unrestricted regression optimum. For fixed $0<p<1$, approaching this optimum also drives closure error to zero. The same interpretation holds under a normalized common weighted law. Thus the contribution lies in the post-training target construction, the induced unconditional optimum, and the analysis of what joint supervision controls. Section 3.2 presents the risk decomposition and bound; Appendix A.3, Eqs. (26)–(27), gives the dropout-probability form. **W2/Q2. Minibatch averaging versus unconditional supervision.** Batch averaging gives equal regression coefficients within a minibatch, with a common scale set by the mean recovered weight. The theory also permits shared per-sample weights under the corresponding weighted law; batch-mean conditional weighting is an implementation choice. To separate the two effects, Table 4 compares native and batch-mean conditional weighting, each with and without the same unconditional supervision term. Each addition preserves its conditional weighting rule and coefficient. Table 4 reports mean ± SD over three runs on SD3.5-Medium at $w=3$ (PaP: Pick-a-Pic; DB: DrawBench). Both $+N$ controls use batch-mean $N$ weights. |$C$ weights|$+N$|PaP PickScore|PaP CLIP|PaP HPSv2.1|DB PickScore|DB CLIP|DB HPSv2.1|DB ImgRwd|DB UniReward|DB DeQA| |---|:---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| |Native|—|22.771 ± 0.107|30.813 ± 0.219|34.352 ± 0.233|23.305 ± 0.091|29.912 ± 0.039|33.255 ± 0.101|1.486 ± 0.018|0.672 ± 0.001|4.241 ± 0.023| |Batch-mean|—|22.360 ± 0.108|30.899 ± 0.295|33.383 ± 0.694|22.975 ± 0.109|29.983 ± 0.033|32.230 ± 0.572|1.439 ± 0.028|0.659 ± 0.006|3.988 ± 0.103| |Native|✓|**23.411** ± 0.077|**31.968** ± 0.222|**36.797** ± 0.097|**23.801** ± 0.059|**30.331** ± 0.023|**34.996** ± 0.006|1.558 ± 0.007|0.691 ± 0.004|4.329 ± 0.019| |Batch-mean|✓|23.326 ± 0.023|31.721 ± 0.028|36.484 ± 0.104|23.690 ± 0.042|30.191 ± 0.180|34.676 ± 0.174|**1.564** ± 0.016|**0.691** ± 0.003|**4.339** ± 0.020| Across three SD3.5-Medium training runs at $w=3$, batch-mean weighting alone raises mean CLIPScore but lowers PickScore and HPSv2.1 on both benchmarks. Adding unconditional supervision improves **all nine metric means under either conditional weighting rule**. With the native conditional loss retained, all nine metrics improve in each of the three paired runs, and this variant attains the highest PickScore and HPSv2.1 means on both benchmarks. These controls support **unconditional supervision as the key component**. **Batch-mean conditional weighting is an implementation choice**: changing conditional weights alone has mixed effects, while the gains from unconditional supervision persist with the native conditional objective unchanged. Section 3.3 clarifies the weighting choice, Section 4.4 discusses the results, and Appendix E.2 specifies the controls. **References** [1] Jonathan Ho and Tim Salimans. *Classifier-Free Diffusion Guidance*. NeurIPS Workshop, 2021. [2] Jincheng Zhong et al. *Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model*. ICLR, 2025.

Thank you to the authors for the clear rebuttal and additional experiments.

With regard to my first concern, the distinction of the relationship of MCReg and conditioning dropout is appreciated. The mathematical equivalence of MCReg and conditioning dropout under the assumption of shared-target regression and the definition of the contribution as post-training specific target construction and marginal closure analysis are particularly useful clarifications.

With regard to my second concern, the ablation study presented in Table 4 clearly isolates the effects of minibatch averaging and unconditional supervision. The continuous improvement when applying the former technique while maintaining the latter shows that the gain in performance comes mainly from the proposed supervision technique.

Thus, I find that both my concerns are well-addressed by the authors, although I still think that there is not enough methodological novelty due to the relation to conditioning dropout. However, the theoretical analysis and empirical improvements under various post-training techniques make the contribution valid.

Therefore, I am maintaining my 3 (Weak Acceptance) score for this paper.

AuthorsOpenReview ↗

Thank you for your constructive feedback and support

We sincerely thank you for your prompt follow-up and positive assessment of our work. We appreciate your careful consideration of our rebuttal and additional experimental results, as well as your constructive feedback and continued support throughout the review process.

Reviewer_iYc6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes a kind of gap that exists in classifier-free guidance in the context of RL. In this context, either the conditional variant or fixed-scale classifier-free guidance is often performed to fine-tune the model, but this leaves an issue for the unconditioned variant. The "gap" (as I have termed it) is cast in probability theory terms in this paper, i.e. the residual between predictions of prompt conditioning and those without. The paper builds on this idea with several formal constructions, ultimately arriving at a loss function that can jointly optimize both conditional and non-conditional branches of the generator. Similar ideas are then integrated into online RL (DiffusionNFT and Flow-GRPO) and distillation. The paper concludes with a series of empirical evaluations that demonstrate performance and attempt to isolate the impacts of the marginal-closure framework.

Critical strengths
  • The paper addresses an interesting and important topic in ML, including a problem at the confluence of RL-based fine tuning (and/or distillation), diffusion-based generators, and classifier-free guidance.

  • The paper identifies an interesting and potentially impactful problem and a sound solution.

Critical weaknesses
  • The paper empirically improves CFG. It is unclear if this truly leads to state of the art performance. For example, AWM (table 1) outperforms all other baselines across all GenEval metrics. Most of the improvements are something akin to "with" and "without" the proposed regularization. This is necessary but perhaps not-quite-sufficient (and is also a bit of self-fulfilling or circular).I am being a bit harsh here, but there are existing baselines that outperform MCReg, or any combination therein. Why is this?
Questions for authors
  1. The missing marginal problem is interesting, but perhaps not new (rather, perhaps this paper is not the first to identify it). Does Sadat et al, ICLR 2025, "No Training, No Problem..." address the same problem?

  2. In MCReg, the paper states that the "The target is detached for both regressions". What does this mean, "to detach"? The population risk looks to be regularized by the unconditioned prediction, but there are not two regressions. They are both regressed at the same time, to the same target. The subsequent theory points to decomposition of excess risk into prompt dependence, marginal, and closure errors. But detachment here simply means that NN and CC are not mixed in the same ''part'' of the risk? The loss that is eventually derived still depends on a linear combination of outputs from both conditional and unconditional models. This is all fine, I am just trying to understand the claim here. Or better yet, what "detach" means because it shows up in several places in the paper.

  3. Are the empirical results invariant to Guidance weight? For example, the paper uses w=3w=3 for CFG, and also for MCReg with CFG. Perhaps for just "with CFG" an optimal guidance weighting might be 5, whereas including MCReg the optimal guidance might be 3. I am of course making up these numbers, but I am (a) curious if there is some invariant here and (b) if not, is this truly a fair comparison? Section 4.4 somewhat addresses this but on a limited set of tasks, so I am curious if there is a rigorous argument about how this trend is true for all other experiments where MCReg is compared to "non MCReg".

  4. In general, how much hyperparameter optimization (or at least sweeps) was done in the various experiments? And what is the justification for a lack of such optimization/sweeps? One justification is trying to get close to 1:1 in terms of compute resources, training time, etc, but it is not clear here.

  5. I suggest expanding the conclusions with a richer discussion (there is a lot of say and it's boiled down to 2-3 sentences). It also does not really say what the paper does or achieves in a meaningful way. To get a bit more room in the paper, the authors could reduce the "Training and Evaluation" paragraph (Line 341) to one simple sentence that points to the appendix (and then move the existing text down where appropriate, i.e. D.1).

  6. I realized (quickly upon first reading, fortunately), that there may be a source of confusion among the community. Let us just say, hypothetically, that a reviewer or reader is a "robotics RL person". Diffusion and flow-matching have been explored extensively in that body of literature. I was following along with the paper nicely until I saw "GRPO" in the intro and then the little discussion about prompts at the end of section 2.1. I do not have a very clean, concrete question here but I wonder if it is possible to make the scope of this paper (or the application, setting within the broader ML community, etc) more clear, earlier on. SFT and several generative tasks were mentioned earlier on in the Introduction, so perhaps shame-on-me for missing this. However, I will leave this comment here if it helps make the paper more clear to the reader/community what it is about.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iYc6

Thank you for raising questions about MCReg's contribution, its connection to ICG, and comparisons across guidance scales. We have clarified the contribution and theoretical relationship, expanded the CFG comparisons, and revised the explanations and Conclusion. Revisions are marked in blue.

W1. Contribution and baseline comparisons.

DiffusionNFT and AWM are both reward-driven diffusion post-training methods. We choose DiffusionNFT as a representative framework commonly used in recent studies. MCReg's joint-supervision mechanism is compatible with reward-objective design.

AWM reports GenEval 0.985. Reported scores depend on training budgets, hyperparameters, and evaluation settings. For example, Choi et al. [1] report DiffusionNFT at 0.95 and AWM at 0.89 after 500 and 200 updates, respectively (Tables 4–5).

We therefore compare DiffusionNFT with and without MCReg under matched 1,000-update budgets and evaluation settings to study the effect of unconditional supervision. At w=3w=3, GenEval rises from 0.905 to 0.981, also exceeding DiffusionNFT without CFG (0.964).

Table 4 retains the native conditional objective, coefficient, and sample weights while adding unconditional supervision. All nine quality metrics improve in each of three paired runs, including ImageReward, UnifiedReward, and DeQA, which were not training rewards. These gains extend beyond the training rewards and closure diagnostic.

Q1. Relationship to ICG.

ICG [2] enables guidance without unconditional training via independent conditions at inference. Its posterior-averaging relation (Lemma A.1) underlies MCReg, but independent sampling need not match the posterior.

MCReg constructs shared targets from NFT's reward loss or OPD teachers' conditional predictions under student rollouts. Joint regression under the sampling law fits both branches without posterior estimation; our excess-risk decomposition bounds closure error.

Sections 1 and 3 cite Lemma A.1; Section 5 distinguishes ICG from MCReg.

Q2. Meaning of detached.

Both prediction losses jointly fit the same target. Detachment holds target-side quantities constant during backpropagation; gradients from both losses update the shared parameters. Section 3.2 now explains this operation.

Q3. Comparisons at the best CFG.

The best CFG varies across methods and metrics. Table 11 in Appendix E.3 reports each metric's highest observed score over w∈{1,2,3,4,5}w\in\{1,2,3,4,5\} and the three runs per method in Tables 2–3: 15 measurements per entry, with CFG in parentheses. Tables 2–3 report fixed-CFG means and standard deviations.

Across all metrics in Table 11, MCReg leads on 9/9 for FLUX.2 Klein Base, 8/9 for SD3.5-Medium, 7/8 for DiffusionOPD, and 6/8 for DanceOPD. The tables below summarize PickScore and HPSv2.1.

Multi-reward DiffusionNFT (from Table 11a).

Model / methodPaP PSPaP HPSDB PSDB HPS
FLUX Original23.5311 (2)36.4474 (1)23.9641 (2)34.5005 (1)
FLUX MCReg23.5619 (2)36.5673 (2)24.0843 (2)34.8090 (2)
SD3 Original23.3801 (1)36.1380 (1)23.7920 (1)34.1923 (1)
SD3 MCReg23.5486 (1)36.8505 (3)23.8552 (2)35.0134 (3)

Multitask OPD (from Table 11b).

Framework / methodDB PSDB HPS
DiffusionOPD Original23.7228 (1)33.2420 (2)
DiffusionOPD MCReg23.8584 (2)34.1860 (4)
DanceOPD Original23.7146 (1)33.4652 (2)
DanceOPD MCReg23.7966 (2)34.3915 (5)

Entries are score (CFG); bold marks the higher score within each setting. PaP: Pick-a-Pic; DB: DrawBench; PS: PickScore; HPS: HPSv2.1. SD3 and FLUX denote SD3.5-Medium and FLUX.2 Klein Base. Section 4.4 discusses these comparisons.

Q4. Hyperparameter sweeps and compute.

We focused our sweeps on regularization strength, a key hyperparameter. We tested four strengths for DiffusionNFT GenEval, eight for OCR, six for FLUX.2 Klein Base, and nine per OPD framework; multi-reward SD3.5-Medium covers ρ=0.0005\rho=0.0005–0.20.2. Repeats use fixed configurations. Variants within each setting use matched update counts. Appendix D.1 gives the training and evaluation protocols. We believe that finer-grained tuning of the regularization strength could further improve MCReg’s performance.

Q5. Conclusion and protocol.

Following your suggestion, the Conclusion now links theory, target constructions, and findings. Section 4.1 outlines the protocol; Appendix D.1 gives details.

Q6. Application scope.

As suggested, the Introduction's first sentence now specifies text-to-image diffusion and flow models.

References

[1] Jaemoo Choi et al. Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design. ICML, 2026.

[2] Seyedmorteza Sadat et al. No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models. ICLR, 2025.

Dear Reviewer iYc6,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

I thank the authors for their thorough response to my (and others') questions and concerns. I also believe the paper is a bit more clear in its current form.

My questions are largely addressed. However, I still have a concern regarding the positioning of this paper relative to existing methods, as well as its performance. For example, it is still unclear whether MCReg produces SotA performance. Yes, it improves DiffusionNFT but does not outperform (for example) AWM. This is fine, a method does not have to outperform everything, on every task. It is just difficult to look at thing empirically when most methods are already "good"; it is also difficult when a method is a direct adaptation of an existing approach (e.g. DiffusionNFT and Flow-GRP-Fast), of course it should improve these.

Again, I appreciate the authors' rebuttal and believe things are more clear. I am inclined to slightly adjust my score.

AuthorsOpenReview ↗

Response to Reviewer iYc6

Thank you for reading our response and revised manuscript so carefully. We appreciate your thoughtful follow-up and your willingness to reconsider your assessment.

MCReg extends existing post-training methods by addressing a specific question: once post-training changes the conditional model, what should its unconditional branch learn for CFG? Our analysis identifies the target, and our experiments show that this choice matters even when the competing method already trains the unconditional branch or explicitly matches the teacher’s guidance direction.

In DiffusionNFT [1], Table 4 isolates the contribution of unconditional supervision. Keeping the native conditional loss and its weights fixed, adding unconditional supervision improves all nine quality metrics at w=3w=3, including DrawBench HPSv2.1 from 33.255 to 34.996. The multitask experiments with DiffusionOPD [2] and DanceOPD [3] let us examine your concern directly. When several methods adapt the same framework for CFG, which supervision target works better, and why?

Teacher-null trains the student’s unconditional branch to reproduce the selected teacher’s unconditional prediction. Positive–Direction Matching (PDM) [4] uses that prediction to form the teacher’s CFG direction, CT−NTC_T-N_T, and trains the student to reproduce the direction. Both are natural choices for transferring an individual teacher’s guidance. In a merged student, however, the mix of tasks and prompts at a given noisy state can differ from that of any individual teacher. MCReg therefore fits both student branches to the same conditional teacher target. Under this objective, the optimal unconditional prediction is the average of the conditional targets at that state under the student’s own rollout distribution.

The Teacher-null ablation isolates this distinction by changing only the unconditional target. Against the common evaluation reference in Table 7, unconditional prediction error RNR_N falls from 0.11530 to 0.05787 in DiffusionOPD and from 0.15550 to 0.08583 in DanceOPD. The table below shows how the target choice affects generation at w=3w=3. Each arrow runs from the comparator to MCReg. The PDM comparisons use the three-run means in Table 3.

ComparatorFrameworkGenEval ↑\uparrowOCR ↑\uparrowDrawBench HPSv2.1 ↑\uparrowMetrics higher with MCReg
Teacher-null (Table 8)DiffusionOPD0.967 →\to 0.9770.965 →\to 0.97333.352 →\to 33.9966/8
Teacher-null (Table 8)DanceOPD0.961 →\to 0.9700.956 →\to 0.96533.271 →\to 34.2727/8
PDM (Table 3)DiffusionOPD0.962 →\to 0.9750.953 →\to 0.97332.995 →\to 33.9487/8
PDM (Table 3)DanceOPD0.959 →\to 0.9710.929 →\to 0.96532.589 →\to 34.2277/8

The practical advantage is that MCReg learns a guidance reference suited to the updated student using the conditional teacher prediction that OPD already computes. Teacher-null and PDM both require the teacher’s unconditional prediction as well. Across both OPD frameworks, MCReg improves most quality metrics over these alternatives, including GenEval, OCR, and guided DrawBench HPSv2.1 in all four comparisons.

We appreciate the opportunity to address this point and hope these comparisons clarify what MCReg adds to existing post-training methods.

References

[1] Kaiwen Zheng et al. DiffusionNFT: Online Diffusion Reinforcement with Forward Process. ICLR, 2026.

[2] Quanhao Li et al. DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models. SIGGRAPH Asia, 2026.

[3] Wei Zhou et al. DanceOPD: On-Policy Generative Field Distillation. arXiv, 2026.

[4] Bingnan Li et al. Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation. arXiv, 2026.

Reviewer_jC7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the degradation of classifier-free guidance (CFG) after diffusion post-training and proposes Marginal Closure Regularization (MCReg) to address the mismatch between conditional and unconditional predictions. The method uses shared targets to supervise both branches and is evaluated on several RL and on-policy distillation frameworks. The results show improvements on many CFG-guided generation metrics.

Critical strengths

The paper provides a useful analysis of why CFG can become less effective after diffusion post-training. MCReg is simple to implement and does not require changes to the model architecture. The experiments cover several RL and distillation frameworks, and the improvements on guided generation metrics, particularly in DiffusionNFT and OPD, are encouraging.

Critical weaknesses
  1. The contribution of marginal closure is not fully isolated. In the DiffusionNFT implementation, the minibatch weighting also changes the conditional training gradient. It is therefore unclear how much of the improvement comes specifically from the additional unconditional supervision. A controlled ablation that keeps the original conditional objective and weighting unchanged would make the results more convincing.

  2. The practical need for inference-time CFG after post-training is not sufficiently established. The experiments mainly evaluate models that still use CFG at inference. However, it is unclear whether the same issue remains important for stronger recent diffusion models or distilled models designed to work without inference-time CFG. RL or OPD may improve conditional generation directly, while guidance distillation can potentially transfer the effect of a fixed CFG scale into the student. The paper should clarify when preserving inference-time CFG is actually beneficial and why MCReg is preferable to these alternatives.

  3. The improvements are not consistent across all metrics and backbones. Although MCReg improves many metrics, some gains are small and some metrics even decrease, particularly on FLUX.2 Klein. The large reduction in closure error does not always translate into a comparable improvement in generation quality. More evidence linking marginal closure to actual generation performance would strengthen the paper.

Questions for authors
  1. Can you provide a controlled ablation that adds unconditional supervision while keeping the original conditional objective and sample weighting unchanged? This would help isolate the contribution of marginal closure from changes to the conditional optimization.

  2. Can you evaluate whether inference-time CFG remains beneficial after RL or OPD on a stronger recent backbone, and compare MCReg with a student distilled from a CFG-guided teacher that does not require guidance at inference? Recent CFG-free distilled models, such as Z-Image-Turbo or FLUX.2 Klein, illustrate the relevance of this question. A directly comparable experiment on the paper’s post-training setup would be especially useful; if these models are not suitable for a controlled comparison, please clarify the scope of the claim and the reason for the choice of backbones.

  3. The FLUX.2 Klein results show a substantial reduction in closure error but only small or mixed changes in generation metrics. Can you provide further analysis of this relationship across guidance scales and backbones?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jC7x

Thank you for your review. We added controlled ablations, CFG-free student comparisons, and analyses of closure and quality across CFG scales and backbones. Revisions are marked in blue.

W1 / Q1. Adding unconditional supervision with the native conditional objective unchanged.

We retain the native conditional loss, per-sample weights, and coefficient, adding only batch-mean-weighted unconditional regression toward the recovered NFT target. At w=3w=3, all nine metrics improve in each of three paired SD3.5-Medium runs.

PaP/DB: Pick-a-Pic/DrawBench.

C+NPaP PickPaP HPSDB PickDB HPSDB ImgRwdDB UniRwdDB DeQA
Native—22.77±.1134.35±.2323.31±.0933.26±.101.49±.020.67±.004.24±.02
Batch-mean—22.36±.1133.38±.6922.98±.1132.23±.571.44±.030.66±.013.99±.10
Native✓23.41±.0836.80±.1023.80±.0635.00±.011.56±.010.69±.004.33±.02
Batch-mean✓23.33±.0236.48±.1023.69±.0434.68±.171.56±.020.69±.004.34±.02

With the conditional objective and weights unchanged, these gains isolate the benefit of N supervision and support it as the key component. Batch-mean conditional weighting is an implementation choice. See Table 4 and Appendix E.2.

W2 / Q2. When CFG remains beneficial, and the CFG-free alternative.

We study reward optimization with DiffusionNFT and conditional-target OPD for merging teacher capabilities. These objectives directly supervise conditional predictions but not the unconditional counterpart used by CFG. MCReg jointly fits both branches to shared targets during post-training, adapting the unconditional prediction to the updated conditional model to improve CFG-guided quality.

DiffusionNFT's CFG-free training uses conditional-only rollouts to optimize rewards. Guidance distillation trains a student to absorb teacher CFG for conditional-only inference; DMD [1] can incorporate guidance through distribution matching for one-step generation. We use undistilled FLUX.2 Klein Base to compare MCReg with a student distilled from the Original RL teacher, holding the backbone and reward-post-training setup fixed. Sections 4.1 and 5 clarify this scope and backbone choice.

Two comparisons directly address your question:

  1. Does CFG help a fixed post-trained model? Increasing ww from 1 to 3 improves all nine metrics in each of the three FLUX MCReg runs reported in Table 2. In contrast, Original HPS decreases in all three runs on both benchmarks. Table 12 in Appendix E.3 reports these within-checkpoint changes.

  2. How does MCReg compare with a guided-teacher CFG-free student? We train a FLUX student by DanceOPD-style on-policy guidance distillation from the Original RL teacher. The target uses teacher guidance w=3w=3; the student uses w=1w=1 at inference. MCReg uses w=3w=3 at inference and scores higher on all nine metrics. We compare retaining CFG with guidance absorption after post-training, while DMD targets one-step generation.

Table 13(b) compares student objectives within SD3.5-Medium multitask DanceOPD: the alternative learns guided predictions from the same three teachers, each using w=3w=3, and uses w=1w=1 at inference. With the same initialization, teachers, task mixture, optimizer, and 2,000 updates, MCReg evaluated at w=3w=3 improves GenEval, OCR, and five of six DrawBench metrics (seven of eight overall).

DrawBench metrics from Table 13:

MethodPickHPSImgRwd
FLUX/CFG-free23.840633.96151.3795
FLUX/MCReg23.978733.97411.4283
DanceOPD/CFG-free23.219231.88221.4000
DanceOPD/MCReg23.710734.27181.5518

Details are in Section 4.4 and Appendices D.1/E.4.

W3 / Q3. Closure and quality across scales and backbones.

Endpoint RabsR_{\rm abs} falls from 0.182 to 0.018 (SD3) and 0.426 to 0.049 (FLUX). The table below reports the corresponding DrawBench differences (MCReg − Original).

Backbone/wwPickScoreHPSv2.1DeQAImgRwd
SD3/1+0.046+0.490+0.0030+0.0444
SD3/3+0.358+1.518+0.0879+0.0718
SD3/5+0.453+1.937+0.1291+0.0799
FLUX/1+0.034−0.407−0.0096+0.0139
FLUX/3+0.110−0.037+0.0067+0.0103
FLUX/5+0.146+0.135+0.0219+0.0192

PickScore, HPS and DeQA differences increase with ww on both backbones; SD3 shows larger gains.

Quality depends on conditional accuracy and branch consistency. When C=C⋆C=C^\star, guided-prediction MSE is (w−1)2Rabs(w-1)^2R_{\rm abs} (Section 3.1). Lower closure lets CFG amplify prompt-dependent information with less distortion from branch mismatch, especially at larger ww. MCReg jointly supervises both branches during post-training to learn accurate conditional predictions and a compatible unconditional guide.

Guided-prediction accuracy also improves: Appendix E.1 reports guided-MSE reductions of 22.7%/25.0% in DiffusionOPD/DanceOPD at w=3w=3; extending this diagnostic to w=5w=5 gives 29.8%/29.7%.

References

[1] Tianwei Yin et al. One-step Diffusion with Distribution Matching Distillation. CVPR, 2024.

Dear Reviewer jC7x,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_bx8aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies why classifier-free guidance (CFG) can reduce image quality after diffusion post-training. CFG combines predictions with and without a prompt. The authors argue that some training objectives do not properly supervise the unconditional prediction. They propose Marginal Closure Regularization (MCReg), which trains both predictions using shared targets in DiffusionNFT and on-policy distillation. The theory explains the relation between the two predictions and how their errors affect guidance. Experiments on SD3.5-Medium and FLUX.2 Klein show improvements in most evaluated guided-generation metrics.

Critical strengths
  • Clear motivation and explanation. The paper identifies a practical problem: stronger guidance can hurt a model after post-training. Its analysis explains why improving the conditional prediction alone may not be enough.

  • Simple and useful design. MCReg reuses the same target to train both predictions. In distillation, it does not require extra queries to the teacher’s unconditional branch.

  • Broad experimental coverage. The paper evaluates several RL and distillation methods, with useful target ablations and guidance-scale tests. For example, MCReg improves guided DiffusionNFT GenEval from 0.905 to 0.981.

Critical weaknesses
  • The connection between closure and image quality is only partly tested. Closure errors are measured at the pure-noise starting point. This does not show whether the two predictions remain consistent throughout the denoising process.

  • The DiffusionNFT comparison changes more than unconditional supervision. Equation 15 also changes the weights of training samples. Without a separate control, it is difficult to tell how much improvement comes from each change.

  • Small gains need stronger statistical support. Most controlled experiments use one training seed. The reported confidence intervals describe prediction diagnostics for fixed checkpoints, rather than variation across independent training runs. This matters especially for the small gains on FLUX.2 Klein.

Questions for authors
  • Can the authors measure or estimate closure errors at intermediate denoising steps? Do these errors explain image quality better than the starting-point measurement?

  • How does a conditional-only DiffusionNFT baseline perform when it uses the same batch-averaged sample weights as MCReg?

  • Can the main experiments be repeated with different training seeds, especially the FLUX.2 Klein comparisons?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bx8a

Thank you for the constructive suggestions. We added the requested diagnostics, controls, and training repeats. Revisions are marked in blue.

W1/Q1. Closure during denoising.

We originally measured closure at pure noise because the state is independent of the prompt, so the prompt posterior equals the known prior. At intermediate times, states carry prompt information, requiring a state-dependent posterior under the rollout distribution.

We therefore evaluated seven additional denoising times on SD3.5-Medium, estimating the posterior with two ResNet18 classifiers [1] calibrated by temperature scaling [2]. Under each rollout distribution, both checkpoints are evaluated on identical held-out states with the same estimated posterior weights.

Table 6 in Appendix D.4 reports the following estimated closure risks in units of 10−310^{-3} (lower is better). Denoising proceeds from t=1t=1 toward t=0t=0.

Original-generated statesMCReg-generated states
Time ttOriginalMCRegOriginalMCReg
1181.418.98181.418.98
0.8973113.3142.821665.93
0.741428.2330.0455.4719.98
0.47697.9977.22814.426.962
0.26083.3082.6795.0863.047
0.14651.9171.4802.5671.698
0.081051.2821.0151.4711.056
0.0089290.69280.60960.44880.4108

MCReg reduces estimated closure at all five sampled middle-to-late times under both rollout distributions. The two earlier times also improve under MCReg rollouts but are exceptions under Original rollouts. The same checkpoint pair improves all six DrawBench quality metrics at w=3w=3. For this pair, endpoint and middle-to-late closure give the same ranking as the quality scores; intermediate measurements show where closure improves. Section 4.4 summarizes the findings; Appendix D.4 gives the estimation protocol.

W2/Q2. Conditional-only training with batch-mean weights.

We added the requested conditional-only baseline with batch-mean weights. Table 4 compares both conditional weighting rules with and without the same unconditional term, retaining each conditional weighting rule and coefficient.

Table 4 reports mean ± SD across three training runs on SD3.5-Medium at w=3w=3.

CC weights+N+NPick-a-PicDrawBench
PickScoreCLIPHPSv2.1PickScoreCLIPHPSv2.1ImgRwdUniRewardDeQA
Native—22.771±0.10730.813±0.21934.352±0.23323.305±0.09129.912±0.03933.255±0.1011.486±0.0180.672±0.0014.241±0.023
Batch-mean—22.360±0.10830.899±0.29533.383±0.69422.975±0.10929.983±0.03332.230±0.5721.439±0.0280.659±0.0063.988±0.103
Native✓23.411±0.07731.968±0.22236.797±0.09723.801±0.05930.331±0.02334.996±0.0061.558±0.0070.691±0.0044.329±0.019
Batch-mean✓23.326±0.02331.721±0.02836.484±0.10423.690±0.04230.191±0.18034.676±0.1741.564±0.0160.691±0.0034.339±0.020

Batch-mean weighting alone raises CLIP but lowers PickScore and HPSv2.1 on both benchmarks. Adding unconditional supervision improves all nine metric means under either weighting rule, as reported in the complete Table 4. These controls support unconditional supervision as the key component; batch-mean conditional weighting is an implementation choice. Section 4.4 discusses the results, and Appendix E.2 specifies the controls.

W3/Q3. Independent training repeats, especially FLUX.

We repeated training with fixed hyperparameters. Tables 2 and 3 now report means and standard deviations across three training runs. The following excerpts show Table 2 results at the same guidance scale, w=3w=3.

Pick-a-PicDrawBench
MethodPickScoreCLIPHPSv2.1PickScoreCLIPHPSv2.1
SD3.5-Medium
DiffusionNFT22.771±0.10730.813±0.21934.352±0.23323.305±0.09129.912±0.03933.255±0.101
MCReg23.391±0.05231.680±0.27036.715±0.11823.766±0.01930.158±0.07134.864±0.155
FLUX.2 Klein Base
DiffusionNFT23.416±0.08329.727±0.11936.023±0.15823.911±0.03728.994±0.11434.207±0.172
MCReg23.512±0.04629.795±0.15036.278±0.33123.995±0.07128.998±0.14334.525±0.477

At w=3w=3, the Table 2 means improve on all nine quality metrics with MCReg on both backbones. We also repeated OPD student training with fixed teachers. In Table 3, MCReg improves seven of eight quality means over Original in DiffusionOPD and six of eight in DanceOPD, and seven over PDM in both frameworks. Tables 2 and 3 report endpoint closure over the same runs as the quality scores. Appendix D.1 provides the training and evaluation protocols.

References

[1] Kaiming He et al. Deep Residual Learning for Image Recognition. CVPR, 2016.

[2] Chuan Guo et al. On Calibration of Modern Neural Networks. ICML, 2017.

Dear Reviewer bx8a,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

Thank you for the time and care you have devoted to reviewing our paper. We appreciate your constructive feedback and the opportunity to clarify our contribution and strengthen the experimental evidence. Below we summarize the main findings from the discussion. Revisions are marked in blue.

1. What does MCReg contribute to diffusion post-training? (dxka, iYc6)

Conditioning dropout in supervised fine-tuning fits the same target with and without the prompt. We ask what the unconditional branch should learn when reinforcement learning updates the conditional predictions of a diffusion model. For DiffusionNFT, we recover a reward-dependent regression target from the conditional loss. At a fixed update, its mean given the noisy state is the optimal unconditional prediction. MCReg’s normalized objective equals the expected conditioning-dropout loss on this target, and Theorem 1 bounds marginal-closure error in terms of the excess regression risk.

In OPD, this reasoning leads us to share conditional teacher targets under the student’s rollout distribution. Teacher-null instead uses the teacher’s unconditional prediction, which can reflect a different prompt distribution. With the same teachers and training budget, MCReg scores higher on six of eight metrics in DiffusionOPD and seven of eight in DanceOPD (Table 8). MCReg also achieves higher mean guided DrawBench PickScore and HPSv2.1 than Positive-Direction Matching in both frameworks (Table 3).

2. Are the DiffusionNFT gains attributable to unconditional supervision? (bx8a Q2, jC7x Q1, dxka Q2)

Table 4 combines the two requested controls. Changing conditional weights alone has mixed effects. Adding unconditional supervision while retaining the original conditional objective and weights improves all nine metric means on SD3.5-Medium at w=3w=3. These gains also hold under batch-mean conditional weighting, supporting unconditional supervision as the key component and batch-mean weighting as an implementation choice.

3. Does closure improve during denoising, and how closely does it track quality? (bx8a Q1, jC7x Q3)

MCReg lowers estimated closure error at all five sampled middle-to-late denoising times under both baseline and MCReg rollouts on SD3.5-Medium (Table 6). The quality gains vary across backbones. At w=3w=3, lower mean endpoint closure error accompanies mean DrawBench PickScore gains of 0.46 on SD3.5-Medium and 0.08 on FLUX (Table 2). Section 3.1 explains how both conditional accuracy and branch consistency contribute to guided prediction accuracy.

4. How do results vary across training runs and guidance scales? (bx8a Q3, iYc6 Q3)

Tables 2–3 report means and standard deviations across three training runs at fixed guidance. For comparisons across guidance scales, Table 11 reports each metric’s highest observed score across three training runs and the common grid w∈{1,2,3,4,5}w\in\{1,2,3,4,5\}. MCReg reaches higher peak PickScore on both benchmarks and both reward-post-trained backbones.

5. Does explicit CFG remain useful relative to guidance distillation? (jC7x Q2)

On each reported FLUX.2 Klein Base MCReg checkpoint, raising inference guidance from w=1w=1 to w=3w=3 improves all nine metrics (Table 12). Table 13 compares MCReg with students that absorb guidance from the baseline RL teacher on FLUX or the same task teachers in DanceOPD. The teachers supply targets at w=3w=3, and the students use w=1w=1 at inference. MCReg evaluated at w=3w=3 scores higher on all nine metrics on FLUX and seven of eight in DanceOPD.

We appreciate dxka’s confirmation that both concerns were well addressed and their continued support. We also thank iYc6 for acknowledging that their questions were largely addressed and for their willingness to reconsider their assessment.

Thank you again for your thoughtful feedback and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.