acceptodds
Under review as a conference paper at ICLR 2027

Analyzing and Stabilizing Classifier Score Distillation for Text-to-Image Synthesis

Abstract

Distribution Matching Distillation (DMD) accelerates diffusion and flow-matching models by compressing their iterative sampling process into a few steps. Recent work shows that its gradient can be decomposed into a DM term and a CFG-augmentation (CA) term, with CA providing a particularly strong distillation signal in text-to-image generation. Yet why CA is effective, and what constraints are needed to stabilize it, remain underexplored. To analyze its effectiveness, we first conduct synthetic experiments illustrating that finer condition groupings yield more discriminative classifier-score directions. We then probe the corresponding score-space mechanism in text-to-image models through interpolation, where retaining more fine-condition score information prolongs the effective distillation phase. These findings further support repositioning classifier score as the central distillation signal, motivating a reconsideration of stabilization within the classifier score distillation (CSD) framework. Within this framing, we ask whether sustaining CA necessarily requires online estimation of the evolving student distribution. We compare distribution matching, including fake and student side tracking strategies, with transition anchoring, which instead constrains student transitions using frozen teacher dynamics without maintaining an online fake-score model. We further equip transition-anchoring variants with a final-step perceptual loss to alleviate CA-induced oversaturation and local-detail degradation. Our empirical results show that both families improve training stability. On FLUX.2, transition anchoring variants achieve competitive performance with a 2.7-4.0x training speedup over vanilla DM, while revealing trade-offs between preference-oriented quality and alignment with the teacher distribution.

Primary area:
generative models
Keywords:
Diffusion models, Step Distillation, Classifier Free Guidance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4qekOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper decompose DMD term to CFG-augmentation (CA) term and a DM term. Since the DM term requires expensive online fake training, the papers investigate wether other losses can replace the DM term.

First, to explain why CA can works they show in a 2D Gaussian-mixture toy that finer condition groupings give more mode-specific classifier-score directions. In FLUX.2-klein, interpolating toward the classifier score of the original prompt, rather than an LLM-coarsened one, delays collapse.

Following this, they ask whether stabilizing CA requires DMD's online fake-score network. Keeping the CA term, they compare two families of stabilizers.

  • Distribution matching: DM with TTUR, IDA, and a new sample-wise Adaptive TE regularizer.
  • Transition anchoring: the student's transitions at its own states are regressed onto frozen-teacher ODE steps. The targets are a K-step rollout (Traj), a single Euler step, i.e. velocity matching (Vel), or a CTM-style consistency target (Cons).

A final-step perceptual loss reduces oversaturation.

Experiments with 4-step students on SD3.5-M/L and FLUX.2-klein-base-4B/9B show that both families prevent CA-only collapse. Anchoring gives higher preference scores (HPS, PickScore) at 2.7–4.0× lower per-update cost than DM with TTUR 5. The DM variants, and especially the hybrid DM+Vel, stay closer to the teacher distribution (lower FID-T).

Critical strengths
  1. An actionable finding. CA can be stabilized without an online fake-score network. Anchoring to frozen-teacher steps at the student's own states prevents collapse on four modern text-to-image models, even with one-step velocity matching, at 2.7–4.0× lower per-update cost than DM at TTUR 5 (Table 2).
  2. A broad, candid comparison. Four backbones, seven stabilizers, six metrics and a long-horizon run (Fig. 5). The trade-off between preference scores and teacher agreement is reported openly, along with useful negative results (e.g., Table 5).
  3. Well written.
Critical weaknesses

1. The evaluation cannot establish which stabilizer is better.

  • Biased metrics. The case for anchoring rests on HPS and PickScore, which are known to reward oversaturation (DRaFT; Pref-GRPO), the very artifact CA produces (Sec. 4.2).
    • Students exceed the teacher's HPS on three of four models, consistent with over-guidance.
    • There is no human study, compositional benchmark (GenEval, DPG), or real-image FID/recall.
  • On teacher agreement, DM wins. The hybrid DM+Vel, which keeps the fake score, has the best FID-T on all four models. At TTUR 1 it costs about as much as Traj (28.0 vs 29.0 s per update), so the speedup holds only against DM at TTUR 5.
  • Weak controls. Hyperparameters are tuned per method with an unstated validation metric. On SD3.5-L, DM, DM+IDA and DM+TE use a 10–20× smaller learning rate than anchoring. Results are single runs over short horizons, with no comparison to released distilled models (TDM, SenseFlow, DMD2).

2. Limited novelty, and the analysis adds little.

  • The toy result follows from known guidance theory (Wu et al., ICML 2024; Chidambaram et al., NeurIPS 2024; neither cited).
  • The interpolation experiment confounds information content with prompt mismatch.
  • The SDS line. The paper dates score distillation to VSD rather than DreamFusion. In this line, CSD isolates the classifier-score term, and NFSD (Katzir et al., ICLR 2024) decomposes the guided score into condition and domain terms and stabilizes it without a learned fake score.

I am open to raising my score after the rebuttal

Questions for authors
  1. How were hyperparameters selected, and with which validation metric? Can you compare anchoring and DM on SD3.5-L at matched learning rates?

  2. Can you add a human preference study, or GenEval/DPG with real-image FID/recall and a saturation statistic, to separate quality from over-guidance?

  3. What is the run-to-run variance, and do the anchoring variants stay stable beyond 10K updates?

  4. Which teacher configuration (CFG scale, steps) produces the anchoring targets? If it uses CFG, how should Fig. 19 (Traj without CA) be read?

  5. In the interpolation experiment, what is the student conditioned on? Can you add a control with identical student and guidance prompts?

  6. NFSD stands to VSD roughly as anchoring stands to DMD.

    • It splits the guided score into condition and domain terms (VSD ≈ domain term + scaled condition term).
    • It replaces VSD's learned auxiliary network with a training-free domain proxy.

    How does your analysis relate to this? How would such a training-free "pseudo-DM" term compare with anchoring and DM as a stabilizer?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4qek (1/3)

We sincerely thank you for the detailed and thoughtful feedback, here we provide our response for your reference.

W1/Q2: On Evaluation bias and over-guidance

Question\textcolor{#1565C0}{\text{Question}} Q2 Can you add a human preference study, or GenEval/DPG with real-image FID/recall and a saturation statistic, to separate quality from over-guidance?

Response\textcolor{#2E7D32}{\text{Response}} We agree that higher preference-model scores alone do not establish better image quality. We therefore add human judgments, GenEval, real-image FID, and mean HSV saturation to isolate quality from potential over-guidance.

Human study

We conducted human study (233 engagements) of overall quality and photographic realism for CA+DM, CA+Traj, and CA+Traj+Perc. Votes were nearly even between CA+DM and CA+Traj, while raters favored CA+Traj+Perc.

ComparisonQualityRealism
CA+Traj vs. CA+DM51.1%47.2%
CA+Traj+Perc. vs. CA+Traj55.8%64.4%
CA+Traj+Perc. vs. CA+DM57.9%60.1%

Evaluation of GenEval, fidelity, and saturation

We also report GenEval and FID to COCO real images. Following APG, we average the HSV saturation channel within each image, then average across images.

MethodGenEval ↑FID-R ↓HSV Sat. ↓
Teacher0.78223.630.3530
CA+DM0.75930.810.4960
CA+Traj0.77440.130.4933
CA+Traj+Perc.0.76131.700.4271

It is observed that both vanilla DM (0.4960) and vanilla Traj (0.4933) incur similar mean saturation, both above the teacher reference, due to the signal of CA. When equiped with Perceptual losses, saturation is alleviated (4.933 to 4.271), and FID-R reduce from 40.13 to 31.70, reaching comparable result to DM variants (30.81.)

We acknowledge that DM retains slightly better teacher agreement and FID-R, standalone Traj and DM receive nearly balanced human votes despite their different preference-model scores. And final-step perceptual supervision improves realism, reduces saturation, and improves FID-R.

W1 On Teacher agreement and efficiency.

Question\textcolor{#1565C0}{\text{Question}} The hybrid DM+Vel, which keeps the fake score, has the best FID-T on all four models. At TTUR 1 it costs about as much as Traj (28.0 vs 29.0 s per update), so the speedup holds only against DM at TTUR 5.

Response\textcolor{#2E7D32}{\text{Response}} We aknowledge the edge of DM on teacher agreement and will revise to script to place the speedup correctly. We'd like to kindly clarify that:

  1. Despite DM's capacity in teacher agreement, TA's variants like Vel can be function as a lightweight accelerator: The reason for TTUR=1 to be stable is the combination of Vel (CA+DM+Vel), whereas vanilla DM TTUR=1 quickly collapse. And the lowest FID is obtained with the combination of DM+Vel instead of DM (TTUR=5) alone.
  2. TA is not far behind on teacher agreement. TA+perceptual loss also reach competitive agreement, CA+Traj+Perc reach 14.12 FID to CA+DM (TTUR=5, 13.97).
MethodCOCO10K FID-T ↓
CA+Traj20.37
CA+Traj+Perc. (final)14.12
CA+DM (TTUR=5)13.97
CA+DM+Vel (TTUR=1)9.76
CA+DM (TTUR=1)Collapse around 500-600 iteration
AuthorsOpenReview ↗

Response to Reviewer 4qek (2/3)

W1/Q1: Hyperparameter selection and learning-rate comparison

Question\textcolor{#1565C0}{\text{Question}} Q1 How were hyperparameters selected, and with which validation metric? Can you compare anchoring and DM on SD3.5-L at matched learning rates?

Response\textcolor{#2E7D32}{\text{Response}} We provide the answer twofold:

  1. We prioritized training stability, once training is stabilized in our evaluated setting (i.e. 1k), we select hyperparameters through a joint evaluation of qualitative samples and metrics (HPS, CLIP, IR, Pick and FID).
  2. We tune the lr of SD35L on DM to provide a competitive baseline. We previously swept the student and fake-score learning rates for CA+DM at TTUR = 5 on SD3.5-L (we add Appendix C.3, Figure 18), and observe early collapse when increasing learning rates (consistent with the finding in Senseflow when applying vanilla DM with larger TTUR). We therefore selected 1e-7 for the student and 5e-7 for the fake-score model for their training stability and competitive performance.

At a shared student learning rate of 2e-6, CA+Traj and CA+DM+Vel remained stable, while the tested CA+DM runs deteriorated. The table reports COCO10K metrics for the stable configurations.

MethodStudent LRFake-score LRTTURHPSv2.1 ↑CLIPScore ↑FID-T ↓Stability
CA+DM (selected LR)1e-75e-750.288531.7013.95Stable on 1K
CA+DM (matched LR)2e-6tuned (Figure 18)5———Collapsed around 400-600
CA+DM+Vel2e-62e-510.306831.767.69Stable on 1K
CA+Traj2e-6——0.314131.749.97Stable on 1K

W1. Additional Baseline (TDM, DMD2, Senseflow)

Response\textcolor{#2E7D32}{\text{Response}} Thank you for pointing, CA+DM baseline inherits TDM. On SD3.5-L, we add DMD2 and evaluate the released ckpt of Senseflow on COCO-10K.

MethodHPSv2.1 ↑CLIPScore ↑PickScore ↑
CA+DM (TDM)0.288531.7022.66
CA+DM (DMD2-style)0.290631.6122.86
CA+Traj0.314131.7422.87
CA+Cons0.317932.1122.57
Senseflow0.304132.3422.82

Our original intend is to provide a controlled ablation under matched budget, thus previously omit the comparison to modern compositional sota baselines.

W2.1: Related work

Response\textcolor{#2E7D32}{\text{Response}} Thank you for mentioning this, we will cite the guidance analyses by Wu et al. [1] and Chidambaram et al. [2], which examine guidance effects on confidence, diversity, and the sampling distribution. Our toy experiments focus on how condition granularity shapes the classifier score during distillation instead of training-free sampling setting. We study this by varying how the same mixture modes are grouped into conditions while keeping the marginal distribution fixed.

We will also correct the attribution of SDS to DreamFusion [3] and also include the discussion to NFSD [4].

References

  • [1] Wu et al. Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models ICML 2024.
  • [2] Chidambaram et al. What Does Guidance Do? A Fine-Grained Analysis in a Simple Setting NeurIPS 2024.
  • [3] Poole et al. DreamFusion: Text-to-3D Using 2D Diffusion ICLR 2023.
  • [4] Katzir et al. Noise-Free Score Distillation ICLR 2024.

W1/Q3: Run-to-run variance and long-horizon stability

Question\textcolor{#1565C0}{\text{Question}} What is the run-to-run variance, and do the anchoring variants stay stable beyond 10K updates?

Response\textcolor{#2E7D32}{\text{Response}} Thank you for the suggestion. On Flux.2, We evaluate across three training seeds on COCO10K on two represented methods (DM,Vel) on two families.

MethodHPSv2.1 ↑ (mean ± std)CLIPScore ↑ (mean ± std)PickScore ↑ (mean ± std)
CA+DM0.2923 ± 0.000931.08 ± 0.0421.859 ± 0.006
CA+Vel0.3084 ± 0.000630.73 ± 0.0522.286 ± 0.017

We further extend the CA+Vel to 12K iterations.

Student updatesHPSv2.1 ↑CLIPScore ↑PickScore ↑
4K0.306930.8322.148
8K0.305530.6922.201
10K0.310230.8822.257
12K0.309930.5722.288

For CA+Vel, the single teacher step supervision manages to stabilize after 10k student update.

AuthorsOpenReview ↗

Response to Reviewer 4qek (3/3)

W1/Q4: Teacher guidance configuration.

Question\textcolor{#1565C0}{\text{Question}} Which teacher configuration (CFG scale, steps) produces the anchoring targets? If it uses CFG, how should Fig. 19 (Traj without CA) be read?

Response\textcolor{#2E7D32}{\text{Response}} For Traj, it's CFG w=4 and K=4 (details in Tab 6). Fig 19 aims to ablate the TA stabilizer w / w/o CA. "Traj w/o CA" removes only the explicit CA loss, rather than teacher guidance.

W2/Q5: Prompt mismatch in score interpolation.

Question\textcolor{#1565C0}{\text{Question}} In the interpolation experiment, what is the student conditioned on? Can you add a control with identical student and guidance prompts?

Response\textcolor{#2E7D32}{\text{Response}} In Fig. 3, the student is conditioned on the fine prompt P. To further reduce to influence of prompt mismatch, we cross the student prompt and the CA prompt using the fine prompt P and its coarse counterpart Pc.

Student promptCA promptHPS retention nAUC ↑Collapse update ↑HPS plateau (50–100)
PPc0.13541050.2383
PP0.44582550.2969
PcPc0.17141050.2680
PcP0.30551950.2690

In Pc (coarse) as the student prompt, CA with P fine prompt still show consistent improvement (nAUC: 0.3055 vs. 0.1714) despite prompt mismatch (Pc on student and P on CA). And for identical Prompt pair, (P,P) surpass (Pc,Pc) (nAUC: 0.4458 vs. 0.1714).

W2/Q6: Relation to NFSD

Question\textcolor{#1565C0}{\text{Question}} How does your analysis relate to NFSD? How would such a training-free "pseudo-DM" term compare with anchoring and DM as a stabilizer?

Response\textcolor{#2E7D32}{\text{Response}} We first make the following comparison to NFSD in formulation during distillation:

CA: distillation engineNFSD: domain correctionDMTA
ϵϕ(c)−ϵϕ(∅)\epsilon_\phi(c)-\epsilon_\phi(\varnothing)ϵϕ(∅)−ϵϕ(c−)\epsilon_\phi(\varnothing)-\epsilon_\phi(c_-)ϵϕ(c)−ϵψ(c)\epsilon_\phi(c)-\epsilon_\psi(c)local velocity, Trajectory ODE, CTM

Here, c−c_- is a negative prompt, which NFSD applies in high-noise level. We then tested an NFSD-inspired regularizer alongside CA in shared few-step T2I training. We report the highest score for CA+NFSD before collapse.

MethodHPS ↑CLIP ↑Stability
CA+NFSD0.266427.54Collapsed at 300 updates
CA+Traj0.310130.90Stable through 1K

The observed collapse shows that the tested NFSD adaptation failed to sustain shared-generator training under this setup. NFSD [4] uses the unconditional prediction at low noise and subtracts a negative-prompt prediction at higher noise. Such term remain effective in the per-instance optimization secenario in 3D, due to NeRF's strong structural bias and the task's objective on a single prompt instance, where the negative prompt matters and provide more structural guidance in high noise level. In shared-generator training across varying samples and prompt, DM instead learns the evolving student score online, while TA constrains the student's transitions using frozen-teacher targets.

We sincerely thank you for your time and valuable suggestions. We are open to any follow-up discussion if you need any.

Dear Reviewer 4qek,

Thank you for your valuable feedback and questions. As the discussion period is near to a close, we would like to kindly extend the reminder and would greatly appreciate your response, at your convenience.

Sincerely, The Authors

Reviewer_S2dbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors study classifier score distillation (CSD), where the CFG-augmentation (CA) term of the DMD gradient drives few-step distillation. CA gives a strong signal in text-to-image generation, but it collapses without regularization. With Gaussian-mixture experiments and score interpolation in text-to-image models, the authors show that finer conditions make CA more effective. For stabilization, they compare variants of distribution matching, which use an online fake-score model, with transition anchoring, which makes the student transitions agree with the frozen teacher’s ODE dynamics. They also add an optional final-step perceptual loss to reduce the oversaturation that CA causes.

Critical strengths
  • The paper asks a clear and useful question: does the CA signal need an online fake-score model to stay stable?
  • The analysis of condition granularity is well controlled. It gives a new explanation of why CA is effective in text-to-image distillation.
  • The comparison of stabilizers is broad and practical. Transition anchoring, which needs no fake-score model, keeps CA stable at a lower cost.
Critical weaknesses
  • The paper puts the contribution in the DMD framework, but the main method is trajectory distillation. The motivation comes from DMD (fake-score estimation, TTUR, the DM/CA split), and Sec. 5.1 continues in this direction. Then Sec. 5.2 changes the family of objectives. Transition anchoring is in effect trajectory/consistency distillation, used as a regularizer for CA. Thus the method reads as an improvement to DMD, but it is really CA together with regression to the teacher trajectory. The related problems are: 1) The introduction and the list of contributions do not state the change. 2) The related work on trajectory and consistency distillation is short, and the paper uses none of it as a baseline. All comparisons use DM variants. Guided trajectory/consistency distillation (with and without CA) and hybrids such as rCM [1] are missing.

  • The analysis (Sec. 4) does not inform the method (Sec. 5). Sec. 4 argues that CA works because fine conditions give more discriminative classifier-score directions, but Sec. 5 does not use this result. None of the stabilizers (transition anchoring, Adaptive TE, the perceptual loss) depends on condition granularity, and no experiment in Sec. 5 changes the granularity. The analysis experiments test only CA-only and CA+DM, so the paper never tests transition anchoring in the toy setting. Thus the analysis explains why CA works, but it does not show how to make CA stable, and the claim of one unified “analyze and stabilize” framework seems weak. The authors must connect the two parts. For example, they can test anchoring across granularity, or give results for each level of prompt specificity (G0–G5). If they cannot connect the two parts, I would suggest that the authors present the paper as an analysis of CA plus a comparison of stabilizers, and change the claims of contribution.

[1] Zheng et al., Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency., ICLR, 2026.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer S2db (1/2)

Thank sincerely thank you for your constructive feedback and review, here we present our response to your question.

W1: Contribution framing and related baselines

Question\textcolor{#1565C0}{\text{Question}} P1 The introduction and the list of contributions do not state the change.

Response\textcolor{#2E7D32}{\text{Response}} We agree that our framing should distinguish a CSD-centered investigation from an improvement to DMD. TA retains the CA objective while constraining student transitions through established teacher-transition losses, rather than estimating a teacher–student score discrepancy. Sec. 5.1 examines distribution-based stabilization and its score tracking requirements and limitation, thus motivating Sec. 5.2 to examine frozen-teacher transition constraints as an alternative. We will make this distinction explicit in the introduction and contributions. (Analyze the mechanism behind Classifier score and stabilize it throughout distillation)

Question\textcolor{#1565C0}{\text{Question}} P2 Guided trajectory/consistency distillation (with and without CA) and hybrids such as rCM [1] are missing.

Response\textcolor{#2E7D32}{\text{Response}} We will 1) expand the trajectory and consistency distillation related work and clarify the connections to our method. 2) Appendix Fig. 19 compares MSE-based Traj (K=4) and Cons with and without CA on FLUX.2-klein-base-4B. (w/o CA removes only the explicit CA loss, teacher guidance remain unchanged within each pair.). Here we provide the standard MS-COCO 10k comparison, along with comparison to rCM (our reproduction on Flux.2).

MethodHPSv2.1 ↑CLIPScore ↑PickScore ↑
Guided Traj (w/o CA)0.244630.8121.63
CA + Traj0.310130.9022.30
Guided Cons (w/o CA)0.216629.9920.92
CA + Cons0.308430.7922.33
rCM0.288331.0121.98
AuthorsOpenReview ↗

Response to Reviewer S2db (2/2)

W2: Connecting analysis and stabilization

Question\textcolor{#1565C0}{\text{Question}} For example, they can test anchoring across granularity, or give results for each level of prompt specificity (G0–G5). If they cannot connect the two parts, I would suggest that the authors present the paper as an analysis of CA plus a comparison of stabilizers, and change the claims of contribution.

Response\textcolor{#2E7D32}{\text{Response}} Thank you for the pointing. We agree that our analysis characterizes CA's effectiveness, while stabilization examines how this signal can be sustained: we 1) toy setting TA+CA 2) t2i setting G0-G5+TA for strengthening the connection between the two, and revised the contribution to avoid overclaiming.

1. Toy setting: finer conditioning reduces the benefit of more teacher steps in TA. In the 16-mode GMM, we compare CA+DM with CA+Traj using K=1,2,4.

GranularityCA+DM W2 ↓CA+Traj K=1 W2 ↓CA+Traj K=2 W2 ↓CA+Traj K=4 W2 ↓
G=21.1251.4511.1500.614
G=40.6480.8830.6530.469
G=80.5290.4420.3970.324
G=160.1880.1960.1910.203

W2 decreases with finer granularity at all three rollout depths, and as G increases, the reliance on trajectory steps decreases, in G=16, K=1 suffice stable distillation, we thus infer that as granularity increases, the role of distillation gradually shift from TA to classifier score (CA), the necessary steps required for TA reduces (e.g. in G=16, K=1 and K=4's W2 is close: 1.96 vs. 0.203), and TA's role gradually shift from distillation to stabilization.

2. T2I setting: the influence of G0-G5 persist when combined with TA We add additional G0-G5 with TA as stabilizer, evaluate COCO-1k at 500 iteration with bs=4. And observe consistent increase-then-decrease trend:

LevelMethodHPSv2.1 ↑CLIP ↑
G0CA+Vel0.271828.73
G1CA+Vel0.280529.59
G2CA+Vel0.291930.05
G3CA+Vel0.306930.54
G4CA+Vel0.301030.78
G5CA+Vel0.292829.97

Under CA+Vel, HPS peaks at G3, while CLIPScore peaks at G4, both decline at greater specificity, matching similar trends on the results on Fig 11 with CA-only.

The experiments above connect the informativeness of CA with the behavior of stabilizers beyond DM. We'll also update the manuscript accordingly.

We appreciate again the effort you put into the review and your constructive suggestions. We are open to follow-up discussion if you need any.

I thank the authors for the detailed response and new additional results. The newly adjusted claims & related works in the main text, results from rCM together with the new toy setting results clarify many of my concerns. Especially the new toy setting establishes the connection between granularity analysis of the paper and the main results. I am raising my score to 3.

Dear Reviewer S2db,

Thank you very much for your recognition for our response and find it useful. Your suggestions have helped us strengthen the presentation and contribution of the paper, along with the connections of the two parts.

Reviewer_Qez4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the effectiveness and stabilization of Classifier Score Distillation (CSD) for few-step text-to-image generation. Building on the decomposition of DMD into distribution matching (DM) and CFG augmentation (CA), the authors investigate why CA provides a strong distillation signal. Controlled Gaussian-mixture experiments and score-space interventions suggest that finer conditioning yields more informative guidance, although CA alone remains unstable. The paper then compares DM-based stabilization, including adaptive tracking-error regularization, with transition anchoring based on frozen teacher trajectories, velocities, and consistency targets. A final-step perceptual loss further mitigates oversaturation and local-detail degradation. Experiments across SD3.5 and FLUX.2-klein models demonstrate favorable trade-offs among training cost, preference metrics, and agreement with the teacher distribution.

Critical strengths
  • Relevant and practically useful research question. Maintaining an online fake-score model introduces substantial computational overhead and optimization complexity. The paper demonstrates that frozen teacher dynamics can provide effective stabilization, offering a useful alternative for large-scale distillation.

  • Thoughtful empirical analysis. The condition-granularity experiments provide an intuitive explanation for the effectiveness of CA. The controlled synthetic setting, analytical-score comparisons, and magnitude-normalized score interpolation strengthen the analysis beyond qualitative observations.

  • Evaluation on recent and capable models. The experiments cover four configurations across SD3.5 and FLUX.2-klein, including models with billions of parameters. This makes the findings practically relevant and provides stronger evidence than evaluation restricted to small or older architectures.

  • Comprehensive comparisons and ablations. The paper examines multiple stabilization objectives, reports both preference-oriented and teacher-distribution metrics, and includes runtime measurements, extended training curves, and perceptual-loss ablations. The results clearly expose the trade-offs rather than suggesting that one method dominates every criterion.

  • Clear presentation and appropriately qualified conclusions. The distinction between a strong condition-driven signal and the constraints needed to stabilize it is useful. The authors also acknowledge the limitations of adaptive TE and the difference between improved preference scores and closer teacher-distribution alignment.

Critical weaknesses
  • Incremental novelty. The approach builds closely on Decoupled DMD and existing trajectory-based objectives. Its distinctive contribution could be clarified further.

  • Mechanistic evidence remains suggestive. The condition-granularity experiments also change conditional generation difficulty or supervision alignment, making the proposed explanation difficult to isolate fully.

  • Evaluation could be strengthened. Quality-versus-wall-clock comparisons, multiple training seeds, and within-prompt diversity measurements would provide a more complete assessment of efficiency and robustness.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer Qez4 (1/2)

We sincerely thank you for your time and the constructive feedback. We provide the following response for your reference.

W1: Contribution and novelty

Question\textcolor{#1565C0}{\text{Question}} Incremental novelty. The approach builds closely on Decoupled DMD and existing trajectory-based objectives. Its distinctive contribution could be clarified further.

Response\textcolor{#2E7D32}{\text{Response}} Thanks for the pointing and the chance for us to reconsider our contribution.

Our contribution is a classifier score-centered investigation, distinguishing the CA-distillation signal from the constraints needed to sustain it.

First, we investigate how condition granularity affect classifier score in distillation through controlled Gaussian mixtures and norm-matched T2I score interpolation, providing an empirically validated answer to DeDMD's remaining question.

Second, we systematically compare DM and TA, finding that even local velocity anchoring can sustain CA. Its combination with DM achieves the lowest FID-T among our evaluated configurations with reduced TTUR=1. We additionally examine final-step perceptual supervision to mitigate over-guidance.

We will clarify these findings while explicitly acknowledging the established trajectory-based objectives underlying TA.

W2: Further Mechanistic evidence

Question\textcolor{#1565C0}{\text{Question}} Mechanistic evidence remains suggestive. The condition-granularity experiments also change conditional generation difficulty or supervision alignment, making the proposed explanation difficult to isolate fully.

Response\textcolor{#2E7D32}{\text{Response}} To reduce the influence of different supervision alignment on FLUX-4B, we provide further evidence on the influence of prompt mismatch between student and CA score prompts. we cross the student and CA prompts at two levels of specificity (P,Pc).

Student promptCA promptHPS retention nAUC ↑Collapse update ↑HPS plateau (50–100)
Fine (P)Coarse (Pc)0.13541050.2383
Fine (P)Fine (P)0.44582550.2969
Coarse (Pc)Coarse (Pc)0.17141050.2680
Coarse (Pc)Fine (P)0.30551950.2690

With the student conditioned on Pc, finer classifier score increases retention nAUC from 0.1714 to 0.3055 and delays collapse from 105 to 195 updates, despite introducing supervision mismatch.

We acknowledge that condition granularity is difficult to isolate fully in real scenarios. Our efforts lies on providing a pratical probe for emphasizing its influence: e.g. analytical scores eliminate teacher score-estimation error (Fig. 10), score-norm matching controls guidance magnitude (Fig. 3).

AuthorsOpenReview ↗

Response to Reviewer Qez4 (2/2)

W3: Evaluation robustness and efficiency

Question\textcolor{#1565C0}{\text{Question}} Evaluation could be strengthened. Quality-versus-wall-clock comparisons, multiple training seeds, and within-prompt diversity measurements would provide a more complete assessment of efficiency and robustness.

Response\textcolor{#2E7D32}{\text{Response}} Thank you for the suggestions. We extend the evaluation along three dimensions.

1. Training seeds. For CA+DM and CA+Vel, we evaluate across three training seeds on COCO10K.

MethodHPSv2.1 ↑CLIPScore ↑PickScore ↑
CA+DM (3 seeds)0.2923 ± 0.000931.08 ± 0.0421.859 ± 0.006
CA+Vel (3 seeds)0.3084 ± 0.000630.73 ± 0.0522.286 ± 0.017

2. Quality versus wall-clock time. On FLUX-4B, we provide the comparison with wall-clock time as measurement.

Method30 min HPS ↑30 min CLIP ↑2h HPS ↑2h CLIP ↑4h HPS ↑4h CLIP ↑
CA+DM (TTUR=5)0.257222.110.283330.820.287731.07
CA+Vel0.301130.970.309230.910.306931.08
CA+Traj0.305131.010.310730.860.310730.94

Compared to CA+DM, transition anchoring variants achieve faster convergence speed. (e.g. 0.3011 vs. 0.2572 in 30min distillation budget.)

3. Within-prompt diversity: limitation shared across distillation
On FLUX-4B, we generate 8 samples for each of 100 COCO10K prompts. We compute DINOv2 ViT-B/14 cosine distance and LPIPS over all 28 (7*8/2) image pairs per prompt, then average across prompts.

MethodDINO Diversity ↑LPIPS ↑
Teacher0.27790.6124
CA+DM0.25800.6094
CA+Traj0.25260.5964
CA+DM+Vel0.26620.5840

Compared to teacher, DM and TA variants show slight degredation on diversity, consistent with previous findings [1] [2] that on-policy based distillation might enhance quality, at the cost of reduced diversity.

[1] Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

[2] Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis

Thank you again for taking the time to review our work and for sharing your thoughtful feedback. We would be happy to discuss follow-up questions if you need any.

Dear Reviewer Qez4,

Thank you for your valuable feedback on our manuscripts. Since the discussion period is near the closing timeline, we'd want to warmly extend the reminder and would sincerely appreciate your response, at your convenience.

Best, The Authors.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response

We sincerely thank ACs and all reviewers for your time and feedback. We are encouraged by the reviewers' recognition of

  • practical stabilization of classifier score distillation using frozen teacher dynamics (Qez4, S2db, 4qek),
  • empirical insights into how condition granularity affects the effectiveness of the classifier-score signal (Qez4, S2db),
  • systematic comparisons of TA to DM across model families and scales, presenting an efficient local velocity stabilizer and a final-step perceptual refinement in TA. (Qez4, 4qek).

More importantly, we appreciate the valuable questions and suggestions raised by the reviewers. Below we summarize our responses for the ACs and reviewers.

  • On contribution and positioning (S2db, Qez4). Our contribution includes an mechanistic analysis of classifier score as a distillation signal with a systematic comparison of TA and DM as stabilizers of CA. We have revised the contribution statements to distinguish these two components. We explicitly acknowledge that TA builds on established teacher-matching objectives, and clarify our study of local velocity anchoring and final-step perceptual supervision within this framework.

  • On more controlled mechanistic analysis (Qez4, 4qek) and the connection between analysis and stabilization (S2db). We independently vary the student and CA prompts to examine the influence of supervision alignment / prompt mismatch. We also test CA with TA across condition granularities in the toy setting and prompt specificity (G0-G5) in T2I, providing more experimental analysis on the connection bewteen CA and TA.

  • On TA–DM trade-offs and practical choice (4qek). We add a human study, FID-R, GenEval, and mean HSV saturation. On Flux.2, final-step perceptual supervision reduces FID-R and saturation relative to CA+Traj. The refined variant also receives a majority of human preferences over CA+Traj and CA+DM for quality and realism. These results support TA with perceptual refinement as a practical option for efficiency and perceptual quality under limited budget, while DM combined with velocity anchoring offers a complementary option when teacher agreement is prioritized.

  • On related work (4qek, S2db). We have extended the related work on trajectory distillation, 2 guidance papers, and the SDS-line before varitional score distillation, along with comparison to NFSD.

Based on the reviewers' suggestions, we revised the paper in the following places.

  1. Introduction and contribution (S2db) . Clarify the roles of mechanistic analysis and stabilization of CA.
  2. Connection between analysis and stabilization (S2db). Add CA+TA experiments in toy (A.1) and T2I (A.2) settings using Traj and Vel.
  3. Related work (4qek, S2db). Add the two guidance papers, clarify SDS and NFSD, and expand trajectory distillation with CTM and flow-based variants.
  4. Human study and new metrics on TA vs. DM (4qek). Extend the evaluation of over-guidance and perceptual supervision (B.2).
  5. LR study of vanilla CA+DM with TTUR (4qek). Add student and fake-score learning-rate sweeps (C.3, Figure 18).
  6. More control on condition granularity (Qez4, 4qek). Add student/CA cross-prompt controls in T2I (A.2).
  7. Seed sensitivity, diversity, additional baselines, longer training, and NFSD comparison (Qez4, 4qek, S2db). Add these evaluations in C.6.

We thank the ACs and reviewers again for your time and efforts in the rebuttal. Detailed responses are provided in the individual replies.

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

We sincerely thank AC for your reminder and appreciate the efforts of reivewers on reviewing our paper. Since we are gradually near the end of discussion period, we'd like to warmly extend the reminder to reviewers, please leave a comment at your convenience, we are open to further discussion if you need any or find it useful.

Thank you again for your time and efforts.

AuthorsOpenReview ↗

Reviewer & Author Discussion Period Update

We sincerely thank the ACs and all reviewers for your time, thoughtful feedback, and efforts throughout the discussion period.

We appreciate Reviewer Qez4's positive evaluation of our work, and Reviewer S2db's follow-up response and recognition of our revisions, especially the strengthened connection between our analysis and stabilization results.

We also thank reviewers (Qez4, S2db, 4qek) for your valuable questions and suggestions, which we have provided detailed point-by-point responses. Specifically, for Reviewer 4qek's concerns, we have expanded the evaluation of TA–DM trade-offs, updated the related-work discussion, and clarified the complementary role of TA and DM, including how velocity anchoring helps stabilize DM under TTUR=1, and the limitation and instability of vanilla DM as a standalone baseline under our controlled experiments.

Depsite we may be unable to get some of the feedback in the reviewer-author discussion period, we hope our responses and clarifications help resolve the remaining concerns. We again appreciate the ACs and reviewers for your time in engaging in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.