- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper decompose DMD term to CFG-augmentation (CA) term and a DM term. Since the DM term requires expensive online fake training, the papers investigate wether other losses can replace the DM term.
First, to explain why CA can works they show in a 2D Gaussian-mixture toy that finer condition groupings give more mode-specific classifier-score directions. In FLUX.2-klein, interpolating toward the classifier score of the original prompt, rather than an LLM-coarsened one, delays collapse.
Following this, they ask whether stabilizing CA requires DMD's online fake-score network. Keeping the CA term, they compare two families of stabilizers.
- Distribution matching: DM with TTUR, IDA, and a new sample-wise Adaptive TE regularizer.
- Transition anchoring: the student's transitions at its own states are regressed onto frozen-teacher ODE steps. The targets are a K-step rollout (Traj), a single Euler step, i.e. velocity matching (Vel), or a CTM-style consistency target (Cons).
A final-step perceptual loss reduces oversaturation.
Experiments with 4-step students on SD3.5-M/L and FLUX.2-klein-base-4B/9B show that both families prevent CA-only collapse. Anchoring gives higher preference scores (HPS, PickScore) at 2.7–4.0× lower per-update cost than DM with TTUR 5. The DM variants, and especially the hybrid DM+Vel, stay closer to the teacher distribution (lower FID-T).
Critical strengths
- An actionable finding. CA can be stabilized without an online fake-score network. Anchoring to frozen-teacher steps at the student's own states prevents collapse on four modern text-to-image models, even with one-step velocity matching, at 2.7–4.0× lower per-update cost than DM at TTUR 5 (Table 2).
- A broad, candid comparison. Four backbones, seven stabilizers, six metrics and a long-horizon run (Fig. 5). The trade-off between preference scores and teacher agreement is reported openly, along with useful negative results (e.g., Table 5).
- Well written.
Critical weaknesses
1. The evaluation cannot establish which stabilizer is better.
- Biased metrics. The case for anchoring rests on HPS and PickScore, which are known to reward oversaturation (DRaFT; Pref-GRPO), the very artifact CA produces (Sec. 4.2).
- Students exceed the teacher's HPS on three of four models, consistent with over-guidance.
- There is no human study, compositional benchmark (GenEval, DPG), or real-image FID/recall.
- On teacher agreement, DM wins. The hybrid DM+Vel, which keeps the fake score, has the best FID-T on all four models. At TTUR 1 it costs about as much as Traj (28.0 vs 29.0 s per update), so the speedup holds only against DM at TTUR 5.
- Weak controls. Hyperparameters are tuned per method with an unstated validation metric. On SD3.5-L, DM, DM+IDA and DM+TE use a 10–20× smaller learning rate than anchoring. Results are single runs over short horizons, with no comparison to released distilled models (TDM, SenseFlow, DMD2).
2. Limited novelty, and the analysis adds little.
- The toy result follows from known guidance theory (Wu et al., ICML 2024; Chidambaram et al., NeurIPS 2024; neither cited).
- The interpolation experiment confounds information content with prompt mismatch.
- The SDS line. The paper dates score distillation to VSD rather than DreamFusion. In this line, CSD isolates the classifier-score term, and NFSD (Katzir et al., ICLR 2024) decomposes the guided score into condition and domain terms and stabilizes it without a learned fake score.
I am open to raising my score after the rebuttal
Questions for authors
-
How were hyperparameters selected, and with which validation metric? Can you compare anchoring and DM on SD3.5-L at matched learning rates?
-
Can you add a human preference study, or GenEval/DPG with real-image FID/recall and a saturation statistic, to separate quality from over-guidance?
-
What is the run-to-run variance, and do the anchoring variants stay stable beyond 10K updates?
-
Which teacher configuration (CFG scale, steps) produces the anchoring targets? If it uses CFG, how should Fig. 19 (Traj without CA) be read?
-
In the interpolation experiment, what is the student conditioned on? Can you add a control with identical student and guidance prompts?
-
NFSD stands to VSD roughly as anchoring stands to DMD.
- It splits the guided score into condition and domain terms (VSD ≈ domain term + scaled condition term).
- It replaces VSD's learned auxiliary network with a training-free domain proxy.
How does your analysis relate to this? How would such a training-free "pseudo-DM" term compare with anchoring and DM as a stabilizer?
4 comments
Response to Reviewer 4qek (2/3)
W1/Q1: Hyperparameter selection and learning-rate comparison
Q1 How were hyperparameters selected, and with which validation metric? Can you compare anchoring and DM on SD3.5-L at matched learning rates?
We provide the answer twofold:
- We prioritized training stability, once training is stabilized in our evaluated setting (i.e. 1k), we select hyperparameters through a joint evaluation of qualitative samples and metrics (HPS, CLIP, IR, Pick and FID).
- We tune the lr of SD35L on DM to provide a competitive baseline. We previously swept the student and fake-score learning rates for CA+DM at TTUR = 5 on SD3.5-L (we add Appendix C.3, Figure 18), and observe early collapse when increasing learning rates (consistent with the finding in Senseflow when applying vanilla DM with larger TTUR). We therefore selected
1e-7for the student and5e-7for the fake-score model for their training stability and competitive performance.
At a shared student learning rate of 2e-6, CA+Traj and CA+DM+Vel remained stable, while the tested CA+DM runs deteriorated. The table reports COCO10K metrics for the stable configurations.
| Method | Student LR | Fake-score LR | TTUR | HPSv2.1 ↑ | CLIPScore ↑ | FID-T ↓ | Stability |
|---|---|---|---|---|---|---|---|
| CA+DM (selected LR) | 1e-7 | 5e-7 | 5 | 0.2885 | 31.70 | 13.95 | Stable on 1K |
| CA+DM (matched LR) | 2e-6 | tuned (Figure 18) | 5 | — | — | — | Collapsed around 400-600 |
| CA+DM+Vel | 2e-6 | 2e-5 | 1 | 0.3068 | 31.76 | 7.69 | Stable on 1K |
| CA+Traj | 2e-6 | — | — | 0.3141 | 31.74 | 9.97 | Stable on 1K |
W1. Additional Baseline (TDM, DMD2, Senseflow)
Thank you for pointing, CA+DM baseline inherits TDM. On SD3.5-L, we add DMD2 and evaluate the released ckpt of Senseflow on COCO-10K.
| Method | HPSv2.1 ↑ | CLIPScore ↑ | PickScore ↑ |
|---|---|---|---|
| CA+DM (TDM) | 0.2885 | 31.70 | 22.66 |
| CA+DM (DMD2-style) | 0.2906 | 31.61 | 22.86 |
| CA+Traj | 0.3141 | 31.74 | 22.87 |
| CA+Cons | 0.3179 | 32.11 | 22.57 |
| Senseflow | 0.3041 | 32.34 | 22.82 |
Our original intend is to provide a controlled ablation under matched budget, thus previously omit the comparison to modern compositional sota baselines.
W2.1: Related work
Thank you for mentioning this, we will cite the guidance analyses by Wu et al. [1] and Chidambaram et al. [2], which examine guidance effects on confidence, diversity, and the sampling distribution. Our toy experiments focus on how condition granularity shapes the classifier score during distillation instead of training-free sampling setting. We study this by varying how the same mixture modes are grouped into conditions while keeping the marginal distribution fixed.
We will also correct the attribution of SDS to DreamFusion [3] and also include the discussion to NFSD [4].
References
- [1] Wu et al. Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models ICML 2024.
- [2] Chidambaram et al. What Does Guidance Do? A Fine-Grained Analysis in a Simple Setting NeurIPS 2024.
- [3] Poole et al. DreamFusion: Text-to-3D Using 2D Diffusion ICLR 2023.
- [4] Katzir et al. Noise-Free Score Distillation ICLR 2024.
W1/Q3: Run-to-run variance and long-horizon stability
What is the run-to-run variance, and do the anchoring variants stay stable beyond 10K updates?
Thank you for the suggestion. On Flux.2, We evaluate across three training seeds on COCO10K on two represented methods (DM,Vel) on two families.
| Method | HPSv2.1 ↑ (mean ± std) | CLIPScore ↑ (mean ± std) | PickScore ↑ (mean ± std) |
|---|---|---|---|
| CA+DM | 0.2923 ± 0.0009 | 31.08 ± 0.04 | 21.859 ± 0.006 |
| CA+Vel | 0.3084 ± 0.0006 | 30.73 ± 0.05 | 22.286 ± 0.017 |
We further extend the CA+Vel to 12K iterations.
| Student updates | HPSv2.1 ↑ | CLIPScore ↑ | PickScore ↑ |
|---|---|---|---|
| 4K | 0.3069 | 30.83 | 22.148 |
| 8K | 0.3055 | 30.69 | 22.201 |
| 10K | 0.3102 | 30.88 | 22.257 |
| 12K | 0.3099 | 30.57 | 22.288 |
For CA+Vel, the single teacher step supervision manages to stabilize after 10k student update.
Response to Reviewer 4qek (3/3)
W1/Q4: Teacher guidance configuration.
Which teacher configuration (CFG scale, steps) produces the anchoring targets? If it uses CFG, how should Fig. 19 (Traj without CA) be read?
For Traj, it's CFG w=4 and K=4 (details in Tab 6). Fig 19 aims to ablate the TA stabilizer w / w/o CA. "Traj w/o CA" removes only the explicit CA loss, rather than teacher guidance.
W2/Q5: Prompt mismatch in score interpolation.
In the interpolation experiment, what is the student conditioned on? Can you add a control with identical student and guidance prompts?
In Fig. 3, the student is conditioned on the fine prompt P. To further reduce to influence of prompt mismatch, we cross the student prompt and the CA prompt using the fine prompt P and its coarse counterpart Pc.
| Student prompt | CA prompt | HPS retention nAUC ↑ | Collapse update ↑ | HPS plateau (50–100) |
|---|---|---|---|---|
| P | Pc | 0.1354 | 105 | 0.2383 |
| P | P | 0.4458 | 255 | 0.2969 |
| Pc | Pc | 0.1714 | 105 | 0.2680 |
| Pc | P | 0.3055 | 195 | 0.2690 |
In Pc (coarse) as the student prompt, CA with P fine prompt still show consistent improvement (nAUC: 0.3055 vs. 0.1714) despite prompt mismatch (Pc on student and P on CA). And for identical Prompt pair, (P,P) surpass (Pc,Pc) (nAUC: 0.4458 vs. 0.1714).
W2/Q6: Relation to NFSD
How does your analysis relate to NFSD? How would such a training-free "pseudo-DM" term compare with anchoring and DM as a stabilizer?
We first make the following comparison to NFSD in formulation during distillation:
| CA: distillation engine | NFSD: domain correction | DM | TA |
|---|---|---|---|
| local velocity, Trajectory ODE, CTM |
Here, is a negative prompt, which NFSD applies in high-noise level. We then tested an NFSD-inspired regularizer alongside CA in shared few-step T2I training. We report the highest score for CA+NFSD before collapse.
| Method | HPS ↑ | CLIP ↑ | Stability |
|---|---|---|---|
| CA+NFSD | 0.2664 | 27.54 | Collapsed at 300 updates |
| CA+Traj | 0.3101 | 30.90 | Stable through 1K |
The observed collapse shows that the tested NFSD adaptation failed to sustain shared-generator training under this setup. NFSD [4] uses the unconditional prediction at low noise and subtracts a negative-prompt prediction at higher noise. Such term remain effective in the per-instance optimization secenario in 3D, due to NeRF's strong structural bias and the task's objective on a single prompt instance, where the negative prompt matters and provide more structural guidance in high noise level. In shared-generator training across varying samples and prompt, DM instead learns the evolving student score online, while TA constrains the student's transitions using frozen-teacher targets.
We sincerely thank you for your time and valuable suggestions. We are open to any follow-up discussion if you need any.
Dear Reviewer 4qek,
Thank you for your valuable feedback and questions. As the discussion period is near to a close, we would like to kindly extend the reminder and would greatly appreciate your response, at your convenience.
Sincerely, The Authors
Response to Reviewer 4qek (1/3)
We sincerely thank you for the detailed and thoughtful feedback, here we provide our response for your reference.
W1/Q2: On Evaluation bias and over-guidance
Response We agree that higher preference-model scores alone do not establish better image quality. We therefore add human judgments, GenEval, real-image FID, and mean HSV saturation to isolate quality from potential over-guidance.
Human study
We conducted human study (233 engagements) of overall quality and photographic realism for CA+DM, CA+Traj, and CA+Traj+Perc. Votes were nearly even between CA+DM and CA+Traj, while raters favored CA+Traj+Perc.
Evaluation of GenEval, fidelity, and saturation
We also report GenEval and FID to COCO real images. Following APG, we average the HSV saturation channel within each image, then average across images.
It is observed that both vanilla DM (0.4960) and vanilla Traj (0.4933) incur similar mean saturation, both above the teacher reference, due to the signal of CA. When equiped with Perceptual losses, saturation is alleviated (4.933 to 4.271), and FID-R reduce from 40.13 to 31.70, reaching comparable result to DM variants (30.81.)
We acknowledge that DM retains slightly better teacher agreement and FID-R, standalone Traj and DM receive nearly balanced human votes despite their different preference-model scores. And final-step perceptual supervision improves realism, reduces saturation, and improves FID-R.
W1 On Teacher agreement and efficiency.
Response We aknowledge the edge of DM on teacher agreement and will revise to script to place the speedup correctly. We'd like to kindly clarify that: