- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies self-referenced training for text-to-image diffusion models. It uses the model's own predictions to create training signals that (1) need no external teacher, demonstration dataset or architectural change, (2) adapt to each sample's difficulty over training history, and (3) adapt to the spectral structure of the model's prediction errors. Each image–caption pair is evaluated at two noise levels with shared weights. The low-noise (easy) path is computed without gradients and serves as a reference, and the high-noise (hard) path provides the training gradient.
Main contributions:
- Advantage-Guided Diffusion (AGD): uses the difference between the prediction error at high noise and at low noise to compute a normalized mini-batch weight. This promotes the gradient of harder samples and down-weights easier ones. There is no direct distillation from easy to hard predictions.
- Temporal Advantage Memory (TAM): an EMA-based per-sample curriculum that sets the gap between the low and high noise levels from the sample's difficulty across training history.
- Spectral Advantage Decomposition (SAD): a frequency-derived spatial weight map applied to the velocity residual during loss computation.
Experiments report improvements over the base diffusion models and over training-acceleration/distillation baselines across backbones, resolutions, datasets and cross-dataset generalization.
Critical strengths
S1. The advantage-based idea is intuitive and well motivated. It brings training-history curricula, spectral bias, and sample-wise and noise-wise weighting into one framework. It is training-only, adds no parameters, and leaves inference unchanged.
S2. The experiments are broad: multiple backbones, resolutions and datasets, a wall-clock-matched comparison, and a longer-training (100k-step) comparison that rules out simple under-training of the baseline. The ablations are well designed and easy to read.
Critical weaknesses
W1. The motivation for SAD is inadequately discussed. It is unclear whether the term sg(P_e,i) in Eq. 13 is meaningful. Because of the coarse-to-fine nature of diffusion, the same frequency pattern may appear in the high-noise residual regardless of the easy counterpart. There is no visualization of M_i (Eq. 15) beyond Fig. 2, where the weight map looks similar to both the easy and hard residual maps. The gate is also positive at all frequencies and is mapped back with an inverse FFT that discards phase, so it is unclear how M_i encodes "which frequencies are hard". The paper itself acknowledges that M_i is not a localized map.
W2. The contribution of the self-referenced contrast is not isolated. In the ablation, "Abs. loss (no contrast)" already recovers most of the gain over uniform weighting (e.g., COCO FID 31.59 → 29.64, versus 29.31 for full SALD), so generic hard-example weighting may explain most of the improvement. This reading depends on Q1: the ablation appears to use |l_a| (the easy-path loss), not |l_b|, so it should be clarified or re-run. Differences are small, from single runs, and have no variance estimates.
W3. The advantage may reflect timestep and noise draw more than sample difficulty. The two paths use different timesteps and independent noise, so l_h − l_e is dominated by the noise-level gap and sampling variance. No evidence shows that A_i tracks per-pair difficulty, or that TAM's memory is populated, since samples may be seen only once on large datasets within 10k steps. Because the weights are not detached, the objective also differs from plain reweighting, and Theorem 1 is a generic SGD result that does not speak to SALD specifically.
W4. The baselines only partly match the claim. The compared methods are mostly training-acceleration or distillation approaches. Difficulty-aware weighting and curriculum methods that the introduction itself discusses (Min-SNR-style weighting, noise selection, curriculum learning, DRO), and a simple loss-weighted or focal baseline, are not compared.
W5. Training setting (post-training vs. from scratch) is not clearly stated. The main text does not say whether SALD is applied to pretrained checkpoints or used to train models from scratch. The learning rates (e.g., 1e-5 to 1e-4) and the baseline FIDs after 10k steps suggest fine-tuning of pretrained SANA, PixArt-Σ and FLUX.2 klein, but this should be stated explicitly. It affects how the results should be read: in a fine-tuning setting, gains may partly reflect domain adaptation or generic loss reweighting, and the claims should be scoped to post-training, not to "diffusion training" in general. It also affects the baseline comparison (what a 10k- or 100k-step budget means) and the interpretation of TAM's curriculum, whose gap schedule starts at 1 and grows over K steps. It is unclear whether the conclusions would hold when training from scratch.
W6. The evidence is limited and hyperparameters may be tuned on the test benchmark.
- There are no seeds or confidence intervals, and the CLIP-Score gains are often within 0.1–0.3.
- There are many hyperparameters (τ, τ_s, α, λ, Δmax, K), apparently chosen in ablations on the benchmarks used for reporting.
- Results are on small datasets (e.g., CUB-200, Flowers) and FID-based, with no alignment benchmark such as GenEval or DPG-Bench.
- The experiments fine-tune pretrained checkpoints, inferred from the learning rates and baseline FIDs. The paper should state the initialization explicitly, and the claims should be scoped to fine-tuning, not diffusion training in general.
Questions for authors
Q1. Is SALD applied to pretrained checkpoints (fine-tuning) or used to train from scratch? Please state the initialization for each backbone in the main text, and indicate whether the conclusions are expected to hold in the other setting.
Q2. At Line 436, is A_i = l_b,i − sg(l_a,i) the same as Eq. 6? If so, I expect the absolute-loss ablation to use |l_b,i|, not |l_a,i|. Could you clarify or re-run it?
Q3. Could you provide an ablation and/or visualization of the sg(P_e,i) term in Eq. 13? How does P_h,i − sg(P_e,i) compare with building the weight map directly from P_h,i? Please also visualize M_i.
Q4. How does SALD compare with hard-path-loss-only weighting, focal-style weighting and Min-SNR, with equal tuning effort and multiple seeds?
Q5. Does A_i correlate with an independent measure of sample difficulty? How often is a sample seen before its TAM memory is used?
Q6. What is the effect of detaching ω_i and M_i? How were the hyperparameter defaults chosen, and do they hold on a held-out benchmark?
9 comments
Author Rebuttal (Part 2/4)
W.3 & Q.5 Does the Advantage Track Sample Difficulty?
Response
We agree that a single is noisy; this motivates TAM's EMA (Section 2.3). We test pair-level signal, prompt specificity, and an exploratory simple–complex comparison (new Appendix D.10, Table 14 and Figure 11):
- Pair-level signal: We measure for 1,000 COCO image–caption pairs using trained SANA 0.6B. We keep the easy and hard timesteps fixed and repeat each measurement with 16 different noise pairs. Differences between image–caption pairs account for 73.3% of the variation in ; only 26.7% comes from changing the noise for the same pair. Thus, contains a repeatable signal about the pair, rather than mainly random noise. Table 3, Case 6 also supports using the same pair as the reference: it gives better FID than using a different pair (29.31 vs. 30.18).
- Prompt specificity: A pair is hard when its caption leaves the image underdetermined. On 100 DALL·E 3 pairs (pretrained SANA 0.6B), we fix the image, timesteps and 16 noise pairs, and only truncate the caption, so any change in comes from the text.
Table: Increase of over the full caption (share of images).
| Caption | |||
|---|---|---|---|
| Empty | +0.0037 (82%) | +0.0132 (80%) | +0.0117 (62%) |
| First 5 words | +0.0027 (80%) | +0.0089 (76%) | +0.0089 (56%) |
| First 10 words | +0.0015 (76%) | +0.0055 (64%) | +0.0062 (57%) |
rises monotonically as the caption becomes less specific (Wilcoxon up to 10 words at ), because the high-noise path relies more on the text (empty caption: +0.023 vs. +0.010).
Simple–complex pilot: We also evaluate five simple and five complex generated image–prompt pairs with pretrained SANA 0.6B, using the same 16 noise pairs across samples. At 25% easy-path noise (, ), complex pairs have a higher mean : 0.1466 vs. 0.0291. At 50% (, ), the means are 0.0385 vs. 0.0330, with substantial sample overlap. Figure 11 in Appendix D.10 shows three simple and three complex examples in a table with images, full generation prompts and both advantage values. These examples were selected to illustrate the higher advantage of complex pairs at both levels (shown-example means: 0.163 vs. 0.033 at 25%, 0.053 vs. 0.023 at 50%); the means above use all ten pilot pairs. This small pilot illustrates an association at these two noise levels, rather than a universal complexity ranking. It changes both image and text; the fixed-image specificity test above isolates the conditioning effect.
On the remaining points:
- TAM usage: With , every CUB-200 image is visited about 7 times within 10k steps (68 at 100k). On COCO (five captions per image), 13.8% of images are revisited by 10k steps and all by 100k. An unvisited entry has , so TAM reduces exactly to the base gap schedule (Eq. 10). Full statistics are in Appendix D.7 (Table 12).
- Non-detached weights: The gradient through adds a term with , which emphasizes samples above the batch loss (derivation in Appendix C.2). Plain reweighting with detached performs worse (Appendix D.9, Table 13).
- Theorem 1: We would like to clarify that Theorem 1 is stated for the SALD objective itself: its stochastic gradient includes all gradient paths through and , so it covers the non-detached objective, not plain reweighting. It is a convergence guarantee, not a speed claim. The new Appendix C.1 (Lemma 5, Theorem 6) analyzes SALD with the online memory: each TAM update moves the objective by at most , so the bound gains only a drift term and recovers Theorem 1 for a fixed memory.
Author Rebuttal (Part 3/4)
W.4 & Q.4 Difficulty-Aware Baselines and Multiple Seeds
Response
We thank the reviewer for this suggestion. The revised manuscript adds Min-SNR and DRO on SANA 0.6B under the same 10k-step budget (Appendix D.5, Table 8). Noise selection is represented by SpeeD (Tables 1-2).
Table: FID↓ of difficulty-aware baselines on SANA 0.6B at (Table 8).
| Method | COCO | MultiGen | Flickr8k | CUB-200 | Flowers | Avg. |
|---|---|---|---|---|---|---|
| Base | 31.59 | 31.93 | 28.83 | 26.06 | 30.64 | 29.81 |
| Min-SNR | 31.49 | 31.34 | 27.39 | 25.57 | 29.75 | 29.11 |
| DRO | 31.74 | 31.50 | 28.70 | 25.12 | 28.24 | 29.06 |
| SALD (ours) | 29.31 | 29.15 | 27.91 | 23.70 | 25.83 | 27.18 |
SALD obtains the best FID on four of five datasets, with Min-SNR lower on Flickr8k, and the best or tied-best CLIP and the best LPIPS on all five. At , SALD also leads on MAGICK (36.74 vs. 37.93 / 38.24) and DALL·E 3 (11.98 vs. 13.69 / 12.82; Table 9).
For hard-path-loss-only weighting, which is the regression analogue of focal-style weighting ( increasing in the sample loss), we select from the same four values and held-out pairs as SALD and run 3 seeds on COCO (Appendix D.5, Table 10).
Table: 3-seed comparison on SANA 0.6B, COCO () (mean ± std).
| Method | FID↓ | CLIP↑ | LPIPS↓ |
|---|---|---|---|
| weighting (best ) | 30.26 ± 0.08 | 26.72 ± 0.06 | 0.7410 ± 0.008 |
| SALD (ours) | 29.30 ± 0.02 | 27.06 ± 0.02 | 0.7377 ± 0.005 |
SALD improves over hard-path weighting by 0.96 FID, beyond the seed standard deviation (±0.08), confirming that the gain is not a single-seed effect.
W.5 & Q.1 Training Setting and Scope of Claims
Response
We thank the reviewer for raising this concern. All experiments fine-tune public pretrained checkpoints; we do not train from scratch (Appendix E). The initializations are the official SANA 0.6B and 1.6B, PixArt- ( and ) and FLUX.2 klein base (4B) checkpoints. Every baseline starts from the same checkpoint with the same data, global batch size (, Appendix D.1) and step budget. The revised Section 3 now states this initialization for every backbone, and the abstract and conclusion scope the claims to post-training of pretrained checkpoints.
On the related concerns:
- Domain adaptation: All methods, including Base, are fine-tuned on identical data for identical steps, so domain adaptation is shared; the differences reflect the objective.
- Generic reweighting: We compare against Min-SNR, DRO and hard-path loss weighting (W.4 & Q.4).
- Budget: The 10k and 100k steps are fine-tuning steps. The gap persists at 100k (Table 7; e.g., 2.58 FID for SANA 1.6B on MultiGen at both 10k and 100k).
- TAM curriculum: The ramp spans the first half of training ( is half the training steps, e.g., 5k of 10k); afterwards is sampled uniformly (Eq. 9).
- From scratch: We do not claim transfer. Early in pretraining both paths have large errors and the advantage may be less informative; we leave this to future work.
Author Rebuttal (Part 1/4)
Dear Reviewer LigD,
Thank you for your detailed review, for finding the advantage-based idea intuitive and well-motivated, and for appreciating how it unifies training-history curricula, spectral bias, and sample-wise weighting within a training-only framework, along with the wall-clock-matched and 100k-step comparisons.
The reviewer's comments raise several important questions about isolating SALD's components, the training setting, and the statistical evidence. We carefully address each of these questions below. All corresponding additions have been incorporated into the revised manuscript (Section 3 and Appendices C.2 and D.5–D.10).
W.1 & Q.3 Motivation for SAD and the Role of the Easy SpectrumResponseWe thank the reviewer for this careful analysis. Our clarification has three parts:
Table: Ablation of the easy-path spectral reference on SANA 0.6B, COCO (512).
The Ph-only gate recovers only a small part of SAD's gain.
W.2 & Q.2 Isolating the Self-Referenced ContrastResponseWe confirm that Ai=ℓb,i−sg(ℓa,i) at Line 436 is Eq. 6, with a and b denoting the easy and hard paths; the revised paper now uses e and h throughout. The submitted ablation indeed used ∣ℓe,i∣, as written. As the reviewer suggests, we re-run it with ∣ℓh,i∣ (Table 3, Case 6), selecting τ from the same grid and held-out pairs as SALD and using 3 seeds on COCO (Appendix D.5, Table 10).
Note that both absolute-loss variants remove the contrast only from AGD's weight: TAM still sets the easy-hard gap and SAD still uses Ph−sg(Pe). They are therefore not generic hard-example weighting. Generic single-path weighting is represented by Min-SNR and DRO (Table 8).
Table: Isolating the easy-hard contrast on SANA 0.6B.
The results provide two observations: