acceptodds
Under review as a conference paper at ICLR 2027

SWIFT: Self-guided Weak-to-strong dIFfusion models via Target extrapolation

Abstract

Diffusion models are typically trained to predict clean samples, while autoguidance improves generation at inference by extrapolating a trained denoiser away from a weaker one. We introduce SWIFT, which moves this weak-to-strong extrap- olation into training: each clean-sample regression target is extrapolated away from the prediction of a frozen weak denoiser, which is discarded after training. Although this deliberately shifts the target away from the posterior mean, a spectral analysis of finite-time population gradient flow shows that a nonempty range of extrapolation strengths can yield a denoiser strictly closer to the original posterior mean than matched standard training. Experiments on CIFAR-10 and ImageNet- 256 show consistent improvements across model scales and latent representations. On ImageNet-256, SWIFT improves unguided FID across every evaluated con- figuration, and the advantage persists with longer training. SWIFT also remains complementary to inference-time autoguidance and classifier free guidance: their combination gives the best FID across all tested configurations. On the DINOv3-L RAEv2 configuration, SWIFT reaches 1.15 gFID without inference-time guidance and 0.99 with retuned guidance.

Primary area:
generative models
Keywords:
diffusion models, guidance, training objective, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MJ7tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SWIFT moves weak-to-strong extrapolation from diffusion sampling into training. A frozen, separately pretrained weak denoiser defines the regression target y = (1 + alpha)x - alpha D_weak for the main model. The weak guide is discarded after training. A finite-time population kernel-gradient-flow analysis establishes a range of positive extrapolation strengths that reduces posterior-mean error relative to ordinary training under the stated assumptions. Experiments cover CIFAR-10 and 19 ImageNet-256 configurations spanning several latent representations and pixel-space models. The paper reports consistent FID improvements, compatibility with independently retuned inference guidance, persistent benefits under longer training, and compute comparisons that include guide preparation.

Critical strengths

The experiments cover multiple representations and training budgets. The finite-time analysis and accounting for guide-preparation costs are informative.

Critical weaknesses

Insufficient methodological novelty. AG already establishes weak-to-strong extrapolation. GFT’s mixed-prediction loss can be rewritten as regression toward an extrapolated target, with corresponding loss weighting. IG also explicitly studies training-time extrapolation (Section 6 of its revised version). SWIFT’s frozen weak reference changes the training dynamics, but the paper does not establish a sufficiently substantial independent contribution.

The closest comparison is missing. Appendix C evaluates IG with ordinary intermediate/final supervision rather than its extrapolated-target training variant. Consequently, the experiments do not establish the benefit over this direct precedent. Broad improvements over standard training do not resolve this contribution gap.

Questions for authors

What substantive contribution remains beyond AG, GFT, and IG’s training-time extrapolation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MJ7t

Thank you for identifying the Section 6 training-time extrapolation in IG. We added this formulation as a separate ImageNet comparison in Table 2. SWIFT improves on it from FID 4.27 to 3.77, with lower estimated total training FLOPs.

1. Direct comparison with IG's training-time extrapolation

Table 2 compares the methods on ImageNet-256 with SD-VAE+reg, 615M parameters, and matched main-model updates. All main models start from random initialization. The relevant entries are:

MethodFID ↓Training FLOPsSampling FLOPs
IG, auxiliary supervision4.791.00×1.00×
IG, Section 6 training-time extrapolation4.271.33×1.00×
SWIFT3.771.22×1.00×

SWIFT improves on IG's Section 6 variant by 0.50 FID, from 4.27 to 3.77, an approximately 11.7% relative reduction. Its estimated total training cost is also lower, 1.22× vs. 1.33×, including guide preparation. Both retain 1.00× standalone sampling FLOPs. The auxiliary-supervision variant gives 4.79; it is listed separately because it trains a different objective.

For the Section 6 comparison, the final target contains the detached EMA final-minus-intermediate prediction gap, and the intermediate output retains ordinary denoising supervision. The intermediate and final EMA outputs share one backbone forward; the cost estimate charges that extra EMA evaluation. Appendix B distinguishes the two objectives, and Appendix E.5 explains the FLOP assumptions. Tables 4–5 in Appendix C remain explicitly identified as adapted Gaussian-mixture illustrations whose IG arm uses auxiliary supervision.

AuthorsOpenReview ↗

Response to Reviewer MJ7t

2. What remains beyond the known extrapolation template?

The contribution is the choice and validation of the training reference, together with a finite-time explanation of why it helps. In x0-prediction notation, suppressing common inputs and loss weights, the two targets are

x~0SWIFT=sg⁡[x0+α(x0−Dweak)],\widetilde x_0^{\rm SWIFT}=\operatorname{sg}\left[x_0+\alpha(x_0-D_{\rm weak})\right],

x~0IG-train=x0+αsg⁡[D‾f−D‾i].\widetilde x_0^{\rm IG\text{-}train}=x_0+\alpha\operatorname{sg}\left[\overline D_f-\overline D_i\right].

SWIFT uses the observed clean label and a separately pretrained, frozen, same-condition weak predictor. IG uses an evolving EMA gap between final and intermediate outputs and retains auxiliary supervision. GFT uses the evolving shared model's unconditional output and its mixture-loss weighting. Guidance distillation uses an extrapolated strong-teacher prediction. These choices determine both the target and its evolution during training.

Appendix B states the exact GFT rewrite, including β² in Eq. (B.2), and the fixed-guide/current-prediction proportional-gradient identity in Eq. (B.3). The revised claim is grounded in the following evidence:

Reference ablation. With architecture, inputs, updates, and strength fixed, the matched ImageNet experiment gives:

Table R1. Reference-source ablation; mean ± SD across three training seeds.

ReferenceUpdated during training?FID without guidance, mean ± SDFID with tuned CFG, mean ± SDTraining seeds
Frozen unconditional weak modelNo4.40 ± 0.072.15 ± 0.053
Frozen conditional weak model, SWIFTNo3.77 ± 0.051.81 ± 0.043
Jointly trained conditional weak modelYes3.95 ± 0.061.93 ± 0.053
Online shared unconditional outputYes4.64 ± 0.082.30 ± 0.063

The frozen conditional reference has the best mean FID in both columns. Freezing the conditional guide improves 3.95 → 3.77 without guidance and 1.93 → 1.81 with tuned CFG. Matching the condition for a frozen guide improves 4.40 → 3.77 and 2.15 → 1.81. These results show that the specific reference design materially affects performance.

Finite-time analysis. Theorem 1 and Appendix A establish a nonempty interval of positive strengths with strictly lower posterior-mean error than ordinary training at matched finite optimization time under the stated RKHS population-gradient-flow assumptions. The analysis explains how an intentionally shifted regression target offsets finite-time underfitting. This is a statement about finite-time estimation with a learned weak reference, and it supplies a mechanism for the design tested above.

Guide selection. Figure 5 studies capacity and maturity. In the additional DINOv2-B comparison, earlier freezing improves student FID from 1.7845 to 1.6576 at half width and from 1.9515 to 1.6970 with GUIDE-w075, using the training-time evaluator. The choice of a smaller but competent, frozen guide is therefore supported by experiments on both capacity and maturity.

3. Benefits over the parent methods and with subsequent AG

The direct comparisons in Table 2 give SWIFT FID 3.77, versus GFT 3.92, MG 4.67, SGG training 4.01, CFG distillation 4.04, and AG distillation 4.22. In particular, SWIFT has lower FID than AG distillation at approximately 48% of its estimated training FLOPs, including preparation, with the same standalone sampling cost. Section 3.2 and Eq. (E.1) explain the target difference.

Figure 4(a) then tests whether the resulting model still benefits from AG. After independently tuning AG for each baseline/SWIFT pair, the FIDs improve 1.70 → 1.63 on DINOv2-B, 2.16 → 1.95 on SD-VAE+reg, 1.79 → 1.62 on Qwen-VAE, and 4.02 → 3.77 on PixNerd. The same weak checkpoint is used within each pair, with the common search rule described in Section 5.2 and Appendix E.5. Thus the improvement persists after baseline AG is optimized.

These results establish practical benefits for both standalone training and subsequent guided sampling, in addition to the direct improvement over IG's closest training-time formulation.

Reviewer_QGEbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SWIFT replaces the regression target with an extrapolated target x₀ + α(x₀ − D_weak(x_t)), where D_weak is a frozen, smaller, under-trained denoiser that is discarded after training. The authors present it as moving autoguidance from sampling into training, at no inference cost, and support it with a finite-training-time analysis in a kernel gradient-flow model. Experiments on CIFAR-10 and 18 ImageNet-256 DiffusionBench configurations report FID improvements with and without guidance.

Critical strengths
  • Simple to implement: only the training target changes, compared to GFT, and inference cost is unchanged.
  • A single fixed α is applied across many tokenizers and backbones.
  • The authors state the theory's limits clearly, including that SWIFT is strictly worse than standard training at convergence.
Critical weaknesses
  1. Limited technical novelty. The objective is the training-time target-extrapolation template of guidance-free training. The paper itself shows that the GFT loss is pointwise equivalent to regressing on q + α(q − u). SWIFT's only change is the reference: a frozen weak conditional model instead of the model's own unconditional prediction, which is the autoguidance choice of Karras et al. (2024) moved into training. The method is therefore essentially GFT-style training combined with the autoguidance reference. Its one claimed distinction, a frozen reference, is weakly supported: on CIFAR-10, frozen and jointly trained guides differ by only 0.015–0.08 FID, below the reported reproduction noise (about 0.049).
  2. The evaluation does not support the main claim.
  • Much of the headline gain is measured against unguided baselines. Since SWIFT is designed to internalize a guidance-like effect, this is not a like-for-like comparison. The relevant comparisons are against guided baselines and against the parent methods.
  • Under these comparisons the evidence is weak. In Fig. 4a, standalone autoguidance beats standalone SWIFT in all five configurations (e.g., 1.70 vs. 1.75, 4.02 vs. 10.31). That SWIFT still helps on top of AG suggests it acts as a partial, internalized version of AG rather than an improvement over it. The trade-off between SWIFT's extra training cost and AG's extra inference cost is not quantified. In addition, the RAEv2 "AG" row uses internal guidance with a truncated backbone, not a separately trained weak model.
  • There is no ImageNet comparison with GFT or MG. These methods are compared only on a 2D toy and a Gaussian mixture.
Questions for authors
  1. On ImageNet, under the same training budget, can you compare SWIFT against GFT (or MG) and against standalone AG, with AG using the same weak model as SWIFT, and report the training-versus-inference cost trade-off?
  2. Can you isolate the source of the gain by comparing a frozen unconditional reference (GFT-style), a frozen weak conditional reference (SWIFT), and a jointly trained reference?
2 comments
AuthorsOpenReview ↗

Response to Reviewer QGEb

Thank you for the review. We added the requested ImageNet comparisons, a three-training-seed reference ablation, and explicit training-versus-sampling costs. The new results directly test the parent training methods and the frozen-reference choice.

1. Direct ImageNet comparisons and training budgets (Weakness 2; Question 1)

Table 2 now compares the methods on ImageNet-256 with SD-VAE+reg, 615M parameters, and an 80-epoch main-training endpoint. All main models and distillation students start from random initialization. Models are evaluated at matched main-model updates; the training-FLOP estimates include preparation. The standalone results are:

Excerpt from Table 2 in the revised manuscript. All rows below use 1.00× standalone sampling FLOPs.

MethodFID ↓Training FLOPs
Ordinary training5.491.00×
GFT3.921.33×
MG4.671.67×
SGG, training variant4.011.33×
IG, auxiliary supervision4.791.00×
IG, training-time extrapolation4.271.33×
CFG distillation4.042.67×
AG distillation4.222.55×
SWIFT3.771.22×

SWIFT has the lowest FID among all seven training alternatives. It also uses fewer estimated total training FLOPs than GFT, MG, SGG, IG's training-time extrapolation, and either distillation method. For example, it improves on GFT's 3.92 FID at 1.22× vs. 1.33× training FLOPs, and on MG's 4.67 at 1.22× vs. 1.67×. The advantage at these matched-update endpoints therefore does not come from spending more total training FLOPs than these methods. IG auxiliary supervision is a cheaper training option, with substantially higher FID, 4.79.

Appendix B specifies the original training objectives, including the weighted GFT identity and both IG formulations. Appendix E.5 gives the objective and FLOP-accounting assumptions. These ImageNet results supplement the adapted Gaussian-mixture illustrations in Appendix C.

2. Why the frozen, same-condition reference matters (Weakness 1; Question 2)

Appendix B explicitly retains the algebraic overlap with GFT, including the β² loss weight in Eq. (B.2). It also states the proportional-gradient identity for the fixed-guide/current-prediction special case in Eq. (B.3). The substantive training choice is the reference used in that correction.

SWIFT anchors the target at the observed clean sample and uses a separately pretrained, frozen reference with the same condition. GFT uses the current shared model's unconditional output; stop-gradient detaches that evaluation within a step, while its shared parameters continue to change. An unconditional reference also introduces a conditional–unconditional contrast into the correction. The following experiment isolates these choices on ImageNet:

Table R1. Reference-source ablation. Main-model architecture, inputs, updates, and strength are held fixed. All target-side reference predictions are detached. Results are mean ± SD across three training seeds.

ReferenceUpdated during training?FID without guidance, mean ± SDFID with tuned CFG, mean ± SDTraining seeds
Frozen unconditional weak modelNo4.40 ± 0.072.15 ± 0.053
Frozen conditional weak model, SWIFTNo3.77 ± 0.051.81 ± 0.043
Jointly trained conditional weak modelYes3.95 ± 0.061.93 ± 0.053
Online shared unconditional outputYes4.64 ± 0.082.30 ± 0.063

SWIFT has the lowest mean FID both without guidance and after tuned CFG. With conditional references, freezing improves 3.95 → 3.77 and 1.93 → 1.81. With frozen references, matching the condition improves 4.40 → 3.77 and 2.15 → 1.81. Thus the ablation supports both freezing and matching conditioning; freezing alone is not the whole design.

This ImageNet experiment provides stronger evidence for the reference choice than the small exploratory CIFAR-10 differences noted in the review. We report training variability separately from sampling variability: five independent 50k-sample evaluations of one fixed checkpoint give 3.77, 3.79, 3.75, 3.78, and 3.76 (sampling seeds 0–4), with mean 3.77 and sample SD approximately 0.016 from the rounded values. This sampling SD is not used as a universal significance threshold for training comparisons.

Theorem 1 and Appendix A supply the finite-time explanation: an earlier weak guide leaves a residual that can offset part of the main model's remaining underfitting. Under the stated RKHS assumptions, a nonempty interval of positive strengths gives strictly lower posterior-mean error at matched finite optimization time. The reference ablation tests the design empirically; the theorem explains when its biased target can improve estimation.

AuthorsOpenReview ↗

Response to Reviewer QGEb

3. Guided baselines, AG distillation, and the training–sampling trade-off (Weakness 2; Question 1)

The revised comparison distinguishes standalone generation from generation with extra sampling guidance. The SD-VAE+reg entries in Table 2 are:

SettingFID ↓Training FLOPsSampling FLOPs
Baseline + AG2.161.13×1.26×
SWIFT3.771.22×1.00×
SWIFT + AG1.951.22×1.26×

Standalone AG has lower FID than unguided SWIFT in this configuration, at additional sampling work. With AG independently tuned for both models, SWIFT + AG improves on baseline + AG from 2.16 to 1.95. The baseline and SWIFT share the same weak checkpoint; scale and interval are selected independently for each main-model checkpoint under the same search rule (Section 5.2 and Appendix E.5).

This improvement holds in all four separate-model configurations in Figure 4(a):

ConfigurationBaseline + AGSWIFT + AG
DINOv2-B1.701.63
SD-VAE+reg2.161.95
Qwen-VAE1.791.62
PixNerd4.023.77

The gain therefore persists after baseline AG is tuned. SWIFT improves the learned model and remains useful when sampling guidance is subsequently applied. The additional CFG comparison also improves 2.32 → 1.81 (Table 8).

AG distillation tests the alternative of learning to reproduce a guided strong teacher while retaining standalone sampling. Table 2 reports 4.22 FID and 2.55× training FLOPs for AG distillation, versus 3.77 and 1.22× for SWIFT, both at 1.00× sampling FLOPs. SWIFT achieves lower FID using approximately 48% of that training budget. CFG distillation gives 4.04 at 2.67× training FLOPs. Section 3.2 and Eq. (E.1) specify the distinction between fitting a guided teacher prediction and extrapolating the observed clean label.

RAEv2 is now identified separately in Figure 4 and Appendix D.2. It uses native internal REPA-head guidance from the same backbone forward; it is not counted among these four separate-denoiser AG comparisons.

4. Full preparation cost and measured resources

Table 2 uses analytical FLOP estimates under Appendix E.5's assumptions. The strong teacher's preparation is charged to distillation; SWIFT needs only the smaller guide. Baseline + AG and SWIFT + AG are charged for their guide preparation, with SWIFT + AG reusing the guide already prepared for training.

The 1.26× AG sampling entry is the full-interval reference cost for the stated weak model at equal solver evaluations. A tuned shorter guidance interval reduces the extra work; the entry is not an end-to-end latency measurement. Figure 1(right) separately measures compute needed to improve on the baseline's best observed FID: SWIFT does so at 0.45× the DINOv2-B baseline's training FLOPs, including preparation.

Table R2. Measured training and sampling resources.

MethodPeak GiBs/updates/sample
Ordinary training≈105≈1.55≈0.080
Baseline + AG≈105≈1.55≈0.101
GFT≈107≈1.84≈0.080
IG-train≈107≈1.88≈0.080
SWIFT≈106≈1.72≈0.080

Measurements use 4×B200, global batch 1024, bf16, no gradient accumulation, and no activation checkpointing. Peak memory is allocated GiB per GPU. s/update is the wall-clock time per optimizer update, measured under the same protocol for all methods. Sampling is batched generation on one B200 with 50 Euler steps, including VAE decoding, expressed as seconds per image.

The resource measurements show unchanged standalone sampling time and an approximately 1 GiB peak-memory increase for SWIFT. Its measured time per update, approximately 1.72 s, is also lower than that of GFT (≈1.84 s) and IG-train (≈1.88 s). The FLOP comparison above provides the full training-budget comparison, including preparation.

Reviewer_Ed2YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Autoguidance (Karras et al., 2024) prepares two models, a weak performanced version and a strong version, and improves the sampling performance of the model by giving guidance in a classifier-free guidance (CFG) manner toward the direction that extrapolates the predictions between these two models. However, this causes two model forwardings and introduces additional overhead, and this paper proposes a method to incorporate this into training. While Autoguidance gives CFG style guidance through the output difference between the weaker model and the original model, this paper uses the weaker model at training time to directly learn the extrapolated objective through flow matching.

Critical strengths

This paper seems to come from a similar motivation to GFT (Chen et al., 2025), and except for the part where a reference model (or what is called a weak denoiser) is already required during training, I think the training objective itself is simple and easy to apply.

Also, by bringing the out-of-distribution issue caused by strong extrapolation in CFG-style guidance models into in-distribution and directly training on it, I think the oversaturation issue occurring in that process or the fidelity under large guidance will be further improved.

Critical weaknesses

A. A weak denoiser is required for training, and I think the complexity required for 2-stage training is not small. No matter how small it is configured, it still needs to be trained well first, and when training a large-scale text-to-image model such as on LAION-5B [a], I think this cost cannot be ignored. Moreover, I am concerned that there is no proper analysis on how well this weaker model should be trained, and that a model trained with 1/4 params and 1/2 epochs was used somewhat arbitrarily. Especially in the case of diffusion models, the training steps required until minimally reasonable images come out during training are quite long, and therefore I think there is a problem that the cost for obtaining an appropriate weaker sample becomes large. In addition, the prediction of the weak denoiser is also needed during training so an additional model has to be loaded; although the authors claim that the overhead from this is small at around 9%, it is disappointing that additional complexity is added to the already heavy diffusion model training.

In particular, I wonder about the robustness of whether the performance of this paper can be obtained even if the weak denoiser is trained roughly. I also wonder if there are controlled experiments on how and how much it should be trained to get how much gain in the overall training process.

B. Furthermore, if training is required, I think this method should also be compared with existing papers that distill guidance: DMD (Yin et al., 2024), Guidance Distillation (Meng et al., 2023), etc. Although these papers did not use an objective that applies guidance to the initial objective, they proceed with distillation during the post-training process and train the models to generate the actual guided target in that process, so I think their final goal is similar to this paper.

Minor Weakness

C. In terms of writing, I think the citations of prior guidance works are insufficient, such as SAG [b], PAG [c], and SEG [d] in inference-time guidance that attempted to give guidance using a weak model, as well as CFG++ [e] which handled the off-manifold issue at sampling time, and Guidance Interval [f] which applied CFG in a limited step interval.

[a] Laion-5b: An open large-scale dataset for training next generation image-text models

[b] Improving sample quality of diffusion models using self-attention guidance

[c] Self-rectifying diffusion sampling with perturbed-attention guidance

[d] Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention

[e] Cfg++: Manifold-constrained classifier free guidance for diffusion models

[f] Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

Questions for authors
  • Even when the model is trained according to the method proposed in this paper, from the fact that applying guidance still further improves the generation quality, it seems that training with the SWIFT method also does not reach the optimal point within finite-time training as derived mathematically. Especially, looking at Figure 5, the fact that the model trained with the extrapolated objective still gets more gain even at later iterations also seems to indicate this. I wonder whether this problem is because the extrapolation scale in the training objective is small so it could not reach the optimal point, or because an appropriate weak denoiser could not be secured so it could not be trained toward the exact direction pointing to the optimal point.

  • As mentioned in Strength, the OOD issue, or off-manifold issue, that occurs when giving guidance in CFG style causes saturation due to overly strong extrapolation. Since all of these are turned into seen data at training time, I wonder whether saturation or artifact issues caused by a high guidance coefficient occur less in that process compared to guidance methods such as CFG and Autoguidance.

  • Additionally, for guidance training, a weak version of itself was used as the weak denoiser like in Autoguidance. I wonder if that choice can be any model, and what results would come out if another model (such as models used in previous literature) is used. Going one step further, I wonder whether a more improved optimal point can be obtained if we iterate this method by using the model trained through this method again as a weak denoiser.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ed2Y

Thank you for the detailed review. We added direct guidance-distillation comparisons, included guide preparation in the cost analysis, and expanded the evidence for guide selection. The main results are in Table 2 and Figures 1, 4, and 5; the additional experiments below address your individual questions.

A1. Guide preparation, training overhead, and total cost

Guide preparation is included in our reported compute savings. In Figure 1(right), SWIFT exceeds the best DINOv2-B baseline FID observed through 320 epochs using 0.45× its training FLOPs, including preparation and the extra guide forward passes. Section 5.3 reports the longer-training comparison.

The approximately 9% figure refers specifically to the extra main-update FLOPs. At the matched-update endpoint in Table 2, the analytical total is approximately 1.22× ordinary training, including guide preparation; standalone sampling remains 1.00×. Appendix E.5 states the FLOP assumptions. Appendix D.3 separately reports measured elapsed time: approximately 44 hours for ordinary training, 49 hours for SWIFT main training, and 12 hours for guide preparation. Thus the preparation cost is charged explicitly rather than absorbed into the 9% figure.

Table R5 reports memory and timing. SWIFT uses approximately 106 vs. 105 GiB peak memory and retains the baseline's measured standalone sampling time, approximately 0.080 seconds/image. The guide is frozen, requires no backward pass, and can be reused across training-coefficient experiments. If AG is subsequently applied, the same guide can also be reused.

A2. How small and how well trained should the guide be?

The guide choice is supported by capacity and maturity ablations, rather than by the default size alone. Figure 5 shows that a smaller but competent guide works best in the tested CIFAR-10 settings; excessively small guides lose effectiveness. The additional ImageNet comparison holds the main model and training α fixed:

Table R1. Guide capacity and maturity on DINOv2-B. The main model is 615M and trains for 80 epochs with α=0.4. The first four rows use the archived training-time evaluator.

GuideFrozen checkpointStudent FID-50k
Default half-widthDefault1.7845
Default half-widthEarlier1.6576
Wider GUIDE-w075Default1.9515
Wider GUIDE-w075Earlier1.6970
Default half-widthLater1.86
Alternative architectureMatched maturity1.85

Earlier freezing improves student FID at both capacities: 1.7845 → 1.6576 and 1.9515 → 1.6970. More guide training is therefore not automatically beneficial. The default half-width guide has approximately 155M parameters and is prepared for half the standard main-training duration; the earlier checkpoint is halfway to that freezing point. GUIDE-w075 denotes the wider guide configuration. These evaluator results are kept separate from the independent re-evaluations in the main result tables.

Together with Figure 5, the results give a practical selection rule: use a smaller guide that has learned useful denoising structure, assess maturity by the resulting student's quality, and freeze the selected checkpoint. The useful guide need not be the most accurate or most extensively trained candidate.

AuthorsOpenReview ↗

Response to Reviewer Ed2Y

B. Guidance distillation and DMD

Table 2 now directly compares the training methods on ImageNet-256 with SD-VAE+reg at matched main-model updates. SWIFT obtains FID 3.77, compared with 4.04 for CFG distillation and 4.22 for AG distillation. Its training cost is approximately 1.22×, compared with 2.67× and 2.55×, including teacher/guide preparation. All three produce standalone sampling models with 1.00× sampling FLOPs. Under this accounting, SWIFT has lower FID than AG distillation using approximately 48% of its training FLOPs.

Section 3.2 and Eq. (E.1) make the target distinction explicit. Guidance distillation fits a guided strong-teacher prediction. SWIFT uses the observed clean sample as the anchor and extrapolates it away from a frozen weak prediction; it does not require a pretrained strong teacher. All main models and distillation students are randomly initialized and trained from scratch. Teacher weights only supply targets.

SWIFT also improves the model to which AG is subsequently applied. Figure 4(a) reports independently tuned baseline + AG versus SWIFT + AG: on SD-VAE+reg, FID improves from 2.16 to 1.95, with improvements in all four separate-model configurations. Thus the evidence supports both a better standalone training method than AG distillation and a further gain when AG is applied after SWIFT.

Table R2. DMD comparison. Preparation and auxiliary-model work are included in training cost; sampling work refers to the deployed generator.

MethodFID-50kTotal training costSampling work
DMD5.441.60×0.02×
SWIFT3.771.22×1.00×

SWIFT has lower FID and training cost in this comparison. DMD provides much cheaper sampling through a one-step generator, while SWIFT retains the original multi-step sampler. The table reports both aspects of that trade-off.

C. Related guidance work

Section 3.2 now discusses SAG, PAG, SEG, CFG++, and guidance applied in a limited noise interval. Appendix B compares the training objectives and distinguishes IG's auxiliary-supervision method from its Section 6 training-time extrapolation. Both IG variants are separate rows in Table 2.

Q1. Does the remaining AG gain mean that training α is too small or the guide is unsuitable?

Theorem 1 establishes a range of strengths that improves finite-time posterior-mean error. Such improvement does not require exact recovery of the posterior mean. In Section 4, the signed error on a learnable mode is proportional to

−e−Tλ+α(1−e−Tλ)e−Tgλ.-e^{-T\lambda}+\alpha(1-e^{-T\lambda})e^{-T_g\lambda}.

The coefficient that cancels this error depends on the mode. One scalar α therefore need not remove every residual, leaving room for sampling guidance. The theorem concerns posterior-mean error under the stated training assumptions; Figure 4 evaluates sampling FID.

The experiments also separate the two coefficients. Figure 5 studies training α and guide choice; Figure 4 studies sampling guidance. Increasing training α from 0.4 to 4.0 worsens standalone FID from 3.77 to 7.44 in Table R3, so a remaining AG gain does not imply that simply increasing training α is beneficial. Guide maturity matters, as Table R1 demonstrates.

For the AG comparison, each baseline/SWIFT pair uses the same weak checkpoint, and each main-model checkpoint selects its best scale and interval under the same search rule (Section 5.2 and Appendix E.5). SWIFT + AG still improves 2.16 → 1.95 on SD-VAE+reg. This comparison already tunes baseline AG; its gain is not explained by an insufficient baseline guidance scale.

AuthorsOpenReview ↗

Response to Reviewer Ed2Y

Q2. Behavior under strong extrapolation

Table R3. Best and stronger settings on SD-VAE+reg. Within each row, reported FID, precision, and recall use the same generated set.

SettingTraining αSampling scaleFIDPrecisionRecall
Baseline + CFG, best01.22.320.800.60
Baseline + CFG, strong04.011.640.820.42
Baseline + AG, best01.62.160.790.64
Baseline + AG, strong04.010.520.840.51
SWIFT, default0.4—3.770.7330.66
SWIFT, strong4.0—7.440.790.64
SWIFT + CFG, best0.41.31.810.800.61
SWIFT + AG, best0.41.21.950.810.62

The setting is 615M, an 80-epoch main-training endpoint, 50k samples, and 50 Euler steps. Strong SWIFT uses its separately trained α=4.0 checkpoint. Its recall changes from 0.66 to 0.64, compared with 0.60 → 0.42 for CFG and 0.64 → 0.51 for AG. At the stronger settings, SWIFT also has lower FID: 7.44 vs. 11.64 and 10.52. These measurements support better retention of coverage in the tested stress settings. SWIFT learns from extrapolated regression targets and then samples with its original model output. CFG and AG instead extrapolate predictions along the sampling trajectory. This difference is consistent with the measured coverage behavior, although training α and sampling scale are separate parameters.

The SWIFT + CFG metrics come from Table 8, and the SWIFT + AG FID comes from Figure 4(a). Dashes mark unreported AG precision/recall. With separately tuned sampling guidance, SWIFT improves 2.32 → 1.81 with CFG and 2.16 → 1.95 with AG.

Q3(a). Guides with different architectures

Table 11 in Appendix F includes NCSN++ and ADM guides for a DDPM++ student. Table R1 adds the ImageNet alternative-architecture result. These experiments show that shared parameters or an identical architecture are unnecessary. The guide must still have compatible noisy inputs, conditioning, and prediction coordinates.

Q3(b). Reusing SWIFT as the next guide

We tested this by freezing each SWIFT model and using it to guide a wider student, with a fresh random initialization at every stage. The comparisons use the same student capacity and 80-epoch student-training duration within each row.

Table R4. Iterative SWIFT and matched-capacity alternatives. These are separate experiments from the fixed-backbone comparison in Table 2.

StageStudent widthOrdinary training FIDAG-distilled FIDSWIFT FID
First1×5.244.224.07
Second1.5×4.613.943.71
Further2×4.143.713.40

The chain starts from a 0.5×-width guide prepared for half a stage's standard main-training duration (FID 13.85). The students have widths 1×, 1.5×, and 2×. Previous-stage SWIFT models supply frozen predictions only; their weights do not initialize the new students.

SWIFT wins at all three capacities, reducing FID relative to ordinary training by approximately 22%, 20%, and 18%. It also outperforms AG distillation by 0.15, 0.23, and 0.31 FID. The 1× SWIFT student (4.07) already outperforms the 2× ordinary model (4.14), and the 1.5× SWIFT student matches the 2× AG-distilled model (3.71). This supports repeated use over the tested capacity-growing chain; each comparison matches student capacity and updates.

Table R5. Training and sampling resources.

MethodPeak GiBs/updates/sample
Ordinary training≈105≈1.55≈0.080
Baseline + AG≈105≈1.55≈0.101
GFT≈107≈1.84≈0.080
IG-train≈107≈1.88≈0.080
SWIFT≈106≈1.72≈0.080

Measurements use 4×B200, global batch 1024, bf16, no gradient accumulation, and no activation checkpointing. Peak memory is allocated GiB per GPU. s/update is the wall-clock time per optimizer update, measured under the same protocol for all methods. Sampling is batched generation on one B200 with 50 Euler steps, including VAE decoding, expressed as seconds per image.

Thank you for the rebuttal. I've read the newly added experimental results, and I think the ImageNet experiments are promising. Also, I feel like this method has clear differences from IG, and therefore this work provides good findings for the community (as many people feel the train-test mismatch in diffusion objectives, I think this paper could offer a meaningful contribution from this perspective). Also, the fact that a roughly trained guidance model leads to better performance is quite counterintuitive, so it is also an interesting result. I look forward to further analysis of this phenomenon, and I would be happy to see this work extended further. I expect the camera-ready version of this paper will include the experimental results from this rebuttal period, and I also hope Section 3.2 can be written more cleanly. (A little concern is the comparison with guidance distillation, where the authors trained the student model from scratch, which may harm the performance slightly, but further experiments are not mandatory for my decision.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.