- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SWIFT moves weak-to-strong extrapolation from diffusion sampling into training. A frozen, separately pretrained weak denoiser defines the regression target y = (1 + alpha)x - alpha D_weak for the main model. The weak guide is discarded after training. A finite-time population kernel-gradient-flow analysis establishes a range of positive extrapolation strengths that reduces posterior-mean error relative to ordinary training under the stated assumptions. Experiments cover CIFAR-10 and 19 ImageNet-256 configurations spanning several latent representations and pixel-space models. The paper reports consistent FID improvements, compatibility with independently retuned inference guidance, persistent benefits under longer training, and compute comparisons that include guide preparation.
Critical strengths
The experiments cover multiple representations and training budgets. The finite-time analysis and accounting for guide-preparation costs are informative.
Critical weaknesses
Insufficient methodological novelty. AG already establishes weak-to-strong extrapolation. GFT’s mixed-prediction loss can be rewritten as regression toward an extrapolated target, with corresponding loss weighting. IG also explicitly studies training-time extrapolation (Section 6 of its revised version). SWIFT’s frozen weak reference changes the training dynamics, but the paper does not establish a sufficiently substantial independent contribution.
The closest comparison is missing. Appendix C evaluates IG with ordinary intermediate/final supervision rather than its extrapolated-target training variant. Consequently, the experiments do not establish the benefit over this direct precedent. Broad improvements over standard training do not resolve this contribution gap.
Questions for authors
What substantive contribution remains beyond AG, GFT, and IG’s training-time extrapolation?
2 comments
Response to Reviewer MJ7t
2. What remains beyond the known extrapolation template?
The contribution is the choice and validation of the training reference, together with a finite-time explanation of why it helps. In x0-prediction notation, suppressing common inputs and loss weights, the two targets are
SWIFT uses the observed clean label and a separately pretrained, frozen, same-condition weak predictor. IG uses an evolving EMA gap between final and intermediate outputs and retains auxiliary supervision. GFT uses the evolving shared model's unconditional output and its mixture-loss weighting. Guidance distillation uses an extrapolated strong-teacher prediction. These choices determine both the target and its evolution during training.
Appendix B states the exact GFT rewrite, including β² in Eq. (B.2), and the fixed-guide/current-prediction proportional-gradient identity in Eq. (B.3). The revised claim is grounded in the following evidence:
Reference ablation. With architecture, inputs, updates, and strength fixed, the matched ImageNet experiment gives:
Table R1. Reference-source ablation; mean ± SD across three training seeds.
| Reference | Updated during training? | FID without guidance, mean ± SD | FID with tuned CFG, mean ± SD | Training seeds |
|---|---|---|---|---|
| Frozen unconditional weak model | No | 4.40 ± 0.07 | 2.15 ± 0.05 | 3 |
| Frozen conditional weak model, SWIFT | No | 3.77 ± 0.05 | 1.81 ± 0.04 | 3 |
| Jointly trained conditional weak model | Yes | 3.95 ± 0.06 | 1.93 ± 0.05 | 3 |
| Online shared unconditional output | Yes | 4.64 ± 0.08 | 2.30 ± 0.06 | 3 |
The frozen conditional reference has the best mean FID in both columns. Freezing the conditional guide improves 3.95 → 3.77 without guidance and 1.93 → 1.81 with tuned CFG. Matching the condition for a frozen guide improves 4.40 → 3.77 and 2.15 → 1.81. These results show that the specific reference design materially affects performance.
Finite-time analysis. Theorem 1 and Appendix A establish a nonempty interval of positive strengths with strictly lower posterior-mean error than ordinary training at matched finite optimization time under the stated RKHS population-gradient-flow assumptions. The analysis explains how an intentionally shifted regression target offsets finite-time underfitting. This is a statement about finite-time estimation with a learned weak reference, and it supplies a mechanism for the design tested above.
Guide selection. Figure 5 studies capacity and maturity. In the additional DINOv2-B comparison, earlier freezing improves student FID from 1.7845 to 1.6576 at half width and from 1.9515 to 1.6970 with GUIDE-w075, using the training-time evaluator. The choice of a smaller but competent, frozen guide is therefore supported by experiments on both capacity and maturity.
3. Benefits over the parent methods and with subsequent AG
The direct comparisons in Table 2 give SWIFT FID 3.77, versus GFT 3.92, MG 4.67, SGG training 4.01, CFG distillation 4.04, and AG distillation 4.22. In particular, SWIFT has lower FID than AG distillation at approximately 48% of its estimated training FLOPs, including preparation, with the same standalone sampling cost. Section 3.2 and Eq. (E.1) explain the target difference.
Figure 4(a) then tests whether the resulting model still benefits from AG. After independently tuning AG for each baseline/SWIFT pair, the FIDs improve 1.70 → 1.63 on DINOv2-B, 2.16 → 1.95 on SD-VAE+reg, 1.79 → 1.62 on Qwen-VAE, and 4.02 → 3.77 on PixNerd. The same weak checkpoint is used within each pair, with the common search rule described in Section 5.2 and Appendix E.5. Thus the improvement persists after baseline AG is optimized.
These results establish practical benefits for both standalone training and subsequent guided sampling, in addition to the direct improvement over IG's closest training-time formulation.
Response to Reviewer MJ7t
Thank you for identifying the Section 6 training-time extrapolation in IG. We added this formulation as a separate ImageNet comparison in Table 2. SWIFT improves on it from FID 4.27 to 3.77, with lower estimated total training FLOPs.
1. Direct comparison with IG's training-time extrapolation
Table 2 compares the methods on ImageNet-256 with SD-VAE+reg, 615M parameters, and matched main-model updates. All main models start from random initialization. The relevant entries are:
SWIFT improves on IG's Section 6 variant by 0.50 FID, from 4.27 to 3.77, an approximately 11.7% relative reduction. Its estimated total training cost is also lower, 1.22× vs. 1.33×, including guide preparation. Both retain 1.00× standalone sampling FLOPs. The auxiliary-supervision variant gives 4.79; it is listed separately because it trains a different objective.
For the Section 6 comparison, the final target contains the detached EMA final-minus-intermediate prediction gap, and the intermediate output retains ordinary denoising supervision. The intermediate and final EMA outputs share one backbone forward; the cost estimate charges that extra EMA evaluation. Appendix B distinguishes the two objectives, and Appendix E.5 explains the FLOP assumptions. Tables 4–5 in Appendix C remain explicitly identified as adapted Gaussian-mixture illustrations whose IG arm uses auxiliary supervision.