- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Past-Self Guidance (PSG), a training scheme that improves flow-based generative models by incorporating an AutoGuidance-inspired temporal contrast into the flow-matching objective. Specifically, PSG perturbs the standard flow-matching velocity target with the difference between the predictions of a recent teacher and a past teacher, and trains the student model to regress toward this modified target. The recent teacher is either the student's exponential moving average (EMA) or a periodically refreshed copy of it, while the past teacher is selected from checkpoints of ordinary flow-matching training at a fixed fraction of the current training step. By internalizing this temporal guidance during training, PSG requires no additional computation at inference time. Experiments demonstrate substantial improvements in generation quality, as measured by FID, across multiple model scales (SiT-S/B/L/XL), with additional improvements demonstrated for diffusion-based DiT models.
Critical strengths
Simplicity and effectiveness. The proposed PSG method is conceptually simple and easy to integrate into existing flow-matching training pipelines, requiring only a modification to the regression target without architectural changes or additional inference-time computation. Despite its simplicity, PSG achieves substantial improvements in generation quality, reducing FID by 34–55% across SiT model scales. Particularly compelling is its effectiveness when applied only during the final 10% of training, yielding a significant FID improvement with approximately 6% additional training compute.
Extensive experimental validation and insightful analyses. The paper presents a comprehensive empirical evaluation across multiple model scales (SiT-S/B/L/XL), generative objectives (flow matching and diffusion), and generation settings (conditional and unconditional). The authors further demonstrate compatibility with existing techniques such as REPA and CFG. Beyond performance comparisons, the paper provides insightful ablation studies investigating the underlying mechanism of PSG. In particular, the authors show that most of the improvement comes from the angular component of the historical prediction difference and that the correction is sample-specific. These findings provide interesting empirical evidence that optimization history contains useful information for improving generative modeling.
Critical weaknesses
Insufficient comparison with closely related guidance-distillation methods. Although the paper demonstrates substantial improvements over vanilla flow matching, it remains unclear whether PSG offers a superior fidelity–diversity tradeoff compared to existing guidance-distillation approaches, particularly CFG-based training contrast. The authors provide a limited comparison in Appendix B.4, where conditional guidance achieves significantly better FID at the expense of recall. However, a more systematic comparison across guidance strengths, ideally examining FID–recall Pareto frontiers under matched computational budgets, would be valuable for establishing the distinct advantages of history-based guidance.
Limited understanding of the underlying mechanism. While the paper provides insightful ablation studies demonstrating the importance of the angular component and sample-specific nature of historical corrections, the underlying reasons for their effectiveness remain unclear. In particular, it is not well understood why extrapolating along historical prediction differences improves generation quality, or what useful information these differences capture about the learned velocity field. The stationary-point analysis establishes a connection to sampling-time guidance but does not fully explain the observed performance improvements.
Questions for authors
- Comparison against CFG-based guidance distillation. In Appendix B.4, conditional guidance achieves substantially lower FID than PSG, albeit with lower recall. Could the authors provide a more systematic comparison across guidance strengths, examining the FID–recall Pareto frontiers of PSG and conditional-guidance training under comparable computational budgets? Does PSG offer a meaningful advantage at matched fidelity or diversity?
- Complementarity with conditional guidance. Could the authors investigate whether temporal guidance provides complementary benefits when combined with CFG-based training contrast? Such an experiment would help establish whether historical prediction differences contain useful information beyond existing guidance signals.
- Mechanism underlying historical guidance. Although the ablations demonstrate the importance of the angular component of historical prediction differences, it remains unclear why this direction improves generation. Could the authors provide further evidence about what information this direction captures? For example, does it systematically correct prediction errors, or does it primarily alter the fidelity–diversity tradeoff?
- Generality beyond ImageNet. While PSG demonstrates consistent improvements across model scales and generative objectives, the experiments primarily focus on ImageNet 256×256. Do the authors have evidence suggesting that PSG generalizes to other datasets or more challenging generation settings, such as text-to-image synthesis?