- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper does not propose a new method; instead, it studies an important question in the field of generative models: whether variations in random seeds significantly affect the FID metric. Through extensive experiments, the paper provides several findings regarding fluctuations and randomness. Based on these findings, the paper suggests an evaluation protocol.
Critical strengths
-
The question studied in the paper uncovers a new blind spot, i.e., how reliable the conclusions produced by the existing FID evaluation metric are, which has strong practical significance.
-
The experimental design is fairly systematic and quantifies the degree of impact, providing strong reference value.
Critical weaknesses
-
The generality of the conclusions is questionable. The authors claim to quantify the randomness of generative model evaluation, but the current experiments mainly focus on diffusion models and do not consider other frameworks; even if only diffusion models are considered, the current methods are primarily based on flow matching, without considering other types of frameworks such as score matching. If the authors believe that the fluctuation of FID mainly originates from the per-step noise in flow matching, and that the reliability issue is relatively less severe in other frameworks, then the scope of generative models covered in the title, abstract, and introduction of the paper should be appropriately narrowed, rather than being treated merely as a research limitation.
-
The suggested evaluation protocol is theoretically reasonable, but it does not fully consider practical situations. Reporting error bars over multiple random seeds implies that large models may need to be trained multiple times, which would multiply the cost. Considering the topic of the paper, there is no need to solve this problem, but it should at least be pointed out; alternatively, the actual reliability gains brought by the increased cost could be further estimated.
Questions for authors
Have the authors tried conducting experiments on other types of generative model frameworks, such as DDPM? If so, what were the results?
3 comments
Thank you for your detailed response. The current revision is meaningful and has addressed some of the concerns, especially my second concern, which has been largely resolved.
However, regarding the first concern, although the drifting model experiments provide some important results, I still believe that the current experiments cannot be claimed as “broad” in the field of generative models. Moreover, the diverse application scenarios listed by the authors do not change the issue that the categories of generative models covered are still relatively limited.
Considering that raising my score would shift my recommendation from rejection to acceptance, I prefer to keep my rating unchanged for now, even though the paper has been improved to some extent.
Thank you for engaging with our response, and for confirming that the revision resolves your second concern. We would like to clarify the remaining point on scope, because we think the new evidence speaks to it more directly than our previous reply made clear.
The Drifting experiment tests the hypothesis behind your scope concern. Your original W1 asked us to narrow the scope if the FID fluctuation "mainly originates from the per-step noise in flow matching" and the issue is "relatively less severe in other frameworks." Drifting is the model family where that hypothesis predicts a smaller lottery. It is trained without any diffusion or flow-matching loss, so it has no per-step regression noise, and it samples in a single step with no ODE integration. Its training-seed CoV is still 1.54%, inside the same 1–2% band as SiT. This is evidence that the lottery does not come from flow matching. That is consistent with our decomposition: random initialisation and data ordering, two of the three sources we identify, are present in any network trained with SGD, whatever its objective.
Our experiments vary the axes along which generative models differ. The three families we calibrate differ in:
- Training objective: flow matching vs. a non-diffusion drifting loss
- Representation: latent vs. pixel space
- Conditioning: class labels vs. free-form text
- Sampler: multi-step ODE vs. one-step
- Metric: Inception FID, GenEval, DPG-Bench
The lottery appears in every one of these settings, with a consistent magnitude wherever we measure FID.
These families account for most current practice. The open-weight models that lead today's benchmarks are trained with flow matching. For example, Qwen-Image is pre-trained with a flow-matching objective, as are SD3, FLUX and the main open video models. In a recent snapshot of the text-to-image arena, FLUX 2 variants alone took four of the top 15 places, and the top fully open-licensed model is Qwen-Image. On ImageNet, the methods competing at the top are SiT/DiT-based flow models and one-step generators such as Drifting, which are exactly the two families we calibrate. We therefore think the title describes the evaluation practice the paper studies. The Limitations section already states which families the ≈4% threshold is calibrated on.
On the scale of the study. We understand your hesitation to change your recommendation, but we would note that the study is already extensive by the standards of evaluation papers in generative modelling:
- Several hundred SiT networks, trained across 76 cells spanning model sizes S to XL and 200k to 2M training steps, with 25 training seeds per setting.
- A two-axis panel of training seeds × generation seeds.
- A text-to-image model retrained from 10 seeds and evaluated under 10 evaluation seeds each.
- 7 independently trained Drifting models.
We are not aware of any prior study that retrains diffusion or flow models at this scale to measure seed variance. Each additional family requires retraining many models from scratch, which is exactly the cost our measured noise floor spares others. We believe the coverage we report is sufficient to support the paper's claims as stated, and we hope you will take this into account in your final assessment.
Thank you again for your time and constructive engagement.
Response to Reviewer 6xAc
Thank you for your review, and for recognising that the paper "uncovers a new blind spot" with "strong practical significance", and especially for noting that you "gained a substantially deeper understanding of the problem or question." Below you address your concerns:
We argue that the reliability issue is not specific to flow matching. The per-step noise we identify is shared by score matching [1]. Denoising score matching, DDPM and flow matching all draw a fresh Gaussian ε at every gradient step and regress onto it; they differ only in the noising path and the loss weighting.
Since the submission, we have also added an experiment with Drifting Models [2], a one-step generator trained without any diffusion loss. We find that it varies by 1.54% across 7 training seeds, which remains inside our 1–2% band, which shows a degree of generalization for our findings. See updated manuscript. Our findings, therefore, hold on three different model families (latent space class conditional flow matching, pixel space T2I and class conditional Drifting).
Indeed, the score and applicability of our paper is quite broad. Flow matching, including rectified flow, is the training objective of many of today's leading generators: SD3 and FLUX for images, Movie Gen for video, Voicebox for speech and π0 for robot control. The SiT recipe we study is also the base of many recent ImageNet results. A calibrated noise floor for this family is directly relevant to a large part of the generative-modeling field.
[1] Generative Modeling by Estimating Gradients of the Data Distribution, Song et al. Neurips 2019
[2] Generative Modeling via Drifting, Deng et al. NeurIPS 2026
Actually, our proposed protocol is much cheaper in practice than our evaluation experiments might suggest. We trained 25 seeds per setting to measure the noise floor, so that others do not have to! Now, everyone can use our estimated noise floor with little cost. A relative gain g needs about (4/g)² runs per method to be significant, so a >4% improvement needs just a single run. Extra runs are only required for close calls, where a single run provides not evidence either way. And because the floor does not change with scale, it can be calibrated on small models, before training the large model. We have added a paragraph clarifying this in the updated manuscript.