- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper models the data distribution and model dynamics using a GMM to analyze how velocity errors at different timesteps affect the final generated images. Based on this analysis, the authors propose a time-decaying weighting scheme to guide FM training and demonstrate improvements across different models and datasets.
Critical strengths
The paper provides a GMM-based perspective for analyzing flow dynamics and investigating how early-stage velocity prediction errors affect the final generated images. The timestep weighting problem studied in this paper is meaningful and plays an important role in Flow Matching training. Moreover, the proposed time-decaying weighting strategy achieves consistent performance improvements across different model scales and datasets.
Critical weaknesses
1. Concerns about the consistency between GMM reference dynamics and the actual model dynamics.
The paper primarily studies endpoint sensitivity using the reference velocity field derived from a GMM. Even in Figure 2, perturbations are propagated using the GMM reference velocity rather than the trained SiT model. The authors do not provide sufficient evidence that these two dynamics exhibit consistent error propagation behavior. In particular, the model's own generation process may correct or attenuate early-stage perturbations. Why not directly use the trained model for subsequent integration to analyze its actual endpoint sensitivity?
2. Insufficient connection between endpoint sensitivity and generation quality.
Previous studies have shown that the high-noise stage mainly determines the global structure, while the low-noise stage primarily refines details. Therefore, it is somewhat intuitive that early-stage perturbations may have a larger impact on the final generated image. However, a larger endpoint deviation does not necessarily imply worse generation quality. For example, a perturbation may simply shift the generation result toward another valid data mode. The paper lacks direct evidence showing how endpoint sensitivity relates to final generation quality and why assigning larger training weights to earlier timesteps is justified.
3. Missing comparison with a simple shifted logit-normal baseline.
Directly adjusting the mean of the logit-normal distribution can also increase the relative training emphasis on earlier timesteps, which is a simple and commonly used approach. However, the paper does not include this baseline. Consequently, it is difficult to determine whether the proposed sensitivity-guided attenuation provides advantages over a straightforward adjustment of the timestep distribution. The necessity of the specific weighting design therefore remains unclear.
Questions for authors
Q1. Generalization of GMM-based sensitivity analysis.
Have the authors considered directly measuring endpoint sensitivity using the learned SiT velocity field instead of the GMM reference dynamics? Would the early-stage error amplification observed in Figure 2 still hold when perturbations are propagated through the trained model itself?
Q2. Connection between endpoint sensitivity and generation quality.
How does the measured endpoint sensitivity relate to actual generation quality? Since a larger endpoint deviation does not necessarily imply a lower-quality sample, could the authors provide further evidence that reducing errors at high-sensitivity timesteps leads to better generation quality?
Q3. Comparison with shifted logit-normal weighting.
Could the authors compare the proposed weighting strategy with a simple shifted logit-normal distribution, where the mean parameter is adjusted to emphasize earlier timesteps? This would help clarify whether the proposed sensitivity-guided attenuation provides benefits beyond a straightforward adjustment of the timestep distribution.
2 comments
Response to Authors
Thank you for the response. Figure S3 addresses my concern about the GMM dynamics, and the shifted logit-normal comparison answers W3.
However, W2 remains unresolved. The paper's logic relies on larger endpoint displacement implying worse generation quality, but this link is not established. The FID gains show that the weighting works, not why it works. Figure S4 shows only a small difference, and the very large early sensitivity in Figure S3 may simply reflect switching to another valid mode. Without direct evidence connecting early-stage errors to generation quality, the motivation for the weighting does not hold. I will therefore maintain my rating.
We sincerely thank you for your thoughtful and constructive feedback, and for recognizing the significance and clarity of our work. Below, we provide point-by-point responses to your questions and concerns.
W1, Q1: Generalization of GMM-based sensitivity analysis.
Response. As described in Section 3.4, we use the same GMM for the dynamics analysis and the perturbation probes. Each data point is the mean of an isotropic Gaussian component, whose variance is set to the average posterior variance of the VAE outputs. The component variance is about 10−4, while the latent data are on a scale of about 1. This motivates using narrow Gaussian components to approximate the target distribution. To further assess how well the GMM approximates the target distribution, we trained two SiT-B models for 250K steps, one on ImageNet latent codes and the other on samples from our GMM approximation. The model trained on the GMM achieved an FID of 35.79, compared with 28.21 for the model trained on ImageNet latents. The FIDs are comparable in scale, suggesting that the GMM retains structure in the target distribution that is relevant to generation.
Velocity prediction errors can accumulate during generation and affect the endpoint, making it harder to isolate the effect of a single local perturbation. In the original probe, we therefore use the GMM reference velocity field to propagate the perturbation to the endpoint and measure its effect under the reference dynamics. To test whether the sensitivity pattern persists under the learned dynamics, we propagated perturbations to the endpoint using the trained SiT velocity field. Figure S3 shows that sensitivity drops near the GMM-selected onset a=0.6 and approaches the within-component reference at later timesteps. This supports qualitative agreement between the learned and GMM reference dynamics. We report this experiment in the revised appendix.
W2, Q2: Connection between endpoint sensitivity and generation quality.
Response. We thank the reviewer for this insightful question. The training objective in flow matching is to approximate a reference flow field. Its connection to generation quality rests on the premise that a learned flow closer to the reference flow produces endpoints closer to the reference endpoints and is expected to improve overall generation quality. This motivates our focus on endpoint sensitivity and on reducing endpoint displacement.
Under the flow matching ODE, there is no theoretical guarantee that a displaced endpoint will land in another valid data mode. We therefore cannot assume this in general when analyzing the flow matching model. There are more evidences illustrate reducing the endpoint displacement improves the generation quality. First, Tables 1, 4, and 5 show that the sensitivity-guided refinement improves generation quality across weighting schemes and datasets. Second, Figure S2 shows that FID is sensitive to the choice of attenuation onset a, which we determine from the endpoint sensitivity analysis. Finally, Figure S4 shows that the refinement reduces endpoint displacement, as expected from its design. Together with the generation results, this provides empirical support for the connection between endpoint sensitivity and generation quality.
W3, Q3: Comparison with shifted logit-normal weighting.
Response: We adopt the standard logit-normal setting that Esser et al. (2024) selected through a hyperparameter sweep. Under matched training and evaluation conditions, we also evaluate several earlier-shifted logit-normal schedules with and without refinement. The table below summarizes the results. The best earlier-shifted setting (ES-LN) yields FID and IS values comparable to those of the standard logit-normal setting (LN). The refinement improves performance with both schedules under matched conditions. Therefore, tuning the shift alone falls short of these gains, demonstrating the necessity of the refinement design.
References