- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper discusses the singularity issue caused by the x-pred+v-MSE loss during the flow-matching training. By considering the second moment of the aligned loss gradient and the mismatched loss gradient, it is rigorously shown that the aligned version has finite second moment, while the mismatched version has infinite second moment, under the usual uniform sampling of training timesteps. The paper then proposed a novel way of sampling the timesteps for training using Logit-Normal endpoint suppression, successfully bounding the second moment of the mismatched version gradient with a finite number. Experiment results on the U-Net and JiT-B/4 model structures across different loss types further validates the theoretical results.
Critical strengths
The motivation of the paper is clear and attractive, which comes from the singularity caused by the derived velocity-prediction loss. Rigorous theoretical derivations are conducted inside of the paper, and the experiments show significant differences in both loss and generation quality between the Uniform and Logit-Normal samplings during the training, in the x-pred+v-MSE case. Different model backbones are used for the experiments, so that the results are more convincing across different model structures.
Critical weaknesses
Some weakness points are listed below:
- For the experiments, it might be helpful if more than 3 different seeds are used to calculate the mean and standard deviation in for example Figure 3.
- Some concepts are not defined clear enough inside of the main paper, for example the aligned loss and the mismatched losses. Worth mentioning that the "aligned loss" corresponds to the x-pred+x-MSE, and the "mismatched loss" corresponds to the x-pred+v-MSE, perhaps in a table.
- I noticed that the ODE samplers/solvers during the inference are different across experiments. Specifically, the U-Net Binary-MNIST experiments used Euler's method and the JiT-B/4 experiments used Heun's method. It is better to use consistent ODE solvers to exclude any possible effects ODE solvers bring during the inference rather than what's in the training.
Questions for authors
Here are several questions:
- I saw inside of the appendix that for in the Logit-Normal endpoint suppression, the experiments chose ; intuitively, since in the mismatched singularity happens around t=1, we would like to avoid that region, so should be away from and should in expectation be negative, which is fine. However, this also indicates that when we choose (eg. ) and some (and think about the case when is small), then will likely to fall into the regime that's close to so that singularity continues to dominate; then
(1) Can you clarify if my understanding and intuitions are correct?
(2) If what I mentioned are correct, then the Logit-Normal endpoint suppression can only conditionally solve the singularity issue, while Proposition 4.3 does not reflect the distinction about the sign of ; some more experiments/theoretical explanations about this part would be very much appreciated. - The usage of inside of the "Endpoint weighting and generation quality" paragraph is not so clear to me; can you elaborate more on what that is and why is that important? A more clear explanation in the later versions of the main paper will be benificial.
I will raise my score to 3 if both questions are addressed. Edit: The authors addressed my questions thoroughly. I've raised my rating to 3.
7 comments
W2 — Definitions of Aligned and Mismatched Supervision
Thank you for suggesting a clearer presentation. We have clarified the definitions in §3.2 and the notation guide in Appendix A. Prediction–loss alignment means that the directly predicted variable and the training supervision occupy the same space. Mismatch occurs when the prediction is converted to another space before applying the training loss.
The main MSE combinations are:
| Direct network prediction | Training supervision | Classification |
|---|---|---|
| Clean signal | Signal-space -MSE | Aligned |
| Velocity | Velocity-space -MSE | Aligned |
| Clean signal | Velocity-space -MSE after conversion | Mismatched |
In our -prediction comparisons, “aligned” means -prediction + -MSE, and “mismatched” means -prediction + converted -MSE. More generally, direct velocity regression also satisfies the alignment principle.
For the same -predictor, exact conversion without a denominator floor gives
The additional endpoint factor introduced by this conversion is the distinction analyzed in the paper (§3.2, Equation (5)). Signal-space BCE provides another form of direct supervision for binary targets through logits.
The revised manuscript uses these terms consistently and includes a notation table for the associated quantities (Appendix A, Table 5).
W3 — Euler-50 versus Heun-50 at Fixed Checkpoints
We have added a fixed-checkpoint solver comparison for all six JiT -prediction configurations. For training seed 0, we evaluate each 400-epoch checkpoint using both Euler-50 and Heun-50. The checkpoint, EMA, generation seed, guidance, generation denominator floor, and FID reference set are held fixed. Each evaluation generates 50,000 images.
FID-50K results (Appendix B.10, Table 16):
| Training configuration | Training sampler | Heun-50 | Euler-50 |
|---|---|---|---|
| Mismatched -MSE, floor | Uniform | 402.37 | 404.58 |
| Mismatched -MSE, floor | Logit-Normal | 15.14 | 14.01 |
| Mismatched -MSE, floor | Uniform | 14.69 | 14.35 |
| Mismatched -MSE, floor | Logit-Normal | 15.04 | 14.01 |
| Aligned -MSE | Uniform | 16.80 | 16.29 |
| Aligned -MSE | Logit-Normal | 24.03 | 23.04 |
Both solvers preserve the severe generation failure of mismatch+Uniform under weak endpoint protection and effective generation for the other configurations. Thus, the principal qualitative contrast persists when the inference solver is changed.
At the standard training floor, the small sampler ranking reverses: Uniform has the lower FID with Heun, whereas Logit-Normal has the lower FID with Euler. Solver choice can therefore affect fine FID rankings without removing the observed endpoint contrast. These are paired evaluations of the same seed-0 checkpoints, rather than multi-seed solver statistics.
Both schemes use 50 integration steps. Our Heun implementation performs 49 Heun updates followed by one Euler update, requiring 99 network evaluations; Euler-50 requires 50. The comparison therefore matches integration steps rather than computational cost.
The original objective–sampler comparisons already used a consistent solver within each task. The additional Euler-50 results also provide a JiT evaluation using the same solver and step count as Binary-MNIST, directly addressing the requested consistency check.
W1 — Ten-Seed Results and Run-to-Run Variability
Thank you for suggesting additional training seeds. We have expanded all eight Binary-MNIST objective–sampler configurations to ten independent training seeds (42–51). The updated FID-50K results are:
Values are means ± sample standard deviations across ten independently trained generators. Every run uses the final 100-epoch checkpoint and the same evaluation protocol, with generation randomness held fixed. Thus, the reported variability reflects training-seed variation.
The strong sampler dependence of mismatched training is reproduced across training seeds: Uniform sampling yields severe generation degradation, whereas Logit-Normal supports effective generation. The aligned objectives support effective generation under both tested samplers. These results support the robustness comparison without implying identical FID across samplers or a universally optimal objective.
Figure 3 now reports ten-seed FID statistics. Its training curves and sample grids retain the explicitly identified seed-42 visualization. All individual seed scores are provided in Appendix B.3, Table 9, and the aggregate statistics appear in Appendix B.8, Table 12.
We also completed ten paired training seeds for the Binary-MNIST fixed-objective control (Appendix B.8, Table 13) and three training seeds for each of the six JiT x-prediction configurations, using a common 400-epoch evaluation protocol (Appendix B.9, Tables 14–15). These additional results quantify variability beyond the Binary-MNIST main comparison.