- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a modification of the flow matching loss where the squared Euclidean norm is replaced with the Euclidean norm. The minimizer of the resulting loss is a conditional geometric median of the distribution of given , which is known to be more robust to corruptions of the data distribution than the conditional mean. The authors derive sufficient conditions under which the minimizer of the Euclidean distance flow matching loss coincides with the minimizer of the traditional flow matching loss, which ensures that Euclidean distance flow matching generates the correct distribution. The authors also prove robustness bounds for Euclidean distance flow matching when the data distribution is corrupted, both for the velocity field and the generated distribution. Experimental results on image datasets, comprising different mechanisms for corrupting the data, complement these robustness guarantees.
Critical strengths
The paper studies an important problem, namely improving the performance of generative models under imperfect training data. The method proposed by the paper is mathematically principled and yields performance guarantees over standard flow matching in certain settings.
Critical weaknesses
The main limitation is that the experimental results do not seem very convincing. Other than for the random noise experiment, the results obtained by Euclidean distance flow matching seem comparable or even worse than those obtained by standard flow matching. Such numerical results make it difficult to endorse a methodology that requires training a velocity field from scratch (as opposed to modifying a pre-trained model).
Another (less significant) weakness is that it seems difficult in general to characterize whether the generated distribution (with the exact Euclidean distance velocity field, under exact integration) is equal to the data distribution. This is guaranteed when the mean and the geometric median of the conditional distribution given are equal for each time the authors prove this holds for Gaussian data, but it is unclear beyond Gaussian data when this holds.
Questions for authors
To be sure that I understand: in your experiments, the architecture and training data are the same for training in both the standard flow matching and Euclidean flow matching settings?
Minor comments:
- I found the proof of Theorem 4.2 quite difficult to follow, even though it was well-written on a per-line basis. It would be helpful to break the proof up into several key steps and explain how the steps combine to prove the claim.
- The upper bound in Theorem 4.2 can be simplified in terms of the magnitude of the contamination:
Writing the result in this way might make the bound clearer.
- Similarly, the bound in Theorem 4.3 can be further upper bounded by
2 comments
Thank you for your reply. Your response helps me understand the trade-off that your model makes, and when it might or might not make sense to use it. I find the contribution sound and interesting, but both the theory and the experiments have weaknesses (as described in my review). I will maintain my score, as I think the paper is closer to borderline.
Weaknesses:
Regarding experimental results:
On an established flow-matching architecture (on this data set) we consider these results convincing since we achieve a lower FID than standard FM for five of the seven contamination mechanisms across all tested contamination rates. The gains are moderate and grow with ϵ, while the standard objective is better for both blur mechanisms. We discuss this in our answer to question 1 from reviewer 3mqt.
Regarding training from scratch: The L2,1 objective changes only the loss, at the same training cost as standard FM, and it targets the setting in which the available training data are themselves contaminated. Its population minimizer does not depend on the initialization, so the loss can also be used to fine-tune a pre-trained model on contaminated data although we have not evaluated this.
Regarding the characterization: A general characterization would be valuable, but it appears to be out of reach. Apart from the Gaussian case, we cannot characterize when the two fields coincide. Note that this is only sufficient, not necessary, for reproducing the target. In general, however, we do not expect the L2,1 flow to reproduce non-Gaussian targets exactly. As in Theorem 4.2, it can shrink minority components of the target, whether they are contamination or not. This is the trade-off stated in the introduction: We give up exact reproduction of the observed distribution in exchange for robustness to contamination.
Question: Yes, the neural network architectures and hyperparameter choices are fixed. Also, both methods in one run see the same training data, because the data pipeline and seeds are controlled.
Minor comments: We thank the reviewer for these helpful suggestions and will incorporate them. In particular, we will restructure the proof and provide explanations of how the statements in the proof will be used.