- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides the analysis on the generalization mechanics of flow matching on discrete/categorical data by systematically removing the two candidate explanations. The authors construct the optimal denoiser and the optimal flow map with rigorous derivation and then design the related experiments. Through solid experiments with matched architecture, parameters and training budget, the authors conclude that the reason for the generalization of flow matching is neither the stochastic target nor the sampler.
Critical strengths
-
The paper is well-structured and easy to follow. The authors constructed the closed-form optimal denoiser and also the optimal flow map with rigorous derivation.
-
Based on these formulations, the authors conducted solid experiments to isolate the reasons of the generalization.
Critical weaknesses
-
The authors claimed in Sec 4.3 that FLM generalize at training-set size on every distribution. However, as shown in Table 6, FLM under and RHM distribution achieves 0% memorization score but only get 23% validation. The generalization should be evaludated based both on the validity and memorization. If most generated samples from FLM are invalid, to claim the generalization property of FLM is unreasonable.
-
The ablation study on the impact of different rollout integration steps on model performance is insufficient. The authors chose the 8 integration steps for all the experiments based on an empirical result that 8 integration steps can achieve over 97% matching rate. However, it remains unclear whether the potential mismatched training points would affect the training procedure and the generalization. While the experiments of larger integration steps with are indeed prohibitive, this concern can be addressed by conducting ablations on varying integration steps (both fewer or larger, e.g., 4 or 16 steps) at smaller training-set size (e.g., ).
Questions for authors
-
I understand that it requires two assumptions (the limit exists and the unique maximizer) to prove Proposition 3.2, but would these assumptions holds in actual experiments?
-
The inference compute differs among the three models. In Appendix C, for example, FLM takes 128 Euler steps, while MDLM uses steps. Could the authors clarify this point?
5 comments
Thanks for the detailed rebuttal. I have carefully read the other reviews and the authors’ responses and my concerns have been addressed.
However, I'm confused about the argument "under a one-hot embedding, every point between two training sequences lies off the support", as stated in the Rebuttal to Reviewer 6rN7. While interpolation between distinct one-hot sequences does leave the discrete support, the interpolation between continuous data is also not guaranteed to remain on the manifold, please see [R1]. Therefore, this observation seems insufficient to claim the difference between one-hot and continuous data. Could the authors clarify this?
Also, I could not locate these additions, discussions and rebuttals with other reviewers in the currently available revised manuscript and I would suggest adding them to the revision.
Overall, I think the rebuttal improves the paper's clarity and I will maintain my current score.
[R1] Kacper Kapusniak et al, "Metric Flow Matching for Smooth Interpolations on the Data Manifold", NeurIPS 2024.
We thank the reviewer for the positive assessment and for engaging in further discussion. We will cite the shared reference in our paper. We address the concern below.
- [1] shows that straight paths can lie outside the data manifold. However, we believe it in fact supports our claim. [1] builds on the manifold hypothesis, under which data such as images, while embedded in a high-dimensional pixel space, lie on a curved low-dimensional manifold.
- [2] measures this directly, finding that natural image datasets have a low intrinsic dimension, and that such low-dimensional data is easier for networks to learn. [2] further notes that this manifold is locally connected, i.e. each natural image is surrounded by other highly similar images. This is why [1] can construct paths between data points that stay on the manifold.
- Under a one-hot embedding, the support is a finite set of isolated points, with no locally connected structure, and no continuous path between two distinct sequences stays on it. This matters because prior explanations of why diffusion models generalize rely on this local structure, e.g. by stitching together local patches of training images [3].
We hope this addresses the reviewer's question. We are finalizing the revised manuscript with the new discussions and results, and will upload it shortly.
[1] Kapuśniak et al. Metric flow matching for smooth interpolations on the data manifold, '24
[2] Pope et al. The intrinsic dimension of images and its impact on learning, '21
[3] Kamb & Ganguli. An analytic theory of creativity in convolutional diffusion models, '24
We thank the reviewer for the careful and positive review, and for finding our experiments well designed.
W1: Evaluating generalization.
We agree that generalization should be evaluated on both validity and memorization. To incorporate this, we report the fraction of samples that are both valid and novel, validity × (1 − memorization). To further strengthen the evaluation, we have also added a fourth distribution, palindromes of length L=16 over V=16 tokens, trained with the recipe of Fig. 6.
Validity / memorization (valid and novel) (%), ∣D∣=104.
Under this combined metric, FLM produces the most valid and novel samples on every distribution, including RHM. We therefore believe the claim that FLM generalizes is supported, and we will qualify it in Sec. 4.3 accordingly.
The low validity of FLM on RHM is due to undertraining. With longer training (1M steps instead of 200k), FLM reaches 90.1% validity on RHM at larger training sets, while staying at 0% memorization. We hypothesize that FLM learns RHM more slowly because it noises every position, which weakens the training signal for the long-range dependency of the grammar. We will report the longer RHM runs and the palindromes in the paper.
W2: Integration steps for FLM-X*.
One-step generalization is stable as target accuracy increases. We first note that a less accurate target is still a valid sample, since the target is the training point with the largest posterior weight at the end of integration. Therefore, integration error only changes which training point a noise sample is assigned to. As suggested, we ran the ablation on Dyck at ∣D∣=105, where target error is largest (Tab. 8), training FLM-X* with targets from 4, 8 and 16 Euler steps.
Dyck (L=10), ∣D∣=105, 16,384 samples (%).
Validity and memorization stay the same within sampling noise as target agreement increases, so the mismatched targets do not affect generalization. We note that while the validity of FLM-X* is low, its valid samples cover the distribution well, with an effective support of 98% on permutation (Reviewer ZwbK, W1). With more samples, the memorization of FLM-X* here is 1.9% rather than the 0.0% in Tab. 3.
Q1: Assumptions of Proposition 3.2.
We thank the reviewer for this question. Both assumptions hold in all our experiments. We checked them numerically on the 1,024 fixed noise samples of Fig. 5, integrating with 1,024 Euler steps up to t=0.999.
Smallest gap between the two largest inner products ⟨xt,x(i)⟩ at t=0.999.
The gap stays well above zero for every sample, so the maximizer is unique (Assumption 2). Moreover, every trajectory converges to a training point, so the limit exists (Assumption 1). We note that one can construct examples that violate Assumption 2, e.g. noise on the diagonal between (1,0) and (0,1), but such instances have probability zero. We will include this discussion after Prop. 3.2.
Q2: Inference compute across paradigms.
We thank the reviewer for raising this point. MDLM's step count is set by its sampler. We use a uniform-order sampler that unmasks one position per step, so it takes exactly L=16 steps and avoids the factorization error of unmasking several tokens at once. To check that a larger budget does not change our conclusions, we also sampled the same checkpoints with the standard ancestral sampler at up to 256 steps.
MDLM on permutation (L=16), validity / memorization (%).
The memorization threshold remains the same under every sampler. More ancestral steps raise validity, but it stays below the uniform sampler at 16 steps. We will state the step counts of each paradigm in App. C and add this comparison.
We hope these responses address the reviewer's concerns, and would be grateful if they could be considered in the assessment.