- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how independent and optimal transport (OT) couplings affect feature learning in flow matching. The authors consider a simplified data model supported on a low-dimensional subspace with a nonlinear feature along one unknown direction. They show that near the data endpoint, the nonlinear component of the population velocity vanishes under independent coupling but remains nonzero under OT coupling. They further analyze gradient flow training with a cubic-neuron model and show that feature learning becomes increasingly slow under independent coupling. The theoretical findings are supported by synthetic experiments and controlled experiments using MNIST and CIFAR-10 decoder manifolds.
Critical strengths
-
Interesting theoretical insight. The paper provides a new perspective on the benefits of OT coupling beyond the commonly discussed reduction in training variance and straighter transport paths. In particular, the analysis connects the choice of coupling to the strength of nonlinear feature-learning signals and the resulting training dynamics.
-
Rigorous theoretical analysis. The paper establishes quantitative differences between independent and OT couplings, including the near-endpoint scaling of nonlinear signal strength and feature-learning time. The synthetic experiments provide supporting evidence for these theoretical predictions.
Critical weaknesses
-
Restrictive assumptions in the theoretical analysis. The main theoretical results rely on strong simplifying assumptions, including a low-dimensional linear subspace, a single nonlinear feature direction, and a highly simplified learning model. In particular, Theorem 2 considers a single cubic neuron trained at a fixed flow time under population gradient flow. While these assumptions make the analysis tractable, it remains unclear to what extent the conclusions generalize to practical flow matching models with more complex data distributions and neural network architectures.
-
Limited connection between the theoretical findings and practical performance. The paper does not establish a clear theoretical or empirical connection between its feature-learning results and the performance of practical flow matching models, which typically use shared parameters and multi-time training. Although the authors introduce theoretical measures of nonlinear signal strength and feature-learning time, it remains unclear whether these quantities predict meaningful improvements in training efficiency or generation quality. Ideally, the analysis would identify a theoretically interpretable and empirically measurable quantity that correlates with practical performance, and validate this relationship in standard flow matching settings. Without such evidence, the practical significance of the theoretical findings remains limited.
Questions for authors
Please see the weakness
3 comments
Thank you for the detailed clarification and additional experiments. However, my main concern remains the practical significance of the findings. While the analysis establishes a connection between nonlinear signal strength and feature-learning time, it is still unclear whether this translates into meaningful improvements in training efficiency or generation quality for practical flow matching models. I believe an experiment on a more realistic benchmark with a standard flow matching architecture would substantially strengthen the paper. I will maintain my score.
Thank you for the follow-up. We respond to the remaining concern below.
Practical benefit of OT coupling. That OT coupling helps standard flow matching models is already documented in other papers: minibatch OT improves the efficiency of training and inference [1] and gives better low-cost sample generation on downsampled ImageNet [2]. These works attribute the gains to lower gradient variance and straighter flows. While our paper does not claim a new practical gain, it identifies an additional mechanism behind the known ones: the nonlinear signal near the data endpoint persists under OT and vanishes under independent coupling.
Why we use controlled experiments. Comparing the two couplings on a realistic benchmark would reproduce the results of [1, 2], but it could not separate our mechanism from variance reduction. Separating them requires the population velocity, which is available in our synthetic setting (the population-label control in Appendix C.4) but not on real data.
What the analysis predicts for practical models. In our experiments, the gap is small when the feature is shared over the full time window (ratio 1.21 in Figure 3) and large when training is concentrated near the endpoint or features are time-specific (ratios 46.4 and 3.65). The signal energy can be estimated on real data to test this prediction. Such a test at realistic scale is a natural next step, which we could not complete in this revision.
[1] Tong et al. Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. TMLR, 2024. [2] Pooladian et al. Multisample Flow Matching: Straightening Flows with Minibatch Couplings. ICML, 2023.
Thank you for the helpful comments. We appreciate your recognition of the analysis linking coupling-dependent nonlinear signals to feature-learning times.
W1 (Assumptions)
Our theoretical setting assumes data supported on a low-dimensional linear subspace, but related low-dimensional assumptions are often used in feature-learning theory [1, 2]. For example, multi-index models assume that the output depends on only a few linear projections of the input, while single-index models restrict this dependence to one direction. In our setting, this subspace model allows us to separate support recovery from nonlinear feature learning.
Moreover, Figures 3–4 extend the comparison to models with features shared across times and CNNs on curved decoder manifolds, and the new Figure 5 (Appendix C.4) uses a standard two-layer tanh MLP with a non-polynomial link. These experiments provide evidence beyond the fixed-time cubic-neuron setting, although extending the theoretical guarantees to these models requires further analysis.
[1] Damian et al. Neural Networks can Learn Representations with Gradient Descent. COLT, 2022.
[2] Jimmy Ba et al. High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation. NeurIPS, 2022.
A single cubic neuron provides a minimal model for analyzing the relationship between nonlinear signal strength and feature-learning time. For the symmetric experimental target, the residual after removing the linear component is odd, making degree three the lowest nonlinear degree that can correlate with it. Symmetry alone does not guarantee nonzero cubic correlation; Theorem 2 explicitly assumes conditions that ensure it near the endpoint. We have added this explanation to the main text.
We distinguish the signal comparison from the training-time guarantee. Under its distributional assumptions, Theorem 1 compares nonlinear signals in the population velocity without restricting the learner's architecture. Theorem 2 gives a training-time guarantee for a cubic neuron. The data-generating function H need not be cubic, provided it satisfies the theorem's assumptions, but training times for more general networks require a separate analysis.
W2(a) (Multi-time training)
We agree that the role of feature sharing needs clarification. Figure 3(b) in Section 4 compares full-window and endpoint-window training on the same target using a cubic-neuron model. Figure 3(c) compares models with separate feature directions for different time bins and a shared support basis. Together, these results suggest that, when the feature direction is shared across times, signals at earlier times can mitigate slow feature learning near the endpoint.
W2(b) (Practical performance)
The quantity our analysis identifies is the nonlinear signal energy EC(t) (Eq. (12)): the mean squared part of the population velocity that no linear predictor can fit. It is defined for any target and architecture, and can be estimated from the gap between the best linear and the best nonlinear regression loss at time t. For the cubic model, the relevant part of this signal is the residual cubic correlation κC(t) (Eq. (20)). In our controlled experiments, the coupling with the larger signal learns the feature faster: for the cubic-neuron model (Figures 2–3) and, in the new Figure 5, for a standard two-layer tanh MLP. We have not tested practical models or generation quality, and we do not claim such gains (Section 5).