- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Function Projection for Flow Matching (FP-FM), a framework for adapting flow-based generative models to unseen target distributions using samples from the target distribution as context. The key idea is to learn a set of basis functions that span the velocity fields associated with training distributions. Given samples from a new target distribution, FP-FM estimates the corresponding velocity field by solving a least-squares projection problem over the learned basis, without requiring gradient-based fine-tuning at inference time. The paper introduces three variants—Static, Temporal, and Dynamic FP-FM—where the projection coefficients are respectively fixed, time-dependent, or state-and-time-dependent, providing different trade-offs between expressivity and computational cost. The method is evaluated on 2D synthetic data, MNIST, and ImageNet under training-distribution and unseen-distribution settings. The results show that the more expressive FP-FM variants, particularly Dynamic FP-FM, achieve competitive or improved precision, recall, and FID on unseen distributions, while avoiding per-target gradient-based fine-tuning.
Critical strengths
-
The proposed formulation is interesting and conceptually clean. Representing distribution-specific velocity fields using a shared set of learned basis functions provides an interesting function-space perspective on adapting flow-matching models. The Static, Temporal, and Dynamic variants also provide a clear progression in expressivity and computational cost.
-
Given context samples from a new target distribution, the method obtains its coefficients through least-squares projection rather than updating model parameters. The experiments demonstrate substantial computational savings over per-target fine-tuning while retaining competitive performance in several settings.
Critical weaknesses
-
The ablation study does not sufficiently validate the method in the most important ImageNet setting. The paper studies the number of context shots and basis functions primarily on MNIST. However, these factors are particularly important for understanding the practicality and scalability of FP-FM on ImageNet. For example, an ImageNet evaluation across different context sizes (e.g., 8/16/32/64/128) would clarify how many target examples are actually required before FP-FM becomes effective.
-
The baseline comparison is insufficient to establish the competitiveness of FP-FM for modern sample-based generative adaptation. The main baselines are unconditional/conditional flow matching, classifier/distribution guidance, and fine-tuning (LoRA for ImageNet). While these baselines provide useful reference points, they do not sufficiently cover recent approaches that also perform example-conditioned or context-based generative adaptation.
-
The strong performance of Dynamic FP-FM is potentially confounded by its tendency to reconstruct the provided context examples. The authors explicitly observe that Dynamic FP-FM “largely reconstructs the images provided in context, introducing only minor visual perturbations,” and later acknowledge that it tends to generate samples similar to the context. This raises an important question regarding whether its strong precision/FID primarily reflects successful modeling of the underlying target distribution or a form of context reconstruction. Although the appendix provides a nearest-neighbor-based overfitting analysis, more comprehensive diversity and memorization evaluations would be necessary to clarify this issue.
-
The evaluation provides limited evidence for adaptation to substantially unseen distributions. In the main ImageNet experiment, the model is trained on 900 ImageNet classes, while the remaining 100 classes are treated as unseen targets. Moreover, FP-FM receives 128 context images from each target class at inference time. It remains unclear how well the proposed basis functions generalize under larger distribution shifts or when substantially fewer target samples are available. This is particularly relevant because the paper acknowledges that the method is expected to perform poorly in few-shot regimes.
Questions for authors
Please see the Critical Weaknesses.
The ablation study does not sufficiently validate the method in the most important ImageNet setting. .... However, these factors are particularly important for understanding the practicality and scalability of FP-FM on ImageNet. For example, an ImageNet evaluation across different context sizes (e.g., 8/16/32/64/128) would clarify how many target examples are actually required before FP-FM becomes effective.
We run your desired ablation on Imagenet. We find that Temporal FP-FM outperforms LoRA finetuning between 32 and 64 context images and beyond. We have added a discussion of these results to the appendix.
We additionally ablate the number of basis functions, k, on imagenet for Temporal FP-FM. We focus on Temporal as it is the best performing variant on ImageNet.
We find that Temporal FP-FM is not too sensitive to the number of basis functions. The difference between 64 and 512 basis functions is only about 7 FID. We have added a discussion of these results to the appendix.
The baseline comparison is insufficient to establish the competitiveness of FP-FM for modern sample-based generative adaptation. The main baselines are unconditional/conditional flow matching, classifier/distribution guidance, and fine-tuning (LoRA for ImageNet). While these baselines provide useful reference points, they do not sufficiently cover recent approaches that also perform example-conditioned or context-based generative adaptation.
We create one additional baseline similar to a set-based autoencoder. It first encodes each image into a latent space using a VAE. Then it passes these latents through a MLP and averages the results, outputting a class representation. This representation is then used as input to a conditional model. We find the following comparison to temporal FP-FM:
If you have any other baselines in mind, we are happy to run anything that fits into the setting.
The strong performance of Dynamic FP-FM is potentially confounded by its tendency to reconstruct the provided context examples. ... Although the appendix provides a nearest-neighbor-based overfitting analysis, more comprehensive diversity and memorization evaluations would be necessary to clarify this issue.
As mentioned in the paper, the expressivity of Dynamic FP-FM allows it to overfit to the provided samples, and therefore it tends to (usually) reproduce these samples with added noise or visual artifacts. For image domains, this is likely unsatisfactory, which is why we suggest practitioners to use Temporal FP-FM in line 439. However, there may be other domains where practitioners prefer fidelity over diversity, in which case it may find uses. Please also see the discussion with reviewer fsZk, where we revised the discussion over overfitting in the appendix.
The evaluation provides limited evidence for adaptation to substantially unseen distributions. In the main ImageNet experiment, the model is trained on 900 ImageNet classes, while the remaining 100 classes are treated as unseen targets. Moreover, FP-FM receives 128 context images from each target class at inference time. It remains unclear how well the proposed basis functions generalize under larger distribution shifts
We retrain a model on all non-animal imagenet classes and evaluate on animals. This is quite a large distribution shift because many animals have idiosyncratic visual features, such as fur texture and eyes. Unsurprisingly, temporal FP-FM has higher FID on the heldout animals than on the training classes. Qualitatively, we observe the model can recreate the textures but fails with global structure. For example, for a dog class, it outputs something that looks like a big ball of fur but fails to correctly create four legs and a head. We have added this ablation to Appendix E.6.