- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes two persistent homology-based regularization objectives for flow matching models. The fidelity loss uses cubical complexes to align the topological features of predicted and target velocity fields, while the diversity loss uses Vietoris–Rips complexes to preserve the topology of velocity predictions across a batch. Experiments on MNIST, MSCOCO, and ImageNet show that these objectives can improve generation quality and diversity through additional fine-tuning, particularly when using RAE latents.
Critical strengths
- The paper explores an interesting direction for flow matching regularization by incorporating topological information beyond standard pointwise losses.
- The proposed batch-level diversity loss is a useful contribution, with encouraging improvements in FID and distribution coverage on RAE-based models.
- The paper provides theoretical analysis, ablation studies, and computational evaluations to support the proposed approach.
Critical weaknesses
- The benefit of topological regularization over simpler geometric constraints is not fully established. Since the diversity loss also includes geometric alignment between samples, it remains unclear how much of the improvement comes specifically from persistent homology.
- The improvements are not consistent across different settings. While the RAE experiments show promising results, the method does not improve performance with SD-VAE latents or FID in the additional SiT/XL-2 experiment. A clearer understanding of when the proposed losses are effective would strengthen the paper.
Questions for authors
- Could the authors compare the proposed diversity loss with a simpler pairwise distance-matching loss, such as minimizing the differences between pairwise cosine distances of predicted and target velocity vectors within a batch, without using persistent homology? This comparison would help clarify the benefit of the proposed topological formulation.
- Could the authors further explain why the method performs differently with RAE and SD-VAE latents? What might account for the limited improvement in the additional SiT/XL-2 experiment?
- The theoretical analysis focuses on simplified forms of the topological objective. To what extent do these results support the multiplicative diversity loss used in the experiments?
- How sensitive is the proposed diversity loss to batch size? Could the authors examine whether smaller batches affect the reliability of the estimated topology and the resulting generation performance?
4 comments
Batch size analysis
We apologize for the lengthy response. We continue in this comment.
Q3. The theoretical analysis focuses on simplified forms of the topological objective. To what extent do these results support the multiplicative diversity loss used in the experiments?
Good question. The short answer is we cannot with certainty guarantee that a similar bound to Claim B.8 exists for the product version of the loss. We only suspect it to be the case due to two facts: (1) empirically, we obtain better results for the product than the sum, and (2) results for the product loss are close but slightly better to the case when we detach the geometrical alignment component so gradients only flow through the topological component (and we do have theoretical guarantees for this case, see Corollary B.4).
In more detail, under the setup of Claim B.8, let Similarly let etc. Then, the proof of the bound for the product loss follows that of Claim B.8, and we have:
where the last term is the joint central moment. This, together with are the two elements not present in the proof of Claim B.8. Now, B and C are not independent, and when expanding this covariance things are difficult to untangle. On the other hand, the joint moment is also very complex. It would be interesting to let an AI agent work on this problem, but we have not had the time and resources to do it. In any case, as we mention in the appendix, we believe the empirical evidence in favor of the product version to be substantial. For any use-case requiring stronger guarantees, the sum version can be used without a problem.
Q4. How sensitive is the proposed diversity loss to batch size?
This is an interesting question, that helps illustrate the way in which the proposed Vietoris-Rips filtration loss regularizes a FM model. Overall, smaller batches do not greatly degrade performance. We provide some analysis below, experiments with RAE baseline, 10k steps ft with div loss only (using JL projections), cfg1.5. For a fair comparison, we use gradient accumulation of 2 when choosing bs 32, and of 4 for bs 16, so that the VR-filtration is computed over a smaller sample, but overall we have comparable gradient scales.
| method | fid | precision | recall | density | coverage |
|---|---|---|---|---|---|
| bs 64 | 7.921 | 0.861 | 0.497 | 1.227 | 0.889 |
| bs 32 | 7.957 | 0.857 | 0.505 | 1.214 | 0.886 |
| bs 16 | 7.949 | 0.859 | 0.504 | 1.223 | 0.885 |
Explanation: When optimizing with MSE, gradients are spread out across pixels, irrespective of their "importance". In essence, our proposed Vietoris-Rips (div) loss assigns topological importance to pixels, so that optimization happens over those that help match the topological signatures of prediction and target. But at this scale (bs << num_classes, with high probability only one sample per class per batch at all batch sizes tested) we only capture global topological features of the data (such as class directions with respect to one another). Intuitively, classes where the model underperforms receive a stronger signal that pushes specific pixels so that it gets correctly positioned with respect to the others in the batch. This is why even with smaller batches, the loss still provides meaningful signals. We do not know what would happen in larger bs regimes, where more fine-grained intra-class topology would start playing a role.
References: [1] Ma, Nanye, et al. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." ECCV 2024.
I appreciate the authors' detailed response and supplementary experiments, which alleviated some of my concerns regarding the simple method comparison and batch size. My primary concern remains that the method's effectiveness seems heavily dependent on the RAE representation; its gains in other latent spaces and the reasons for this are not yet fully understood. Therefore, I am maintaining my original rating for now, while acknowledging that the supplementary experiments further support the method's value in the RAE setting.
Thank you for taking the time to consider our response and supplementary experiments. We appreciate that the additional results have helped alleviate some of your initial concerns.
Regarding the dependence on RAE representations, we have revised the manuscript to better articulate the contribution of our work, highlighting the effectiveness of the proposed regularization in the RAE setting while more explicitly discussing the limitations of our evaluation in VAE spaces.
With respect to the underlying reasons for this lack of generalization, as we mentioned in our previous response, we suspect high-dimensional artifacts in the persistence diagrams (PDs) to play a role. To further investigate this possibility, we conducted an additional experiment using SiT/XL-2 (following the same setup as in our previous response), trained for 15k steps with MSE and MSE + Div, this time applying a Johnson–Lindenstrauss projection to 128 dimensions (rather than 512 dimensions in the previous experiment).
| Method | FID | Precision | Recall | Density | Coverage |
|---|---|---|---|---|---|
| MSE | 3.8143 | 0.8336 | 0.6699 | 1.2053 | 0.9507 |
| MSE + Div (l2 dist & proj) | 3.8070 | 0.8204 | 0.6771 | 1.1471 | 0.9417 |
Interestingly, we begin to observe a behavior more similar to that in our RAE experiments, with regularization improving Recall and FID at the expense of Precision. While these results are still preliminary and insufficient to establish the potential effectiveness of our method in VAE spaces, they suggest that dimensionality reduction, together with the right choice of metric, could be a promising way to address this issue. We have incorporated this discussion into the revised manuscript (Appendices B.3 and B.4)..
Thank you again for your constructive feedback.
Comparison with simpler constrains, and batch size analysis
We thank the reviewer for adding relevant questions and comments. We address your concerns below.
W1/Q1. The benefit of topological regularization over simpler geometric constraints is not fully established.
We thank the reviewer for the suggestion to compare with simpler regularizers. We add results below. We compare (1) cosine distance regularization (pred vs tgt), (2) random pairwise cosine distance (between randomly selected pairs in the batch, n=16), (3) geometric alingment component of the diversity loss only (equivalent to pairwise cosine distance matching, but between critical indices), (4) geometric alignment component + MSE and (5) only the diversity loss. All results for 10k finetuning steps on the DiTDH-S RAE baseline, cfg 1.5.
We observe the div loss achieves better FID and Recall than other regularization methods. MSE + Fid wins in precision and density (higher loss signal). Note: due to computational limitations, we had to compute the diversity loss with Johnson-Lindenstrauss projection, more details below. In most of our experiments, the method after projection is an upper bound for the results without projection, so we could probably expect even better numbers. MSE + Div (without proj.) achieves FID of 7.203, see paper.
W2/Q2. The improvements are not consistent across different settings (VAE latents).
We acknowledge this limitation, and are aware of its importance. We admit we have no clear solution at this point. We add some results we got below, as well as our research process and the hypothesis we tested in case it can shed some light on the issue.
We believe the root cause to be the curse of dimensionality when computing VR-filtrations, as discussed in Appendix B.3. Given that VR computations depend only on pairwise distances, noise in high dimensional spaces can produce large artifacts in the PD that renders it unusable. In RAE spaces, where cosine distances are meaningful, this problem is alleviated, but we do not have that privilege in VAE or Pixel space. We've taken this limitation very seriously, and have proposed two solutions: (1) project data to a lower dimensional space via a random sparse projection (leveraging Johnson-Lindenstrauss lemma, which ensures such a projection nearly preserves pairwise distances), and (2) using different distance metrics less sensitive to this issue, such as spectral distances. Both approaches work good with RAE latents (improved computational performance, at the price of slight degradation of the improvements).
However, this does not seem to be enough. Here some results with SiT/XL-2 [1], a famous VAE state-of-the-art model:
Corresponding to 8-10k steps of finetuning, CFG 1.5. It may be (1) that 10k ft steps for a model pretrained at a complete different scale (SiT/XL is trained 7M steps) is not enough to appreciate differences wrt MSE, or (2) that neither spectral, l2 or cosine distances are optimal for VAE spaces (in fact, cosine even increases recall). In any case, we do not think there is a fundamental limitation with our method (which is a very general idea that should work as long as we can compute meaningful PDs), but rather a lack of understanding of the right approach to extract topological information from VAE spaces, a direction in which more work is needed. Due to computational limitations we could not test 1, and are still trying alternatives to 2. Nonetheless, we still believe this work to be a useful proof-of-concept, that consistently and cheaply improves models in RAE space (see response to reviewer M4jn for statistical tests), and a step towards geometrical regularization of FM models.