- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes geometric PCA (GPCA) that respects the underlying information geometry for discrete data and further uses it to derive a Riemannian geometry for discrete flow matching. Instead of directly modeling on the simplex, the authors run the generative framework on the latent space, which may provide more meaningful semantics. On binarized MNIST, the paper showed improved FID; other tasks also showed better performance.
Critical strengths
- The paper is mathematically solid to me. Its proposed use of exponential geometry respects the underlying information geometry for discrete data, potentially benefiting the generative process.
- For the proposed geometric PCA, it improves on standard PCA in terms of final generative performance.
- The paper benchmarks on a wide variety of generative tasks across different domains, and improvements can be shown across tasks.
- Code is provided for reproducibility.
Critical weaknesses
- I find it very hard to identify the true contribution of this paper, even though I have experience with discrete flow matching models. The authors seem to claim their contribution as proposing a "geometric PCA" and using the exponential geometry for discrete flow matching. However, after reading the whole paper, these two contributions are already well-explored in the existing literature (see below), and the more credible contribution to me is the combination of the latent approach with the e-geometry, which, to the best of my knowledge, is indeed new.
- The geometric PCA does not provide solid theoretical or practical motivations and justifications.
- Generalized PCA already supplies the categorical low-rank representation and nonlinear decoder, as the paper acknowledges. As the simplex is embeddable in the ambient space (either directly as the simplex or as the sphere to preserve the metric), there is no motivation to support geometric PCA.
- Further, more advanced dimension reduction techniques like t-SNE or UMAP theoretically preserve local geometry, a much stronger theoretical guarantee that can be transferred to smooth Riemannian manifolds, as they are locally Euclidean anyway.
- Recent "continuous discrete language models" like [1] predominantly use low-dim embeddings to achieve very good performance. Compared to fixed PCA vectors, I believe learnable embeddings provide a better option than fixed PCA vectors.
- Additionally, the discrete data are one-hot vectors instead of continuous vectors. It remains unclear whether PCA on such data can lead to meaningful results.
- Last, only ordinary PCA is compared. No ablation study on other popular dimension reduction techniques is provided. PCA dimension is also chosen empirically.
- The exponential geometry is already studied in previous work, and I see no new methodological novelty in this part. As the paper already cited, the most relevant work of Assignment Flow and Alpha-Flow already used such geometry for generative modeling. Specifically, both benchmark the binarized MNIST task, and Assignment Flow additionally has the segmentation results. However, they are never compared.
- Unlike the standard Fisher-Rao geometry, the e-geometry is undefined at the boundary. The authors suggest using a small cap on the one-hot target. However, this may impact both the theoretical results in Prop. 3.5, as well as empirical performance. Although this value is chosen differently for different tasks, the paper does not have an ablation on that.
- The binarized MNIST FID is not comparable to baselines due to unmatched sampling numbers. The cited SFM uses 1k generations, whereas this paper uses 10k, which may account for the drastic drop in FID.
- As mentioned above, highly related work like Assignment Flow and Alpha-Flow is only cited but not compared, despite the overlap in the tasks.
- Although quite a lot of tasks are benchmarked, scalability remains one concern as the paper did not extend to larger vocabularies such as natural language generation. This is, though, more of a limitation should other concerns be addressed.
[1] Hu, K., Qiu, L., Lu, Y., Zhao, H., Li, T., Kim, Y., ... & He, K. (2026). Elf: Embedded language flows. arXiv preprint arXiv:2605.10938.
Questions for authors
Please refer to the weaknesses above.
Thank you for the clarification. However, regarding the claim of “removing redundant degrees of freedom,” there is still a lack of evidence demonstrating that the removed DoF are indeed redundant. How do you ensure that the reduced representation is better than the original one?