acceptodds
Under review as a conference paper at ICLR 2027

Generative Modeling of Discrete Data Using Geometric Latent Subspaces

Abstract

We propose a geometric latent-subspace framework for generative modeling of discrete data. Specifically, we introduce latent subspaces in the exponential parameter space of product manifolds of categorical distributions as a novel approach to learning low-dimensional representations of high-dimensional discrete data. The resulting low-dimensional latent space captures statistical dependencies and removes redundant degrees of freedom among the categorical variables. We equip the parameter domain with a Riemannian geometry such that the latent subspace and induced data manifold are related isometrically, enabling consistent flow matching. Exploiting this structure, we propose a geometry-aware dimensionality reduction objective, called geometric PCA (GPCA), which we formulate as a regularized cross-entropy minimization that encourages small Riemannian distances between the data and their reconstructions. In particular, under the induced geometry, geodesics correspond to straight lines in the latent parameter space, allowing flow matching to be performed directly in reduced coordinates. Empirical results show that low-dimensional latent representations suffice to accurately model high-dimensional discrete data and enable substantially more computationally efficient flow matching.

Primary area:
generative models
Keywords:
discrete data, generative models, flow matching, geometric PCA, latent space models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WwnYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes geometric PCA (GPCA) that respects the underlying information geometry for discrete data and further uses it to derive a Riemannian geometry for discrete flow matching. Instead of directly modeling on the simplex, the authors run the generative framework on the latent space, which may provide more meaningful semantics. On binarized MNIST, the paper showed improved FID; other tasks also showed better performance.

Critical strengths
  • The paper is mathematically solid to me. Its proposed use of exponential geometry respects the underlying information geometry for discrete data, potentially benefiting the generative process.
  • For the proposed geometric PCA, it improves on standard PCA in terms of final generative performance.
  • The paper benchmarks on a wide variety of generative tasks across different domains, and improvements can be shown across tasks.
  • Code is provided for reproducibility.
Critical weaknesses
  • I find it very hard to identify the true contribution of this paper, even though I have experience with discrete flow matching models. The authors seem to claim their contribution as proposing a "geometric PCA" and using the exponential geometry for discrete flow matching. However, after reading the whole paper, these two contributions are already well-explored in the existing literature (see below), and the more credible contribution to me is the combination of the latent approach with the e-geometry, which, to the best of my knowledge, is indeed new.
  • The geometric PCA does not provide solid theoretical or practical motivations and justifications.
    • Generalized PCA already supplies the categorical low-rank representation and nonlinear decoder, as the paper acknowledges. As the simplex is embeddable in the ambient space (either directly as the simplex or as the sphere to preserve the metric), there is no motivation to support geometric PCA.
    • Further, more advanced dimension reduction techniques like t-SNE or UMAP theoretically preserve local geometry, a much stronger theoretical guarantee that can be transferred to smooth Riemannian manifolds, as they are locally Euclidean anyway.
    • Recent "continuous discrete language models" like [1] predominantly use low-dim embeddings to achieve very good performance. Compared to fixed PCA vectors, I believe learnable embeddings provide a better option than fixed PCA vectors.
    • Additionally, the discrete data are one-hot vectors instead of continuous vectors. It remains unclear whether PCA on such data can lead to meaningful results.
    • Last, only ordinary PCA is compared. No ablation study on other popular dimension reduction techniques is provided. PCA dimension is also chosen empirically.
  • The exponential geometry is already studied in previous work, and I see no new methodological novelty in this part. As the paper already cited, the most relevant work of Assignment Flow and Alpha-Flow already used such geometry for generative modeling. Specifically, both benchmark the binarized MNIST task, and Assignment Flow additionally has the segmentation results. However, they are never compared.
  • Unlike the standard Fisher-Rao geometry, the e-geometry is undefined at the boundary. The authors suggest using a small cap on the one-hot target. However, this may impact both the theoretical results in Prop. 3.5, as well as empirical performance. Although this value is chosen differently for different tasks, the paper does not have an ablation on that.
  • The binarized MNIST FID is not comparable to baselines due to unmatched sampling numbers. The cited SFM uses 1k generations, whereas this paper uses 10k, which may account for the drastic drop in FID.
  • As mentioned above, highly related work like Assignment Flow and Alpha-Flow is only cited but not compared, despite the overlap in the tasks.
  • Although quite a lot of tasks are benchmarked, scalability remains one concern as the paper did not extend to larger vocabularies such as natural language generation. This is, though, more of a limitation should other concerns be addressed.

[1] Hu, K., Qiu, L., Lu, Y., Zhao, H., Li, T., Kim, Y., ... & He, K. (2026). Elf: Embedded language flows. arXiv preprint arXiv:2605.10938.

Questions for authors

Please refer to the weaknesses above.

Reviewer_V56gOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a latent-subspace framework for generative modeling of discrete data that combines a regularized variant of generalized PCA—termed Geometric PCA (GPCA)—with Conditional Flow Matching (CFM). Specifically, the authors equip the product manifold of categorical distributions with a pullback Euclidean metric (ee-metric) under the additive log-ratio coordinate chart, which makes the linear latent subspace in natural parameters isometric to the induced data manifold. Because geodesics under this metric correspond to straight lines in natural coordinates, flow matching can be performed directly in the reduced latent coordinate space. The approach is evaluated on synthetic distributions, binarized MNIST and Fashion-MNIST, downsampled Cityscapes, and biological DNA sequence datasets.

Critical strengths
  1. Interesting Problem Formulation: Reducing the dimensionality and computational overhead of continuous-state flow matching for high-dimensional discrete data via geometric subspace projection is an interesting and well-motivated research direction.

  2. Tractable Pullback Geometry for Flow Matching: Equipping the simplex interior with the pullback Euclidean metric (ee-metric) yields straight-line geodesic interpolants in natural coordinates (Proposition 4.1) and allows the authors to derive a clean upper bound relating latent reconstruction error to the ambient flow matching objective (Proposition 3.4 and Corollary 4.3).

  3. Computational Speedups on Compressible Image Datasets: On highly compressible binary image datasets such as binarized MNIST (d=32d=32 or 6464), operating in the reduced coordinate space Z\mathcal{Z} allows a compact MLP to train significantly faster than an ambient-space U-Net while achieving competitive FID scores.

Critical weaknesses
  1. Unsupported Claims Regarding Statistical Dependencies and Redundant Degrees of Freedom: The Abstract (Lines 015–017) and Section 5 (Lines 354–356) claim that the low-dimensional GPCA latent space "captures statistical dependencies and removes redundant degrees of freedom among the categorical variables" and resolves the "independence assumption between classes" of prior product-manifold methods. This claim is not supported by either theoretical or empirical results:
  • Theoretically: Every point on the induced manifold M=φ−1(U)⊂Scn\mathcal{M} = \varphi^{-1}(\mathcal{U}) \subset \mathcal{S}_c^n still parameterizes a factorized product distribution p1⊗⋯⊗pnp_1 \otimes \cdots \otimes p_n across the nn categorical variables (as explicitly shown in Appendix C, Eqs. 50–51). Non-factorizing joint distributions are only approximated via the pushforward mixture distribution of the flow over M\mathcal{M}, which is already true for standard flow matching on Scn\mathcal{S}_c^n without GPCA (Lines 947–955).

  • Empirically: Beyond low-rank reconstruction error (MHD) and 2-variable toy histograms, there are no quantitative evaluations (e.g., mutual information, higher-order marginals, or conditional dependency metrics) demonstrating that GPCA captures statistical dependencies or removes redundant degrees of freedom better than existing methods.

  1. Unclear Superiority Over Generalized PCA (Collins et al., 2001): The proposed GPCA objective (Eq. 13) simply takes the classic Generalized PCA objective of Collins et al. (2001) (Eq. 6, KL/cross-entropy minimization in natural parameters) and adds an ℓ2\ell_2 penalty λde2\lambda d_e^2 on ϵ\epsilon-smoothed one-hot data. The superiority of this modification over Generalized PCA is not convincingly established:
  • In Section 6.1.1 and Appendix E (Table 4), the authors only compare GPCA against standard linear PCA on one-hot vectors, omitting a direct comparison against Generalized PCA.

  • In the ablation tables comparing λ>0\lambda > 0 (GPCA) against λ=0\lambda = 0 (Generalized PCA / Cross-Entropy / NLL minimization, Tables 1, 3, 17, and 18), unregularized Generalized PCA (λ=0\lambda = 0) consistently achieves better (lower) Cross-Entropy and Mean Hamming Distance (MHD) than GPCA. The ee-distance de2d_e^2 only grows large for λ=0\lambda = 0 because discrete one-hot data lie on the simplex boundary where exact natural parameters diverge to ±∞\pm\infty; adding λde2\lambda d_e^2 merely anchors the logits to the arbitrary smoothing hyperparameter ϵ=10−2\epsilon = 10^{-2} (Eq. 63) at the expense of worse discrete reconstruction accuracy.

  1. Failure to Utilize Latent Space Z\mathcal{Z} and Poor Performance on Standard Discrete Benchmarks: While the primary motivation of the paper is performing flow matching in the reduced coordinate space Z=Rd\mathcal{Z} = \mathbb{R}^d (Eq. 19, Remark 4.2), the approach struggles on standard non-image discrete benchmarks:
  • On the Promoter and Enhancer (Melanoma, FlyBrain) DNA benchmarks (Appendix J), the data are poorly compressible by GPCA, suffering from high base-pair reconstruction error (MHD of 45.8% at d=256d=256 and 29.0% at d=512d=512 on Promoter, Table 21).

  • Crucially, on all three DNA datasets, the authors do not perform flow matching in the latent coordinate space Z\mathcal{Z}, instead falling back to training in the ambient natural-parameter subspace U\mathcal{U} and explicitly conceding that these experiments "cannot directly take advantage of operating in Z\mathcal{Z}" or realize its computational savings (Lines 2079–2081, 2149–2151).

  • Even at large dd (d=1024d=1024 and d=500d=500), GPCA + FM underperforms DirichletFM, SFM, and LinearFM on Promoter SP-MSE (Table 24) and performs substantially worse than DirichletFM on Melanoma and FlyBrain FBD (29.6 vs. 7.3 and 17.3 vs. 6.8, Table 25). Similarly, on the Toy Data KL benchmark (Figure 4), GPCA + FM (which reports the minimum KL over 5 seeds rather than the mean) trails Fisher FM (OT) across most category sizes.

  1. Elementary Pullback Geometry, Category Asymmetry, and Non-Amortized Encoding: Defining a Riemannian metric as the pullback of the Euclidean metric under a coordinate chart φ\varphi (Definition 3.1) tautologically makes φ\varphi an isometry to Euclidean space; thus, Propositions 3.2–3.4 and 4.1 are immediate consequences of standard linear interpolation in logit space. Furthermore, the additive log-ratio transformation φ(p)=(log⁡(pi/pc))i∈[c−1]\varphi(p) = (\log(p_i/p_c))_{i \in [c-1]} (Eq. 1) is asymmetric with respect to the reference category cc, making the ee-metric dependent on category permutation. Finally, because GPCA optimizes (V,Z)(V, Z) transductively via full-batch Adam over NN training samples (taking up to 47 minutes on Promoter before generative training begins, Table 5), it lacks an amortized encoder for unseen data (Eq. 7).
Questions for authors
  1. Statistical Dependencies: Can the authors provide rigorous theoretical justification or quantitative empirical evidence showing how the linear subspace U⊂Rn×(c−1)\mathcal{U} \subset \mathbb{R}^{n \times (c-1)} captures cross-node statistical dependencies and removes redundant degrees of freedom, given that every point in M⊂Scn\mathcal{M} \subset \mathcal{S}_c^n remains a factorized product distribution?

  2. Superiority Over Generalized PCA: Since λ=0\lambda = 0 (Generalized PCA of Collins et al., 2001) achieves superior Cross-Entropy and Hamming reconstruction error in Tables 1, 17, and 18, what is the fundamental advantage of GPCA over Generalized PCA coupled with standard logit clipping or temperature scaling? How sensitive is GPCA to the smoothing hyperparameter ϵ\epsilon in Eq. (63)?

  3. Applicability of Latent-Space Flow Matching (Eq. 19) to DNA Benchmarks: Why was flow matching performed in U\mathcal{U} rather than the reduced latent space Z\mathcal{Z} for the unconditional Melanoma and FlyBrain DNA experiments (Section J.0.2)? What is the generation performance (FBD and SP-MSE) when actually training the flow matching model in Z\mathcal{Z} using Eq. (19)?

  4. Permutation Sensitivity of the ee-Metric: Because the additive log-ratio chart (Eq. 1) divides by the cc-th category probability pθ(c)p_\theta(c), the induced ee-distance ded_e is not invariant to permutations of the cc categories. How does the choice of reference category cc affect GPCA reconstruction and flow matching trajectories when c>2c > 2?

2 comments

Thank you for the clarification. However, regarding the claim of “removing redundant degrees of freedom,” there is still a lack of evidence demonstrating that the removed DoF are indeed redundant. How do you ensure that the reduced representation is better than the original one?

We agree that dimensionality reduction alone does not establish that the discarded degrees of freedom are statistically redundant. Our approach follows the motivation of [1], which seeks a low-dimensional subspace of natural parameters that explains the observations by maximizing their likelihood under an exponential family model. In this sense, redundancy refers to the possibility of explaining the observed configurations well using fewer coordinates. Our reconstruction results across different datasets and latent dimensions provide empirical evidence that the observed data can be represented using fewer coordinates, although the achievable reconstruction accuracy depends on the dataset and latent dimension. Figure 7 in Appendix C illustrates this idea through a one-dimensional manifold learned from samples of a non factorizing joint distribution.

For flow matching, Proposition 3.4 shows that small geometric reconstruction errors at the endpoints keep the corresponding geodesic interpolants close, while Corollary 4.3 provides a corresponding bound for the flow matching objective. Once the subspace is learned, the objective on U\mathcal U can be written equivalently in the reduced coordinates Z\mathcal Z using the norm induced by VV, as described in Remark 4.2. Our aim is to exploit low-dimensional structure when it is present. Training the vector field in fewer coordinates offers a computational advantage, as illustrated by the runtime comparisons in Appendix F. Moreover, as discussed in Section 6.3, the reduced formulation allows us to reuse the same flow matching architecture and training configuration across datasets, simplifying implementation.

[1] M. Collins et al. A Generalization of Principal Components Analysis to the Exponential Family, 2001.

Reviewer_AyFCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method for generative modeling of discrete data. In particular, authors run flow matching on a low-dimensional manifold found by running generalized PCA in the logit parameterization of the discrete distribution space. To make flow matching more efficient in this setting, authors propose a geometric regularizer with a controllable penalty. Authors evaluate the method on a set of synthetic data and image data modeling problems, comparing to a set of discrete distribution modeling baselines.

Critical strengths

The idea to run flow matching in the latent space of generalized PCA is novel (to the best of this reviewer's knowledge). Non-trivial technical work required to make flow matching efficient in this setting.

Critical weaknesses

The paper is rather dense, which makes it hard to read. Sections 2-4 intersperse the high-level narrative with background knowledge and deep theoretical analysis. I would encourage the authors to streamline the high-level narrative in these sections, and move more technical details to the appendix.

The experimental results could better demonstrate an improvement upon baselines. While the Table 2 shows an improved FID on binarized MNIST, Figure 4 suggests results on other problems might be more mixed. I would encourage the authors to move more results from the appendix to the main text, and discuss the method's performance compared to the baselines in more detail. In its current form, I believe the paper doesn't convince the reader that the method brings a significant improvement on a wide set of important problems.

Questions for authors

In section 6.2, which image model do authors use to compute FID? If one trained on natural images (like the Inception model), is there a reason to believe the FID is faithful to binary image quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.