- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces RiCoTA as a framework for generative modelling of data living on Riemannian manifolds. The idea is that a uniform discretization of latent space may be very geometrically inefficient -- equal changes in latent coordinates may correspond to very different changes in the data space. The method therefore uses pullback geometry to define canonical coordinates and uses decoder sensitivity to allocate finer cells in sensitive regions. It autoregressively models the joint distribution over these discrete cells. During generation, RiCoTA samples a sequence of cell indices, samples values within selected cells, and makes a single decoder pass.
Critical strengths
Strength 1. It is a conceptually appealing and well-motivated idea. Since there could be a mismatch between a uniform discretization of latent space and the non-uniform geodesics induced by the decoder, allocating resolution according to the decoder's sensitivty makes sense.
Strength 2. The paper tests the proposed mechanism with a good effort. For example, the full versus identity comparisons are better than simply comparing against unrelated generative models (as they keep the architecture, prior, optimization, and code budget fixed while changing the cell allocation).
Critical weaknesses
Weakness 1. The practical utility and scalability of the design -- and hence the significance -- are not very convincing. The DNA study partly addresses this, but it has substantial modifications to tackle the higher dimensionality. I therefore find it unclear whether the basic formulation may scale to problems beyond the low-dimensional manifolds considered here. While some scaling aspects are discussed in the text, the paper would benefit from empirical results showing efficiency and quality change with latent dimensions.
Weakness 2. Some parts are overly dense and difficult to follow. Several sections appear fully AI-written; I recommend the authors to rewrite or remove overly dense parts and ensure everything is very clear. Generally, the paper needs a better, easier and more clean presentation.
Questions for authors
Question 1. The main experiments use a very small latent dimensionality. How does efficiency and quality compare with other generators when becomes significantly larger? When might autoregressive generation become the dominant cost?
Question 2. The full-vs-identity results support the authors' mechanism, but the improvements of Table 5 are very modest compared to the larger gains over baselines. How much of the overall improvement in the quality--speed tradeoff comes from geometry-aware companding rather than the broader encoder–decoder architecture and AR prior? It would strengthen the paper to more clearly demonstrate whether the benefits of their main novelty justify its additional complexity.
4 comments
Response to Reviewer ZeZe (2/2)
Q2 — How much does geometry-aware companding contribute to the quality–speed tradeoff?
Replacing Identity’s uniform probability cells with RiCoTA’s geometry-aware cells improved all three mean quality metrics in every manifold family. Across families, the largest absolute difference in mean generation time between Identity and RiCoTA was 0.06 ms per 100k samples (Table C).
Manifold Free-Form Flows (M-FFF) was the fastest competing baseline on average across the 13 shared benchmarks in our main experiments. It uses an autoencoder (AE) and decodes a continuous prior sample in one pass. To test a conventional quantizer, we trained a model with finite scalar quantization (FSQ) [1] and matched its decoder backbone and autoregressive (AR) prior architecture to RiCoTA’s.
Table C. Generation quality and time per 100k samples. Entries give local ↑ / HD95 ↓ / MMD ↓ · time (ms). For each dataset, we used mean quality scores across five training seeds and median generation time. We then weighted datasets equally within each family. The last row gives the relative increase and HD95 and MMD reductions from Identity to RiCoTA.
| Model or comparison | Sphere | SO(3) | Product tori | Hyperbolic |
|---|---|---|---|---|
| M-FFF | 0.6277 / 11.939 / 0.0719 · 18.98 ms | 0.1941 / 59.302 / 0.2171 · 84.78 ms | 0.5263 / 43.290 / 0.1067 · 22.65 ms | — |
| FSQ | 0.5940 / 9.020 / 0.0628 · 8.84 ms | 0.4132 / 4.976 / 0.0216 · 30.05 ms | 0.4390 / 18.402 / 0.0299 · 5.85 ms | 0.5033 / 0.135 / 0.0262 · 7.25 ms |
| Identity (RiCoTA with uniform cells) | 0.6747 / 9.495 / 0.0516 · 9.05 ms | 0.4654 / 5.180 / 0.0198 · 30.22 ms | 0.5399 / 19.379 / 0.0313 · 6.01 ms | 0.6503 / 0.140 / 0.0294 · 7.27 ms |
| RiCoTA | 0.6999 / 7.678 / 0.0445 · 9.05 ms | 0.4780 / 4.893 / 0.0180 · 30.28 ms | 0.5501 / 17.236 / 0.0238 · 5.99 ms | 0.6593 / 0.132 / 0.0228 · 7.31 ms |
| RiCoTA vs. Identity (relative improvement) | 3.7% / 19.1% / 13.7% | 2.7% / 5.5% / 9.2% | 1.9% / 11.1% / 24.0% | 1.4% / 5.9% / 22.2% |
M-FFF vs. FSQ. Even with conventional quantization, FSQ generated samples faster than M-FFF on the sphere, SO(3) and product tori. Its mean local exceeded M-FFF’s only on SO(3).
FSQ vs. Identity. The two models use different quantization schemes but share the same AE–AR design and decode once per sample after drawing the tokens autoregressively. At nearly identical generation times, Identity had higher mean local in all four manifold families, while FSQ had lower mean HD95 in all four.
Identity vs. RiCoTA. Across the four manifold families, replacing Identity’s uniform cells with RiCoTA’s geometry-aware cells increased mean local by 1.4–3.7% and reduced mean HD95 by 5.5–19.1% and mean MMD by 9.2–24.0%. RiCoTA estimates decoder sensitivity during Phase 1 to set the boundaries. At generation time, both models use their stored boundaries, take the same number of autoregressive steps, and decode once. Their measured generation times were nearly identical (Table C).
References
[1] Mentzer, F., et al. Finite Scalar Quantization: VQ-VAE Made Simple. ICLR, 2024.
W2 — Clarifying the presentation of the methods and theoretical analysis
We revised the presentation of our method and theoretical analysis. In Section 2, we expanded the overview of the two training phases. The revised Figure 1 caption describes how output geometry guides the choice of latent axes and cell boundaries and how RiCoTA uses the learned prior to generate samples. Section 3.2 now opens by explaining how decoder sensitivity is measured and how it guides cell placement. Section 3.4 explains what the autoencoder and the prior each learn before presenting their training objectives.
Thanks for your effort. Looking at the pdf, I believe the manuscript is going toward a better state now, but I still believe it falls short of typical ICLR-quality, so I still lean weak reject.
I do think the overview figure could be substantially improved, I see this was also mentioned by another reviewer, and I agree. I also think the appendices have become quite large, and the readability of them could be substantially improved. Making a much more focused new version may be in your best interest.
Best of luck with the paper, I highlight that it's an interesting idea.
Thank you for your feedback
Thank you for taking the time to read the revised manuscript and share your feedback. In the final version, we will improve the overview figure and shorten the appendices to make them easier to read.
Response to Reviewer ZeZe (1/2)
W1 · Q1 — Quality and speed versus other generators at larger latent dimensions, and the AR bottleneck
Thank you for taking the time to review our paper.
Increasing RiCoTA’s latent dimension m0 slowed generation without a consistent gain in quality (Table A). On each dataset, we evaluated RiCoTA at the latent dimension used in the main experiments (1×) and at 2×, 4× and 8× that dimension, keeping the data manifold and the number of cells per coordinate fixed. The nominal bit budget therefore grew with m0, and each additional coordinate required one more sequential token draw before decoding. Some quality metrics improved at 2× or 4×. At 8×, however, generating 100k samples took 3.19–6.11 times as long as at 1×, and all five datasets showed lower mean local F1(0.5) and higher mean HD95 and MMD.
RiCoTA first took longer to generate 100k samples than M-FFF at 4× the base dimension on Fire and Protein Glycine and at 8× on RNA and SO(3)-Line. At 8×, Fire and RNA were the only two of these four datasets on which RiCoTA still led M-FFF on all three quality metrics. On the hyperbolic dataset, 8× RiCoTA remained faster than G-LSD, with similar local F1(0.5) and lower HD95 and MMD.
Table A. Generation quality and time per 100k samples as RiCoTA’s latent dimension increases. For each dataset, we tested RiCoTA at 1×, 2×, 4× and 8× the latent dimension used in the main experiments. Quality values are five-seed means and times are medians.
When autoregressive sampling dominates. At the base latent dimension (1×), autoregressive (AR) token sampling took more than half the end-to-end generation time on Fire, Protein Glycine, RNA, and SO(3)-Line. On the hyperbolic dataset, this ratio first exceeded 50% at 4× the base dimension. At 8×, the ratio ranged from 78.5% to 94.4% across the five datasets (Table B). We added these results to Appendix H.6 of the revised paper (Table 32).
Table B. AR token sampling time per 100k samples. For each dataset, 1× denotes the latent dimension used in the main experiments. Entries give median AR time (ms) and, in parentheses, its percentage of the median end-to-end generation time. Bold marks the first tested dimension at which this ratio exceeded 50%.