- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles a meaningful problem and proposes a variational multi-chart framework for learning structured latent representations without supervision. The proposed latent-evidence routing and portal-based cross-chart interpolation are interesting, and the reported experiments show improvements in generative quality over VAE and VaDE baselines, along with qualitatively promising interpolation results.
However, I believe the current experiments more convincingly establish improved generative quality from a multi-chart representation than the effectiveness of the proposed chart-selection and portal-based interpolation mechanisms.
The most important missing evidence is whether latent-evidence routing learns better manifold representations than existing routing strategies, independently of the balancing regularization, and whether the proposed portals offer measurable advantages over prior cross-chart interpolation methods.
Critical strengths
Learning multiple local latent representations without supervision is a meaningful problem. The proposed latent-evidence routing is an interesting alternative to reconstruction-based chart assignment, and the portal-based interpolation provides a promising way to connect independently learned charts. The improvements in FID and qualitative interpolation results are encouraging, although more targeted evaluation is needed to establish the benefits of these two contributions.
Critical weaknesses
Regarding the first contribution, chart-selection routing, the current experiments primarily demonstrate improvements in sample FID, but provide limited evidence that the proposed routing strategy learns better manifold representations or more effectively avoids chart collapse than existing alternatives. Could the authors consider the following additional comparisons to better isolate and validate this contribution?
| Category | Suggested comparison |
|---|---|
| Additional metrics | Report unfaithfulness and coverage, following CAE (Schonsheck et al., 2019), to evaluate whether the proposed routing strategy improves manifold fidelity and coverage beyond sample quality as measured by FID. |
| Baseline | The authors mention that CAE could not be successfully trained due to chart collapse. Could they clarify whether this occurred when reproducing CAE's original experimental setup or when adapting it to the current architecture and training protocol? If a direct comparison is infeasible, could VAA at least be evaluated under CAE's original experimental protocol, enabling a comparison with the results reported in Table 1 of CAE? |
| Ablation | The effect of latent-evidence routing appears potentially confounded with the additional chart-balancing regularization (). Although Table 5 compares ELBO routing with latent-evidence routing, it remains whether balancing regularization (Appendix D2) was ablated here. Could the authors clarify this, so that readers can disentangle contributions of ELBO vs. latent-evidence routing, with and without the same balancing regularizer? |
| Diagnostics | To substantiate the claim that latent-evidence routing prevents chart collapse, could the authors report per-chart utilization and assignment entropy for both routing strategies? These statistics would help distinguish successful chart specialization from two undesirable outcomes: collapse onto a few charts and nearly uniform routing across all charts for every input. |
Together, these experiments would help establish not only whether latent-evidence routing improves generative performance, but also whether it produces meaningfully better chart assignments and manifold representations, independently of the balancing regularizer.
Regarding the second contribution, portal-based interpolation, the paper compares its portal-based interpolation qualitatively against straight-line interpolation in a vanilla VAE, but does not compare against existing cross-chart interpolation methods, such as Atlas Generative Models (Stolberg-Larsen & Sommer, 2022).
The latter evaluates graph-based geodesic interpolation on MNIST and Fashion-MNIST, reporting distributions of interpolation path lengths and qualitative interpolation results (Figure 3). Could the authors provide a comparable evaluation of the proposed portal-based interpolation?
More importantly, would it be possible to compare the proposed portal-based transitions with the point-based graph interpolation strategy from Atlas Generative Models using the same trained VAA charts and endpoint pairs? This would help isolate the contribution of the proposed portal mechanism from differences in the underlying learned representations.
In addition to path length, it would be helpful to report transition continuity and, for the synthetic double-torus experiment where ground-truth geometry is available, distance to the true manifold. Together, these metrics would help establish whether portal-based interpolation offers measurable advantages over existing cross-chart interpolation approaches.
Questions for authors
Questions listed in the above section
3 comments
Adherence to a known manifold
For the double-torus dataset the ground-truth manifold is known, so we can measure exactly how far interpolated points lie from it. We sampled 240 random point pairs on the surface and interpolated between them with a single-chart autoencoder (a standard VAE) and with our multi-chart model (4 charts, as in Figure 1). For each path, we record the largest distance of any interpolated point to the true surface and what percentage of paths have a maximum distance above a certain threshold:
Maximum distance to the surface per path (mesh units, 240 paths):
| Method | p5 | Median | p95 | Max |
|---|---|---|---|---|
| Portal (ours) | 0.0026 | 0.0031 | 0.0037 | 0.0044 |
| Single chart | 0.0027 | 0.0034 | 0.0427 | 0.1022 |
Paths leaving the surface (max distance > threshold):
| Method | > 0.005 | > 0.01 | > 0.02 | > 0.05 |
|---|---|---|---|---|
| Portal (ours) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
| Single chart | 53 (22%) | 35 (15%) | 28 (12%) | 9 (4%) |
The two methods have nearly identical typical paths (similar median), but they differ sharply in the tail. On a threshold that sits above our maximum distance (0.0044), the single chart baseline leaves the surface in 22% of the cases. This is the failure mode a single chart cannot avoid on a surface of genus 2: no single global parameterization covers it, so some straight-line interpolations in latent space must cut through regions that are off the manifold. Portals avoid this by switching charts before the path leaves a chart's valid region. We will add this information to the supplemental material.
This experiment also speaks to the faithfulness and coverage of the model. The distance to the true surface is a ground-truth version of unfaithfulness: it measures directly how far decoded points stray from the manifold, rather than estimating it through nearest neighbours in the training set. Interpolated points are the hardest case, since they come from latent regions between the encodings of real data. On coverage, the four charts split the surface almost exactly evenly (see also Figure 1), each representing about a quarter of the data. The model therefore represents the whole surface rather than collapsing onto part of it. And because the endpoints span every pair of charts, the paths cross the whole surface. That our model stays on the manifold along all of them means faithfulness holds across the full extent of the surface.
Dear reviewer kqid:
As the discussion period ends tomorrow, we hope our rebuttal has addressed your concerns. We would appreciate it if you could reassess our work in light of our responses. If any concerns remain, we would be happy to address them.
Thank you for your time and consideration.
We thank the reviewer for the detailed and constructive suggestions. Following them, we ran the routing x balance ablation, a direct comparison with the point-based interpolation of Stolberg-Larsen & Sommer on the same trained charts and endpoint pairs, and a distance-to-manifold evaluation on the double torus.
Comparison to existing atlas transition methods
We have added a direct comparison with the graph-based cross-chart interpolation of Atlas Generative Models (Stolberg-Larsen & Sommer, 2022), the method discussed in Section 2. Both methods operate on the same trained atlas, so only the transition mechanism differs. For 100 random test pairs, we compare our portals against their kNN graph (k=20, 2000-point pool) at three values of their chart-membership threshold ε, a value that needs to be tuned. We report LPIPS at the chart switch (pointwise transition error), the largest LPIPS step along the path (smoothness), and summed LPIPS along the path (path length).
Mean ± std over pairs connected by all methods
The graph-based method trades transition quality against path quality through ε: small ε connects all pairs but gives worse transitions, while larger ε gives sharper transitions at the cost of longer paths and, on MNIST, 22% of pairs left disconnected. Portals avoid this trade-off. Their transition error is comparable to the best ε setting, while their paths are shorter than every setting, and every pair is connected. We will add this comparison to the paper.
Impact of a balance term on collapse
We agree that the original submission did not establish why related work such as MLMM-VAE collapsed. These methods collapsed when we ran the authors' code on our image datasets. To isolate the routing mechanism, the submitted paper already integrates MLMM-VAE routing within our framework (Table 5, ELBO routing). We will make this more explicit in the revision.
To further separate the routing rule from the balance regularizer, we ran a 2x2 grid crossing the MLMM-VAE router and ours with and without the γ balance term (Eq. 19), using the same backbone, iteration count and per-dataset hyperparameters as in the paper
Mean ± std over 3 seeds. A chart is dead if it receives <1% of the data. For CelebA, the ablation uses the configuration of Table 5 (DCGAN backbone, C = 10, iteration counts matched within each dataset), which differs from the Table 1 configuration
Collapse is driven by the balance term, not the router. Without γ, both routers collapse (on Fashion-MNIST, ELBO routing even leaves fewer dead charts); with γ, neither does. We therefore attribute MLMM-VAE's collapse mainly to the absence of a usage regularizer, and we will revise the manuscript so that it does not suggest our routing alone prevents collapse.
Given γ, the router determines sample quality. On CelebA, γ balances usage under ELBO routing but barely improves FID (48.3 → 47.6), while our routing reaches 42.9. On Fashion-MNIST ours is also better (37.6 vs 39.7); on MNIST the two are on par. We will add this table and discussion to the appendix.