acceptodds
Under review as a conference paper at ICLR 2027

Variational Atlas Autoencoders and Self-Supervised Chart Selection

Abstract

Complex, high-dimensional data lie on manifolds that a single global latent coordinate system cannot represent without distortion. We introduce Variational Atlas Autoencoders, a mixture-of-charts framework that learns an atlas of local latent representations, each responsible for one region of the data manifold. The central difficulty in learning an atlas is deciding which chart should be responsible for which data point without labels. We show that this decision need not be a heuristic. By treating the chart index as a latent variable, the evidence lower bound yields the training objective and casts chart assignment as posterior inference, valid for any routing distribution. Within this framework, we propose latent-evidence routing, which assigns a point to a chart in proportion to the evidence of its encoding under that chart’s latent prior. This has a simple interpretation: the closer a point lands to the center of a chart’s latent space, and the more confidently it is placed there, the more likely it is to belong to that chart. Chart assignment thus emerges as posterior inference, depends on the encoder alone, and is fully self-supervised, so the charts discover structure in the data on their own, separating classes and subclasses such as distinct types of shoes or faces without access to labels. We further recover an atlas from the trained charts by identifying their overlaps and transition maps, enabling continuous interpolation within and across charts that stays close to the data manifold rather than traversing unsupported regions of latent space. Compared to capacity-matched baselines, Variational Atlas Autoencoders yields better-structured latent spaces and higher sample quality on image datasets including CelebA, while sampling requires evaluating only a single chart’s decoder.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
manifold learning, generative modelling, representation learning, variational autoencoders, mixture of experts

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_kqidOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles a meaningful problem and proposes a variational multi-chart framework for learning structured latent representations without supervision. The proposed latent-evidence routing and portal-based cross-chart interpolation are interesting, and the reported experiments show improvements in generative quality over VAE and VaDE baselines, along with qualitatively promising interpolation results.

However, I believe the current experiments more convincingly establish improved generative quality from a multi-chart representation than the effectiveness of the proposed chart-selection and portal-based interpolation mechanisms.

The most important missing evidence is whether latent-evidence routing learns better manifold representations than existing routing strategies, independently of the balancing regularization, and whether the proposed portals offer measurable advantages over prior cross-chart interpolation methods.

Critical strengths

Learning multiple local latent representations without supervision is a meaningful problem. The proposed latent-evidence routing is an interesting alternative to reconstruction-based chart assignment, and the portal-based interpolation provides a promising way to connect independently learned charts. The improvements in FID and qualitative interpolation results are encouraging, although more targeted evaluation is needed to establish the benefits of these two contributions.

Critical weaknesses

Regarding the first contribution, chart-selection routing, the current experiments primarily demonstrate improvements in sample FID, but provide limited evidence that the proposed routing strategy learns better manifold representations or more effectively avoids chart collapse than existing alternatives. Could the authors consider the following additional comparisons to better isolate and validate this contribution?

CategorySuggested comparison
Additional metricsReport unfaithfulness and coverage, following CAE (Schonsheck et al., 2019), to evaluate whether the proposed routing strategy improves manifold fidelity and coverage beyond sample quality as measured by FID.
BaselineThe authors mention that CAE could not be successfully trained due to chart collapse. Could they clarify whether this occurred when reproducing CAE's original experimental setup or when adapting it to the current architecture and training protocol? If a direct comparison is infeasible, could VAA at least be evaluated under CAE's original experimental protocol, enabling a comparison with the results reported in Table 1 of CAE?
AblationThe effect of latent-evidence routing appears potentially confounded with the additional chart-balancing regularization (γ=50\gamma=50). Although Table 5 compares ELBO routing with latent-evidence routing, it remains whether balancing regularization (Appendix D2) was ablated here. Could the authors clarify this, so that readers can disentangle contributions of ELBO vs. latent-evidence routing, with and without the same balancing regularizer?
DiagnosticsTo substantiate the claim that latent-evidence routing prevents chart collapse, could the authors report per-chart utilization and assignment entropy for both routing strategies? These statistics would help distinguish successful chart specialization from two undesirable outcomes: collapse onto a few charts and nearly uniform routing across all charts for every input.

Together, these experiments would help establish not only whether latent-evidence routing improves generative performance, but also whether it produces meaningfully better chart assignments and manifold representations, independently of the balancing regularizer.

Regarding the second contribution, portal-based interpolation, the paper compares its portal-based interpolation qualitatively against straight-line interpolation in a vanilla VAE, but does not compare against existing cross-chart interpolation methods, such as Atlas Generative Models (Stolberg-Larsen & Sommer, 2022).

The latter evaluates graph-based geodesic interpolation on MNIST and Fashion-MNIST, reporting distributions of interpolation path lengths and qualitative interpolation results (Figure 3). Could the authors provide a comparable evaluation of the proposed portal-based interpolation?

More importantly, would it be possible to compare the proposed portal-based transitions with the point-based graph interpolation strategy from Atlas Generative Models using the same trained VAA charts and endpoint pairs? This would help isolate the contribution of the proposed portal mechanism from differences in the underlying learned representations.

In addition to path length, it would be helpful to report transition continuity and, for the synthetic double-torus experiment where ground-truth geometry is available, distance to the true manifold. Together, these metrics would help establish whether portal-based interpolation offers measurable advantages over existing cross-chart interpolation approaches.

Questions for authors

Questions listed in the above section

3 comments

We thank the reviewer for the detailed and constructive suggestions. Following them, we ran the routing x balance ablation, a direct comparison with the point-based interpolation of Stolberg-Larsen & Sommer on the same trained charts and endpoint pairs, and a distance-to-manifold evaluation on the double torus.

Comparison to existing atlas transition methods

We have added a direct comparison with the graph-based cross-chart interpolation of Atlas Generative Models (Stolberg-Larsen & Sommer, 2022), the method discussed in Section 2. Both methods operate on the same trained atlas, so only the transition mechanism differs. For 100 random test pairs, we compare our portals against their kNN graph (k=20, 2000-point pool) at three values of their chart-membership threshold ε, a value that needs to be tuned. We report LPIPS at the chart switch (pointwise transition error), the largest LPIPS step along the path (smoothness), and summed LPIPS along the path (path length).

DatasetMethodTransition LPIPS ↓Smoothness ↓Path length ↓Connected
MNISTOurs0.040 ± 0.0360.042 ± 0.0370.119 ± 0.072100/100
SLS, ε=0.010.057 ± 0.0480.070 ± 0.0590.187 ± 0.122100/100
SLS, ε=0.050.045 ± 0.0460.060 ± 0.0640.201 ± 0.14378/100
SLS, ε=0.10.037 ± 0.0400.050 ± 0.0580.203 ± 0.13978/100
Fashion-MNISTOurs0.111 ± 0.0800.156 ± 0.1080.523 ± 0.345100/100
SLS, ε=0.010.130 ± 0.0800.183 ± 0.1120.633 ± 0.458100/100
SLS, ε=0.050.111 ± 0.0690.164 ± 0.1060.718 ± 0.551100/100
SLS, ε=0.10.097 ± 0.0560.152 ± 0.0960.718 ± 0.533100/100

Mean ± std over pairs connected by all methods

The graph-based method trades transition quality against path quality through ε: small ε connects all pairs but gives worse transitions, while larger ε gives sharper transitions at the cost of longer paths and, on MNIST, 22% of pairs left disconnected. Portals avoid this trade-off. Their transition error is comparable to the best ε setting, while their paths are shorter than every setting, and every pair is connected. We will add this comparison to the paper.

Impact of a balance term on collapse

We agree that the original submission did not establish why related work such as MLMM-VAE collapsed. These methods collapsed when we ran the authors' code on our image datasets. To isolate the routing mechanism, the submitted paper already integrates MLMM-VAE routing within our framework (Table 5, ELBO routing). We will make this more explicit in the revision.

To further separate the routing rule from the balance regularizer, we ran a 2x2 grid crossing the MLMM-VAE router and ours with and without the γ balance term (Eq. 19), using the same backbone, iteration count and per-dataset hyperparameters as in the paper

  • ELBO: per-chart ELBO routing, as in MLMM-VAE (Alberti et al., 2024);
  • Ours: latent-evidence (prior-marginal) routing.
DatasetRouterDead charts, no γDead charts, γFID, no γFID, γ
MNIST (C=10)ELBO8.3 ± 0.6017.2 ± 4.012.9 ± 0.6
Ours8.3 ± 0.6017.0 ± 4.113.7 ± 1.1
Fashion-MNIST (C=20)ELBO10.0 ± 2.0037.3 ± 0.539.7 ± 0.3
Ours14.3 ± 0.6042.1 ± 1.437.6 ± 0.6
CelebA (C=10)ELBO9.0 ± 0.0048.3 ± 0.047.6 ± 0.1
Ours9.0 ± 0.0048.4 ± 0.242.9 ± 0.3

Mean ± std over 3 seeds. A chart is dead if it receives <1% of the data. For CelebA, the ablation uses the configuration of Table 5 (DCGAN backbone, C = 10, iteration counts matched within each dataset), which differs from the Table 1 configuration

Collapse is driven by the balance term, not the router. Without γ, both routers collapse (on Fashion-MNIST, ELBO routing even leaves fewer dead charts); with γ, neither does. We therefore attribute MLMM-VAE's collapse mainly to the absence of a usage regularizer, and we will revise the manuscript so that it does not suggest our routing alone prevents collapse.

Given γ, the router determines sample quality. On CelebA, γ balances usage under ELBO routing but barely improves FID (48.3 → 47.6), while our routing reaches 42.9. On Fashion-MNIST ours is also better (37.6 vs 39.7); on MNIST the two are on par. We will add this table and discussion to the appendix.

Adherence to a known manifold

For the double-torus dataset the ground-truth manifold is known, so we can measure exactly how far interpolated points lie from it. We sampled 240 random point pairs on the surface and interpolated between them with a single-chart autoencoder (a standard VAE) and with our multi-chart model (4 charts, as in Figure 1). For each path, we record the largest distance of any interpolated point to the true surface and what percentage of paths have a maximum distance above a certain threshold:

Maximum distance to the surface per path (mesh units, 240 paths):

Methodp5Medianp95Max
Portal (ours)0.00260.00310.00370.0044
Single chart0.00270.00340.04270.1022

Paths leaving the surface (max distance > threshold):

Method> 0.005> 0.01> 0.02> 0.05
Portal (ours)0 (0%)0 (0%)0 (0%)0 (0%)
Single chart53 (22%)35 (15%)28 (12%)9 (4%)

The two methods have nearly identical typical paths (similar median), but they differ sharply in the tail. On a threshold that sits above our maximum distance (0.0044), the single chart baseline leaves the surface in 22% of the cases. This is the failure mode a single chart cannot avoid on a surface of genus 2: no single global parameterization covers it, so some straight-line interpolations in latent space must cut through regions that are off the manifold. Portals avoid this by switching charts before the path leaves a chart's valid region. We will add this information to the supplemental material.

This experiment also speaks to the faithfulness and coverage of the model. The distance to the true surface is a ground-truth version of unfaithfulness: it measures directly how far decoded points stray from the manifold, rather than estimating it through nearest neighbours in the training set. Interpolated points are the hardest case, since they come from latent regions between the encodings of real data. On coverage, the four charts split the surface almost exactly evenly (see also Figure 1), each representing about a quarter of the data. The model therefore represents the whole surface rather than collapsing onto part of it. And because the endpoints span every pair of charts, the paths cross the whole surface. That our model stays on the manifold along all of them means faithfulness holds across the full extent of the surface.

Dear reviewer kqid:

As the discussion period ends tomorrow, we hope our rebuttal has addressed your concerns. We would appreciate it if you could reassess our work in light of our responses. If any concerns remain, we would be happy to address them.

Thank you for your time and consideration.

Reviewer_HCv7OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a multi-chart VAE with latent-evidence routing to assign inputs to charts. After training, it constructs portals and a chart graph for cross-chart interpolation. Experiments on a synthetic surface, MNIST, Fashion-MNIST, and CelebA evaluate generation, reconstruction, and interpolation.

Critical strengths

The Fashion-MNIST generation result is encouraging: sample FID improves from 45.07 for VaDE to 37.73 with matched parameter counts and training iterations. This provides some evidence that the proposed model is useful for generation.

Critical weaknesses
  1. Connecting local representations and interpolating across charts are not new ideas. Section 2 (p. 3, lines 130–141) already discusses chart autoencoders with decoder–encoder transition maps and the graph-based atlas interpolation of Stolberg-Larsen and Sommer (2022). I understand that this paper introduces a different routing score and represents overlaps as distributions rather than individual points (Section 3.3, p. 7). However, the paper does not make clear why these changes amount to a substantial advance. What becomes possible, or meaningfully better, because of these particular choices?

  2. Figure 6 shows sequences of plausible intermediate images, but I am not sure what problem these sequences solve or how their usefulness should be measured. A visually plausible path does not by itself establish that the model has recovered meaningful transitions in the data. The generation results do not resolve this concern: sampling selects a chart and decodes a latent without using the portals or transition graph (Section 3.2 and Algorithm 2). Thus, the FID improvements cannot establish the value of the bridging mechanism. I would like to see a clearer connection between this part of the method and a measurable benefit.

  3. Section 3.3 describes searching for a route through intermediate charts and fitted portals, while the Figure 6 caption explicitly states that the VAE baseline follows a straight line. One method can take a detour around low-density regions; the other cannot. This changes both the representation and the interpolation procedure, so it is difficult to tell which accounts for the better-looking results. Matching parameter counts and training iterations does not fix this difference. A single-chart baseline with geometry-aware path selection would help establish whether multiple charts are actually responsible for the improvement.

  4. Section 4 and Appendix D.5 state that CAE and MLMM-VAE were excluded because most charts became inactive during training. This is disclosed, but the resulting comparisons with a plain VAE and VaDE do not establish an advantage over existing multi-chart models. The graph-based atlas method discussed in Section 2 is a relevant missing comparison. The evidence therefore leaves the benefit over the closest alternatives unresolved.

Questions for authors
  1. What specific benefit does the distributional portal provide over the point-based connections in the earlier atlas interpolation method?
  2. What evidence demonstrates the usefulness of the cross-chart connections beyond visually plausible interpolation sequences, given that unconditional sampling does not use them?
  3. Does the interpolation advantage remain when the single-chart baseline also uses a geometry-aware path rather than a straight line?
  4. How does the proposed method compare with the existing graph-based atlas interpolation method discussed in Section 2?
5 comments

We thank the reviewer for the careful review and the clear questions. They led to a direct comparison with the graph-based atlas interpolation of Stolberg-Larsen & Sommer (Q1, Q4) and a new on-manifold test on the double torus (Q2)

Q1: What specific benefit does the distributional portal provide over the point-based connections in the earlier atlas interpolation method?

Q1.1: Why represent overlaps as distributions?

In an atlas, transition maps are defined on overlaps, which are regions of the manifold, not isolated points (Sec. A.1). A portal q_ab captures such a region: its mean locates the overlap and its covariance describes its extent, so the transition map ψ_ab (Eq. 8) is fitted over a neighbourhood and its round trip is checked there (Eq. 9). Unlike the isolated links of Stolberg-Larsen & Sommer, which offer only a discrete set of crossing points, a portal provides a continuous region with a differentiable density on which the transition map is fitted. The crossing point can therefore be optimized with gradients rather than chosen combinatorially, which opens the way to optimizing interpolation paths in latent space, including where they cross between charts. More broadly, continuous overlaps with differentiable transition maps are what latent generative models such as latent diffusion would need to move trajectories between charts, as is standard for processes on manifolds expressed in local charts. We leave both directions to future work. A region also removes the dependence on data in the overlap. Our router partitions the data fairly sharply, so few training points are claimed by two charts at once, and point-based links depend on those rare points and a threshold ε. Portals are instead fitted where the router ties, which exists whether or not any training point lies there. They are defined by the model itself, and every chart-graph edge P(b | a) (Eq. 10) comes with a portal through which it can be traversed. One limitation: the portal objective preserves q_ab as a distribution rather than enforcing pointwise invertibility (Sec. 3.3, Appendix C), so in Q1.2 we measure the pointwise transition error directly.

Q1.2: What measurable benefit do portals give over point-based connections?

To answer this directly, we implemented the point-based graph interpolation of Stolberg-Larsen & Sommer (SLS) on the same trained VAA charts and the same 100 endpoint pairs, so that only the transition mechanism differs. Following their method, we encode N = 2000 data points, build a kNN graph in each chart’s latent space with decoded curve-length edge weights, add cross-chart edges at data points routed above a threshold ε to both charts, and find shortest paths. We sweep ε and report four metrics for a fixed number of frames: Transition LPIPS (perceptual distance between the decoded images on either side of a chart crossing), Smoothness (largest LPIPS between consecutive frames), path length (sum of LPIPS between consecutive frames), connectedness (number of endpoint pairs between which a connection exists):

DatasetMethodTransition LPIPS ↓Smoothness ↓Path length ↓Connected
MNISTOurs0.040 ± 0.0360.042 ± 0.0370.119 ± 0.072100/100
SLS, ε=0.010.057 ± 0.0480.070 ± 0.0590.187 ± 0.122100/100
SLS, ε=0.050.045 ± 0.0460.060 ± 0.0640.201 ± 0.14378/100
SLS, ε=0.10.037 ± 0.0400.050 ± 0.0580.203 ± 0.13978/100
Fashion-MNISTOurs0.111 ± 0.0800.156 ± 0.1080.523 ± 0.345100/100
SLS, ε=0.010.130 ± 0.0800.183 ± 0.1120.633 ± 0.458100/100
SLS, ε=0.050.111 ± 0.0690.164 ± 0.1060.718 ± 0.551100/100
SLS, ε=0.10.097 ± 0.0560.152 ± 0.0960.718 ± 0.533100/100

Mean ± std over pairs connected by all methods

The results match the motivation in Q1.1. The point-based graph faces a trade-off in ε: raising it improves transitions but lengthens paths and, on MNIST, disconnects 22% of pairs, while lowering it restores connectivity at the cost of worse transitions. Portals avoid this trade-off because they do not rely on shared data points. They connect every pair, give the shortest paths in perceptual length (about 40% shorter on MNIST and 17–27% on Fashion-MNIST), and match the best ε setting of the point-based graph in transition error and smoothness. They also need no stored data, since interpolation is a shortest-path search over the C-node chart graph (Eq. 10). We will add this comparison to the revised paper.

Q2: What evidence demonstrates the usefulness of the cross-chart connections beyond visually plausible interpolation sequences, given that unconditional sampling does not use them?

The reviewer is right that unconditional sampling does not use the portals (Algorithm 2), so sample FID cannot establish their value. Sampling within a single chart, however, is the simplest use of the atlas. Downstream methods that move along continuous paths in latent space, such as latent diffusion models, flow-based models or latent-space optimization, would need to cross between charts. The portals and transition maps are what make such crossings possible (see also Q1.1).

FID is also unsuitable for evaluating interpolations in general. It compares distributions, whereas an interpolation path visits only a narrow subset of the manifold, passing through the limited regions where charts meet. For MNIST, Fashion-MNIST and CelebA the true data manifold is unknown, so whether a path stays on it cannot be measured directly. We therefore go beyond visual inspection with the quantitative path metrics of Q1.2. Transition LPIPS measures the perceptual discontinuity at each chart crossing, smoothness measures the worst jump along a path, and path length measures the total perceptual change. On these metrics portals match or improve on the existing point-based method while connecting every endpoint pair. On the double torus, where the manifold is known exactly, we can additionally measure directly whether paths stay on it. To address this, we computed the maximum distance to the true surface along each of 240 interpolation paths:

Methodp5Medianp95Max
Portal (ours)0.00260.00310.00370.0044
Single chart0.00270.00340.04270.1022

Paths leaving the surface (max distance > threshold):

Method> 0.005> 0.01> 0.02> 0.05
Portal (ours)0 (0%)0 (0%)0 (0%)0 (0%)
Single chart53 (22%)35 (15%)28 (12%)9 (4%)

No portal path leaves the surface, whereas 22% of single-chart paths do. We will add these results to the revised paper.

Q3. Does the interpolation advantage remain when the single-chart baseline also uses a geometry-aware path rather than a straight line?

We would like to clarify that VAA does not use geometry-aware search within charts. It interpolates along straight lines inside each chart, and the only search is over the sequence of C charts. The advantage therefore does not come from detouring around low-density regions.

Our results also suggest that path search is not what drives the improvement. The method of Stolberg-Larsen & Sommer in Q1.2 is a geometry-aware path search: shortest paths on a kNN graph of encoded data with decoded curve-length weights. Applied to the same charts, it still produces longer paths than our straight-line interpolation through portals (Q1.2 table), with no better transitions or smoothness than the best of its ε settings. Given the atlas, straight lines suffice.

On manifolds with non-trivial topology, a single chart cannot match an atlas regardless of the path rule. A single 2D chart cannot cover the double torus without a tear, so for endpoints on opposite sides of it, any single-chart path must either leave the surface or detour around the tear. The double-torus results in Q2 are consistent with this: no portal path leaves the surface, whereas 22% of single-chart paths do.

We will clarify in the revised manuscript that interpolation within each chart is along straight lines, and that the improvement over the single-chart baseline comes from the atlas rather than from path search.

Q4: How does the proposed method compare with the existing graph-based atlas interpolation method discussed in Section 2?

We ran this comparison and report it in Q1.2. On the same trained charts and endpoint pairs, our portals connect every endpoint pair and give perceptually shorter paths than the graph-based method of Stolberg-Larsen & Sommer at every threshold ε, with comparable transition error and smoothness. The point-based graph, in contrast, must trade transition quality against connectivity.

Regarding the other multi-chart baselines (W4): the ELBO routing column of Table 5 is the MLMM-VAE router implemented within our framework. Our new routing × balance ablation shows that MLMM-VAE’s chart collapse is caused by the absence of a balance term, not by its router. With the balance term, both routers avoid collapse, and latent-evidence routing gives better sample quality on CelebA (42.9 vs. 47.6, in the Table 5 configuration) and Fashion-MNIST (37.6 vs. 39.7) and is on par on MNIST. We will make these comparisons explicit in the revised manuscript.

Dear reviewer HCv7:

As the discussion period ends tomorrow, we hope our rebuttal has addressed your concerns. We would appreciate it if you could reassess our work in light of our responses. If any concerns remain, we would be happy to address them.

Thank you for your time and consideration.

Reviewer_F4VWOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem that traditional Variational Autoencoders (VAEs) encode all data onto a single, continuous latent space. If the data is distinct and/or discontinuous, then assuming that it can all be represented by a single map can cause issues to rise when performing tasks such as interpolation. To avoid making this assumption, the authors propose Variational Atlas Autoencoders (VAAs), which divides the latent space into multiple local overlapping maps that make up the entire latent space in combination. The authors clearly state four core contributions: they divide the single latent space into multiple separate charts, each with its own VAE; they propose latent-evidence reasoning, using solely the encoder rather than pre-existing labels to assign images to charts; they address cases when an image is assigned equal probabilities to multiple charts by introducing portals; lastly, they show their model has better sampling skills than the VAE and VaDE.

The authors’ final contribution is a demonstration of the Variational Atlas Autoencoder’s superior sampling over Variational Autoencoder and Variational Deep Embedding models on the MNIST, Fashion-MNIST, and CelebA datasets. These models are all scaled to have similar amounts of parameters to ensure they are the same in size. Although the traditional VAE had the lowest reconstruction FID due to using all of its network for every image, the VAA model has the lowest sample FID scores across the datasets, which is attributed to its better latent structure. The authors also claim that the VAA can categorize data subclasses into separate charts without supervision and yields realistic samples during interpolation in contrast to the VAE’s poorer samples.

Critical strengths

The problem defined by this paper is mathematically and conceptually well-defined. Using differential geometry and probabilistic machine learning, the authors provide a rigorous generative formulation, a principled training objective, and explicit closed-form scores for routing data. Mathematical decompositions are given when necessary and adequately explained, as I did not need to rely on outside sources to understand how the authors derived their equations. Conceptually and logically, this paper is consistently well-written and well-explained, and following the authors’ reasoning proved to not be challenging.

The authors make an explicit argument that their work is novel. Through the related work section, they state what literature and findings have led up to their proposal, their individual shortcomings, and how Variational Atlas Autoencoders address these challenges. Furthermore, the results section demonstrates empirical rigor and novel improvements, with the Variational Autoencoder and Variational Deep Embedding chosen as appropriate baselines to demonstrate that the VAA improves on its predecessors’ sampling abilities. Particularly choosing the VaDE, a close architecture to the VAA, is an appropriate choice as the comparison demonstrates that it is not enough to split the latent space but retain a single encoder-decoder pair.

Critical weaknesses

This is a minor limitation given that the paper focuses on introducing an architectural breakthrough, but I would have appreciated some exploration of the tradeoff of increasing the number of charts, C. I see that the authors choose C=20 in the experimental analysis of the VAA via grid search, but I wonder how the model may have performed for significantly larger values on the chosen datasets. It is also disadvantageous for the paper to not quantify the extra post-processing time that forming portals and computing the transition graph takes.

Questions for authors

I am interested in the potential limitations of scaling the number of encoders. It was mentioned that encoding a new data point requires either running all C encoders or training a smaller classifier. How computationally expensive would each of these tasks be as we scale up in C?

1 comment

We thank the reviewer for their careful reading and positive assessment of our work. We appreciate their comments on the clarity of the formulation and on the choice of baselines. We address their questions on scaling the number of charts below.

How computationally expensive would each of these tasks [running all C encoders or training a smaller classifier] be as we scale up in C?

In our architecture, running all C encoders is a single forward pass. The convolutional encoder is shared, each chart owns only a small linear head, and all heads are evaluated together in one batched operation. The heads shrink as 1/C, so the encoder's size is essentially independent of C. Routing needs only the encoder outputs: each chart's score is computed in closed form from its posterior mean and variance, and no decoding is required. Encoding and routing a new point therefore cost essentially the same for any number of charts. Once the chart is chosen, only that chart's decoder is used.

Where the routing distribution p(c | x) is a simple function of x, it could still make sense to train a separate, lightweight routing model that distills it, either jointly during training or afterwards. It is not yet clear for which problems learning the routing function separately from the encoder would pay off, and we leave this to future work.

Exploration of the trade-off in the number of charts C

While developing the method, we observed that with too few charts, distinct subgroups are forced to share a chart, so its latent distribution contains several separate modes and prior samples fall in the gaps between them. With too many charts, the cost is mostly redundancy. Choosing C somewhat too large is therefore less harmful than choosing it too small: on MNIST and Fashion-MNIST, even C = 100 samples better than C ≤ 5. On CelebA the good range is narrower, with the best results at C = 10–20. We will add additional quantitative results to the supplemental regarding scaling the number of charts.

Quantify extra post-processing time (cross-chart transitions)

Building the atlas is a one-off post-processing step after training. For the configurations in the paper, it takes about 2 minutes on MNIST and 8 minutes on Fashion-MNIST and CelebA. Nearly all of this time is spent fitting portals, one per ordered pair of charts, so the cost scales super-linearly in C. This step was not optimised during development: pairs are processed one at a time, although they are independent and could be fitted in parallel. We will add these timings to the revised paper.

Reviewer_N2yzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses several local latent spaces, called charts, to represent complex data. Each chart has an encoder and a decoder. The model assigns each input a weight for each chart. Charts receive higher weights when their encoded representations better match their assumed latent distributions. The rule does not use class labels.

After training, the method fits overlap regions and connects the charts in a graph. These connections let the model generate intermediate samples between inputs that belong to different charts. Experiments use a double-torus surface, MNIST, Fashion-MNIST, and CelebA.

Critical strengths

The problem is valuable for representation learning. A single latent space can represent separate data regions poorly. Local charts offer a useful way to separate these regions and preserve their structure.

The routing rule is simple and practical. It uses encoder outputs that the model already computes. This choice avoids a direct dependence on each decoder's current reconstruction quality. The routing comparisons in Table 5 help assess this design choice.

The overlap construction gives the model a useful function beyond clustering. The chart graph permits paths across several local representations. The sample-quality improvement on Fashion-MNIST is also substantial relative to the reported baselines.

Critical weaknesses

The distinct contribution over earlier atlas models is not yet convincing. The paper builds on mixtures of autoencoders and existing chart-based interpolation methods. Its main new parts are the routing rule and the overlap construction. However, the main comparisons exclude the closest multi-chart baselines after training failures. Those failures may show a real benefit, but the paper does not establish their cause well enough to support that conclusion.

The evidence for broad practical benefit is limited. On CelebA, sample FID is 27.84 for this method and 28.01 for VaDE. This difference is small relative to the reported variation. The cross-chart interpolation examples are promising, but they do not establish an advantage over a previous atlas method. Also, the portal objective preserves a distribution under a round trip, rather than each point, as Section 3.3 acknowledges. A comparison of path quality and pointwise transition error would directly test the value of the new overlap construction.

Questions for authors

Do the closest multi-chart baselines still collapse with the same balance loss and comparable tuning effort? What improvement do portals give over an existing atlas transition method, in path quality and pointwise transition error?

2 comments

We thank the reviewer for carefully reading and reviewing our manuscript. Their questions prompted two additional experiments, a direct comparison with an existing atlas transition method and an ablation of the balance term, which we believe strengthen the paper. We address each point below. Regarding their note on the comparison with VaDE, we agree that this difference (27.84±0.42 vs. 28.01±0.26) is within noise, and will revise the abstract and Section 4 so they no longer claim an improvement on CelebA.

Comparison to existing atlas transition methods

We have added a direct comparison with the graph-based cross-chart interpolation of Atlas Generative Models (Stolberg-Larsen & Sommer, 2022. SLS), the method discussed in Section 2. Both methods operate on the same trained atlas, so only the transition mechanism differs. For 100 random test pairs, we compare our portals against their kNN graph (k=20, 2000-point pool) at three values of their chart-membership threshold ε, a value that needs to be tuned. We report LPIPS at the chart switch (pointwise transition error), the largest LPIPS step along the path (smoothness), and summed LPIPS along the path (path length).

DatasetMethodTransition LPIPS ↓Smoothness ↓Path length ↓Connected
MNISTOurs0.040 ± 0.0360.042 ± 0.0370.119 ± 0.072100/100
SLS, ε=0.010.057 ± 0.0480.070 ± 0.0590.187 ± 0.122100/100
SLS, ε=0.050.045 ± 0.0460.060 ± 0.0640.201 ± 0.14378/100
SLS, ε=0.10.037 ± 0.0400.050 ± 0.0580.203 ± 0.13978/100
Fashion-MNISTOurs0.111 ± 0.0800.156 ± 0.1080.523 ± 0.345100/100
SLS, ε=0.010.130 ± 0.0800.183 ± 0.1120.633 ± 0.458100/100
SLS, ε=0.050.111 ± 0.0690.164 ± 0.1060.718 ± 0.551100/100
SLS, ε=0.10.097 ± 0.0560.152 ± 0.0960.718 ± 0.533100/100

Mean ± std over pairs connected by all methods

The graph-based method trades transition quality against path quality through ε: small ε connects all pairs but gives worse transitions, while larger ε gives sharper transitions at the cost of longer paths and, on MNIST, 22% of pairs left disconnected. Portals avoid this trade-off. Their transition error is comparable to the best ε setting, while their paths are shorter than every setting, and every pair is connected. We will add this comparison to the paper.

Impact of a balance term on collapse

We agree that the original submission did not establish why related work such as MLMM-VAE collapsed. These methods collapsed when we ran the authors' code on our image datasets. To isolate the routing mechanism, the submitted paper already integrates MLMM-VAE routing within our framework (Table 5, ELBO routing). We will make this more explicit in the revision.

To further separate the routing rule from the balance regularizer, we ran a 2x2 grid crossing the MLMM-VAE router and ours with and without the γ balance term (Eq. 19), using the same backbone, iteration count and per-dataset hyperparameters as in the paper:

  • ELBO: per-chart ELBO routing, as in MLMM-VAE (Alberti et al., 2024);
  • Ours: latent-evidence (prior-marginal) routing.
DatasetRouterDead charts, no γDead charts, γFID, no γFID, γ
MNIST (C=10)ELBO8.3 ± 0.6017.2 ± 4.012.9 ± 0.6
Ours8.3 ± 0.6017.0 ± 4.113.7 ± 1.1
Fashion-MNIST (C=20)ELBO10.0 ± 2.0037.3 ± 0.539.7 ± 0.3
Ours14.3 ± 0.6042.1 ± 1.437.6 ± 0.6
CelebA (C=10)ELBO9.0 ± 0.0048.3 ± 0.047.6 ± 0.1
Ours9.0 ± 0.0048.4 ± 0.242.9 ± 0.3

Mean ± std over 3 seeds. A chart is dead if it receives <1% of the data. For CelebA, the ablation uses the configuration of Table 5 (DCGAN backbone, C = 10, iteration counts matched within each dataset), which differs from the Table 1 configuration

Collapse is driven by the balance term, not the router. Without γ, both routers collapse (on Fashion-MNIST, ELBO routing even leaves fewer dead charts); with γ, neither does. We therefore attribute MLMM-VAE's collapse mainly to the absence of a usage regularizer, and we will revise the manuscript so that it does not suggest our routing alone prevents collapse.

Given γ, the router determines sample quality. On CelebA, γ balances usage under ELBO routing but barely improves FID (48.3 → 47.6), while our routing reaches 42.9. On Fashion-MNIST ours is also better (37.6 vs 39.7); on MNIST the two are on par. We will add this table and discussion to the appendix.

Dear reviewer N2yz:

As the discussion period ends tomorrow, we hope our rebuttal has addressed your concerns. We would appreciate it if you could reassess our work in light of our responses. If any concerns remain, we would be happy to address them.

Thank you for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all four reviewers for their careful reading and constructive feedback.

The reviewers recognized the relevance of learning multiple local latent representations without supervision where a single latent space represents separate data regions poorly (N2yz, kqid); praised the principled ELBO-derived formulation and closed-form routing score with clear derivations (F4VW) and the simple and practical routing mechanism for self-supervised chart selection based on latent evidence (N2yz, kqid); and praised the functions beyond clustering such as building a chart graph to form an understanding of local representations (N2yz, kqid). Moreover reviewers appreciated the clarity of the manuscript and the improved quality of the results.

The reviewers raised some concerns regarding why closely related methods collapsed (MLMM-VAE, ELBO routing), and whether they still collapse with the same balance loss and comparable tuning effort (N2yz, HCv7, kqid). Related to this, they asked to disentangle the effect of the routing rule from the balance term and to report chart utilization. Reviewers HCv7, N2yz and kqid moreover asked us for quantitative evidence that this approach would benefit interpolation beyond the visual results. F4VW asked about the trade-off in the number of charts and how the cost of routing and atlas construction scales with C.

New experiments and revisions

Experiments

  • Routing x balance ablation, crossing both routers with and without the balance term γ (N2yz, HCv7, kqid)
  • Comparison with the point-based interpolation of Stolberg-Larsen & Sommer (2022) on the same trained charts and endpoint pairs (N2yz, HCv7, kqid)
  • Distance to the true manifold on the double torus (HCv7, kqid)
  • Chart-count scaling, including training and atlas construction time (F4VW)

Revisions

  • Make explicit that the ELBO routing in Table 5 is the MLMM-VAE router, and no longer suggest that our routing alone prevents collapse
  • Remove the claimed improvement over VaDE on CelebA (N2yz)
  • Clarify the interpolation procedure of each method, including the Figure 6 caption (HCv7)

Concluding remarks

We thank the reviewers again for their thoughtful feedback, which has substantially strengthened the paper. The new experiments clarify both contributions. First, our formulation yields a balance term that is necessary to avoid chart collapse, and given this term, latent-evidence routing gives better sample quality than the ELBO routing of MLMM-VAE. Second, distributional portals connect charts more reliably and with perceptually shorter paths than the point-based atlas interpolation of Stolberg-Larsen & Sommer, and keep interpolations on the true manifold where it is known. We have also corrected claims that were stronger than the evidence, namely that our routing alone prevents collapse and the improvement over VaDE on CelebA.

We hope these additions address the reviewers’ concerns. Each reviewer’s questions are answered in detail in our individual responses. If you find that we have addressed your concerns, we would kindly ask you to consider revising your score, and we are happy to answer any further questions during the discussion period.

Dear reviewers,

If you believe the authors have addressed your questions, please respond to the rebuttal.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.