acceptodds
Under review as a conference paper at ICLR 2027

Learning Manifold-Faithful Representations

Abstract

Autoencoders learn latent representations for reconstructing observed data. However, reconstruction training constrains the decoder only at encoded data points, providing no guarantee that perturbations of these codes will decode to samples that remain on the data manifold. Consequently, even a small step away from an encoded point can push the model into an unconstrained region of the latent space, causing the decoded output to degrade. To address this limitation, we introduce manifold-faithful traversal, which constrains movement through the latent space to follow the learned data manifold. By keeping the trajectory on the manifold, latent codes can be repeatedly perturbed while producing plausible intermediate samples without accumulating off-manifold drift over successive steps. We propose a two-phase training framework that first learns a reconstructive representation and then trains the decode–encode mapping to correct perturbed latent codes toward the manifold. Because the true latent manifold is unknown, we approximate this projection using the nearest clean latent code in each training batch. Unlike disentanglement, which seeks latent directions associated with identifiable semantic factors, manifold-faithful traversal does not require specifying what changes along a trajectory; it only requires that the trajectory remain on the data manifold. Experiments on synthetic data and Colored MNIST show that the proposed technique substantially reduces accumulated off-manifold drift and produces coherent trajectories in both random and target-directed walks. We further show that a learned traversal policy together with a memory-efficient training scheme lets the approach scale to larger latent widths and longer trajectories, remaining feasible and effective where the original mechanism is not. Such stable multi-step traversal potentially enables latent space exploration for generative modeling and downstream tasks such as data augmentation that depend on outputs remaining realistic.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
latent space geometry, manifold learning, representation learning, unsupervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9NYrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces manifold-faithful traversal, enabling stable traversal of autoencoder latent spaces. It constrains latent perturbations to remain close to the learned data manifold. The authors propose a two-phase training framework in which a reconstructive autoencoder is first learned, followed by training the decode–encode to project perturbed latent codes toward the manifold using the nearest clean latent code. Experiments on synthetic data and Colored MNIST show that the method reduces drift under both random and target-directed traversal.

Critical strengths

Using a separate decode–encode mapping and the nearest sample point for correction is an intuitive and reasonable approach. The authors also provide corresponding theoretical analysis and validate the method on two simple and intuitive datasets.

Critical weaknesses

My main concern is the relevance of the problem studied in the context of current AI research. The self-supervised learning capability of traditional autoencoder architectures has received relatively limited attention in recent years, partly due to their limited generalization ability. This may limit the practical impact of the proposed work on the current research community. This concern is also reflected in the relatively limited number of closely related recent publications at major conferences. Moreover, the idea of introducing an additional decode–encode mapping for correction is not particularly uncommon, and using similar sample points for such correction is also a relatively straightforward idea.

Questions for authors
  1. Could the authors further clarify the importance and relevance of this research in the context of current AI research?
  2. Could the authors provide a more comprehensive review of related work that adds additional components to autoencoders, such as decode–encode mappings, and clarify the novelty of the proposed approach relative to these existing methods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9NYr

We thank the reviewer for finding the approach intuitive and reasonable, and for the positive assessment of soundness and presentation.

Q1. Relevance. Autoencoders remain central to generative modeling. Latent diffusion and flow models generate in the latent space of a pretrained autoencoder (Rombach et al., 2022; Peebles & Xie, 2023; Esser et al., 2024), so their samples land on latents the decoder never saw during reconstruction training. The autoencoder in these systems is itself an active research topic at major venues. Recent work studies how its latent space supports generation (Yao et al., CVPR 2025), regularizes its geometry (Kouzelis et al., ICML 2025; Skorokhodov et al., ICML 2025), compresses it further (Chen et al., ICLR 2025), and trains it jointly with the generator (Leng et al., ICCV 2025). Sparse autoencoders are a standard tool for interpreting language models (Huben et al., ICLR 2024), and the dynamics of iterated encoding and decoding, the setting closest to ours, was an oral at ICLR 2026 (Fumero et al.). The specific question of decoding off the embedded codes goes back to Berthelot et al. (2019), who showed that standard objectives do not make decoded interpolations realistic. Our method addresses this directly, and can also be read as latent augmentation, where a model trained with Lproj\mathcal{L}_{\mathrm{proj}} provides small latent steps whose decodes stay near the data manifold (Eq. 8, Table 1). We added this motivation to the introduction.

Q2. Novelty relative to relevant work. We agree decode-encode consistency terms are common, as discussed in Sec. 7, they differ from ours in the target, and we now make this explicit. They differ from ours in what goes into the cycle and what it is pulled toward.

LossInput to the cycleTargetOptimum
Latent cycle (BicycleGAN, MUNIT, DRIT)E(G(z))E(G(z)), zz from the priorthe same zzidentity on the prior's support
ALI / BiGANjoint pairsadversarial, no pointwise targetinverse pair in distribution
Latent cycle at random zzrandom zzthe same zzidentity on the latent space
Denoising (Lden\mathcal{L}_{\mathrm{den}})zi+σεz_i+\sigma\varepsilonoriginal code ziz_iposterior mean, off a curved manifold
Ours (Lproj\mathcal{L}_{\mathrm{proj}})zi+σεz_i+\sigma\varepsilonnearest codeprojection onto the manifold

Every other loss in the table either sends the cycle back to where it started or matches the distributions without a pointwise target. Their optimum is the identity on the prior or the latent space, or an average of codes that sits off a curved manifold (denoising). Ours is the only one that sends a perturbed code to the data manifold rather than back to its own origin. The change is small, from the original code to the nearest real code, and it is what turns the optimum into a projection with a convergence rate rather than a conditional mean (revised Prop. 2). Empirically, walks trained with this target stay at a bounded distance from the true manifold, while the latent-cycle objective, the closest decode-encode alternative, keeps drifting (Table 1, distance 2.96 against 41.6 after 40 steps).

Any model trained to return perturbed points to the data, such as autoencoders behind latent diffusion models, faces the same choice of target, and Prop. 2 shows which one lands on the data manifold. We see this simple change with a provable effect as a main contribution of the paper, and the revised Sec. 4.1 states it explicitly.

We hope the added context shows that the question matters for how autoencoders are used today, and that the nearest-code target is a precise change with provable consequences rather than a generic decode-encode term. If so, we would be grateful if the reviewer would consider raising the score.

Thank you very much for the authors' update. I acknowledge that I am not an expert in this research area. However, conducting experiments only on Swiss Roll and Colored MNIST gives me the impression that this work may have limited impact on addressing frontier research problems.

Based on the authors' response, I believe that the proposed method has considerable potential to contribute to several important and emerging research problems. Combined with theoretical guarantees, this could make for a promising paper. Therefore, conducting experiments demonstrating the generalization capability of the proposed method on frontier problems would make the work much more convincing.

Alternatively, could the authors provide some references showing that evaluating methods only on Swiss roll and MNIST remains a common practice in this research community, e.g., ICLR, ICML?

AuthorsOpenReview ↗

Response to Reviewer 9NYr

We thank the reviewer for the follow-up, and for seeing potential in the method and its theory.

Is evaluation on synthetic data and MNIST common practice? Yes, for work whose central claim is about the geometry of the data manifold. Recent examples at ICLR and ICML, in autoencoders and in flow matching, are evaluated at the same scale as ours.

  • Zhan et al. (ICLR 2026), Bi-Lipschitz autoencoders with an injectivity guarantee, evaluated on the swiss roll, dSprites and rotated MNIST.
  • Diepeveen et al. (ICML 2025), score-based pullback Riemannian geometry and Riemannian autoencoders, evaluated on synthetic manifolds, synthetic image manifolds and MNIST.
  • Chen & Lipman (ICLR 2024, oral), flow matching on general geometries, evaluated on Earth-science datasets of at most 13k points, meshes and maze-shaped manifolds.
  • Nazari et al. (ICML 2023), geometric autoencoders, evaluated on MNIST, Fashion-MNIST and single-cell data.

The reason these works and ours rely on such datasets was stated by Berthelot et al. (ICLR 2019), the closest prior work on decoding between codes. Whether a decoded point lies on the data manifold can only be measured when the manifold is known. They built a synthetic dataset and evaluated interpolation quantitatively there. Our evaluation follows the same logic. The swiss roll is where the distance to the true manifold is exact, and Colored MNIST shows that the effect transfers to images.

What this setting lets us measure. Because the manifold is known, we can check Definition 1 exactly at every frame of a walk, not only at its end. We used this in a controlled comparison, now in App. C.4, where both objectives are fine-tuned from the same model and only the training target and the noise scale differ (10 seeds, three latent widths). A walk counts as a success only if it reaches the target and every decoded frame stays within a tolerance of the true manifold. At σ=1.0\sigma = 1.0, ours succeeds in 80.8% of walks against 40.5% for denoising (gap 40.3 points, 95% CI 32.7 to 47.1), in all 10 seeds and at every latent width and step size we tested (Table 4). Together with Prop. 2 and its numerical check (Table 3), this isolates the effect of the target, which is only possible when the manifold is known.

Frontier problems. We agree that scaling up is the important next step, and a natural one. The works above show how this tends to happen. Flow matching on general geometries was introduced on small manifolds (Chen & Lipman, ICLR 2024) and extended to protein backbone generation in the same year (Bose et al., ICLR 2024). Our objective is in a similar position. It applies wherever a model is trained to return perturbed points to the data, including the autoencoders behind latent diffusion and image tokenizers (general response, point 4), and Sec. 6 gives an initial recipe for larger scales, with a memory bank that supplies the reference codes and a learned direction for traversal. We could not run this reliably within the discussion period, and the paper names it as the main direction for future work.

We hope these references show that our evaluation follows the standard of this line of work, and that the theory and the controlled results make the contribution clear. If so, we would be grateful if the reviewer would consider raising the score.

Reviewer_iUWoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to learn manifold-faithful representation by denoising autoencoder latent code to its nearest neighbor instead of itself. It proposes theories to prove that the training is feasible, and how would it help the latent traversal during test time. On 2D swiss roll and colored mnist, the method demonstrates manifold-adhering representations after training.

Critical strengths
  1. The problem statement is clear.
  2. The design choice is simple and well-motivated.
  3. The paper is not that hard to follow.
  4. The limitation section is highly appreciated.
Critical weaknesses
  1. I do not understand well why the Proposition 2 derives a result that says a denoising AE target L_den here, can have degenerated solution. According to score matching and related derivations like in [1], a denoising AE naturally gives the score of a distribution, whose optimum is a conditional expectation. It seems to me that the assumption is that g(z, ϵ\epsilon) factors through z and ϵ\epsilon when only z+ϵ\epsilon is visible to the E*D function, which in general is non-identifiable (note that this identifiability has nothing to do with the Where we differ section). I think this part lacks certain clarity, or it largely undermines the theoretical motivation.
  2. I'm not sure about the motivation of the paper to answer: "can we repeatedly move through a learned latent space while remaining on the data manifold?". It seems to be well-answered by many existing papers like [2-6], and the paper also shows significant overlap with them. For example in Section 6, the method of building the graph resembles that in [6]. I hope the authors provide discussions related to those works and clarify the contribution, and provide empirical comparisons to this line of work.
  3. Related to 2, the paper does not provide a large-enough scale experiments on real-world datasets, given the fact that many previous works can already perform manifold-faithful latent traversals [7-9] after learning on large datasets.
  4. I'm not sure about the role of the proposed L_proj, given that the snap rate during learning can be as small as 15%. It means that 85% of the training data is doing denosing AE in the latent space, which (related to point 1) raises the concern of the proved degenerated solutions of denoising AE.

[1] What regularized auto-encoders learn from the data-generating distribution. JMLR, 2014.

[2] Latent space oddity: on the curvature of deep generative models. ICLR 2018.

[3] Metrics for deep generative models. AISTATS 2018.

[4] The Riemannian geometry of deep generative models. CVPRW 2018.

[5] Fast approximate geodesics for deep generative models. ICANN 2019.

[6] (Deep) generative geodesics. ICML 2024 Workshop.

[7] Do not escape from the manifold: discovering the local coordinates on the latent space of GANs. ICLR 2022.

[8] Latent traversals in generative models as potential flows. ICML 2023.

[9] Understanding the latent space of diffusion models through the lens of Riemannian geometry. NeurIPS 2023.

Questions for authors

Please see Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iUWo (1/2)

We thank the reviewer for the positive comments on clarity and design, and in particular for W1, which identified an oversight in Prop. 2 and led to a stronger paper.

W1. Proposition 2. The reviewer is right, E∘DE \circ D observes only z+εz + \varepsilon, from which zz cannot be recovered, so g(z,ϵ)=zg(z,\epsilon)=z is not realizable, and the optimum of Lden\mathcal{L}_{\mathrm{den}} is the conditional expectation E[z∣z+ϵ]\mathbb{E}[z \mid z + \epsilon], as in Alain & Bengio (2014). We have rewritten Prop. 2 accordingly (revised Sec. 4.1, statement in point 2 of the general response). The corrected statement keeps, and sharpens, the motivation. The denoising optimum is a softmax-weighted average of nearby codes, so on a curved manifold it lies strictly inside it, and on the unit circle it stays at distance at least σ2/(2r+σ2)\sigma^2/(2r+\sigma^2) however many codes are available (Prop. 2b). The projection target is a deterministic function of the input given the reference set, so the Lproj\mathcal{L}_{\mathrm{proj}} optimum converges to the true projection at rate O(log⁡n/n)O(\log n/n) (Prop. 2c).

App. B checks this on the unit circle. We train a small MLP with each target and measure its distance 1−∥f(zˉ)∥1-\|f(\bar z)\| to the circle, averaged over held-out noisy inputs zˉ\bar z and three seeds.

σ\sigmaDenoising, predicted (n→∞n\to\infty)Denoising, n=100n=100Denoising, n=10,000n=10{,}000Projection, s=16s=16Projection, s=64s=64Projection, s=1024s=1024
0.10.00500.00570.00510.00320.00100.0000
0.20.02020.02090.02080.00820.00150.0000

(ss is the reference-set size.) The denoising offset matches the prediction and does not shrink with 100 times more data, while the projection offset vanishes as the reference set grows. We also corrected the corresponding step in the proof of Prop. 3.

AuthorsOpenReview ↗

Response to Reviewer iUWo (2/2)

W2. Relation to [2-6]. These works compute paths through a fixed, already trained model (pullback-metric geodesics in Arvanitidis et al., 2018, Chen et al., 2018, and Shao et al., 2018, and graph shortest paths in Chen et al., 2019, and Kim et al., 2024). We ask whether the model itself can be trained so that its decode-encode map retracts onto the data manifold, which is a different question. Without such training, our Lrec\mathcal{L}_{\mathrm{rec}} baseline shows the decoder is unconstrained off the code set, which is the scenario Arvanitidis et al. (2018) handle through an uncertainty-aware metric. The approaches are complementary, where a geodesic method can be run on a model trained with Lproj\mathcal{L}_{\mathrm{proj}}. On Sec. 6, the reviewer is right that kk-nearest-neighbor shortest paths are a standard approximation of geodesics, and we now cite Tenenbaum et al. (2000) there and Chen et al. (2019) and Kim et al. (2024) in Sec. 7. The graph is used only to supervise a direction policy during training, and inference uses the policy and E∘DE \circ D without a graph. A like-for-like empirical comparison is difficult, since these methods compute paths on a fixed model rather than train one, and the two can be combined.

W3. Scale and [7-9]. Choi et al. (2022), Song et al. (2023), and Park et al. (2023) traverse the latents of pretrained GANs or diffusion models (e.g. the W\mathcal{W} space of StyleGAN2 in Choi et al., 2022), where the generators were trained on a dense latent prior. Their scale is inherited from that pretraining, and their contribution is an analysis of a fixed model. Autoencoder latents do not have that, which is why off-code decoding is an issue, and the setting matters in practice because latent diffusion operates in autoencoder latents (general response, point 4). Ours is a training objective, so it must be validated where its effect can be isolated, and our synthetic experiment is the setting where distance to the data manifold can be measured exactly rather than through a proxy. We agree that natural-image validation is the important next step and state it as the main limitation.

W4. Snap rate of 15-30%. This connects directly to W1. The network observes only z+εz + \varepsilon, never which samples are the 70-85% whose targets coincide, so the loss does not split into a denoising part and a projection part. Denoising asks for the origin, which cannot be identified from the input, so its optimum averages nearby codes and lies off a curved manifold. Lproj\mathcal{L}_{\mathrm{proj}} asks for the nearest code, which the input determines, so its optimum is the projection (Prop. 2). When σ\sigma is small compared with the spacing of the codes, the two targets usually coincide, and the difference lies on exactly the inputs a traversal step produces when it moves between codes. The snap rate measures how often training sees such inputs. Sec. 3.2 now states this explicitly.

We thank the reviewer again for W1, which led to a corrected and stronger Prop. 2. As the main soundness concern is now resolved, we would be grateful if the reviewer would reconsider the score, and we are glad to discuss any remaining point.

Thank you for the detailed rebuttal and revisions. I appreciate the correction of Proposition 2 and the expanded related-work discussion.

However, I remain unconvinced that this theoretical distinction establishes a sufficiently novel or practically significant contribution. My concern is that learning a mapping that pulls perturbed inputs toward the data manifold is a longstanding objective of denoising autoencoders and related manifold-learning methods. The corrected Proposition 2 shows that the unconstrained MSE-optimal denoiser has a nonzero projection bias on the unit circle, but this does not imply that denoising cannot produce approximately manifold-faithful trajectories. Definition 1 only requires proximity to the data manifold within a tolerance, rather than exact nearest-point projection. Therefore, the advantage should lie in empirical validation, which shows limited evidence. Specifically, most samples still use the original denoising AE objective (which can learn a manifold-oriented score direction), in the most extreme cases reported in Appendix A, only 0.7% of the samples have modified targets. Although the learned optima could have changed, it is important to establish how much of the empirical benefit can actually be attributed to the changed targets. The potential problems could be inferred from the empirical results in Table 1, that the denoising baseline achieves a lower off-manifold distance than the proposed method. This could be a trade-off of the manifold fidelity and target reaching. Scaling remains to be another significant problem.

Overall, with the corrected prop. 2, I see the proposed approach primarily as an incremental modification of a classical denoising objective, without demonstrating that this modification provides a meaningful advantage over existing manifold-restoration and traversal methods. I keep my score for now.

AuthorsOpenReview ↗

Response to Reviewer iUWo

Thank you for the follow-up and the close reading. All new results are now in the revised PDF (Sec. 5.2, App. A.2, App. C.4).

1. Denoising autoencoders already pull toward the data. We agree that this is the goal of denoising, and it is why denoising is so widely used. The point of Prop. 2 is that the objective does not learn what it is used for. Its direction is indeed the score, fden(zˉ)=∑jwj(zˉ)zj=zˉ+σ2∇log⁡pσ(zˉ)f_{\mathrm{den}}(\bar z)=\sum_j w_j(\bar z) z_j=\bar z+\sigma^2\nabla\log p_\sigma(\bar z) with pσp_\sigma the density of zˉ\bar z, so it points toward the data, but it stops at the posterior mean, which lies off a curved manifold (our reply to W1). What we want from denoising (a pull onto the data) and what it learns (a pull to an average near the data) differ, and the nearest-code target closes this gap. Figs. 7 to 9 (App. A.2) illustrate this.

On the unit circle in Prop. 2b. The averaging in Prop. 2a holds on any manifold. The circle is the setting where its effect has a closed form, so we can bound the gap below by σ2/(2r+σ2)>0\sigma^2/(2r+\sigma^2)>0 however much data is available. App. B (Table 3) confirms that it does not shrink with 100 times more data, while the projection offset falls to zero.

2. Definition 1 only asks for proximity within a tolerance. We agree, and this is exactly what we now measure. A walk counts as a faithful success only if it reaches the target and every decoded frame stays within τ\tau of the true manifold, which is Definition 1 along the whole path, with τ\tau a multiple of the model's own decoding error d0d_0 (see point 3).

3. Attribution and the 0.7% cell. Thank you for this note, which we now address explicitly. The network sees only zˉ\bar z, so the loss does not split into a denoising part and a projection part. It is one objective with a different optimum (Prop. 2). To test it directly, both objectives are fine-tuned from the same Phase-I model with identical settings, so only the target and σ\sigma differ (synthetic data, N=2,4,8N = 2, 4, 8, 10 seeds each, App. C.4). Differences are ours minus denoising, with 95% intervals over seeds.

σ\sigma (× code std)targets changedfaithful success, denoising / oursdifference [95% CI]max distance along path, denoising / ours
0.133%82.6 / 83.1%+0.5+0.5 [−1.3-1.3, 2.3]1.35 / 1.37
0.374%61.8 / 76.9%+15.1+15.1 [10.7, 19.5]1.15 / 1.12
1.096%40.5 / 80.8%+40.3+40.3 [32.7, 47.1]1.60 / 1.14
2.098%67.3 / 76.9%+9.7+9.7 [5.4, 15.0]1.72 / 1.22
3.099%61.6 / 76.7%+15.1+15.1 [7.5, 22.5]2.09 / 1.24

The table shows that our objective is better, by up to 40 points at σ=1.0\sigma = 1.0, in all 10 seeds, at half and double the step size, and at the stricter tolerance 2d02d_0 (64.4% against 23.8%). The 0.7% cell came from our calibration in one Colored MNIST setting, and this controlled comparison now answers the attribution question directly.

4. Table 1 and the trade-off. There is no trade-off. In Table 1 the denoising baseline uses a larger σ\sigma (App. A.1, calibration), which is why it moves less and plateaus lower. At matched σ\sigma, denoising both reaches the target less often and drifts further from the manifold, and its maximum distance along the path grows from 1.35 to 2.09 with σ\sigma while ours stays at 1.1 to 1.2 (Table above and Sec. 5.2). On Colored MNIST the true manifold is unknown, so we rely on visual evidence and a proxy. Figs. 12 and 13 (Table 1 settings) show ours reaching the target where denoising stalls in the targeted walk and barely changes in the undirected one. Under the same controls the proxy agrees. At σ=1.0\sigma = 1.0 ours reaches the target within 20% more often (31% against 12%).

5. Scaling. We agree it is the main next step. Recent geometric work at ICLR and ICML is evaluated at the same scale as ours (e.g. Zhan et al., ICLR 2026, Diepeveen et al., ICML 2025). Sec. 6 gives a recipe for larger scales, as a starting point for applying the objective there. We could not run this reliably within the discussion period.

To summarize, and on the overall assessment. The modification is simple to implement, but its effect is not incremental. It changes what is learned, from a posterior mean that lies off a curved manifold to the projection onto it. Against denoising, the standard manifold-restoration objective, the matched experiment confirms that ours is distinct and stronger at both what denoising is meant to do, staying on the manifold, and traversing it. Ours keeps 77 to 83% faithful success, while denoising falls to 40 to 67% and drifts from the manifold, so there is no trade-off. Traversal methods are complementary, since ours changes only the training target and can be added to any autoencoder they use. We would be glad to clarify any remaining point. We hope this resolves the remaining concerns, and if so we would be grateful if the reviewer would reconsider their score.

Reviewer_UWq9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Ordinary autoencoders are trained solely using a reconstruction loss, enabling it to be well-behaved when given inputs the data points that indeed lie on the manifold. However, their behaviors are less ideal for off-manifold input. The paper proposes to additionally employ a nearest neighbor projection step to train the model upon reconstructions, so as to encourage the model to generate outputs that lie on the manifold. Empirical evidence on synthetic dataset and Colored MNIST demonstrate that the proposed technique allows trajectories that better remain on manifold for both random and target-directed walks.

Critical strengths

The paper considers an interesting problem, namely how to train an autoencoder, so as to ensure that the output remains on the target manifold even when given as input data points that lie outside the manifold. This problem is relevant and somewhat overlooked by recent research.

Critical weaknesses
  1. It is unclear to me what is the interpolation scheme used for target-directed walks. Suppose it is the linear interpolation between the two endpoints, then it is unclear whether it will lead to trajectories that meaningfully remain on the manifold.

  2. From my perspective, the paper studies latent geometries in autoencoder models, but did not study and discuss previous works on it. Some interesting references are [1] [2] [3] [4], where people look into neural networks including autoencoders from the perspective of Riemannian geometry. These works essentially noted the issue that the model might not be well-behaved given inputs that lie outside the data manifold, and attempted to avoid traversing these regions using a Bayesian treatment and Riemannian geometry. To me, the paper is much closer to these works than the literature on identifiability that the paper cited, though I am less familiar with the latter literature.

[1] Metrics for Probabilistic Geometry, Tosi et al., UAI 2014

[2] The Riemannian Geometry of Deep Generative Models, Shao et al, 2018

[3] Latent Space Oddity: on the Curvature of Deep Generative Models, Arvantidis et al., ICLR 2018

[4] Pulling back information geometry, Arvanitidis et al., AISTATS 2022

Questions for authors

Please see Critical Weaknesses above.

  1. What interpolation scheme was used for target-directed walks? Whether and how did the authors ensure that this interpolation scheme results in interpolations that remain reasonable, even when the curvature of the ground truth manifold is highly non-trivial?

  2. Can the authors discuss the connections of the proposed approach to works on latent geometries, where the boundary value problems are essentially the same problem that the target-directed walks that the current work is trying to solve?

Minor typo: Section 5.2, Quantitative: Off-Manifod Distance, should be Manifold instead of Manifod

4 comments
AuthorsOpenReview ↗

Response to Reviewer UWq9

We thank the reviewer for finding the problem relevant and overlooked, and for the precise questions and pointers to the latent-geometry literature, which we now discuss explicitly (see the general response).

Q1. Interpolation scheme for targeted walks. The walk (Eq. 9) is defined for an arbitrary unit direction utu_t, and our guarantees do not depend on how it is chosen (e.g. Thm. 6 holds for any direction). Staying on the manifold is therefore a property of the learned projection, not of the interpolation scheme, where each step is mapped back by E∘DE \circ D before the next one is taken, so the walk never follows the straight segment between the endpoints. The direction rule only determines how efficiently, and whether, the target is reached. In Sec. 5 we use the Euclidean direction to the target, recomputed at every step, and we now state this rule in Sec. 5.1. The Euclidean direction is a simple heuristic, and Sec. 6 introduces a learned direction policy in its place. We have also reworded Sec. 6, which could be read as implying that Eq. 9 itself uses the Euclidean direction.

Q2. Connection to latent geometry and boundary value problems. We agree the targeted walk relates to the geodesic boundary-value problem (BVP), and now we say so. The difference is when the geometry enters. The BVP is solved post hoc on a trained model, finding a short curve under its pullback metric. Ours is a training-time objective that reshapes the model so that its decode-encode map is a retraction, after which a simple step-and-project rule is sufficient. Similarly, recent work analyses the vector field that iterated encoding and decoding induces in a trained model (Fumero et al., ICLR 2026). We instead train that field so that the data manifold is its attracting set. Our claim concerns the validity of every intermediate decode, not geodesic optimality, and the approaches complement each other, where a geodesic solver can be run on a model trained with Lproj\mathcal{L}_{\mathrm{proj}}.

We have also fixed the typo the reviewer pointed out.

We hope the clarified walk rule and the new discussion of latent geometry answer both questions. If they do, we would be grateful if the reviewer would reconsider the soundness rating and score, and we would welcome any further suggestions.

AuthorsOpenReview ↗

Response to Reviewer UWq9

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our reply answers your questions. We have since uploaded a revised PDF, with changes in blue. It states the walk rule in Sec. 5.1, discusses pullback-metric geodesics and their boundary-value problem in Sec. 7, and adds three results that bear on soundness. These are a corrected and stronger Prop. 2 with a numerical check (App. B), illustrations of how the objective keeps walks on the manifold (App. A.2, Figs. 7 to 9), and a controlled comparison at matched noise scale, where our walks stay on the manifold and reach the target twice as often as denoising at σ=1.0\sigma = 1.0 (App. C.4). If these address your concerns, we would be grateful if you would reconsider your soundness rating and score, and we are happy to clarify anything further.

I thank the authors for their responses. However, I agree with Reviewer iUWo that the proposed approach is relatively straightforward. As the proposed approach appears reasonably scalable, for a future version I encourage the authors to consider expanding the empirical validations to larger scale problems, e.g. for latent diffusion that the authors noted.

AuthorsOpenReview ↗

Response to Reviewer UWq9

Thank you for the reply and for the encouraging suggestion. We agree the objective is simple, and we see this as a strength, since it is easy to adopt. In this paper we aimed first to establish the finding where it can be measured exactly, that denoising learns a target off the manifold while a simple change of target learns the projection onto it. Since denoising is so widely used, we see this as a useful contribution to the community. It also shows that simple walks can stay on the manifold and reach their destination, as the walk guarantees in Sec. 4 and the experiments show, while denoising falls clearly short even in these simple settings. We see this gap at small scale as a promising signal for larger ones. Scaling this up is the natural next step, though not a direct one. Sec. 6 (Towards Large Scale) takes the first steps in this direction, and we agree latent diffusion is the right setting to build towards.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers for their careful reading. We are glad they found the problem relevant and overlooked by recent work (UWq9), the design simple and well-motivated (iUWo), and the approach intuitive and reasonable (9NYr). We summarize the main changes below. The revised PDF marks all changes in blue.

1. What the paper contributes. Existing work on latent geometry analyses a trained model post hoc, such as geodesics under a pullback metric (Tosi et al., 2014; Arvanitidis et al., 2018; Chen et al., 2018; Shao et al., 2018), shortest paths on a graph of samples (Tenenbaum et al., 2000; Chen et al., 2019; Kim et al., 2024), traversal directions in a pretrained generator (Choi et al., 2022; Song et al., 2023; Park et al., 2023), or the vector field induced by iterated encoding and decoding (Fumero et al., 2026). We ask a different question, can an autoencoder be trained so that its decode-encode map is itself a retraction onto the data manifold, so that simple step rules stay on the manifold? Our answer is a single change of regression target, from the perturbed code's own origin to its nearest code in a randomly drawn reference set (the batch, or a fresh subsample of a memory bank, Sec. 6), which may be the original code itself. The two views are complementary, post-hoc geodesic methods can be run on a model trained with Lproj\mathcal{L}_{\mathrm{proj}}.

2. Theory (iUWo, W1). We are particularly grateful to Reviewer iUWo, who pointed out that the original Prop. 2 overlooked that E∘DE \circ D observes only z+εz + \varepsilon. Correcting it gave a stronger result than the one it replaces, and we now see the new Prop. 2 (Sec. 4.1, proof in App. B) as a central contribution of the paper. The other results do not rely on the original Prop. 2, and their statements are unchanged. In the proof of Prop. 3 we fixed a typo in one step and updated the last step to refer to the new Prop. 2. Let z1,…,znz_1,\dots,z_n be codes drawn i.i.d. on M\mathcal{M}, zz one of them, and zˉ=z+σε\bar z = z+\sigma\varepsilon.

(a) The denoising optimum is the posterior mean (Alain & Bengio, 2014), a softmax-weighted average of nearby codes, and its loss cannot reach zero,

fden(zˉ)=∑j=1nwj(zˉ) zj,wj(zˉ)∝e−∥zˉ−zj∥2/2σ2,Lden⋆>0.f_{\mathrm{den}}(\bar z)=\sum_{j=1}^n w_j(\bar z)\,z_j,\qquad w_j(\bar z)\propto e^{-\|\bar z-z_j\|^2/2\sigma^2},\qquad \mathcal{L}_{\mathrm{den}}^\star>0 .

(b) On the unit circle, with zˉ=reiϕ\bar z = re^{i\phi}, this average stays off the manifold however many codes are available,

lim⁡n→∞d(fden(zˉ),M) ≥ σ22r+σ2 > 0.\lim_{n\to\infty} d\big(f_{\mathrm{den}}(\bar z),\mathcal{M}\big)\ \ge\ \frac{\sigma^2}{2r+\sigma^2}\ >\ 0 .

(c) The nearest-code target is determined by the input and the reference set, so nothing is averaged over unseen origins, and the Lproj\mathcal{L}_{\mathrm{proj}} optimum converges to the true projection as the reference set grows,

∥fproj(zˉ)−ΠM(zˉ)∥ ≤ ℓn−1+2δ = O ⁣(log⁡nn)(δ=1/n).\big\|f_{\mathrm{proj}}(\bar z)-\Pi_{\mathcal{M}}(\bar z)\big\|\ \le\ \ell_{n-1}+2\delta\ =\ O\!\left(\frac{\log n}{n}\right)\quad(\delta=1/n).

The argument does not depend on the autoencoder, and applies to any model trained to return perturbed points to the data when the aim is to land on the data manifold. App. B confirms all three parts numerically (see our reply to iUWo, W1).

AuthorsOpenReview ↗

General Response (2/2)

3. Related work (all reviewers). Sec. 7 now contrasts post-hoc navigation of a fixed model (pullback-metric geodesics, graph shortest paths, and traversal directions in pretrained GANs and diffusion models, including all references suggested by the reviewers) with training the model so that traversal is stable. It also discusses denoising-autoencoder Markov chains (Bengio et al., 2013) and interpolation regularizers (Berthelot et al., 2019).

4. Relevance (9NYr, iUWo). Autoencoders sit at the core of current generative models. Latent diffusion and flow models generate in the latent space of a pretrained autoencoder (Rombach et al., 2022; Peebles & Xie, 2023; Esser et al., 2024), and the image and video tokenizers behind them are autoencoders (Yu et al., 2024; Chen et al., 2025). Their samples land on latents the decoder never saw during reconstruction training. Recent work shows that reconstruction quality does not determine how well a latent space supports generation (Yao et al., 2025), and regularizes latent geometry directly (Kouzelis et al., 2025; Skorokhodov et al., 2025) or tunes the autoencoder jointly with the generator (Leng et al., 2025). Sparse autoencoders are also a standard tool for reading and steering language models (Huben et al., 2024). Closest to our setting, the latent dynamics of iterating an autoencoder's encode-decode map was an oral at ICLR 2026 (Fumero et al., 2026). We added this motivation to the introduction.

5. Scale (iUWo, 9NYr). The traversal methods cited as large-scale analyse or steer pretrained generators, so their scale is inherited from that pretraining. Our contribution is a training objective, which must be trained and is therefore validated where its effect can be isolated. Our central quantity, distance to the data manifold, can be measured exactly only when the manifold is known, which is why the synthetic experiment is our primary evidence and Colored MNIST shows transfer to images. Geometric analyses of autoencoders are commonly validated at this scale (Chen et al., 2018; Nazari et al., 2023). We agree natural-image validation is the important next step and state it as the main limitation.

Summary. The paper makes three contributions. (1) A training objective that turns the decode-encode map into a retraction onto the data manifold, using only the nearest code in a random reference set. (2) A theory that explains why this target, and not denoising, lands on the manifold. The denoising optimum stays off a curved manifold however much data is available, while the projection optimum converges to the projection at rate O(log⁡n/n)O(\log n/n). (3) Walks that stay bounded where reconstruction and cycle-consistency training drift (Table 1), with a learned direction and a memory bank that keep the method feasible as latent width and trajectory length grow (Sec. 6). We believe the revision resolves the reviewers' main concerns, and that the review process has made the paper substantially stronger, most of all through the new Prop. 2, and we would be grateful if the reviewers would take it into account in their final assessment.

AuthorsOpenReview ↗

Summary of the Discussion and Revision

We thank the reviewers and the AC. As our latest replies were posted near the end of the discussion, we summarize where things stand.

The main soundness concern is resolved. Reviewer iUWo identified an oversight in the original Prop. 2. The corrected Prop. 2 is stronger. The denoising optimum is a posterior mean that stays off a curved manifold however much data is available, while the nearest-code optimum converges to the projection at rate O(log⁡n/n)O(\log n/n). App. B verifies all three parts of it numerically, and the other results are unchanged.

New controlled evidence. With both objectives fine-tuned from the same model and only the target differing at each noise scale (10 seeds, App. C.4), our walks reach the target while staying on the true manifold twice as often as denoising at σ=1.0\sigma = 1.0 (80.8% against 40.5%), and stay closer to the manifold. This answers the attribution and trade-off questions (iUWo).

All other questions are answered in the revision. The walk rule and the relation to geodesic boundary-value problems (UWq9) are stated in Sec. 5.1 and Sec. 7. The relevance and the scale of evaluation (9NYr) are addressed with recent ICLR and ICML work evaluated at the same scale, and the motivation is added to the introduction. New illustrations of the theory are in App. A.2, and all changes are marked in blue.

On simplicity and scale (UWq9, iUWo). The main contribution is scientific. Denoising is widely regarded as the way to pull perturbed points back to the data, and we show, by theory (Prop. 2, Sec. 4) and a controlled experiment (App. C.4), that what it learns lies off the manifold, while a simple change of target learns the projection onto it. This finding does not depend on scale, and that denoising already falls clearly short in these controlled settings is, in our view, a promising signal for larger ones. Large-scale validation, like any large-scale training, requires substantial compute and engineering, and we see it as the natural next step, though not a direct one. Sec. 6 (Towards Large Scale) takes the first steps in this direction, with a memory bank so that the reference set grows without growing the batch, and a learned direction, as a starting point towards latent diffusion autoencoders.

The revision keeps the method and experiments of the original submission, and adds a sharper Prop. 2 and controlled evidence in their support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.