acceptodds
Under review as a conference paper at ICLR 2027

Encoder–Decoder Manifold Alignment for Idempotent Generation

Abstract

Recently, several learning paradigms have been introduced to enforce idempotency in generative models. The goal of this property is to ensure that repeated application of a model leaves samples unchanged once they lie on the target data manifold. In this work, we explore enforcing idempotency in encoder–decoder models. We argue that a key reason for the failure of generative models on some tasks, such as editing, is a geometric mismatch between the manifolds learned by the encoder and decoder. The encoder projects inputs onto one latent manifold, while the decoder implicitly learns to reconstruct data from a different manifold. This discrepancy prevents the model from learning truly idempotent mappings. To address this issue, we propose a new training framework that explicitly closes this gap by forcing the encoder and decoder to learn consistent representations of the same underlying data manifold. By aligning the geometry of these components, our method encourages stable projections. Empirically, we show that our approach achieves significantly lower idempotency error and consistently regenerates identical outputs under repeated application, compared to existing methods. We demonstrate the effectiveness of the proposed framework on both image generation and image editing tasks. Finally, we show that enforcing idempotency in this manner improves identity preservation and information stability, leading to more realistic and controllable generative editing models.

Primary area:
generative models
Keywords:
generative models, autoencoders, neural representations, editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5ANcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main contribution of the paper is Eq. (9). The method consists of adding an auxiliary loss, Lidem\mathcal{L}_{idem}, in addition to the reconstruction loss. Eq (9) is very related to Eq. (1) of IGN (Shocher et al., 2023); indeed, the first part is similar to the ideas of the left part of Eq (1). The new proposed loss work for reconstruction-based generative modeling (i.e., autoencoder-type approaches) and is forcing the model to:

  • re-encoder the generated data back to the same point that was used for generation i.e. E(D(z))E(D(z)) must map to z, where z=E(x)z=E(x).
  • the other part of the loss is doing the same, but enforcing basically the same loss, but sampling the latent codes z into a local neighbor defined by z_p = \sigma_z \eps + average_z, where the average of the code and std. dev are computed in the mini batch. and \eps is white noise.
  • z = E(x) is detached so as to avoid collapse. A mild analysis is performed on this topic.

The paper reports results that show:

  • small 3D geometric experiments
  • toy data like MNIST, CelebA, LFW; so tiny MNIST + faces data

The method shows improvements when re-encoding the data multiple times against baselines. The article compares with IGN only on Colored MNIST and dSprites.

Critical strengths
  • Idempotency is a nice property. It is not a new field (there are a few papers that exploit this in literature already but I agree with the paper that this part of gen. modeling is less explored.

  • All the experiments provided in the paper show that what is proposed solve the problem of idempotency. While this is a strength, it is also linked to the biggest limit: if I enforce idempotency, which benefits do I have on the downstream tasks of the gen. model?

  • One of the downstream tasks seems to be that editing works better on the proposed methods yet the experiments are not large scale

Critical weaknesses

Limited novelty and practical impact

Eq 9 is the main contribution of the work, but is of limited novelty. As reported in the related work, idempotency has already been explored. The paper claims the novelty is in the latent space, but this is too weak a contribution.

We enforce it in latent space, as a property of the encoder and decoder pair.

The other problem is impact: let's assume the novelty is sufficient. I believe a killer question for the paper is, "Beyond the small results reported on editing, which are the other impactful applications of the method?"

The problem of this paper is

Reconstruction stability after 30 encoder–decoder iterations.

Who is going to use the encoder-decoder after looping the AE 30x times? What is the application for this? does the improvement at x1 (1st decoding step improves?)

This is the main limitation of the contribution.

Claim partially supported

The encoder projects inputs onto one latent manifold, while the decoder implicitly learns to reconstruct data from a different manifold.

This claim is partially supported by Fig. 2 and only covers AE and VAE approaches. For example, is this true for DPPM? The claim needs to be toned down to say this works only for reconstruction-based generative models.

By aligning the geometry of these components, our method encourages stable projections.

This is partially supported by Fig. 3, but only on the 3D dataset, not on high-dimensional real data (images).

model-agnostic and can be added to any encoder–decoder objective

even to DDPM? (no VAE). From the experiments, it cannot work on DPPM unless the diffusion is latent in VAE+diffusion

On the mathy side, the article talks about idempotency, but if neural nets are assumed to be non-linear, it is better to talk about retraction.

Paper quality so and so

Fig 1 is not well described. I have to make an effort to read it (i.e., no variable notation in the figure). The figure is not very clear. Where are the repeated generations?

L 252 If Eq. 4, this is just one, but I have seen it in multiple parts.

Weak Experiments

poor image quality; the quality of the results is at the 2019 quality Tab. 3 is an easy win against the baseline.


overall assessment: The paper is overall interesting, it is good, but not good enough. My rating is around borderline as of now, but given that I have to choose an option, I'd choose weak reject. In general, for this kind of submission, the experimental result and impact have to be much stronger to pass. Note that if this were the first paper on idempotency, my rating would have been different.

Questions for authors
  • Can you do fig. 3 at large scale on high-dimensional real data (images)? Maybe start from CIFAR, and then go to ImageNet.

  • Why do we not build an encoder which is by construction e=D^-1(x)? A similar argument to normalizing flows. is this possible? trade-off with the proposed solutions?

  • This work [A] may allow you to develop better you theory/experiments, rethinking applying the AE multiple times as a vector field.

  • The method proposed works under the assumption

Assumption: If D_θ is L-Lipschitz,

Can we explain under which models these assumptions hold?

Counterargument for idempotency:

  • The encoder should take real data. This assumes we will use the encoder on real data.
  • With idempotency, you force it to work on generated data. You force the reconstruction to match the encoding of the real data. How does the approach motivate this tradeoff?

[A] Fumero et al. "Navigating the latent space dynamics of neural models." ICLR 2026

1 comment
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for their careful reading and detailed questions. We have addressed the main theoretical contribution and practical motivation in our general response. Here, we clarify the remaining theoretical points and provide additional experimental evidence.


Projection and Retraction

We thank the reviewer for raising this distinction; this was an astute observation. A retraction is any smooth X⟶XX \longrightarrow X transform PP that maps data onto a lower-dimensional manifold and satisfies P2=PP^2 = P. However, a reconstruction-optimal autoencoder is more specifically targeted at a metric projection (also P2=PP^2=P), where any XX is mapped to its closest point on the manifold under the ℓ2\ell_2 metric. While a retraction permits any path for a point XX to its image X^\hat{X}, the metric projection specifically requires it to be the closest point. Since autoencoders are trained to minimize ∥X−D(E(X))∥2\Vert{}X - D(E(X))\Vert{}^2, the unconstrained theoretical optimum is indeed a metric projection.

Even so, the reviewer's observation remains valid: with continuous but derivative-discontinuous activations such as ReLU, the ideal closest-point projection may itself be discontinuous. Consequently, the optimal feasible continuous network solution may behave as a retraction that approximates a metric projection almost everywhere.

However, as our empirical experiments demonstrate, actual neural network training learns neither a pure metric projection nor a strict retraction. Regardless, the theoretical and practical conclusions of our paper hold independently of this distinction. We will update the manuscript to note this nuance.


Figure 1 and Presentation

Figure 1 illustrates our main hypothesis about the alignment between the encoder and decoder manifolds. When the manifolds are aligned, the encoder and decoder learn consistent representations of the data, and the relationships between points are preserved. When they are misaligned, the decoder maps latent codes in a way that is inconsistent with how the encoder placed them, which changes the relative positions and distances between points.

The figure shows this mismatch within a single pass through the model. It does not show repeated generations directly, since the mismatch it depicts is the cause of the drift we measure when the model is applied to the same input multiple times. As our theoretical argument shows, the first reconstruction does not need to be exact for an autoencoder to be idempotent. Even when the model cannot preserve all information from the original input, once a sample is projected onto the learned output manifold, subsequent applications should leave it unchanged.

We will revise the figure and caption to make this distinction clearer, explain the illustrated distances, and correct the repeated "Eq. equation" references.


Weak Experiments

The experiments in Table 3 are controlled comparisons designed to isolate the effect of our regularizer. We use the same architectures, data, and training settings for the baseline and idempotent models. Their purpose is to evaluate the stability of the reconstruction rather than to achieve the best possible image quality.

Table 3 shows that our regularizer reduces the variance across reconstruction iterations for all metrics and datasets. On LFW, although the mean FID and LPIPS are slightly higher, our method achieves substantially lower variance and improves MSE. Figure 8 illustrates better preservation of identity and image details over 10, 20, 30, and 40 reconstruction iterations.

We additionally evaluated the Stable Diffusion VAE used by InstructPix2Pix at 512×512512\times512 resolution. The baseline and idempotent models have identical architectures and training settings.

RoundModelPSNR ↑SSIM ↑LPIPS ↓CLIP-I ↑FID ↓Drift ↓
1Pretrained24.850.7830.1180.97714.710.138
Baseline25.470.8010.1040.98711.890.158
Ours25.640.8070.1020.98711.610.033
3Pretrained22.080.7110.1960.95528.290.306
Baseline23.030.7440.1630.97421.550.277
Ours24.020.7700.1390.97917.780.082
5Pretrained19.660.6240.2920.91247.050.442
Baseline20.770.6710.2490.94637.130.379
Ours22.420.7250.1870.96925.570.124
10Pretrained15.440.4260.4840.765108.450.693
Baseline16.450.4730.4460.80799.710.599
Ours19.420.6190.3100.92046.690.211

Our method improves reconstruction quality and reduces latent drift across all evaluated rounds. After ten iterations, FID decreases from 99.71 to 46.69, while relative latent drift decreases from 0.599 to 0.211 compared with the fine-tuned baseline.

Reviewer_fm3wOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a latent-consistency regularizer for encoder–decoder models, including VAEs, to improve stability under repeated reconstruction and support downstream tasks such as image editing. It investigates the relationship between idempotency and encoder–decoder manifold alignment. The authors argue that geometric mismatch between the encoder and decoder can cause latent representations and reconstructed content to drift when the same model is repeatedly applied. Building on latent-cycle consistency, the proposed method augments the base training objective with a decode–encode consistency loss using stop-gradient targets and additional sampled latent codes. This encourages consistency both at encoded training examples and beyond them, with the aim of aligning the encoder and decoder representations. The evaluation includes synthetic manifold analyses, comparisons with alternative consistency objectives on Colored MNIST, and reconstruction and editing experiments across multiple datasets and architectures. The authors report reduced drift under repeated reconstruction, improved controlled attribute-editing success, and improved source-preservation metrics in natural-image editing relative to the evaluated baselines.

Critical strengths
  • The paper is well structured and clearly written, with an intuitive problem formulation that is easy to follow.
  • The proposed regularizer is simple and can be incorporated into existing encoder–decoder training objectives without requiring a new architecture.
  • The evaluation covers multiple datasets and architectures, with useful comparisons of consistency objectives and a sensitivity study of the regularization weight. These provide evidence supporting the method’s reconstruction-stability benefits.
Critical weaknesses
  • The evidence for the main claim, that better encoder–decoder alignment leads to better editing, is thin. The broadest comparison of objectives (Cycle, Latent (both), Latent (decoder), Ours without sampling) appears only for repeated reconstruction (Fig. 5). The editing experiments compare against far fewer methods: Fig. 7 has only VAE and VAE+IGN, and Table 2 has only Baseline vs. Idempotent. So the paper shows that the method reduces drift, but not that this reduced drift is what improves editing, or that other consistency objectives with similar stability wouldn’t edit just as well.

  • The novelty and the reasons behind it need clearer positioning. As Sec. 2.3 itself says, the latent cycle E(D(z)) ≈ z has been used for many years (BicycleGAN, MUNIT, InfoGAN/ALI, H3AE). Compared with the “Latent (both)” baseline, the contribution comes down to (a) the stop-gradient and (b) sampling codes around the encoded latents. For a paper like this, stating precisely what earlier latent-cycle methods were missing, and showing why each part matters, is more important than state-of-the-art numbers. Comparing several sampling distributions and analysing the differences would be one way to provide this explanation.

  • Suggestions for the rebuttal: 1) Add Cycle, Latent (both), Latent (decoder) and Ours (no sampling) to Fig. 7, and if compute allows, to Table 2. Fig. 7 is cheap to extend; one or two extra rows in Table 2 would already help. 2) Explain why alignment helps editing in particular, and why the other objectives don’t help as much. If either of these can be added in the rebuttal, I would comfortably raise my score.

Questions for authors
  • Eq. 10 samples from a Gaussian fitted to the batch’s latent mean and std, while Algorithm 1 samples around each example’s posterior N(μ, s²σ²). Which one was used in each experiment, and what value of κ? Why not use the prior N(0, I)?
  • What happens if you combine the data-space cycle loss with your latent loss? Do they conflict, or do they complement each other?
Reviewer_HAjtOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper blames editing failures in encoder–decoder models on a geometric mismatch: the manifold the encoder projects onto differs from the one the decoder reconstructs from. It casts idempotency as a necessary condition for f to be a projection, and shows reconstruction loss does not enforce the latent cycle. The fix is a latent-consistency loss. A degeneracy analysis shows the stop-gradient is needed, else the loss can be cheated by shrinking the latent. Diagnostics: round-trip Jacobian singular values on six surfaces, and kNN/neighbor/contraction under repeated reconstruction. Tried on a VAE, VQ-VAE-2, DCGAN vs IGN, and the InstructPix2Pix VAE.

Critical strengths

Simple, model-agnostic method with careful analysis. The loss needs no extra networks, and the stop-gradient analysis (removing the degenerate latent-scaling direction) shows real understanding.

Critical weaknesses
  • Benchmark-dependent, mixed gains. Editing improves a lot only on BM-Bench, marginal on I2EBench/MagicBrush; reconstruction basically unchanged; VQ-VAE-2 gets worse on FID/LPIPS. So the "broadly improved" claim is only partly supported.
  • Incremental novelty, un-isolated causal claim. The latent cycle is well known (ALI/BiGAN, cycle consistency, H3AE); novelty is just the stop-gradient + neighborhood formulation. And "lower idempotency error → better editing" is only correlational: InstructPix2Pix is retrained with the new VAE, so idempotency is never isolated as the cause.
Questions for authors

Can you isolate idempotency's causal effect on editing (e.g. vary λ, show a monotone link between idempotency error and editing success at matched reconstruction)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.