- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The main contribution of the paper is Eq. (9). The method consists of adding an auxiliary loss, , in addition to the reconstruction loss. Eq (9) is very related to Eq. (1) of IGN (Shocher et al., 2023); indeed, the first part is similar to the ideas of the left part of Eq (1). The new proposed loss work for reconstruction-based generative modeling (i.e., autoencoder-type approaches) and is forcing the model to:
- re-encoder the generated data back to the same point that was used for generation i.e. must map to z, where .
- the other part of the loss is doing the same, but enforcing basically the same loss, but sampling the latent codes z into a local neighbor defined by z_p = \sigma_z \eps + average_z, where the average of the code and std. dev are computed in the mini batch. and \eps is white noise.
- z = E(x) is detached so as to avoid collapse. A mild analysis is performed on this topic.
The paper reports results that show:
- small 3D geometric experiments
- toy data like MNIST, CelebA, LFW; so tiny MNIST + faces data
The method shows improvements when re-encoding the data multiple times against baselines. The article compares with IGN only on Colored MNIST and dSprites.
Critical strengths
-
Idempotency is a nice property. It is not a new field (there are a few papers that exploit this in literature already but I agree with the paper that this part of gen. modeling is less explored.
-
All the experiments provided in the paper show that what is proposed solve the problem of idempotency. While this is a strength, it is also linked to the biggest limit: if I enforce idempotency, which benefits do I have on the downstream tasks of the gen. model?
-
One of the downstream tasks seems to be that editing works better on the proposed methods yet the experiments are not large scale
Critical weaknesses
Limited novelty and practical impact
Eq 9 is the main contribution of the work, but is of limited novelty. As reported in the related work, idempotency has already been explored. The paper claims the novelty is in the latent space, but this is too weak a contribution.
We enforce it in latent space, as a property of the encoder and decoder pair.
The other problem is impact: let's assume the novelty is sufficient. I believe a killer question for the paper is, "Beyond the small results reported on editing, which are the other impactful applications of the method?"
The problem of this paper is
Reconstruction stability after 30 encoder–decoder iterations.
Who is going to use the encoder-decoder after looping the AE 30x times? What is the application for this? does the improvement at x1 (1st decoding step improves?)
This is the main limitation of the contribution.
Claim partially supported
The encoder projects inputs onto one latent manifold, while the decoder implicitly learns to reconstruct data from a different manifold.
This claim is partially supported by Fig. 2 and only covers AE and VAE approaches. For example, is this true for DPPM? The claim needs to be toned down to say this works only for reconstruction-based generative models.
By aligning the geometry of these components, our method encourages stable projections.
This is partially supported by Fig. 3, but only on the 3D dataset, not on high-dimensional real data (images).
model-agnostic and can be added to any encoder–decoder objective
even to DDPM? (no VAE). From the experiments, it cannot work on DPPM unless the diffusion is latent in VAE+diffusion
On the mathy side, the article talks about idempotency, but if neural nets are assumed to be non-linear, it is better to talk about retraction.
Paper quality so and so
Fig 1 is not well described. I have to make an effort to read it (i.e., no variable notation in the figure). The figure is not very clear. Where are the repeated generations?
L 252 If Eq. 4, this is just one, but I have seen it in multiple parts.
Weak Experiments
poor image quality; the quality of the results is at the 2019 quality Tab. 3 is an easy win against the baseline.
overall assessment: The paper is overall interesting, it is good, but not good enough. My rating is around borderline as of now, but given that I have to choose an option, I'd choose weak reject. In general, for this kind of submission, the experimental result and impact have to be much stronger to pass. Note that if this were the first paper on idempotency, my rating would have been different.
Questions for authors
-
Can you do fig. 3 at large scale on high-dimensional real data (images)? Maybe start from CIFAR, and then go to ImageNet.
-
Why do we not build an encoder which is by construction e=D^-1(x)? A similar argument to normalizing flows. is this possible? trade-off with the proposed solutions?
-
This work [A] may allow you to develop better you theory/experiments, rethinking applying the AE multiple times as a vector field.
-
The method proposed works under the assumption
Assumption: If D_θ is L-Lipschitz,
Can we explain under which models these assumptions hold?
Counterargument for idempotency:
- The encoder should take real data. This assumes we will use the encoder on real data.
- With idempotency, you force it to work on generated data. You force the reconstruction to match the encoding of the real data. How does the approach motivate this tradeoff?
[A] Fumero et al. "Navigating the latent space dynamics of neural models." ICLR 2026
Response (1/2)
We thank the reviewer for their careful reading and detailed questions. We have addressed the main theoretical contribution and practical motivation in our general response. Here, we clarify the remaining theoretical points and provide additional experimental evidence.
Projection and Retraction
We thank the reviewer for raising this distinction; this was an astute observation. A retraction is any smooth X⟶X transform P that maps data onto a lower-dimensional manifold and satisfies P2=P. However, a reconstruction-optimal autoencoder is more specifically targeted at a metric projection (also P2=P), where any X is mapped to its closest point on the manifold under the ℓ2 metric. While a retraction permits any path for a point X to its image X^, the metric projection specifically requires it to be the closest point. Since autoencoders are trained to minimize ∥X−D(E(X))∥2, the unconstrained theoretical optimum is indeed a metric projection.
Even so, the reviewer's observation remains valid: with continuous but derivative-discontinuous activations such as ReLU, the ideal closest-point projection may itself be discontinuous. Consequently, the optimal feasible continuous network solution may behave as a retraction that approximates a metric projection almost everywhere.
However, as our empirical experiments demonstrate, actual neural network training learns neither a pure metric projection nor a strict retraction. Regardless, the theoretical and practical conclusions of our paper hold independently of this distinction. We will update the manuscript to note this nuance.
Figure 1 and Presentation
Figure 1 illustrates our main hypothesis about the alignment between the encoder and decoder manifolds. When the manifolds are aligned, the encoder and decoder learn consistent representations of the data, and the relationships between points are preserved. When they are misaligned, the decoder maps latent codes in a way that is inconsistent with how the encoder placed them, which changes the relative positions and distances between points.
The figure shows this mismatch within a single pass through the model. It does not show repeated generations directly, since the mismatch it depicts is the cause of the drift we measure when the model is applied to the same input multiple times. As our theoretical argument shows, the first reconstruction does not need to be exact for an autoencoder to be idempotent. Even when the model cannot preserve all information from the original input, once a sample is projected onto the learned output manifold, subsequent applications should leave it unchanged.
We will revise the figure and caption to make this distinction clearer, explain the illustrated distances, and correct the repeated "Eq. equation" references.
Weak Experiments
The experiments in Table 3 are controlled comparisons designed to isolate the effect of our regularizer. We use the same architectures, data, and training settings for the baseline and idempotent models. Their purpose is to evaluate the stability of the reconstruction rather than to achieve the best possible image quality.
Table 3 shows that our regularizer reduces the variance across reconstruction iterations for all metrics and datasets. On LFW, although the mean FID and LPIPS are slightly higher, our method achieves substantially lower variance and improves MSE. Figure 8 illustrates better preservation of identity and image details over 10, 20, 30, and 40 reconstruction iterations.
We additionally evaluated the Stable Diffusion VAE used by InstructPix2Pix at 512×512 resolution. The baseline and idempotent models have identical architectures and training settings.
Our method improves reconstruction quality and reduces latent drift across all evaluated rounds. After ten iterations, FID decreases from 99.71 to 46.69, while relative latent drift decreases from 0.599 to 0.211 compared with the fine-tuned baseline.