acceptodds
Under review as a conference paper at ICLR 2027

Exact and Inexact Marginal Penalties: When Does the Wasserstein Autoencoder Recover the Wasserstein Distance?

Abstract

Primal optimal transport (OT) minimizes a transport cost over couplings with prescribed marginals. Parameterizing a coupling by a conditional distribution enforces one marginal by construction and leaves the other as an explicit constraint. The Wasserstein autoencoder relaxes this remaining constraint into a divergence penalty. We characterize when the resulting penalized objective recovers the OT cost at a finite coefficient . On bounded domains and under an active constraint, -divergences with differentiable at , as well as squared maximum mean discrepancy, strictly underestimate the cost for every finite coefficient. For satisfying a generalized Pinsker inequality, the optimal-value gap is as . In contrast, the square roots of such -divergences and the 1-Wasserstein distance recover the OT value and global minimizers above explicit finite thresholds, and for square roots of the Kullback-Leibler and chi-square divergences exactness extends to unbounded priors. The dichotomy follows from comparing reconstruction descent with penalty growth in the Wasserstein space of joint distributions. Uniform exactness over a decoder class further yields statistical guarantees for learning the pushforward model. Simulations with computable ground truth illustrate the dichotomy, while CelebA experiments compare latent matching and local image redundancy across penalty classes.

Primary area:
optimization
Keywords:
Optimal transport, Exact penalty methods, Infinite-dimensional optimization, Wasserstein autoencoders, f-divergences

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_MctyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper deals with the theoretical question whether the OT solution of an encoder arising in Wasserstein autoencoders is consistent with a regularized version of the problem. In the regularized version, one minimizes a transport cost but relaxes the marginal constraint to a soft penalty. The paper studies for which "discrepancies" such regularized version admits the same optimal encoder as the original OT problem. Somewhat surprisingly, it does not work for f-divergences but for its square root. The paper validates its theory on analytic low dimensional example and CelebA.

Critical strengths
  1. The proofs and arguments seem correct as far as I checked.

  2. The placement wrt prior work is rigorous, table S7 is good to read.

  3. It is interesting to see that this argument works for the square root of such f divergences.

Critical weaknesses
  1. I do not think the Wasserstein autoencoder formulation is SOTA or used much anymore. Most of the literature now uses conditional normalizing flows/diffusion models for which this theory is largely irrelevant since they fix the marginals automatically. The case of generic autoencoders is not covered since the paper explicitly studies the OT formulation. Generally, it is not known whether minimizing a Wasserstein inspired loss term (either via the dual or primal formulation) really leads to recovery of the actual transports for a learned NN. Most strongly, my recommendation would change if the authors could argue that it is relevant to more papers than I currently believe. I currently do not think it is impactful/significant enough to be published at ICLR.

  2. I find the use of GANs to approximate the square-root of the f-divergence needs some justification, in particular since for instance Wasserstein GANs do not approximate the Wasserstein distance [1]. It seems to be done even in the low dimensional examples and therefore one could also resort to binning?

  3. The writing is very hard to follow, many sentences are hard to parse for me personally and made it quite hard to read. Examples include: " The proofs proceed by perturbing a feasible,nearly optimal joint distribution ... mixture segment,where .. is the joint distribution induced by the quantile encoder that restricts the prior to the θ-fraction (θ∈(0,1)) of latent codes minimizing the reconstruction cost c(x,g(z)),yielding a latent marginal π1♯ρ with density at most 1/θ with respect to PZ "

"Fixed-decoder simulations compare reconstruction estimates with computable OT ground truth and illustrate both sides of the finite-coefficient dichotomy. "

  1. Some of the mathematical proofs are standard, e.g., Proposition 3.9 but are not cited or deemed as such. I would appreciate if it was clearer whether standard results were marked as such (see [2])

[1] Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance), Stanczuk et al, 2021

[2] On Choosing and Bounding Probability Metrics, Gibbs and Su, 2002.

Questions for authors

see weaknesses

Reviewer_B5CsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks when the marginal penalty in a WAE recovers the underlying OT objective at a finite coefficient λ\lambda. It shows that smooth ff-divergences and squared MMD are inexact for every finite λ\lambda, while square roots of suitable ff-divergences and the W1W_1 penalty are exact above explicit thresholds. I think the population results are correct. My concerns are elsewhere. The methodological step forward is modest, the claimed statistical improvement isn’t really supported, and I have doubts about parts of the experiments.

Critical strengths
  • Separating formulation bias from network approximation, divergence estimation and optimization error is useful.
  • The quantile-encoder construction is a nice technical contribution.
  • The strict inexactness result and the 1/λ1/\lambda bias bounds explain clearly why a finite penalty can't remove the bias.
Critical weaknesses

Methodological novelty The positive results combine standard exact-penalty arguments with familiar transport and Pinsker-type inequalities, and Wasserstein-penalized autoencoder identities already appear in Patrini et al. (2020). The paper mentions this, but to me the inexactness construction is the main new contribution. I would like to see a detailed comparison with Patrini et al.

The generalization bound doesn't need exactness Take a coupling between PXP_X and P^Xn\hat P_X^n and move the data coordinate along it while keeping the latent coordinate fixed. The aggregate posterior, and hence the marginal penalty, stays the same. If X\mathcal X has diameter KK, this gives ∣Vλ(P,g)−Vλ(P′,g)∣≤pKp−1W1(P,P′)|V_\lambda(P,g)-V_\lambda(P',g)| \le pK^{p-1}W_1(P,P') for any marginal penalty and any λ\lambda, uniformly over decoders taking values in X\mathcal X. So Proposition 4.2 is correct, but its conclusion doesn’t depend on exactness. Exactness is only needed to identify the penalized objective with the OT cost. Please correct me if I’ve missed something.

Statistical improvement Proposition 4.3 bounds distribution-learning error by a decoder approximation term plus an empirical Wasserstein error. The approximation/misspecification term can remain nonzero, and the stated convergence rate follows under model specification. The paper mentions expressive neural decoders but doesn't analyse encoder approximation, decoder approximation and estimation error together. Recent works, such as Chakraborty and Bartlett (2024), by contrast, explicitly control neural approximation error and balance it against statistical error through network-size choices, under their assumptions. Comparing the faster statistical term here with their overall rate is misleading. The present result concerns an idealized estimator and does not establish a stronger learning guarantee for neural WAEs. The discussion of Husain et al. (2019) also needs to be more detailed. Their Theorem 13 allows any positive coefficient, with extra conditions needed to read it as a WAE guarantee. Their exactness threshold shouldn't be mixed up with their generalization theorem.

Practical benefit The CelebA coefficient is selected using square-root JS and then reused across penalties. Since taking a square root changes the effective regularization strength, each baseline should receive its own coefficient sweep. The improvements are primarily in latent matching under the logit-normal prior. FID is about the same, and the Gaussian-prior results are mixed.

Table S.6 also seems unnecessarily long.

Questions for authors
  1. How are the reverse-divergence experiments defined, given that their population penalties are infinite for the stated encoder architecture?
  2. Is the auxiliary variable in Equation (7) clipped or otherwise stabilized? What effect does this have on exactness?
  3. What statistical improvement remains when neural approximation errors are included in the comparison?
  4. Do the CelebA advantages persist when each penalty receives an independently tuned coefficient?
Reviewer_1H7zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To my understanding, this paper studies whether the soft penalties used in Wasserstein Autoencoders to match the aggregated posterior to the prior can recover the original constrained optimal transport objective. The authors show that commonly used penalties, such as KL, JS, and squared MMD, cannot exactly recover the Wasserstein objective for any finite penalty coefficient under certain assumptions. On the other hand, using the square root of these divergences, or the Wasserstein-1 distance, allows exact recovery above a finite threshold.

The paper also provides theoretical bounds on the approximation error of the inexact penalties, and extends the analysis to decoder learning, deriving some statistical generalization guarantees. Finally, the authors provide synthetic experiments and experiments on CelebA to illustrate the theoretical results and compare the different penalty choices.

Critical strengths

I think the main strength of the paper is its theoretical contribution. In particular, the authors provide a characterization of which marginal penalties can exactly recover the original optimal transport objective for a finite penalty coefficient, and which cannot. To my understanding, the most interesting result is that commonly used divergences such as KL and JS are never exact under the stated assumptions, regardless of the penalty coefficient, while taking their square root allows exact recovery above a certain threshold. The authors also quantify the approximation error of the inexact penalties, showing how it decreases as the penalty coefficient increases. While the general intuition is related to existing results in exact penalty theory, I think applying it to the WAE formulation and obtaining explicit guarantees is an interesting contribution. The synthetic experiments also help illustrate these theoretical results by comparing the different penalties against known optimal transport costs.

Critical weaknesses

My main concern is the significance of the contribution, especially in the context of modern generative modeling. While I find the theoretical results interesting, the underlying intuition seems closely related to existing results in exact penalty theory, and I am not fully convinced that the extensions presented here represent a sufficiently substantial contribution. More importantly, it is not clear to me how relevant these results are in practice. The theoretical guarantees rely on rather restrictive assumptions, such as unrestricted encoders, exact divergence estimation, and global optimization. These assumptions are quite far from the actual setting in which generative models are trained. Furthermore, the exactness thresholds depend on quantities such as the Lipschitz constant of the decoder, which are generally difficult to estimate or control for neural networks. I would have liked to see a more convincing discussion of why exact recovery of the Wasserstein objective should be important in practical generative modeling.

I also find the empirical evaluation rather limited. The synthetic experiments illustrate the theoretical findings, but are restricted to very low-dimensional settings. The CelebA experiments are more relevant, but the improvements are mostly limited to latent distribution matching, while the generation quality, measured by FID, remains essentially unchanged. Moreover, the results are mixed depending on the choice of prior. It is therefore difficult to conclude that the proposed penalties provide a meaningful advantage over standard alternatives. I also wonder whether the differences would remain if the penalty coefficients were independently tuned for each method.

Finally, I think the presentation is a major weakness of the paper. The writing is unnecessarily complicated, and I found the paper quite difficult to follow, even at a conceptual level. The introduction introduces a large amount of mathematical notation and technical details without first providing a clear intuition of the problem and why it matters. The main idea behind the results is relatively simple, but it gets lost in the formalism. I also found it difficult to understand what the actual novelty is compared to previous work, and what the reader should take away from the theoretical analysis beyond the exactness characterization itself. Overall, I think the paper would benefit substantially from a clearer structure, a more accessible explanation of the main results, and a stronger motivation of their relevance to machine learning.

Questions for authors
  1. In Section 5.1, the Wasserstein-1 penalty often fails to pass the equivalence test, despite being theoretically exact. The authors attribute this to instability in the Lipschitz critic, but is there any further evidence supporting this explanation? Could other factors contribute to this behavior?

  2. The square-root divergences are implemented using an additional optimization variable, as described in Equation 7. How does this affect training stability and computational cost compared to the standard divergence penalties? I think this would be an interesting aspect to investigate, especially since the proposed alternatives should ideally be practical to use.

  3. I found the results in Figure 3 a bit difficult to interpret. While the standard JS penalty produces more concentrated latent representations, it also achieves lower intra-cluster MS-SSIM than its square-root counterpart, suggesting less similarity between nearby decoded samples. Could the authors explain this behavior and how it relates to the claimed improvements in latent matching?

1 comment

Thank you for the detailed responses and clarifications. I appreciate the additional experiments, particularly the extended ablations on penalty coefficients, which help clarify the practical differences between the proposed penalties. The explanations addressing my specific questions were also helpful. That said, my main concerns remain. While the theoretical results are interesting, the empirical advantages appear limited. In particular, the additional experiments suggest that both penalties achieve comparable performance when properly tuned, with the main benefit of the square-root penalties being reduced sensitivity to coefficient selection. While this is a useful property, I am still not convinced that it represents a sufficiently significant practical contribution. I therefore appreciate the authors' efforts but will maintain my original rating.

Reviewer_Rq5TOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when marginal penalties in Wasserstein autoencoders recover the original optimal transport objective for a finite penalty parameter. The main contribution is a characterization of exact and inexact penalties. The authors establish that differentiable f-divergences and squared MMD are inexact under an active marginal constraint (Prop. 3.4) and also provide quantitative bounds on the resulting gap (Thm 3.5). They additionally establish finite exactness thresholds for square-root divergences and Wasserstein penalties. The paper further derives statistical guarantees and presents numerical experiments.

Critical strengths
  • Their question is well-motivated question: how penalized Wasserstein autoencoders relates to optimal transport.
  • Prop 3.4 and Thm 3.5 clarifies the distinction between finite exactness and asymptotic convergence and quantifying the approximation error.
  • The mathematical analysis appears rigorous, however I didn’t check all proofs
  • The paper is generally easy to follow, and the numerical experiments help illustrate the theoretical findings.
Critical weaknesses

Weaknesses

  • Limited novelty.* Thm 3.7 and Cor 3.8 largely follow from existing exact-penalty theory and transportation inequalities. Exactness for Wasserstein penalties was already established in Patrini et al 2020. The main novelty seems to be the inexactness results and their quantitative bounds.
  • Prop 4.2 does not require exactness.* The generalization bound follows from a coupling argument preserving the latent marginal and thus the penalty. It therefore appears to hold for arbitrary marginal penalties, independently of exactness.
  • Limited practical relevance.* The results concern unrestricted encoders and global minimizers of population objectives. Their implications for parametrized neural networks and approximate optimization remain unclear.
Questions for authors
  1. What is new in Thm 3.7 and Cor 3.8 compared to Patrini et al and classical exact-penalty theory?
  2. Is exactness necessary for Prop 4.2 or does the bound hold for arbitrary marginal penalties?
  3. How relevant are the exactness results for neural encoders and approximate optimization?
  4. Do the experimental advantages persist when the penalty parameter is tuned separately for each method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.