acceptodds
Under review as a conference paper at ICLR 2027

Gromov-Wasserstein Information Bottleneck for Distance-Preserving Representation Learning

Abstract

Preserving useful relational geometry is a fundamental challenge in representation learning for generative and predictive tasks. However, task losses and distributional constraints do not directly specify how an encoder transforms relationships among samples. In this work, we study geometry-aware representation learning from an information-theoretic perspective and develop the Gromov-Wasserstein Information Bottleneck (GWIB). Drawing on the information bottleneck principle, we derive an upper bound for an empirical kernelized information quantity involving the Gromov-Monge gap. The gap measures the encoder's excess structural cost relative to an optimal Gromov-Wasserstein coupling between the same input and encoded distributions. For counterfactual regression, we couple within-group geometry with cross-group fused-GW alignment through a shared transport plan. For autoencoders, the geometric term complements native reconstruction and prior-matching objectives. Experiments on IHDP and ACIC support the coupled causal construction through treatment-effect comparisons and ablations. Image experiments compare GWIB-augmented autoencoders with their base models on FashionMNIST and CIFAR-10, assessing generation, reconstruction, and representation utility.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Gromov-Wasserstein Distance, Information Bottleneck, Representation Learning, Optimal Transport, Counterfactual Regression, Geometric Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4SQeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for learning representations that can better preserve data geometry in the latent space. Based on the information bottleneck, a novel upper bound in terms of the Gromov-Monge gap is derived, which is further bounded by fused Gromov-Wasserstein alignment, and therefore the Gromov-Wasserstein Information Bottleneck (GWIB) is proposed. Experiments were primarily conducted on the CelebA dataset, where the proposed method achieves better generation quality compared with multiple baselines.

Critical strengths

(S1) This paper tackles an important challenge in representation learning: preserving meaningful relationships/geometry in the data space. While the previous information bottleneck focuses on learning compressed representations that are most useful for downstream tasks, the proposed new bottleneck in this paper fills the gap in data geometry preservation.

(S2) Experimental results are very promising. The ablation studies especially show that the proposed modules significantly improve the data generation quality.

Critical weaknesses

(W1) This paper appears to have been done in a rush and lacks proofreading, which makes it difficult to follow. For example, the abbreviation "WAE" is used in the introduction, but the full term is introduced formally much later in Section 4 (line 266); "OT" (line 280) is never explained (optimal transport?); and IMQ-MMD is never cited.

(W2) As a result of (W1), for the key Theorem 1 proposed, I could not find what exact distance metrics (dXd_X and dZd_Z) were used in the implementation. More importantly, what are the performance differences when choosing different distance metrics?

(W3) For evaluating the claim that the proposed method better preserves data geometry, while t-SNE plots are provided, additional quantitative results would strengthen the paper. Furthermore, geometric preservation should be directly confirmed first, before evaluating the data generation.

Questions for authors

(Q1) The authors are encouraged to clarify the points I mentioned in the weaknesses, which will greatly affect my score at the current stage.

Reviewer_4dkfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to combine representation learning according to the information bottleneck (IB) principle with a regularizer based on the Gromov-Wasserstein (GW) gap. This is motivated by showing that the GW gap is an upper bound for the mutual information between the data and representation (theorem 1). In addition, the theorem derives how the GW gap can be approximated from the empirical distribution of the training data. This theory is then applied to autoencoder training (sec. 4.1 and 5.1) and treatment effect estimation (sec. 4.2 and 5.2). Experiments for both applications demonstrate improvements in important metrics over existing algorithms.

Critical strengths

GW regularization is a very promising idea that has recently been advanced to the point where real-world applications come within reach. Its combination with the IB principle is a potentially very interesting addition to the toolbox.

Critical weaknesses

Unfortunately, the paper does not work in its present form – it attempts way more than can be reasonably achieved within the 9-page limit of an ICLR publication. Hence, it is very hard to read and leaves many critical gaps in its story.

  • The GW approach is still relatively new and has not entered mainstream machine learning theory and practice. Yet, sections 3.2 to 3.4 require deep familiarity with GW to be comprehensible at all. Typical participants of a conference like ICLR, which addresses a general machine learning audience, will not have this knowledge, and one cannot assume that they will first study several prior publications in order to understand what the paper is even talking about. The theory should instead be presented in a much more self-contained manner.
  • In particular, section 3.4 simply states that a certain formula involving the “fused GW distance” is a useful upper bound for the true GW gap. This section entirely lacks motivation, justification, derivations and a discussion of implications.
  • By penalizing the mutual information between X and Z, the information bottleneck objective encourages reliance of the model on highly specialized features and, in the worst case, clever Hans-type short-cut predictors. These will not generalize well and lack robustness under even slight distribution shifts. Modern representation learning has therefore moved towards self-supervised (i.e. task independent) learning on large datasets. While it is conceivable that GW regularization is a suitable strategy to reduce the overfitting tendency of IB, this combination is not an obvious choice and requires much more justification – why is it a good idea, exactly which benefits and shortcomings can be expected, and do these expectations actually manifest in practice? Figure 1 is meaningless in this respect.
  • The exact theory requires many approximations to become tractable, e.g. kernel density estimation of p(X,Z) and its marginals (eqs. 3 and 4), upper bounding of the mutual information by the GW gap (eq. 5), approximation of the GW discrepancy from finite data (eq. 6), heuristics for the GW matching problem (Alg. 1 and 2 in appendix E), upper bounding the GW gap again in terms of the fused GW distance (eqs. 8 and 9), restriction to additive Gaussian noise in the responses Y (eq. 2) etc. Each of these approximations comes with hyperparameters and introduces errors into the method. However, the paper only reports end-to-end accuracy, so that readers gain no insight into how big these errors are, which approximations are more or less critical, whether they remain stable under retraining and/or hyperparameter changes, how they may interact etc. To arrive at a trustworthy method, it is necessary to systematically establish the validity and accuracy of each approximation in isolation and in combination on suitable synthetic data with ground-truth.
  • The main paper says next to nothing about the numerical realization of the method, e.g. whether eq. (6) is evaluated on the entire dataset or on random batches, how various optimization objectives are actually minimized etc. While it is ok for the details to be relegated to the appendix, important high-level considerations, core decisions, critical caveats and possible limitations must be discussed in the main text.
  • The descriptions of the applications are so brief that the standard approaches for the respective tasks and the open issues to be addressed by IB+GW cannot be understood without detailed prior knowledge of these problems. In sections 4.1 and 5.1, better explanations and justifications are needed even more urgently because the quality of all models (including the new one) in figure 2 is clearly far from state-of-the-art – you must clarify why this experiment is nonetheless interesting. In case of the treatment effect estimation in sections 4.2 and 5.2, a shared encoder ϕ\phi for the treated and control groups is only a proxy for the actual goal of eliminating selection bias – it will, for example, fail in the presence of unobserved confounders for the treatment decisions. Moreover, it remains unclear why the IB+GW approach would be inherently better in finding a good shared encoder than alternatives.

Verdict: The paper is very hard to understand and lacks critical justification and systematic experimentation to obtain scientific insight rather than mere numbers. In my opinion, this is hard/impossible to fix within the 9 page limit of a conference publication. The authors should consider writing a journal paper with a much larger page limit, or at least expand and split the text into multiple conference papers.

Questions for authors

No questions

Reviewer_PB8DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GWIB, a regularizer that encourages an encoder to keep the distances between data points when it maps them to a latent space. The idea is motivated by the information bottleneck. On the theory side, Theorem 1 shows that a kernel-based estimate of the information between inputs and codes is bounded by the Gromov–Monge gap plus a constant. Proposition 1 then gives an upper bound on the gap that is easier to use in training. This bound combines the encoder's distance distortion with a term that measures how well the codes match a reference distribution. The method is used in two settings. In Wasserstein autoencoders, the reference distribution is the fixed Gaussian prior and the regularizer replaces the usual MMD penalty. In treatment-effect estimation (C-GWIB), the treated and control groups act as each other's reference; the method computes one soft matching between treated and control individuals and uses it for all the comparisons between the two groups. The paper reports better image generation than WAE-MMD, SWAE and VAE on CelebA 64×64, and the lowest treatment-effect errors among 13 baselines on IHDP and ACIC.

Critical strengths

The paper tackles a real and relevant problem. Reconstruction or prediction losses, together with matching the overall distribution of codes, leave the encoder free to distort how data points relate to each other in the latent space. This matters for generative models and for treatment-effect estimation, where pushing the treated and control groups too hard to look alike is a known issue. A single tool that addresses both would be of practical interest. The most interesting technical idea is the reference relaxation in Proposition 1. The Gromov–Monge gap needs the GW distance between the data and its own codes, which is awkward to use during training. The paper replaces it with the encoder's distance distortion plus an alignment to a reference distribution, using the triangle inequality for GW. This is simple and elegant, and it is what lets the same regularizer work with a fixed prior in autoencoders and with two groups that are learned together in treatment-effect estimation, through one shared matching between the groups. As far as I know, using this kind of regularizer for treatment-effect estimation is new. The math is correct as far as I could check. I went through the proof of Theorem 1 step by step, including a numerical check of each inequality, and found no errors; the proof of Proposition 1 also holds. The authors are also open about some limits of their results, for example that the constant in Theorem 1 cannot be ignored (after Eq. 7) and that the training objective of C-GWIB, which reuses one matching between the two groups for all comparisons, is not guaranteed to remain an upper bound on the gap (Appendix D.3). The CelebA experiments are carefully done and well documented. All methods use the same backbone, data splits and training length, checkpoints are chosen by a fixed rule, and Appendix J describes the metric code in detail, so the results are easy to check and reproduce. Under this protocol the gains are clear. FID drops from 60.7–67.7 for the baselines to 51.5, the differences across seeds are small, and reconstruction quality (PSNR, SSIM) improves slightly as well.

Critical weaknesses

My main concern is that the treatment-effect results don't agree with each other across the paper. The baseline numbers in Table 3 are often worse than anything those same baselines get in the paper's own sensitivity analysis over the regularization weight (Appendix G.2). For example, CFR-Wass has in-sample PEHE 1.529 on ACIC in Table 3, but about 1.22–1.32 for every weight in Fig. 6. On IHDP, Fig. 7 shows CFR-Wass at λ=1\lambda = 1 and ESCFR at λ=0.1\lambda = 0.1 reaching about 0.47 out-of-sample PEHE, which is better than the 0.699 reported for C-GWIB in Table 3. In the other direction, none of the β\beta settings in Table 7 reproduces the C-GWIB numbers in Table 3. Two further details add to this. On IHDP, C-GWIB's standard deviations (0.0002 to 0.0006) are much smaller than those of almost every neural baseline, often by a factor of 10 to 100, which is unusual for networks trained with different seeds. And the paper doesn't say which metric was used to pick hyperparameters, which matters here because the true counterfactual outcomes are available in these benchmarks. There may well be a simple explanation, but until these points are cleared up I can't rely on the treatment-effect claims, which are a large part of the paper's contribution. My second concern is that the novelty is smaller than the paper suggests, and the closest methods are not compared. The Gromov–Monge gap and its basic property (Definition 1 and Theorem 2) come from Uscidda et al. (2025). The autoencoder version is also very close to Gromov–Wasserstein Autoencoders (Nakagawa et al., ICLR 2023), which likewise combine a distance-preservation penalty with prior matching; that paper isn't cited. Neither method appears in the CelebA comparison, whose baselines (WAE-MMD, SWAE, VAE) don't try to preserve distances at all. The paper's own ablation hints at why this matters. In Table 2, the "FGW only" variant already reaches FID 55.2, against 51.5 for full GWIB, so most of the improvement over WAE-MMD (67.7) comes from ingredients that already exist. Finally, the link to the information bottleneck is weaker than the framing suggests. The Gromov–Monge gap is zero both for an encoder that keeps all distances (where the kernel information can come close to its maximum, log N) and for one that maps everything to a single point (where it is zero), so on its own it does not measure how much information is kept. In its current form, the constant in Theorem 1 can also be larger than log N, a value the kernel information never exceeds anyway (this follows from Eq. 4). For instance, the constant grows without limit when the input and latent spaces are large compared with the kernel bandwidth. In that case the bound adds little, although a tighter constant may be possible. In addition, the method trains on relaxations of the gap rather than on the gap itself. In the autoencoder, the loss penalizes the raw distance distortion R together with the alignment to the prior, and R also depends on the normalization constants chosen for the two spaces (Appendix J). For C-GWIB, Appendix D.3 notes that the trained objective is not guaranteed to remain an upper bound. The authors are open about some of these limits, and the method may well be useful, but it is better described as a geometric regularizer than as an information bottleneck.

Questions for authors
  1. How did you choose hyperparameters for C-GWIB and for each baseline in Table 3? Which validation metric did you use, and did you use the true counterfactual outcomes at any point?
  2. Why are many baseline numbers in Table 3 outside the ranges shown in Figs. 6–7? Which settings (λGWIB\lambda_{GWIB} and β\beta) gave the C-GWIB row in Table 3, and why isn't that row in Table 7?
  3. What changes between the five seeds, and why are C-GWIB's standard deviations on IHDP so much smaller than the baselines'? How many IHDP realizations and which ACIC setting(s) did you use?
  4. Could you compare with GWAE (Nakagawa et al., 2023) on CelebA, using the same setup and the same tuning budget? It is the closest exisiting method to GWIB and the current baselines don’t try to preserve distances, so this comparison would show whether GWIB’s gains come from its own new elements rather than from distance preservation in general.
  5. Were the Table 2 variants (MMD+R, FGW only) tuned on their own, or did they reuse GWIB's weights?
  6. How are the prior samples in the FβF_{\beta} term drawn (how many per batch, and are they resampled at every step)?
1 comment

I realize that my review became a bit long and AI heavy, so I would like to stress some main points:

The paper addresses an interesting problem in representation learning by proposing a regularization method that aims to preserve distances between data points in the input space and in the latent space. To achieve this, the authors introduce a reference to give a bound on the Gromov–Monge gap. Such a reference can be a fixed prior or a treated/controls group in the treatment effects use case.

The mathematical derivation of the upper bound is solid, and the proposed analogy with the information bottleneck to motivate the gap is an interesting idea.

However, I have some major concerns:

Novelty: The idea of combining distance preservation with prior matching is not new and is already present in GWAE (Nakagawa et al., ICLR 2023). This work should be properly cited and compared with in the experiments, as it represents the closest existing baseline to the proposed model.

Soundness of the results: In the treatment effects experiment, there seems to be a lack of consistency between the numbers in Table 3 and those in the sensitivity analysis (Figures 6-7 and Table 7). Moreover, it is not stated which hyperparameter selection metric is used, and the reported C-GWIB standard deviations are much lower than those of all the other neural network baselines (especially for IHDP). For these reasons, I find it difficult to trust the results without further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.