- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for learning representations that can better preserve data geometry in the latent space. Based on the information bottleneck, a novel upper bound in terms of the Gromov-Monge gap is derived, which is further bounded by fused Gromov-Wasserstein alignment, and therefore the Gromov-Wasserstein Information Bottleneck (GWIB) is proposed. Experiments were primarily conducted on the CelebA dataset, where the proposed method achieves better generation quality compared with multiple baselines.
Critical strengths
(S1) This paper tackles an important challenge in representation learning: preserving meaningful relationships/geometry in the data space. While the previous information bottleneck focuses on learning compressed representations that are most useful for downstream tasks, the proposed new bottleneck in this paper fills the gap in data geometry preservation.
(S2) Experimental results are very promising. The ablation studies especially show that the proposed modules significantly improve the data generation quality.
Critical weaknesses
(W1) This paper appears to have been done in a rush and lacks proofreading, which makes it difficult to follow. For example, the abbreviation "WAE" is used in the introduction, but the full term is introduced formally much later in Section 4 (line 266); "OT" (line 280) is never explained (optimal transport?); and IMQ-MMD is never cited.
(W2) As a result of (W1), for the key Theorem 1 proposed, I could not find what exact distance metrics ( and ) were used in the implementation. More importantly, what are the performance differences when choosing different distance metrics?
(W3) For evaluating the claim that the proposed method better preserves data geometry, while t-SNE plots are provided, additional quantitative results would strengthen the paper. Furthermore, geometric preservation should be directly confirmed first, before evaluating the data generation.
Questions for authors
(Q1) The authors are encouraged to clarify the points I mentioned in the weaknesses, which will greatly affect my score at the current stage.
I realize that my review became a bit long and AI heavy, so I would like to stress some main points:
The paper addresses an interesting problem in representation learning by proposing a regularization method that aims to preserve distances between data points in the input space and in the latent space. To achieve this, the authors introduce a reference to give a bound on the Gromov–Monge gap. Such a reference can be a fixed prior or a treated/controls group in the treatment effects use case.
The mathematical derivation of the upper bound is solid, and the proposed analogy with the information bottleneck to motivate the gap is an interesting idea.
However, I have some major concerns:
Novelty: The idea of combining distance preservation with prior matching is not new and is already present in GWAE (Nakagawa et al., ICLR 2023). This work should be properly cited and compared with in the experiments, as it represents the closest existing baseline to the proposed model.
Soundness of the results: In the treatment effects experiment, there seems to be a lack of consistency between the numbers in Table 3 and those in the sensitivity analysis (Figures 6-7 and Table 7). Moreover, it is not stated which hyperparameter selection metric is used, and the reported C-GWIB standard deviations are much lower than those of all the other neural network baselines (especially for IHDP). For these reasons, I find it difficult to trust the results without further clarification.