- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the influence of encoders' geometric properties on trimodal contrastive learning and identifies encoder Jacobian conditioning as a key factor. Mathematical analysis is provided. Based on this finding, the paper proposes geometry-preserving encoders (GPEs). GPEs introduce simple modifications like LeakyReLU activations and residual paths, yet yield improvements in most retrieval tasks and some downstream tasks. Experiments are conducted on trimodal contrastive learning tasks using a synthetic benchmark and healthcare datasets.
Critical strengths
- The paper highlights the importance of encoders' geometry for trimodal contrastive learning, and links encoder trainability to Jacobian conditioning and dynamical isometry. The authors use theoretical analysis and empirical experiments to substantiate their claims.
- The proposed practical interventions based on residual paths and LeakyReLU activations are easy to integrate into common encoders and simple to implement.
Critical weaknesses
- The title and abstract indicate that the paper contributes to multimodal contrastive learning, but the evidence for the generalizability of the proposed method is limited. Although the proposed method and theoretical analysis in this paper don't depend on healthcare-specific properties, all real-world experiments are carried out on healthcare datasets, despite the availability of non-healthcare trimodal datasets like Symile-M3, CMU-MOSI, MOSEI, etc.. The applicability to settings beyond healthcare remains unclear, and the claims about the method’s generalizability should be qualified to reflect the empirical scope.
Questions for authors
- The motivation of this work focuses on proposing a different perspective for multimodal representation learning, as much of the previous work focuses on objective design and representation geometry. While cross-objective experiments show that several multimodal contrastive objectives benefit from GPEs, could the authors also discuss whether GPEs can still provide additional gains when combined with methods that jointly model embedding norms and directions? In particular, to what extent do these approaches alleviate overlapping optimization difficulties, and are their benefits expected to be complementary?
With the rebuttal phase coming to an end, we thank the reviewer again for their time and feedback. Overall, the new results support our central findings.
We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.