- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SmoothAlign, a term that regulates contrastive loss functions to take care of the semantic similarity of hard negatives.
Critical strengths
The paper obtains promising results in the retrieval task.
The method to compute the weight is simple and interesting.
Critical weaknesses
Weaknesses 1&2:
The paper lacks a discussion on works strictly related to the method itself. Indeed, the problem faced by this paper is a well-known problem in the literature, and there already exist several works that address it. Examples are (among many Others):
Jiang et al., “Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation,” ACL 2023
Gao et al., “SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger,” AAAI 2024
Wang et al., “Sample-Specific Debiasing for Better Image-Text Models,” Machine Learning for Healthcare Conference 2023
Byun et al., “MAFA: Managing False Negatives for Vision-Language Pre-training,” CVPR 2024
Nevertheless, the paper does not compare the proposed method with the relevant literature and this leads to two major concerns:
-
The paper novelty is limited, as the problem they name already exists in the literature. The contribution is then reframed as another, among the existing ones, method for better modeling the negatives in contrastive learning.
-
it is hard to understand whether the proposed methods is more effective than the existing ones, since authors do not run appropriate comparisons.
Weakness 3:
The notation should be lighter, and \hat{r}_ij appears eq.4 but it is not defined and never appears again in the paper. I imagine that \hat{r}_ij becomes \tilde{P}_ij later in the paper, but this is not stated. Authors should carefully check the definitions and the equations and considerably make it lighter.
Weakness 4:
The paper lacks more detailed analysis. While the method seems effective for retrieval, very few clues are provided on the learned latent space. How similar objects are mapped in the learned space? Are similar instances placed closer than in InfoNCE or GRAM latent spaces? Also, it is not clear why retrieval should improve. Intuitively, smoothalign should place similar object closer than contrastive methods based on infoNCE or GRAM, but this may make retrieval harder: retrieval is based on instance rankings, why placing all the similar instances close improve retrieval?
Weakness 5:
Why applying the smoothing to L_dam too? L_dam is computed from a multimodal encoder that reranks the pairs trhough attention mechanisms, why the authors apply smoothaling to it? I don't get the reason behind it.
Questions for authors
N/A