- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper aims to propose a new metric to curate image-text pair datasets from noisy-webly collected image-text pairs. If I understand correctly, they attempt to incorporate the sensitivity to the caption-perturbation to filter a well-aligned image-text pairs. Their empirical results suggest that the proposed approach sometimes outperforms the baseline model in vision-language alignment benchmark dataset. However, overall, they did not present their approach well, and it was hard to understand the reason why their proposed approach can outperform the baseline methods.
Critical strengths
- The idea of using the sensitivity value to get well-aligned image-text pairs might be novel. However, the intuition behind why the method can outperform the existing approach was not very clear.
Critical weaknesses
- The empirical effectiveness of the proposed method is limited. Their proposed approach does not much improve the performance in some cases.
- The baseline is limited. They list a lot of related work, but only a few naive baselines are used for comparison.
- Other than the quality of image-text pair alignment can affect the pre-training of CLIP-like model. As shown in [1, 2], ensuring the diversity of the data is critical in image-text pair pre-training. They are missing this kind of baseline and miss the detailed discussion.
- Poor presentation quality. Especially their writing is poor. It was hard to understand their main claim throughout the paper. Also, the figure needs improvement to better help readers to understand the paper.
- There can be some bug or mistakes in their experiments. In Table 1 and 2, imagenet zero-shot performance is around 7%. This value is too low.
[1] SemDeDup: Data-efficient learning at web-scale through semantic deduplication [2] MoDE: CLIP Data Experts via Clustering
Questions for authors
It would be great if the authors answer whether my understanding of their methodology is correct.