- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HN-CLIP for dense-caption retrieval, where closely related descriptions require fine-grained discrimination. It computes caption similarities using the current text encoder, detaches the similarity matrix, masks its diagonal, and adds adaptive margins to negative scores. Similar captions therefore require greater separation in matching scores. Training optionally incorporates token-level alignment, while inference retains the dual-encoder architecture. Evaluation covers four benchmarks, integration with six fine-tuning frameworks, cross-dataset transfer, and loss ablations.
Critical strengths
- Replacing only the global loss improves plain fine-tuning on Long-DCI by 8.19 and 7.58 Recall@1 in the two retrieval directions. It also improves in-domain results across all six fine-tuning frameworks, without requiring additional architectural modules.
- The method reuses existing text embeddings without adding parameters, auxiliary data, or inference overhead. Table A4 reports 17 minutes for DCI training, close to 16 minutes for plain fine-tuning and below 41 minutes for GOAL and 92 minutes for StructXLIP.
Critical weaknesses
- Gienapp et al. [1] use text similarity to require stronger discrimination between similar positive and negative texts, and AVIGATE [2] uses textual and visual similarities to set different separation requirements for different negatives. The core idea of adjusting separation requirements according to sample similarity therefore has limited novelty, and the additions mainly concern the specific margin formulation and its application to dense image-text retrieval.
- Section 2.4 and Appendix C update the model exclusively with HN-CLIP and merely measure standard InfoNCE. Small InfoNCE gradients along this trajectory do not establish premature failure under independent InfoNCE training. HN-CLIP can retain a substantial loss when the positive already outranks every in-batch negative but has not satisfied the additional margin, this does not directly establish the claimed optimization failure caused by near-duplicate.
[1] Lukas Gienapp, Niklas Deckers, Martin Potthast, and Harrisen Scells. “Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins.” [2] Boseung Jeong, Jicheol Park, Sungyeon Kim, and Suha Kwak. “Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval.”
Questions for authors
In the existing training logs for independent InfoNCE fine-tuning in Table 2, does the loss also approach zero? At that stage, does retrieval performance continue to improve or plateau?