acceptodds
Under review as a conference paper at ICLR 2027

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

Abstract

Dense-caption retrieval requires distinguishing images paired with highly similar, detail-rich descriptions. We empirically examine contrastive fine-tuning from a strong pretrained dual encoder and observe that the standard InfoNCE loss can become small on sampled batches while held-out retrieval errors remain. Through an analysis of gradient dynamics, we find that InfoNCE rapidly loses optimization signal: within the first epoch, its loss falls below on of sampled batches, and over the run its gradient norm below in of measurements, even as held-out retrieval continues to improve. We relate this premature saturation to the abundance of near-duplicate captions in dense-caption datasets, where the most confusable negatives remain unresolved. We therefore introduce HN-CLIP, which uses the text encoder's own text–text geometry to construct per-negative adaptive similarity margins. A detached caption-similarity matrix is added to the negative logits, assigning larger margins to more similar captions without mining, synthesizing, or resampling negatives. The objective requires only one similarity matrix and a masked logit addition during training, with no auxiliary data, additional parameters, offline preprocessing, or inference-time overhead. Across four dense-caption retrieval benchmarks, HN-CLIP improves over the strongest competitors by – Recall@1 while training faster than GOAL and faster than StructXLIP. It also improves all six tested fine-tuning frameworks in-domain and surpasses full-data GOAL and StructXLIP with only of the DOCCI and DCI training data, demonstrating strong data efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dense-caption retrieval, contrastive learning, hard negatives, metric learning, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WhvfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HN-CLIP for dense-caption retrieval, where closely related descriptions require fine-grained discrimination. It computes caption similarities using the current text encoder, detaches the similarity matrix, masks its diagonal, and adds adaptive margins to negative scores. Similar captions therefore require greater separation in matching scores. Training optionally incorporates token-level alignment, while inference retains the dual-encoder architecture. Evaluation covers four benchmarks, integration with six fine-tuning frameworks, cross-dataset transfer, and loss ablations.

Critical strengths
  1. Replacing only the global loss improves plain fine-tuning on Long-DCI by 8.19 and 7.58 Recall@1 in the two retrieval directions. It also improves in-domain results across all six fine-tuning frameworks, without requiring additional architectural modules.
  2. The method reuses existing text embeddings without adding parameters, auxiliary data, or inference overhead. Table A4 reports 17 minutes for DCI training, close to 16 minutes for plain fine-tuning and below 41 minutes for GOAL and 92 minutes for StructXLIP.
Critical weaknesses
  1. Gienapp et al. [1] use text similarity to require stronger discrimination between similar positive and negative texts, and AVIGATE [2] uses textual and visual similarities to set different separation requirements for different negatives. The core idea of adjusting separation requirements according to sample similarity therefore has limited novelty, and the additions mainly concern the specific margin formulation and its application to dense image-text retrieval.
  2. Section 2.4 and Appendix C update the model exclusively with HN-CLIP and merely measure standard InfoNCE. Small InfoNCE gradients along this trajectory do not establish premature failure under independent InfoNCE training. HN-CLIP can retain a substantial loss when the positive already outranks every in-batch negative but has not satisfied the additional margin, this does not directly establish the claimed optimization failure caused by near-duplicate.

[1] Lukas Gienapp, Niklas Deckers, Martin Potthast, and Harrisen Scells. “Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins.” [2] Boseung Jeong, Jicheol Park, Sungyeon Kim, and Suha Kwak. “Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval.”

Questions for authors

In the existing training logs for independent InfoNCE fine-tuning in Table 2, does the loss also approach zero? At that stage, does retrieval performance continue to improve or plateau?

Reviewer_wfmROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HN-CLIP for dense-caption image–text retrieval. Motivated by the observation that standard InfoNCE can become nearly inactive on sampled batches while retrieval errors remain, the method adds detached caption–caption similarities to negative logits, imposing larger separation margins on semantically similar captions. It optionally combines this objective with token-level alignment. Experiments cover four retrieval settings, integration into six fine-tuning frameworks, cross-dataset transfer, data efficiency, and gradient dynamics.

Critical strengths

S1. A simple objective modification delivers substantial and broadly supported improvements. HN-CLIP reuses existing text embeddings without additional parameters, negative generation, or inference overhead. Its gains across eight retrieval directions, integration into six fine-tuning frameworks, and multi-seed results establish practical value. Controlled comparisons of loss components, fixed/adaptive margins, and frozen/recomputed geometry further support the design. This combination of simplicity and empirical effectiveness is my main reason for recommending acceptance.

S2. The optimization analysis provides useful insight beyond benchmark improvements. Section 2.4 measures both objectives on the same batches along an HN-CLIP training trajectory, showing that the proposed loss retains gradient signal when standard InfoNCE becomes small. The theoretical analysis explains how caption-dependent margins redistribute gradient weight toward confusable negatives. Together, these results offer an informative account of why modifying negative treatment can improve dense-caption retrieval.

Critical weaknesses

W1. The advantage over established negative-separation objectives is not sufficiently demonstrated. DiHT [X1] increases hard-negative weights, while CE-CLIP [X2] and SPARCL [X3] introduce adaptive-margin objectives for CLIP. HN-CLIP’s detached caption-derived margins are distinct, but comparisons against these objectives under matched backbone, data, and training budgets are needed to establish their specific advantage. Representation-level evidence, such as embedding visualizations or clustering analyses accompanied by quantitative positive–hard-negative gap measurements, would also help verify that the proposed mechanism improves separation. This matters because negative separation is the central contribution, beyond the strong overall retrieval results.

W2. The connection between the proposed mechanism and cross-domain transfer remains insufficiently explained. Tables A8/A12 establish transfer performance, but do not directly show why source-domain negative margins improve target-domain discrimination. Table 3(a) also shows that the default boost underperforms (gamma=0) on Urban-1K, indicating that stronger separation can hurt transfer. An analysis of target-domain hard-negative gaps or error types would clarify when the mechanism generalizes and when it over-specializes. This limits the explanation of the transfer results, rather than negating the reported in-domain benefits.

[X1] Filip Radenović et al. Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training. CVPR, 2023.

[X2] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding. CVPR, 2024.

[X3] Haoxin Li and Boyang Li. Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data. CVPR, 2025.

Questions for authors
  1. Are controlled comparisons with DiHT’s HN-NCE or related adaptive-ranking objectives available? Please clarify which differences specifically motivate the caption-derived margin.
  2. Do existing measurements show how positive–hard-negative similarity gaps change relative to plain fine-tuning or other negative-separation methods? Representation visualizations and clustering analyses would be useful additional validation.
  3. How do the authors reconcile the cross-domain gains with the Urban-1K degradation relative to (gamma=0)? Is there evidence identifying which target-domain confusions improve or worsen?
Reviewer_gfbrOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces HN-CLIP. It uses the text encoder’s text–text geometry to construct per-negative adaptive similarity margins. The similarity margins are then used to improve the long-caption retrieval.

Critical strengths
  • the proposed adaptive similarity margin from text encoder's text-text geometry demonstrated its effectiveness in performance improvement in long-text retrieval tasks, even with a plug-and-play manner
  • the evaluation and ablation experiments are comprehensive
  • the performance improvement of retrieval especially R@1 is significant
Critical weaknesses

None

Questions for authors
  • The authors should consider writing a better conclusion section to summarize the contributions and insights offered by this paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.