acceptodds
Under review as a conference paper at ICLR 2027

CoverCLIP: Coverage-Aware Visual and Textual Alignment

Abstract

Contrastive Language–Image Pre-training (CLIP) performs strongly on image–text retrieval and zero-shot classification. However, existing CLIP training objectives treat all semantically correct captions as equally valid, largely ignoring their semantic coverage. Ideally, for a given image, a caption covering more visible semantic information should receive a higher image–text similarity score. To explicitly measure this, we introduce CoverageBench, a manually verified benchmark containing images derived from Visual Genome. Each image is annotated with four visual semantic atoms (e.g., object, attribute, relation, etc.) and paired with four captions with increasing semantic coverage, denoted as \(C_1, C_2, C_3, C_4\). Models are expected to assign similarity scores satisfying \(s(I, C_1) < s(I, C_2) < s(I, C_3) < s(I, C_4)\). Evaluating on CoverageBench, we find that existing CLIP models struggle to consistently distinguish correct captions according to their semantic coverage. To address this limitation, we propose CoverCLIP, a coverage-aware training framework built on three components. First, we construct Coverage-1M, which consists of 710K samples with graded semantic coverage captions, 228K samples with long-short captions, and 76K samples with highly descriptive captions, providing explicit supervision for graded semantic coverage. Second, we introduce a step-wise mixed training strategy that incorporates semantic coverage-aware supervision into standard contrastive learning. Finally, we design coverage-ranking and hard-negative separation objectives to further enhance coverage-aware supervision. Experiments demonstrate that CoverCLIP improves CoverageBench performance while remaining competitive on image–text retrieval and zero-shot classification. Interestingly, when applied to text-to-image generation by replacing the CLIP text encoder in SDXL, CoverCLIP transfers its coverage awareness to the generation process, enabling the model to progressively generate more detailed images as semantic coverage increases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Coverage-aware Supervision, Fine-grained Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RiuZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoverCLIP, a coverage-aware vision-language alignment framework that aims to improve CLIP's sensitivity to graded semantic coverage. The authors first identify a limitation of existing CLIP models: their difficulty in consistently assigning higher image-text similarity scores to captions containing more correct visual information. To systematically evaluate this issue, they introduce CoverageBench, a manually verified benchmark containing 1,300 images with captions at four progressively increasing coverage levels. They further construct Coverage-1M, a large-scale training dataset with coverage-graded captions, and propose a mixed training framework incorporating coverage-ranking and hard-negative separation objectives. Experiments on image-text retrieval, zero-shot classification, and text-to-image generation demonstrate improvements in coverage sensitivity while largely maintaining general vision-language alignment capabilities.

Critical strengths
  1. Valuable problem identification and a systematic research framework.

The paper identifies a meaningful and relatively underexplored limitation of existing CLIP models regarding graded semantic coverage. More importantly, the authors provide a comprehensive framework that integrates problem formulation, benchmark construction, training data preparation, methodological design, and experimental validation. The introduction of CoverageBench and Coverage-1M provides valuable resources for studying coverage-aware vision-language alignment. Overall, the work is well organized and presents a coherent research pipeline.

  1. Reasonably comprehensive and informative experimental evaluation.

The paper conducts experiments across multiple tasks, including image-text retrieval, zero-shot classification, and text-to-image generation. The results demonstrate consistent improvements on CoverageBench while maintaining competitive performance on conventional downstream tasks. In addition, the ablation studies investigate the contributions of different training data sources, loss functions, ranking strategies, and hyperparameters. These experiments provide reasonable empirical support for the effectiveness of the proposed framework.

Critical weaknesses
  1. Insufficient evidence of broader fine-grained understanding and advantages over alternative vision-language models.

Although CoverCLIP achieves clear improvements on the proposed CoverageBench, its advantages in broader fine-grained vision-language understanding remain insufficiently demonstrated. The main comparisons are primarily conducted against CLIP and its fine-tuned variants, such as Long-CLIP and SmartCLIP. Comparisons with stronger or architecturally different vision-language alignment models are limited. Moreover, the improvements on the proposed coverage metric do not necessarily establish corresponding improvements in general fine-grained semantic understanding. While the text-to-image generation experiments provide additional evidence, the quantitative gains over the strongest baseline are relatively modest. More comprehensive comparisons and evaluations on independent fine-grained or compositional understanding benchmarks would strengthen the practical significance of the proposed approach.

  1. Limited methodological novelty.

The proposed framework mainly combines several established techniques, including mixed-data training, conventional contrastive learning, margin-based ranking, hard-negative separation, and stop-gradient operations. Although their integration is reasonable and empirically effective, the individual components are largely based on existing approaches, and the overall framework does not introduce a substantially new modeling mechanism. Consequently, the methodological contribution appears relatively incremental. The primary novelty lies more in the problem formulation and dataset construction than in the proposed learning algorithm.

Questions for authors
  1. Generalization of CoverageBench. CoverageBench is constructed exclusively from Visual Genome and uses exactly four semantic atoms per image. How well does the benchmark generalize to images from different domains or scenarios involving substantially more semantic information? Could the authors discuss the limitations of this fixed four-level evaluation protocol?

  2. Potential linguistic shortcuts. Although the authors control caption lengths, the average caption length still increases slightly with semantic coverage. Could the authors evaluate a text-only or length-based baseline to further rule out potential linguistic shortcuts in CoverageBench?

  3. Trade-off between coverage and retrieval. In Table 16(a), the Coverage + ShareGPT-4V configuration achieves a higher Exact Chain accuracy (54.8%) than the full CoverCLIP configuration (51.2%), although the latter achieves better retrieval performance. Could the authors further explain this trade-off and clarify how the final data mixing ratio was selected?

  4. Contribution of coverage-specific objectives. Table 16(b) shows that mixed-data CLIP training without the proposed auxiliary objectives already improves Exact Chain accuracy from 29.7% to 44.7%. Could the authors further clarify the relative importance of the coverage-aware training data and the proposed optimization objectives?

  5. Practical significance of generation improvements. Although the generation results show statistically significant improvements, the numerical gains in Table 4 are relatively small. Could the authors provide additional evaluations based on human judgments or explicit concept-presence verification to better establish the practical benefits of CoverCLIP in text-to-image generation?

Reviewer_y9XDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CoverCLIP trains image–text alignment to rank correct captions by how much visible content they cover. It introduces a human-checked benchmark of 1,300 images and a large training set with graded captions. Mixed-data training and ranking objectives raise Exact Chain accuracy to 51.2%, with additional retrieval and image-generation evaluations.

Critical strengths

Ordering multiple true captions by coverage tests a meaningful property that standard correct-versus-incorrect contrastive evaluation misses.

Human checks and training ablations improve the benchmark’s credibility and distinguish gains from added data versus specialized losses.

Critical weaknesses

Average caption length and shared generation templates could reveal coverage level without image understanding; length-matched, text-only, and image-shuffled controls are important.

The benchmark shares scene source and atom-based caption format with training. Non-overlapping images alone do not show transfer to new domains or naturally written descriptions.

A mixed-data baseline already scores 44.7% Exact Chain versus 51.2% with the specialized objectives. The SDXL comparison shows only a small gain over SmartCLIP on a frozen-embedding metric, without direct human confirmation.

Questions for authors

How does Exact Chain change under length-matched, human-written captions from an independent image source, including text-only and shuffled-image controls?

Could you separate the effect of the added graded data from each loss under the same training budget?

Do blinded human raters confirm that later semantic atoms are present and correctly rendered in generated images?

3 comments
AuthorsOpenReview ↗

Polite follow-up: request for reviewers’ feedback on our responses!

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Thank you for the thoughtful rebuttal. The response has addressed my concerns, and I will keep my current score unchanged.

We sincerely thank you for your positive feedback and recognition of our rebuttal. We appreciate your time and thoughtful consideration of our work.

Reviewer_CHhBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether CLIP-style models can distinguish semantically correct captions according to their degree of visual semantic coverage. The authors introduce CoverageBench, a benchmark of 1,300 manually verified images with progressively more comprehensive captions, and Coverage-1M, a dataset providing graded coverage supervision. They further propose CoverCLIP, which combines mixed training with ranking and hard-negative separation losses. Experiments demonstrate improvements on CoverageBench while maintaining competitive retrieval performance.

Critical strengths

1.Interesting problem formulation

Graded semantic coverage provides a useful evaluation perspective beyond existing benchmarks that primarily distinguish correct captions from semantically corrupted ones.

2.Careful benchmark and dataset construction

CoverageBench employs multi-stage filtering, VLM-based verification, and human review. Coverage-1M also provides a potentially valuable resource for studying fine-grained image-text alignment.

Critical weaknesses

1.Insufficient validation of the central assumption and benchmark

The paper assumes that captions covering more visual facts should consistently receive higher image-text similarity scores. However, additional visual details do not necessarily imply greater relevance, and the connection between semantic coverage and similarity remains insufficiently justified. Moreover, despite controlling caption lengths, potential linguistic shortcuts have not been adequately investigated. These concerns directly affect whether CoverageBench measures the intended visual understanding capability.

2.Limited evidence of significance and methodological contribution

The evaluation lacks comparisons with recent strong vision-language encoders such as SigLIP 2 and FG-CLIP 2, making it difficult to assess whether the identified limitation persists in current models. Furthermore, the proposed method mainly combines established training techniques. Its improvement over Long-CLIP on CoverageBench is relatively modest (51.2% vs. 49.7%), without statistical significance testing or multi-seed evaluation. Consequently, the broader significance and robustness of the proposed approach remain unclear.

Questions for authors
  1. Can the authors further justify the assumed monotonic relationship between semantic coverage and image-text similarity? Would text-only or image-shuffling controls reveal potential shortcuts in CoverageBench?

  2. How do recent models such as SigLIP 2 and FG-CLIP 2 perform on CoverageBench? These comparisons would help establish the relevance of the proposed problem to current vision-language models.

  3. Is the improvement over Long-CLIP statistically significant and consistent across independent training runs?

1 comment
AuthorsOpenReview ↗

Polite follow-up: request for reviewers’ feedback on our responses!

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Comments on the paper

1 comment

We sincerely thank you for your positive feedback and recognition of our rebuttal ! We appreciate your time and thoughtful consideration of our work !

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.