- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MCDisp-Align, a probabilistic vision-language representation learning framework that leverages the semantic dispersion across multiple captions of the same image to guide image distribution learning. The authors identify two limitations in existing approaches: deterministic methods may cause semantic distortion by aligning distinct captions to a single image point, and existing probabilistic methods may cause semantic collapse by aligning the image distribution with individual caption distributions that cover only partial semantics. MCDisp-Align addresses this through probabilistic representation, caption-set distribution construction via moment matching, and a combined distribution matching objective. Experiments on MSCOCO and Flickr30K show strong retrieval and improvements over fine-tuned CLIP on standard retrieval metrics.
Critical strengths
- The motivation behind the method is reasonable. The overall method is simple and uses the second-order statistical information of multiple captions as the supervised signal for image distribution learning.
- The reported retrieval improvements are encouraging.
Critical weaknesses
- Some details need to be clarified. Does CLIP-FT also use exactly the same caption-set contrastive learning objective for training? It is unclear whether the improvements come from modeling caption dispersion, or more from introducing caption-set training.
- Table 4 in the paper only compares the complete model with the model that removes the entire . I hope the author removes KL alignment, coverage, subspace alignment, and variance regularization separately
- The paper lacks sensitivity experiments on the number of captions and has not studied the impact of increasing the number of captions on model performance. As the number of captions increases, can the model obtain more effective semantic information to continuously improve performance?
Questions for authors
None
3 comments
The response solved some of my concern. I will keep my positive rating.
Thank You for Your Consideration
Dear Reviewer,
Thank you for taking the time to re-evaluate our response. We are glad the additional analyses addressed your concerns. All new experiments and clarifications from the discussion will be incorporated into the revision, and we remain happy to provide any further details.
Best regards,
The Authors
Response to Reviewer fTHS
1. Summary
Thank you for your constructive review. We added a five-caption CLIP control, caption-count sensitivity experiments, and separate loss ablations to address your concerns. We present the settings, results, and conclusions below in the order of W1, W2, and W3.
2. Caption-Set Training in CLIP-FT (W1)
The original CLIP-FT uses paired image–caption training. To test five-caption supervision, we added CLIP-FT (SET) with the same backbone and fine-tuning recipe, using all five captions for contrastive learning. The additional control uses 5000 MSCOCO images and center-cosine scoring. Table R3.1 identifies the paper results and the supplementary control separately.
Table R3.1. Caption-set retrieval with the five-caption CLIP control (R@1, %).
Under the original paper protocol, MCDisp-Align improves set I2T/T2I R@1 over CLIP-FT by 2.70/7.80 percentage points. Together with the within-model alignment ablation, these gains support the effectiveness of distribution alignment.
3. Sensitivity to the Number of Training Captions (W2)
To test caption-count sensitivity, we trained MCDisp-Align using the first one, first three, or all five captions in a fixed order, with the remaining training settings unchanged. All runs use the same 11814 training-unseen MSCOCO evaluation images, five test captions per image, and paper Eq. (4). Results are in Table R3.2.
Table R3.2. Training-caption sensitivity with fixed test inputs (%).
Increasing training captions from one to five improves I2T/T2I R@1 by 8.23/13.20 percentage points, with further gains from three to five captions. This demonstrates that MCDisp-Align continues to benefit from additional training captions within the tested range.
4. Component-Wise Loss Ablations (W3)
To address your request for separate loss ablations, we removed each alignment component individually, removed all alignment terms, and tested diagonal-only covariance. Training uses MSCOCO, ViT-L/14, 10 epochs, and seed 42; evaluation uses 11814 training-unseen images, five captions per image, and paper Eq. (4). Table R3.3 reports R@1; mR averages I2T and T2I R@1/5/10.
Table R3.3. Caption-set retrieval with component ablations (R@1, %).
The full MCDisp-Align achieves higher I2T/T2I R@1 than all listed ablation variants in Table R3.3. This highlights the importance of combining the alignment terms and low-rank covariance for image–text distribution alignment.
5. Closing Summary
Overall, the added control clarifies caption-set training, the caption-count study establishes continued gains from one to five training captions, and the ablations provide a separate result for each alignment component. We will include these experiments and clarify their settings in the revised manuscript. Thank you again for your helpful suggestions.