acceptodds
Under review as a conference paper at ICLR 2027

MCDisp-Align: Multi-Caption Semantic Dispersion-Guided Distribution Alignment for One-to-Many Vision-Language Representation Learning

Abstract

Vision-language representation learning supports a wide range of multimodal applications by aligning visual and textual semantics, but existing methods still face two key challenges. Deterministic methods embed samples as single points, which tends to compress differences among semantically distinct descriptions and cause semantic distortion. Conversely, while probabilistic methods model samples as distributions, aligning a broader image distribution with caption distributions that primarily contain semantically similar points can restrict the image's semantic range, leading to semantic collapse. To address these challenges, we propose MCDisp-Align, a probabilistic framework that leverages multi-caption semantic dispersion to guide image distribution learning. MCDisp-Align constructs a caption-set distribution that captures the shared semantic center, semantic range, and dominant directions of variation across captions. By combining set-level distribution alignment with caption-level coverage, it preserves semantic differences among individual captions to reduce semantic distortion, while guiding the image distribution to cover complementary semantic perspectives and mitigate semantic collapse. We evaluate MCDisp-Align on image-text retrieval along with geometric analyses on MSCOCO and Flickr30K. The results demonstrate improved retrieval performance alongside greater semantic dispersion among caption representations of the same image and clearer separation across images, effectively balancing semantic coverage and discrimination. Code will be made publicly available upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Learning, Probabilistic Embeddings, Contrastive Learning, Cross-Modal Alignment, Image-Text Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fTHSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MCDisp-Align, a probabilistic vision-language representation learning framework that leverages the semantic dispersion across multiple captions of the same image to guide image distribution learning. The authors identify two limitations in existing approaches: deterministic methods may cause semantic distortion by aligning distinct captions to a single image point, and existing probabilistic methods may cause semantic collapse by aligning the image distribution with individual caption distributions that cover only partial semantics. MCDisp-Align addresses this through probabilistic representation, caption-set distribution construction via moment matching, and a combined distribution matching objective. Experiments on MSCOCO and Flickr30K show strong retrieval and improvements over fine-tuned CLIP on standard retrieval metrics.

Critical strengths
  1. The motivation behind the method is reasonable. The overall method is simple and uses the second-order statistical information of multiple captions as the supervised signal for image distribution learning.
  2. The reported retrieval improvements are encouraging.
Critical weaknesses
  1. Some details need to be clarified. Does CLIP-FT also use exactly the same caption-set contrastive learning objective for training? It is unclear whether the improvements come from modeling caption dispersion, or more from introducing caption-set training.
  2. Table 4 in the paper only compares the complete model with the model that removes the entire LalignL_{\text{align}}. I hope the author removes KL alignment, coverage, subspace alignment, and variance regularization separately
  3. The paper lacks sensitivity experiments on the number of captions and has not studied the impact of increasing the number of captions on model performance. As the number of captions increases, can the model obtain more effective semantic information to continuously improve performance?
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer fTHS

1. Summary

Thank you for your constructive review. We added a five-caption CLIP control, caption-count sensitivity experiments, and separate loss ablations to address your concerns. We present the settings, results, and conclusions below in the order of W1, W2, and W3.

2. Caption-Set Training in CLIP-FT (W1)

The original CLIP-FT uses paired image–caption training. To test five-caption supervision, we added CLIP-FT (SET) with the same backbone and fine-tuning recipe, using all five captions for contrastive learning. The additional control uses 5000 MSCOCO images and center-cosine scoring. Table R3.1 identifies the paper results and the supplementary control separately.

Table R3.1. Caption-set retrieval with the five-caption CLIP control (R@1, %).

ModelI2T R@1T2I R@1
CLIP-FT82.5076.10
MCDisp-Align85.2083.90
CLIP-FT (SET)83.7681.44

Under the original paper protocol, MCDisp-Align improves set I2T/T2I R@1 over CLIP-FT by 2.70/7.80 percentage points. Together with the within-model alignment ablation, these gains support the effectiveness of distribution alignment.

3. Sensitivity to the Number of Training Captions (W2)

To test caption-count sensitivity, we trained MCDisp-Align using the first one, first three, or all five captions in a fixed order, with the remaining training settings unchanged. All runs use the same 11814 training-unseen MSCOCO evaluation images, five test captions per image, and paper Eq. (4). Results are in Table R3.2.

Table R3.2. Training-caption sensitivity with fixed test inputs (%).

Training captions / imageI2T R@1T2I R@1
156.9952.16
362.6661.65
565.2265.36

Increasing training captions from one to five improves I2T/T2I R@1 by 8.23/13.20 percentage points, with further gains from three to five captions. This demonstrates that MCDisp-Align continues to benefit from additional training captions within the tested range.

4. Component-Wise Loss Ablations (W3)

To address your request for separate loss ablations, we removed each alignment component individually, removed all alignment terms, and tested diagonal-only covariance. Training uses MSCOCO, ViT-L/14, 10 epochs, and seed 42; evaluation uses 11814 training-unseen images, five captions per image, and paper Eq. (4). Table R3.3 reports R@1; mR averages I2T and T2I R@1/5/10.

Table R3.3. Caption-set retrieval with component ablations (R@1, %).

ConfigurationI2T R@1T2I R@1
Full MCDisp-Align85.2083.90
w/o all alignment terms66.2264.99
w/o KL alignment67.6367.51
w/o caption-center coverage65.7965.80
w/o subspace alignment67.1866.61
w/o variance regularization62.2262.85
Diagonal-only covariance (r=0r = 0)67.0566.73

The full MCDisp-Align achieves higher I2T/T2I R@1 than all listed ablation variants in Table R3.3. This highlights the importance of combining the alignment terms and low-rank covariance for image–text distribution alignment.

5. Closing Summary

Overall, the added control clarifies caption-set training, the caption-count study establishes continued gains from one to five training captions, and the ablations provide a separate result for each alignment component. We will include these experiments and clarify their settings in the revised manuscript. Thank you again for your helpful suggestions.

The response solved some of my concern. I will keep my positive rating.

AuthorsOpenReview ↗

Thank You for Your Consideration

Dear Reviewer,

Thank you for taking the time to re-evaluate our response. We are glad the additional analyses addressed your concerns. All new experiments and clarifications from the discussion will be incorporated into the revision, and we remain happy to provide any further details.

Best regards,

The Authors

Reviewer_N3nkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MCDisp-Align, a probabilistic framework for learning one-to-many image–text relationships. It constructs a caption-set Gaussian through moment matching and uses its statistics to supervise the image distribution’s center, spread, and variation subspace. A caption-center coverage constraint encourages inclusion without forcing caption centers to coincide. Experiments on MSCOCO and Flickr30K evaluate caption-set retrieval, standard retrieval, and caption-space geometry.

Critical strengths

S1. The proposed direction is plausible and conceptually well motivated. Distinguishing the semantic scope of an image from that of an individual caption is sensible. Combining within-caption variance with between-caption dispersion provides a coherent supervision target, while detached alignment and bounded caption-center coverage avoid explicitly forcing all caption centers to coincide.

S2. The method demonstrates useful retrieval improvements without substantially contracting caption representations. Caption-set T2I R@1 improves over CLIP-FT by 7.8 points on MSCOCO and 2.5 points on Flickr30K. Standard retrieval also improves across all six MSCOCO metrics. Tables 4-5 show that adding alignment improves retrieval while leaving average caption dispersion largely unchanged, supporting the practical utility of the objective.

Critical weaknesses

W1. The experiments do not convincingly establish that the motivating semantic distortion and semantic collapse are consequential failures of the baseline. The caption-set T2I gains are meaningful, but improvements over CLIP-FT are less pronounced elsewhere: Flickr30K caption-set I2T R@1 improves by only 0.3 points, and MSCOCO standard I2T/T2I R@1 improves by 2.0/0.7 points. Small gains alone do not invalidate the hypothesis, but the paper does not directly connect baseline retrieval errors to the proposed failure mechanisms. WSD/BSD describe average geometry without establishing whether meaningful attributes or relations were lost. Larger caption dispersion alone does not establish better semantic preservation, since it may also reflect variation unrelated to meaningful visual distinctions. Moreover, caption dispersion changes little when the alignment loss is removed. The current evidence therefore supports an effective training objective more clearly than it establishes and resolves the stated semantic failures.

W2. The benchmarks do not adequately stress the semantic diversity central to the motivation. MSCOCO and Flickr30K provide multiple captions, but multiplicity does not necessarily imply substantially different semantic coverage. Even Figure 1(a) largely repeats the same horses-pulling-a-carriage content, with relatively small changes in descriptive emphasis. Evaluation on a dataset or controlled subset containing more complementary objects, attributes, and relations is needed to test the proposed advantage. This should examine both final retrieval performance and representation geometry, including image-distribution coverage of distinct caption meanings and separation from hard negatives, with results stratified by caption-set semantic diversity.

Questions for authors
  1. Can you directly demonstrate baseline errors attributable to semantic distortion or semantic collapse, and show that MCDisp-Align corrects them? In addition to aggregate distances, please examine whether specific attribute or relation distinctions become less distinguishable under CLIP-FT and are recovered by the proposed method.
  2. Can you evaluate a dataset or controlled subset with genuinely complementary caption content, and report how the gain over CLIP-FT changes with semantic diversity? Please include both retrieval results and analyses of image-distribution coverage and hard-negative separation.
  3. Which alignment components are responsible for the improvements? Table 4 removes the entire alignment objective. Please separately ablate KL alignment, caption-center coverage, subspace alignment, and variance regularization, and compare the full covariance with a diagonal-only variant. These controls would clarify whether modeling caption variation directions is necessary beyond center alignment and regularization.
5 comments
AuthorsOpenReview ↗

Response to Reviewer N3nk

1. Summary

Thank you for your thoughtful suggestions on semantic preservation and caption diversity. We added a caption-count experiment with fixed test inputs, a diversity-stratified study on 400 MSCOCO images, and separate component ablations. Detailed responses follow in the order of W1, W2, Q1, Q2, and Q3.

2. Retrieval Evidence Beyond Geometric Dispersion (W1)

To test whether multi-caption learning improves actual retrieval, we trained MCDisp-Align with one, three, or five captions per image, using nested caption subsets and the same training recipe. Evaluation is fixed to the same 11814 training-unseen MSCOCO images, five captions per image, and paper Eq. (4). Table R2.1 reports the results.

Table R2.1. Training-caption count with fixed MSCOCO evaluation inputs (%).

Training captions / imageI2T R@1T2I R@1
156.9952.16
362.6661.65
565.2265.36

With identical test inputs, increasing training captions from one to five improves I2T/T2I R@1 by 8.23/13.20 percentage points and mR by 7.66 percentage points. These retrieval gains demonstrate the effectiveness of multi-caption training for image–text matching.

3. Performance Across Caption Differences (W2)

To examine performance across caption diversity, we selected 400 training-unseen MSCOCO images and grouped them by caption-text statistics, independently of retrieval scores. The low-, medium-, and high-difference groups contain 134, 133, and 133 images. All models use the same 400-image or 400-caption-set candidate pool and center-cosine scoring. Table R2.2 reports I2T/T2I R@1.

Table R2.2. Caption-difference stratification; each cell is I2T/T2I R@1 (%).

ModelLow (134)Medium (133)High (133)All (400)
CLIP-FT94.78 / 97.0196.99 / 93.2396.24 / 94.7496.00 / 95.00
CLIP-FT (SET)94.78 / 99.2596.99 / 93.9895.49 / 95.4995.75 / 96.25
MCDisp-Align91.79 / 93.2894.74 / 89.4796.24 / 95.4994.25 / 92.75

In the high-difference group, MCDisp-Align obtains 96.24/95.49 R@1: it matches CLIP-FT in I2T and improves T2I by 0.75 pp; it improves I2T over CLIP-FT (SET) by 0.75 pp and matches T2I. Its relative performance is strongest in this group, consistent with the intended use of caption dispersion.

4. Specific Attribute and Relation Recovery (Q1)

The experiments establish retrieval gains while preserving caption variation. Recovery of a specific attribute or relation is a separate fact-level question requiring annotated minimal pairs and verified baseline-error corrections. We identify that analysis as future work and will distinguish the demonstrated retrieval findings from attribute-specific recovery in the revised claims.

5. Complementarity, Coverage, and Hard Negatives (Q2)

Table R2.2 provides the controlled, diversity-stratified retrieval comparison under a common 400-candidate pool and identical scoring. The high-difference group shows a 0.75-pp T2I improvement over CLIP-FT and a 0.75-pp I2T improvement over CLIP-FT (SET).

The groups reflect caption-text differences. The 1000-image checkpoint diagnostic measures coverage across its sample as a whole. Fact-annotated complementarity, diversity-specific coverage, and separation from structured hard negatives require targeted tests, which we identify as a follow-up study.

6. Contributions of the Alignment Components (Q3)

To address your request, we separately removed the four alignment components and compared low-rank-plus-diagonal with diagonal-only covariance. The runs use MSCOCO, ViT-L/14, 10 epochs, seed 42, and the same 11814-image, five-caption evaluation with paper Eq. (4). Table R2.3 reports R@1; mR averages I2T and T2I R@1/5/10.

Table R2.3. Component-wise ablations on caption-set retrieval (%).

ConfigurationI2T R@1T2I R@1
Full MCDisp-Align85.2083.90
w/o all alignment terms66.2264.99
w/o KL alignment67.6367.51
w/o caption-center coverage65.7965.80
w/o subspace alignment67.1866.61
w/o variance regularization62.2262.85
Diagonal-only covariance (r=0r = 0)67.0566.73

The full MCDisp-Align achieves higher I2T/T2I R@1 than all listed ablation variants in Table R2.3. This highlights the importance of combining the alignment terms and low-rank covariance for image–text distribution alignment.

7. Closing Summary

Overall, the additional experiments establish retrieval gains from more training captions, characterize performance across caption differences, and report separate component outcomes. We will incorporate these findings and distinguish retrieval evidence from fact-level semantic diagnostics in the revision. Thank you for your suggestions.

Thank you for the additional experiments and detailed responses. I appreciate the authors' efforts. However, I remain seriously concerned that the rebuttal does not address my central concerns regarding the proposed semantic distortion and semantic collapse. In particular, the new results in Table R2.2 raise further questions about the paper's core motivation.

W1. Evidence of semantic distortion and semantic collapse. My original concern was not whether training with more captions improves retrieval performance. Increasing the number of training captions provides additional supervision, so the improvements in Table R2.1 do not establish that semantic distortion or semantic collapse actually occurs in existing methods, or that MCDisp-Align resolves these failures. I specifically asked for quantitative and qualitative evidence showing that baseline representations lose meaningful attribute or relation distinctions, or that image distributions fail to cover distinct caption meanings, and that MCDisp-Align corrects these failures. The authors instead defer this analysis to future work. Since these failure mechanisms are central to the paper's motivation, I consider this a significant unresolved concern. Can the authors directly demonstrate the existence of these semantic failures in the baselines and show that MCDisp-Align addresses them, rather than only reporting retrieval gains from additional caption supervision?

W2. Diversity-stratified results and the core hypothesis. I find the new results in Table R2.2 particularly concerning. If greater caption diversity causes the semantic failures motivating this work, one would expect some evidence that existing methods struggle more with such diversity. However, CLIP-FT does not show a clear degradation from low to high caption difference, and its retrieval performance remains consistently high. More importantly, MCDisp-Align underperforms CLIP-FT in both directions for the low- and medium-difference groups. Even in the high-difference group, it only matches CLIP-FT in I2T and improves T2I by 0.75 percentage points, equivalent to approximately one additional correct retrieval among 133 queries. Across all 400 queries, MCDisp-Align performs worse than CLIP-FT in both directions. These results do not convincingly support the claimed advantage of modeling diverse caption semantics and may instead suggest a trade-off that the paper does not explain. Moreover, the groups are defined by caption-text differences rather than verified semantic complementarity. If this grouping does not capture meaningful differences in objects, attributes, or relations, the experiment may not adequately test the hypothesis. If it does, the results raise serious doubts about the claimed benefits. How do the authors reconcile these results with the core motivation of MCDisp-Align? Do they indicate that the diversity grouping is insufficient to capture the proposed semantic failures, or that the method does not consistently provide the claimed benefits even when caption diversity is high?

Overall, while I acknowledge the additional experiments, I do not find that W1 or W2 has been sufficiently addressed. My concern is no longer limited to missing experimental evidence. The newly reported results themselves raise substantial questions about whether the proposed method actually resolves the semantic failures that motivate its design.

AuthorsOpenReview ↗

Response to Reviewer N3nk

1. Summary

We thank the reviewer for the careful re-examination of our evidence. Determining exactly which meanings a learned embedding preserves is an open interpretability problem, and established practice probes specific semantic distinctions through observable image–caption matching behavior (e.g., Winoground, ARO). We agree that aggregate retrieval gains alone leave your question open, so we examined individual caption scores, retrieval ranks, and distributional coverage on 400 MSCOCO images held out from fine-tuning.

2. Evidence of semantic distortion and semantic collapse. (W1)

Distortion (deterministic CLIP). The score spread across an image's five reference captions grows from 0.025 (zero-shot) to 0.056 after one fine-tuning epoch and 0.061 after full fine-tuning, and the most distinctive captions lose the most (Spearman ρ = 0.25). Example: for image 000000028858, four "yard chairs" captions score 0.61–0.68, while the valid "A woman opening the trunk of her car" is left at 0.006. MCDisp-Align recovers such dropped meanings in other cases: for image 000000427083, early pairwise fine-tuning ranks the only mouse-mentioning caption far down the gallery; MCDisp-Align ranks it near the top. Table 3 adds geometric evidence: MCDisp-Align retains greater within-image caption dispersion than CLIP-FT (0.408 vs. 0.320) with cross-image separation maintained.

Collapse (probabilistic models). We measured whether each caption center falls inside its image's coverage region (d² ≤ D). ProbVLM and PCME++ cover 0% of their own caption centers (median d²/D: 1.6×10⁴ and 20); MCDisp-Align covers 100% (3×10⁻⁵), because its KL and coverage terms align the image covariance with the caption-set statistics. The learned regions also admit fact-swapped captions, so region membership alone cannot establish semantic correctness; we will state this limitation in the revision.

3. Diversity-stratified results and the core hypothesis. (W2)

We would like to state our position explicitly. In the first-round response we reported the observed pattern that MCDisp-Align's relative performance was strongest in the high-difference group; we did not claim a monotone advantage that grows with caption diversity, and neither the paper nor the rebuttal claims that greater caption diversity degrades baseline retrieval. The claim we do make is a hypothesis about representations: MCDisp-Align uses differences among valid captions to guide the range of meanings the image representation covers. We are glad to discuss this point. Under this hypothesis, strong CLIP-FT retrieval across groups is fully compatible — captions sharing "folded clothes" can identify the correct image even when the mouse caption ranks poorly. Table R2.2 groups captions by textual difference and scores the shared center: CLIP-FT leads in the low- and medium-difference groups and overall; in the high-difference group the two models are comparable at set level. Since the grouping does not verify which captions contribute distinct facts, it cannot isolate complementary-meaning handling. The caption-level analysis above addresses this directly: where a valid description loses alignment, where MCDisp-Align recovers it, and how broadly each model covers its reference captions.

Reviewer_mVpnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the one-to-many vision-language representation learning problem where a single image is associated with multiple semantically distinct captions, aiming to mitigate both semantic distortion in deterministic methods caused by compressing diverse captions into a single point and semantic collapse in probabilistic methods caused by directly aligning an image distribution with individual, narrowly scoped caption distributions. To address these challenges, the authors propose MCDisp-Align, a probabilistic framework that leverages multi-caption semantic dispersion to guide image distribution learning. The empirical effectiveness and geometric properties of the framework are validated on MSCOCO and Flickr30K through one-to-many retrieval, standard image-text retrieval, text-space dispersion analyses.

Critical strengths

S1: The paper is intuitively motivated and directly addresses the core bottlenecks of cross-modal alignment by clearly decomposing the challenges in one-to-many image-text matching into text-side semantic distortion and image-side semantic collapse. Image semantics essentially constitute the union rather than the intersection of multiple complementary text semantics, thereby making the strategy of utilizing multi-text set statistics to guide image distribution learning both theoretically sound and justified.

S2: The design of the asymmetric gradient-stopping mechanism is conceptually clear and well-aligned with the goal of preserving text semantic diversity. Specifically, within the distribution alignment loss, the diagonal KL divergence, coverage constraint, and subspace alignment terms treat the text-set statistics as a one-way supervisory signal using stop-gradient operations (sg(⋅)\mathrm{sg}(\cdot)) to update only the image branch. This prevents the distribution alignment loss from forcibly collapsing different text centers of the same image into a single point, thereby effectively supporting the preservation of text-side semantic variance.

Critical weaknesses

W1: The core loss function $ \mathcal{L} _ {align} $ exhibits several mathematical inconsistencies and architectural disconnects from the main pipeline, leaving both the low-rank matrix UiU_i and the text variance vtv^t virtually idle during training and inference. Specifically, UiU_i is entirely omitted during standard retrieval inference and set-level similarity computation in Eq. 4. It also lacks gradient paths in both Lmatch\mathcal{L} _ {match} and KLdiag\mathrm{KL} _ {diag}, relying solely on a subspace alignment term that is enabled only in the final epochs with a negligible weight lacking scale constraints. Furthermore, the coverage constraint term β[d2/D−1]+\beta[d^2/D - 1] _ + remains perpetually zero in high-dimensional spaces under normal feature scales. In addition, the target diagonal variance in KLdiag\mathrm{KL} _ {diag} leads to double-counting cross-text variance against Σiv\Sigma_i^v. Finally, the text variance vi,ktv_{i,k}^t is isolated from alignment gradients due to stop-gradient operations sg(⋅)\mathrm{sg}(\cdot), causing it to degenerate into a trivial constant near v0=0.04v_0 = 0.04.

W2: The main one-to-many retrieval experiment suffers from a severe training-evaluation circular bias that favors the proposed method unfairly, as Section 3.2 defines this task by pre-aggregating five reference captions per image into a set representation, which is an artificial setup unavailable in real-world independent text retrieval scenarios where queries and candidates are processed separately. Furthermore, because MCDisp-Align is explicitly optimized via Lmatch\mathcal{L}_{match} to align images with the aggregated mean of five text centers during training, whereas baseline models like CLIP-FT and ProLIP-FT are typically fine-tuned on single image-text pairs, its performance advantage stems primarily from this direct coupling between the training objective and the evaluation protocol rather than any intrinsic superiority of the probabilistic distribution alignment framework.

W3: The standard retrieval evaluation suffers from selective baseline reporting and severe performance anomalies, as evidenced by the omission of Flickr30K comparisons in Table 2 alongside a drastic performance drop in the ablation variant (w/o Lalign\mathcal{L} _ {align}) compared to CLIP-FT, which reveals that Lmatch\mathcal{L} _ {match} alone harms single-text retrieval and Lalign\mathcal{L}_{align} merely compensates for this collapse. Furthermore, backbone inconsistencies, such as utilizing ViT-H/14 for ProLIP versus ViT-L/14 for MCDisp-Align, coupled with the absence of fair comparisons against classical probabilistic baselines like PCME++ and ProbVLM on the same backbone, further undermine the empirical credibility of the evaluation.

W4: The ablation study fails to validate the individual efficacy of the key modules, as Section 3.5 only presents a single experiment that removes Lalign\mathcal{L} _ {align} as a whole, omitting any step-by-step breakdown to isolate the contributions of the coverage constraint (β\beta), the low-rank subspace alignment (γ\gamma), the rank rr, and the stop-gradient operation sg(⋅)\mathrm{sg}(\cdot). Furthermore, the ablation results in Table 4 are restricted entirely to standard retrieval, completely omitting any evaluation on the paper's core task of one-to-many retrieval under the ablation of Lalign\mathcal{L} _ {align}.

Questions for authors

Please refer to the "Critical Weaknesses" section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mVpn (Part 1/2)

1. Summary

Thank you for your careful review. To address your concerns about parameter training, caption-set supervision, baseline coverage, and component contributions, we examined training paths and checkpoint statistics, added a five-caption CLIP control and ViT-L/14 probabilistic baselines, and conducted separate loss ablations. De tailed responses follow in the order of your review.

2. Training of the Low-Rank Factor and Text Variance (W1)

To address your concern about parameter supervision, we examined the gradient paths and a trained checkpoint on 1000 MSCOCO images. Both the low-rank factor and text variance receive training signals; the corresponding component ablations are reported in Table R1.3.

(1) Low-rank factor and inference. Coverage and subspace alignment train UiU_i through the full image covariance and its directional basis, respectively, even though UiU_i is not directly included in retrieval scores. Subspace alignment follows initial matching, and its coefficient balances loss scales without determining parameter-gradient magnitude.

(2) Coverage activation. The checkpoint median, 90th percentile, and maximum d2D\frac{d^2}{D} are 0.0243, 0.0427, and 0.1107, all below the coverage threshold of 1. This confirms caption-center coverage at the trained checkpoint, while Table R1.3 evaluates the contribution of this constraint during training.

(3) Covariance accounting. Within-caption uncertainty and between-caption variation are distinct components of total covariance. Between-caption variation enters the KL target once. The diagonal KL matches the image base variance to the target diagonal; subspace alignment separately guides the low-rank directions.

(4) Text variance and stop-gradient. Stop-gradient detaches the caption-derived alignment targets. Matching and variance regularization continue to train the text-variance head. Its predictions vary around the 0.04 anchor: the checkpoint mean is 0.0397, with 10th/90th percentiles of 0.0356/0.0440. We will clarify these distinct training roles in the revision.

3. Caption-Set Evaluation and Training-Objective Control (W2)

To address the difference in training objectives, we added CLIP-FT (SET), which uses all five captions for contrastive learning with the same backbone and fine-tuning recipe as CLIP-FT. All methods receive identical caption groups within each caption-set evaluation protocol.

Table R1.1 shows that MCDisp-Align outperforms CLIP-FT (SET) in both retrieval directions. This demonstrates the value of distribution alignment beyond simply including all captions in contrastive training.

Table R1.1. Original paper results and supplementary caption-set control (set R@1, %).

ModelI2T R@1T2I R@1
CLIP-FT82.5076.10
MCDisp-Align85.2083.90
CLIP-FT (SET)83.7681.44

4. Standard Retrieval and Probabilistic Baselines (W3)

To address your concern about the matching objective, we compare the same MCDisp-Align architecture with and without alignment on MSCOCO (paper Table 4). Adding alignment improves standard I2T/T2I R@1 by 8.80/7.20 percentage points and brings both metrics above CLIP-FT (paper Table 2). The within-model improvement demonstrates the effectiveness of distribution alignment under a fixed matching objective.

To address baseline coverage and backbone consistency, we evaluated ProbVLM and PCME++ with ViT-L/14 on the same 5000-image MSCOCO pool, using center-cosine scoring and each method's training recipe. Table R1.2 reports the additional results.

Table R1.2. Additional ViT-L/14 probabilistic baselines on MSCOCO (set R@1, %).

ModelI2T R@1T2I R@1
ProbVLM62.1050.60
PCME++87.3683.98

The additional evaluation includes both ProbVLM and PCME++ with a ViT-L/14 backbone. This expands the comparison with established probabilistic methods and addresses your concern about baseline coverage.

ProLIP backbone control. Separately, the newly trained ViT-L/14 ProLIP variant obtains 34.46/32.74 under the supplementary cosine protocol. We distinguish this variant from the paper's ViT-H/14 ProLIP-FT checkpoint by reporting its backbone and training source.

AuthorsOpenReview ↗

Response to Reviewer mVpn (Part 2/2)

5. Individual Component Ablations on Caption-Set Retrieval (W4)

To address your request for component ablations, we removed each alignment term, all alignment terms together, and the low-rank covariance factor. The ablation runs use MSCOCO, ViT-L/14, 10 epochs, and seed 42, with evaluation on 11814 training-unseen images and five captions per image. Table R1.3 includes the full MCDisp-Align results from the original draft for comparison.

Table R1.3. Component-wise ablations under the supplementary caption-set protocol (%).

ConfigurationI2T R@1T2I R@1
Full MCDisp-Align85.2083.90
w/o all alignment terms66.2264.99
w/o KL alignment67.6367.51
w/o caption-center coverage65.7965.80
w/o subspace alignment67.1866.61
w/o variance regularization62.2262.85
Diagonal-only covariance (r=0r = 0)67.0566.73

The full MCDisp-Align achieves higher I2T/T2I R@1 than all listed ablation variants in Table R1.3. This demonstrates the importance of the alignment components and low-rank covariance for distribution alignment.

Table R1.3 reports individual supplementary outcomes. The full-model 85.20/83.90 result from the original draft is included for reference under the original protocol. Stop-gradient is held fixed, with its training role clarified in W1.

6. Closing Summary

Overall, the added controls address caption-set supervision and baseline coverage, while the diagnostics and separate ablations clarify the distribution parameters' roles. The original within-model comparison establishes the retrieval contribution of alignment. We will incorporate these results and clarify the protocols in the revised manuscript. Thank you again for your constructive suggestions.

Thank you for the rebuttal and the revised manuscript. Rather than resolving my original concerns, the newly reported results introduce serious cross-protocol inconsistencies and directly contradict the paper's central methodological and empirical claims.

  1. Invalid cross-protocol comparison in Table R1.3 and self-refuting matched ablations. In Table R1.3, you place the original-protocol score of Full MCDisp-Align (85.20 / 83.90) directly above six ablated variants evaluated under the supplementary 11,814-image protocol to claim that every alignment term and the low-rank covariance factor substantially improve retrieval. Comparing results across incompatible evaluation protocols is scientifically invalid, and in Revised Table 8 the full-model row is omitted altogether. Both Table 8 and Table 9 use the exact same settings. In Table 9, the full five-caption model of Full MCDisp-Align with K=5K=5 obtains only 0.6522 / 0.6536 I2T/T2I R@1. Comparing this matched full model against Table 8 reveals that removing nearly any proposed component improves performance. Thus, under the controlled 11,814-image protocol, the proposed KL alignment, coverage constraint, subspace alignment, and low-rank covariance all fail to help, and in fact hurt, caption-set retrieval. Moreover, the requested stop-gradient ablation and component-wise ablations on standard retrieval remain unaddressed.
  1. The revision scatters MSCOCO evaluations across multiple incompatible protocols without providing a single unified comparison across all models. Specifically, the proposed method suffers an overall defeat on the only unified baseline control in Table 10. In Appendix G, where CLIP-FT, CLIP-FT (SET), and MCDisp-Align are actually evaluated under one shared protocol in Table 10, MCDisp-Align ranks last overall with scores of 94.25 and 92.75. Furthermore, there is a clear cross-protocol contradiction between Table R1.1 and Table 7. Table R1.1 directly compares MCDisp-Align (85.20 / 83.90), against the CLIP-FT (SET) (83.76 / 81.44), to claim superiority. Yet Revised Table 7 explicitly separates them into different blocks because the evaluation protocols differ, and it omits MCDisp-Align from the 5,000-image center-cosine protocol altogether. Finally, MCDisp-Align is surpassed by PCME++ on the same ViT-L/14 backbone. In Table R1.2 and Revised Table 7, PCME++ with ViT-L/14 achieves 87.36 and 83.98 on the 5,000-image MSCOCO pool, exceeding the original-protocol scores of MCDisp-Align at 85.20 and 83.90 in both directions. Despite this, the Abstract and Conclusion remain completely unchanged and still claim the best or tied-best one-to-many retrieval scores among all compared methods.

Because the matched ablations in Tables 8 and 9 show that the proposed alignment terms and low-rank covariance degrade performance, the only unified baseline comparison in Table 10 shows MCDisp-Align losing overall to both CLIP-FT (SET), and PCME++ surpasses it on the same ViT-L/14 backbone, the core claims of the paper are not supported by the evidence. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.