acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Set Representations with Partial Optimal Transport for Image–Text Retrieval

Abstract

Image–text retrieval involves incomplete semantic correspondence: an image may contain multiple entities and contextual details, while its caption describes only a subset. Capturing this structure requires both semantically distinct representations and a matching mechanism that accommodates unmentioned content. We propose a framework that independently represents each image and caption as a weighted set of von Mises–Fisher components. Each component is parameterized by a direction, a concentration, and a mixture weight, and cross-modal compatibility is computed through analytic distributional overlap. To encourage distinct semantic roles, we use limited entity annotations during training to supervise component support and cross-modal identity through permutation-invariant assignments. We then perform asymmetric partial alignment using optimal transport with an image-side dustbin, allowing a prescribed fraction of image mass to remain unmatched while preserving the relative coverage of caption components. Entity annotations are used only for training; inference requires only the full image and caption, with no entity inputs or auxiliary grounding modules. We evaluate retrieval alongside component localization, entity correspondence, and deletion-based selectivity to examine whether the learned components support meaningful partial alignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Image–Text Retrieval, von Mises–Fisher Mixture Representations, Entity-Guided Semantic Learning, Asymmetric Partial Alignment, Optimal Transport.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8g9QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a probabilistic, set-based image–text retrieval framework that models partial correspondence between an image and a caption. Rather than representing each modality with a single embedding, the method independently encodes images and texts as sets of weighted von Mises–Fisher components, each with a direction, concentration, and learned transport mass. Cross-modal component compatibility is defined by analytic distributional overlap, while one-sided partial optimal transport determines how these components contribute to the final retrieval score. A fixed matched-mass ratio allows part of the image mass to remain unmatched through a text-side dustbin, reflecting the fact that a caption may describe only a subset of the visible scene. The paper also studies an optional entity-supervised extension using region–phrase annotations. Experiments evaluate retrieval performance, structural design choices, caption-coverage effects, transport allocation, and component behavior. The main empirical findings show that the complete model underperforms PCME++ in retrieval, that low-coverage captions do not exhibit a statistically established larger benefit from partial OT, and that improved local entity support does not necessarily translate into larger transport participation or retrieval gains.

Critical strengths

The main strengths of the paper are that (i) it presents a conceptually well-motivated formulation of partial image–text correspondence, and (ii) it provides careful analyses that distinguish local representation quality from actual participation in retrieval scoring.

First, the paper addresses a real mismatch in image–text retrieval: captions often describe only a subset of the content present in an image. The proposed framework models this explicitly by separating component compatibility from participation in matching. Image and text inputs are represented as sets of probabilistic vMF components; distributional overlap determines component compatibility, and learned masses determine how much each component can participate in the transport plan. The one-sided partial OT formulation further allows some image mass to remain unmatched rather than forcing every visual component to correspond to the caption. This is a clean and conceptually meaningful formulation of the caption-scope problem.

Second, the paper goes beyond aggregate retrieval metrics and examines how different parts of the framework actually behave. In particular, the entity-supervision experiments reveal an interesting dissociation: spatial supervision increases text support from 4.07% to 64.31%, yet the corresponding GT-slot mass and correct-edge mass decrease substantially. This suggests that improving what a component represents does not necessarily make that component more influential in the final retrieval score. The paper is also commendably explicit about inconclusive results, including the lack of established evidence that partial OT helps low-coverage captions more than high-coverage captions. These analyses provide useful insight into the interaction between representation, correspondence, and scoring.

Critical weaknesses

The central empirical weakness is that the experiments do not convincingly establish the practical benefit of the proposed partial-matching mechanism. The complete model performs worse than the independently re-evaluated PCME++ baseline on every reported retrieval metric, including mean R@1 gaps of approximately 4.8 points on COCO 5K and CxC. More importantly, the paper’s main motivation is that partial OT should be useful when a caption covers only part of an image, but the caption-coverage experiment does not support this hypothesis. On fixed representations, partial versus full OT differs by only +0.26 R@1 for low-coverage captions, with a confidence interval that includes zero, and no larger low-coverage-specific advantage is established. The unmatched-mass sweep is also non-monotonic, and the prescribed 25% unmatched capacity is not empirically optimal. Together, these results make it difficult to conclude that partial matching itself is the source of meaningful retrieval gains.

A second concern is that several central modeling assumptions remain insufficiently validated. The method relies on multiple latent probabilistic components and a globally fixed matched-mass ratio, yet the experiments do not establish that either choice is necessary. The component-count study evaluates only K = 2, 4, and 8; K = 1 is omitted, so the experiments do not demonstrate that a set representation is actually superior to a single-component probabilistic representation. In fact, increasing K to 8 substantially reduces retrieval performance. Similarly, the matched-mass ratio ρ is fixed globally rather than adapted to the semantic coverage of each caption, even though varying caption scope is the primary motivation for the method. Finally, most training comparisons use a single seed, and the reported bootstrap intervals capture sample variation but explicitly exclude training-seed and model-selection uncertainty. These limitations make the robustness and necessity of the central design choices difficult to assess.

Questions for authors

Can the authors more directly demonstrate that partial OT is beneficial because of partial caption coverage? The main motivation is that captions often describe only a subset of image content, yet the coverage-stratified experiment does not show a statistically significant larger benefit for low-coverage captions. Could the authors train full-OT and partial-OT models independently under the same protocol and compare them separately within low-, medium-, and full-coverage subsets? The current fixed-representation score-switching analysis may underestimate the effect of learning representations jointly with partial matching.

Why is the matched-mass ratio fixed globally rather than adapted to each image–caption pair? Since captions can vary substantially in semantic coverage, a single value such as ρ = 0.75 seems only indirectly connected to the motivating problem. Could the authors estimate ρ from the caption/image representations or learn it jointly with the model? A comparison between fixed and adaptive matched mass would help determine whether pair-specific partial correspondence is important.

Can the authors include K = 1 in the component-count ablation? The current sweep over K = 2, 4, and 8 does not establish that multiple components are necessary, and K = 8 performs substantially worse. A K = 1 probabilistic baseline under the same scoring and training protocol would clarify whether the set representation itself provides value beyond the probabilistic embedding formulation.

How robust are the results across training seeds? The complete model, structural ablations, and entity-supervised variants are primarily reported from single-seed runs, while the bootstrap intervals capture only fixed-checkpoint sample variation. Could the authors provide mean and standard deviation across several training seeds for the main retrieval comparison and the key partial-versus-full OT ablations? This is particularly important because many observed differences are small.

Can the authors provide a more controlled comparison against PCME++ or other trained probabilistic baselines? The paper correctly notes that the current PCME++ comparison uses a different optimizer, schedule, augmentation strategy, and auxiliary objectives, which prevents attributing the performance gap to the proposed representation or matching mechanism. A matched training protocol would make it easier to isolate the contribution of probabilistic sets and partial OT.

Could the authors directly supervise or regularize transport participation in the entity-supervised extension? Spatial supervision dramatically improves local support but causes GT-slot and correct-edge masses to decrease. Since the paper’s main conceptual distinction is between content support and matching participation, adding a loss that explicitly encourages transport mass on entity-aligned components could test whether the observed dissociation is merely an optimization artifact or an inherent property of the formulation.

What is the practical computational cost of using OT at retrieval time? Although image and text component sets can be precomputed independently, every candidate image–text pair still requires a component compatibility matrix and a transport solve. Could the authors report retrieval latency or throughput relative to single-vector or simpler set-based scoring methods? This would help assess whether the additional matching flexibility is practical for large-scale retrieval.

Reviewer_DnGKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper is about text-to-image retrieval where the query and the images are represented by sets/sequences of tokens. The claim is that local matching and partial matching are required, therefore the authors propose representing text/image examples by a set of latent components formulated an optimal transport problem. Its formulation allows partial matching and its result is used to estimate the final similarity for retrieval.

Critical strengths

The paper has a very clear formulation. They separating compatibility (closed-form vMF log-overlap) from participation (masses) and combine them through OT with dustbins to allow for one-sided partial matching. This is principled and well motivated by the caption-scope problem.

The paper is very well written and a pleasure to follow. Maybe sometimes a bit too dense though.

Focusing on local matching and on partial support is a key in the task and it makes sense to focus on this.

Critical weaknesses

W1 - There are no comparisons with the closest methods. The following are discussed but are not part of experiments: OMIT, DivE, PVSE and MaxMatch.

W2 - The only comparison with a SoA method/score is in Table 1, with a single approach, and the achieved performance is inferior. I do not understand what is the point the author are trying to make here, and their comment in that paragraph is just hand-wavy.

W3 - Poor evaluation settings: only CLIP ViT-B/32 and COCO/Flickr are used, there are not larger backbones or modern VLM encoders.

W4 - The residual contribution compared to existing OT-based methods is never discussed. I have not really identified myself (off-the-shelf I am not familiar in detail with the existing methods) and therefore I identify the novelty as weak.

W5 - estimating the components is a critical part of the method but details about it are left only for the supplementary.

W6 - the approach is not scalable, while a 2 stage approach with full efficient ranking (eg global descriptors) and re-ranking with OT is not discussed.

Questions for authors

Major reasons for rejecting are W1, W2 and W4. The authors should address 1,2 with experiments and 4 with justification. More minor are W3,W5, W6: experiments for W3 would be appreciated, and possibly for W6.

Reviewer_NxyPOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies image–text retrieval when a caption describes only part of the image. The authors represent images and texts as sets of weighted von Mises–Fisher components. Component compatibility is computed using distributional overlap, while learned component masses and one-sided partial optimal transport determine how much each component contributes to the matching score. A fixed matched-mass ratio allows part of the image representation to remain unmatched.

The paper evaluates retrieval performance on COCO, CxC, ECCV Caption, and Flickr30k, and analyzes the effects of component count, unmatched mass, caption coverage, and entity supervision. The results show that the proposed system underperforms PCME++ under different training protocols. The analyses also find that better local content support does not necessarily lead to greater matching participation or improved retrieval performance.

Critical strengths
  1. The paper addresses a meaningful aspect of image–text retrieval: a caption may describe only part of an image, so unmentioned visual content should not necessarily be treated as irrelevant. The proposed framework makes a useful distinction between component compatibility and matching participation. In particular, the combination of probabilistic overlap, learned component masses, and partial transport provides a way to examine how different components contribute to the retrieval score.

  2. The experimental analysis is relatively careful about its limitations. The authors report that the proposed method does not outperform PCME++, acknowledge that the current component-count study cannot establish the benefit of multiple components, and do not claim a low-coverage advantage when the confidence intervals fail to support one. The entity-supervision experiments also reveal an interesting result: improving local content support can coincide with a reduction in matching participation. I consider this analysis useful, even though the method's retrieval performance is not competitive with the main baseline.

Critical weaknesses
  1. The experiments do not sufficiently establish the benefit of the proposed representation and matching design.

The framework combines probabilistic set representations, learned component masses, and one-sided partial OT, but the evidence for several key design choices is limited. The component-count study only considers K = 2, 4, and 8, without a K = 1 baseline. It is therefore unclear whether multiple probabilistic components are necessary.

The structural ablation in Table 2 also shows small and inconsistent gains from the unmatched channel and multilayer routing. For example, the combined configuration improves COCO 5K R@1 from 40.446 to 40.730 over the base OT setting, while ECCV Caption mAP@R decreases. All training comparisons use a single random seed.

The fixed matched-mass ratio is another important limitation. Although partial matching is motivated by captions describing different amounts of image content, the model uses the same ratio for all image–text pairs. Table 3 does not establish a larger benefit for low-coverage captions. The paper acknowledges this limitation, but the current evidence still leaves the practical advantage of the proposed partial-matching design unclear.

  1. The distinct contribution over prior work and the retrieval effectiveness are not sufficiently demonstrated.

Several elements of the framework have appeared in previous work. PCME and PCME++ investigate probabilistic image–text representations, PVSE and DivE use multiple embeddings, and OMIT already applies partial optimal transport to image–text matching. The proposed combination of vMF components, learned masses, and one-sided partial transport is different from these approaches, but its specific advantage is not sufficiently established experimentally.

Table 1 shows that the complete model performs below PCME++ on every reported metric. The authors correctly acknowledge that the training protocols differ, so this result does not isolate the effect of the proposed representation or transport mechanism. However, it also does not demonstrate a competitive retrieval advantage.

More targeted comparisons are needed to show whether the probabilistic components and learned mass allocation improve retrieval over simpler representations or alternative partial-matching formulations under comparable training conditions.

Questions for authors
  1. Could the authors report K = 1 results under the same training and evaluation protocol? This would help establish whether representing each modality with multiple probabilistic components provides an advantage over a single-component representation.

  2. Could the authors provide a more controlled comparison of partial OT, full OT, and alternative component representations, preferably with results across multiple training seeds? In particular, it would be useful to isolate the contributions of probabilistic compatibility and learned component masses, rather than evaluating only their combined effect.

  3. What is the rationale for using a fixed matched-mass ratio of ρ = 0.75 for all image–text pairs? Given that the coverage analysis does not establish a larger benefit for low-coverage captions, have the authors considered whether a caption-dependent ratio would be more appropriate?

  4. How does the proposed method compare with a simpler or more directly comparable partial-matching formulation, such as OMIT, under a common backbone and training setup? This would help clarify the contribution of the learned probabilistic components and mass constraints beyond existing partial-OT approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.