- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a probabilistic, set-based image–text retrieval framework that models partial correspondence between an image and a caption. Rather than representing each modality with a single embedding, the method independently encodes images and texts as sets of weighted von Mises–Fisher components, each with a direction, concentration, and learned transport mass. Cross-modal component compatibility is defined by analytic distributional overlap, while one-sided partial optimal transport determines how these components contribute to the final retrieval score. A fixed matched-mass ratio allows part of the image mass to remain unmatched through a text-side dustbin, reflecting the fact that a caption may describe only a subset of the visible scene. The paper also studies an optional entity-supervised extension using region–phrase annotations. Experiments evaluate retrieval performance, structural design choices, caption-coverage effects, transport allocation, and component behavior. The main empirical findings show that the complete model underperforms PCME++ in retrieval, that low-coverage captions do not exhibit a statistically established larger benefit from partial OT, and that improved local entity support does not necessarily translate into larger transport participation or retrieval gains.
Critical strengths
The main strengths of the paper are that (i) it presents a conceptually well-motivated formulation of partial image–text correspondence, and (ii) it provides careful analyses that distinguish local representation quality from actual participation in retrieval scoring.
First, the paper addresses a real mismatch in image–text retrieval: captions often describe only a subset of the content present in an image. The proposed framework models this explicitly by separating component compatibility from participation in matching. Image and text inputs are represented as sets of probabilistic vMF components; distributional overlap determines component compatibility, and learned masses determine how much each component can participate in the transport plan. The one-sided partial OT formulation further allows some image mass to remain unmatched rather than forcing every visual component to correspond to the caption. This is a clean and conceptually meaningful formulation of the caption-scope problem.
Second, the paper goes beyond aggregate retrieval metrics and examines how different parts of the framework actually behave. In particular, the entity-supervision experiments reveal an interesting dissociation: spatial supervision increases text support from 4.07% to 64.31%, yet the corresponding GT-slot mass and correct-edge mass decrease substantially. This suggests that improving what a component represents does not necessarily make that component more influential in the final retrieval score. The paper is also commendably explicit about inconclusive results, including the lack of established evidence that partial OT helps low-coverage captions more than high-coverage captions. These analyses provide useful insight into the interaction between representation, correspondence, and scoring.
Critical weaknesses
The central empirical weakness is that the experiments do not convincingly establish the practical benefit of the proposed partial-matching mechanism. The complete model performs worse than the independently re-evaluated PCME++ baseline on every reported retrieval metric, including mean R@1 gaps of approximately 4.8 points on COCO 5K and CxC. More importantly, the paper’s main motivation is that partial OT should be useful when a caption covers only part of an image, but the caption-coverage experiment does not support this hypothesis. On fixed representations, partial versus full OT differs by only +0.26 R@1 for low-coverage captions, with a confidence interval that includes zero, and no larger low-coverage-specific advantage is established. The unmatched-mass sweep is also non-monotonic, and the prescribed 25% unmatched capacity is not empirically optimal. Together, these results make it difficult to conclude that partial matching itself is the source of meaningful retrieval gains.
A second concern is that several central modeling assumptions remain insufficiently validated. The method relies on multiple latent probabilistic components and a globally fixed matched-mass ratio, yet the experiments do not establish that either choice is necessary. The component-count study evaluates only K = 2, 4, and 8; K = 1 is omitted, so the experiments do not demonstrate that a set representation is actually superior to a single-component probabilistic representation. In fact, increasing K to 8 substantially reduces retrieval performance. Similarly, the matched-mass ratio ρ is fixed globally rather than adapted to the semantic coverage of each caption, even though varying caption scope is the primary motivation for the method. Finally, most training comparisons use a single seed, and the reported bootstrap intervals capture sample variation but explicitly exclude training-seed and model-selection uncertainty. These limitations make the robustness and necessity of the central design choices difficult to assess.
Questions for authors
Can the authors more directly demonstrate that partial OT is beneficial because of partial caption coverage? The main motivation is that captions often describe only a subset of image content, yet the coverage-stratified experiment does not show a statistically significant larger benefit for low-coverage captions. Could the authors train full-OT and partial-OT models independently under the same protocol and compare them separately within low-, medium-, and full-coverage subsets? The current fixed-representation score-switching analysis may underestimate the effect of learning representations jointly with partial matching.
Why is the matched-mass ratio fixed globally rather than adapted to each image–caption pair? Since captions can vary substantially in semantic coverage, a single value such as ρ = 0.75 seems only indirectly connected to the motivating problem. Could the authors estimate ρ from the caption/image representations or learn it jointly with the model? A comparison between fixed and adaptive matched mass would help determine whether pair-specific partial correspondence is important.
Can the authors include K = 1 in the component-count ablation? The current sweep over K = 2, 4, and 8 does not establish that multiple components are necessary, and K = 8 performs substantially worse. A K = 1 probabilistic baseline under the same scoring and training protocol would clarify whether the set representation itself provides value beyond the probabilistic embedding formulation.
How robust are the results across training seeds? The complete model, structural ablations, and entity-supervised variants are primarily reported from single-seed runs, while the bootstrap intervals capture only fixed-checkpoint sample variation. Could the authors provide mean and standard deviation across several training seeds for the main retrieval comparison and the key partial-versus-full OT ablations? This is particularly important because many observed differences are small.
Can the authors provide a more controlled comparison against PCME++ or other trained probabilistic baselines? The paper correctly notes that the current PCME++ comparison uses a different optimizer, schedule, augmentation strategy, and auxiliary objectives, which prevents attributing the performance gap to the proposed representation or matching mechanism. A matched training protocol would make it easier to isolate the contribution of probabilistic sets and partial OT.
Could the authors directly supervise or regularize transport participation in the entity-supervised extension? Spatial supervision dramatically improves local support but causes GT-slot and correct-edge masses to decrease. Since the paper’s main conceptual distinction is between content support and matching participation, adding a loss that explicitly encourages transport mass on entity-aligned components could test whether the observed dissociation is merely an optimization artifact or an inherent property of the formulation.
What is the practical computational cost of using OT at retrieval time? Although image and text component sets can be precomputed independently, every candidate image–text pair still requires a component compatibility matrix and a transport solve. Could the authors report retrieval latency or throughput relative to single-vector or simpler set-based scoring methods? This would help assess whether the additional matching flexibility is practical for large-scale retrieval.