- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a volume-free, distance-based objective for learning box-to-box relations, specifically intersection, separation, and directed containment, in image-text representation learning. Hard intersection volume gives zero gradients for disjoint boxes, while the analyzed product-form smoothed-volume surrogate has gradients that can decay rapidly with accumulated separation and the number of separated dimensions. The experiments show improved cross-modal retrieval performance and also evaluate whether positive and negative pairs become geometrically joint or disjoint. While this paper further reports results on CxC human semantic-similarity judgments and set-theoretic recommendation queries, these evaluations do not establish whether the learned containment direction corresponds to human judgments of semantic entailment or informational scope, nor whether the number of intersecting boxes constitutes a calibrated measure of uncertainty.
Critical strengths
Directly Addressing Multiplicative Gradient Attenuation:
The formulation addresses the multiplicative gradient attenuation of volume-based box objectives by replacing coordinate-wise products with a norm for intersection/separation and an average for containment. The theoretical analysis provides dimension-dependent gradient bounds without the exponential product attenuation analyzed for the smoothed-volume objective. The experiments at provide empirical evidence that the resulting objective can learn useful box relations under the evaluated settings.
Interpretable Relation-Specific Objectives:
The mapping from training assignments to geometric adjustments is direct: same-index image-text pairs are encouraged to intersect, different-index pairs are encouraged to separate, and positive pairs are additionally subjected to a fixed image-inside-text inclusion objective. Signed gaps and containment violations make the optimization mechanism easier to inspect than an objective expressed only through intersection volume. The P-Joint and N-Joint analyses provide useful evidence that the intended joint/disjoint geometry is more reliably learned than by the evaluated box baselines.
Critical weaknesses
Potential Semantic Mismatch in In-Batch Negative Pairing:
The intersection objective treats every different-index image-text pair in a batch as a negative pair and pushes it toward geometric separation. This is potentially too restrictive for datasets such as COCO, where captions or images from different annotated pairs may still be semantically compatible. Because intersection is interpreted as semantic correspondence, false negatives affect not only ranking supervision but also the claimed geometry.
Unverified Semantic Validity of the Fixed Containment Direction:
The paper applies , to every positive pair, following prior work. It is unclear whether this fixed direction corresponds to a consistent semantic relation in caption datasets. A caption may omit visible content, while also introducing attributes or relations that are not visually verifiable. Although adding the inclusion loss improves retrieval, the current experiments do not distinguish semantic entailment from alternative explanations such as regularization or systematic control of relative box sizes. These comparisons are needed to determine whether the gain comes from containment direction or from box-size regularization.
The inclusion loss can be reduced by enlarging the outer text box, shrinking the inner image box, or moving their boundaries into containment. As the reported P-Joint/N-Joint trade-off as the inclusion weight increases implies that stronger inclusion can also increase unintended intersections, it would be useful to report box-width distributions, volume or width statistics by modality, and controls that match box sizes without imposing semantic containment.
Questions for authors
Handling False Negatives:
How sensitive is the learned joint/disjoint geometry to false negatives introduced by treating all different-index pairs as negatives? Have you considered pseudo-positive mining, multiple-positive supervision, or soft semantic-relevance targets? In particular, could the CxC similarity annotations be used for training, rather than only evaluation, to test whether soft relevance supervision reduces geometrically disjoint but semantically plausible pairs?
Evaluation of Semantic Entailment:
Did you evaluate whether the learned containment relation correlates with human or model-based judgments of caption specificity, visual entailment, or informational scope? Please also report controlled comparisons among , , bidirectional containment, and intersection-only training.
Interpretation and Calibration of Uncertainty:
The proposed distances do not retain the conditional-volume interpretation used in probabilistic box embeddings. What interpretation should therefore be assigned to “matching uncertainty,” which is measured by the number of intersecting boxes? Have you evaluated whether this quantity is calibrated against empirical ambiguity, multiple human-approved matches, or prediction error? If downstream applications require conditional probabilities, would an additional calibration model be necessary?
6 comments
MHYv-W2. Unverified Semantic Validity of the Fixed Containment Direction.
Our image-within-text direction is adopted as an inductive bias, rather than a strict semantic entailment for every image–caption pair. As described in Sec. 3.3 (Lines 245–246), this choice follows prior works [1, 2, 3] that model captions as generally less specific than their paired images. To validate whether this assumption is meaningful in practice, we examine whether the containment direction matters, whether the gain can instead be explained by box-size regularization, and whether the learned containment is consistent with the semantic hierarchies within both modalities.
Experimental Setup. To this end, we compare our original model, which uses forward inclusion (image-within-text), with a reverse-inclusion variant (text-within-image) and a box-size regularization variant. For the box-size regularization variant, we consider two settings: one matches the target box sizes to those of our original model (size-matched), while the other only encourages image boxes to be smaller than text boxes (unmatched). We evaluate all models on COCO-1K, COCO-5K, and CxC, and report their box-size statistics for each modality.
Results. As shown in Tab. R3-W2a, forward inclusion outperforms reverse inclusion across all benchmarks, indicating that the direction matters. The size-matched model achieves comparable, though slightly lower, performance than ours, indicating that, as the reviewer suggested, the retrieval gain largely stems from the size asymmetry induced by the inclusion loss. However, the size-matched model relies on target sizes taken from the box sizes already learned by our model. When only the asymmetry is enforced without these learned targets (unmatched), performance drops on all benchmarks (Tab. R3-W2a and R3-W2b), whereas the inclusion loss learns the asymmetry without such targets. Moreover, on HierarCaps [4], the containment learned by our model closely follows both the human-verified caption specificity and the visual information hierarchy of nested image crops (Kendall’s for caption specificity and visual entailment = 0.90 and 0.82 for ours, vs. −0.75 and 0.62 for size-matched and 0.88 and −0.35 for unmatched; see Tab. R3-Q2). Together, these results show that our inclusion loss learns, without target sizes, both the cross-modal size asymmetry that contributes to the retrieval gains and the within-modality containment that box sizes alone cannot express. We will incorporate these results into the revised manuscript.
Table R3-W2a. Semantic alignment performance of forward, reverse inclusion, and box-size regularization. Each cell reports R@1 (I2T, T2I) / RSUM.
| Models | COCO-1K (R@1 / RSUM) | COCO-5K (R@1 / RSUM) | CxC (R@1 / RSUM) |
|---|---|---|---|
| Ours (forward inclusion) | 82.7, 70.0 / 539.0 | 63.8, 49.4 / 458.0 | 65.4, 51.3 / 467.7 |
| Ours (reverse inclusion) | 81.4, 69.1 / 536.1 | 61.6, 47.8 / 451.4 | 63.4, 49.7 / 461.6 |
| Box-size regularization (unmatched) | 81.1, 69.1 / 536.0 | 61.5, 47.8 / 450.5 | 63.0, 49.8 / 460.5 |
| Box-size regularization (size-matched) | 82.3, 69.4 / 538.3 | 63.4, 48.6 / 456.3 | 64.8, 50.4 / 465.4 |
Table R3-W2b. Box-size statistics (mean std) for forward, reverse inclusion, and box-size regularization on COCO-5K.
| Models | Image width (mean std) | Text width (mean std) |
|---|---|---|
| Ours (forward inclusion) | 0.103 0.04 | 1.134 0.43 |
| Ours (reverse inclusion) | 0.747 0.38 ( forward model) | 0.096 0.02 ( forward model) |
| Box-size regularization (unmatched) | 0.096 0.02 ( forward model) | 2.114 0.77 ( forward model) |
| Box-size regularization (size-matched) | 0.102 0.03 ( forward model) | 1.135 0.37 ( forward model) |
-[1] Desai, Karan, et al. "Hyperbolic image-text representations." International Conference on Machine Learning. PMLR, 2023.
-[2] Ramasinghe, Sameera, et al. "Accept the modality gap: An exploration in the hyperbolic space." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
-[3] Pal, Avik, et al. "Compositional entailment learning for hyperbolic vision-language models." International Conference on Learning Representations. 2025.
- [4] Alper, Morris, and Hadar Averbuch-Elor. "Emergent visual-semantic hierarchies in image-text representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
MHYv-Q1. Handling False Negatives.
We have addressed this in our response to MHYv-W1.
MHYv-Q2. Evaluation of Semantic Entailment.
Following the reviewer’s suggestion, we additionally evaluate whether the learned containment relations capture hierarchical relationships within each modality. Our model achieves the highest correlation with both caption specificity and visual information hierarchy, compared with models trained with reverse containment (text-within-image), bidirectional containment, intersection-only objectives, or box-size regularization.
Experimental Setup. We use HierarCaps [4], a benchmark that provides four levels of captions for each image, ranging from general (i.e., low specificity) to specific descriptions (i.e., high specificity), with a human-verified ordering. We conduct two evaluations. For caption specificity, we use the human-verified caption ordering as the reference. For visual entailment, we construct four nested crops of each HierarCaps image and use the ordering by the amount of visual information as the reference, where a crop with less visual information is regarded as more general. For each model, we derive an ordering from its containment relations by comparing the containment degrees between two boxes in each direction and assigning the hierarchical order based on the stronger containment. We then measure the agreement between the model-derived and reference orderings using Kendall’s tau [5]. Finally, we compare our model trained with image-within-text inclusion (ours) against models trained with reverse containment, bidirectional containment, intersection-only training, and box-size regularization (size-matched and unmatched; see MHYv-W2).
Results. As shown in Tab. R3-Q2, our model achieves the highest correlation for both caption specificity ( = 0.90) and visual entailment ( = 0.82). Reverse and bidirectional containment yield substantially lower correlations for visual entailment ( = 0.18 and −0.21), suggesting that the direction of containment plays an important role in capturing the hierarchy of visual information. Intersection-only training, which imposes no containment, also falls short of our model, particularly for visual entailment ( = 0.56). Notably, both box-size regularization models show substantially lower correlations than ours on at least one of the two hierarchies, indicating that these hierarchies are not captured by box sizes alone, even when the sizes are matched to those learned by our model. Overall, these results show that the learned containment of our model is the most consistent with both the hierarchy of caption specificity and that of visual information.
Table R3-Q2. Kendall’s between learned containment orderings and reference orderings, given by the HierarCaps caption hierarchy for caption specificity and by nested image crops for visual entailment. Box size reg. replaces the inclusion loss with a size prior, where the matched variant uses our learned box sizes as targets and the unmatched variant only enforces image text.
| Kendall () | Ours (forward) | Ours (reverse) | Bidirectional (both) | Intersection-only | Size regularization (size-matched) | Size regularization (unmatched) |
|---|---|---|---|---|---|---|
| Caption specificity | 0.90 | 0.68 | 0.79 | 0.84 | -0.75 | 0.88 |
| Visual entailment | 0.82 | 0.18 | -0.21 | 0.56 | 0.62 | -0.35 |
- [5] Kendall, Maurice G. "A new measure of rank correlation." Biometrika 30.1-2 (1938): 81-93.
Author Response to Reviewer Comments
MHYv-W1. Does in-batch negative pairing separate semantically compatible pairs and distort the learned geometry?
We understand the reviewer’s concern that treating every different-index image–text pair as a negative may push semantically compatible pairs toward geometric separation, affecting not only ranking supervision but also the claimed geometry. To directly examine this concern, we conduct a controlled evaluation under our COCO setup, testing whether semantically compatible different-index pairs can still be learned as intersecting boxes despite in-batch negative pairing. If such supervision distorted the claimed geometry, these pairs would tend to become disjoint. Conversely, if the geometry is preserved, it should yield intersections for semantically compatible pairs while keeping incompatible pairs disjoint.
Setup. We conduct two complementary evaluations using human relevance annotations from CxC following the reviewer’s suggestion.
Geometric results (Tab. R3-W1a). Our original model achieves 89.0% Relevant Joint and 90.3% Irrelevant Disjoint on CxC. These results show that treating different-index pairs as negatives during training does not prevent the model from learning intersections for semantically compatible pairs, while preserving separation for incompatible pairs. The FN model with 16 injected pairs maintains similarly high rates (91.6% and 88.8%, respectively). With 32 injected FN pairs, Relevant Joint increases to 97.3% but Irrelevant Disjoint decreases to 60.9%, indicating reduced geometric discrimination only under heavier FN contamination.
Retrieval results (Tab. R3-W1b). The PP models, which explicitly treat semantically compatible pairs as positives, do not consistently improve retrieval over our original model. Moreover, introducing 16 FN pairs causes only modest performance drops (at most 1.6 RSUM), while larger degradations occur with 32 pairs. These findings provide complementary evidence that the ranking supervision remains reasonably robust to moderate false-negative contamination.
Overall, these experiments show that in-batch negative pairing does not inherently distort the learned geometry. Our model learns selective intersections for semantically compatible pairs, although excessive false-negative supervision can degrade this discrimination. We will include these results and clarify the distinction between training supervision and the learned geometry in the revised manuscript. We also appreciate the reviewer’s suggestion of soft relevance supervision and will discuss it as a future direction.
Table R3-W1a. Comparison of joint/disjoint geometry on the CxC test set, showing the proportions (%) of relevant and irrelevant pairs that are joint or disjoint.
Table R3-W1b. Comparison of semantic correspondence on COCO-1K, COCO-5K, and CxC. Each cell reports R@1 (I2T, T2I) / RSUM.