acceptodds
Under review as a conference paper at ICLR 2027

Learning Box Relations as Distances for Multi-modal Representation

Abstract

Multimodal representation learning seeks to capture semantic correspondence beyond pairwise similarity, including intersection, inclusion, and separation. Point embeddings model correspondence through proximity, whereas box embeddings can encode these relations and compose them into more complex structures. Existing box methods, however, primarily rely on volume-based objectives whose coordinatewise products attenuate learning signals as dimensionality or box separation increases. We introduce a new volume-free, distance-based objective that learns box-to-box relations through intersection-aware and directed inclusion losses. Our objective avoids this attenuation and theoretically guarantees box intersection and inclusion. Experiments show that our method reliably learns joint, disjoint, and inclusion relations, achieves the best cross-modal retrieval performance on COCO-1K, COCO-5K, and RedCaps, and shows the closest alignment with human semantic similarity on CxC. On MovieLens-1M, our learned relations compose to answer set-theoretic queries, including combinations not observed during learning. Together, these results establish distance-based box relation learning as a reliable foundation for multimodal correspondence and set-theoretic reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
box embedding, multi-modal representation, image-text alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MHYvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a volume-free, distance-based objective for learning box-to-box relations, specifically intersection, separation, and directed containment, in image-text representation learning. Hard intersection volume gives zero gradients for disjoint boxes, while the analyzed product-form smoothed-volume surrogate has gradients that can decay rapidly with accumulated separation and the number of separated dimensions. The experiments show improved cross-modal retrieval performance and also evaluate whether positive and negative pairs become geometrically joint or disjoint. While this paper further reports results on CxC human semantic-similarity judgments and set-theoretic recommendation queries, these evaluations do not establish whether the learned containment direction corresponds to human judgments of semantic entailment or informational scope, nor whether the number of intersecting boxes constitutes a calibrated measure of uncertainty.

Critical strengths

Directly Addressing Multiplicative Gradient Attenuation:

The formulation addresses the multiplicative gradient attenuation of volume-based box objectives by replacing coordinate-wise products with a norm for intersection/separation and an average for containment. The theoretical analysis provides dimension-dependent gradient bounds without the exponential product attenuation analyzed for the smoothed-volume objective. The experiments at D=1024D = 1024 provide empirical evidence that the resulting objective can learn useful box relations under the evaluated settings.

Interpretable Relation-Specific Objectives:

The mapping from training assignments to geometric adjustments is direct: same-index image-text pairs are encouraged to intersect, different-index pairs are encouraged to separate, and positive pairs are additionally subjected to a fixed image-inside-text inclusion objective. Signed gaps and containment violations make the optimization mechanism easier to inspect than an objective expressed only through intersection volume. The P-Joint and N-Joint analyses provide useful evidence that the intended joint/disjoint geometry is more reliably learned than by the evaluated box baselines.

Critical weaknesses

Potential Semantic Mismatch in In-Batch Negative Pairing:

The intersection objective treats every different-index image-text pair in a batch as a negative pair and pushes it toward geometric separation. This is potentially too restrictive for datasets such as COCO, where captions or images from different annotated pairs may still be semantically compatible. Because intersection is interpreted as semantic correspondence, false negatives affect not only ranking supervision but also the claimed geometry.

Unverified Semantic Validity of the Fixed Containment Direction:

The paper applies Bimage⊆BtextB_{\mathrm{image}} \subseteq B_{\mathrm{text}}, to every positive pair, following prior work. It is unclear whether this fixed direction corresponds to a consistent semantic relation in caption datasets. A caption may omit visible content, while also introducing attributes or relations that are not visually verifiable. Although adding the inclusion loss improves retrieval, the current experiments do not distinguish semantic entailment from alternative explanations such as regularization or systematic control of relative box sizes. These comparisons are needed to determine whether the gain comes from containment direction or from box-size regularization.

The inclusion loss can be reduced by enlarging the outer text box, shrinking the inner image box, or moving their boundaries into containment. As the reported P-Joint/N-Joint trade-off as the inclusion weight increases implies that stronger inclusion can also increase unintended intersections, it would be useful to report box-width distributions, volume or width statistics by modality, and controls that match box sizes without imposing semantic containment.

Questions for authors

Handling False Negatives:

How sensitive is the learned joint/disjoint geometry to false negatives introduced by treating all different-index pairs as negatives? Have you considered pseudo-positive mining, multiple-positive supervision, or soft semantic-relevance targets? In particular, could the CxC similarity annotations be used for training, rather than only evaluation, to test whether soft relevance supervision reduces geometrically disjoint but semantically plausible pairs?

Evaluation of Semantic Entailment:

Did you evaluate whether the learned containment relation correlates with human or model-based judgments of caption specificity, visual entailment, or informational scope? Please also report controlled comparisons among Bimage⊆BtextB_{\mathrm{image}} \subseteq B_{\mathrm{text}}, Btext⊆BimageB_{\mathrm{text}} \subseteq B_{\mathrm{image}}, bidirectional containment, and intersection-only training.

Interpretation and Calibration of Uncertainty:

The proposed distances do not retain the conditional-volume interpretation used in probabilistic box embeddings. What interpretation should therefore be assigned to “matching uncertainty,” which is measured by the number of intersecting boxes? Have you evaluated whether this quantity is calibrated against empirical ambiguity, multiple human-approved matches, or prediction error? If downstream applications require conditional probabilities, would an additional calibration model be necessary?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer Comments

MHYv-W1. Does in-batch negative pairing separate semantically compatible pairs and distort the learned geometry?

We understand the reviewer’s concern that treating every different-index image–text pair as a negative may push semantically compatible pairs toward geometric separation, affecting not only ranking supervision but also the claimed geometry. To directly examine this concern, we conduct a controlled evaluation under our COCO setup, testing whether semantically compatible different-index pairs can still be learned as intersecting boxes despite in-batch negative pairing. If such supervision distorted the claimed geometry, these pairs would tend to become disjoint. Conversely, if the geometry is preserved, it should yield intersections for semantically compatible pairs while keeping incompatible pairs disjoint.  

Setup. We conduct two complementary evaluations using human relevance annotations from CxC following the reviewer’s suggestion.

  • First, to directly assess the learned geometry, we evaluate our original model on image–text pairs in the CxC test set. We measure Relevant Joint, the proportion of semantically compatible pairs whose boxes intersect, and Irrelevant Disjoint, the proportion of incompatible pairs whose boxes are disjoint. Together, these metrics assess whether the model learns selective geometric correspondences beyond the explicitly matched training pairs.
  • Second, to examine the effect of false-negative (FN) supervision, we train two additional variants. The FN model deliberately introduces semantically compatible different-index pairs as negatives, while the pseudo-positive (PP) model introduces the same pairs as positives. We select these pairs from the CxC validation set using human relevance scores ≥3\geq 3 and introduce 16 or 32 pairs per batch (12.5% or 25% of the batch size). All other settings follow our original COCO training protocol (Sec. 4.1). We evaluate both geometric relations and retrieval performance on the held-out test sets.  

Geometric results (Tab. R3-W1a). Our original model achieves 89.0% Relevant Joint and 90.3% Irrelevant Disjoint on CxC. These results show that treating different-index pairs as negatives during training does not prevent the model from learning intersections for semantically compatible pairs, while preserving separation for incompatible pairs. The FN model with 16 injected pairs maintains similarly high rates (91.6% and 88.8%, respectively). With 32 injected FN pairs, Relevant Joint increases to 97.3% but Irrelevant Disjoint decreases to 60.9%, indicating reduced geometric discrimination only under heavier FN contamination.  

Retrieval results (Tab. R3-W1b). The PP models, which explicitly treat semantically compatible pairs as positives, do not consistently improve retrieval over our original model. Moreover, introducing 16 FN pairs causes only modest performance drops (at most 1.6 RSUM), while larger degradations occur with 32 pairs. These findings provide complementary evidence that the ranking supervision remains reasonably robust to moderate false-negative contamination.  

Overall, these experiments show that in-batch negative pairing does not inherently distort the learned geometry. Our model learns selective intersections for semantically compatible pairs, although excessive false-negative supervision can degrade this discrimination. We will include these results and clarify the distinction between training supervision and the learned geometry in the revised manuscript. We also appreciate the reviewer’s suggestion of soft relevance supervision and will discuss it as a future direction.  
 

Table R3-W1a. Comparison of joint/disjoint geometry on the CxC test set, showing the proportions (%) of relevant and irrelevant pairs that are joint or disjoint.

ModelsRelevant Joint (↑)Relevant Disjoint (↓)Irrelevant Disjoint (↑)Irrelevant Joint (↓)
Ours89.0 %11.0 %90.3 %9.7 %
FN model (16 pairs)91.6 %8.4 %88.8 %11.2 %
FN model (32 pairs)97.3 %2.7 %60.9 %39.1 %
PP model (16 pairs)89.2 %10.8 %92.2 %7.8 %
PP model (32 pairs)93.6 %6.4 %85.4 %14.6 %
 

Table R3-W1b. Comparison of semantic correspondence on COCO-1K, COCO-5K, and CxC. Each cell reports R@1 (I2T, T2I) / RSUM.

ModelsCOCO-1K (R@1 / RSUM)COCO-5K (R@1 / RSUM)CxC (R@1 / RSUM)
Ours82.7, 70.0 / 539.063.8, 49.4 / 458.065.4, 51.3 / 467.7
FN model (16 pairs)82.2, 69.6 / 538.363.3, 49.1 / 456.864.8, 51.0 / 466.1
FN model (32 pairs)82.0, 69.2 / 537.262.2, 48.4 / 452.763.7, 50.3 / 462.7
PP model (16 pairs)83.0, 69.8 / 538.963.5, 49.0 / 457.664.9, 50.8 / 466.9
PP model (32 pairs)82.2, 68.9 / 536.663.3, 48.0 / 453.664.9, 49.8 / 463.1

MHYv-W2. Unverified Semantic Validity of the Fixed Containment Direction.

Our image-within-text direction is adopted as an inductive bias, rather than a strict semantic entailment for every image–caption pair. As described in Sec. 3.3 (Lines 245–246), this choice follows prior works [1, 2, 3] that model captions as generally less specific than their paired images. To validate whether this assumption is meaningful in practice, we examine whether the containment direction matters, whether the gain can instead be explained by box-size regularization, and whether the learned containment is consistent with the semantic hierarchies within both modalities.  

Experimental Setup. To this end, we compare our original model, which uses forward inclusion (image-within-text), with a reverse-inclusion variant (text-within-image) and a box-size regularization variant. For the box-size regularization variant, we consider two settings: one matches the target box sizes to those of our original model (size-matched), while the other only encourages image boxes to be smaller than text boxes (unmatched). We evaluate all models on COCO-1K, COCO-5K, and CxC, and report their box-size statistics for each modality.  

Results. As shown in Tab. R3-W2a, forward inclusion outperforms reverse inclusion across all benchmarks, indicating that the direction matters. The size-matched model achieves comparable, though slightly lower, performance than ours, indicating that, as the reviewer suggested, the retrieval gain largely stems from the size asymmetry induced by the inclusion loss. However, the size-matched model relies on target sizes taken from the box sizes already learned by our model. When only the asymmetry is enforced without these learned targets (unmatched), performance drops on all benchmarks (Tab. R3-W2a and R3-W2b), whereas the inclusion loss learns the asymmetry without such targets. Moreover, on HierarCaps [4], the containment learned by our model closely follows both the human-verified caption specificity and the visual information hierarchy of nested image crops (Kendall’s τ\tau for caption specificity and visual entailment = 0.90 and 0.82 for ours, vs. −0.75 and 0.62 for size-matched and 0.88 and −0.35 for unmatched; see Tab. R3-Q2). Together, these results show that our inclusion loss learns, without target sizes, both the cross-modal size asymmetry that contributes to the retrieval gains and the within-modality containment that box sizes alone cannot express. We will incorporate these results into the revised manuscript.  
 

Table R3-W2a. Semantic alignment performance of forward, reverse inclusion, and box-size regularization. Each cell reports R@1 (I2T, T2I) / RSUM.

ModelsCOCO-1K (R@1 / RSUM)COCO-5K (R@1 / RSUM)CxC (R@1 / RSUM)
Ours (forward inclusion)82.7, 70.0 / 539.063.8, 49.4 / 458.065.4, 51.3 / 467.7
Ours (reverse inclusion)81.4, 69.1 / 536.161.6, 47.8 / 451.463.4, 49.7 / 461.6
Box-size regularization (unmatched)81.1, 69.1 / 536.061.5, 47.8 / 450.563.0, 49.8 / 460.5
Box-size regularization (size-matched)82.3, 69.4 / 538.363.4, 48.6 / 456.364.8, 50.4 / 465.4
 

Table R3-W2b. Box-size statistics (mean ±\pm std) for forward, reverse inclusion, and box-size regularization on COCO-5K.

ModelsImage width (mean ±\pm std)Text width (mean ±\pm std)
Ours (forward inclusion)0.103 ±\pm 0.041.134 ±\pm 0.43
Ours (reverse inclusion)0.747 ±\pm 0.38 (>> forward model)0.096 ±\pm 0.02 (<< forward model)
Box-size regularization (unmatched)0.096 ±\pm 0.02 (≈\approx forward model)2.114 ±\pm 0.77 (>> forward model)
Box-size regularization (size-matched)0.102 ±\pm 0.03 (≈\approx forward model)1.135 ±\pm 0.37 (≈\approx forward model)
 

-[1] Desai, Karan, et al. "Hyperbolic image-text representations." International Conference on Machine Learning. PMLR, 2023.
-[2] Ramasinghe, Sameera, et al. "Accept the modality gap: An exploration in the hyperbolic space." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
-[3] Pal, Avik, et al. "Compositional entailment learning for hyperbolic vision-language models." International Conference on Learning Representations. 2025.
- [4] Alper, Morris, and Hadar Averbuch-Elor. "Emergent visual-semantic hierarchies in image-text representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

MHYv-Q1. Handling False Negatives.

We have addressed this in our response to MHYv-W1.  
 
 

MHYv-Q2. Evaluation of Semantic Entailment.

Following the reviewer’s suggestion, we additionally evaluate whether the learned containment relations capture hierarchical relationships within each modality. Our model achieves the highest correlation with both caption specificity and visual information hierarchy, compared with models trained with reverse containment (text-within-image), bidirectional containment, intersection-only objectives, or box-size regularization.  

Experimental Setup. We use HierarCaps [4], a benchmark that provides four levels of captions for each image, ranging from general (i.e., low specificity) to specific descriptions (i.e., high specificity), with a human-verified ordering.  We conduct two evaluations. For caption specificity, we use the human-verified caption ordering as the reference. For visual entailment, we construct four nested crops of each HierarCaps image and use the ordering by the amount of visual information as the reference, where a crop with less visual information is regarded as more general. For each model, we derive an ordering from its containment relations by comparing the containment degrees between two boxes in each direction and assigning the hierarchical order based on the stronger containment. We then measure the agreement between the model-derived and reference orderings using Kendall’s tau [5]. Finally, we compare our model trained with image-within-text inclusion (ours) against models trained with reverse containment, bidirectional containment, intersection-only training, and box-size regularization (size-matched and unmatched; see MHYv-W2).  

Results. As shown in Tab. R3-Q2, our model achieves the highest correlation for both caption specificity (τ\tau = 0.90) and visual entailment (τ\tau = 0.82). Reverse and bidirectional containment yield substantially lower correlations for visual entailment (τ\tau = 0.18 and −0.21), suggesting that the direction of containment plays an important role in capturing the hierarchy of visual information. Intersection-only training, which imposes no containment, also falls short of our model, particularly for visual entailment (τ\tau = 0.56). Notably, both box-size regularization models show substantially lower correlations than ours on at least one of the two hierarchies, indicating that these hierarchies are not captured by box sizes alone, even when the sizes are matched to those learned by our model. Overall, these results show that the learned containment of our model is the most consistent with both the hierarchy of caption specificity and that of visual information.  
 

Table R3-Q2. Kendall’s τ\tau between learned containment orderings and reference orderings, given by the HierarCaps caption hierarchy for caption specificity and by nested image crops for visual entailment. Box size reg. replaces the inclusion loss with a size prior, where the matched variant uses our learned box sizes as targets and the unmatched variant only enforces image << text.

Kendall τ\tau (↑\uparrow)Ours (forward)Ours (reverse)Bidirectional (both)Intersection-onlySize regularization (size-matched)Size regularization (unmatched)
Caption specificity0.900.680.790.84-0.750.88
Visual entailment0.820.18-0.210.560.62-0.35
 

- [5] Kendall, Maurice G. "A new measure of rank correlation." Biometrika 30.1-2 (1938): 81-93.

Reviewer_12q9OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(−1)
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes a volume-free, distance-based box embedding framework for learning intersection, disjointness, and directed inclusion relations in multimodal representation learning. It introduces an intersection-aware distance constructed from signed gaps and an inclusion distance based on containment violations, together with geometric properties and gradient analysis. The main motivation is to avoid the gradient attenuation caused by coordinate-wise products in conventional volume-based box objectives.

Experiments cover COCO, CxC, RedCaps, and MovieLens-1M. The method generally outperforms the compared baselines on cross-modal retrieval and performs well on set-theoretic compositional queries that are not directly supervised during training. Overall, the method is well motivated and empirically promising, but some theoretical claims and the empirical validation of the inclusion relation require further clarification.

Critical strengths
  1. The motivation is clear and directly targets the optimization difficulty of volume-based box learning. Instead of relying on coordinate-wise volume products, the method constructs intersection and inclusion distances directly from signed gaps and containment violations. The design is simple and has a clear geometric interpretation.
  2. The experimental evaluation is broad and consistent. The method achieves competitive results on COCO, CxC, and RedCaps, and clearly outperforms the compared box baselines on unseen compositional queries in MovieLens-1M.
Critical weaknesses
  1. Theorem 2 does not fully justify the claimed gradient behavior. Its lower bound still contains �0��0� and �1��1�, which can approach zero when the classifier saturates or the center distance becomes large. Thus, the current result does not by itself establish that the gradient is non-vanishing or that its dependence on �2��2� is at most polynomial without additional assumptions. In addition, Theorem 1 mainly analyzes the multiplicative structure of the smoothed volume objective; the paper does not provide a direct analysis or comparison showing that the same issue persists for a numerically stable log-space volume objective.
  2. The empirical validation of the inclusion relation is indirect. The paper mainly reports retrieval performance and P-Joint/N-Joint, but P-Joint/N-Joint measures box intersection rather than whether �3��3� is actually satisfied. It does not report the forward containment rate on matched pairs, the reverse containment rate, or the false-containment rate on negative pairs. In addition, Tables 4 and 5 use COCO-5K results to select �4��4� and �5��5�; the paper should clarify whether these hyperparameters were selected exclusively on a validation split rather than the test set.
Questions for authors
  1. The gradient lower bound in Theorem 2 depends on �0��0� and �1��1�. Do the claims of a non-vanishing gradient and at-most-polynomial decay with �2��2� require additional assumptions, such as bounded center norms or operation away from saturation? Could the authors also provide a theoretical or empirical comparison with a numerically stable log-space volume objective?
  2. Could the authors report the �3��3� containment rate on matched pairs, the reverse �4��4� rate, and the false-containment rate on negative pairs? Please also clarify which data split was used to select �5��5� and �6��6� in Tables 4 and 5, and whether the COCO-5K test set was involved in hyperparameter selection.
6 comments
AuthorsOpenReview ↗

Author Response to Reviewer Comments

12q9-W1a. Clarify whether gradient can vanish due to the (1−p1-p) and (1−tanh⁡(∣∣ca−cb∣∣2)1-\tanh(||c_a-c_b||_2)) terms, and whether its dependence on the dimension DD is at most polynomial without additional assumption.

Thank you for this important question. We first clarify that a small gradient is not inherently problematic; what matters is why it becomes small. Once a pair has been sufficiently learned, its gradient is naturally expected to decrease. In particular, 1−p1−p approaches zero only as pp approaches 11, i.e., when the model already assigns the correct relation with high confidence. This is fundamentally different from a global attenuation mechanism that suppresses gradients regardless of the learning state, such as an unfavorable scaling with the embedding dimension DD. Our concern in Theorem 2 is the latter: whether increasing DD itself causes the gradient to vanish independently of how well the pair has been learned.  

Setup. To analyze the dependence on DD, let LD=ℓD(⋅;α,β,γ)∣α,β,γ>0\mathcal{L}_D = \\{ \ell_D(\cdot; \alpha, \beta, \gamma) | \alpha, \beta, \gamma > 0 \\} denote the loss family at dimension DD. We then fix arbitrary α,β,γ>0\alpha, \beta, \gamma > 0 and analyze the corresponding sequence ℓD(⋅;α,β,γ)D\\{\ell_D(\cdot; \alpha, \beta, \gamma)\\}_D across dimensions.  

Regarding the (1−p1-p) factor: Let z=yα(β−dinter)z = y\alpha(\beta−d_\text{inter}), so that p=σ(z)p = \sigma(z) and 1−p=1/(1+ez)1−p = 1 / (1+e^z). Since 0<dinter<20 < d_\text{inter} < 2, we have β−2<β−dinter<β\beta−2 < \beta−d_\text{inter} < \beta, and therefore ∣β−dinter∣≤max⁡(∣β∣,∣β−2∣)|\beta−d_\text{inter}| ≤ \max(|\beta|,|\beta−2|). Because y∈−1,+1y \in \\{−1,+1\\}, z≤∣z∣=α∣β−dinter∣≤αmax⁡(∣β∣,∣β−2∣)z \leq |z| = \alpha|\beta−d_\text{inter}| ≤ \alpha\max(|\beta|, |\beta−2|). Let k=αmax⁡(∣β∣,∣β−2∣)>0k = \alpha\max(|\beta|, |\beta−2|) > 0. For fixed α\alpha and β\beta, kk is a positive constant independent of DD. Hence, 1−p=1/(1+ez)≥1/(1+ek)>01−p = 1 / (1+e^z) ≥ 1 / (1+e^k) > 0. Thus, 1−p1−p has a strictly positive lower bound independent of DD and therefore cannot vanish as DD increases.  

Regarding the (1−tanh⁡(∣∣ca−cb∣∣2))(1-\tanh(||c_a - c_b||_2)) factor: In our paper, we actually l2l_2-normalize the box centers, ensuring that their pairwise distance satisfies ∣∣ca−cb∣∣2≤2||c_a - c_b||_2 \leq 2. Consequently, this factor is bounded below by 1−tanh⁡(∣∣ca−cb∣∣2)≥1−tanh⁡(2)≈0.035971 - \tanh(||c_a - c_b||_2) \geq 1 - \tanh(2) \approx 0.03597, preventing it from vanishing due to increasing center separation. Importantly, this normalization is already part of our implementation and is documented in Lines 1489-1495 in Supp. A.5.4.  

Regarding the dependence on DD: Therefore, both 1−p1−p and 1−tanh⁡(∣∣ca−cb∣∣2)1−\tanh(||c_a−c_b||_2) are bounded below by positive constants independent of DD. Moreover, since C=sg(∣∣ca−cb∣∣2)+Dlog⁡2/γ=O(D)C = \text{sg}(||c_a - c_b||_2) + \sqrt{D} \log 2 / \gamma = O(\sqrt{D}) for fixed γ\gamma, the lower bound in Theorem 2 scales as Ω(D−1)\Omega(D^{-1}). Hence, the gradient has at most polynomial dependence on DD and does not suffer from exponential attenuation with increasing dimension.

12q9-W1b. Gradient behavior of numerically stable log-space volume objectives.

We would like to clarify that the submitted manuscript includes direct theoretical and empirical analyses of log-space volume objectives. The theoretical analysis is briefly discussed in the main text (Lines 50-51) and detailed in Supp. A.3, while the empirical comparisons are presented in Fig. 1 of the main text and Fig. A1 (Supp. A.4.1), both referenced in the main text (Lines 127-128 and Line 154, respectively).  

Importantly, our analyses distinguish the multiplicative gradient attenuation of raw smoothed volume from the different gradient limitations that may arise after logarithmic transformation or normalization. For the reviewer’s convenience, we summarize the results below:
 

Theoretical analysis (Supp. A.3).

  • Log-space-hard-volume: Directly applying the logarithm to the hard intersection volume (VV) results in an undefined gradient when two boxes are non-overlapping, and the gradient diverges as their overlap approaches zero (Eq. A45). Specifically, ∇(log⁡V)=(1/V)∇V\nabla(\log V) = (1/V) \nabla V, which has a singularity as (VV) approaches zero. Thus, as the positive overlap approaches zero, the gradient diverges to ∞\infty, resulting in severe instability. For this reason, log⁡V\log V itself is generally not used directly as a loss.
  • Log-smoothed-volume: Taking the exact logarithm of smoothed volume removes the multiplicative attenuation across coordinates for separated boxes. However, gradients approach zero for deeply overlapping coordinates and remain sensitive to the smoothing parameter (Eq. A44). Moreover, as discussed in the empirical analysis below, the gradient can vanish in practical training settings as the embedding dimension or the separation between boxes increases.
  • Stabilized-log volume: Adding ϵ>0\epsilon > 0 to the volume VV before taking the logarithm (i.e., log⁡(V+ϵ)\log(V+\epsilon)) avoids the singularity but can reintroduce small-volume attenuation: the resulting gradient satisfies ∣∇log⁡(V+ϵ)∣=∣(∇V)∣/(V+ϵ)≤(1/ϵ)∣∇V∣|\nabla\log(V+\epsilon)| = |(\nabla V)| / (V+\epsilon) \leq (1/\epsilon) |\nabla V|. Since ∣∇V∣|\nabla V| has an exponentially decaying upper bound by Theorem 1, ∣∇log⁡(V+ϵ)∣|\nabla\log(V+\epsilon)| inherits the same exponential attenuation bound (Eqs. A46-A48).
  • Log-normalized probability objective: We also analyzed the logarithm of the volume-based conditional probabilities (Eq. 2 in Sec. 2.1), which is a representative normalized-probability objective. In this form, the gradients of the numerator (joint volume) and denominator (marginal volume) can exactly cancel out, resulting in a zero gradient even when boxes are non-overlapping (Eq. A58). We further showed that this problem cannot be resolved by applying a stabilizing function such as the sigmoid (Eq. A70).
     

Empirical analysis (Fig. 1 and Fig. A1). We further investigated a commonly used form of smoothed-volume-based objectives that are designed to facilitate optimization, and found it can still suffer from vanishing gradients and fail to learn box relationships as the embedding dimension or the separation between boxes increases.

  • Figure 1 (Sec. 2.3): In Fig. 1, we examine whether gradients of the smoothed-log-normalized probability objectives (i.e., log-volume objectives in the figure) vanish as the embedding dimension or box separation increases. The results show that gradients fall below the numerical scale relevant to AdamP updates as either factor increases, demonstrating that the gradient problem identified in our theoretical analysis still persists.
  • Figure A1 (Supp. A.4.1): In Fig. A1, we examine whether smoothed-log-normalized-probability objectives can learn the desired joint relationships for positive box pairs and disjoint relationships for negative pairs as the embedding dimension or box separation increases. The results show that these objectives fail to learn the desired relationships as either factor increases, indicating that smoothed-log-normalized-probability objectives do not ensure stable learning of box relationships in practice.
     

In summary, our existing analyses show that log-space transformations can alleviate the coordinatewise multiplicative attenuation of raw volume, but do not universally eliminate gradient attenuation, cancellation, or the resulting optimization difficulties in practical volume-based formulations. We will make these analyses more prominent in the revised manuscript and clarify the distinct failure modes of the different formulations.

12q9-W2a. The empirical validation of the inclusion relation is indirect.

We thank the reviewer for the feedback. As also discussed in our response to reviewer aV6f (W5b), we conducted additional experiments validating the effectiveness of the image-within-text inclusion assumption through retrieval performance and alignment with semantic hierarchies. To further address the reviewer’s concern, we directly evaluate whether the learned boxes satisfy the intended containment relation Box(image) ⊆\subseteq Box(text). Specifically, we measure forward (Box(image) ⊆\subseteq Box(text)) and reverse (Box(text) ⊆\subseteq Box(image)) containment rates for matched and unmatched image-text pairs on COCO-5k. Since our inclusion loss is applied in the image-within-text direction, it should increase forward containment among positive pairs while keeping reverse containment low. For negative pairs, which should be disjoint, containment should remain low in both directions. To isolate the effect of the inclusion loss, we compare our full model against the Intersection-only variant.  

As shown in Tab. R2-W2a, our inclusion loss substantially increases forward containment among matched pairs from 0.5% to 53.1%, while reverse containment remains at 0.0%. This indicates that the inclusion loss primarily encourages the intended containment direction. Meanwhile, false-containment among unmatched pairs remains low (2.5% in the forward direction, and 0.0% in the reverse direction). Overall, these results provide direct geometric evidence that our inclusion loss encourages the intended asymmetric containment relation, complementing the retrieval and semantic-hierarchy evaluations reported in our responses to other reviewers (Tabs. R1-W5c and R3-Q2, respectively).  

In the revised manuscript, we will include the containment analysis (Tab. R2-W2a).  
 

Table R2-W2a. Comparison of models in terms of forward and reverse containment rates on positive pairs and false-containment rates on negative pairs.

MetricPair typeDirectionOursIntersection-only
Containment ratePositive pairsForward (↑)53.1 %0.5 %
Reverse (↓)0.0 %0.0 %
False-containment rateNegative pairsForward (↓)2.5 %1.1 %
Reverse (↓)0.0 %0.0 %
 
 

12q9-W2b. Clarify whether γ\gamma and λ\lambda hyperparameters were selected exclusively on a validation split.

We apologize for the confusion caused by a typo: Tabs. 4 and 5 in the paper report results on the COCO-1K validation set, not COCO-5K. Both γ\gamma and λ\lambda were selected exclusively using this validation set, based on two criteria: P-Joint > 90% and N-Joint < 10%. The test sets were not used for hyperparameter selection. We provide the full details of each selection and the corresponding results across the validation and test sets below.  

  • Selection of γ\gamma (Tab. R2-W2b). We first evaluate γ\gamma using the intersection-only model. We select γ\gamma = 20, the only value that satisfies both criteria on the validation set. The selected value also satisfies both criteria on the COCO-1K and COCO-5k test sets.
  • Selection of λ\lambda (Tab. R2-W2c). With γ\gamma = 20 fixed, we evaluate the weight λ\lambda of the inclusion loss. We select λ\lambda = 0.6, the only nonzero value that satisfies both criteria on the validation set. The selected λ\lambda also satisfies both criteria on the COCO-1K and 5K test sets.  

In the revised manuscript, we will include the full ablation results across validation and test sets (Tabs. R2-W2b and W2c), and correct the split labels of Tabs. 4 and 5.  
 

Table R2-W2b. Full ablation results for γ\gamma, including the original results reported in our paper on the COCO-1K validation set and evaluations on the COCO-1K and 5K test sets.

γ\gammaP / N-joint (1K val)P / N-joint (1K test)P / N-joint (5K test)
1099.7 / 17.299.6 / 18.399.6 / 18.2
1598.7 / 11.199.1 / 11.598.9 / 11.4
2095.6 / 3.595.4 / 3.895.6 / 3.8
2567.2 / 0.570.1 / 0.668.4 / 0.6
3066.1 / 0.468.1 / 0.566.2 / 0.5
 

Table R2-W2c. Full ablation results for λ\lambda, including the original results reported in our paper on the COCO-1K validation set and evaluations on the COCO-1K and 5K test sets.

λ\lambdaP / N-joint (1K val)P / N-joint (1K test)P / N-joint (5K test)
0.095.6 / 3.595.4 / 3.895.6 / 3.8
0.287.5 / 3.788.3 / 4.286.9 / 4.4
0.487.5 / 4.088.4 / 5.087.8 / 5.0
0.694.0 / 8.593.3 / 9.993.21 / 9.7
0.896.1 / 18.697.3 / 19.296.9 / 19.4
1.098.3 / 31.998.4 / 32.998.2 / 31.2
Reviewer_aV6fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes volume-free distance objectives for learning intersection and inclusion relations between box embeddings. It analyzes gradient attenuation in volume-based formulations and evaluates the proposed losses on multimodal retrieval, semantic similarity, and compositional recommendation.

Critical strengths
  1. The paper addresses a meaningful optimization problem in box embeddings and provides a clear geometric formulation.
  2. The intersection and directed inclusion losses offer a unified approach to learning symmetric and asymmetric relations.
  3. Experiments cover several applications, with consistent improvements over the reported retrieval baselines and substantial gains on compositional recommendation.
Critical weaknesses
  1. The gradient claims require qualification. Theorem 2 contains the factor 1−tanh⁡(∥ca−cb∥2)1-\tanh(\|c_a-c_b\|_2), which can decay exponentially with center separation. Therefore, removing coordinatewise products does not eliminate all separation-dependent gradient attenuation. The distinction between strictly positive and practically useful gradients should be made explicit.

  2. The analysis of volume-based objectives is not fully conclusive. Theorem 1 concerns smoothed volume, whereas practical methods may optimize log-volumes or normalized probabilities. These transformations can change gradient behavior substantially. The referenced supplementary analysis is important for establishing how broadly the criticism applies.

  3. Geometric certificates do not guarantee successful learning. The intersection results provide sufficient distance thresholds, and the inclusion result characterizes zero loss. They do not establish that optimization reaches these conditions. The abstract’s claim of theoretically guaranteed relations should distinguish geometric characterization from optimization convergence.

  4. Baseline fairness and statistical reliability need stronger evidence. Some box baselines perform extremely poorly, particularly OptBox on COCO. A shared optimization setup may disadvantage methods requiring different initialization or smoothing settings. Method-specific tuning and results across multiple seeds are needed, especially for the small RedCaps gains.

  5. The semantic and compositional interpretations need further validation. Treating all unmatched pairs as negatives conflicts with the motivation of one-to-many correspondence. Likewise, imposing image-within-text containment is a modeling assumption rather than a universally valid semantic relation. MovieLens experiments would also benefit from strong recommendation models combined with explicit attribute filtering.

Questions for authors
  1. Do the improvements persist with individually tuned box baselines?
  2. How sensitive are the results to false negatives and the chosen inclusion direction?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer Comments

aV6f-W1. The gradient claims require qualification.

Thank you for the helpful comment. In our paper, we actually ℓ2\ell_2-normalized the box centers, ensuring that their pairwise distance satisfies R=∣ca−cb∣2≤2R=|c_a-c_b|_2\leq 2. Consequently, the factor highlighted by the reviewer is bounded below by 1−tanh⁡(R)≥1−tanh⁡(2)≈0.035971-\tanh(R)\geq 1-\tanh(2)\approx 0.03597, preventing it from vanishing due to increasing center separation. Importantly, this normalization is already part of our implementation and is documented in Lines 1489-1495 in Supp. A.5.4.  
 
 

aV6f-W2. The analysis of volume-based objectives is not fully conclusive.

We would like to clarify that we have already conducted theoretical and empirical analyses of the log-volume and normalized probabilities, as briefly discussed in the main text (L50–51) and detailed in Supp. A.3, Fig. 1 (Sec. 2.3), and Fig. A1 (Supp. A.4.1). These analyses examine how logarithmic transformations and normalization affect gradient behavior and identify the remaining optimization limitations.

For the reviewer's convenience, we summarize our theoretical analyses of log-volume objectives and log-normalized probability objectives, followed by our empirical analysis, showing that these variants do not fundamentally resolve the gradient issues.  

Log-volume objectives (Supp. A.3). We analyzed three variants of log-volume objectives. Our results show that applying a logarithm to the volume does not eliminate the gradient issues.

  • Log-hard-volume: Directly applying the logarithm to the hard intersection volume VV results in an undefined gradient when two boxes are non-overlapping, and the gradient diverges as their overlap approaches zero (Eq. A45). Specifically, ∇(log⁡V)=(1/V)⋅∇V\nabla (\log V) = (1/V)\cdot\nabla V so it has a singularity as VV approaches zero. That is, as the positive overlap approaches zero, the gradient diverges to ∞\infty, resulting in severe instability. For this reason, log⁡V\log V itself is generally not used directly as a loss.
  • Log-smoothed-volume: Applying the logarithm to the smoothed volume keeps the gradient bounded for non-overlapping boxes. However, for deeply overlapping boxes, the gradient approaches zero as the overlap increases (Eq. A44). Also, as discussed in empirical analyses paragraph below, the gradient can vanish in practical training settings as the embedding dimension or the separation between boxes increases.
  • Stabilized-log-volume: Adding a small constant ϵ>0\epsilon>0 to the volume before applying the logarithm (i.e., log⁡(V+ϵ)\log (V+\epsilon)) prevents gradient explosion. The resulting gradient satisfies ∣∇log⁡(V+ϵ)∣=∣(∇V)/(V+ϵ)∣≤(1/ϵ)∣∇V∣|\nabla \log (V+\epsilon)| = |(\nabla V)/(V+\epsilon)|\leq(1/\epsilon)|\nabla V| . Since ∣∇V∣|\nabla V| has an exponentially decaying upper bound by Theorem 1, ∣∇log⁡(V+ϵ)∣|\nabla \log (V+\epsilon)| inherits the same exponential attenuation bound (Eqs. A46–A48).  

Log-normalized probability objectives (Supp. A.3). We also analyzed the log form of the volume-based conditional probability (Eq. 2 in Sec. 2.1), a representative normalized-probability objective. In this form, the gradients of the numerator (joint volume) and denominator (marginal volume) exactly cancel out, resulting in a zero gradient even when boxes are non-overlapping (Eq. A58). We further showed that this problem cannot be resolved by applying a stabilizing function such as the sigmoid (Eq. A70).  

Empirical analysis. We further empirically demonstrated that smoothed-log-normalized-volume objectives, which are commonly used in practical training, can still fail to learn box relationships or suffer from vanishing gradients as the embedding dimension or the separation between boxes increases.

  • Figure 1 (Sec. 2.3): In Fig. 1, the models labeled as using log-volume objectives are trained using the GumbelBox loss, which employs a smoothed-log-normalized-volume formulation. In the result, we observe that, as the embedding dimension or the separation between boxes increases, gradients vanish to zero under AdamP, providing empirical evidence that the gradient vanishes even under objectives with practical modifications such as normalization and logarithmic transformations.
  • Figure A1 (Supp. A.4.1): This experiment evaluates whether GumbelBox using smoothed-log-normalized-volume objectives and other proposed box-learning baselines can learn the desired relationships: joint relationships between positive box pairs and disjoint relationships between negative box pairs. As shown in Fig. A1, as the embedding dimension or the separation between boxes increases, these objectives fail to learn the ideal joint/disjoint relationships between box pairs.  

In summary, we have already conducted analyses of the volume-based variants raised by the reviewer. Due to the page limit, we only briefly mentioned these analyses in the main text and will discuss them in detail in the revised paper.

aV6f-W3. Geometric certificates do not guarantee successful learning.

Thank you for highlighting this important distinction. Our theoretical results characterize the geometric conditions for box intersection and inclusion (Propositions 1-2), while also establishing nonzero gradient bounds that facilitate learning these relations (Theorem 2 and Proposition 3), which the volume-based objectives analyzed in Theorem 1 and Theorem 3 (Supp. A.3.1) do not provide, rather than guaranteeing optimization convergence. We will clarify this distinction in the abstract accordingly.
 
 
 

aV6f-W4. Baseline fairness and statistical reliability needs stronger evidence.

Thank you for raising these points. Both method-specific hyperparameter tuning and multi-seed evaluations are already included in our submitted supplementary material, as detailed below.

Regarding baseline fairness, we conducted method-specific hyperparameter searches for all box-based baselines, including additional experiments with volume lower bounds to prevent box collapse (Tab. A7 of Supp. A.4.4). In particular, the reported OptBox result on COCO corresponds to its best retrieval performance among the evaluated configurations, rather than its default settings.

Regarding statistical reliability, we reported mean performance and variation across multiple random seeds for all methods (Tabs. A3–A6 of Supp. A.4.3). Notably, on RedCaps, our method achieves an RSUM of 368.73±0.2368.73 \pm 0.2, compared with 365.70±1.2365.70 \pm 1.2 for PCME, the strongest baseline in terms of RSUM.

We will make these existing experimental results more prominent in the main paper.

aV6f-W5a. Compatibility of negative-pair training with one-to-many correspondence.

We understand the reviewer’s concern that treating all unmatched pairs as negatives may conflict with our goal of learning one-to-many semantic correspondence. To directly examine this potential conflict, we conduct a controlled evaluation under our COCO setup, testing whether semantically relevant unmatched pairs can still be learned as intersecting boxes despite negative-pair supervision. If such supervision prevented one-to-many correspondence learning, these pairs would tend to become disjoint. Conversely, successful learning should yield intersections for semantically relevant unmatched pairs while keeping irrelevant pairs disjoint.  

Setup. We conduct two complementary evaluations using human relevance annotations from CxC.

  • First, to directly assess the learned geometry, we evaluate our original model on unmatched image-text pairs in the CxC test set. We measure Relevant Joint, the proportion of semantically relevant unmatched pairs whose boxes intersect, and Irrelevant Disjoint, the proportion of irrelevant unmatched pairs whose boxes are disjoint. Together, these metrics assess whether the model learns selective geometric correspondences beyond the explicitly matched training pairs.
  • Second, to examine the effect of false-negative (FN) supervision, we train two additional variants. The FN model deliberately introduces semantically relevant unmatched pairs as negatives, while the pseudo-positive (PP) model introduces the same pairs as positives. We select these pairs from the CxC validation set using human relevance scores ≥3\geq 3 and introduce 16 or 32 pairs per batch (12.5% or 25% of the batch size). All other settings follow our original COCO training protocol (Sec. 4.1). We evaluate both geometric relations and retrieval performance on the held-out test sets.  

Geometric results (Tab. R1-W5a). Our original model achieves 89.0% Relevant Joint and 90.3% Irrelevant Disjoint on CxC. These results show that treating unmatched pairs as negatives during training does not prevent the model from learning intersections for semantically relevant unmatched pairs, while preserving separation for irrelevant pairs. The FN model with 16 injected pairs maintains similarly high rates (91.6% and 88.8%, respectively). With 32 injected FN pairs, Relevant Joint increases to 97.3%, but Irrelevant Disjoint decreases to 60.9%, indicating reduced geometric discrimination under heavier FN contamination.  

Retrieval results (Tab. R1-W5b). The PP models, which explicitly treat semantically relevant unmatched pairs as positives, do not consistently improve retrieval over our original model. Moreover, introducing 16 FN pairs causes only modest performance drops (at most 1.6 RSUM), while larger degradations occur with 32 FN pairs. These findings provide complementary evidence that our model remains reasonably robust to moderate false-negative contamination.  

Overall, these experiments show that negative-pair supervision does not inherently preclude learning one-to-many geometric correspondences. Our model learns selective intersections for semantically relevant unmatched pairs, although excessive false-negative supervision can degrade this discrimination. We will include these results and clarify the distinction between training supervision and learned geometric relations in the revised manuscript.  
 

Table R1-W5a. Joint/disjoint ratios (%) of our model, the FN models, and the PP models, evaluated on unmatched image–text pairs from the CxC test set.

ModelsRelevant Joint (↑\uparrow)Relevant Disjoint (↓\downarrow)Irrelevant Disjoint (↑\uparrow)Irrelevant Joint (↓\downarrow)
Ours89.0 %11.0 %90.3 %9.7 %
FN model (16 pairs)91.6 %8.4 %88.8 %11.2 %
FN model (32 pairs)97.3 %2.7 %60.9 %39.1 %
PP model (16 pairs)89.2 %10.8 %92.2 %7.8 %
PP model (32 pairs)93.6 %6.4 %85.4 %14.6 %
 

Table R1-W5b. Comparison of semantic correspondence among our model, the FN models, and the PP models on COCO-1K, COCO-5K, and CxC. Each cell reports R@1 (I2T, T2I) / RSUM.

ModelsCOCO-1K (R@1 / RSUM)COCO-5K (R@1 / RSUM)CxC (R@1 / RSUM)
Ours82.7, 70.0 / 539.063.8, 49.4 / 458.065.4, 51.3 / 467.7
FN model (16 pairs)82.2, 69.6 / 538.363.3, 49.1 / 456.864.8, 51.0 / 466.1
FN model (32 pairs)82.0, 69.2 / 537.262.2, 48.4 / 452.763.7, 50.3 / 462.7
PP model (16 pairs)83.0, 69.8 / 538.963.5, 49.0 / 457.664.9, 50.8 / 466.9
PP model (32 pairs)82.2, 68.9 / 536.663.3, 48.0 / 453.664.9, 49.8 / 463.1

Comments on the paper

1 comment

Dear Reviewers, Area Chairs, and Senior Area Chairs,

As the author-reviewer discussion period approaches its end on November 18 at 11:59 pm AOE, we would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our paper. We have carefully considered all comments and have revised the manuscript accordingly, with the changes highlighted in red. We summarize the main revisions below.

Theoretical analysis

  • Gradient bounds (12q9, aV6f): Clarified that dimensional scaling of Theorem 2 holds for l2l_2 normalized centers, as 1−p1-p and 1−r1-r are bounded below independently of DD (Sec. 3.4) and added the corresponding derivation (Supp. A.1.2). We also revised the abstract and contributions to state precisely what our analysis establishes: the geometric conditions for intersection and inclusion, and non-vanishing gradients for learning these relations.
  • Log-space volume objectives (12q9, aV6f): Clarified that the gradient attenuation is observed even for the smoothed log-normalized-volume objective used in practice (Fig. 1 and Sec. 2.3), and linked our existing analyses of log-space variants and the controlled study (Supp. A.3 and A.4.1).

Additional analyses

  • In-batch negatives (MHYv, aV6f): Showed that the semantically relevant unmatched pairs still intersect under in-batch negative supervision, and analyzed the effect of false-negative and pseudo-positive supervision (Supp. A.5.1).
  • Inclusion direction (MHYv, aV6f, 12q9): Compared forward, reverse, and bidirectional inclusion with box-size regularization, including alignment with caption-specificity and visual-entailment hierarchies on HierarCaps (Sec 4.5; Supp. A.5.4), and directly evaluated forward / reverse containment rates of the learned boxes (Supp. A.5.3).
  • Matching uncertainty (MHYv): Clarified its interpretation in terms of prediction difficulty and semantic ambiguity (Sec. 4.3), and added its correlation with human-approved matches and top-1 errors, together with its downstream use (Supp. A.5.2).

Experimental clarifications

  • Hyperparameter selection (12q9): Clarified that γ\gamma and λ\lambda are selected on the COCO-1K validation set (Tabs. 4-5), and added validation and test results across splits (Supp. A.5.5).
  • Baselines and reliability (aV6f): Added filtering-based baselines for set-theoretic recommendation (Sec. 4.4, Tab 3, Supp. A.4.2), and noted that all results are averaged over multiple random seeds with standard deviations (Sec. 4.1; Supp. A.4.3).

We understand that some reviewers may not have had the opportunity to reply to our rebuttals. We would be grateful if you could take these revisions into account in your final assessment. If any concerns or questions remain, we would be happy to address them before the discussion period ends.

Thank you again for your time and valuable feedback.

Best regards,

The Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.