acceptodds
Under review as a conference paper at ICLR 2027

Learning Spatially-aware Representations from Scene Graphs Using Optimal Transport

55 views

Abstract

Visual similarity is typically considered from the semantic content of images, while the structure and the spatial organization of objects are often discarded. Spatial information curated from objects (such as bounding box coordinates and relative position relations between objects) is usually absent from the neural model similarity assessment. In the meantime, scene graphs computed from images provide an intermediate representation for explicitly modeling the structure of objects in images, yet their spatial configurations remain underexplored, limiting their usability for spatially-aware computer vision tasks like retrieval based on object arrangement. To address this gap and in this effort to better account for spatial information in images, we propose and formalize the Spatial Scene Graph (SSG), which encodes absolute and relative positions between objects as node and edge attributes, respectively. Building on this structured representation, we propose SPOT (SPatial scene graph Optimal Transport), a novel criterion for spatial similarity between SSGs, and indirectly between images. SPOT allows both the ability to evaluate neural models based on the spatial similarity found in the images and the training of neural backbones to learn more spatially-aware representations. We experimentally demonstrate these claims across two spatially-aware downstream tasks: content-based image retrieval and visual question answering.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spatial Relation, scene graph, spatial similarity

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4j9COpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that visual similarity in deep learning is typically assessed semantically, by shared object content, while the spatial arrangement of objects is largely ignored. They make three contributions:

  • Spatial Scene Graph (SSG): A scene graph where nodes encode objects' absolute geometry (normalized bounding box center/size), and edges encode relative position using Force Banners, a descriptor capturing directional and distance relations between objects.

  • SPOT: A training-free measure of spatial similarity between SSGs, formulated as a fused Gromov–Wasserstein optimal transport problem solved using the with Frank-Wolfe method. It is a convex combination of a node cost (bounding box differences) and a relational cost (circular Wasserstein distance between the edges’ angular profiles), weighted by α = 0.5.

  • S3PE: A model-agnostic self-supervised method that uses SPOT rankings to mine positive and negative triplets from a dataset and trains encoders with a cosine triplet loss. It is applied to a fine-tuned DINOv2 image encoder and to GATv2-based GNNs operating on either semantic or spatial scene graphs.

The experimental section evaluates on PSG (human annotated scene graphs + object masks), and score against both semantic relevance (SPICE) and spatial relevance (SPOT). They also test the method's ability to perform spatial VQA. The authors report that S3PE outperforms CLIP, DINOv2, RelTR and IRSGS on spatial retrieval at the cost of semantic retrieval, and that frozen S3PE-Image embeddings outperform prior other spatially aware representations (C-SIP) on VQA.

Critical strengths
  • The method put forward is novel and useful, and is demonstrated on downstream tasks (spatial VQA, and scene classification from scene graphs alone).

  • The authors measure image similarity by the spatial arrangement of objects through a principled formulation. Using optimal transport, they can compare scene graphs with different numbers of objects, and the resulting transport plan is interpretable.

  • It is clearly shown that spatial similarity is a distinct notion from semantic similarity which current methods miss: Encoders such as CLIP and DINOv2 barely beat random on spatial retrieval. The authors evaluate not only spatial relevance but also the method’s impact on semantic relevance, exposing a potential trade-off.

  • The method is model-agnostic and can be applied to images, semantic graphs and spatial graphs.

  • The paper is well organized and the method is easy to follow: each component (SSG, then SPOT, then S3PE) builds on the previous one, and the experimental goals are clearly stated.

Critical weaknesses
  • The authors list as a contribution that SPOT is experimentally demonstrated as an evaluation criterion for spatial similarity in retrieval tasks. However, this is only shown qualitatively. All the quantitative spatial results (e.g. Table 1) use SPOT itself as the reference ranking, and S3PE is trained on triplets mined from SPOT, so S3PE outperforming the baselines on spatial relevance shows that it learned to reproduce SPOT, rather than that SPOT captures spatial similarity. In order for the paper to support SPOT as an evaluation criterion, it should be validated against an independent reference.

  • VQA is the only evaluation that does not use SPOT as the reference, but S3PE-Image is only evaluated against C-SIP and C-SIP++. The paper should state which backbone C-SIP uses: if it differs from DINOv2, the improvement could come from the backbone rather than from SPOT supervision. In either case, a frozen DINOv2 baseline (and CLIP, as in Table 1) under the same protocol would show how much SPOT fine-tuning adds over the pretrained encoder. The classifier used on top of the concatenated image and question embeddings should also be described

  • The alpha term is only measured qualitatively, at three values. A quantitative sweep over α, e.g. on SharvitSR or VQA would establish a clearer relationship/tradeoff between absolute and relative position information.

With respect to presentation/organization of the paper:

  • A figure showing banners and the derived angular profiles for a few configurations would help someone not familiar with Force Banner
  • The introduction would benefit from a concrete motivating application, along with the downstream results (VQA, scene classification). In particular, when is spatial-only similarity desirable?
  • Table 1 is dense. The metrics need more explanation, as does why random scores so high on spatial NDCG.
Questions for authors
  • SSGs and SPOT encode spatial configuration only in the 2D image plane, so depth relations (e.g. “in front of”, “behind”) and viewpoint changes are not captured, and bounding box size conflates object size with distance from the camera. Could the authors clarify this scope, and discuss whether adding depth information (e.g. from monocular depth estimation) to the node features would be feasible?

  • Why was Frank–Wolfe chosen as the solver? The paper does not motivate this choice or discuss alternatives. What is the runtime per SPOT computation, and in total for the landmark matrix? Would an entropic-regularized solver (Peyré et al., 2016) make computing the exact full SPOT matrix feasible, removing the need for the landmark approximation in Appendix C?

  • In what applications is spatial-only similarity preferable to semantic similarity, and could the two be combined?

  • How sensitive are results to the negative pool range (ranks 1000–5000) and the positive pool size (100)?

5 comments

(W1) : The authors list as a contribution that SPOT is experimentally demonstrated as an evaluation criterion for spatial similarity in retrieval tasks. However, this is only shown qualitatively. All the quantitative spatial results (e.g. Table 1) use SPOT itself as the reference ranking, and S3PE is trained on triplets mined from SPOT, so S3PE outperforming the baselines on spatial relevance shows that it learned to reproduce SPOT, rather than that SPOT captures spatial similarity. In order for the paper to support SPOT as an evaluation criterion, it should be validated against an independent reference.

We thank the reviewer for raising this important point. We agree that evaluating SPOT against an independent spatial similarity reference, particularly human judgements, would provide stronger validation of the proposed criterion. However to the best of our knowledge, there is no established reference metric specifically designed to evaluate the similarity of global spatial configurations between objects across scenes, as existing visual similarity metrics predominantly focus on semantic content. This lack of an independent spatial similarity criterion is precisely one of the challenges motivating our work. We therefore provide controlled qualitative experiments on synthetic and natural scenes in Appendix F, illustrating how SPOT retrieves scenes with similar spatial configurations. We also investigate the usefulness of SPOT-guided representations through an independent downstream task, Visual Question Answering (VQA). Unlike our retrieval evaluation, VQA relies on annotated answers rather than SPOT-based relevance scores. Under the same evaluation protocol, S3PE-Image achieves 80.71% accuracy on spatial relation questions, compared to 67.61% for the frozen DINOv2 baseline. These results provide complementary quantitative evidence that SPOT-guided supervision benefits spatial scene understanding beyond reproducing SPOT-based retrieval rankings. While they do not directly establish agreement between SPOT and human judgments of spatial similarity, they support the practical usefulness of the spatial information captured through SPOT.

(W2) : VQA is the only evaluation that does not use SPOT as the reference, but S3PE-Image is only evaluated against C-SIP and C-SIP++. The paper should state which backbone C-SIP uses: if it differs from DINOv2, the improvement could come from the backbone rather than from SPOT supervision. In either case, a frozen DINOv2 baseline (and CLIP, as in Table 1) under the same protocol would show how much SPOT fine-tuning adds over the pretrained encoder. The classifier used on top of the concatenated image and question embeddings should also be described

We thank the reviewer for their feedback which strengthens our experiments. We have added a DINOv2 baseline to the VQA experiment (See Table 3 of the revised submission) under the same protocol used for S3PE-Image. This setup leads the frozen DINOv2 model to reach an accuracy of 67.61, below our S3PE-Image, showing the interest of using spatially-aware representations to answer related questions. The classifier used after the concatenation is a 3-layer feed-forward network with a classification layer. We clarify this setup in the main paper (Section 6.3).

(W3) : The alpha term is only measured qualitatively, at three values. A quantitative sweep over α\alpha, e.g. on SharvitSR or VQA would establish a clearer relationship/tradeoff between absolute and relative position information.

We thank the reviewer for this suggestion. We have added a quantitative sensitivity analysis of α∈{0,0.25,0.5,0.75,1}\alpha \in \{0, 0.25, 0.5, 0.75, 1\} on the PSG test set, using 500 query graphs and 200 candidates per query, repeated three times (see Appendix E of the revised manuscript).

Compared to α=0.5\alpha=0.5, moderate variations yield high Spearman rank correlations of 0.917±0.0030.917 \pm 0.003 (α=0.25\alpha=0.25) and 0.929±0.0030.929 \pm 0.003 (α=0.75\alpha=0.75), with Top-25 overlaps of 75.0%75.0\% and 79.9%79.9\%, respectively. In contrast, using only relative (α=0\alpha=0) or absolute (α=1\alpha=1) spatial information leads to more distinct rankings, with Top-25 overlaps of 40.9%40.9\% and 64.0%64.0\%.

These results show that moderate variations of α\alpha preserve substantial ranking agreement, while excluding either component has a stronger impact. We therefore retain α=0.5\alpha=0.5 as a balanced default, without claiming that it is empirically optimal.

For the presentation and organization of the paper, We thank the review for pointing out the lack in clarity in some parts of the paper. We have added qualitative examples of Force Banners in Appendix G to highlight the content of the Force Banner. Furthermore, we have reduced the density of Table 1 to improve clarity.

(Q1) : SSGs and SPOT encode spatial configuration only in the 2D image plane, so depth relations (e.g. “in front of”, “behind”) and viewpoint changes are not captured, and bounding box size conflates object size with distance from the camera. Could the authors clarify this scope, and discuss whether adding depth information (e.g. from monocular depth estimation) to the node features would be feasible?

We agree that the current formulation of SPOT focuses on spatial configurations in the 2D image plane. It does not explicitly model depth, physical object size, or viewpoint invariant 3D spatial relationships. In particular, bounding box dimensions reflect the apparent size of objects in the image and may therefore be influenced by their distance from the camera.

We would like to emphasize that SPOT is designed as a generic framework for comparing spatial configurations across different image domains, rather than being restricted to natural images acquired from a particular viewpoint. For instance, our cross-domain experiment on the STAR remote sensing dataset (Appendix I) demonstrates its applicability to aerial imagery, where scenes are observed from an overhead perspective and spatial configurations can be meaningfully characterized in the 2D image plane without explicit depth information.

Extending SPOT to incorporate depth information is conceptually feasible within the existing optimal transport. For example, monocular depth estimates could be aggregated over object segmentation masks and incorporated into the node attributes. This would require adapting the node cost function to account for depth differences, while preserving the overall SPOT formulation.

Such an extension would be particularly relevant for spatial relationships involving depth, such as in front of and _behind, which cannot always be reliably distinguished from 2D geometry alone. Incorporating relative depth information could provide a more fine-grained characterization of these relationships. We therefore consider depth-aware spatial representations a promising direction for future work, beyond the scope of the present study.

(Q2) : Why was Frank–Wolfe chosen as the solver? The paper does not motivate this choice or discuss alternatives. What is the runtime per SPOT computation, and in total for the landmark matrix? Would an entropic-regularized solver (Peyré et al., 2016) make computing the exact full SPOT matrix feasible, removing the need for the landmark approximation in Appendix C?

We chose Franke--Wolfe as the OT solver as it is commonly applied for Gromov-Wasserstein algorithms. The main bottleneck of the computation of the full SPOT matrix is the number of OT solve required. However, entropic regularization (such as the Sinkhorn algorithm) benefit from large graph structures to solve faster and does not reduce the number of OT solves. Due to the structure of the scene graphs of the training set of PSG, entropic-regularized solver does not enable the computational feasibility of the full SPOT matrix.

We performed a full SPOT matrix computation on 1,000 random spatial graphs of PSG using consumer-grade CPU. The full (1000,1000) matrix computation with Franke--Wolfe solver took 2 hours and 21 minutes; using entropic-regularized solver computed the matrix in 2 hours and 16 minutes. By considering proportional computation of the choice of the kk landmarks considered, and the 20-fold fewer OT solves, the matrix of (N,k)(N, k) SPOT values was computed in 6 minutes, and a few seconds for the full matrix approximation.

(Q3) : In what applications is spatial-only similarity preferable to semantic similarity, and could the two be combined?

Spatial-only similarity can be preferable to semantic-only similarity in many applications such as medical imagery. Specifically, blood vessels characterization in retinal imaging [1], and lung cancer classification in histopathological analyses [2] can greatly benefit from the spatial configuration of the objects (blood vessel and cells respectively). In remote sensing imagery, spatial similarity can be leveraged for retrieval similar Local Climate Zones (LCZ) and help urbanization planning [3].

Furthermore, spatial and semantic similarity can be combined to retrieve images that are both semantically relevant and spatially similar to a query. To demonstrate this practical application, we added a two-stage retrieval experiment in Section 6.2 of the revised manuscript. CLIP first retrieves the top-100 semantically relevant candidates, which are then reranked according to their SPOT distances to the query. This approach improves spatial NDCG@25 from 69.09 to 86.83 (+17.74 points), while semantic NDCG@25 only decreases from 62.33 to 58.62 (-3.71 points). These results illustrate how SPOT can complement existing semantic retrieval methods by prioritizing spatial configurations while retaining much of the initial semantic relevance.

[1]: Clément, Michaël, et al. "Directional enlacement histograms for the description of complex spatial configurations between objects." IEEE TPAMI (2016): 2366-2380.

[2]: Desharnais, Lysanne, et al. "Spatially mapping the tumour immune microenvironments of non-small cell lung cancer." Nature communications 16.1 (2025): 1345.

[3]: Zhu, Xiao Xiang, et al. "The urban morphology on our planet–Global perspectives from space." Remote Sensing of Environment 269 (2022): 112794.

(Q4) : How sensitive are results to the negative pool range (ranks 1000–5000) and the positive pool size (100)?

We thank the reviewer for this suggestion. We conducted an additional sensitivity analysis on the PSG dataset for all three S3PE variants (Image, SG, and SSG), varying the positive pool size (50, 100, and 200 nearest neighbors) and the negative sampling range (ranks 500--2000, 1000--5000, and 5000--10000), while keeping the remaining training and evaluation settings unchanged.

Th ablation study has been added to Appendix D.2 of the revised manuscript.

As shown in Table 4, S3PE-SSG exhibits limited sensitivity to these parameters, with NDCG@25 ranging from 86.5586.55 to 87.2787.27 and MP@100 from 30.4530.45 to 31.2931.29. S3PE-SG also remains relatively stable, with NDCG@25 between 77.1877.18 and 79.5279.52. In contrast, S3PE-Image is more sensitive to negative sampling, particularly when using harder negatives (ranks 500--2000), which reduce NDCG@25 from 73.7873.78 to 64.1364.13.

Overall, the original configuration (100 positive neighbors and negative ranks 1000--5000) provides the best results for S3PE-Image and S3PE-SSG, while remaining competitive for S3PE-SG. These findings indicate that the sensitivity to triplet sampling depends on the input representation, with spatial scene graph embeddings showing the greatest stability.

Reviewer_WBfCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Spatial Scene Graphs (SSGs), which represent object geometry through node attributes and fine-grained spatial relationships using Force Banner edge attributes. Authors propos SPOT, an optimal transport criterion for comparison SSGs, and S3PE, a framework that uses SPOT-guided triple supervision to learn image and graph embeddings. Experiments demonstrate improved spatial image retrieval and visual question answering.

Critical strengths

The main idea of the integration of Force Banners into scene graph edge representations is interesting, as it captures finer spatial repationships than categorical labels such as "left of" or "above". The combination of this representation with optimal transport and SPOT-guided learning forms a cogenernt approach. Its contributions lies in explicitly addressing a representation gap through a dedicated representation, similarity criterion, and learning framework, rather than primaily relying on the capabilities of a pretrained foundation model.

Critical weaknesses
  • The main weaknesses concern computational cost and evaluation. Constructing SPOT-based supervision requires substantial optimal transport computation, even with the landmark approximation.
  • The retrieval evaluation mainly measures how well the model reproduces the SPOT rankings used for training. Evaluation against human judgements or another spatial similatiry criterion would proved stronger evidence.
  • Comparison with more recent spatial representation and reasoning methods would also help establish the contribution relative to current work. -Please also revise Figure 3 .
Questions for authors
  • How much preprocessing time and memory are required to construct the SPOT-based supervision, and how does this cost scale with dataset and graph size?
3 comments

(W1): The main weaknesses concern computational cost and evaluation. Constructing SPOT-based supervision requires substantial optimal transport computation, even with the landmark approximation.

We agree that the cost for training models with SPOT is substantial, but it remains a one-time offline cost. Once built, the SPOT matrix can be re-used for training models. Thanks to the supervision strategy, these trained models enable strong spatially-aware image (or graph) representation that do not require any SPOT computation at inference time.

(W2): The retrieval evaluation mainly measures how well the model reproduces the SPOT rankings used for training. Evaluation against human judgements or another spatial similatiry criterion would proved stronger evidence.

We thank the reviewer for raising this important point. We agree that an independent evaluation, particularly through human judgments, would provide stronger evidence for the validity of SPOT as a spatial similarity criterion.

Our SPOT-based retrieval evaluation measures how well the learned representations reproduce the spatial similarity rankings defined by SPOT on the PSG test set, rather than directly validating SPOT against human judgments. Conducting a reliable human evaluation would require a dedicated annotation study, which is beyond the scope of the present revision.

To our current knowledge, SPOT is the only similarity criterion that focuses on the spatial configuration of objects in images. We evaluated existing models using two distinct tasks: retrieval and visual-question answering. The VQA experiment does not leverage SPOT as an evaluation metric (i.e. VQA is evaluated with the accuracy metric). This experiment enables to evaluate models without using SPOT. Specifically, as requested by Reviewer 4j9C, we have added a frozen DINOv2 baseline for the VQA downstream task. The results show that finetuning models with SPOT enables better performances on questions related to the spatial configurations of the objects in scenes.

While these results provide complementary evidence of the usefulness of SPOT-guided representations, we acknowledge that they do not directly establish agreement between SPOT and human perceptions of spatial similarity. We therefore consider human-annotated spatial retrieval evaluation an important direction for future work.

(W3): Comparison with more recent spatial representation and reasoning methods would also help establish the contribution relative to current work.

We thank the reviewer for this proposition. We have included C-SIP (published in 2025) as an additional baseline model for the retrieval task in Table 1 of the revised submission. While the model aims for spatially-aware image representations, it focuses solely on a pair of salient objects per image. Aiming for global spatially-aware image representations is a relatively new task in computer vision, leading to few models, aside from IRSGS, that focus on this task.

Concerning Figure 3, We thank the reviewer for this suggestion regarding Figure 3. We have revised the figure to clarify the S3PE training framework, explicitly illustrating that the encoder can take either an image or a scene graph as input.

(Q1) : How much preprocessing time and memory are required to construct the SPOT-based supervision, and how does this cost scale with dataset and graph size?

Given N the number of graphs in the training set, the computation memory is O(N2)\mathcal{O}(N^2). The computation time is O(N2⋅T⋅n3)\mathcal{O}(N^2 \cdot T\cdot n^3) with nn the average number of nodes per graph and TT the number of solver steps until convergence, but this time complexity becomes O(N2⋅T⋅e2)\mathcal{O}(N^2 \cdot T\cdot e^2) with ee the average number of edges per graph (See Appendix B). The time cost is however reduced to O(N⋅k⋅n2)\mathcal{O}(N\cdot k \cdot n^2) with the approximation-based method described in Appendix C, where kk denotes the number of landmarks used.

Scaling with dataset size, the total time scales quadratically (O(N2)\mathcal{O}(N^2)) when considering full matrix computation, but becomes linear (O(N⋅k)\mathcal{O}(N\cdot k)) with the landmark method. When graphs become larger, the time complexity is O(T⋅e2)\mathcal{O}(T\cdot e^2).

Reviewer_WBfCOpenReview ↗

reply to rebuttal

Thanks to the authors for clarifying my questions. Specifically, regarding Question 2, my concern is resolved, and I agree with the evaluation method using visual question answering tasks. The additional experimental results using C-SIP are also very useful. Even though the computation time and memory cost grow as the graph becomes larger, the contribution of this method is clear and interesting.

We appreciate the reviewer's constructive feedback and are glad that our clarifications and additional experiments helped address their concerns.

Reviewer_GsdrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers the problem of defining and computing a similarity metric between two scene graphs, called SPOT. Such a scene graph metric induces a similarity metric between images (in symbols, similarity(image1, image2) = SPOT(scenegraph(image 1), scenegraph(image2)). The paper describes two use cases for the induced graph metric: 1) retrieve similar graphs to a query graph 2) generate triplets to train a graph encoder to produce graph embeddings that minimize a triplet loss (called S3PE). The paper introduces three main innovations for computing the SPOT scene graph metric. 1) for each object (node) adding the coordinates of its bounding box as a node feature vector. 2) for each pair of linked objects (edge), they add the force banner between the objects as an edge feature vector. 3) They formulate computing spatial similarity as solving an optimal transport problem. Two experiments are reported on image retrieval and on Visual Question Answering (VQA). The main conclusion from the retrieval experiment is that augmenting the scene graph with spatial features leads to a similarity metric that is more sensitive to spatial relations (as opposed to semantic content). The VQA experiments show a 3% improvement over a C-SIP baseline.

Critical strengths

Novelty. There is a lot of novelty in how the SPOT scene graph metric is computed. Adding the bounding box as a node feature is not a deep idea, and it seems to have been used before (https://ojs.aaai.org/index.php/AAAI/article/view/6783), but in the context of scene graph similarity I believe it is new. The Force Banner to represent spatial relations between two objects is new and sophisticated, I was happy to learn about this concept. Using optimal transport to compute graph similarity is evaluated already in the predecessor paper by Yoon et al. AAAI 2021 (named GWL) and there is quite a bit of previous work on optimal transport between graphs. It's a neat idea but the authors of the present paper did not explain how their use of optimal transport is novel. Evaluation: I found the evaluation made a convincing case that spatial relations are captured in SPOT.

Critical weaknesses

Motivation. It is somewhat unclear what the task addressed is. If it is the general problem of defining a similarity metric for images, there are many proposals not addressed in this paper (e.g. FIT). I suspect the authors would say it is the more specific problem of how to define a similarity metric for scene graphs. But what is the motivation for this task? If the final motivation is to define similarities between images, which is what the experiments suggest, one can question whether extracting an intermediate discrete representation will work better than an end-to-end approach. E.g. S3P3 vs. CLIP. Generally the experience of machine learning has been that end-to-end beats intermediate representations, since errors in the intermediate representation extraction step propagate. Rich Sutton might call that a "bitter lesson" https://en.wikipedia.org/wiki/Bitter_lesson. I am sympathetic to the idea of using intermediate discrete representations, mainly because human vision seem to do that. But the paper does not motivate the use of scene graphs in itself. Anyway since there is previous literature on defining similarity between scene graphs, I will take that as the target task in itself.

update: the authors have clarified that the problem is to learn a similarity metric and image representations that quantify/capture spatial relations. It is neither a general concept of similarity between images (like SSIM or cosine distance between clip embeddings) nor is it a similarity metric between entire scene graphs.

Evaluation results. I found the distinction between the semantic content of an image and the spatial relations in it illuminating and this is something I learned from the paper. For example in Figure 4 we see that CLIP retrieves cats in response to a query picture of a cat, whereas SPOT retrieves an image of a plane, but with the plane in a similar position to the cat. However, it seems to me that ultimately we want to retrieve images based on their semantic content rather than spatial layout (e.g., we want cat images not plane images). As the authors note, SPOT and S3P3 do not do well on semantic content. (I applaud their honesty). The spatial relevance metric where S3P3 does work well (87.27) takes SPOT as the ground truth. Since S3P3 is learned based on SPOT that is hardly a fair comparison. In fact I was impressed by the score of IRSGS-GIN considering it is not exposed to information captured by SPOT. I also think that for positioning the paper, it would have been good to include baselines based on Yoon et al. in an A-B design where the only substantial change from Yoon et al is adding the spatial node/edge features. I would hope that with such an incremental design you can better results both for semantic and for spatial relevance.

[update: I see new results that clarify the relationship to Yoon et al.]

Questions for authors
  1. Can you confirm that you are not arguing that using a scene graph is better than directly defining image similarity?
  2. How would S3P3 compare to a baseline based on the methods in Yoon et al with a spatially augmented scene graph? Specifically: 2a. Use their GWL method for optimal transport. What is the difference between their GWL method and your OT? Is it a straightforward extension to bounding box coordinates and force banners? 2b. What would happen if you applied the GCN/GIN from Yoon et al with the new spatial node/edge features?
  3. On the last note, why did you introduce a new graph encoder for S3P3 (see appendix D)? Rather than using existing graph encoders like Yoon et al did?
4 comments

Motivation. It is somewhat unclear what the task addressed is. If it is the general problem of defining a similarity metric for images, there are many proposals not addressed in this paper (e.g. FIT). I suspect the authors would say it is the more specific problem of how to define a similarity metric for scene graphs. But what is the motivation for this task? If the final motivation is to define similarities between images, which is what the experiments suggest, one can question whether extracting an intermediate discrete representation will work better than an end-to-end approach.

We thank the reviewer for their careful assessment and constructive remarks. We agree that the motivation and scope of our work could have been made clearer in the original submission, and have revised the introduction and conclusion accordingly. Our objective is to explicitly model spatial configuration similarity, independently of object semantics. In particular, we aim to compare how objects are positioned, sized, and arranged within a scene, regardless of their semantic categories. This motivates our use of Spatial Scene Graphs (SSGs), which provide a structured representation of absolute object geometry through bounding boxes and relative spatial information through Force Banners. SPOT is designed to quantify this specific notion of spatial similarity, while S3PE investigates whether such spatial information can be transferred to learned neural representations, including representations obtained directly from RGB images.

We agree with the reviewer's observation that image retrieval should usually prioritize semantic content over spatial layout. However, our goal is to investigate spatial similarity as a distinct retrieval criterion, and not to replace semantic retrieval. We emphasize that spatial and semantic similarity address different, complementary objectives.

In this context, the example in Figure 4, where SPOT retrieves an airplane in response to a query containing a cat, illustrates the intended behavior of our method rather than a failure of semantic retrieval. Despite belonging to different semantic categories, the images exhibit similar spatial configurations. This example demonstrates that SPOT can retrieve scenes based on their geometric organization rather than primarily on object identity.

In practical applications, spatial and semantic similarity could be combined to retrieve images that satisfy both geometric and semantic constraints. To illustrate the complementarity of both spatial and semantic retrieval, we provided an additional experiment using CLIP embeddings (that tend to focus on semantic content of the image) and reranking the top-100 candidates with respect to the SPOT similarities to the query (spatial retrieval). We detail the experiment in Section 6.2 of the revised submission.

Regarding existing image similarity metrics, we appreciate the suggestion to better position SPOT relative to previous approaches. If the reviewer is referring to the Fréchet Inception Distance (FID), the main distinction concerns the level of comparison: FID measures discrepancies between image distributions, whereas SPOT compares the spatial configurations of objects between individual image pairs. Our contribution therefore addresses a more specific notion of similarity rather than general-purpose visual similarity.

Regarding intermediate representations, we agree with the reviewer that end-to-end tends to beat intermediate representations and would like to emphasize that S3PE models are trainable in an end-to-end fashion.

Regarding the use of SPOT as the ground truth, we also agree with the reviewer that it does not offer fair comparison of the methods. As a relatively new task, Spatial similarity between images (with respect to the object configurations) offer no existing evaluation metric. We experimentally demonstrated the use of SPOT in downstream tasks without leveraging SPOT as the ground truth in both the VQA experiment (Section 6.2) and in the Appendix I on our cross-domain evaluation of S3PE.

(Q1): Can you confirm that you are not arguing that using a scene graph is better than directly defining image similarity?

We do not claim that scene graph-based representations are inherently superior to approaches that learn image similarity directly from RGB inputs. Our motivation for using Spatial Scene Graphs (SSGs) is methodological. SSGs explicitly encode object geometry through bounding boxes and relative geometric information through Force Banners, allowing us to define a spatial similarity measure and derive a geometry-driven supervision signal. In fact, our S3PE-Image model bridges these two approaches. This model is trained using triplets selected with SPOT values, but operates directly on RGB images at inference time, without requiring scene graph extraction. We acknowledge that errors in object detection or scene graph construction may affect the quality of the supervision signal during training. Our contribution is therefore not to establish the superiority of scene graphs over end-to-end visual models, but to investigate how explicit spatial relationships can guide the learning of spatially-aware representations.

(Q2): How would S3P3 compare to a baseline based on the methods in Yoon et al with a spatially augmented scene graph? Specifically: 2a. Use their GWL method for optimal transport. What is the difference between their GWL method and your OT? Is it a straightforward extension to bounding box coordinates and force banners? 2b. What would happen if you applied the GCN/GIN from Yoon et al with the new spatial node/edge features?

2a: We thank the reviewer for allowing us to draw the differences between GWL and SPOT. GWL, introduced by [1] and evaluated as a baseline by Yoon et al., and SPOT both rely on Gromov--Wasserstein optimal transport, but differ in their objectives and formulations. GWL jointly performs graph matching and node embedding learning through a Gromov--Wasserstein-based objective. In contrast, SPOT is a training-free spatial dissimilarity criterion that explicitly combines absolute object geometry, encoded by bounding boxes, with relative spatial configurations, encoded by Force Banners. In particular, SPOT defines node-level costs from bounding box differences and relational costs from the circular Wasserstein distance between normalized Force Banner angular profiles. These costs are integrated through an edge-attributed optimal transport formulation. Our contribution therefore lies in the spatial representation and the associated cost formulation, rather than in introducing a new general-purpose optimal transport algorithm.

2b: We thank the reviewer for their insightful proposition: we added experimental results of the IRSGS variants using spatial scene graphs as inputs in Table 1. The models using semantic scene graphs as inputs are better than their spatial variants with respect to semantic retrieval. Conversely, Spatial scene graphs offer better spatial retrieval scores than semantic scene graphs. This further highlights the interest of using such graphs for spatially-related tasks instead of semantic scene graphs that do not provide fine-grained spatial information.

[1]: Xu, Hongteng, et al. "Gromov-wasserstein learning for graph matching and node embedding." International conference on machine learning. PMLR, 2019.

(Q3): On the last note, why did you introduce a new graph encoder for S3P3 (see appendix D)? Rather than using existing graph encoders like Yoon et al did?

We introduced a new graph encoder for S3PE instead of reusing IRSGS as GIN and GCN are graph convolutions that do not natively leverage edge attributes during message passing. Edge attributes are valuable information when leveraging Spatial Scene Graphs as inputs, which led us to the different graph encoder (GATv2) with circular padding for input edges projection. We have added a version of IRSGS using GATv2 using either the semantic or spatial scene graph as input in Table 1 for fair comparison. Regarding the previous question (Q2b), when using spatial scene graphs, we performed dot product between the node embedding and edge attributes before the GIN (and GCN) message passing algorithms to account for edge values (Force Banners) and provide fair comparisons.

Reviewer_GsdrOpenReview ↗

reply to reply to my questions

thanks for your replies. I've got clarity on what the task is now (spatial relations). I believe you have introduced creative methods for solving this task. I'm less convinced that the vision community will care about this task to be blunt. Unless you can show that the spatial relation can add an extra information channel that can be combined with other visual information to solve standard tasks.For example, your VQA experiments improve nicely on the C-SIP baseline but SOTA are vision-language models. I still upgraded my rating because I like to be open-minded about introducing new tasks.

I'm happy with the answer to my questions. E.g. I can see that Gatv2 is good for rich edge features. My bad about FIT/FID. I accept that the optimal transport cost is quite different from yoon et al. even if the problem formulation looks similar at a high level.

We thank the reviewer for their positive and constructive feedback.

Regarding the applicability of spatial relations to standard vision tasks, our additional re-ranking experiment (Section 6.2 and Table 2 of the revised article) illustrates how spatial similarity can complement semantic image retrieval.

Comments on the paper

1 comment

We thank the reviewers for their insightful feedback and questions about our work. We are pleased that they found SPOT to be novel (Gsdr), useful (4j9C), and interesting (WBfC) that addresses a representation gap (WBfC). We are encouraged that reviewers found the experiments to be a convincing case that spatial relations are captured by SPOT (Gsdr) with experimental goals clearly stated (4j9C).

We revised the submission by taking into account the various comments of the Reviewers. Modifications are written in blue. The main revisions comprise:

  • We have reformulated the motivation to clarify the task we address
  • Additional baseline models in Table 1 (Gsdr, WBfC);
  • An additional experiment showing the complementarity of spatial and semantic retrieval using CLIP embeddings (Gsdr,4j9C);
  • Additional ablation studies in Appendices D and E (4j9C);

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.