- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors argue that visual similarity in deep learning is typically assessed semantically, by shared object content, while the spatial arrangement of objects is largely ignored. They make three contributions:
-
Spatial Scene Graph (SSG): A scene graph where nodes encode objects' absolute geometry (normalized bounding box center/size), and edges encode relative position using Force Banners, a descriptor capturing directional and distance relations between objects.
-
SPOT: A training-free measure of spatial similarity between SSGs, formulated as a fused Gromov–Wasserstein optimal transport problem solved using the with Frank-Wolfe method. It is a convex combination of a node cost (bounding box differences) and a relational cost (circular Wasserstein distance between the edges’ angular profiles), weighted by α = 0.5.
-
S3PE: A model-agnostic self-supervised method that uses SPOT rankings to mine positive and negative triplets from a dataset and trains encoders with a cosine triplet loss. It is applied to a fine-tuned DINOv2 image encoder and to GATv2-based GNNs operating on either semantic or spatial scene graphs.
The experimental section evaluates on PSG (human annotated scene graphs + object masks), and score against both semantic relevance (SPICE) and spatial relevance (SPOT). They also test the method's ability to perform spatial VQA. The authors report that S3PE outperforms CLIP, DINOv2, RelTR and IRSGS on spatial retrieval at the cost of semantic retrieval, and that frozen S3PE-Image embeddings outperform prior other spatially aware representations (C-SIP) on VQA.
Critical strengths
-
The method put forward is novel and useful, and is demonstrated on downstream tasks (spatial VQA, and scene classification from scene graphs alone).
-
The authors measure image similarity by the spatial arrangement of objects through a principled formulation. Using optimal transport, they can compare scene graphs with different numbers of objects, and the resulting transport plan is interpretable.
-
It is clearly shown that spatial similarity is a distinct notion from semantic similarity which current methods miss: Encoders such as CLIP and DINOv2 barely beat random on spatial retrieval. The authors evaluate not only spatial relevance but also the method’s impact on semantic relevance, exposing a potential trade-off.
-
The method is model-agnostic and can be applied to images, semantic graphs and spatial graphs.
-
The paper is well organized and the method is easy to follow: each component (SSG, then SPOT, then S3PE) builds on the previous one, and the experimental goals are clearly stated.
Critical weaknesses
-
The authors list as a contribution that SPOT is experimentally demonstrated as an evaluation criterion for spatial similarity in retrieval tasks. However, this is only shown qualitatively. All the quantitative spatial results (e.g. Table 1) use SPOT itself as the reference ranking, and S3PE is trained on triplets mined from SPOT, so S3PE outperforming the baselines on spatial relevance shows that it learned to reproduce SPOT, rather than that SPOT captures spatial similarity. In order for the paper to support SPOT as an evaluation criterion, it should be validated against an independent reference.
-
VQA is the only evaluation that does not use SPOT as the reference, but S3PE-Image is only evaluated against C-SIP and C-SIP++. The paper should state which backbone C-SIP uses: if it differs from DINOv2, the improvement could come from the backbone rather than from SPOT supervision. In either case, a frozen DINOv2 baseline (and CLIP, as in Table 1) under the same protocol would show how much SPOT fine-tuning adds over the pretrained encoder. The classifier used on top of the concatenated image and question embeddings should also be described
-
The alpha term is only measured qualitatively, at three values. A quantitative sweep over α, e.g. on SharvitSR or VQA would establish a clearer relationship/tradeoff between absolute and relative position information.
With respect to presentation/organization of the paper:
- A figure showing banners and the derived angular profiles for a few configurations would help someone not familiar with Force Banner
- The introduction would benefit from a concrete motivating application, along with the downstream results (VQA, scene classification). In particular, when is spatial-only similarity desirable?
- Table 1 is dense. The metrics need more explanation, as does why random scores so high on spatial NDCG.
Questions for authors
-
SSGs and SPOT encode spatial configuration only in the 2D image plane, so depth relations (e.g. “in front of”, “behind”) and viewpoint changes are not captured, and bounding box size conflates object size with distance from the camera. Could the authors clarify this scope, and discuss whether adding depth information (e.g. from monocular depth estimation) to the node features would be feasible?
-
Why was Frank–Wolfe chosen as the solver? The paper does not motivate this choice or discuss alternatives. What is the runtime per SPOT computation, and in total for the landmark matrix? Would an entropic-regularized solver (Peyré et al., 2016) make computing the exact full SPOT matrix feasible, removing the need for the landmark approximation in Appendix C?
-
In what applications is spatial-only similarity preferable to semantic similarity, and could the two be combined?
-
How sensitive are results to the negative pool range (ranks 1000–5000) and the positive pool size (100)?
5 comments
(Q1) : SSGs and SPOT encode spatial configuration only in the 2D image plane, so depth relations (e.g. “in front of”, “behind”) and viewpoint changes are not captured, and bounding box size conflates object size with distance from the camera. Could the authors clarify this scope, and discuss whether adding depth information (e.g. from monocular depth estimation) to the node features would be feasible?
We agree that the current formulation of SPOT focuses on spatial configurations in the 2D image plane. It does not explicitly model depth, physical object size, or viewpoint invariant 3D spatial relationships. In particular, bounding box dimensions reflect the apparent size of objects in the image and may therefore be influenced by their distance from the camera.
We would like to emphasize that SPOT is designed as a generic framework for comparing spatial configurations across different image domains, rather than being restricted to natural images acquired from a particular viewpoint. For instance, our cross-domain experiment on the STAR remote sensing dataset (Appendix I) demonstrates its applicability to aerial imagery, where scenes are observed from an overhead perspective and spatial configurations can be meaningfully characterized in the 2D image plane without explicit depth information.
Extending SPOT to incorporate depth information is conceptually feasible within the existing optimal transport. For example, monocular depth estimates could be aggregated over object segmentation masks and incorporated into the node attributes. This would require adapting the node cost function to account for depth differences, while preserving the overall SPOT formulation.
Such an extension would be particularly relevant for spatial relationships involving depth, such as in front of and _behind, which cannot always be reliably distinguished from 2D geometry alone. Incorporating relative depth information could provide a more fine-grained characterization of these relationships. We therefore consider depth-aware spatial representations a promising direction for future work, beyond the scope of the present study.
(Q2) : Why was Frank–Wolfe chosen as the solver? The paper does not motivate this choice or discuss alternatives. What is the runtime per SPOT computation, and in total for the landmark matrix? Would an entropic-regularized solver (Peyré et al., 2016) make computing the exact full SPOT matrix feasible, removing the need for the landmark approximation in Appendix C?
We chose Franke--Wolfe as the OT solver as it is commonly applied for Gromov-Wasserstein algorithms. The main bottleneck of the computation of the full SPOT matrix is the number of OT solve required. However, entropic regularization (such as the Sinkhorn algorithm) benefit from large graph structures to solve faster and does not reduce the number of OT solves. Due to the structure of the scene graphs of the training set of PSG, entropic-regularized solver does not enable the computational feasibility of the full SPOT matrix.
We performed a full SPOT matrix computation on 1,000 random spatial graphs of PSG using consumer-grade CPU. The full (1000,1000) matrix computation with Franke--Wolfe solver took 2 hours and 21 minutes; using entropic-regularized solver computed the matrix in 2 hours and 16 minutes. By considering proportional computation of the choice of the landmarks considered, and the 20-fold fewer OT solves, the matrix of SPOT values was computed in 6 minutes, and a few seconds for the full matrix approximation.
(Q3) : In what applications is spatial-only similarity preferable to semantic similarity, and could the two be combined?
Spatial-only similarity can be preferable to semantic-only similarity in many applications such as medical imagery. Specifically, blood vessels characterization in retinal imaging [1], and lung cancer classification in histopathological analyses [2] can greatly benefit from the spatial configuration of the objects (blood vessel and cells respectively). In remote sensing imagery, spatial similarity can be leveraged for retrieval similar Local Climate Zones (LCZ) and help urbanization planning [3].
Furthermore, spatial and semantic similarity can be combined to retrieve images that are both semantically relevant and spatially similar to a query. To demonstrate this practical application, we added a two-stage retrieval experiment in Section 6.2 of the revised manuscript. CLIP first retrieves the top-100 semantically relevant candidates, which are then reranked according to their SPOT distances to the query. This approach improves spatial NDCG@25 from 69.09 to 86.83 (+17.74 points), while semantic NDCG@25 only decreases from 62.33 to 58.62 (-3.71 points). These results illustrate how SPOT can complement existing semantic retrieval methods by prioritizing spatial configurations while retaining much of the initial semantic relevance.
[1]: Clément, Michaël, et al. "Directional enlacement histograms for the description of complex spatial configurations between objects." IEEE TPAMI (2016): 2366-2380.
[2]: Desharnais, Lysanne, et al. "Spatially mapping the tumour immune microenvironments of non-small cell lung cancer." Nature communications 16.1 (2025): 1345.
[3]: Zhu, Xiao Xiang, et al. "The urban morphology on our planet–Global perspectives from space." Remote Sensing of Environment 269 (2022): 112794.
(Q4) : How sensitive are results to the negative pool range (ranks 1000–5000) and the positive pool size (100)?
We thank the reviewer for this suggestion. We conducted an additional sensitivity analysis on the PSG dataset for all three S3PE variants (Image, SG, and SSG), varying the positive pool size (50, 100, and 200 nearest neighbors) and the negative sampling range (ranks 500--2000, 1000--5000, and 5000--10000), while keeping the remaining training and evaluation settings unchanged.
Th ablation study has been added to Appendix D.2 of the revised manuscript.
As shown in Table 4, S3PE-SSG exhibits limited sensitivity to these parameters, with NDCG@25 ranging from to and MP@100 from to . S3PE-SG also remains relatively stable, with NDCG@25 between and . In contrast, S3PE-Image is more sensitive to negative sampling, particularly when using harder negatives (ranks 500--2000), which reduce NDCG@25 from to .
Overall, the original configuration (100 positive neighbors and negative ranks 1000--5000) provides the best results for S3PE-Image and S3PE-SSG, while remaining competitive for S3PE-SG. These findings indicate that the sensitivity to triplet sampling depends on the input representation, with spatial scene graph embeddings showing the greatest stability.
(W1) : The authors list as a contribution that SPOT is experimentally demonstrated as an evaluation criterion for spatial similarity in retrieval tasks. However, this is only shown qualitatively. All the quantitative spatial results (e.g. Table 1) use SPOT itself as the reference ranking, and S3PE is trained on triplets mined from SPOT, so S3PE outperforming the baselines on spatial relevance shows that it learned to reproduce SPOT, rather than that SPOT captures spatial similarity. In order for the paper to support SPOT as an evaluation criterion, it should be validated against an independent reference.
We thank the reviewer for raising this important point. We agree that evaluating SPOT against an independent spatial similarity reference, particularly human judgements, would provide stronger validation of the proposed criterion. However to the best of our knowledge, there is no established reference metric specifically designed to evaluate the similarity of global spatial configurations between objects across scenes, as existing visual similarity metrics predominantly focus on semantic content. This lack of an independent spatial similarity criterion is precisely one of the challenges motivating our work. We therefore provide controlled qualitative experiments on synthetic and natural scenes in Appendix F, illustrating how SPOT retrieves scenes with similar spatial configurations. We also investigate the usefulness of SPOT-guided representations through an independent downstream task, Visual Question Answering (VQA). Unlike our retrieval evaluation, VQA relies on annotated answers rather than SPOT-based relevance scores. Under the same evaluation protocol, S3PE-Image achieves 80.71% accuracy on spatial relation questions, compared to 67.61% for the frozen DINOv2 baseline. These results provide complementary quantitative evidence that SPOT-guided supervision benefits spatial scene understanding beyond reproducing SPOT-based retrieval rankings. While they do not directly establish agreement between SPOT and human judgments of spatial similarity, they support the practical usefulness of the spatial information captured through SPOT.
(W2) : VQA is the only evaluation that does not use SPOT as the reference, but S3PE-Image is only evaluated against C-SIP and C-SIP++. The paper should state which backbone C-SIP uses: if it differs from DINOv2, the improvement could come from the backbone rather than from SPOT supervision. In either case, a frozen DINOv2 baseline (and CLIP, as in Table 1) under the same protocol would show how much SPOT fine-tuning adds over the pretrained encoder. The classifier used on top of the concatenated image and question embeddings should also be described
We thank the reviewer for their feedback which strengthens our experiments. We have added a DINOv2 baseline to the VQA experiment (See Table 3 of the revised submission) under the same protocol used for S3PE-Image. This setup leads the frozen DINOv2 model to reach an accuracy of 67.61, below our S3PE-Image, showing the interest of using spatially-aware representations to answer related questions. The classifier used after the concatenation is a 3-layer feed-forward network with a classification layer. We clarify this setup in the main paper (Section 6.3).
(W3) : The alpha term is only measured qualitatively, at three values. A quantitative sweep over α, e.g. on SharvitSR or VQA would establish a clearer relationship/tradeoff between absolute and relative position information.
We thank the reviewer for this suggestion. We have added a quantitative sensitivity analysis of α∈{0,0.25,0.5,0.75,1} on the PSG test set, using 500 query graphs and 200 candidates per query, repeated three times (see Appendix E of the revised manuscript).
Compared to α=0.5, moderate variations yield high Spearman rank correlations of 0.917±0.003 (α=0.25) and 0.929±0.003 (α=0.75), with Top-25 overlaps of 75.0% and 79.9%, respectively. In contrast, using only relative (α=0) or absolute (α=1) spatial information leads to more distinct rankings, with Top-25 overlaps of 40.9% and 64.0%.
These results show that moderate variations of α preserve substantial ranking agreement, while excluding either component has a stronger impact. We therefore retain α=0.5 as a balanced default, without claiming that it is empirically optimal.
For the presentation and organization of the paper, We thank the review for pointing out the lack in clarity in some parts of the paper. We have added qualitative examples of Force Banners in Appendix G to highlight the content of the Force Banner. Furthermore, we have reduced the density of Table 1 to improve clarity.