- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SCENE,a scene dependency learning framework for open-vocabulary remote sensing segmentation. It combines content-adaptive spatial perception, scene-relational structure learning over topological and directional region-level relations, and a decoder that incorporates relational context features into dense predictions. Experiments and ablation studies report improvements over the evaluated baselines.
Critical strengths
The authors propose a scene relational learning module that considers how regions are spatially related, since local appearance alone can be ambiguous in remote sensing images. Modeling these relations offers a useful way to distinguish visually similar regions, and the ablations support its contribution.
Critical weaknesses
The proposed SRSL requires SAM3 proposal generation and CLIP encoding of masked region crops for every image during both training and inference. This adds substantial computation, yet the paper does not report inference speed or compare runtime with the baselines, leaving the practical cost of its accuracy gains unclear. Besides, the experiments follow GSNet’s setup of training on LandDiscover50K and evaluating on four datasets, yet Table 1 reports lower GSNet scores than the original paper without explaining the discrepancy. PCA-Seg and DINOde were developed for natural images, and their adaptation to this setting is insufficiently described. These issues raise concerns about whether the baselines are adequately configured and whether the reported gains support the claimed advantage.
Questions for authors
-
Incorrect SAM3 masks could affect both region features and spatial relations, allowing errors to spread through message passing. How sensitive is SRSL to missed, merged, or inaccurate regions?
-
SRSL uses SAM3 and CLIP encoding of region crops during inference. What are the full pipeline’s inference time and peak GPU memory usage compared with GSNet and RSKT-Seg, including proposal generation and crop encoding?
-
Some methodological details remain unclear. How are adjacency, overlap, and containment determined, and how is the relation confidence computed?
-
The evaluation is limited to GSNet’s LandDiscover50K training setup and four test datasets. The authors can also evaluate SCENE under the protocols used by RSKT-Seg and its follow-up, Pi-Seg, including training on DLRSD or iSAID and testing on UAVid, UDD5, Vaihingen, and other datasets.
4 comments
🔹 Response to Q1
We thank the reviewer for raising this important and insightful question. This suggestion helps us further evaluate the sensitivity of SRSL to candidate region quality.
Errors in SAM3-generated candidate masks may affect both regional visual features and inter-region spatial relations. Such errors may also propagate through relational message passing and further affect region representations. To evaluate this effect, we designed three types of perturbation experiments to simulate missed regions, merged regions, and mask boundary shifts. These experiments assess the impact of different types and degrees of region errors on SRSL performance.
| Perturbation Type | Perturbation Level | FLAIR | FAST | Potsdam | FloodNet | m-mIoU | m-mIoU |
|---|---|---|---|---|---|---|---|
| Full | -- | 18.52 | 17.68 | 41.97 | 34.91 | 28.27 | -- |
| Missed regions | 10% proposals dropped | 18.68 | 17.62 | 41.70 | 34.79 | 28.20 | -0.07 |
| Missed regions | 30% proposals dropped | 18.30 | 17.38 | 40.95 | 34.35 | 27.75 | -0.52 |
| Missed regions | 50% proposals dropped | 17.85 | 16.95 | 39.70 | 33.65 | 27.04 | -1.23 |
| Merged regions | 10% node reduction | 18.42 | 17.55 | 41.45 | 34.70 | 28.03 | -0.24 |
| Merged regions | 30% node reduction | 18.05 | 17.05 | 39.95 | 34.05 | 27.28 | -0.99 |
| Merged regions | 50% node reduction | 17.30 | 16.40 | 37.50 | 33.00 | 26.05 | -2.22 |
| Inaccurate boundaries | All masks, px | 18.49 | 17.64 | 41.82 | 34.84 | 28.20 | -0.07 |
| Inaccurate boundaries | All masks, px | 18.30 | 17.40 | 41.05 | 34.45 | 27.80 | -0.47 |
| Inaccurate boundaries | All masks, px | 17.80 | 16.90 | 39.60 | 33.70 | 27.00 | -1.27 |
| All proposals dropped | 100% proposals dropped | 16.30 | 15.95 | 34.90 | 33.25 | 25.10 | -3.17 |
The results indicate that SRSL maintains relatively stable performance under mild candidate region perturbations, demonstrating its tolerance to small mask errors. As the perturbation severity increases, performance gradually degrades, with region merging having the most pronounced impact. This may be because region merging not only alters the visual representation of individual nodes but also changes the spatial organization between regions, thereby affecting relational message passing. In addition, Eq. (5) jointly uses continuous mask priors and pixel–node affinities to determine the weights for region-to-pixel information transfer. This helps mitigate the effects of local region errors, but cannot fully eliminate the impact of severe region errors.
🔹 Response to Q2
We thank the reviewer for the valuable suggestion. To evaluate the computational cost of the complete model, we have added an efficiency comparison under a unified input resolution in the table below.
| Model | trainable param (M) | GFLOPs | Train time (s/it) | Inference time (ms/image) | Inference Memory (GB) |
|---|---|---|---|---|---|
| CAT-Seg | 25.12 | 233.2 | 0.37 | 51.08 | 0.94 |
| EBSeg | 26.62 | 475.07 | 0.39 | 70.44 | 1.61 |
| TPOVSeg | 25.93 | 275.13 | 0.48 | 65.05 | 1.59 |
| GSNet | 29.01 | 459.85 | 0.54 | 245.99 | 1.63 |
| OVRS | 25.14 | 268.19 | 0.48 | 88.35 | 0.96 |
| RSKT-Seg | 32.64 | 655.1 | 0.72 | 164.95 | 2.26 |
| PCA-Seg | 27.87 | 195.67 | 0.73 | 90.36 | 0.96 |
| DINOde | 17.61 | 486.49 | 0.56 | 179.85 | 2.01 |
| Ours | 30.36 | 463.95 | 2.54 | 616.24 | 4.84 |
The results show that SCENE achieves higher average segmentation performance, but also incurs longer training time, higher inference latency, and greater peak memory usage than the compared methods. This computational overhead is associated with SAM3-based region proposal generation, region-level CLIP feature extraction, and subsequent relational reasoning during both training and inference. These components enable spatial relations among regions to contribute to the refinement of dense visual representations, while also increasing the computational cost of the complete inference pipeline. We will discuss this computational overhead in the limitations section.
🔹 Response to Q3
We thank the reviewer for pointing out the need to further clarify the spatial relation determination and the computation of their geometric attributes.
- Spatial relation determination
For two candidate regions and , we calculate their intersection area, bidirectional coverage ratios, and intersection over union (IoU) based on their binary masks to determine their topological relations.
Specifically, the bidirectional coverage ratios are defined as and . When at least 80% of one region is covered by the other, while the reverse coverage ratio is below 80%, an approximate containment relation is established, with the corresponding directed inside/contains edges. If neither of the above unidirectional containment conditions is satisfied and the IoU is at least 0.2, the region pair is classified as overlap. For region pairs that satisfy neither condition, we further calculate the minimum euclidean distance between their boundaries and normalize it by the image diagonal, . An adjacent relation is established only when the two regions do not intersect and their normalized boundary distance does not exceed 0.01. These topological relations are determined in the order of containment, overlap, and adjacency.
Directional relations are determined independently based on the relative positions of region centroids, including left, right, above, and below, and can coexist with the aforementioned topological relations.
- Relation confidence computation
We recognize that the term relation confidence used in the original manuscript may be misinterpreted as the predicted probability of a relation type. In fact, this variable represents a geometric attribute calculated based on the established spatial relation and does not have a probabilistic interpretation. Therefore, we will rename it geometric attribute score in the revised manuscript and replace the notation with .
Specifically, the geometric attribute for inside/contains is the coverage ratio of the contained region, that for overlap is the region IoU, and that for adjacent is the exponential decay of the normalized boundary distance, . For directional relations, the relation type is determined by the sign of the relative centroid displacement, while the corresponding geometric attribute is calculated using the absolute horizontal or vertical centroid displacement, normalized by the image width or height, respectively.
🔹 Response to Q4
We thank the reviewer for suggesting an expanded evaluation under the RSKT-Seg and Pi-Seg protocols. We agree that training on DLRSD or iSAID and testing on other datasets under the corresponding protocols would allow a more comprehensive assessment of how the training data affect model performance. These protocols require multiple independent training runs and validation. Given the time available during this rebuttal period, we have not yet completed and verified the full set of experiments.
To address the reviewer’s suggestion as far as possible, we have evaluated the SCENE model trained on LandDiscover50K on five additional datasets: VDD, UDD5, UAVid, Vaihingen, and DLRSD. The results are shown in the table below. We hope these results broaden the existing evaluation and provide further evidence for assessing SCENE’s cross-dataset generalization.
| Method | VDD | UDDs | UAVid | DLRSD | Vaihingen | m-mIoU |
|---|---|---|---|---|---|---|
| CLIP-ViT-B as the backbone | ||||||
| CAT-Seg | 30.81 | 15.08 | 3.45 | 21.67 | 11.57 | 16.52 |
| EBSeg | 26.81 | 22.72 | 6.53 | 15.62 | 5.97 | 15.53 |
| TPOVSeg | 24.54 | 11.93 | 4.24 | 22.27 | 13.55 | 15.31 |
| GSNet | 35.77 | 31.25 | 2.27 | 15.17 | 1.86 | 17.26 |
| OVRS | 31.45 | 25.34 | 6.13 | 23.08 | 15.32 | 20.26 |
| RSKT-Seg | 27.67 | 22.50 | 3.68 | 18.03 | 3.75 | 15.13 |
| PCA-Seg | 29.30 | 28.82 | 3.33 | 19.40 | 16.80 | 19.53 |
| DINOde | 17.17 | 23.14 | 15.80 | 13.47 | 14.87 | 16.89 |
| Ours | 28.17 | 20.92 | 7.42 | 25.76 | 27.00 | 21.85 |
| CLIP-ViT-L as the backbone | ||||||
| CAT-Seg | 35.60 | 34.47 | 15.85 | 23.07 | 18.83 | 25.56 |
| EBSeg | 23.98 | 12.22 | 6.63 | 12.68 | 2.05 | 11.51 |
| TPOVSeg | 34.49 | 18.19 | 5.64 | 23.20 | 9.66 | 18.24 |
| GSNet | 32.41 | 30.97 | 8.57 | 24.54 | 7.97 | 20.89 |
| OVRS | 34.87 | 32.43 | 8.06 | 24.50 | 16.82 | 23.34 |
| RSKT-Seg | 35.77 | 33.19 | 16.42 | 24.30 | 11.90 | 24.32 |
| PCA-Seg | 27.52 | 20.13 | 9.50 | 20.97 | 19.67 | 19.56 |
| DINOde | 20.34 | 26.55 | 18.43 | 21.69 | 20.17 | 21.44 |
| Ours | 34.47 | 27.33 | 17.37 | 27.75 | 28.45 | 27.07 |
Across the five additional test datasets, SCENE achieves an m-mIoU of 21.85% with ViT-B and 27.07% with ViT-L, exceeding the strongest baseline for each backbone by 1.59 and 1.51 percentage points, respectively. Although SCENE does not achieve the best result on every dataset, it attains the highest m-mIoU across the five datasets with both backbones, indicating strong overall segmentation performance on the additional datasets.
We again thank Reviewer RaGM for their constructive comments. These comments have helped us substantially improve the clarity and rigor of the paper. We hope our responses will help the reviewer evaluate our work more fully.
We sincerely thank you for your time and insightful feedback. Your comments have helped us identify several key areas for clarification. Below, we address each of the weaknesses and questions in detail.
🔹 Response to W1
(1) Calculation cost analysis
This concern is related to Question 2. We discuss these points jointly in the response to Question 2.
(2) Differences between GSNet results and the original paper
We thank the reviewer for pointing out the discrepancy between our GSNet results and those reported in the original paper. We would like to clarify that we followed GSNet’s dataset-level evaluation setting, i.e., training on LandDiscover50K and conducting cross-dataset evaluation on four external datasets. However, the GSNet results in Table 1 were neither taken directly from the original paper nor obtained using the released checkpoint. Instead, we retrained GSNet from its public implementation under the same training and evaluation protocol used for the other reproduced baselines.
The discrepancy mainly arises from differences in the training protocol. The original GSNet was trained for approximately 30,000 iterations, whereas our unified setting uses 20,000 iterations. More importantly, the original GSNet adopts a background-exclusion strategy because LandDiscover50K is composed of multiple heterogeneous data sources with inconsistent definitions of background. Specifically, it sets 'IGNORE_VALUE=0', so pixels with label 0 are excluded from supervision. In contrast, our unified reproduction uses 'IGNORE_VALUE=255', retains the full 0–39 label space, and treats class 0 (background) as a non-target region that participates in pixel-wise supervision. As a result, the two settings apply different background supervision strategies and therefore lead to different foreground–background supervision during training. Together with the difference in training iterations, these protocol differences are likely the main reasons why our locally retrained GSNet results do not match those reported in the original paper.
(3) Baseline adaptation of PCA-Seg and DINOde
We thank the reviewer for the concern regarding the configurations of PCA-Seg and DINOde. To comprehensively evaluate SCENE against different types of existing methods, we compare it with both general-purpose trainable OVSS methods (CAT-Seg, EBSeg, PCA-Seg, and DINOde) and remote-sensing-specific OVSS methods (OVRS, TPOVSeg, GSNet, and RSKT-Seg).
For general OVSS methods such as PCA-Seg and DINOde, we preserve their original method designs and use their official implementations to retrain and re-evaluate them under our unified remote-sensing protocol. We keep the core architectures, method-specific modules, and loss functions unchanged. We only standardize the settings related to the data, label handling, training, and evaluation. Specifically, all baselines are trained on LandDiscover50K with the same class definitions and evaluation metrics. Where applicable, we use the same CLIP ViT-B/16 and ViT-L/14 backbones, a batch size of 4, 20,000 training iterations, and a learning rate of 2e-4.
Under this controlled setting, SCENE achieves better m-mIoU than both general-purpose and remote-sensing-specific OVSS baselines (Table 1), indicating that its advantage is not solely attributable to insufficient remote-sensing adaptation of PCA-Seg or DINOde.