acceptodds
Withdrawn from ICLR 2027

Not in Isolation: Learning from Scene Dependencies for Open-Vocabulary Remote Sensing Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although existing open-vocabulary semantic segmentation (OVSS) methods have made significant progress, segmenting complex remote sensing images remains challenging. Regions of the same semantic class often varies widely in scale and shape, while different classes may exhibit similar local appearances. These ambiguities can be reduced by using spatial relationships between regions and their surroundings as complementary cues. However, such scene dependencies remain insufficiently modeled in OVSS. Therefore, we propose SCENE, a scene dependency learning framework that incorporates scene dependencies into open-vocabulary visual representations through content-adaptive multi-granularity perception at the location level and spatial relation modeling at the region level. Specifically, Content-Adaptive Spatial Perception adaptively aggregates information across multiple spatial granularities for each position based on image content; Scene-Relational Structure Learning treats category-agnostic regions as entities, propagates region information through multiple types of spatial relations, and maps the resulting relational representations into pixel space to generate dense relational context; Structure-Guided Fusion Decoder further uses relational context to guide the refinement of spatial perception representations and dense visual representations and performs progressive decoding with multi-level features to produce segmentation predictions. Extensive experiments on four remote sensing benchmarks demonstrate that SCENE effectively improves OVSS performance and generalization across datasets. Code will be available at https://anonymous.4open.science/r/Anonymous-DD21.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Semantic Segmentation, Remote Sensing, Vision-Language Models, Segment Anything Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RaGMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SCENE,a scene dependency learning framework for open-vocabulary remote sensing segmentation. It combines content-adaptive spatial perception, scene-relational structure learning over topological and directional region-level relations, and a decoder that incorporates relational context features into dense predictions. Experiments and ablation studies report improvements over the evaluated baselines.

Critical strengths

The authors propose a scene relational learning module that considers how regions are spatially related, since local appearance alone can be ambiguous in remote sensing images. Modeling these relations offers a useful way to distinguish visually similar regions, and the ablations support its contribution.

Critical weaknesses

The proposed SRSL requires SAM3 proposal generation and CLIP encoding of masked region crops for every image during both training and inference. This adds substantial computation, yet the paper does not report inference speed or compare runtime with the baselines, leaving the practical cost of its accuracy gains unclear. Besides, the experiments follow GSNet’s setup of training on LandDiscover50K and evaluating on four datasets, yet Table 1 reports lower GSNet scores than the original paper without explaining the discrepancy. PCA-Seg and DINOde were developed for natural images, and their adaptation to this setting is insufficiently described. These issues raise concerns about whether the baselines are adequately configured and whether the reported gains support the claimed advantage.

Questions for authors
  1. Incorrect SAM3 masks could affect both region features and spatial relations, allowing errors to spread through message passing. How sensitive is SRSL to missed, merged, or inaccurate regions?

  2. SRSL uses SAM3 and CLIP encoding of region crops during inference. What are the full pipeline’s inference time and peak GPU memory usage compared with GSNet and RSKT-Seg, including proposal generation and crop encoding?

  3. Some methodological details remain unclear. How are adjacency, overlap, and containment determined, and how is the relation confidence computed?

  4. The evaluation is limited to GSNet’s LandDiscover50K training setup and four test datasets. The authors can also evaluate SCENE under the protocols used by RSKT-Seg and its follow-up, Pi-Seg, including training on DLRSD or iSAID and testing on UAVid, UDD5, Vaihingen, and other datasets.

4 comments

We sincerely thank you for your time and insightful feedback. Your comments have helped us identify several key areas for clarification. Below, we address each of the weaknesses and questions in detail.

🔹 Response to W1

(1) Calculation cost analysis

This concern is related to Question 2. We discuss these points jointly in the response to Question 2.


(2) Differences between GSNet results and the original paper

We thank the reviewer for pointing out the discrepancy between our GSNet results and those reported in the original paper. We would like to clarify that we followed GSNet’s dataset-level evaluation setting, i.e., training on LandDiscover50K and conducting cross-dataset evaluation on four external datasets. However, the GSNet results in Table 1 were neither taken directly from the original paper nor obtained using the released checkpoint. Instead, we retrained GSNet from its public implementation under the same training and evaluation protocol used for the other reproduced baselines.

The discrepancy mainly arises from differences in the training protocol. The original GSNet was trained for approximately 30,000 iterations, whereas our unified setting uses 20,000 iterations. More importantly, the original GSNet adopts a background-exclusion strategy because LandDiscover50K is composed of multiple heterogeneous data sources with inconsistent definitions of background. Specifically, it sets 'IGNORE_VALUE=0', so pixels with label 0 are excluded from supervision. In contrast, our unified reproduction uses 'IGNORE_VALUE=255', retains the full 0–39 label space, and treats class 0 (background) as a non-target region that participates in pixel-wise supervision. As a result, the two settings apply different background supervision strategies and therefore lead to different foreground–background supervision during training. Together with the difference in training iterations, these protocol differences are likely the main reasons why our locally retrained GSNet results do not match those reported in the original paper.


(3) Baseline adaptation of PCA-Seg and DINOde

We thank the reviewer for the concern regarding the configurations of PCA-Seg and DINOde. To comprehensively evaluate SCENE against different types of existing methods, we compare it with both general-purpose trainable OVSS methods (CAT-Seg, EBSeg, PCA-Seg, and DINOde) and remote-sensing-specific OVSS methods (OVRS, TPOVSeg, GSNet, and RSKT-Seg).

For general OVSS methods such as PCA-Seg and DINOde, we preserve their original method designs and use their official implementations to retrain and re-evaluate them under our unified remote-sensing protocol. We keep the core architectures, method-specific modules, and loss functions unchanged. We only standardize the settings related to the data, label handling, training, and evaluation. Specifically, all baselines are trained on LandDiscover50K with the same class definitions and evaluation metrics. Where applicable, we use the same CLIP ViT-B/16 and ViT-L/14 backbones, a batch size of 4, 20,000 training iterations, and a learning rate of 2e-4.

Under this controlled setting, SCENE achieves better m-mIoU than both general-purpose and remote-sensing-specific OVSS baselines (Table 1), indicating that its advantage is not solely attributable to insufficient remote-sensing adaptation of PCA-Seg or DINOde.

🔹 Response to Q1

We thank the reviewer for raising this important and insightful question. This suggestion helps us further evaluate the sensitivity of SRSL to candidate region quality.

Errors in SAM3-generated candidate masks may affect both regional visual features and inter-region spatial relations. Such errors may also propagate through relational message passing and further affect region representations. To evaluate this effect, we designed three types of perturbation experiments to simulate missed regions, merged regions, and mask boundary shifts. These experiments assess the impact of different types and degrees of region errors on SRSL performance.

Perturbation TypePerturbation LevelFLAIRFASTPotsdamFloodNetm-mIoUΔ\Delta m-mIoU
Full--18.5217.6841.9734.9128.27--
Missed regions10% proposals dropped18.6817.6241.7034.7928.20-0.07
Missed regions30% proposals dropped18.3017.3840.9534.3527.75-0.52
Missed regions50% proposals dropped17.8516.9539.7033.6527.04-1.23
Merged regions10% node reduction18.4217.5541.4534.7028.03-0.24
Merged regions30% node reduction18.0517.0539.9534.0527.28-0.99
Merged regions50% node reduction17.3016.4037.5033.0026.05-2.22
Inaccurate boundariesAll masks, ±2\pm 2 px18.4917.6441.8234.8428.20-0.07
Inaccurate boundariesAll masks, ±4\pm 4 px18.3017.4041.0534.4527.80-0.47
Inaccurate boundariesAll masks, ±8\pm 8 px17.8016.9039.6033.7027.00-1.27
All proposals dropped100% proposals dropped16.3015.9534.9033.2525.10-3.17

The results indicate that SRSL maintains relatively stable performance under mild candidate region perturbations, demonstrating its tolerance to small mask errors. As the perturbation severity increases, performance gradually degrades, with region merging having the most pronounced impact. This may be because region merging not only alters the visual representation of individual nodes but also changes the spatial organization between regions, thereby affecting relational message passing. In addition, Eq. (5) jointly uses continuous mask priors and pixel–node affinities to determine the weights for region-to-pixel information transfer. This helps mitigate the effects of local region errors, but cannot fully eliminate the impact of severe region errors.


🔹 Response to Q2

We thank the reviewer for the valuable suggestion. To evaluate the computational cost of the complete model, we have added an efficiency comparison under a unified input resolution in the table below.

Modeltrainable param (M)GFLOPsTrain time (s/it)Inference time (ms/image)Inference Memory (GB)
CAT-Seg25.12233.20.3751.080.94
EBSeg26.62475.070.3970.441.61
TPOVSeg25.93275.130.4865.051.59
GSNet29.01459.850.54245.991.63
OVRS25.14268.190.4888.350.96
RSKT-Seg32.64655.10.72164.952.26
PCA-Seg27.87195.670.7390.360.96
DINOde17.61486.490.56179.852.01
Ours30.36463.952.54616.244.84

The results show that SCENE achieves higher average segmentation performance, but also incurs longer training time, higher inference latency, and greater peak memory usage than the compared methods. This computational overhead is associated with SAM3-based region proposal generation, region-level CLIP feature extraction, and subsequent relational reasoning during both training and inference. These components enable spatial relations among regions to contribute to the refinement of dense visual representations, while also increasing the computational cost of the complete inference pipeline. We will discuss this computational overhead in the limitations section.

🔹 Response to Q3

We thank the reviewer for pointing out the need to further clarify the spatial relation determination and the computation of their geometric attributes.

  • Spatial relation determination

For two candidate regions MiM_i and MjM_j, we calculate their intersection area, bidirectional coverage ratios, and intersection over union (IoU) based on their binary masks to determine their topological relations.

Specifically, the bidirectional coverage ratios are defined as pi∣j=∣Mi∩Mj∣∣Mi∣p_{i|j} = \frac{|M_i \cap M_j|}{|M_i|} and pj∣i=∣Mi∩Mj∣∣Mj∣p_{j|i} = \frac{|M_i \cap M_j|}{|M_j|}. When at least 80% of one region is covered by the other, while the reverse coverage ratio is below 80%, an approximate containment relation is established, with the corresponding directed inside/contains edges. If neither of the above unidirectional containment conditions is satisfied and the IoU is at least 0.2, the region pair is classified as overlap. For region pairs that satisfy neither condition, we further calculate the minimum euclidean distance between their boundaries and normalize it by the image diagonal, H2+W2\sqrt{H^2+W^2}. An adjacent relation is established only when the two regions do not intersect and their normalized boundary distance does not exceed 0.01. These topological relations are determined in the order of containment, overlap, and adjacency.

Directional relations are determined independently based on the relative positions of region centroids, including left, right, above, and below, and can coexist with the aforementioned topological relations.

  • Relation confidence computation

We recognize that the term relation confidence used in the original manuscript may be misinterpreted as the predicted probability of a relation type. In fact, this variable represents a geometric attribute calculated based on the established spatial relation and does not have a probabilistic interpretation. Therefore, we will rename it geometric attribute score in the revised manuscript and replace the notation cijc_{ij} with cijrc_{ij}^{r}.

Specifically, the geometric attribute for inside/contains is the coverage ratio of the contained region, that for overlap is the region IoU, and that for adjacent is the exponential decay of the normalized boundary distance, e−dˉij0.01e^{-\frac{\bar{d}_{ij}}{0.01}}. For directional relations, the relation type is determined by the sign of the relative centroid displacement, while the corresponding geometric attribute is calculated using the absolute horizontal or vertical centroid displacement, normalized by the image width or height, respectively.

🔹 Response to Q4

We thank the reviewer for suggesting an expanded evaluation under the RSKT-Seg and Pi-Seg protocols. We agree that training on DLRSD or iSAID and testing on other datasets under the corresponding protocols would allow a more comprehensive assessment of how the training data affect model performance. These protocols require multiple independent training runs and validation. Given the time available during this rebuttal period, we have not yet completed and verified the full set of experiments.

To address the reviewer’s suggestion as far as possible, we have evaluated the SCENE model trained on LandDiscover50K on five additional datasets: VDD, UDD5, UAVid, Vaihingen, and DLRSD. The results are shown in the table below. We hope these results broaden the existing evaluation and provide further evidence for assessing SCENE’s cross-dataset generalization.

MethodVDDUDDsUAVidDLRSDVaihingenm-mIoU
CLIP-ViT-B as the backbone
CAT-Seg30.8115.083.4521.6711.5716.52
EBSeg26.8122.726.5315.625.9715.53
TPOVSeg24.5411.934.2422.2713.5515.31
GSNet35.7731.252.2715.171.8617.26
OVRS31.4525.346.1323.0815.3220.26
RSKT-Seg27.6722.503.6818.033.7515.13
PCA-Seg29.3028.823.3319.4016.8019.53
DINOde17.1723.1415.8013.4714.8716.89
Ours28.1720.927.4225.7627.0021.85
CLIP-ViT-L as the backbone
CAT-Seg35.6034.4715.8523.0718.8325.56
EBSeg23.9812.226.6312.682.0511.51
TPOVSeg34.4918.195.6423.209.6618.24
GSNet32.4130.978.5724.547.9720.89
OVRS34.8732.438.0624.5016.8223.34
RSKT-Seg35.7733.1916.4224.3011.9024.32
PCA-Seg27.5220.139.5020.9719.6719.56
DINOde20.3426.5518.4321.6920.1721.44
Ours34.4727.3317.3727.7528.4527.07

Across the five additional test datasets, SCENE achieves an m-mIoU of 21.85% with ViT-B and 27.07% with ViT-L, exceeding the strongest baseline for each backbone by 1.59 and 1.51 percentage points, respectively. Although SCENE does not achieve the best result on every dataset, it attains the highest m-mIoU across the five datasets with both backbones, indicating strong overall segmentation performance on the additional datasets.


We again thank Reviewer RaGM for their constructive comments. These comments have helped us substantially improve the clarity and rigor of the paper. We hope our responses will help the reviewer evaluate our work more fully.

Reviewer_e8A3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCENE for open-vocabulary remote sensing segmentation, combining multiscale visual information with spatial relationships between regions to reduce recognition ambiguity. CASP performs content-adaptive multiscale aggregation of RemoteCLIP features. SRSL organizes category-agnostic regions generated by SAM3 into a graph with topological and directional relations, then maps the relation-enhanced region representations back to pixel space. SGFD uses relational context to refine visual features and incorporates text embeddings for decoding. The model is trained on LandDiscover50K and evaluated on four remote sensing datasets. The paper also includes ablations of components, relation types, scale weights, and fusion strategies. The reported results achieve the highest average mIoU among the compared methods across the four datasets with both CLIP backbone configurations.

Critical strengths
  1. The paper addresses a meaningful problem. Open-vocabulary remote sensing segmentation faces substantial scale variation, similar local appearances, and confusion between categories. Investigating how spatial information between regions can assist recognition is worthwhile. The paper has a clear motivation and potential practical value.

  2. The reported experiments show performance gains. The authors evaluate the method on four remote sensing datasets with two CLIP backbones. In Table 1, the average mIoU reaches 28.27 with ViT-B and 26.96 with ViT-L, exceeding the strongest reported baseline averages by 1.47 and 1.95 percentage points, respectively. The main paper and appendix also provide ablations of relation types, scale weights, and fusion strategies, supporting the effectiveness of the overall system and several of its design choices.

Critical weaknesses

1. CASP and SGFD use fairly conventional designs, with limited novelty as methodological contributions. CASP primarily applies dynamic weighting to average-pooled features from windows of different sizes, followed by attention and residual updates. SGFD combines relational context and visual features through convolutions, gating, and residual connections. These are established operations, and their combination here is relatively straightforward. They are reasonable choices for handling scale variation and contextual information in remote sensing, but I find the methodological novelty of these two modules limited. Tables 8 and 10 demonstrate the benefits of adaptive weights, the two scale branches, and the chosen refinement direction, which I acknowledge. However, these comparisons are largely internal to the proposed framework. Direct comparisons with representative existing multiscale aggregation and feature fusion approaches are missing. The results therefore establish that the designs are useful, but do not sufficiently establish their added contribution over existing alternatives. I view these modules mainly as task-specific engineering improvements.

2. The experiments establish component effectiveness more clearly than they validate the paper's central claims. The authors motivate adaptive scale aggregation and region relation modeling through variation in scale and shape, similar local appearances, and confusion between adjacent regions. However, Tables 7 and 8 mainly compare overall mIoU across configurations. These results do not sufficiently establish whether the gains specifically address the stated difficulties or whether more general contextual enhancement could yield similar improvements. The examples in Figures 1 and 3 and the scale-weight visualization in Figure 4 provide some support, but targeted analyses by object scale, scene complexity, or easily confused categories are still missing. Such analyses would connect the motivation, design choices, and observed gains more directly. In addition, I suggest considering evaluation on another benchmark, such as OVRSISBenchV2 introduced in the recent Pi-Seg work. In my experience using the GSNet evaluation protocol, the results exhibit substantial variability. Additional results under another evaluation protocol would make the experimental conclusions more convincing.

Questions for authors
  1. For CASP and SGFD, what are the key technical differences from existing adaptive multiscale aggregation and gated fusion approaches? Could the authors compare against representative replacement modules with similar parameter budgets to establish the added value of the proposed designs, beyond comparisons that remove branches, fix weights, or change the refinement direction?

  2. Could the authors provide analyses that more directly test the stated motivation? For example, an analysis of CASP's gains by object scale and a comparison of category confusions before and after adding SRSL for adjacent regions with similar local appearances would be useful. Table 7 already includes shared-relation and other controls. Showing which errors full SRSL corrects relative to these controls would help establish whether explicit spatial relations address the specific difficulties emphasized in the paper.

3 comments

We thank the reviewer for the careful review and thoughtful suggestions. These comments help us better elaborate our contributions and enrich our experimental discussions. We hope the responses below address your concerns.


🔹 Response to W1 & Q1

We thank the reviewer for the constructive comments on the designs and methodological contributions of CASP and SGFD. We clarify their technical differences from existing approaches and introduce representative module replacement experiments to further validate their effectiveness.

(1) Differences from existing adaptive multi-scale aggregation methods

Existing methods: Representative adaptive multi-scale methods typically extract features using different receptive fields or convolutional kernels. They then use branch selection, attention, or adaptive weighting to regulate the contributions of different scales.

CASP: CASP directly predicts weights for candidate aggregation windows at each spatial location, producing two complementary spatially aware representations under the fine and coarse granularity modes. Furthermore, it independently models inter-position associations for the two granularities. Specifically, the Query and Value are generated from the current-granularity features, while the Key is jointly constructed from the current-granularity features and the underlying remote-sensing visual features. This design guides the selective aggregation of information from relevant spatial locations.

(2) Differences from existing gated feature fusion methods

Existing methods: Typically learn affinity-based weights between input features to regulate their contributions to the fused representation, enabling adaptive feature interaction and fusion.

SGFD: SGFD uses relational context derived from region-graph reasoning as a shared structural guidance signal to separately refine the CLIP general visual representation and the CASP spatially aware representation. Specifically, for CLIP features, a gating mechanism jointly conditioned on visual features and relational context selectively incorporates structural information. In contrast, the CASP branch uses relational context to further refine the spatially aware representation. Therefore, the same relational context serves as a shared structural condition for differentiated updates of the CLIP and CASP representations.

(3) Representative module replacement experiments

To evaluate our designs against existing alternatives, we introduce four module replacement experiments. For CASP, we replace it with Selective Kernel (SK) [1] and the MSCA module in SegNeXt [2], representing adaptive receptive-field selection and multi-scale convolutional attention, respectively. For SGFD, we use Attentional Feature Fusion (AFF) [3] and GA from BiSeNet V2 [4], representing attention-based feature fusion and guided cross-branch aggregation, respectively. All other model components and training settings remain unchanged.

As shown in the table, CASP+SGFD achieves the highest m-mIoU, outperforming the MSCA, SK, AFF, and GA replacement variants by 1.03, 0.88, 1.76, and 1.09 percentage points, respectively. Although some replacement modules achieve higher mIoU on individual datasets, CASP+SGFD delivers the best overall m-mIoU. Meanwhile, all variants have comparable parameter counts and GFLOPs, demonstrating the effectiveness of CASP+SGFD over these representative alternatives under similar parameter and theoretical computation budgets.

MethodsFLAIRFASTPotsdamFloodNetm-mIoU (%\%)Params (M)DFLOPs
MSCA+ SGFD18.0617.5139.8733.5127.24310.67463.89
SK+ SGFD19.1617.6637.9434.8027.39310.67463.89
CASP +AFF18.0716.1138.1933.6726.51310.50463.76
CASP + GA17.9816.6939.1434.9027.18310.95464.03
SGFD+ CASP18.5217.6841.9734.9128.27310.80463.95

References

[1] Li X, Wang W, Hu X, et al. Selective kernel networks[C]//2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2019: 510-519.

[2] Guo M H, Lu C Z, Hou Q, et al. Segnext: Rethinking convolutional attention design for semantic segmentation[J]. Advances in neural information processing systems, 2022, 35: 1140-1156.

[3] Dai Y, Gieseke F, Oehmcke S, et al. Attentional feature fusion[C]//2021 IEEE winter conference on applications of computer vision (WACV). IEEE, 2021: 3559-3568.

[4] Yu C, Gao C, Wang J, et al. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation[J]. International journal of computer vision, 2021, 129(11): 3051-3068.

Thank you for the detailed response. The additional replacement experiments help address some of my concerns. However, my main concern regarding methodological novelty remains. The individual designs of CASP and SGFD largely rely on existing techniques. While I understand that the authors consider their combination a novel approach, the motivation behind some specific design choices is still not sufficiently clear.

For example, CASP adopts an asymmetric Q/K/V construction, where Q and V are derived from FmF_m, while K incorporates both FmF_m and FbF_b. Since the authors emphasize this as a key distinction from existing methods, I believe a more convincing validation would be to compare different Q/K/V configurations using FmF_m and FbF_b, or at least include a targeted ablation to demonstrate why the proposed asymmetric design is preferable to conventional alternatives.

Additionally, while the reported parameter counts and GFLOPs are comparable across the replacement variants, this does not necessarily imply comparable inference latency. The additional region proposal generation and relational reasoning may introduce non-negligible runtime overhead. At this stage, I am inclined to keep the rating unchanged.

We greatly appreciate your time in reading our response and providing further feedback, which is encouraging for us. We understand your decision to keep the current rating and sincerely thank you for your careful evaluation. Your comments help us re-examine our manuscript and clarify directions for future revisions. Thank you again for your time, patience and help.

Reviewer_3W9FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCENE for open-vocabulary remote sensing segmentation. It combines content-adaptive spatial aggregation (CASP), spatial relation modeling over category-agnostic region proposals (SRSL), and a structure-guided fusion decoder (SGFD). The authors claims that scene context can help distinguish regions with similar local appearances. Experiments conducted on FLAIR, FAST, Potsdam, and FloodNet show improved average segmentation performance with two CLIP backbones, supported by component ablations, seen/unseen class evaluation, and qualitative comparisons.

Critical strengths
  1. The method addresses a well-motivated problem in remote sensing: local appearance alone can be ambiguous, while spatial relationships provide complementary evidence.
  2. The empirical evaluation covers four datasets and two backbone sizes.
Critical weaknesses
  1. The central methodological contribution needs a sharper definition. CASP combines adaptive multiscale aggregation with attention, SRSL performs relation-aware message passing, and SGFD applies feature refinement and progressive decoding. The implementation is described in detail, but the paper is less explicit about which modeling principle constitutes the central contribution and how the three components jointly realize that principle.
  2. The practical cost of the complete pipeline is not yet clear. SCENE uses SAM3, RemoteCLIP, and CLIP features extracted from region crops. Figures 5 and 6 provide useful SRSL parameter counts and training-time measurements, but do not establish end-to-end inference cost relative to the baselines. A compact comparison of latency and memory under the same input resolution would help assess whether the accuracy gains justify the added computation.
  3. Directional relations introduce an unresolved dependence on image orientation. The graph distinguishes left, right, above, and below using image coordinates. These relations change when an image is rotated, even though the underlying scene semantics can remain unchanged. Given the paper’s emphasis on orientation variation in remote sensing, the assumptions behind this coordinate-dependent representation are insufficiently discussed.
  4. The connection between proposal topology and semantic scene dependencies is underdeveloped. SRSL treats all retained masks as scene entities, but their intended semantic granularity is not defined. Containment, overlap, and adjacency between proposals describe the organization of those masks; their connection to the semantic relationships needed for class disambiguation deserves a clearer explanation.
Questions for authors
  1. What orientation assumptions justify the directional relations, and how should the relational representations and segmentation predictions behave when the same scene is viewed under a different image rotation?
  2. Why does CASP adapt the mixture of window sizes while retaining fixed square support, and which part of the module specifically handles irregular boundaries or prevents neighboring regions from being mixed during aggregation?
  3. How are individual class embeddings combined with dense visual features, and how does the decoder produce predictions for an arbitrary number of candidate classes? Are the decoding parameters shared across classes?
2 comments

We sincerely appreciate the reviewer’s careful and expert review. These suggestions help us better clarify the core modeling principle and refine the technical design of our SCENE method. We will update the manuscript accordingly to incorporate these improvements. We hope the responses below can address your concerns.

🔹 Response to W1

We thank the reviewer for the valuable comment. SCENE’s core modeling principle is to leverage scene dependencies to enhance pixel-level category discrimination. In remote sensing scenes, instances of the same semantic class may vary in scale and shape, while different classes may have similar local appearances. Relying solely on local visual information can therefore lead to incomplete class responses and confusion between similar classes. Guided by this principle, CASP captures context suited to different spatial positions, SRSL extracts explicit spatial relationships between regions, and SGFD incorporates the inferred relationships into pixel-level visual representations, allowing both forms of scene dependency to contribute to category discrimination.

Specifically, CASP dynamically adjusts the weights of candidate spatial windows according to visual content and selectively aggregates information from related positions to accommodate variations in object scale and shape and improve the completeness of class responses. SRSL constructs a relation graph from class-agnostic candidate regions and reasons over their topological and directional relationships, providing structural information to help distinguish classes with similar local appearances. SGFD uses region-level relational context to guide the updates of both CLIP dense visual representations and CASP spatial representations, enabling regional organization to contribute to pixel-level visual–text matching before producing the segmentation result through progressive decoding.


🔹 Response to W2

We appreciate the reviewer’s valuable suggestion. We supplement the end-to-end inference efficiency comparison under a unified input resolution in the table below.

ModelTrainable param (M)GFLOPsTrain time (s/it)Inference time (ms/image)Inference Memory (GB)
CAT-Seg25.12233.200.3751.080.94
EBSeg26.62475.070.3970.441.61
TPOVSeg25.93275.130.4865.051.59
GSNet29.01459.850.54245.991.63
OVRS25.14268.190.4888.350.96
RSKT-Seg32.64655.100.72164.952.26
PCA-Seg27.87195.670.7390.360.96
DINOde17.61486.490.56179.852.01
Ours30.36463.952.54616.244.84

The results show that SCENE achieves better average segmentation performance but incurs higher inference latency and peak GPU memory usage than the compared methods. This additional overhead is associated with SAM3-based region proposal generation, region-level CLIP feature extraction, and relational reasoning. We will discuss this computational overhead in the limitations section.


🔹 Response to W4

We appreciate the reviewer’s valuable comment. We clarify the granularity of candidate regions and how their spatial relationships contribute to class discrimination as follows.

  • Granularity of candidate regions. SRSL uses the retained SAM3 masks as class-agnostic modeling units. Their granularity depends on the spatial extent of each mask, without assuming that every region corresponds to a complete object or a fixed semantic level.

  • Connection between spatial relationships and class discrimination. Containment, overlap, adjacency, and directional relationships describe the spatial arrangement of regions; they do not inherently represent predefined semantic relationships. SRSL uses these relationships to guide the exchange of regional visual features and learns during training to update region representations using visual information from spatially related regions. The updated representations are then mapped to pixel space and incorporated into visual–text class matching to help distinguish classes with similar local appearances.

🔹 Response to W3 & Q1

We appreciate this valuable suggestion, which helps us clarify the assumptions of directional relationship modeling.

SRSL does not assume a fixed geographic orientation for remote sensing images. Directional relationships are defined in the current image coordinate system to capture relative positions between regions that topological relationships alone cannot express. They do not assume an association between a particular semantic class and a fixed direction. The ablation results in Table 7 of the original manuscript show that adding directional relationships to topological relationships increases m-mIoU from 25.16% to 28.27%, supporting the complementary value of directional information.

When an image is rotated, SRSL reconstructs the spatial relation graph from the rotated region positions. The directional relationships change accordingly and are used in relational reasoning. Therefore, relation representations may vary with image orientation while still providing spatial context for class discrimination at the corresponding positions. This coordinate-dependent encoding does not guarantee strict rotation equivariance.

To assess the effect of this coordinate dependence on segmentation performance, we evaluated the same model weights on four benchmark datasets at rotations of 0°, 90°, 180°, and 270°. The corresponding average mIoU values were 28.27%, 28.72%, 28.34%, and 28.45%, with a maximum difference of 0.45 percentage points. Although performance varies across datasets and orientations, the average performance across the four datasets remains broadly stable, with no consistent decline as the rotation angle increases. These results indicate that SCENE maintains relatively stable overall segmentation performance under the tested rotations.

RotationFLAIRFASTPotadamFloodNetm-mIoU (%)
0°18.5217.6841.9734.9128.27
90°18.9617.1242.0036.7828.72
180°19.3717.6141.8934.4728.34
270°18.5617.1341.2336.8728.45

🔹 Response to Q2

We thank the reviewer for this valuable question, which allows us to clarify our design rationale.

  • CASP combines fixed square windows with adaptive scale weighting to maintain efficient spatial aggregation while accommodating the varying contextual needs of different locations. For example, recognizing road boundaries requires attention to changes in nearby regions, whereas understanding road extension and continuity requires broader spatial context. Therefore, CASP uses fixed square windows to extract features over different spatial ranges and dynamically adjusts their contributions based on visual content, enabling content-adaptive spatial perception.

  • CASP enhances local visual representations through fine-grained perception and content-guided spatial interaction, but does not explicitly perform boundary alignment or cross-region feature isolation during window aggregation. The improved discrimination between adjacent classes benefits from SCENE's integration of content-adaptive spatial perception with dense relational context generated through inter-region relational reasoning, followed by structure-guided fusion and refinement of visual representations.


🔹 Response to Q3

We greatly appreciate the reviewer for raising these questions.

  • Combination of class embeddings and dense visual features: Given KK candidate classes, we first compute the pixel-wise cosine similarity between each class embedding tkt_k and the dense CLIP feature CclipC_{clip} to obtain the class relevance map SkS_k. Each SkS_k is encoded using a shared projection layer and then fused with the structure-guided refined dense CLIP visual features (Eq. 11) and spatial perception features (Eq. 12) to form class-conditioned features.

  • Prediction for an arbitrary number of candidate classes: For KK candidate classes, we construct KK class-conditioned features XkX_k. We merge the class dimension into the batch dimension and feed them into the decoder in parallel. The decoder outputs are reshaped back into a logit tensor of size B×K×H×WB \times K \times H \times W. Therefore, changing the number of candidate classes only alters the class dimension KK, without requiring modifications to the network architecture.

  • Parameter sharing across classes: The relevance-map projection layer, feature fusion layers, two-stage upsampling decoder, and prediction head are shared across all classes. Class distinctions are solely introduced by the corresponding text embedding tkt_k and class relevance map SkS_k.


We hope these clarifications address the reviewer's concerns and highlight our contributions. We appreciate the feedback and hope our responses will be considered in the re-evaluation of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.