acceptodds
Withdrawn from ICLR 2027

CompatLift: Anchored Feature Lifting for Remote Sensing Open-Vocabulary Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Remote-Sensing Open-Vocabulary Semantic Segmentation (RS-OVSS) requires fine spatial detail to delineate small objects and narrow structures, while extracting high-resolution features with the backbone incurs substantial computation. Feature upsampling provides a more efficient alternative, but existing learned feature upsamplers are mainly developed for pretrained visual encoders and may produce representations that are incompatible with a frozen encoder–decoder segmentation pipeline. We study this problem in a SAM 3-based RS-OVSS pipeline and observe that directly replacing native features with learned upsamplers substantially degrades segmentation performance, whereas bicubic interpolation largely preserves the behavior of the frozen decoder. Motivated by this, we propose CompatLift, a lightweight feature upsampler framework that learns image-guided residual updates anchored by the bicubic reference. The upsampler module is trained without segmentation annotations using high-resolution features from the same frozen SAM 3 encoder, while the backbone and prediction heads remain frozen. We further introduce a simple score calibration step to compensate for the confidence shift caused by feature lifting. Extensive experiments on representative eight remote-sensing benchmarks show that CompatLift improves SAM 3 segmentation performance by lifted higher-resolution, decoder-compatible feature representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote sensing, Open-vocabulary semantic segmentation, Feature upsampling, SAM 3

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m76ROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CompatLift, a lightweight feature upsampling framework for remote-sensing open-vocabulary semantic segmentation. It addresses feature compatibility issues in frozen prediction pipelines by combining image-guided residual updates with a bicubic feature anchor. The method uses high-resolution features from the same frozen encoder for supervision and applies score calibration to compensate for confidence shifts. Experiments on eight remote-sensing benchmarks show that CompatLift improves the average mIoU from 53.4 to 54.2 over the original SegEarth-OV3 baseline.

Critical strengths

The method is lightweight and supported by extensive evaluation. CompatLift requires only 0.111M trainable parameters and no segmentation annotations. Experiments across eight remote-sensing benchmarks demonstrate consistent overall improvements, while the ablation studies provide evidence for the contributions of residual mixing and score calibration.

Critical weaknesses
  1. The paper does not sufficiently demonstrate the extent to which the identified problem generalizes beyond the specific architecture considered. Although investigating feature upsampling with frozen predictors is a reasonable setting, it remains unclear whether the observed performance degradation reflects a general compatibility issue between feature upsampling and frozen predictors or primarily arises from representation mismatch at the specific feature interfaces of SegEarth-OV3.

  2. The core idea of CompatLift is to use bicubic features as an anchor and incorporate only a fraction of the learned update into this reference representation. More importantly, Table 2 shows that applying the same residual mixing strategy to existing upsampling methods can also substantially recover their performance. For example, the average mIoU of AnyUp increases from 28.2 to 52.5, while that of JAFAR increases from 27.0 to 52.5. These results provide strong support for the paper's feature compatibility hypothesis, but they also raise an important question: does the main contribution of this work lie in a new feature lifting method, or primarily in a generic and conservative feature mixing mechanism? The paper does not yet clearly distinguish the respective contributions of the compatibility phenomenon itself, generic residual mixing, and the learned module in CompatLift.

  3. The proposed method achieves an average improvement of 0.8 mIoU over the original SegEarth-OV3 baseline (53.4 to 54.2). More importantly, compared with the bicubic baseline equipped with the same score calibration, the full CompatLift improves performance only from 53.7 to 54.2, corresponding to a gain of 0.5 mIoU. Furthermore, Table 3 shows that these improvements are not consistent across datasets: performance slightly decreases on Potsdam, while no improvement is observed on Vaihingen or UDD5. The paper does not report results from multiple random seeds, standard deviations, or confidence intervals. Since CompatLift itself requires optimization-based training, reporting only a single run is insufficient to determine whether such modest improvements consistently exceed the variability introduced by training randomness.

  4. There is an inconsistency in the way conference names are written in the references.

Questions for authors
  1. What evidence supports the generalizability of the identified feature compatibility problem beyond the specific SegEarth-OV3 architecture?

  2. Given that CompatLift improves average mIoU by only 0.5 points over the calibrated bicubic baseline, can the authors provide results from multiple random seeds or other evidence demonstrating that this improvement is robust to training randomness?

Reviewer_gFvQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompatLift, a feature upsampling module for remote-sensing open-vocabulary segmentation with frozen SegEarth-OV3. It mixes image-guided upsampled features with bicubic features, trains the module using higher-resolution features from the same encoder, and applies score calibration to adjust background rejection.

Critical strengths

1 Clear writing and informative figures. The paper is well organized, and the method is easy to follow. The framework figures clearly illustrate the training and inference procedures and help explain how the bicubic reference, learned upsampling branch, and score calibration fit together.

2 Lightweight module and broad evaluation. The module is lightweight and requires no segmentation annotations. It improves over the native model on seven of eight datasets. The authors also evaluate residual mixing with competing upsamplers, providing a useful comparison of the effect of retaining a reference representation.

Critical weaknesses

1 Limited methodological novelty beyond combining existing techniques. The main components closely follow existing work. Image-guided JBU upsampling follows FeatUp, supervision using higher-resolution features from the same frozen encoder has already been explored in LiFT, and feature preservation under frozen predictors has also been studied. The main additions are a weighted mixture of bicubic and learned features and a constant score offset. As presented, the method appears primarily to adapt and combine existing techniques within SegEarth-OV3. The paper does not clearly establish an independent methodological contribution beyond this combination. The authors should explain what new design addresses a difficulty that existing methods cannot readily handle, and why this constitutes a substantive advance.

2 Modest performance gains limit the demonstrated contribution. The method is mainly evaluated on SegEarth-OV3, where the complete model improves average mIoU by only 0.8 points over native inference and by approximately 0.5 points over calibrated bicubic upsampling. These results show some improvement, but do not demonstrate a substantial performance advantage. Given the incremental nature of the design, the gains do not yet make a convincing case for the method's contribution. SegEarth-OV3 is already a strong baseline, which may leave limited room for improvement. Adapting the approach to other baselines, such as SegEarth-OV, could help establish whether these modules can deliver more pronounced gains and better demonstrate their practical value.

Questions for authors
  1. What is the key methodological advance over adapting existing feature upsamplers with bicubic residual mixing and score calibration? In particular, how does CompatLift compare with FeatUp/JBU or LiFT when they use the same supervision, mixing, and calibration protocol?

  2. Have you considered adapting the approach to SegEarth-OV or other segmentation baselines? Results on another baseline could clarify whether the modest gains reflect limited headroom in SegEarth-OV3 or limited benefit from the proposed modules.

Reviewer_bryPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates feature upsampling for remote-sensing open-vocabulary segmentation within the frozen SAM 3-based SegEarth-OV3 pipeline. It observes that directly substituting features from existing learned upsamplers can substantially degrade segmentation performance. The proposed CompatLift combines an image-guided feature proposal with a bicubic anchor through residual mixing. It is trained on unlabeled imagery using higher-resolution features from the same frozen encoder, with an additional confidence correction before background rejection. Experiments on eight benchmarks report an average mIoU of 54.2, compared with 53.4 for native SegEarth-OV3 and 53.7 for calibrated bicubic upsampling.

Critical strengths

1.A practically relevant problem supported by informative controls. Table 2 demonstrates that increasing feature resolution through learned upsampling does not necessarily improve a frozen downstream predictor. The native, bicubic, and residual-mixing comparisons make this observation concrete and provide useful guidance for integrating feature upsamplers into existing segmentation systems.

2.A compact solution with validation across multiple datasets. CompatLift introduces only 0.111M trainable parameters and requires no segmentation annotations or updates to the pretrained prediction components. It improves over the native pipeline on seven of eight benchmarks, including gains of 2.4 and 2.0 mIoU points on OpenEarthMap and iSAID, respectively. These results support its practical utility within the studied setting.

Critical weaknesses

1.The analysis of the mechanism behind the observed phenomenon is shallow. The paper documents that learned upsamplers degrade the frozen pipeline and that lifting shifts decoder confidence downward, but it does not explain why. Bicubic resizing with score calibration (53.7) already exceeds both the native model (53.4) and CompatLift without calibration (53.6). So a substantial part of the observed “incompatibility” is a confidence shift that also occurs with parameter-free interpolation, which points to the enlarged feature grid itself rather than the learned features.

2.Methodological novelty is limited, and the work reads more like a technical report. The methodological contribution is primarily an incremental combination of established components: residual learning on top of an interpolated reference (standard since VDSR-style super-resolution), the FeatUp JBU operator, LiFT-style same-encoder multi-resolution supervision, and a global scalar offset on the rejection threshold. Their combination does not yield a new principle.

Questions for authors

1.In Table 2, is score calibration applied to AnyUp, JAFAR, and ViT-Up, with and without residual mixing? Calibration alone raises bicubic from 52.8 to 53.7, so reporting all baselines with a per-method δ would make the comparison more informative. 2. Table 4 reports only parameter counts. Could you report end-to-end latency, FLOPs, and memory for native, bicubic, and CompatLift, together with the encoder input resolution used for each? 3. The motivation emphasizes small objects and narrow structures. Could you report boundary-oriented metrics or per-class IoU for small classes, or results on the building and road extraction benchmarks evaluated in SegEarth-OV3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.