acceptodds
Withdrawn from ICLR 2027

SFC-SAM3: Specializing Foundation Priors via Cross-Modal Coordination for Referring Image Segmentation in Agricultural Remote Sensing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Referring image segmentation (RIS) grounds free-form language in pixel-level masks, but extending it to agricultural remote sensing exposes a substantial mismatch between generic foundation-model priors and domain-specific visual and linguistic structure. Agricultural scenes contain large and weakly bounded regions, while referring expressions combine phenology, morphology, and spatial relations. To address these challenges, we introduce SFC-SAM3, a parameter-efficient foundation-prior framework for agricultural referring remote sensing image segmentation (RRSIS) that specializes complementary structural, visual, and linguistic priors. Rather than retraining foundation backbones, SFC-SAM3 coordinates SAM3, DINOv3, and CLIP through a frequency-aware PromptGenerator, a multi-layer DINOv3 FPN bridge, a CLIP token-level text bridge, and a residual mask refinement head, together with LoRA residuals on the SAM3 encoder. Training follows two stages: joint fine-tuning of the adapters and SAM3 mask decoder, followed by parameter-efficient continued training with the decoder frozen. Across the two-stage training procedure, SFC-SAM3 updates only 4.3M parameters (0.35% of the 1.21B model). On FarmSeg-VL, SFC-SAM3 reaches 87.04% mIoU, compared with 86.11% for RMSIN trained on the same data. The advantage becomes clearer off the training distribution: without training on the target datasets, SFC-SAM3 reaches 41.14% mIoU on the RRSIS-D land subset and 19.81% on the RefSegRS land/ground subset, compared with 5.65% and 13.99%, respectively, for the strongest evaluated baselines. These results point to a practical lesson for agricultural RRSIS: task-structured specialization of complementary foundation priors can preserve strong in-domain performance while substantially improving transfer to unseen land-oriented remote-sensing scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
referring image segmentation, remote sensing, foundation models, cross-modal learning, vision-language learning, parameter-efficient fine-tuning, agricultural remote sensing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ak1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SFC-SAM3, a parameter-efficient framework for agricultural referring remote sensing image segmentation (RRSIS). The core motivation is to bridge the domain gap between generic foundation priors and agricultural scenes. To achieve this, the authors coordinate three frozen foundation models—SAM3, DINOv3, and CLIP—through an elaborate combination of a frequency-aware PromptGenerator (PromptGen-FFT), a multi-layer DINO-FPN bridge, a token-level text bridge, a residual mask refinement head, and LoRA layers on the SAM3 image encoder. While the method achieves competitive performance on FarmSeg-VL and cross-domain subsets, the framework suffers from severe over-engineering and lacks conceptual elegance.

Critical strengths
  1. Strong Empirical Performance in a Specialized Domain: The method demonstrates solid quantitative results on the in-domain agricultural benchmark (FarmSeg-VL) and exhibits decent zero-shot transfer capabilities on land-related cross-domain subsets.
  2. Parameter-Efficient Design: The two-stage training scheme combined with lightweight adapters and LoRA keeps the trainable parameter ratio low (0.35% to 0.54%), which is practically favorable.
Critical weaknesses
  1. The framework suffers from heavy over-engineering. Instead of a unified and parsimonious design principle, the method appears as an ad-hoc combination of multiple existing components (e.g., SAM3, DINOv3, CLIP, an FFT high-pass branch, an FPN bridge, a text token bridge, a residual refinement head, and LoRA). This component-stacking approach lacks structural elegance and raises concerns about its conceptual originality.
  2. The paper's motivations appear tailored to justify pre-selected modules (e.g., adding a residual head for fuzzy boundaries, an FFT branch for periodic textures, and a text bridge for long captions). Furthermore, the paper lacks a deep mechanistic analysis of how features from these multiple foundation models interact within the shared representation space.
  3. The progressive ablation study (Table 2) shows an incremental performance increase from 1.01% to 87.04% mIoU through continuous module stacking. Rather than solely demonstrating component complementarity, this pattern highlights the underlying fragility of the base model, which relies on adding numerous auxiliary components and feature bridges to drive up performance. This violates the principle of parsimony and leaves the individual necessity of each component ambiguous within an over-parameterized pipeline.
Questions for authors

Given that the evaluation relies strictly on a cumulative progressive ablation (Table 2), have you conducted leave-one-out experiments (e.g., independently removing the FFT branch, the DINO-FPN bridge, or the text token bridge)? Clarifying the isolated marginal utility of each component is essential to determine whether the performance gains stem from a cohesive structural design or simply from over-parameterized feature stacking.

Reviewer_iGqSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SFC-SAM3, a parameter-efficient framework for agricultural referring remote sensing image segmentation. It combines SAM3, DINOv3, and CLIP through a frequency-aware prompt generator, multi-layer visual feature fusion, and other components. A two-stage training strategy is employed to adapt the foundation models. Experiments on FarmSeg-VL, RRSIS-D, and RefSegRS demonstrate competitive in-domain accuracy and improved cross-domain performance.

Critical strengths
  1. Encouraging cross-domain results. The proposed method achieves substantially better mIoU than the evaluated baselines on the RRSIS-D and RefSegRS land-related subsets.

  2. Practical integration of foundation models. The framework integrates pretrained visual, linguistic, and structural representations through trainable adapters, achieving competitive in-domain performance with a relatively small number of trainable parameters.

Critical weaknesses
  1. The proposed method mainly combines existing techniques, including FFT-based features, DINOv3 feature fusion, and CNN-based mask refinement, without introducing a sufficiently distinctive methodological contribution.

  2. Insufficient component ablations. Table 2 only presents cumulative ablations, where multiple modules and training tricks (augmentation, EMA, TTA, and loss functions) are introduced simultaneously, making it difficult to identify the actual contribution of each component.

  3. Insufficient comparisons with relevant methods. Important related methods such as RS2-SAM2 [1], RSRefSeg 2 [2], and UGFF-VLM [3] are not adequately compared or discussed, and a simple SAM3 + LoRA baseline under the same training settings is also missing.

  4. Unclear contribution of language understanding. Since FarmSeg-VL primarily focuses on farmland segmentation, it remains unclear how much the model actually relies on referring expressions, which could be examined using generic prompts, shuffled descriptions, or image-only inputs.

  5. Questionable cross-domain evaluation results. The extremely low performance of LAVT (1.16% mIoU) and RMSIN (2.20%) on the selected RRSIS-D subset requires further explanation, particularly regarding domain shifts and evaluation settings.

  6. Incomplete benchmark evaluation. The paper neither specifies the exact categories and sample counts of the selected "land-related" subsets nor evaluates the proposed method under the standard RRSIS-D and RefSegRS training and testing protocols.

  7. Unclear computational efficiency. Despite using three large foundation backbones and four-way TTA, the paper only reports trainable parameter counts without providing inference FLOPs, latency, or GPU memory consumption.

  8. Missing implementation details. The CAPA presence head appears in Eq. (12) and Table 2 without a clear architectural definition or explanation of its role, raising concerns about reproducibility.

  9. The mathematical notations, equations, and definitions throughout the manuscript clearly lack careful consideration and polishing, and are cluttered with verbose variable names copied directly from the raw source code.

[1] Rong F, Lan M, Zhang Q, et al. RS2-SAM2: Customized SAM2 for referring remote sensing image segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(11): 8751-8759.

[2] Chen K, Liu C, Chen B, et al. Rsrefseg 2: Decoupling referring remote sensing image segmentation with foundation models[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 64: 1-20.

[3] Tan K, Wu Y, Yang H, et al. UGFF-VLM: Uncertainty-Guided and Frequency-Fused Vision-Language Model for Remote Sensing Farmland Segmentation[J]. Remote Sensing, 2026, 18(2): 282.

Questions for authors

Please see the weaknesses.

Reviewer_4pnoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SFC‑SAM3, a parameter‑efficient framework for agricultural remote‑sensing referring segmentation. It leverages lightweight adapters to specialize frozen SAM3, DINOv3 and CLIP under a two‑stage PEFT schedule. It obtains competitive in‑domain results and better cross‑dataset generalization on farmland‑focused benchmarks.

Critical strengths
  1. The paper addresses a practically relevant problem in agricultural remote sensing. Combining pretrained segmentation, visual representation, and language models through lightweight adaptation is a reasonable engineering approach to handling limited task-specific training data.

  2. The reported results suggest practical utility. SFC-SAM3 achieves competitive in-domain performance on FarmSeg-VL and substantial improvements over the evaluated baselines on the selected cross-domain subsets, particularly RRSIS-D. These results indicate the potential value of the complete system for agricultural segmentation under domain shift.

Critical weaknesses
  1. The methodological novelty is limited. The frequency-aware prompt generator closely follows the structure available in SAM-Adapter: Fourier high-pass preprocessing, fusion with projected patch embeddings, layer-specific lightweight MLPs, and a shared up-projection for additive prompt injection. The remaining components primarily use familiar mechanisms, including weighted aggregation of intermediate visual features, token-level cross-attention, LoRA, and convolutional residual mask refinement. The paper is motivated by difficulties in interpreting crop-state descriptions, segmenting texture-poor farmland, and resolving spatial references. However, its response largely follows a standard engineering strategy: combining texture and visual feature extractors with a text feature extractor, then integrating the resulting features into an existing segmentation backbone. The manuscript does not clearly identify a new mechanism specifically developed to address these challenges or sufficiently establish a substantive methodological advance over existing adaptation and fusion approaches.

  2. Aspects of the experimental analysis could be strengthened to clarify the source of the gains. Table 2 introduces several changes together, such as the text bridge with EMA/TTA and DINO-FPN with OHEM and mask refinement. Its initial improvement also combines PromptGen-FFT with CAPA rather than isolating the frequency prior. Furthermore, all ablations are conducted on FarmSeg-VL, although cross-domain transfer is the most notable result. High-pass preprocessing could plausibly reduce sensitivity to certain appearance shifts, while the satellite-pretrained DINOv3 backbone offers another possible explanation for the transfer improvements. A matched SAM3 adaptation baseline and selected cross-domain ablations would help distinguish the benefits of individual established components from those of the proposed coordination. Reporting the exact category lists and sample counts for the cross-domain subsets would also improve reproducibility. These are secondary concerns and do not negate the potential practical value of the reported results.

  3. The paper offers limited new insight into the motivating challenges. The connection between the stated difficulties and the proposed mechanisms remains largely intuitive. Beyond reporting the performance of the complete system, the paper provides limited additional understanding of agricultural referring segmentation.

Minor Comments:

  1. The definition of M_r in Section 3.2 could be more precise. A centred rectangular low-pass mask with relative area r = 0.25 does not uniquely specify its construction. Please provide an explicit definition or pseudocode describing the rectangle dimensions, boundary conventions, and frequency-coordinate arrangement, including whether the spectrum is shifted before applying the mask.

  2. The freezing and fine-tuning annotations in Figure 1 could be clearer. The snowflake on the SAM3 image encoder suggests a frozen module, while the LoRA label indicates adaptation within it. These are compatible if the original weights are frozen and only the added LoRA parameters are trained, but the illustration could distinguish the two more explicitly.

Questions for authors
  1. What is the principal methodological advance over SAM-Adapter and existing foundation-model adaptation and fusion approaches? Please distinguish inherited components from new mechanisms, particularly within the frequency-aware prompt generator and the sequential DINO/text coordination.

  2. What specific, generalizable finding does the paper establish about adapting foundation models to agricultural referring segmentation, beyond the performance of the complete system? Which evidence connects that finding to the motivating challenges?

  3. Do you have cross-domain comparisons that isolate the frequency branch and the DINO bridge under matched training and inference settings? These would help assess whether the transfer gains depend on the proposed coordination or are largely explained by individual components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.