acceptodds
Withdrawn from ICLR 2027

CONCORD-SAM3: Native Evidence Coordination for Training-Free Open-Vocabulary Remote Sensing Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-vocabulary remote sensing semantic segmentation aims to assign pixel-level labels to arbitrary textual categories beyond predefined land-cover taxonomies. However, independent prompt responses lack explicit cross-class coordination, while textual descriptions alone do not provide image-specific visual references for diverse within-class appearances. We introduce CONCORD-SAM3, a training-free framework built on frozen SAM 3 that coordinates native evidence across competing classes and within-image appearances. Dual-Head Evidence Coordination (DHEC) calibrates class scores using shared visual–text descriptors and accumulates pixelwise instance support through a bounded noisy-OR rule. Positive semantic–instance soft-union residuals are gated by query presence and semantic support within native masks. Multi-Prototype Visual Calibration (MPVC) constructs multiple prototypes per class from the current image's multi-scale features. A prediction-margin gate strengthens visual calibration at ambiguous pixels while reducing intervention where scores are well separated. The framework uses one image encoding per image or crop and one prompt evaluation per supplied query, without target-domain annotations, parameter updates, auxiliary backbones, re-querying, or transformed views. Evaluations on 16 datasets demonstrate the effectiveness of our method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Training-Free Open-Vocabulary Segmentation, Remote Sensing Semantic Segmentation, SAM 3, Evidence Coordination, Multi-Prototype Visual Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ABZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CONCORD-SAM3, a training-free open-vocabulary remote-sensing segmentation framework built on frozen SAM 3. It combines Dual-Head Evidence Coordination (DHEC), which recalibrates cross-class scores and aggregates reliable instance evidence, with Multi-Prototype Visual Calibration (MPVC), which constructs image-specific class prototypes for further refinement. The method requires no parameter updates, auxiliary backbones, or repeated prompting, and achieves incremental improvements across 16 remote-sensing segmentation and extraction benchmarks.

Critical strengths
  1. Practical training-free design. CONCORD-SAM3 reuses native SAM 3 predictions and intermediate features without parameter updates, auxiliary backbones, additional image encoding, or repeated prompt evaluations, making the framework computationally practical and easy to apply.

  2. Broad empirical evaluation. The method is evaluated across 16 remote-sensing segmentation and extraction benchmarks and shows consistent improvements, providing reasonably strong evidence for its robustness and generality across diverse datasets.

Critical weaknesses
  • Although the paper targets training-free remote-sensing OVSS, the proposed DHEC and MPVC are largely generic calibration and prototype-refinement techniques, without explicitly addressing remote-sensing-specific challenges such as extreme scale variation, dense objects, or domain differences.

  • The method is heavily heuristic. The paper introduces a complicated multi-stage fusion pipeline, but many key design choices are only supported by intuitive explanations, without clearly justifying why the adopted formulations are necessary or preferable to simpler alternatives. In addition, the corresponding ablations are insufficient. Specific concerns include:

    1. In Accumulating Reliable Instance Evidence, the source and processing of oqjo_{qj} are not sufficiently clear. It is also unclear how the binary mask MqjM_{qj} is obtained from the soft mask and what threshold is used. If MqjM_{qj} is produced by hard thresholding, potentially useful low-confidence soft evidence may be discarded. The necessity of this binary masking operation and its threshold sensitivity should therefore be evaluated.
    2. In CCEC, the method specifically uses the 18th vision block and the Q/K projections from the first fusion encoder layer to construct visual-text descriptors, but the paper does not explain why these particular layers are chosen. Comparisons across different vision and fusion layers are needed.
    3. AGRF combines noisy-OR accumulation, semantic agreement, soft union, presence gating, and positive-only residual fusion, resulting in a highly hand-crafted pipeline.
    4. CCEC further introduces LogMeanExp aggregation, τs=0.1\tau_s = 0.1, multiplicative fusion bcπcλb_c \pi_c^\lambda, and the weighting factor λ\lambda, but these choices are not sufficiently justified. The paper should explain why multiplicative fusion is preferred over simpler additive fusion and provide sensitivity analyses for key hyperparameters.
    5. MPVC also relies on several fixed heuristic choices, including Top-30% pixel selection, exactly three prototypes per class, farthest-point initialization, and two clustering iterations. These choices may be particularly questionable for remote-sensing categories with highly different spatial extents, especially very small objects or absent classes. Ablations on the selected pixel ratio, number of prototypes, and clustering iterations are necessary.
    6. The final margin gate introduces additional hyperparameters α\alpha and TmT_m, together with the specific exponential form exp⁡(−Δ/Tm)\exp(-\Delta/T_m), without explaining why this formulation is preferable to linear, sigmoid, or hard gating. Moreover, prototype cosine similarity and segmentation confidence are not naturally calibrated to the same score space, so directly interpolating them also requires further justification.
Questions for authors

It is unclear how the query vocabulary is constructed for multi-class datasets. Specifically, does each image use the complete category vocabulary of the dataset, including classes absent from that image, or only the categories present in its ground-truth annotations?

Reviewer_qCqJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to better use the outputs and features of frozen SAM 3 for remote-sensing semantic segmentation. The authors argue that independently evaluated class queries need cross-class coordination, and that text queries alone cannot capture the different appearances of a category within an image. They address these issues with DHEC, which calibrates class scores and incorporates instance evidence, and MPVC, which refines predictions using image-specific visual prototypes. The complete method reports improvements over SegEarth-OV3 across 16 datasets without additional image encoding or prompt evaluations.

Critical strengths
  1. The paper identifies a useful problem: obtaining a response for each text query does not by itself resolve how competing responses should be combined into a semantic segmentation map. Coordinating the semantic, instance, and visual evidence already available in SAM 3 is a sensible approach. Reusing these outputs without additional image encoding or model training also makes the method practically appealing.

  2. The method achieves strong empirical performance across multiple remote-sensing datasets, covering multi-class semantic segmentation, building extraction, and road extraction. It consistently improves over SegEarth-OV3 on all reported datasets, supporting its effectiveness across different remote-sensing settings.

Critical weaknesses
  1. The methodological novelty appears limited. The method combines cross-class score calibration, semantic-instance residual fusion, and image-specific prototype refinement using outputs and features already available in SAM 3. The overall design is reasonable and achieves good results, but the contribution currently appears to be an incremental inference-time improvement, with limited evidence of a distinct new mechanism or insight. In particular, CCEC and AGRF are always ablated together, and the multi-prototype design is not compared with a single-prototype alternative. This makes it difficult to identify which choices constitute the substantive contribution and which provide auxiliary adjustments. The experiments establish the effectiveness of the complete pipeline more clearly than its methodological novelty.

  2. The connection between the motivation and the proposed designs is not sufficiently established. The lack of coordination between independent queries and the presence of multiple within-class appearances are plausible starting points. However, the paper does not sufficiently demonstrate how these issues account for major errors in SAM 3’s remote-sensing predictions, or whether the proposed components specifically resolve them. For example, there is no direct analysis of conflicting class responses, nor a visualization showing that the three prototypes capture distinct within-class appearances. The clearest improvements in Figure 4 involve fewer scattered false predictions, whose connection to cross-class competition and appearance diversity remains unclear. The motivation therefore provides a plausible explanation for the components, but lacks direct evidence establishing why these particular designs are needed.

  3. A suggestion: Figure 2 essentially repeats the data already presented in the table and does not provide any new information. Instead of repeating this information, it would be better to include additional evidence—beyond performance-based experiments—to substantiate the stated motivation and the role of the proposed components.

Questions for authors
  1. What is the main methodological advance beyond combining score calibration, residual fusion, and prototype-based refinement? Which aspect of the proposed coordination is essential to the improvement over SegEarth-OV3?

  2. Can the authors show concrete cases where independently evaluated class queries produce conflicting responses, and demonstrate how CCEC resolves them? Results for CCEC alone, AGRF alone, and their combination, including λ = 0, would help establish this connection.

  3. Do the three prototypes capture distinct appearances within a class? A comparison with a single-prototype version, together with visualizations of the corresponding image regions and prediction changes, would help determine whether modeling appearance diversity explains the gain.

  4. The paper has introduced numerous components without providing an analysis of the computational overhead associated with each. Can you substantiate the claim that this approach is more efficient than SegEarth-OV3? Specifically, which aspects contribute to the reduction in overhead? I would like to see a detailed analysis here. If these additions actually increase overhead, the performance advantage over SegEarth-OV3 may be marginal; this trade-off warrants further discussion.

Reviewer_VfBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CONCORD-SAM3, a training-free framework for open-vocabulary remote sensing semantic segmentation based on the frozen SAM3 model. The proposed framework consists of two main components: Dual-Head Evidence Coordination (DHEC) and Multi-Prototype Visual Calibration (MPVC). DHEC coordinates semantic and instance-level predictions through cross-concept score calibration and agreement-gated residual fusion, while MPVC constructs image-specific visual prototypes to refine class predictions based on prediction uncertainty. The framework reuses native SAM3 predictions and features without additional training, auxiliary backbones, or repeated model inference. Extensive experiments across 16 remote sensing datasets demonstrate effective performance improvements over existing training-free OVSS methods.

Critical strengths
  1. Practical training-free framework. The proposed method leverages native SAM3 predictions and features without requiring parameter updates or auxiliary models, providing a practical approach to open-vocabulary remote sensing segmentation.

  2. Comprehensive experimental evaluation. Experiments on 16 remote sensing datasets, spanning multi-class semantic segmentation and building and road extraction tasks, demonstrate consistent performance improvements over existing training-free methods.

Critical weaknesses

The justification for several key methodological choices could be further strengthened, and more fine-grained ablation studies would help better demonstrate the effectiveness of individual components. In addition, further clarification is needed regarding the fairness of experimental comparisons and the remote sensing-specific motivation underlying the proposed method.

Questions for authors

1.Justification of Noisy-OR aggregation. The proposed Noisy-OR rule increases the aggregated confidence with the number of contributing instance responses, without explicitly accounting for their redundancy or reliability. Consequently, multiple overlapping low-confidence predictions may yield a higher aggregated score than a single high-confidence prediction, potentially amplifying redundant or incorrect evidence. The authors should further justify the reasonableness of this aggregation strategy.

2.The rationale behind several design choices requires further clarification. For example, Eqs. (3) and (5) retain only positive residuals while excluding negative ones. The authors should further explain the motivation and justification for this asymmetric design.

3.Unclear synonym generation and comparison fairness. The proposed method incorporates synonyms into class text prompts, but the manuscript does not clarify how these synonyms are obtained or whether this strategy follows existing work. Since synonym-based prompt expansion may improve segmentation performance, it is unclear whether comparisons with methods that do not employ the same strategy are fair. The authors should clarify how the synonyms are obtained and ensure fair comparisons by adopting consistent text prompting strategies across methods.

4.Insufficient ablation studies. Table 3 only evaluates the contributions of DHEC and MPVC, without separately examining the effectiveness of their key components. For example, the individual contributions of Cross-Concept Evidence Calibration (CCEC) and Agreement-Gated Residual Fusion (AGRF) remain unclear. Furthermore, the effects of the number of visual prototypes and the farthest-point sampling strategy have not been investigated. More fine-grained ablation studies are needed to validate the effectiveness of these design choices.

5.Remote sensing-specific motivation and generalization. The proposed method is evaluated exclusively on remote sensing datasets, but it remains unclear how its design specifically addresses the characteristics and challenges of remote sensing imagery. Given that the proposed components appear to rely on general prediction calibration and feature refinement mechanisms, their applicability beyond remote sensing imagery warrants investigation. The authors should either clarify the remote sensing-specific motivation behind the proposed design or conduct additional experiments on segmentation benchmarks from other domains (e.g., natural image datasets such as PASCAL Context, COCO-Stuff, Cityscapes, and others) to assess its generalizability.

6.Comparison with recent methods. The proposed method shares similarities with TMPA [1] (CVPR 2026), as both leverage visual information to calibrate predictions and improve open-vocabulary remote sensing segmentation. However, the manuscript does not include a comparison with TMPA.The authors should discuss the methodological differences and provide experimental comparisons to better establish the novelty and effectiveness of the proposed method.

[1] Test-Time Multi-Prompt Adaptation for Open-Vocabulary Remote Sensing Image Segmentation. CVPR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.