acceptodds
Under review as a conference paper at ICLR 2027

ActiveSAM: Fast and Accurate Open-Vocabulary Semantic Segmentation with Frozen SAM 3

Abstract

Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then uses evidence-proportional grounding to estimate an image-conditioned active set from a low-resolution presence preview. Only retained prompts receive full-resolution mask prediction, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation. To resolve overlapping concept responses, exclusive concept decoding compares each pixel's joint score vector with class signatures estimated once per vocabulary from unlabeled images. ActiveSAM requires no weight updates, no oracle class-presence labels and no per-dataset hyperparameter tuning. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by +2.1 mIoU on average while running much faster, with 7.3-12.2x speedups on large-vocabulary datasets. ActiveSAM also achieves the highest accuracy under image corruptions that simulate real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary segmentation, foundation models, training-free inference, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oBwYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ActiveSAM, a training-free inference pipeline for applying frozen SAM 3 to open-vocabulary semantic segmentation. It combines prompt canonicalization and expansion, a low-resolution presence preview that filters prompts before full-resolution decoding, bucketed decoding, and a signature-based rule for resolving overlapping class responses. The method is evaluated on eight segmentation benchmarks and reports higher mean mIoU and substantially higher throughput than SAM 3 and SegEarth-OV3.

Critical strengths

(i) The empirical coverage is unusually broad for this setting. The paper evaluates eight benchmarks, includes matched SAM 3-based comparisons, separates the accuracy-oriented and efficiency-oriented components, and reports both clean and corrupted-image results. The EPG ablation is particularly useful: it shows that prompt filtering changes mean accuracy very little while materially reducing per-image computation.

(ii) The paper addresses a real inefficiency in concept-prompted segmentation. Reusing one image encoding and avoiding full decoding for most vocabulary entries is simple, but practically consequential when the vocabulary is large.

Critical weaknesses

(i) The deployment claim does not account for the method's per-vocabulary calibration stage. The full method estimates response signatures and class masses by running the pipeline over 9858 COCO images for every vocabulary. This cost is excluded from the reported FPS numbers. It is therefore unclear when the claimed speedup is actually realized, especially in the motivating setting where vocabularies are dynamic or used for only a modest number of images. More importantly, the evaluation covers eight fixed benchmark vocabularies; it does not test vocabulary edits, rare concepts, or vocabularies whose classes have poor support in COCO. The paper currently establishes fast amortized inference after substantial offline calibration, not efficient segmentation under a genuinely changing open vocabulary.

(ii) The accuracy contribution is much less clean than the headline comparison suggests. EPG is essentially accuracy-neutral, ECD adds only 0.7 mean mIoU and is concentrated on Cityscapes, while the largest CPE contribution comes from an unspecified hand-written lexical canonicalizer. The manuscript gives only an example and says that fixed string rules are used; it does not list the rules or establish that they were designed independently of the benchmark labels and errors. Since the total gain over SegEarth-OV3 is 2.1 points, benchmark-aware name rewriting and different post-processing can plausibly explain a meaningful fraction of the margin. A matched comparison that adds the same active-prompt/bucketing machinery to the closest SAM 3 baseline, while holding prompt rewriting and final decoding fixed, is needed to isolate the contribution.

Questions for authors

(i) What are the wall-clock time and GPU cost of estimating signatures from the 2k and 10k pools for each vocabulary? Please report end-to-end or amortized cost as a function of the number of inference images and the frequency of vocabulary changes.

(ii) Can the authors provide the complete canonicalization rules and results on previously unseen, long-tail, or incrementally edited vocabularies? Were any rules introduced after examining benchmark-specific failures?

(iii) Under identical prompts and final decoding, how much speed and accuracy are obtained by applying only the proposed active-prompt selection and bucketing to SAM 3 or SegEarth-OV3?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer oBwY [1/3]

We thank the reviewer for the valuable feedback and comments, which have definitely helped us improve the quality of the paper. We have carefully updated the revised manuscript to include all the mentioned suggestions and changes. We further clarify each of your questions in the review.

W1: The deployment claim does not account for the method's per-vocabulary calibration stage. The full method estimates response signatures and class masses by running the pipeline over 9858 COCO images for every vocabulary. This cost is excluded from the reported FPS numbers. It is therefore unclear when the claimed speedup is actually realized, especially in the motivating setting where vocabularies are dynamic or used for only a modest number of images. More importantly, the evaluation covers eight fixed benchmark vocabularies; it does not test vocabulary edits, rare concepts, or vocabularies whose classes have poor support in COCO. The paper currently establishes fast amortized inference after substantial offline calibration, not efficient segmentation under a genuinely changing open vocabulary.

Thanks for the comment and suggestion. We provide detailed clarifications for each point below.

a) The offline calibration is used only for ECD. Without ECD, ActiveSAM already achieves 65.27 mean mIoU, compared with 63.9 for SegEarth-OV3. For the full method, signatures are estimated once per vocabulary and cached for subsequent images. We report this preparation cost separately from online FPS and include it when evaluating total inference time.

When reducing the calibration pool from 10k to 2k images, mean mIoU changes only from 66.02 to 66.00 (Table 5). The 2k pool takes 4.7–9.8 minutes per vocabulary on one RTX 5090 (Table 16 gives the details for both the 10k and 2k pools). Including this cost, full ActiveSAM becomes faster than SegEarth-OV3 after 221–1,184 images, depending on the vocabulary. Q1 reports the per-vocabulary costs and calculation. For shorter workloads, ActiveSAM can run without ECD or signature estimation.

b) Following your suggestion on testing edits, rare concepts, or vocabularies whose classes have poor support in COCO, we have evaluated Context-459 and ADE20K-847, expanding Context59 from 59 to 459 classes and ADE20K from 150 to 847 classes on the same validation images. Neither expanded vocabulary was used during development. We keep the original hyperparameters and canonicalization mapping unchanged. That mapping applies to only 7 of 459 Context-459 names and 14 of 1,281 ADE20K-847 names, including aliases.

We have also tested an automatic canonicalization (WordNet-based) that requires no manually defined name mappings, described in Q2 and more details in Appendix I of the revised manuscript.

Context-459ADE20K-847
SegEarth-OV3, mIoU18.7814.47
ActiveSAM without ECD, fixed mapping, mIoU18.9714.80
ActiveSAM without ECD, automatic canonicalization, mIoU19.2014.52
Mean classes decoded per image / vocabulary size38.9 / 459113.0 / 847
SegEarth-OV3, FPS0.130.046
ActiveSAM, FPS (with ECD)2.370.83
Speedup over SegEarth-OV318.3x18.0x

ActiveSAM decodes only 38.9 of 459 classes and 113.0 of 847 classes per image, while both configurations without ECD achieve comparable mIoU to SegEarth-OV3. The measured online speedups are 18.3x and 18.0x faster than SegEarth-OV3.

On these expanded vocabularies, 201 of 459 and 452 of 847 classes are flagged uncalibrated (Classes with limited COCO support). The 2k calibration pass takes 16 and 39 minutes, respectively. We also evaluate the effect of enabling ECD under both canonicalization methods:

CanonicalizationContext-459 without ECDContext-459 with ECDADE20K-847 without ECDADE20K-847 with ECD
Fixed mapping18.9718.3214.8013.86
Automatic19.2018.5314.5213.61

Prompt selection remains effective without calibration on both expanded vocabularies. ECD gives lower accuracy here, where many classes have insufficient support in the COCO calibration pool. For these settings, using ActiveSAM without ECD gives the better result. We have added this analysis into Appendix Q of the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer oBwY [2/3]

W2: The accuracy contribution is much less clean than the headline comparison suggests. EPG is essentially accuracy-neutral, ECD adds only 0.7 mean mIoU and is concentrated on Cityscapes, while the largest CPE contribution comes from an unspecified hand-written lexical canonicalizer. The manuscript gives only an example and says that fixed string rules are used; it does not list the rules or establish that they were designed independently of the benchmark labels and errors. Since the total gain over SegEarth-OV3 is 2.1 points, benchmark-aware name rewriting and different post-processing can plausibly explain a meaningful fraction of the margin. A matched comparison that adds the same active-prompt/bucketing machinery to the closest SAM 3 baseline, while holding prompt rewriting and final decoding fixed, is needed to isolate the contribution.

Thanks for your detailed suggestion. Following it, we have evaluated the matched comparison and also tested whether the canonicalization benefit depends on the fixed mapping.

ConfigurationMean mIoU
SegEarth-OV3 (Table 1)63.9
+ prompt selection and bucketing (same prompts, thresholds, and argmax)64.09
+ canonical names65.00
+ context tokens (complete CPE, still using baseline decoding)65.28
+ ECD (full ActiveSAM)66.02

Prompt selection and bucketing alone give 6.8× faster inference with comparable accuracy (64.09 versus 63.9). CPE then improves mIoU to 65.28 with the baseline decoding unchanged, and ECD further improves it to 66.02.

We also test a WordNet-based alternative that automatically splits compound names, separates hyphenated names, and standardizes spelling without any manually defined per-name mappings. With context tokens enabled and ECD disabled, mean mIoU is 64.53 without canonicalization, 65.28 with the fixed mapping, and 65.39 with automatic canonicalization. With full ActiveSAM and re-estimated signatures, the automatic version reaches 66.11 mIoU, compared with 66.02 for the original mapping. This shows that the gain does not depend on the particular predefined mappings.

We have added the complete mapping rule and the automatic procedure in Appendix I.

Q1: What are the wall-clock time and GPU cost of estimating signatures from the 2k and 10k pools for each vocabulary? Please report end-to-end or amortized cost as a function of the number of inference images and the frequency of vocabulary changes.

Thanks for your question. Following your comment, we have measured the one-time cost of signature estimation separately from online FPS. The measured times on one RTX 5090 are shown below.

V21P60C-OV20P59C-SCityADE
2k pool, wall-clock (min)4.77.16.04.76.99.85.39.7
10k pool, wall-clock (min)20.732.826.520.331.647.722.946.1

Calibration is performed only once per vocabulary, and the resulting signatures are cached for subsequent inference.

For the question on total cost and vocabulary changes, let TT denote the one-time calibration cost, and tAt_A and tSt_S denote the per-image inference times of ActiveSAM and SegEarth-OV3. For a vocabulary used on nn images, the total and average costs are

CA(n)=T+ntA,tˉA(n)=tA+Tn.C_A(n)=T+nt_A,\qquad \bar t_A(n)=t_A+\frac{T}{n}.

Including calibration, ActiveSAM becomes faster than SegEarth-OV3 when

n>TtS−tAn > \frac{T}{t_S-t_A}

When vocabularies change, the calibration cost is incurred once for each new vocabulary. Based on our measurements, the full method recovers its calibration cost after 221–1,184 images with the 2k pool, or approximately 1,100–5,300 images with the 10k pool. For shorter workloads, ActiveSAM can run without ECD or calibration, achieving 65.27 mean mIoU versus 63.9 for SegEarth-OV3. We have added this analysis to Appendix K in the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer oBwY [3/3]

Q2: Can the authors provide the complete canonicalization rules and results on previously unseen, long-tail, or incrementally edited vocabularies? Were any rules introduced after examining benchmark-specific failures?

Thanks for your question. We have added the complete canonicalization mapping (and WordNet version) into Appendix I of the revised manuscript.

For previously unseen and expanded vocabularies, W1 reports our new experiments on Context-459 and ADE20K-847, extending the original vocabularies from 59 to 459 and from 150 to 847 classes. We keep the original mapping and hyperparameters unchanged. Without ECD, ActiveSAM achieves 18.97 and 14.80 mIoU, compared with 18.78 and 14.47 for SegEarth-OV3.

Regarding the canonicalization rule, no mapping entries were introduced after examining benchmark-specific failures. To further test whether the improvement depends on the predefined mappings, we evaluate an automatic WordNet-based alternative without manually defined name entries. With full ActiveSAM, it achieves 66.11 mIoU, compared with 66.02 for the original mapping. This shows that the improvement does not depend on the particular predefined entries. We have added this evaluation in Appendix L in the revised manuscript.

Q3: Under identical prompts and final decoding, how much speed and accuracy are obtained by applying only the proposed active-prompt selection and bucketing to SAM 3 or SegEarth-OV3?

Thanks for your question. We have evaluated the setting when only prompt selection and bucketing to SegEarth-OV3, keeping its prompts, thresholds, and argmax decoding unchanged. The results are shown below.

V21P60C-OV20P59C-SCityADEMean
SegEarth-OV3, mIoU (Table 1)79.853.472.096.859.242.869.737.663.9
+ prompt selection and bucketing, mIoU79.8854.1372.2396.5859.8042.6669.6837.7264.09
Speed vs SegEarth-OV33.9×6.5×9.6×4.9×6.5×11.4×2.2×9.4×6.8×

Prompt selection and bucketing alone give 6.8x faster inference with comparable accuracy (64.09 versus 63.9 mean mIoU). The speedup reaches 11.4× on COCO-Stuff. This comparison isolates the efficiency benefit without changing the baseline's prompts or final decoding. We have added this evaluation to Appendix P in the revised manuscript.

AuthorsOpenReview ↗

Follow-up to Reviewer oBwY

We thank the reviewer again for the careful assessment and constructive feedback.

Following your suggestions, we have added detailed analyses of calibration costs, including their impact on total inference time and a calibration-free alternative. We have also evaluated two larger vocabularies with long-tail classes and conducted matched baseline comparisons and joint ablations to isolate the contributions of prompt selection and bucketing, CPE, and ECD. In addition, we provide the complete canonicalization mapping, clarify that no rules were added after examining benchmark-specific failures, and evaluate automatic WordNet-based canonicalization. All of these experiments and analyses are included in the revised manuscript.

We hope our response and additional experiments address your concerns. We would welcome any further feedback or questions you may have.

Reviewer_chXEOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ActiveSAM, a training-free framework for open-vocabulary semantic segmentation that leverages SAM 3's presence predictions to selectively perform full-resolution mask decoding, improving inference efficiency. ActiveSAM achieves state-of-the-art performance across eight benchmarks, reaching 66.02% mIoU while delivering 7.3× to 12.2× inference speedup on different datasets. Furthermore, it demonstrates strong robustness to distribution shifts, outperforming conventional CLIP-based methods.

Critical strengths
  1. The paper provides comprehensive coverage of relevant literature in this field and clearly distinguishes the proposed approach from prior methods in terms of underlying principles.

  2. Notably, ActiveSAM achieves an impressive trade-off between inference speed and segmentation accuracy across eight OVSS benchmarks.

Critical weaknesses
  1. In the proposed method, Eq. (2) is used to select classes that are likely to be present in an image. However, low-resolution predictions at 336 × 336 may not accurately identify all ground-truth classes. In particular, small objects and rare categories may be incorrectly filtered out, making it impossible to recover these classes during subsequent high-resolution decoding. Therefore, reporting the recall rates for small objects and long-tail categories is necessary to further evaluate the reliability of the proposed prompt selection strategy.

  2. Lack of component-wise computational efficiency analysis. Although the experiments provide extensive evaluations of the overall inference efficiency of the proposed method, the computational costs of individual components are not reported or analyzed.

  3. The visualization results presented in the appendix on datasets such as VOC21 and ADE20K fail to effectively demonstrate the superiority of the proposed method.

Questions for authors

Please refer to "Critical Weaknesses".

4 comments
AuthorsOpenReview ↗

Responses to Reviewer chXE [1/2]

We thank the reviewer for the valuable feedback and suggestions, which have definitely helped us improve the quality of the paper. We have carefully updated the revised manuscript and uploaded a revision to accommodate all the mentioned suggestions and changes. We further provide detailed clarifications for each of your questions:

W1: In the proposed method, Eq. (2) is used to select classes that are likely to be present in an image. However, low-resolution predictions at 336 × 336 may not accurately identify all ground-truth classes. In particular, small objects and rare categories may be incorrectly filtered out, making it impossible to recover these classes during subsequent high-resolution decoding. Therefore, reporting the recall rates for small objects and long-tail categories is necessary to further evaluate the reliability of the proposed prompt selection strategy.

Thanks for your insight and suggestion. Following it, we have further evaluated retention for small objects and long-tail classes across all eight benchmarks, as shown below. (Table 11 already reported overall class retention after both EPG stages)

Retention (%)V21P60C-OV20P59C-SCityADE
All ground-truth image–class pairs (Table 11)97.493.493.999.493.479.899.093.6
Small objects93.593.193.799.893.181.999.796.2
Long-tail classes90.390.790.599.390.784.898.680.1

Retention measures whether a ground-truth class remains available for prediction after filtering. For small objects, we use connected components with an area below 1,024 pixels, following COCO's small-object area threshold. For long-tail classes, we select the least frequent third of each vocabulary, based on the number of validation images containing each class.

Small-object retention remains close to overall retention across all eight benchmarks. Since selection operates at the class level, evidence from any instance can retain its class for full-resolution decoding. Long-tail retention ranges from 80.1% to 99.3%.

On ADE20K and COCO-Stuff, where long-tail retention is lowest, decoding every prompt improves mIoU by only 0.02 and 0.34, respectively (Table 10). Across all benchmarks, the accuracy difference is at most 0.34 mIoU. Table 12 further shows that increasing the preview resolution from 336 to 672 changes mean mIoU by only 0.04.

We have added this analysis to Appendix N of the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer chXE [2/2]

W2: Lack of component-wise computational efficiency analysis. Although the experiments provide extensive evaluations of the overall inference efficiency of the proposed method, the computational costs of individual components are not reported or analyzed.

Thanks for your comment. Following your suggestion, we have measured the inference time of each component across all eight benchmarks, as shown below. (Table 3 already measures the effects of prompt selection and bucketing)

Stage (ms/image)V21P60C-OV20P59C-SCityADE
Image loading16.417.622.418.117.623.253.418.2
EPG preview: 336 image encoder9.18.58.69.18.58.612.69.1
EPG preview: presence of all prompts9.019.127.19.619.054.210.946.6
1008 image encoder34.134.134.134.034.134.235.634.3
Fusion encoder, admitted prompts with CPE tokens12.833.019.710.731.059.541.471.1
EPG one-layer screening1.93.42.41.83.35.74.16.7
Full decoder and heads, screened prompts18.245.324.716.642.272.261.992.1
Head fusion, upsampling and resizing0.51.10.80.51.12.16.62.6
ECD0.60.60.60.50.50.64.20.8
Other3.33.43.73.33.44.09.54.3
End to end105.9166.1144.1104.2160.5264.4240.4285.7

The EPG preview takes 18–63 ms/image. Full-resolution fusion and decoding increase from 27 ms on VOC20 to 163 ms on ADE20K, reflecting the number of retained prompts. CPE prompts are cached, with their additional tokens accounted for in the fusion time. ECD takes only 0.5–4.2 ms/image, at most 1.8% of inference time. Its signatures are estimated offline once per vocabulary, taking 4.7–9.8 minutes with the 2k-image calibration pool.

We have added this analysis to Appendix O of the revised manuscript.

W3: The visualization results presented in the appendix on datasets such as VOC21 and ADE20K fail to effectively demonstrate the superiority of the proposed method.

Thanks for your comment. We clarify that the original visualizations use a small random subset of images, so not every example shows a large difference from SegEarth-OV3. Examples such as VOC21 (row 2) and ADE20K (row 6) still show that ActiveSAM better separates foreground from background and avoids incorrect foreground predictions. Following your suggestion, we have replaced the previous visualization with more examples that make these improvements easier to notice. The updated visualizations are provided in Appendix R in the revised manuscript.

Thank you for the authors' response. However, after carefully reading it, one concern remains particularly puzzling to me. In my view, the weaknesses I highlighted are all quite important. Given that the main paper is still well under 10 pages, I find it difficult to understand why the experiments addressing Questions 1 and 2 have been placed entirely in the appendix rather than incorporated into the main paper. Taking into account the comments from the other two reviewers, I have decided to maintain my original score.

Dear Reviewer chXE,

Thank you very much for your follow-up and thoughtful feedback. Following your valuable suggestion, we have carefully revised Section 4.2 to include two important analyses: "Class retention by component size and class frequency" (W1) and "Time per stage" (W2) together with their corresponding experimental tables directly in the main paper, and we have uploaded the updated manuscript.

We hope these revisions address your remaining concern. We would be grateful if you could take another look at the revised manuscript and share any further feedback. We sincerely appreciate your time and constructive suggestions, which have helped us further improve the quality and completeness of our paper.

Reviewer_Ad1TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActiveSAM turns a frozen SAM 3 into an "active-vocabulary" segmenter through three training-free components: CPE canonicalizes class names and appends vocabulary-nearest and WordNet-hypernym tokens; EPG estimates class presence at 336×336 with the presence head only, decodes just the prompts passing τ=0.1\tau=0.1 at 1008×1008, and reuses features across buckets (K=32K=32) with a first-layer early exit; ECD estimates per-class response signatures on 9,858 unlabeled COCO images and replaces per-pixel argmax with a prior-weighted nearest-signature rule. Reported: 66.02 mean mIoU over eight benchmarks (+2.1 over SegEarth-OV3), 7.3–12.2× speedup on large-vocabulary datasets (7.5× averaged), highest mean mIoU under four ImageNet-C corruptions, shared hyperparameters throughout.

Critical strengths
  • Good system insight. Moving SAM 3's presence signal from post-decode filtering to pre-decode admission is a reasonable and effective restructuring: at τ=0.1\tau=0.1 only 7.3/81 prompts survive on COCO-Object and 30.8/150 on ADE20K while 93.8% of ground-truth image–class pairs remain available.
  • Broad, shared-hyperparameter ablations. Components, CPE sub-components, preview on/off, admission statistics, six hyperparameters plus two rule switches (14 variants), calibration pool source, preview resolution and corruption robustness — with one hyperparameter set across all eight datasets.
  • Candid leakage handling. COCO train2017 is stated to be disjoint from val2017, and the "signatures estimated from evaluation images" reference row (66.40, only +0.38) is reported separately with an explicit statement that the main results do not use it.
Critical weaknesses
  1. The absolute timing used to support the acceleration claim cannot be reproduced from the paper's own FPS data. Table 10's caption and column are self-consistent (each value equals w/ preview ÷ w/o preview), but the text then states that "EPG reduces the time per image from 0.521 to 0.184 seconds" averaged over the eight benchmarks, whereas the FPS figures imply ≈0.385 → 0.161 s; Table 12 separately gives 0.135 s over only five datasets. This sits in the sentence supporting the central efficiency claim, so the accounting must be rebuilt and unified.

  2. ECD's benefit is small and its coverage is never reported. Removing ECD costs only −0.7 mIoU on average, concentrated in Cityscapes (−3.3) and Context60 (−1.5), and the paper reports no variance, CIs or seeds anywhere. Since signatures are distilled from the frozen model's own predictions, the paper must report the per-benchmark fraction of classes flagged uncalibrated (nk<N=2000n_k<N=2000) — if it is high, most ECD pixels do nothing and the Cityscapes gain is unexplained.

  3. The two most direct controls are missing. The paper's two directions each cover half of "presence filtering + per-pixel argmax" (Table 4's "−ECD" still includes CPE; Table 7's "without CPE" still includes ECD), and there is no speed–accuracy curve for thresholding SAM 3's native presence head directly. Without these it cannot be shown that the 2.1-point gain is not simply split between the two parts.

  4. The applicable range is stated selectively. The abstract mentions only "7.3–12.2× on large-vocabulary datasets", whereas Cityscapes achieves 1.25× in Table 10 — precisely the autonomous-driving setting the motivation invokes, where the premise "the image contains a small subset of the vocabulary" is least likely to hold. Three baseline accountings (vs SAM 3: 5.2–34.5×; vs SegEarth-OV3: 2.2–12.2×; internal: 1.25–4.12×) are also mixed within one discussion.

Questions for authors
  1. Please give the raw measurement behind 0.521 → 0.184 s and unify the per-image accounting with 0.135 s (Table 12, five datasets) and ≈0.161 s (implied by 6.23 FPS): which steps are included, and what is the aggregation for the "7.5×" average?
  2. Please report ECD's effective coverage (classes flagged uncalibrated, fraction of pixels falling back) and the classes/pixel counts behind the +3.3 Cityscapes gain; and explain how Eq. 4's distance remains comparable across values of ∣A(I)∣|A(I)| (3.7 classes on VOC21 vs 30.8 on ADE20K).
  3. Please add "presence filtering + per-pixel argmax" and "CPE and ECD both removed", plus a native presence-threshold sweep, and report multi-run means and standard deviations for all key results.
  4. Please give CPE's complete canonicalization rule table η\eta and the WordNet version (this item is ~77% of CPE's gain), and specify the evaluation protocol for VOC20 = 97.08 relative to ReME (93.2), SegEarth-OV3 (96.8) and your own VOC21 (83.42).
5 comments
AuthorsOpenReview ↗

Responses to Reviewer Ad1T [1/4]

We thank the reviewer for the detailed and constructive comments, which have definitely helped us improve the quality of the paper. We have carefully revised our manuscript and uploaded a revision to accommodate all your mentioned suggestions. Below, we further clarify each of your questions in detail:

W1&Q1: The absolute timing used to support the acceleration claim cannot be reproduced from the paper's own FPS data. Table 10's caption and column are self-consistent (each value equals w/ preview ÷ w/o preview), but the text then states that "EPG reduces the time per image from 0.521 to 0.184 seconds" averaged over the eight benchmarks, whereas the FPS figures imply ≈0.385 → 0.161 s; Table 12 separately gives 0.135 s over only five datasets. This sits in the sentence supporting the central efficiency claim, so the accounting must be rebuilt and unified. Please give the raw measurement behind 0.521 → 0.184 s and unify the per-image accounting with 0.135 s (Table 12, five datasets) and ≈0.161 s (implied by 6.23 FPS): which steps are included, and what is the aggregation for the "7.5×" average?

Thanks for the constructive comment. We clarify that the reported times are reproducible from Table 10, and we describe the steps we use below:

a) Eight benchmarks: 0.521 → 0.184 s/image.

Using Table 10 in the order VOC21, VOC20, Cityscapes, Context59, Context60, COCO-Object, COCO-Stuff and ADE20K, we calculate:

tˉw/o preview=18(15.09+14.40+13.19+12.39+12.40+11.81+10.92+11.05)=0.5205…≈0.521 s/image.\bar t_{\mathrm{w/o\ preview}}=\frac{1}{8}\left(\frac{1}{5.09}+\frac{1}{4.40}+\frac{1}{3.19}+\frac{1}{2.39}+\frac{1}{2.40}+\frac{1}{1.81}+\frac{1}{0.92}+\frac{1}{1.05}\right)=0.5205\ldots\approx\mathbf{0.521}\ \mathrm{s/image}.

tˉw/ preview=18(19.37+19.54+13.98+16.33+16.12+17.16+13.79+13.51)≈0.1841≈0.184 s/image.\bar t_{\mathrm{w/\ preview}}=\frac{1}{8}\left(\frac{1}{9.37}+\frac{1}{9.54}+\frac{1}{3.98}+\frac{1}{6.33}+\frac{1}{6.12}+\frac{1}{7.16}+\frac{1}{3.79}+\frac{1}{3.51}\right)\approx0.1841\approx\mathbf{0.184}\ \mathrm{s/image}.

b) Five benchmarks: 0.135 s/image.

Table 12 specifies VOC21, Context60, COCO-Object, VOC20 and Context59. Applying the same averaging formula to their Table 10 entries gives

tˉfive=15(19.37+16.12+17.16+19.54+16.33)≈0.135 s/image.\bar t_{\mathrm{five}}=\frac{1}{5}\left(\frac{1}{9.37}+\frac{1}{6.12}+\frac{1}{7.16}+\frac{1}{9.54}+\frac{1}{6.33}\right)\approx\mathbf{0.135}\ \mathrm{s/image}.

This matches Table 12 at the reported precision.

For the eight-benchmark comparison, 0.161 s/image comes from dividing 1 by the mean FPS (6.23). Our reported 0.184 s/image instead averages the per-benchmark times (1/FPS), giving each benchmark equal weight.

c) Aggregation for the "7.5×" average.

We clarify that this compares ActiveSAM with SegEarth-OV3. We calculate the speedup on each benchmark and then take the equal-weight arithmetic mean:

sˉ=18∑d=18FPSActiveSAM,dFPSSegEarth,d.\bar s=\frac{1}{8}\sum_{d=1}^{8}\frac{\mathrm{FPS}_ {\mathrm{ActiveSAM},d}}{\mathrm{FPS}_ {\mathrm{SegEarth},d}}.

sˉ=18(9.372.19+9.541.92+3.981.81+6.330.86+6.120.84+7.160.63+3.790.31+3.510.35)≈7.46×≈7.5×.\bar s=\frac{1}{8}\left(\frac{9.37}{2.19}+\frac{9.54}{1.92}+\frac{3.98}{1.81}+\frac{6.33}{0.86}+\frac{6.12}{0.84}+\frac{7.16}{0.63}+\frac{3.79}{0.31}+\frac{3.51}{0.35}\right)\approx7.46\times\approx7.5\times.

AuthorsOpenReview ↗

Responses to Reviewer Ad1T [2/4]

W2 & Q2: ECD's benefit is small and its coverage is never reported. Removing ECD costs only −0.7 mIoU on average, concentrated in Cityscapes (−3.3) and Context60 (−1.5), and the paper reports no variance, CIs, or seeds anywhere. Since signatures are distilled from the frozen model's own predictions, the paper must report the per-benchmark fraction of classes flagged uncalibrated (nk<N=2000n_k < N = 2000) — if it is high, most ECD pixels do nothing and the Cityscapes gain is unexplained. Please report ECD's effective coverage (classes flagged uncalibrated, fraction of pixels falling back) and the classes/pixel counts behind the +3.3 Cityscapes gain; and explain how Eq. 4's distance remains comparable across values of ∣A(I)∣|\mathcal{A}(I)| (3.7 classes on VOC21 vs 30.8 on ADE20K).

Thanks for your detailed comment and suggestions. Following them, we have conducted careful analyses for each point, and we clarify them one by one below.

a) ECD coverage and pixels falling back.

We measure the fraction of uncalibrated classes, pixels falling back to argmax, and ECD's accuracy contribution across all eight benchmarks.

V21P60C-OV20P59C-SCityADE
Classes216081205917119150
Classes flagged uncalibrated (nk<Nn_k < N)00000404
Fraction of classes flagged uncalibrated (%)0.000.000.000.000.002.340.002.67
Pixels falling back because a class has insufficient calibration support (% of image pixels)0.000.000.000.000.000.000.000.03
Pixels falling back because the two signatures select different classes (% of image pixels)1.67.44.216.76.513.67.410.7
mIoU gain of ECD (Table 4) and 95% CI+0.15 [-0.46, +0.72]+1.54 [+0.72, +2.09]+0.41 [-0.02, +0.81]-0.02 [-0.09, +0.05]-0.05 [-0.26, +0.18]+0.64 [+0.43, +0.86]+3.31 [+2.98, +3.62]+0.03 [-0.65, +0.80]

Only 8 of 581 class entries are flagged uncalibrated, with none on six benchmarks. COCO-Stuff has four (floor-other, ground-other, solid-other, textile-other), and ADE20K has four (pool table, screen door, trade name, CRT screen), representing at most 2.7% of any vocabulary. The four COCO-Stuff classes are predicted on essentially no pixels. Insufficient calibration support causes only 0.03% of ADE20K pixels and 0.00% elsewhere to keep the original argmax prediction. Pixels also fall back when the two signature estimates select different classes, as reported separately in the table.

b) Variance, CIs, and seeds.

ActiveSAM is training-free and uses fixed cached signatures, with no random sampling during inference. Following your suggestion, we repeated the evaluation three times, obtaining 66.02 mean mIoU in each run.

Run 1Run 2Run 3Mean ± std
Mean mIoU (8 benchmarks)66.016266.016266.017566.02 ± 0.0007

The results show negligible variation across runs. We also compute 95% CIs using 2,000 paired bootstrap resamples per benchmark. ECD improves mean mIoU by +0.75 (95% CI [+0.57, +0.91]).

c) Cityscapes gain (+3.31 mIoU).

ECD changes only 0.32% of valid pixels, including 0.24% corrected from wrong to right and 0.05% changed from right to wrong. The main improvement comes from separating rider and person.

With ECD, 1.37M more pixels are predicted as rider and 1.37M fewer as person, with 1.21M of the added rider predictions correct. Correct rider pixels increase from 0.27M to 1.48M out of 1.97M ground-truth rider pixels. Rider IoU increases from 13.1 to 66.1 and person IoU from 73.5 to 80.0. Together, these classes explain +3.13 of the +3.31 mIoU gain, since each contributes 1/191/19 to Cityscapes mIoU. Wall adds +0.14, and no class loses more than 0.5 IoU. No Cityscapes class is uncalibrated.

d) Eq. 4 and active-set size.

Eq. 4 compares candidates on the same pixel and active coordinates, not distances across datasets. Writing the score for class kk as FkF_k, the difference between candidates kk and jj is

Fk−Fj=log⁡qkmkqjmj−12σ2∑c∈A(I)(Rkc−Rjc)(Rkc+Rjc−2Sc(p)).F_k-F_j=\log\frac{q_km_k}{q_jm_j}-\frac{1}{2\sigma^2}\sum_{c\in\mathcal A(I)}(R_{kc}-R_{jc})(R_{kc}+R_{jc}-2S_c(p)).

Additional classes affect the comparison through differences between the two signatures. Their measured mean differences are 0.20–0.89 on the candidates' own coordinates, versus 0.001–0.013 on other coordinates. The same σ\sigma is used across benchmarks.

To further validate this, we evaluate ECD on images with more than 20 active classes. ECD improves mIoU over argmax in every evaluated group: (N/A means the benchmark has no images in this range.)

Active classesContext60Context59COCO-ObjectCOCO-StuffADE20K
21–40+3.25+0.25+1.40+0.53+0.34
More than 40N/AN/AN/A+0.69+0.22

We have added all of these evaluations into Appendix J of the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer Ad1T [3/4]

W3 & Q3: The two most direct controls are missing. The paper's two directions each cover half of "presence filtering + per-pixel argmax" (Table 4's "−ECD" still includes CPE; Table 7's "without CPE" still includes ECD), and there is no speed–accuracy curve for thresholding SAM 3's native presence head directly. Without these, it cannot be shown that the 2.1-point gain is not simply split between the two parts. Please add "presence filtering + per-pixel argmax" and "CPE and ECD both removed", plus a native presence-threshold sweep, and report multi-run means and standard deviations for all key results.

Thanks for the detailed suggestion. Following it, we have evaluated the requested joint ablation and presence-threshold sweep, keeping the baseline's prompts and final decoding fixed.

a) Presence filtering + per-pixel argmax.

We evaluate the setting with CPE and ECD both removed. All mIoU results use the full validation sets. Without ECD, we use argmax and the published background thresholds. When every prompt is decoded, our implementation reproduces the official SegEarth-OV3 results within 0.08 mIoU on every benchmark.

EPGCPEECDV21P60C-OV20P59C-SCityADEMean
offoffoff (SegEarth-OV3, Table 1)79.853.472.096.859.242.869.737.663.9
onoffoff79.8054.1372.1196.7059.8042.7469.6837.7264.09
ononoff (Table 4, −ECD)83.2755.0572.2297.1060.4944.2669.7340.0065.27
onoff (Table 4, −CPE)on79.0256.0772.8496.7060.2543.4070.0938.8264.65
ononon (ActiveSAM)83.4256.5972.6397.0860.4444.9073.0440.0366.02

Presence filtering with argmax reaches 64.09 mean mIoU, compared with 63.9 for SegEarth-OV3. CPE adds +1.18 with ECD removed and +1.37 with ECD enabled. ECD adds +0.56 without CPE and +0.75 with CPE, with a +0.19 interaction. Both components improve accuracy even when the other is removed. We have added this evaluation to Appendix M of the revised manuscript.

b) Native presence-threshold sweep.

We apply SAM 3's presence score before full mask decoding, while keeping SegEarth-OV3's prompts, thresholds, and argmax unchanged, without CPE or ECD. This avoids decoding prompts that are filtered out, unlike thresholding only after full decoding.

τ\tauSegEarth-OV300.020.050.10.20.30.5
Mean mIoU63.963.9064.0564.0964.0963.8363.4962.45
Speed vs SegEarth-OV31.00×2.49×5.30×6.14×6.80×7.39×7.73×8.10×

At τ=0\tau=0, no prompts are filtered, and bucketing alone gives a 2.49× speedup. At τ=0.1\tau=0.1, selection and bucketing together give a 6.80× speedup, with 64.09 mIoU versus 63.9 for SegEarth-OV3. Higher thresholds increase speed but reduce accuracy. We have added this evaluation to Appendix P of the revised manuscript.

c) Repeated evaluations and uncertainty.

As reported in W2/Q2, three repeated evaluations of full ActiveSAM give 66.0162, 66.0162, and 66.0175 mean mIoU across eight benchmarks. We also report paired-bootstrap intervals for the component comparisons. CPE contributes +1.18 (95% CI [+0.99, +1.43]) with ECD removed. CPE and ECD together contribute +1.93 (95% CI [+1.67, +2.23]) relative to the configuration without either component, with a positive improvement on all eight benchmarks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their careful assessments, constructive comments, and valuable suggestions. Their feedback has helped us strengthen the experimental evaluation and improve the clarity of our paper. We have carefully revised the manuscript, with the new and modified content highlighted in blue. The main additions and clarifications are summarized below.

  1. Inference efficiency and matched comparisons. We clarified the inference-time calculations and the reported average speedups. We also added matched baseline comparisons and a native presence-threshold sweep, showing that prompt selection and bucketing alone achieve a 6.8× speedup with comparable accuracy to SegEarth-OV3 (Appendix P, Tables 21 and 22).

  2. Canonicalization and component contributions. We provided the complete canonicalization mapping and evaluated an automatic WordNet-based alternative, which achieves 66.11 mean mIoU across eight benchmarks, compared with 66.02 for the fixed mapping (Appendices I and L, Tables 13 and 17), showing that proposed method work well with both fix and automatic canonicalization. We also added joint ablations of CPE and ECD, confirming that both components contribute to accuracy (Appendix M, Table 18).

  3. ECD reliability and statistical analysis. We reported ECD coverage, repeated evaluations, confidence intervals, and an analysis of the Cityscapes improvement. Only 8 of 581 class entries lack sufficient calibration support, and ECD improves mean mIoU by 0.75 with a 95% confidence interval of [+0.57, +0.91]. We further examined ECD on images with larger active sets (Appendix J, Tables 14 and 15).

  4. Small-object retention and component-wise timing. We analyzed class retention for small components and long-tail categories across all eight benchmarks and measured the inference time of each component (Section 4.2, Tables 19 and 20).

  5. Calibration costs and expanded vocabularies. We measured the offline calibration cost and analyzed when the total inference-time savings recover this cost (Appendix K, Table 16). We also evaluated Context-459 and ADE20K-847, two larger vocabularies not used during development. Without ECD, ActiveSAM outperforms SegEarth-OV3 on both vocabularies using either fixed or automatic canonicalization. Even with ECD enabled, it achieves approximately 18x faster inference (Appendix Q, Table 23).

  6. Additional qualitative comparisons. We updated the original qualitative results with new visualizations covering all eight benchmarks, showing improvements in foreground segmentation, class separation, and scene labeling (Appendix R).

We hope these additional experiments, analyses, and clarifications address the reviewers' concerns and improve the quality and completeness of our work. We greatly appreciate the reviewers' time and thoughtful feedback. Detailed responses to each reviewer's comments are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.