- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes ActiveSAM, a training-free inference pipeline for applying frozen SAM 3 to open-vocabulary semantic segmentation. It combines prompt canonicalization and expansion, a low-resolution presence preview that filters prompts before full-resolution decoding, bucketed decoding, and a signature-based rule for resolving overlapping class responses. The method is evaluated on eight segmentation benchmarks and reports higher mean mIoU and substantially higher throughput than SAM 3 and SegEarth-OV3.
Critical strengths
(i) The empirical coverage is unusually broad for this setting. The paper evaluates eight benchmarks, includes matched SAM 3-based comparisons, separates the accuracy-oriented and efficiency-oriented components, and reports both clean and corrupted-image results. The EPG ablation is particularly useful: it shows that prompt filtering changes mean accuracy very little while materially reducing per-image computation.
(ii) The paper addresses a real inefficiency in concept-prompted segmentation. Reusing one image encoding and avoiding full decoding for most vocabulary entries is simple, but practically consequential when the vocabulary is large.
Critical weaknesses
(i) The deployment claim does not account for the method's per-vocabulary calibration stage. The full method estimates response signatures and class masses by running the pipeline over 9858 COCO images for every vocabulary. This cost is excluded from the reported FPS numbers. It is therefore unclear when the claimed speedup is actually realized, especially in the motivating setting where vocabularies are dynamic or used for only a modest number of images. More importantly, the evaluation covers eight fixed benchmark vocabularies; it does not test vocabulary edits, rare concepts, or vocabularies whose classes have poor support in COCO. The paper currently establishes fast amortized inference after substantial offline calibration, not efficient segmentation under a genuinely changing open vocabulary.
(ii) The accuracy contribution is much less clean than the headline comparison suggests. EPG is essentially accuracy-neutral, ECD adds only 0.7 mean mIoU and is concentrated on Cityscapes, while the largest CPE contribution comes from an unspecified hand-written lexical canonicalizer. The manuscript gives only an example and says that fixed string rules are used; it does not list the rules or establish that they were designed independently of the benchmark labels and errors. Since the total gain over SegEarth-OV3 is 2.1 points, benchmark-aware name rewriting and different post-processing can plausibly explain a meaningful fraction of the margin. A matched comparison that adds the same active-prompt/bucketing machinery to the closest SAM 3 baseline, while holding prompt rewriting and final decoding fixed, is needed to isolate the contribution.
Questions for authors
(i) What are the wall-clock time and GPU cost of estimating signatures from the 2k and 10k pools for each vocabulary? Please report end-to-end or amortized cost as a function of the number of inference images and the frequency of vocabulary changes.
(ii) Can the authors provide the complete canonicalization rules and results on previously unseen, long-tail, or incrementally edited vocabularies? Were any rules introduced after examining benchmark-specific failures?
(iii) Under identical prompts and final decoding, how much speed and accuracy are obtained by applying only the proposed active-prompt selection and bucketing to SAM 3 or SegEarth-OV3?
4 comments
Responses to Reviewer oBwY [2/3]
W2: The accuracy contribution is much less clean than the headline comparison suggests. EPG is essentially accuracy-neutral, ECD adds only 0.7 mean mIoU and is concentrated on Cityscapes, while the largest CPE contribution comes from an unspecified hand-written lexical canonicalizer. The manuscript gives only an example and says that fixed string rules are used; it does not list the rules or establish that they were designed independently of the benchmark labels and errors. Since the total gain over SegEarth-OV3 is 2.1 points, benchmark-aware name rewriting and different post-processing can plausibly explain a meaningful fraction of the margin. A matched comparison that adds the same active-prompt/bucketing machinery to the closest SAM 3 baseline, while holding prompt rewriting and final decoding fixed, is needed to isolate the contribution.
Thanks for your detailed suggestion. Following it, we have evaluated the matched comparison and also tested whether the canonicalization benefit depends on the fixed mapping.
| Configuration | Mean mIoU |
|---|---|
| SegEarth-OV3 (Table 1) | 63.9 |
| + prompt selection and bucketing (same prompts, thresholds, and argmax) | 64.09 |
| + canonical names | 65.00 |
| + context tokens (complete CPE, still using baseline decoding) | 65.28 |
| + ECD (full ActiveSAM) | 66.02 |
Prompt selection and bucketing alone give 6.8× faster inference with comparable accuracy (64.09 versus 63.9). CPE then improves mIoU to 65.28 with the baseline decoding unchanged, and ECD further improves it to 66.02.
We also test a WordNet-based alternative that automatically splits compound names, separates hyphenated names, and standardizes spelling without any manually defined per-name mappings. With context tokens enabled and ECD disabled, mean mIoU is 64.53 without canonicalization, 65.28 with the fixed mapping, and 65.39 with automatic canonicalization. With full ActiveSAM and re-estimated signatures, the automatic version reaches 66.11 mIoU, compared with 66.02 for the original mapping. This shows that the gain does not depend on the particular predefined mappings.
We have added the complete mapping rule and the automatic procedure in Appendix I.
Q1: What are the wall-clock time and GPU cost of estimating signatures from the 2k and 10k pools for each vocabulary? Please report end-to-end or amortized cost as a function of the number of inference images and the frequency of vocabulary changes.
Thanks for your question. Following your comment, we have measured the one-time cost of signature estimation separately from online FPS. The measured times on one RTX 5090 are shown below.
| V21 | P60 | C-O | V20 | P59 | C-S | City | ADE | |
|---|---|---|---|---|---|---|---|---|
| 2k pool, wall-clock (min) | 4.7 | 7.1 | 6.0 | 4.7 | 6.9 | 9.8 | 5.3 | 9.7 |
| 10k pool, wall-clock (min) | 20.7 | 32.8 | 26.5 | 20.3 | 31.6 | 47.7 | 22.9 | 46.1 |
Calibration is performed only once per vocabulary, and the resulting signatures are cached for subsequent inference.
For the question on total cost and vocabulary changes, let denote the one-time calibration cost, and and denote the per-image inference times of ActiveSAM and SegEarth-OV3. For a vocabulary used on images, the total and average costs are
Including calibration, ActiveSAM becomes faster than SegEarth-OV3 when
When vocabularies change, the calibration cost is incurred once for each new vocabulary. Based on our measurements, the full method recovers its calibration cost after 221–1,184 images with the 2k pool, or approximately 1,100–5,300 images with the 10k pool. For shorter workloads, ActiveSAM can run without ECD or calibration, achieving 65.27 mean mIoU versus 63.9 for SegEarth-OV3. We have added this analysis to Appendix K in the revised manuscript.
Responses to Reviewer oBwY [3/3]
Q2: Can the authors provide the complete canonicalization rules and results on previously unseen, long-tail, or incrementally edited vocabularies? Were any rules introduced after examining benchmark-specific failures?
Thanks for your question. We have added the complete canonicalization mapping (and WordNet version) into Appendix I of the revised manuscript.
For previously unseen and expanded vocabularies, W1 reports our new experiments on Context-459 and ADE20K-847, extending the original vocabularies from 59 to 459 and from 150 to 847 classes. We keep the original mapping and hyperparameters unchanged. Without ECD, ActiveSAM achieves 18.97 and 14.80 mIoU, compared with 18.78 and 14.47 for SegEarth-OV3.
Regarding the canonicalization rule, no mapping entries were introduced after examining benchmark-specific failures. To further test whether the improvement depends on the predefined mappings, we evaluate an automatic WordNet-based alternative without manually defined name entries. With full ActiveSAM, it achieves 66.11 mIoU, compared with 66.02 for the original mapping. This shows that the improvement does not depend on the particular predefined entries. We have added this evaluation in Appendix L in the revised manuscript.
Q3: Under identical prompts and final decoding, how much speed and accuracy are obtained by applying only the proposed active-prompt selection and bucketing to SAM 3 or SegEarth-OV3?
Thanks for your question. We have evaluated the setting when only prompt selection and bucketing to SegEarth-OV3, keeping its prompts, thresholds, and argmax decoding unchanged. The results are shown below.
| V21 | P60 | C-O | V20 | P59 | C-S | City | ADE | Mean | |
|---|---|---|---|---|---|---|---|---|---|
| SegEarth-OV3, mIoU (Table 1) | 79.8 | 53.4 | 72.0 | 96.8 | 59.2 | 42.8 | 69.7 | 37.6 | 63.9 |
| + prompt selection and bucketing, mIoU | 79.88 | 54.13 | 72.23 | 96.58 | 59.80 | 42.66 | 69.68 | 37.72 | 64.09 |
| Speed vs SegEarth-OV3 | 3.9× | 6.5× | 9.6× | 4.9× | 6.5× | 11.4× | 2.2× | 9.4× | 6.8× |
Prompt selection and bucketing alone give 6.8x faster inference with comparable accuracy (64.09 versus 63.9 mean mIoU). The speedup reaches 11.4× on COCO-Stuff. This comparison isolates the efficiency benefit without changing the baseline's prompts or final decoding. We have added this evaluation to Appendix P in the revised manuscript.
Follow-up to Reviewer oBwY
We thank the reviewer again for the careful assessment and constructive feedback.
Following your suggestions, we have added detailed analyses of calibration costs, including their impact on total inference time and a calibration-free alternative. We have also evaluated two larger vocabularies with long-tail classes and conducted matched baseline comparisons and joint ablations to isolate the contributions of prompt selection and bucketing, CPE, and ECD. In addition, we provide the complete canonicalization mapping, clarify that no rules were added after examining benchmark-specific failures, and evaluate automatic WordNet-based canonicalization. All of these experiments and analyses are included in the revised manuscript.
We hope our response and additional experiments address your concerns. We would welcome any further feedback or questions you may have.
Responses to Reviewer oBwY [1/3]
We thank the reviewer for the valuable feedback and comments, which have definitely helped us improve the quality of the paper. We have carefully updated the revised manuscript to include all the mentioned suggestions and changes. We further clarify each of your questions in the review.
Thanks for the comment and suggestion. We provide detailed clarifications for each point below.
a) The offline calibration is used only for ECD. Without ECD, ActiveSAM already achieves 65.27 mean mIoU, compared with 63.9 for SegEarth-OV3. For the full method, signatures are estimated once per vocabulary and cached for subsequent images. We report this preparation cost separately from online FPS and include it when evaluating total inference time.
When reducing the calibration pool from 10k to 2k images, mean mIoU changes only from 66.02 to 66.00 (Table 5). The 2k pool takes 4.7–9.8 minutes per vocabulary on one RTX 5090 (Table 16 gives the details for both the 10k and 2k pools). Including this cost, full ActiveSAM becomes faster than SegEarth-OV3 after 221–1,184 images, depending on the vocabulary. Q1 reports the per-vocabulary costs and calculation. For shorter workloads, ActiveSAM can run without ECD or signature estimation.
b) Following your suggestion on testing edits, rare concepts, or vocabularies whose classes have poor support in COCO, we have evaluated Context-459 and ADE20K-847, expanding Context59 from 59 to 459 classes and ADE20K from 150 to 847 classes on the same validation images. Neither expanded vocabulary was used during development. We keep the original hyperparameters and canonicalization mapping unchanged. That mapping applies to only 7 of 459 Context-459 names and 14 of 1,281 ADE20K-847 names, including aliases.
We have also tested an automatic canonicalization (WordNet-based) that requires no manually defined name mappings, described in Q2 and more details in Appendix I of the revised manuscript.
ActiveSAM decodes only 38.9 of 459 classes and 113.0 of 847 classes per image, while both configurations without ECD achieve comparable mIoU to SegEarth-OV3. The measured online speedups are 18.3x and 18.0x faster than SegEarth-OV3.
On these expanded vocabularies, 201 of 459 and 452 of 847 classes are flagged uncalibrated (Classes with limited COCO support). The 2k calibration pass takes 16 and 39 minutes, respectively. We also evaluate the effect of enabling ECD under both canonicalization methods:
Prompt selection remains effective without calibration on both expanded vocabularies. ECD gives lower accuracy here, where many classes have insufficient support in the COCO calibration pool. For these settings, using ActiveSAM without ECD gives the better result. We have added this analysis into Appendix Q of the revised manuscript.