- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SHAPE, a training-free open-vocabulary semantic segmentation pipeline built from three stages. The Edge expert transfers DINO structure into CLIP feature aggregation, largely following CASS; the Node expert averages class scores over DINO-feature nearest neighbors; and the Region expert performs agreement-gated score sharing within spectral groups. Results are reported on eight benchmarks with several CLIP backbones, together with custom patch- and region-consistency diagnostics.
Critical strengths
(i) The paper studies a concrete failure mode of dense CLIP prediction and makes a reasonable distinction between isolated patch errors and region fragmentation. The GT-defined diagnostics are more informative than relying on mIoU alone.
(ii) The authors evaluate eight datasets and three CLIP backbones, and the Node/Region composition ablation is directionally consistent across the reported benchmarks.
Critical weaknesses
(i) The main comparison is not credible without a tuning protocol. Appendix Table 5 reveals a separate triplet of text/fusion hyperparameters for every benchmark, and the spectral rescaling is also changed specifically for VOC20 and Cityscapes. The paper never states what data or criterion selected these values. This is decisive because the claimed gains over the strongest baseline are only 0.1-1.3 mIoU without background and 0.2-0.9 with background. Such margins are easily explained by per-dataset tuning, particularly when the baselines are quoted from prior work rather than retuned with an equivalent search budget. If validation or test labels informed these settings, the headline comparison is invalid; if they did not, the manuscript needs to describe the held-out selection procedure and report its variance.
(ii) The incremental contribution beyond CASS is small and is not compared against sufficiently matched refinements. The Edge expert adopts CASS's head matching, spectral rescaling, and text refinement. The new pieces are essentially one-step kNN score averaging and agreement-gated regional averaging, which improve the common Edge output by 1.0 mIoU on average. There is no matched CASS+generic smoothing/graph propagation/CRF baseline using the same multi-scale inference, text settings, DINO features, and background calibration. The paper also omits runtime and memory despite adding overlapping multi-scale crops, a second backbone, nearest-neighbor search, eigendecomposition, and clustering. As written, it is impossible to tell whether the proposed hierarchy is better than straightforward post-processing or whether its small gains justify the added machinery.
Questions for authors
(i) Exactly how were the per-dataset values in Appendix Table 5 and the dataset-specific spectral mapping selected? Did any ground-truth labels from the reported evaluation splits enter this process? Please report results with one shared configuration or with a clearly separated validation protocol.
(ii) Can the authors compare against CASS augmented with standard, matched score refinements such as spatial/VFM kNN smoothing, graph diffusion, or a dense CRF, all using the same text prompts, scales, and background threshold?
(iii) What are the end-to-end latency, peak memory, and FLOPs relative to CASS and the strongest training-free baselines?
To Reviewer #shYi:
We thank the reviewer for the careful reading and for recognizing the distinction between isolated patch errors and region fragmentation and the value of the GT-defined diagnostics. We ran all requested experiments on the full validation splits and address each point below.
W1 / Q1: How were the per-dataset values in Appendix Table 5 and the spectral mapping selected? Did evaluation labels enter this process?
No evaluation labels were used. For the dataset-specific values in Table 5 (λtext, λglobal, htext), the spectral mapping on VOC20/Cityscapes, and the Edge parameters rE=30 and λfuse=10, we followed the official CASS configuration, without any additional search. Our comparison with CASS therefore uses identical dataset-specific settings. The parameters introduced by SHAPE (K, αN, M, αR, τR) are shared across datasets, and Table 9 shows that they are not per-dataset optima.
As requested, we also report one shared configuration for all datasets (λtext = 0.03, λglobal = 0.25, htext = 0.10, rescaling everywhere), with CASS re-run under exactly the same configuration and scales. Under both the dataset-specific and the shared configuration, SHAPE outperforms CASS on all eight datasets.
W2 / Q2: Matched CASS + generic refinements.
As requested, we additionally applied each refinement and our Node → Region to the CASS output (before any post-processing), with identical prompts, scales {1.0,1.5}, background threshold, DINO features, and logit scale.
Cost. On a single RTX 3080 (batch size 1, scales {1.0,1.5}), SHAPE requires 5.4 / 26.0 TFLOPs per image on ADE20K / Cityscapes (CASS: 4.3 / 6.9) with peak memory 2.2 / 2.7 GB (CASS: 2.9 / 1.2). The components that SHAPE adds on top of the Edge expert (kNN, eigendecomposition, k-means, gated sharing) take only 0.12 / 2.15 s per image.
We hope these experiments and the clarified protocol address your concerns, and we welcome any further questions.