acceptodds
Under review as a conference paper at ICLR 2027

SHAPE: Structural Hierarchy of Affinities for Patch Enhancement in Training-Free Open-Vocabulary Semantic Segmentation

Abstract

Training-free Open-Vocabulary Semantic Segmentation (OVSS) assigns text-specified class labels to image pixels without additional training. In this setting, frozen CLIP provides visual–text alignment, while a vision foundation model (VFM) supplies structural cues for localization. Despite this, predictions still suffer from patch noise, where isolated patches disagree with surrounding context, and region noise, where coherent object regions fragment. To address these issues, we formulate structural refinement as support selection: the VFM determines which patches contribute to each update, while CLIP remains the sole source of class evidence. We propose Structural Hierarchy of Affinities for Patch Enhancement (SHAPE), which introduces three complementary experts at different structural levels—Edge, Node, and Region—corresponding to relation, patch, and region support, respectively. The Edge Expert leverages VFM-derived patch affinities as relation-level support to aggregate CLIP visual features. The Node Expert reduces patch noise by refining each patch's class scores using its patch-specific VFM neighbors. Finally, the Region Expert reduces residual region noise by sharing class scores within coherent regions among patches whose predictions agree. Across eight benchmarks with a ViT-B/16 backbone, SHAPE achieves the best mIoU among weakly supervised and training-free methods on every dataset, reaching average scores of 44.0 and 47.9 without and with background classes, respectively. Code will be made publicly available upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Semantic Segmentation, Training-Free Learning, Vision-Language Models, Vision Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_shYiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SHAPE, a training-free open-vocabulary semantic segmentation pipeline built from three stages. The Edge expert transfers DINO structure into CLIP feature aggregation, largely following CASS; the Node expert averages class scores over DINO-feature nearest neighbors; and the Region expert performs agreement-gated score sharing within spectral groups. Results are reported on eight benchmarks with several CLIP backbones, together with custom patch- and region-consistency diagnostics.

Critical strengths

(i) The paper studies a concrete failure mode of dense CLIP prediction and makes a reasonable distinction between isolated patch errors and region fragmentation. The GT-defined diagnostics are more informative than relying on mIoU alone.

(ii) The authors evaluate eight datasets and three CLIP backbones, and the Node/Region composition ablation is directionally consistent across the reported benchmarks.

Critical weaknesses

(i) The main comparison is not credible without a tuning protocol. Appendix Table 5 reveals a separate triplet of text/fusion hyperparameters for every benchmark, and the spectral rescaling is also changed specifically for VOC20 and Cityscapes. The paper never states what data or criterion selected these values. This is decisive because the claimed gains over the strongest baseline are only 0.1-1.3 mIoU without background and 0.2-0.9 with background. Such margins are easily explained by per-dataset tuning, particularly when the baselines are quoted from prior work rather than retuned with an equivalent search budget. If validation or test labels informed these settings, the headline comparison is invalid; if they did not, the manuscript needs to describe the held-out selection procedure and report its variance.

(ii) The incremental contribution beyond CASS is small and is not compared against sufficiently matched refinements. The Edge expert adopts CASS's head matching, spectral rescaling, and text refinement. The new pieces are essentially one-step kNN score averaging and agreement-gated regional averaging, which improve the common Edge output by 1.0 mIoU on average. There is no matched CASS+generic smoothing/graph propagation/CRF baseline using the same multi-scale inference, text settings, DINO features, and background calibration. The paper also omits runtime and memory despite adding overlapping multi-scale crops, a second backbone, nearest-neighbor search, eigendecomposition, and clustering. As written, it is impossible to tell whether the proposed hierarchy is better than straightforward post-processing or whether its small gains justify the added machinery.

Questions for authors

(i) Exactly how were the per-dataset values in Appendix Table 5 and the dataset-specific spectral mapping selected? Did any ground-truth labels from the reported evaluation splits enter this process? Please report results with one shared configuration or with a clearly separated validation protocol.

(ii) Can the authors compare against CASS augmented with standard, matched score refinements such as spatial/VFM kNN smoothing, graph diffusion, or a dense CRF, all using the same text prompts, scales, and background threshold?

(iii) What are the end-to-end latency, peak memory, and FLOPs relative to CASS and the strongest training-free baselines?

1 comment

To Reviewer #shYi:

We thank the reviewer for the careful reading and for recognizing the distinction between isolated patch errors and region fragmentation and the value of the GT-defined diagnostics. We ran all requested experiments on the full validation splits and address each point below.

W1 / Q1: How were the per-dataset values in Appendix Table 5 and the spectral mapping selected? Did evaluation labels enter this process?

No evaluation labels were used. For the dataset-specific values in Table 5 (λtext\lambda_{\text{text}}, λglobal\lambda_{\text{global}}, htexth_{\text{text}}), the spectral mapping on VOC20/Cityscapes, and the Edge parameters rE=30r_E = 30 and λfuse=10\lambda_{\text{fuse}} = 10, we followed the official CASS configuration, without any additional search. Our comparison with CASS therefore uses identical dataset-specific settings. The parameters introduced by SHAPE (KK, αN\alpha_N, MM, αR\alpha_R, τR\tau_R) are shared across datasets, and Table 9 shows that they are not per-dataset optima.

As requested, we also report one shared configuration for all datasets (λtext\lambda_{\text{text}} = 0.03, λglobal\lambda_{\text{global}} = 0.25, htexth_{\text{text}} = 0.10, rescaling everywhere), with CASS re-run under exactly the same configuration and scales. Under both the dataset-specific and the shared configuration, SHAPE outperforms CASS on all eight datasets.

W2 / Q2: Matched CASS + generic refinements.

As requested, we additionally applied each refinement and our Node →\rightarrow Region to the CASS output (before any post-processing), with identical prompts, scales {1.0,1.5}\{1.0, 1.5\}, background threshold, DINO features, and logit scale.

Refinement on CASSVOC20VOC21PC59PC60CityADEC-ObjC-StfAvg
None (CASS)86.1965.4740.8638.1141.0320.8337.7326.2444.56
Gaussian (σ=1\sigma = 1)86.4164.0939.5836.8738.1319.9136.8925.6243.44
Dense CRF84.1262.1238.4033.9636.9019.2134.8424.7641.79
PAMR85.6565.4540.7137.9440.8620.7537.7826.3944.44
VFM kNN ×\times 186.4866.2441.1938.4041.4520.9838.1926.5344.93
VFM kNN ×\times 386.5266.8141.3338.5041.4120.9238.3726.7145.07
VFM kNN ×\times 586.3966.9441.3138.4441.1320.7838.3326.7645.01
Diffusion (a=0.8a = 0.8)87.2867.0541.3638.4741.1020.9638.7826.8145.23
Diffusion (a=0.95a = 0.95)87.2167.0840.8237.7138.5020.2938.3026.5844.56
Node →\rightarrow Region86.8966.9041.4238.6241.6821.0338.5326.7345.23
  1. Generic post-processing does not help. Dense CRF lowers mIoU on every dataset, Gaussian smoothing on seven of eight, and PAMR is neutral to negative; RGB-boundary refinement cannot correct the semantic errors of the coarse CLIP score map. Appending CRF or PAMR to our refinement also lowers mIoU (ADE20K 21.03 →\rightarrow 19.00 / 20.81), so SHAPE uses neither.
  2. The gain comes from VFM structure: only refinements on the VFM graph improve the CASS average.
  3. Node →\rightarrow Region achieves the highest average (45.23, matched only by diffusion at a hand-picked a=0.8a = 0.8) and the best results on PC59, PC60, Cityscapes, and ADE20K. With one pass and one fixed setting, it improves all eight datasets with the largest worst-case gain.
  4. In a paired bootstrap over images (1,000 resamples), Node →\rightarrow Region improves over CASS on all eight datasets (p<0.001p < 0.001) and over kNN ×\times 3 on seven of eight (p<0.05p < 0.05).

Cost. On a single RTX 3080 (batch size 1, scales {1.0,1.5}\{1.0, 1.5\}), SHAPE requires 5.4 / 26.0 TFLOPs per image on ADE20K / Cityscapes (CASS: 4.3 / 6.9) with peak memory 2.2 / 2.7 GB (CASS: 2.9 / 1.2). The components that SHAPE adds on top of the Edge expert (kNN, eigendecomposition, k-means, gated sharing) take only 0.12 / 2.15 s per image.

We hope these experiments and the clarified protocol address your concerns, and we welcome any further questions.

Reviewer_wmuzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of patch noise and region noise existing in feature optimization using the frozen CLIP + VFM scheme for training-free open-vocabulary semantic segmentation (OVSS). It proposes the Structural Hierarchy of Affinities for Patch Enhancement (SHAPE) method. An Edge expert is designed to guide CLIP feature aggregation and reduce cross-class feature leakage; a Node expert corrects the scores of isolated blocks to mitigate block noise; a Region expert is used to reduce region noise and prevent the propagation of cross-class errors. The effectiveness of the method is verified on 8 benchmarks and achieves state-of-the-art results.

Critical strengths
  1. The paper is clearly structured and the motivation is sound. Three complementary experts are designed at the Edge, Node and Region levels to tackle feature-level leakage, patch-level misclassification and region-level fragmentation, respectively. The resulting pipeline achieves progressive refinement from feature aggregation to score calibration.
  2. The proposed method achieves consistently strong results across multiple benchmark datasets, validating the effectiveness and generalization of the proposed framework.
Critical weaknesses
  1. The overall performance gains of the paper are relatively small (+1.1 without background mean, +0.8 with background). The marginal improvements on some datasets (VOC20 +0.1, COCO-Object +0.2) weaken the contributions of the paper.
  2. The paper involves many hyper-parameters(Table-5). Although Table-9 provides three coverage analyses, the sensitivity analysis is still insufficient. It is necessary to illustrate that the performance improvement does not rely on dataset-wise parameter tuning.
Questions for authors
  1. The overall performance gains of the paper are relatively small (+1.1 without background mean, +0.8 with background). The marginal improvements on some datasets (VOC20 +0.1, COCO-Object +0.2) weaken the contributions of the paper.
  2. The paper involves many hyper-parameters(Table-5). Although Table-9 provides three coverage analyses, the sensitivity analysis is still insufficient. It is necessary to illustrate that the performance improvement does not rely on dataset-wise parameter tuning.
  3. The Edge expert is directly built based on CASS spectral distillation, and its contribution needs to be further quantified.
  4. The paper lacks descriptions of relevant computational overhead, including the spectral decomposition of the affinity matrix, k-means clustering, and the full pipeline of the three experts.
2 comments

To Reviewer #wmuz:

We thank the reviewer for appreciating the clear structure, the complementary design of the three experts, and the consistently strong results. We address each point below.

W1 / Q1: The overall gains are relatively small (VOC20 +0.1, COCO-Object +0.2).

These margins are measured against the best baseline for each dataset, which is a different method on each benchmark (CASS on VOC20, GLA-CLIP on COCO-Object).

  1. Context. In Table 1, consecutive training-free state-of-the-art methods improve the average by +0.6 per step (PEARL 41.7 →\rightarrow GLA-CLIP 42.3 →\rightarrow CASS 42.9), whereas SHAPE adds +1.1 (44.0); in Table 2, SHAPE is best on all three benchmarks (47.9 vs. 47.1).
  2. Consistency. With identical hyperparameters and scales, SHAPE outperforms CASS on all eight datasets under both the dataset-specific and a shared configuration (W2), and Node and Region improve every dataset by +0.38 to +1.86.
  3. Where the gains concentrate. Against CASS, the gains reach +2.5 on Cityscapes and VOC21, where patch and region noise are most frequent; VOC20 is close to saturation (87.8 →\rightarrow 87.9).
  4. Beyond mIoU. On Cityscapes, region fragmentation drops from 5.86 to 4.98 and patch deviation from 0.032 to 0.024 (Table 4).

W2 / Q2: Many hyperparameters; dependence on dataset-wise tuning.

For the dataset-specific values in Appendix Table 5, including the spectral mapping on VOC20/Cityscapes, we followed the official CASS configuration, without any additional search and without using evaluation labels. All parameters introduced by SHAPE are shared across datasets. We also re-ran SHAPE and CASS with one configuration for all eight datasets:

SettingMethodScalesVOC20VOC21PC59PC60CityADEC-ObjC-StfAvg
Dataset-specificCASS (reported)1.087.864.440.137.939.620.336.626.744.18
SHAPE1.088.0965.2340.3437.9339.6720.5136.6627.0944.44
SharedCASS1.0, 1.586.1965.4740.8638.1141.0320.8337.7326.2444.56
SHAPE1.0, 1.587.0965.5141.0038.2241.0420.9437.9726.5244.79

In addition to Table 9, the comparison in our response to Reviewer QsLV (W1) shows that Node →\rightarrow Region reaches its results with one fixed setting, without any per-dataset sweep.


Q3: Quantify the contribution of the Edge expert beyond CASS.

The Edge expert builds on the spectral transfer used in CASS; what distinguishes SHAPE from CASS is its results:

  • Accuracy. SHAPE outperforms CASS on all eight datasets under both the dataset-specific and the shared configuration (table under W2), with gains of up to +2.5 on Cityscapes and VOC21 in the main results.
  • Significance. In a paired bootstrap over images (1,000 resamples), applying Node →\rightarrow Region to the CASS output improves it significantly on all eight datasets (p<0.001p < 0.001).
  • Segmentation quality. Node and Region improve all eight datasets by +0.38 to +1.86 on top of the Edge expert and reduce region fragmentation on Cityscapes from 5.86 to 4.98 (Table 4).
  • Edge alone. With our two-scale inference, the Edge expert alone already exceeds the reported CASS results on seven of eight datasets (Table 7, e.g., Cityscapes 40.9 vs. 39.6), before Node and Region add a further +1.0 on average.

Q4: Computational overhead.

On a single RTX 3080 (batch size 1, scales {1.0,1.5}\{1.0, 1.5\}), SHAPE requires 5.4 / 26.0 TFLOPs per image on ADE20K / Cityscapes (CASS: 4.3 / 6.9) with peak memory 2.2 / 2.7 GB (CASS: 2.9 / 1.2). Of the 7.1 / 33.2 s per image, the Edge expert with sliding-window inference takes 7.0 / 31.1 s, while Node and Region together add only 0.12 / 2.15 s, including spectral grouping with an 8-dimensional embedding and k-means with M=16M = 16 per scale.

We welcome any further comments, questions, and discussions from you.

Thank you for the response and the additional experiments, which clarify the shared-hyperparameter behavior, noise metrics, and runtime overhead. However, the accuracy gains on several datasets remain modest, the multi-scale inference brings high computational overhead, and the standalone contribution of the Edge expert beyond CASS is still mixed with multi-scale aggregation. Taking these into account, I will maintain my original score.

Reviewer_QsLVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SHAPE, a training-free framework for open-vocabulary semantic segmentation that aims to reduce patch noise and region noise in CLIP predictions. The method uses frozen CLIP features for semantic classification and a vision foundation model (VFM) to provide structural guidance. SHAPE consists of three sequential experts: the Edge Expert uses VFM affinities to guide CLIP feature aggregation, the Node Expert refines patch-level class scores through VFM-based nearest neighbors, and the Region Expert improves regional consistency using spectral clustering and agreement-gated score sharing. Experiments on eight benchmark settings show consistent improvements over existing training-free methods, with additional analyses demonstrating reductions in patch-level inconsistency and region-level fragmentation.

Critical strengths

Clear motivation and problem formulation. The paper identifies two common issues in training-free OVSS, namely patch noise and region noise. The distinction between these two types of errors is intuitive, and using VFM structural information to refine CLIP predictions is a reasonable approach.

Consistent performance improvements across benchmarks. SHAPE achieves the best mIoU across all eight evaluated benchmark settings. The consistent improvements across different datasets and CLIP backbones support the effectiveness of the proposed refinement strategy.

Detailed ablation studies and noise analysis. The paper provides ablations of the Node and Region Experts and introduces GT-defined noise metrics to evaluate patch-level and region-level consistency. These experiments help explain how the individual components affect the segmentation results.

Critical weaknesses

The overall method design appears incremental. The Edge Expert largely builds upon CASS, while the Node and Region Experts perform score refinement through nearest-neighbor averaging and spectral grouping, respectively. Although the individual components are reasonable, their combination appears to be a sequence of relatively straightforward refinement operations. The paper does not sufficiently establish why these three separate experts are necessary, or whether a simpler unified affinity-based refinement could achieve similar results.

The performance gains are relatively small compared with the additional complexity. Although SHAPE achieves the best results across all eight benchmarks, the improvements over the strongest baselines are relatively modest, ranging from 0.1 to 1.3 mIoU. Meanwhile, the method involves two foundation models, multi-scale inference, nearest-neighbor retrieval, and spectral clustering. The paper does not provide runtime or memory comparisons to justify this additional computational cost.

The robustness of the method under different settings is not sufficiently established. The method uses several dataset-specific hyperparameters, particularly for text refinement and global-local score fusion. While this does not contradict the training-free setting, it raises questions about how much tuning is needed when applying SHAPE to a new dataset. The paper would benefit from a clearer explanation of the hyperparameter selection protocol and an evaluation using a shared configuration across datasets.

Questions for authors

What is the key advantage of using separate Node and Region Experts rather than a single affinity-based score refinement method? For example, could repeated neighborhood-based score propagation achieve comparable results without introducing spectral clustering?

Could the authors report the inference time and GPU memory usage of SHAPE compared with CASS and other strong training-free baselines? Given the relatively small mIoU improvements, understanding the additional computational cost would help assess the practical value of the method.

The Region Expert uses prediction agreement to determine whether patches within a spectral group should share their scores. However, high agreement does not necessarily imply correct predictions. How does the method handle groups where most patches consistently predict an incorrect class, and could the refinement reinforce such errors?

2 comments

To Reviewer #QsLV:

We thank the reviewer for recognizing our clear motivation, the consistent improvements across benchmarks and backbones, and the GT-defined noise analysis. We address each point below; all new experiments use the full validation splits.

W1 / Q1: Why three experts? Could a single repeated propagation achieve comparable results?

The experts act on different objects, and each consumes the previous output. Edge acts on features inside CLIP attention. Node corrects isolated patch errors (patch noise) with each patch's KK nearest VFM neighbors, a local operation that cannot repair an error covering a whole region. Region corrects fragmented regions (region noise) by gated score sharing within VFM spectral groups; its gate reads the group majority, so Node must first remove outliers. On Cityscapes, SHAPE reduces both noise types (Table 4): patch deviation 0.032 →\rightarrow 0.024, region fragmentation 5.86 →\rightarrow 4.98.

We additionally applied repeated kNN propagation and graph diffusion to the Edge expert output under identical settings (one shared configuration, see W3):

RefinementCityADEFixed setting
None39.9120.56–
Node only (kNN ×\times 1)40.6220.83✓
Region only40.5420.75✓
kNN ×\times 3 / ×\times 540.90 / 40.8120.91 / 20.84✗
Diffusion, a=0.8/0.95a = 0.8 / 0.9540.37 / 38.1520.86 / 20.39✗
Node →\rightarrow Region (ours)41.0420.94✓
  1. Node and Region are complementary: their combination outperforms either one alone (Cityscapes 40.62 / 40.54 →\rightarrow 41.04; ADE20K 20.83 / 20.75 →\rightarrow 20.94). Region also improves repeated propagation (kNN ×\times 3 →\rightarrow Region: Cityscapes 40.90 →\rightarrow 41.05; ADE20K 20.91 →\rightarrow 20.95), so its group-level correction is not reproduced by more propagation.
  2. With one fixed setting and no sweep, Node →\rightarrow Region achieves these results (+1.13 and +0.38 over the unrefined output).

W2 / Q2: Small gains relative to complexity; runtime and memory.

In Table 1, consecutive training-free state-of-the-art methods improve the average by +0.6 per step (PEARL 41.7 →\rightarrow GLA-CLIP 42.3 →\rightarrow CASS 42.9), whereas SHAPE adds +1.1 (44.0). In Table 2, SHAPE is best on all three benchmarks (47.9 vs. 47.1 for GLA-CLIP). The +0.1 (VOC20) and +0.2 (COCO-Object) margins are against a different best baseline on each dataset; against CASS, the gains reach +2.5 on Cityscapes and VOC21, where patch and region noise are most frequent.

On a single RTX 3080 (batch size 1, scales {1.0,1.5}\{1.0, 1.5\}), SHAPE requires 5.4 / 26.0 TFLOPs per image on ADE20K / Cityscapes (CASS: 4.3 / 6.9) with peak memory 2.2 / 2.7 GB (CASS: 2.9 / 1.2), and takes 7.1 / 33.2 s per image. Node and Region add only 0.12 / 2.15 s and negligible memory; most of the runtime is spent in the per-window processing of the Edge expert.

W3: Hyperparameter selection; a shared configuration.

For the dataset-specific values in Appendix Table 5 (λtext\lambda_{\text{text}}, λglobal\lambda_{\text{global}}, htexth_{\text{text}}, and the spectral mapping on VOC20/Cityscapes), we followed the official CASS configuration, without any additional search and without using evaluation labels. The parameters introduced by SHAPE (KK, αN\alpha_N, MM, αR\alpha_R, τR\tau_R) are shared across all datasets, and Table 9 shows that they are not per-dataset optima. We also ran SHAPE and CASS with one configuration for all datasets (λtext\lambda_{\text{text}} = 0.03, λglobal\lambda_{\text{global}} = 0.25, htexth_{\text{text}} = 0.10, rescaling everywhere):

Setting (scales)CASS (Avg)SHAPE (Avg)
Dataset-specific (1.0)44.1844.44
Shared (1.0, 1.5)44.5644.79

In the shared configuration, Node and Region improve every dataset by +0.38 to +1.86, so the gains do not depend on dataset-wise settings.


Q3: Can the agreement gate reinforce consistently wrong predictions?

For every spectral group (both scales, after Node), we compared the group consensus with the majority GT label of its patches and counted the patches that Region flips.

  1. Blocked groups are 1.6$\times$ to 5.2$\times$ more often wrong than passed groups on all eight datasets (e.g., Cityscapes 72.5% vs. 15.6%).
  2. Region fixes 1.6$\times$ to 2.4$\times$ more patches than it harms. Compared with sharing in all groups without the gate, the gate raises this ratio on 7 of 8 datasets (Cityscapes 1.3 →\rightarrow 2.1), and patches harmed inside wrong groups are at most 0.6% of all patches.
  3. The convex blend (αR=0.25\alpha_R = 0.25) only overwrites low-margin patches, so confident correct patches are not flipped.

Removing the gate lowers Cityscapes mIoU from 41.04 to 40.41.

We welcome any further comments, questions, and discussions from you.

Thank you for the detailed response and additional experiments, which have addressed my concerns regarding the hyperparameter settings, computational cost, and the agreement gate. However, I still have concerns about the methodological novelty, so I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.