- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits how training-free CLIP-based open-vocabulary semantic segmentation responds to the inference vocabulary when everything else (OpenCLIP ViT-B/16, templates, sliding window, pooling) is frozen. Three perturbation axes are used: WordNet synonyms filtered by CLIP-text cosine, first-sense WordNet hypernyms, and injection of 50 to 200 absent class names stratified by text similarity. Main claims: on VOC-21, engineered vs. plain name files differ by 11.8 to 20.6 mIoU; full synonym substitution costs 4 to 6 mIoU; injected distractors raise target-class mIoU (+4.5 to +12.8, 10 draws, paired bootstrap on 1,349 images) while pixel accuracy and all-class mIoU collapse, which the paper calls "metric-interface incoherence"; text-side whitening fails; a text-only background-negative synthesis (VABS) recovers most of the engineered-vocabulary gain, as do matched random words. A long appendix adds unmodified author-code anchors (ProxyCLIP, LPOSS, SC-CLIP, Trident, RF-CLIP, CorrCLIP, SCI-CLIP), detector-family transfers, cross-lingual probes, and many pre-registered negative results.
Critical strengths
-
Relevant problem, carefully run distractor study. The class-name file is effectively part of a training-free method, and the recommendation to release it is sound. The V1.1 distractor matrix uses the full split, 10 frozen vocabulary draws per cell, and paired image-bootstrap intervals with shared multiplicities, which is good statistical hygiene.
-
Honest controls. The random-word control (random negatives recover +12.6 of the +16.1 macro VABS gain) and the author-code anchors (which reproduce published numbers and then show the effect direction) are informative and candidly reported.
Critical weaknesses
-
The headline findings are either known or mechanical. (a) That VOC-21 results of training-free methods hinge on how "background" is expressed is well established (background prompt sets in CLIP-ES and CaR; the VOC-20 and Context-59 conventions exist for this reason), and the paper's own data confirm it is VOC-specific: author-code official-vs-plain gaps are +3.3 (SC-CLIP) and +2.0 (Trident) on COCO-Object and +0.2 on Context-60. Name sensitivity of CLIP-style models (Parisot et al., CVPR 2023; Han et al., ICCV 2023) and metric pathologies under overlapping vocabularies (SG-IoU, Open mIoU, OVDEval) are also prior work. (b) The "score paradox" is largely an accounting artifact. All-class mIoU equals 21/(21+|D|) times target-class mIoU exactly (Sec. 3; for example 42.04 times 21/71 = 12.43 in Table 3), so its collapse carries no information. In VOC, "background" means "none of the 20 classes," so a distractor label such as "wall" on a background pixel is not an error; counting the 87 to 91% background absorption as degradation contradicts the benchmark's own definition. Under VOC-consistent accounting (folding non-target predictions into background, as the official files and VABS do), what remains is foreground absorption (29 to 35%), and the paper does not show how much of that is genuinely wrong rather than hyponym or part relations (clothing words taking person pixels, "screen" taking tvmonitor pixels, by the paper's own account).
-
Two of the three perturbation axes are not validated, and the ranking claims lack statistical support. Synonyms are chosen from any WordNet synset without word-sense disambiguation, the human validity gate was not passed, and the searched vocabularies show the result (cat replaced by "guy", dog by "frump", bird by "skirt", chair by "chairwoman"); the random-suite substitutions used in Table 2 are not listed. The hypernym axis collapses by construction because merged-credit scoring is deferred. The core matrix uses single 300-image subsets with a stated 3.1 mIoU noise floor, comparable to the 4 to 6 point synonym effects; the "model selection depends on the file" evidence (a 1.8 vs. 0.1 gap) is inside that floor; and the SCLIP vs. NACLIP "reversal" involves two near-identical final-block variants 1.3 mIoU apart, with no cross-method confidence intervals.
Questions for authors
-
For Table 3, can you report background IoU and foreground mIoU separately, and the results when distractor predictions are folded into background?
-
Does the distractor effect appear on background-free benchmarks (ADE-150, COCO-Stuff-171, Context-59)?
-
Can you list the synonym substitutions used in Table 2 and report results on a human-validated subset?
-
How many near-tercile distractors are hypernyms, hyponyms, parts, or synonyms of a target class, and what are the results with them removed?
-
Can you give paired bootstrap confidence intervals for the SCLIP minus NACLIP difference in each cell?