acceptodds
Under review as a conference paper at ICLR 2027

VocabAudit: Auditing Inference-Vocabulary Sensitivity in Training-Free Open-Vocabulary Segmentation

Abstract

Open-vocabulary semantic segmentation (OVSS) is evaluated with a class-name list that directly controls test-time predictions, yet this list is often treated as fixed metadata. We study vocabulary sensitivity by freezing the images, model weights, prompt templates, and dense-inference pipeline while varying only the inference vocabulary. On a 300-image VOC-21 matrix, replacing an engineered vocabulary with plain dataset names reduces mIoU by 11.8–20.6 points. Filtered synonym substitutions cost up to 6 points and produce a 3.7-point spread across vocabulary draws. We then run VocabAudit V1.1 on 1,349 VOC images, three methods, six vocabulary-expansion conditions, and ten independent vocabulary draws per condition. Adding classes absent from the VOC-21 label set increases target-class mIoU by 4.49–12.75 points, with all 18 paired image-bootstrap intervals excluding zero, and reverses the cell-mean SCLIP–NACLIP ordering in every condition. At the same time, the expanded vocabulary substantially changes the predictions: with 200 added classes, pixel accuracy falls to 16.84–24.05%, and distractor labels receive 71.62–91.48% of benchmark-background pixels and 18.81–35.42% of foreground pixels. All-class mIoU also falls to 4.04–4.59, partly because the metric assigns zero IoU to label-set-absent classes. These results show that target class mIoU alone can obscure changes in the prediction interface and in pixel-level errors under vocabulary expansion. Reliable OVSS evaluation should therefore release the exact inference vocabulary, report sensitivity across controlled vocabulary draws, distinguish target-class from all-class accounting, and include an explicit pixel-flow ledger.

Primary area:
datasets and benchmarks
Keywords:
Open-Vocabulary Semantic Segmentation, Evaluation and Benchmarking, Vocabulary Robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LhuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits how training-free CLIP-based open-vocabulary semantic segmentation responds to the inference vocabulary when everything else (OpenCLIP ViT-B/16, templates, sliding window, pooling) is frozen. Three perturbation axes are used: WordNet synonyms filtered by CLIP-text cosine, first-sense WordNet hypernyms, and injection of 50 to 200 absent class names stratified by text similarity. Main claims: on VOC-21, engineered vs. plain name files differ by 11.8 to 20.6 mIoU; full synonym substitution costs 4 to 6 mIoU; injected distractors raise target-class mIoU (+4.5 to +12.8, 10 draws, paired bootstrap on 1,349 images) while pixel accuracy and all-class mIoU collapse, which the paper calls "metric-interface incoherence"; text-side whitening fails; a text-only background-negative synthesis (VABS) recovers most of the engineered-vocabulary gain, as do matched random words. A long appendix adds unmodified author-code anchors (ProxyCLIP, LPOSS, SC-CLIP, Trident, RF-CLIP, CorrCLIP, SCI-CLIP), detector-family transfers, cross-lingual probes, and many pre-registered negative results.

Critical strengths
  1. Relevant problem, carefully run distractor study. The class-name file is effectively part of a training-free method, and the recommendation to release it is sound. The V1.1 distractor matrix uses the full split, 10 frozen vocabulary draws per cell, and paired image-bootstrap intervals with shared multiplicities, which is good statistical hygiene.

  2. Honest controls. The random-word control (random negatives recover +12.6 of the +16.1 macro VABS gain) and the author-code anchors (which reproduce published numbers and then show the effect direction) are informative and candidly reported.

Critical weaknesses
  1. The headline findings are either known or mechanical. (a) That VOC-21 results of training-free methods hinge on how "background" is expressed is well established (background prompt sets in CLIP-ES and CaR; the VOC-20 and Context-59 conventions exist for this reason), and the paper's own data confirm it is VOC-specific: author-code official-vs-plain gaps are +3.3 (SC-CLIP) and +2.0 (Trident) on COCO-Object and +0.2 on Context-60. Name sensitivity of CLIP-style models (Parisot et al., CVPR 2023; Han et al., ICCV 2023) and metric pathologies under overlapping vocabularies (SG-IoU, Open mIoU, OVDEval) are also prior work. (b) The "score paradox" is largely an accounting artifact. All-class mIoU equals 21/(21+|D|) times target-class mIoU exactly (Sec. 3; for example 42.04 times 21/71 = 12.43 in Table 3), so its collapse carries no information. In VOC, "background" means "none of the 20 classes," so a distractor label such as "wall" on a background pixel is not an error; counting the 87 to 91% background absorption as degradation contradicts the benchmark's own definition. Under VOC-consistent accounting (folding non-target predictions into background, as the official files and VABS do), what remains is foreground absorption (29 to 35%), and the paper does not show how much of that is genuinely wrong rather than hyponym or part relations (clothing words taking person pixels, "screen" taking tvmonitor pixels, by the paper's own account).

  2. Two of the three perturbation axes are not validated, and the ranking claims lack statistical support. Synonyms are chosen from any WordNet synset without word-sense disambiguation, the human validity gate was not passed, and the searched vocabularies show the result (cat replaced by "guy", dog by "frump", bird by "skirt", chair by "chairwoman"); the random-suite substitutions used in Table 2 are not listed. The hypernym axis collapses by construction because merged-credit scoring is deferred. The core matrix uses single 300-image subsets with a stated 3.1 mIoU noise floor, comparable to the 4 to 6 point synonym effects; the "model selection depends on the file" evidence (a 1.8 vs. 0.1 gap) is inside that floor; and the SCLIP vs. NACLIP "reversal" involves two near-identical final-block variants 1.3 mIoU apart, with no cross-method confidence intervals.

Questions for authors
  1. For Table 3, can you report background IoU and foreground mIoU separately, and the results when distractor predictions are folded into background?

  2. Does the distractor effect appear on background-free benchmarks (ADE-150, COCO-Stuff-171, Context-59)?

  3. Can you list the synonym substitutions used in Table 2 and report results on a human-validated subset?

  4. How many near-tercile distractors are hypernyms, hyponyms, parts, or synonyms of a target class, and what are the results with them removed?

  5. Can you give paired bootstrap confidence intervals for the SCLIP minus NACLIP difference in each cell?

Reviewer_hCdPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VocabAudit, a systematic evaluation framework for investigating inference-vocabulary sensitivity in training-free open-vocabulary semantic segmentation. By keeping model weights and inference configurations fixed while varying class names, the authors analyze the effects of synonym substitution, semantic granularity changes, and distractor-class injection. Experiments across several segmentation methods and benchmarks reveal substantial performance variations and changes in method rankings. The paper further investigates the limitations of existing evaluation metrics and explores several potential remedies, including vocabulary-adaptive background synthesis and text embedding transformations.

Critical strengths
  1. Practical significance. The paper addresses an important but often overlooked issue in open-vocabulary segmentation, demonstrating that inference vocabulary selection can substantially affect reported performance.

  2. Extensive empirical analysis. The experiments cover multiple segmentation approaches, vocabulary perturbations, and evaluation settings. The additional evaluations with official implementations strengthen the empirical evidence.

  3. Interesting observations. The analysis of distractor-class injection reveals a meaningful discrepancy between target-class mIoU and pixel-level prediction quality, providing useful insights into existing evaluation practices.

Critical weaknesses
  1. The methodological novelty is relatively limited. The influence of class naming, prompt formulations, and vocabulary design on vision-language models has been investigated in previous studies, including RENOVATE, FLOSS, and related evaluation work. Although this paper provides a more systematic analysis, its main contribution remains an empirical audit rather than a fundamentally new evaluation methodology or effective robustness solution.

  2. The semantic validity of vocabulary perturbations requires stronger justification. Synonym substitutions rely primarily on WordNet and CLIP similarity thresholds, which do not necessarily guarantee semantic equivalence in visual contexts. Similarly, the automatic hypernym mapping may introduce substantial changes in category semantics. Human-validated vocabulary perturbations would make the robustness conclusions more convincing.

  3. The interpretation of vocabulary-expansion results remains challenging. The dramatic reduction in all-class mIoU is partly determined by the metric definition, which assigns zero IoU to absent classes. Moreover, categories absent from benchmark annotations may still correspond to meaningful visual content. Additional semantic validation is needed to distinguish genuine prediction errors from incomplete label coverage and metric-induced effects.

  4. The evidence for method-ranking reversals could be strengthened. Although the paper reports consistent changes in cell-mean rankings, some individual vocabulary draws do not exhibit the same ordering. The reported bootstrap intervals primarily characterize changes relative to the baseline rather than uncertainty in cross-method performance differences. Direct statistical comparisons between competing methods would better support the ranking-instability claims.

  5. The generalizability of the main findings remains insufficiently established. The comprehensive V1.1 evaluation focuses on VOC-21 and three methods, while several additional experiments rely on smaller subsets or different inference configurations. Furthermore, the large performance gap between engineered and plain vocabularies is strongly influenced by background modeling. More controlled cross-dataset evaluations would help isolate vocabulary sensitivity from dataset-specific and background-calibration effects.

Questions for authors
  1. Can the authors provide human-validated synonym and distractor vocabularies to better establish the semantic validity of the proposed perturbations?

  2. Are the observed ranking reversals statistically significant when jointly accounting for image sampling and vocabulary-draw variability?

  3. How much of the performance difference between engineered and plain vocabularies can be attributed specifically to background expansion rather than foreground class naming?

Reviewer_jd3HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how the vocabulary supplied to an open-vocabulary segmentation model affects its predictions and evaluation. The authors keep the images, model weights, prompts, and inference pipeline fixed while changing the class names through synonym substitution, broader terms, or additional categories. They find large differences between engineered and plain vocabularies, changes in method rankings, and cases where adding categories improves target-class mIoU while reducing pixel accuracy. The paper also tests several possible repairs and recommends reporting vocabulary sensitivity alongside conventional benchmark scores.

Critical strengths
  1. The paper addresses a relevant evaluation problem in OVSS: the vocabulary has a significant impact on results, yet this impact is rarely evaluated in benchmark comparisons. The paper's recommendation to include the class name list as part of the evaluated system is highly relevant.

  2. The vocabulary comparisons, keeping the inference pipeline fixed, are informative. The main expansion experiment, covering 1,349 VOC images, three methods, six expansion conditions, and ten vocabulary draws per condition, is particularly comprehensive.

  3. The paper identifies several effects that can affect practical conclusions. In particular, the performance gap between engineered and plain names is larger than many differences between methods. Results also show that the ranking of methods can change from one vocabulary to another.

  4. The pixel-flow analysis, which shows that tracking foreground and background pixels assigned to added labels is more informative than reporting mIoU alone, is also interesting. Likewise, the background controls provide useful context and show that a large part of the gain comes from background calibration and not from sophisticated semantic selection.

Critical weaknesses
  1. My main concern is the interpretation of predictions assigned to added categories. A category absent from VOC’s label set may still be present in the image. For instance, predicting “grass” in a region annotated as background could be correct from an open-vocabulary perspective. Therefore, the drop in pixel accuracy does not necessarily mean a worse segmentation.

  2. The naming experiments do not always preserve the intended concept. For example, WordNet replacements may use a different word sense, such as "car" (automobile) to "railroad car". The incomplete human-validated alias study only partially addresses this issue.

  3. The decrease in all-class mIoU is, in large part, a consequence of its definition. Since every added class contributes zero (even when it is never predicted), the score must therefore decrease as vocabulary grows. Although the authors acknowledge this, the resulting “collapse” is over-emphasized in the paper.

  4. The contribution relative to RENOVATE and previous work on vocabulary expansion needs to be clearer. The controlled audit and pixel-flow accounting are useful additions, but the paper should show which conclusions remain after applying existing semantic or mask-wise evaluation protocols.

Questions for authors
  1. Can you estimate how often added-category predictions correctly describe content annotated as background? A manually checked subset or a densely annotated benchmark would help distinguish semantic errors from limitations of VOC’s labels.

  2. Do the naming effects persist with human-validated, sense-preserving synonyms? For the granularity experiment, what happens when the ground truth is merged to match the broader categories?

  3. Which findings remain under existing open-vocabulary evaluation protocols and a metric that does not penalize added classes that are never predicted? In particular, do the method-ranking changes persist?

  4. Does the repeated-draw expansion experiment show the same behavior with official engineered vocabularies and on another dataset? This would clarify how much the result depends on the plain-name background representation.

  5. What do you consider the main new finding beyond RENOVATE and earlier studies of vocabulary expansion and semantic ambiguity? A direct experimental comparison using their evaluation protocols would help make this distinction concrete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.