acceptodds
Withdrawn from ICLR 2027

HIVE: Holistic Image-Adaptive Vocabulary Evolution for Training-Free Dense Vision-Language Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benefiting from pretrained vision-language models, training-free open-vocabulary semantic segmentation (OVSS) predicts pixel-level labels for arbitrary categories without task-specific optimization. Existing training-free OVSS methods commonly perform dense prediction using independently processed local views and a fixed set of category names. However, this paradigm suffers from two limitations. First, processing local views independently prevents the model from capturing cross-view dependencies, limiting contextual reasoning for dense prediction. Second, fixed category names describe semantic classes generically and fail to cover the image-specific appearances of target objects, weakening visual-text alignment. To overcome these limitations, we propose olistic mage-adaptive ocabulary volution (), a training-free framework that jointly improves visual representations and textual vocabularies. Specifically, HIVE introduces Holistic Visual Reasoning (HVR) yielding holistic representations through local-global-local interaction. Furthermore, Image-adaptive Vocabulary Evolution (IVE) constructs a dedicated vocabulary set for each image. It first employs an off-the-shelf MLLM to generate image-conditioned category aliases and then iteratively evolutes the vocabulary set by selecting beneficial aliases under semantic and structural visual guidance. Experiments across eight evaluation protocols show that HIVE achieves an average mIoU of 56.2%, outperforming existing training-free OVSS methods and establishing new state-of-the-art performance. Our code will be publicly available after acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary semantic segmentation, training-free paradigm, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hjCNOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HIVE is a training-free framework for open-vocabulary semantic segmentation that enhances visual representations through local-global-local interaction and improves vision-language alignment by evolving image-adaptive vocabularies with MLLM-generated category aliases. Experiments across eight evaluation protocols achieve an average mIoU of 56.2%, establishing a new state of the art among training-free OVSS methods.

Critical strengths

HIVE achieves strong empirical performance across eight standard OVSS protocols, reaching an average mIoU of 56.2% and outperforming the reported competing methods on every protocol. Evaluations cover diverse natural-image and urban-scene datasets. Experiments with alternative MLLMs also consistently improve over the baseline, suggesting that HIVE's vocabulary adaptation approach is not tied to a single generator.

Critical weaknesses
  1. HIVE introduces per-image MLLM queries and additional local computation compared to VIP and other baselines. This seems to introduce substantial additional overhead, which requires further evaluation.
  2. The improvement from 55.7 for IVE-only to 60.3 for full HIVE adds both HVR and final random-walk refinement, leaving their respective contributions unclear.
Questions for authors
  1. HIVE uses 336×336 local windows, whereas the original VIP uses 224×224. How much do the local windows affect the results?
Reviewer_MPWvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HIVE proposes a training-free framework for open-vocabulary semantic segmentation with two components: Holistic Visual Reasoning (HVR) and Image-adaptive Vocabulary Evolution (IVE). HVR enhances local features through local-global-local interactions, while IVE uses a multimodal language model to generate image-conditioned category aliases and selects them based on semantic and structural consistency. Affinity-guided random walks further refine predictions. Experiments across eight protocols from six datasets report an average mIoU of 56.2%, exceeding VIP by 3.8 percentage points. Component ablations, design comparisons, and qualitative results support the effectiveness of the framework without requiring task-specific parameter updates.

Critical strengths
  1. This paper is generally well written and easy to follow, with figures and tables clearly explaining the methodology and results.

  2. HIVE consistently improves segmentation accuracy across eight evaluation protocols, outperforming the strongest reported baseline by 3.8% mIoU on average.

  3. The evaluation includes component ablations, alternative interaction schemes, vocabulary selection strategies, and qualitative comparisons, providing evidence for key design choices.

Critical weaknesses

My main concerns center on the novelty of the contributions and whether the experiments adequately establish their individual value.

  1. Insufficient differentiation from existing methods. Using LLMs to expand category names is an established idea, and cross-view information exchange is also widely explored in segmentation. Although the introduction and related work briefly distinguish HIVE from previous approaches, the methodological differences are not analyzed in sufficient depth. More detailed discussion and controlled comparisons with representative vocabulary-expansion and cross-view methods are needed to establish a substantive contribution beyond combining existing ideas.
  2. Insufficient evaluation of prediction refinement. Affinity-guided random walks are an established segmentation technique, alongside alternatives such as dense CRF and PAMR. However, the paper does not compare these refinement strategies. Table 2 includes a random-walk-only ablation but does not report the complete HVR+IVE pipeline without final random-walk refinement. Furthermore, the main comparisons do not clearly establish consistent refinement settings across methods. Evaluating HIVE and representative baselines with and without comparable refinement would help separate the proposed modules’ contributions from improvements attributable to standard post-processing.
  3. Missing quantitative analysis of inference cost. HIVE requires MLLM-based alias generation, iterative candidate evaluation, and random-walk propagation. Appendix G acknowledges additional computation from vocabulary evolution, but the paper provides no quantitative assessment of this overhead. The authors should report end-to-end inference latency, memory consumption, and a breakdown of the major components, together with comparisons against the baseline and representative prior methods under comparable conditions. Without these measurements, the trade-off between accuracy and computational cost remains unclear. Inference efficiency is particularly important when assessing the practical value of a training-free method.
Questions for authors

Please see weaknesses.

Reviewer_KTRGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on the training-free open-vocabulary semantic segmentation (OVSS) task and proposes a training-free framework, HIVE, to improve visual representations and textual vocabularies. Specifically, HIVE introduces local-global-local interactions in image features and iteratively constructs a dedicated vocabulary set. The experiments show that the proposed method can significantly improve the baseline.

Critical strengths
  1. The proposed method is well motivated and clearly described.
  2. The experiments on different datasets show that the proposed method can outperform the previous methods.
Critical weaknesses
  1. The proposed Holistic Visual Reasoning (HVR) module seems to be incremental compared with previous methods, such as HRSeg(MM, 2025) and LoGoSeg(AAAI 2026). The idea of extracting both local and global visual features and modeling local-global interactions has been explored in previous studies. The authors should clarify the key technical differences between HVR and these existing approaches to better justify its novelty. Moreover, the HVR module only improves the baseline by 1.6%, which raises concerns about its effectiveness.
  2. The analysis of the IVE module is not extensive. The IVE module is one of the most important contributions of this submission and improves the baseline by 6.2%. However, further analysis is needed to better prove its effectiveness. For example, the performance of different iterations should be shown. It would be better to show some examples of how the updated vocabulary set helps to achieve better segmentation results.
  3. Since the HVR module introduces additional interactions and the IVE updates the vocabulary iteratively, it is better to show the average processing time of end-to-end inference.
Questions for authors

Please refer to the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.