acceptodds
Under review as a conference paper at ICLR 2027

Hyper-RIS: Hypergraph Consistency for Training-Free Referring Image Segmentation

Abstract

Training-free referring image segmentation (RIS) typically first generates class-agnostic mask candidates and then uses a pretrained vision–language model to select the candidate that best matches the referring expression. However, existing methods mainly score candidates independently or model only low-order interactions, making them prone to errors when multiple visually similar candidates compete for the same expression. We introduce Hyper-RIS, which improves candidate selection by explicitly modeling high-order consistency among candidate proposals. Our framework consists of two main components. First, Multi-Factor Scoring integrates complementary semantic cues from the subject, attributes, relations, and complete expression with dense spatial evidence to produce a unified score for each candidate. Second, Consistency Hypergraph Modeling uses the candidate scores and visual features to capture structured interactions among proposals. It first establishes pairwise visual context and then groups visually related candidates into hyperedges, where visually coherent groups provide consistency-weighted messages to their members. Candidate-wise reliability further suppresses conflicting group evidence, and the resulting high-order context is used to refine the proposal scores. Extensive experiments demonstrate that Hyper-RIS achieves state-of-the-art performance across multiple referring image segmentation benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hypergraph Reasoning, Referring Image Segmentation, High-Order Consistency, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jJX7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the HYPER-RIS framework to address the challenge in training-free referring image segmentation (RIS), where independent candidate scoring or low-order (pairwise) interactions often fail to distinguish between visually similar distractors.

  1. The framework consists of two core stages. The first stage, Multi-Factor Scoring, decomposes a referring expression into four semantic views—subject, attributes, relations, and the full sentence—using syntactic dependency analysis. These are then linearly fused with dense spatial heatmaps (derived from GEM and TIPSv2, incorporating positional priors and conditional area-based refinement) to generate an initial score for each candidate mask produced by SAM. The second stage, Consistency Hypergraph Modeling, constructs pairwise visual contexts and kk-nearest neighbor hyperedges based on the visual features of candidate regions. It aggregates high-order group information using a trimmed mean weighted by the visual consistency within each hyperedge. Finally, a node reliability gate is employed to inject residuals—derived from both high-order and pairwise contexts—into the initial scores, enabling a re-ranking process.
  2. The framework was evaluated on four standard benchmarks: RefCOCO, RefCOCO+、RefCOCOg, and PhraseCut. Experimental results demonstrate that HYPER-RIS achieves state-of-the-art (SOTA) performance in a training-free, zero-shot setting, significantly outperforming methods like RefChess and HybridGL (with mIoU improvements of 3.15–4.86 percentage points on the RefCOCO series). Furthermore, the study validates the method’s generalizability and inference efficiency through experiments involving different encoders (CLIP, FG-CLIP, and TIPSv2) as well as a stage-wise computational time analysis.
Critical strengths
  1. The paper not only demonstrates state-of-the-art (SOTA) performance using powerful features from TIPSv2 and GEM, but also—as shown in Table 6—rigorously controls for the visual encoder variable (comparing CLIP, FG-CLIP, and TIPSv2). Furthermore, Table 5 presents a controlled baseline in which the hypergraph is replaced by second-order graph propagation (HG→G\text{HG}\rightarrow\text{G}). These results provide compelling evidence that the performance gains are indeed partly attributable to the high-order candidate region re-ranking mechanism, rather than relying solely on a more robust visual backbone.

  2. Consistency Hypergraph Modeling operates exclusively through algebraic computations on candidate mask scalar scores and pre-extracted visual features, requiring no additional forward-pass network inference. The breakdown of computational time provided in Table 8 of the appendix is ​​highly transparent; it shows that hypergraph construction and selection account for only about 5.9% (approximately 55.8 ms) of the total end-to-end runtime (942 ms), demonstrating superior computational efficiency compared to RefChess, which is based on Monte Carlo Tree Search (MCTS).

Critical weaknesses
  1. In Equations (7)–(9), hyperedges are constructed solely based on the unconditional visual feature similarity of masks (Vij=fi⊤fjV_{ij} = f_i^\top f_j). The "TrimMean" (trimmed mean) of the scores of members within a hyperedge is then added—with positive weights (α=1.5,β=1.0\alpha=1.5, \beta=1.0)—to the scores of the individual nodes themselves. This implies that visually similar objects of the same category (e.g., multiple donuts or zebras with similar appearances in a scene) are grouped into the same high-weight hyperedge, where they share a common positive group reward. While this "homophily smoothing" can suppress background noise (by lowering the high scores of outliers belonging to non-target categories), it logically reduces the relative score margin between actual target objects and similar distractors within the same hyperedge; it fails to generate a discriminative contrast among similar objects of the same class. The authors overstate this smoothing and noise-reduction mechanism as "fine-grained disambiguation for resolving competition among similar candidates," which represents a flawed interpretation of the underlying mechanism.

  2. According to Appendix A, SAM employs an 8×88\times 8 grid combined with a high threshold (0.7) and a 90% inclusion-based deduplication process, resulting in a very small number of candidate masks NN per image (typically only 5–15). However, the authors set the relative neighbor ratio ρ\rho to an exceptionally high value of 0.7 (Equation 7: k=⌈0.7(N−1)⌉k = \lceil 0.7(N-1) \rceil). This means that each hyperedge encompasses more than 70% of all candidate masks in the entire image! Such excessively large hyperedges not only compromise the semantic purity of local groupings—inevitably incorporating significant amounts of background or unrelated objects—but also cause the trimmed mean mem_e of every hyperedge to converge toward the same value. This directly explains why, as shown in Table 5, removing the node reliability gating mechanism ("w/o NR") results in almost no performance change on the RefCOCO, RefCOCO+, and RefCOCOg datasets (in fact, for RefCOCO+, removing NR leads to a slight increase from 51.81 to 51.84). It also accounts for why the sensitivity curves for the temperature parameters τedge\tau_{edge} and τnode\tau_{node} in Figures 6–8 appear as nearly flat, horizontal lines.

  3. In the "Multi-Factor Scoring" component, the authors introduce a significant number of hard-coded heuristic rules tailored to the annotation preferences found in the RefCOCO dataset (Section 3.2). For instance, depth-related terms like "front" and "back" are rigidly mapped to the bottom and top of the image plane (represented by 1−η1-\eta and η\eta, respectively); "closer" and "further" are strictly tied to adjustments in object area; and a score amplification factor—up to a maximum of bmax=3b_{max}=3—is applied only when a response exceeds the median (Equations 3–4). Such rules are highly prone to failure in scenarios involving top-down or bottom-up viewing angles, non-standard compositions, or complex 3D environments. Furthermore, the paper entirely lacks independent ablation studies for the "positional prior" (Bq\mathcal{B}_q) and the "conditional size adjustment" (bib_i), making it impossible to determine the actual performance gains contributed by these hand-crafted priors.

  4. The most challenging aspect of Referring Image Segmentation (RIS) lies in understanding spatial and interactive relationships between multiple objects. Yet, in Equation (1), the authors acknowledge that the relational phrase TrelT_{rel} is evaluated solely by calculating the cosine similarity against the isolated masked image features fif_i of the candidate object (with the background replaced by the CLIP mean value). There is no incorporation of features from reference or context objects, nor any explicit modeling of spatial-geometric relationships. Once the background and reference objects are masked out, relying exclusively on single-object feature matching to interpret relational phrases—such as "left of the yellow one" or "near the bench and tree"—is semantically unsound. The ablation study in Table 4 also confirms this shortcoming: removing the relation factor (λrel=0\lambda_{rel}=0) has almost no impact on performance (a change of only 0.12% on RefCOCO, and an actual improvement of 0.07% on PhraseCut).

  5. In Table 1, the authors label the pre-trained models used for HYPER-RIS as "SAM, TIPSv2." However, when generating dense heatmaps (Equation 2), HYPER-RIS actually integrates both GEM (ViT-B/16) and TIPSv2 (B/14)—two vision-language models. More importantly, regarding the fair backbone comparison in Table 6: when TIPSv2 is replaced with a standard CLIP, do the authors still retain the GEM ViT-B/16 heatmap branch? If GEM is kept, then the comparison under the "CLIP" column in Table 6 (50.68 vs. 49.48 for HybridGL) is effectively a comparison between a "CLIP + GEM" dual-model setup and the baseline; this undermines the persuasiveness of a fair comparison across different encoders.

Questions for authors
  1. When the true target i∗i^* and a visually highly similar distractor jj of the same category (e.g., "candidate 1" and "candidate 3" in Figure 1b) exhibit an extremely high visual similarity Vi∗jV_{i^*j}, they will inevitably share nearly identical sets of kk-nearest neighbors in Equation (7). Consequently, they belong to the same hyperedge and receive nearly identical positive contextual messages, gg and hh. In such a scenario, how does Equation (11) manage to reverse the initial local score ranking between "candidate 1" and "candidate 3"? Is this reversal merely a result of "candidate 3" happening to be connected to certain low-scoring background masks, rather than the hypergraph truly "understanding" the fine-grained attribute—in this case, a "small hole" in a "white donut"?

  2. Following 8×88\times 8 SAM sampling, filtering with a 0.7 threshold, and deduplication based on 90% coverage (Appendix A), what is the average number of candidate masks NN per image across the four datasets? Given that each hyperedge encompasses the vast majority of candidate masks when ρ=0.7\rho=0.7, why does such a coarse global grouping outperform finer-grained local clustering (e.g., ρ∈[0.2,0.4]\rho \in [0.2, 0.4])? Furthermore, the curves for τedge\tau_{edge} and τnode\tau_{node} in Figures 6–8 are entirely flat; does this imply that the hyperedge consistency weighting and node reliability gating effectively degenerate into simple constant scaling during actual inference?

  3. In Table 4, removing the heatmap component (λheat=0\lambda_{heat}=0) causes the mIoU on the RefCOCO dataset to plummet by 14.04 percentage points (dropping from 58.87 to 44.83). Since SiheatS_i^{heat} incorporates the GEM heatmap, the TIPSv2 heatmap, the positional prior Bq\mathcal{B}_q (including front/back mappings), and the conditional size adjustment bib_i, could the authors provide fine-grained ablation results for these four sub-modules? This would clarify whether the performance gains stem from the pre-trained dense features (GEM/TIPSv2) or from the hand-crafted rules regarding coordinates and areas.

  4. In Table 6, when the encoder of HYPER-RIS is replaced with CLIP or FG-CLIP, is HGEM\mathcal{H}_{GEM} (GEM ViT-B/16) from Equation (2) still utilized? Furthermore, regarding the comparative baselines HybridGL and RefChess, how were their original feature extraction and spatial modules adapted when switched to use TIPSv2?

Reviewer_h2ofOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hyper-RIS, a training-free referring image segmentation framework that reranks SAM-generated mask proposals using hypergraph-based candidate consistency. It first combines semantic matching of different linguistic factors with dense spatial heatmap evidence to obtain initial proposal scores. A consistency hypergraph is then constructed from visual similarities among candidates, allowing group-level score aggregation and reliability-gated score refinement. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and PhraseCut demonstrate improvements over existing training-free approaches. Component ablations, graph-based comparisons, and encoder-controlled experiments further evaluate the effectiveness of the proposed framework.

Critical strengths
  1. Consistent empirical improvements across multiple benchmarks. Hyper-RIS outperforms recent training-free RIS methods across all eight RefCOCO-family splits in both mIoU and oIoU. These improvements provide meaningful evidence of the effectiveness of the overall framework.

  2. Relevant controlled experiments and ablations. The paper includes comparisons across different visual encoders (Table 6), a direct hypergraph-to-graph replacement experiment (Table 5), and extensive component and hyperparameter analyses. These experiments strengthen the evaluation and demonstrate that the performance gains cannot be attributed solely to the choice of pretrained visual encoder.

Critical weaknesses
  1. Limited methodological novelty of the hypergraph formulation. Hypergraph construction and node–hyperedge–node aggregation are well-established techniques in prior work, including Learning with Hypergraphs (NeurIPS 2006), Contextual Hypergraph Modeling for Salient Object Detection (ICCV 2013), and HGNN (AAAI 2019). Hyper-RIS mainly adapts these ideas to proposal reranking through consistency weighting, trimmed-mean aggregation, and reliability-gated score refinement. The distinct algorithmic contribution beyond existing hypergraph reasoning mechanisms is therefore relatively limited.

  2. Limited improvements over conventional graph propagation. Although the full framework achieves strong overall results, Table 5 shows that replacing hypergraph aggregation with graph propagation reduces mIoU by only 0.22–0.94 percentage points across the four benchmarks. These results support the usefulness of contextual score propagation but provide less convincing evidence that the proposed high-order mechanism is substantially more effective than simpler pairwise alternatives.

  3. Spatial relationship modeling is largely implicit. The paper emphasizes resolving complex relational ambiguities, but the proposed hyperedges are constructed from pairwise visual feature similarities rather than explicit spatial relationships among objects. Relation phrases are matched through vision–language similarity (Eq. 1), while position-aware priors (Eq. 3) mainly encode coarse image-level positions. Consequently, the hypergraph primarily aggregates candidate scores rather than explicitly reasoning about relations such as "left of," "behind," or "between." It remains unclear how the proposed mechanism specifically addresses complex spatial relationships beyond existing RIS methods.

  4. Insufficient quantitative evidence for improvements on relational expressions. Figures 4 and 9–12 provide useful qualitative examples involving spatial and multi-object relationships. However, the paper does not report quantitative results separately for absolute positions, pairwise spatial relationships, or more complex multi-object relations. Moreover, removing the explicit relation-matching factor in Table 4 produces relatively small performance changes, although other components may still encode relational information. A targeted evaluation would be necessary to establish whether hypergraph reasoning provides particular benefits for the relational ambiguities motivating the method.

  5. Unclear effectiveness of high-order modeling for different candidate-set sizes. Under Eq. (7) with the default neighborhood ratio, candidate sets containing 2–4 proposals collapse into a single global hyperedge. In this case, the hypergraph refinement becomes equivalent, in terms of proposal ranking, to pairwise propagation with an adjusted coefficient. The paper does not report the distribution of candidate-set sizes or analyze how the improvement over graph propagation varies with the number of proposals. This makes it difficult to assess how frequently the proposed higher-order mechanism provides genuinely distinct information.

Questions for authors
  1. Spatial relationship modeling: How does Hyper-RIS explicitly or implicitly distinguish relational expressions such as "A left of B" from absolute-position descriptions such as "A on the left side of the image"? Which part of the hypergraph formulation captures these relational constraints?

  2. Relational evaluation: Can the authors report results grouped by spatial relationship complexity, comparing Multi-Factor Scoring, graph propagation, and the full hypergraph method? This would help establish whether the proposed high-order reasoning specifically improves relational grounding.

  3. High-order contribution: Given the relatively small improvements over HG→G in Table 5, can the authors clarify whether the graph baseline was comparably optimized and provide evidence that the gains arise from high-order aggregation rather than score smoothing?

  4. Candidate-set size: What proportion of evaluation samples contain 2–4 candidates, and how does the performance improvement over graph propagation vary with candidate-set size?

Reviewer_WExaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hyper-RIS, a training-free framework for zero-shot referring image segmentation, which improves candidate selection in the presence of multiple visually similar candidates through two components: Multi-Factor Scoring and Consistency Hypergraph Modeling. Multi-Factor Scoring integrates semantic cues from subject, attributes, relations, and the complete expression with dense spatial evidence to measure the matching degree between candidate masks and the referring expression. Consistency Hypergraph Modeling constructs hyperedges based on the visual similarity of candidate masks, aggregates high-order context using group consistency, and refines candidate scores through node reliability-gated residual correction. The method outperforms existing zero-shot RIS approaches on RefCOCO/+/g. The paper also provides relatively comprehensive ablation experiments, parameter sensitivity analysis, encoder comparisons, and efficiency analysis to validate the proposed framework.

Critical strengths
  1. The research problem is clearly defined and has practical value. The authors address the challenges faced by candidate-based zero-shot methods and, in the introduction, clearly explain the limitations of existing methods from the perspective of candidate scoring: one type of method scores candidate masks independently, making it difficult to distinguish visually similar candidates, while another type, although introducing interactions between candidates, does not explicitly model a group of mutually related proposals as a unified reasoning unit. The research motivation is reasonable.
  2. The overall experimental results are competitive. The authors evaluated the proposed method on the RefCOCO family, and its performance consistently improved compared to existing methods, demonstrating the effectiveness of the approach. Testing on the PhraseCut dataset further proved that the method has a certain degree of generalization capability.
Critical weaknesses
  1. According to Table 5, the extra benefits of hypergraph compared to graph are relatively small. In contrast, high-order context contributes more. It's recommended to provide visualization experiments to analyze in which scenarios hypergraph has more advantages over graph.
  2. In Section 3.3, hyperedge grouping uses visual similarity, while node reliability measures the dispersion of incident hyperedge messages. If there are multiple visually similar incorrect distractors with high initial scores, they might form a wrong group, and in this case, the hypergraph could reinforce the errors. It's recommended to include some failure case analyses to examine the limitations of the method.
Questions for authors
  1. In Table 5, the gain of Full relative to HG→G is relatively small in RefCOCO+ and RefCOCOg. Could you provide some visual analysis to demonstrate the superiority of hypergraph over graph?
  2. When the method faces multiple visually similar incorrect distractors with high initial scores, will the hypergraph reinforce the errors? Can an analysis of failure cases be added to illustrate the limitations of the method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.