acceptodds
Withdrawn from ICLR 2027

MulGeCa: Anchoring Scattered Semantic Representations to Physical Entities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent open-vocabulary segmentation methods leverage pretrained vision-language models (VLMs) to recognize arbitrary text queries beyond the constraints of closed-set categories. Despite their promise, adapting image-level vision-language representations to dense pixel-level prediction remains fundamentally challenging. Semantic representations from VLMs are often scattered and weakly structured, making it difficult to bind open-vocabulary concepts to their corresponding physical entities. To solve this problem, our key insight is that category-agnostic depth cues provide spatial constraints that can bind semantic responses to depth-consistent physical support. To this end, we propose MulGeCa, which leverages estimated relative depth to hierarchically calibrate scattered VLM semantics at the scene, region, and instance levels. Specifically, MulGeCa incorporates a global-geometry text embedding calibrator to inject dynamic geometric context into text prompts, a geometric-contextual anchor aggregator to modulate attention mechanisms with geometric-consistency biases, and a centroid-purified mask refiner to suppress boundary noise via centripetal weighted aggregation. Extensive experiments on challenging benchmarks demonstrate that MulGeCa achieves superior performance, with particular improvements in cluttered scenarios. Notably, our method exhibits strong robustness in complex scenes, providing an effective pathway for anchoring scattered semantic representations to depth-consistent regions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vRbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MulGeCa uses estimated depth to improve open-vocabulary segmentation through scene-conditioned text embeddings, depth-guided feature aggregation, and mask feature reweighting. Experiments examine cross-dataset performance and the effects of depth quality, model capacity, and individual components.

Critical strengths
  • Using depth to reduce mixing between nearby objects is well motivated, and the method improves its segmentation baseline across several benchmarks.
  • The appendix provides useful controls for auxiliary feature capacity, depth quality, computational cost, and variation across training runs.
Critical weaknesses
  • DFormerv2 already uses depth-based geometric relations to guide self-attention, but is not discussed. Vireo also deserves more than a passing citation given its use of depth and language for open-vocabulary segmentation. These connections matter for judging the novelty.
  • CPMR is compared with different kernels, but not with a strong learned mask-pooling alternative. Mask-Adapter is cited and included in the cost analysis; a matched accuracy comparison would better establish the value of the proposed pooling rule.
  • The MAFT+ Base results differ from those in the official repository. I suggest distinguishing reproduced from published numbers and clarifying whether the baseline and MulGeCa share the same training and inference settings.
  • The main performance claims are stronger than the full results support. The large model shows mixed gains over MAFT+ and trails stronger methods in Appendix Table 19. This should be reflected in the main discussion.
  • Some details needed to reproduce the geometry experiments remain unclear, including depth normalization, and the strength of the noise perturbations. These settings should be specified explicitly.
Questions for authors
  • Were the MAFT+ Base results reproduced under the same settings as MulGeCa, and what explains the difference from the released baseline?
Reviewer_fQeCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MulGeCa, a multi-level geometric calibration framework for open-vocabulary semantic segmentation. The authors argue that pretrained vision-language models (VLMs) produce spatially scattered semantic representations, making it difficult to associate semantic concepts with physical entities. Hence, they propose to use scene-level geometric context, region-level contextual anchor, and instance-level centroid-purified mask refiner. Experimental results demonstrate improvements over the adopted baseline on several common benchmarks.

Critical strengths
  1. A coherent multi-level integration strategy. The framework incorporates geometric information at the scene, region, and mask-representation levels rather than simply concatenating depth and RGB features. The proposed hierarchy is understandable, and the three modules target different aspects of the segmentation pipeline.
  2. Comprehensive ablations and supplementary experiments. The paper includes component-wise ablations, architectural comparisons, different depth estimators, backbone comparisons, and additional computational measurements.
  3. The paper is well-organized and easy to follow.
Critical weaknesses
  1. The claimed semantic-to-physical anchoring is not directly validated. The central claim of the paper is that MulGeCa addresses the failure of VLMs to bind scattered semantic representations to physical entities. However, the experiments primarily evaluate conventional semantic segmentation accuracy using mIoU.
  2. The performance improvement is marginal and some recent baselines are not evaluated. On large VLM, the proposed model cannot even beat CAT-Seg [1], which is a classic model in OVSS.
  3. The methodological novelty is incremental. The multi-level integration is not new in OVSS, e.g., DiSa [2].

[1] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation [2] DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation

Questions for authors
  1. Can the authors provide quantitative evidence that MulGeCa improves semantic-to-physical correspondence beyond standard mIoU improvements?
  2. Can the authors elaborate their unique contributions?
Reviewer_Yq7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies open-vocabulary semantic segmentation with CLIP-based mask classification. The authors argue that CLIP features behave like a bag of words and do not bind concepts to physical objects, especially in cluttered scenes. MulGeCa adds estimated monocular depth from Depth Anything V2 to MAFT+ at three levels. At the scene level, a depth-conditioned meta-network shifts the prompt tokens, and a learned gate fuses the result with the static text embedding. At the region level, a depth-affinity bias is added to the attention of a frozen DINOv2, and CLIP features are injected by cross-attention. At the instance level, mask pooling is replaced by Gaussian weighting on a learned response map. The model is trained on COCO-Stuff and tested zero-shot on five benchmarks. With ConvNeXt-B it beats MAFT+ on all five. With ConvNeXt-L the changes are at most +0.4 and are negative on PC-459 and PAS-20. The appendix adds five-seed ablations, depth-source and noise controls, efficiency tests, a plug-in test on CAT-Seg, and failure cases.

Critical strengths
  1. A simple, modular idea. Estimated depth serves as a category-agnostic prior, CLIP's text encoder stays frozen, and no extra segmentation labels are needed. In one plug-in test, G2TEC also raises CAT-Seg from 31.8 to 34.7 on A-150 in Table 20, which suggests that the scene-level component can transfer beyond MAFT+. This matters because an idea that attaches to an existing decoder is easy to adopt and to test.

  2. A broad set of controls. Table 16 reports five seeds, with A-150 going from 33.63 ± 0.15 to 35.27 ± 0.10. Table 10 swaps depth estimators, adds noise, and shuffles depth, and shuffled depth falls back to the no-depth score of 32.7. App. D.7 also states that the Large results trail CAT-Seg and DEDOS. This matters because the shuffled-depth result is the best evidence that aligned depth helps, even though it comes from a 10K-iteration setting.

Critical weaknesses

Critical Weaknesses

  1. Key numbers conflict across tables, so the headline gain is not yet reliable.

PC-459 is the main case. MulGeCa-B scores 19.0 in Tables 1, 12, 19 and 21. This is the +4.8 gain over MAFT+ at 14.2 that the text highlights. But Table 23 gives 14.96 at the default affinity floor of 0.05, and every row of Table 24 lies between 14.50 and 14.67. At the default settings, these tables agree with Table 1 within 0.3 on A-150, PC-59 and PAS-20. Unless the protocols differ materially, the PC-459 gain would be about 0.5 to 0.8 points, not 4.8. The 19.0 also equals CAT-Seg's Base score on PC-459 in Table 19.

The "strongest" claims are uneven. Sec. 4.2 says MulGeCa is strongest on A-847, A-150 and PC-59 under both Base and Large. This holds within Table 1, which omits CAT-Seg, DEDOS and SCAN. Table 19 shows CAT-Seg-L higher on all three. App. D.7 concedes this, but Sec. 4.2 is not qualified. App. D.7 also says Base leads on all five benchmarks, while Table 21 shows SCAN at 97.0 against 95.9 on PAS-20.

The parameter counts do not add up. The rows of Table 8 sum to 386.10M total and 208.93M trainable, not the stated 385.56M and 207.14M. It is also unclear whether the MAFT+ total in Table 6 is counted on the same basis.

This matters because the largest gain is the main evidence for the method, and these conflicts leave its size unknown.

  1. The gain is small, and its attribution to geometry is not isolated from added capacity.

The gain is small at scale. With ConvNeXt-L the changes over MAFT+ are +0.4, -0.4, +0.1, +0.2 and -1.1. Tables 6 and 8 show an added 88.22M depth encoder and an 85.94M DINOv2. Trainable parameters rise from 109.98M to 207.14M. The main text does not report this.

The capacity controls are the right idea but not conclusive. Tables 14 and 15 report single numbers without variance. They attribute only 0.8 to 1.3 mIoU to depth over the matching RGB variant. On the text side, App. D.5 shows that an RGB stream alone lifts the static reference from 31.6 to 34.1, and aligned depth adds 0.8 more. That reference is 2.0 below the MAFT+ baseline of 33.6, so it is unclear how much of the G2TEC gain over MAFT+ belongs to depth. Table 10 uses 10K iterations and has no MAFT+ reference.

The module gains are sub-additive. In Table 16, single modules give +1.31, +1.19 and +1.38. All three give +1.64. Every pair lies within 0.1 of the best single module.

CPMR does not operate on raw depth. It computes its centroid on a learned response map, as App. E states. With raw depth as input, all kernels score roughly 27 to 29.5 in Fig. 5a, read from bars without numeric labels. All sit below the 33.6 baseline. The figure does not name the dataset, but the 35.3 bar in Fig. 5c implies A-150.

The evidence supports useful depth-assisted gains. It does not yet support the stronger claim that multi-level geometric anchoring is the main cause.

Questions for authors

Q1. Which PC-459 value is correct for MulGeCa-B, 19.0 or about 14.5 to 15.0? What differs between Table 1 and Tables 23 and 24? Can you report PC-459 across several seeds?

Q2. Were the MAFT+ numbers in Table 1 reproduced with your schedule, or taken from the original paper? The baseline row of Table 16 rounds to the same values.

Q3. How does depth enter the network? Sec. 3.1 says CLIP-V processes both RGB and depth, but Table 8 lists a separate 88.22M Depth SPM. In the RGB rows of Tables 14 and 15, is the depth branch removed or only fed zeros? Why does the static reference at 31.6 sit 2.0 below the MAFT+ baseline at 33.6?

Q4. Can you provide one capacity-matched control at the full schedule with several seeds? It should compare aligned depth, shuffled depth, and an RGB-only branch of equal size, with everything else fixed.

Q5. Why is FPS higher than MAFT+, 19.77 against 13.18, with 2.2 times the FLOPs? Table 13 gives 35.0 mIoU at 384 × 384 and 35.3 at 640 × 640. Which resolution produced Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.