acceptodds
Withdrawn from ICLR 2027

RIMAR: Riemannian Metric Attention and Regularization for Training-Free Open-Vocabulary Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free open-vocabulary semantic segmentation converts frozen vision-language features into pixel predictions. Existing inference methods typically use different geometric quantities for feature comparison and spatial refinement. We introduce RIMAR, which uses a Riemannian metric derived from spatial variation in a multi-layer feature field as a common geometry for both. Bidirectional Affine Attention (BAFAT) fits closed-form affine query-key mappings using normalized metric-area weights. Its area-weighted proposal is fused with native attention before value aggregation. The retained metric supplies directional distances to a refinement graph alongside RGB differences. Semantic information from the input and overlapping-window integration produce observations refined by fixed-budget Gaussian and heavy-tailed updates derived from maximum a posteriori (MAP) objectives. RIMAR requires no segmentation training, auxiliary visual backbone, or camera calibration. For a continuous BAFAT-MAP reference family, we establish reparameterization equivariance of scores and MAP solution sets under consistent transport of features, sampling measures, and regularization fields. With frozen CLIP ViT-B/16, RIMAR averages mIoU and pixel accuracy across 13 panoramic, fisheye, and perspective benchmark settings, exceeding PEARL by and percentage points, respectively, under a common evaluation protocol. Paired projection and sampling studies show higher mean accuracy and prediction agreement than PEARL and NACLIP, alongside residual sensitivity. Controlled comparisons show an upstream advantage over PEARL under a common refiner and a -point mIoU improvement over a jointly simplified geometry-independent variant.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary semantic segmentation, training-free inference, vision-language models, Riemannian geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qWYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

RIMAR is a training-free framework for open-vocabulary semantic segmentation that unifies feature comparison and spatial refinement using a Riemannian metric derived from multi-layer feature variations. It combines bidirectional affine attention with geometry-aware graph refinement, requiring no segmentation training or additional visual backbone. Experiments across 13 benchmark settings achieve 41.4% mIoU and 70.6% pixel accuracy, outperforming PEARL under a common evaluation protocol.

Critical strengths

RIMAR achieves strong results across 13 panoramic, fisheye, and perspective settings, improving mean mIoU over PEARL by 2.9 points while also achieving lower inference latency under matched conditions.

Critical weaknesses
  1. The necessity of the more elaborate geometric and alignment designs is not sufficiently established. Although the complete system performs well, several simpler alternatives achieve similar results. In Table 11, bidirectional affine alignment scores 41.394 mIoU, compared with 41.386 for bidirectional orthogonal alignment and 41.328 for self-self attention. In Table 4, adding feature-distance guidance to RGB-only refinement provides no clear additional average gain. The joint simplification in Sec. 4.3 also trails the full system by only 0.444 points. These results leave the practical benefit of the additional designs unclear.
  2. The pipeline contains many interacting components, but the provenance of its empirical parameter choices is unclear. Beyond geometric attention, RIMAR includes adaptive layer mixing, prototype adaptation, local/context integration, and two-stage refinement. These introduce choices such as the score scale, blend coefficient, heavy-tail scale, posterior resolution, and iteration budgets. Sec. C.4 explicitly states that the original search procedure and development split for several choices cannot be identified. This makes it difficult to assess how the configuration was selected and whether the reported gains remain stable under reasonable parameter changes.
Questions for authors
  1. Could you clarify exactly what is removed or replaced in the “w/o BAFAT” and “w/ flat metric” variants in Table 13, and how they differ from the self-self substitution in Table 11 and the joint simplification in Sec. 4.3?
  2. What data and selection criteria were used to choose the empirical parameters listed in Sec. C.4? Could you report a sensitivity study for the score scale, blend coefficient, and heavy-tail scale?
Reviewer_H3GvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a shared feature-geometric construction whose area measure weights bidirectional affine attention and whose directional lengths guide spatial score refinement within a frozen-model predictor. The authors establish a reparameterization-equivariance result for scores and MAP solution sets of a continuous BAFAT-MAP reference family under explicit transport assumptions. An evaluation is conducted across 13 perspective, panoramic, and fisheye benchmark settings, with controlled comparisons separating upstream aggregation, refinement, and the roles of geometric and RGB cues.

Critical strengths
  1. The methodology section is presented in detail, with mathematical formulations illustrating each step of the proposed method.

  2. The compared methods are relatively advanced, and the experimental settings are more diverse.

Critical weaknesses
  1. The motivation lacks in-depth analysis. The transition from existing methods to the proposed Riemannian metric is not well justified. Although the authors explain what a spatial metric can provide, they fail to clarify why these properties can address the limitations of existing methods and how they are connected. Furthermore, the authors repeatedly emphasize the importance of the spatial metric in the motivation. However, Table 4 shows that introducing feature-based geometric guidance into the RGB-based setting fails to improve performance and instead leads to a slight degradation.

  2. The experimental details necessary for reproducibility are insufficient. In addition, the paper lacks a comprehensive computational efficiency analysis of RIMAR, including component-wise efficiency evaluations and computational cost comparisons with existing methods.

  3. The related work section is too brief, particularly the Geometry subsection. The limited number of cited studies is insufficient to adequately distinguish the proposed method from prior work. Besides, Figures 1 and 2 are poorly designed and lack sufficient visual clarity and refinement.

  4. The writing quality of this paper leaves considerable room for improvement.

Questions for authors

Please refer to "Critical Weaknesses".

Reviewer_rtayOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

RIMAR is a training-free open-vocabulary segmentation method that constructs a shared Riemannian metric from multi-layer CLIP features. It uses metric-area weights for bidirectional affine attention and combines metric-derived distances with RGB cues for Gaussian and heavy-tailed spatial refinement. With frozen CLIP, RIMAR achieves 41.4% average mIoU across 13 panoramic, fisheye, and perspective evaluation settings, outperforming PEARL by 2.9 percentage points.

Critical strengths
  1. Strong overall empirical performance across diverse imaging conditions. RIMAR achieves an average mIoU of 41.4% across 13 benchmark settings covering panoramic, fisheye, and perspective imagery, outperforming PEARL by 2.9 percentage points and achieving the highest mIoU in 11 of the 13 settings. These results provide substantial evidence for the effectiveness of the complete system.

  2. A practical training-free formulation with favorable measured latency. The method uses frozen CLIP encoders without segmentation-specific training, auxiliary visual backbones, or camera calibration. Under the reported H100 measurement protocol, the complete predictor also achieves lower latency than PEARL (108.66 versus 153.17 ms/image), supporting its practical potential despite the more elaborate pipeline.

Critical weaknesses
  1. The necessity of the additional design choices and the attribution of the core gains remain unclear. Compared with related methods such as PEARL and NACLIP, RIMAR introduces more interdependent aggregation, alignment, and refinement steps. Although the full system’s performance improvements, including a 0.444-point mIoU gain over the jointly simplified variant, support its overall effectiveness, the results do not sufficiently establish why several of the more elaborate design choices are needed. For example, in Table 11, bidirectional affine alignment improves over bidirectional orthogonal alignment by only 0.008 mIoU points. In Table 4, adding the geometric-distance term does not improve average performance when RGB guidance is already enabled. These findings leave the advantages of some core design choices over simpler alternatives insufficiently established. Further controlled simplification studies would help identify the main sources of improvement and determine whether a simpler pipeline could retain most of the performance.

  2. The presentation of the method is not sufficiently intuitive. Although the paper provides an overall pipeline diagram, stage-specific schematics, and an algorithm description, Figure 1 compresses several distinct operations into individual nodes, requiring substantial cross-referencing between the figures, equations, and appendix to reconstruct the complete data flow. In particular, the paper lacks concrete visual explanations of how the shared metric produces area weights and directional coupling, and how these mechanisms affect actual predictions. Annotating key intermediate quantities in the pipeline diagram, visualizing the geometry-derived quantities on real images, and providing side-by-side comparisons of intermediate predictions across the refinement stages would help readers understand the core design and the role of each stage.

Questions for authors
  1. Could the authors provide concrete visual examples illustrating the roles of the main components? For the same representative input, visualizations of the metric-derived area weights and directional coupling, together with predictions before and after the key refinement stages, would help explain how these components affect the output and which types of errors they address.

  2. What evidence establishes the practical advantage of the core geometric design? Table 11 shows nearly identical performance for bidirectional affine and orthogonal alignment, while Table 4 shows no average benefit from geometric-distance guidance when RGB guidance is enabled. In light of these findings, what practical benefits justify the chosen formulation? Could the authors further isolate the benefit of sharing the metric between attention aggregation and spatial refinement, beyond the improvements of the complete system?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.