acceptodds
Withdrawn from ICLR 2027

Relational Structure as a Prior for Weakly-Supervised Part Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study weakly-supervised part semantic segmentation using only image-level object categories and the semantic parts present for each object, without part-level spatial supervision. In this setting, the main difficulty is to establish correspondence between fine-grained part concepts and visual regions when several visually similar parts may coexist within the same object. We observe that although their locations are unknown, part concepts are not independent: pretrained part-text representations already encode relations among them. We therefore use this relational structure as a prior for part-level visual-semantic alignment. Our method identifies part-specific visual support through relative competition among the parts present in the same object, forms image-specific visual prototypes, and adapts part representations with a shared orthogonal transformation that preserves their relational geometry. Experiments on PP-116 and ADE-234 show clear improvements under this weak supervision setting, supporting the use of pretrained relational structure for learning fine-grained part correspondence without part-level spatial annotations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Part Semantic Segmentation, Weakly Supervised Learning, Visual-Semantic Alignment, Relational Structure, Orthogonal Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y1N4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper studies part semantic segmentation using only image-level object labels and part-presence labels, without pixel-level part annotations.

  • It constructs part-specific visual prototypes by comparing relative similarities among parts within the same object.

  • A shared orthogonal transformation aligns pretrained part-text embeddings with visual prototypes while preserving their pairwise semantic relations.

Critical strengths
  • The paper addresses a practical problem of learning part segmentation without part-level spatial annotations.

  • The proposed method combines relative part competition and orthogonal mapping in a simple and straightforward framework.

  • The ablation studies separately demonstrate the effectiveness of relative scoring and the additional benefits of orthogonal constraints.

Critical weaknesses
  • Both relative similarity-based selection and orthogonal mapping (2016) build on established techniques, limiting the methodological novelty. Furthermore, it remains unclear whether the improvement from orthogonal mapping comes from preserving relational structure or simply restricting the flexibility of the transformation.

  • The method assumes that the relational structure of pretrained part-text embeddings is consistent with the underlying visual part structure. Since orthogonal mapping strictly preserves these relations, it may struggle to correct mismatches between the two representation spaces.

  • The improvements on ADE-234 are relatively limited. The mIoU remains low at 6.38, while mACC decreases from 20.06 to 15.82 compared with Talk2DINO-Part. Further analysis would help explain this behavior. In addition, it is unclear whether the approximately 1-point mIoU improvement is statistically meaningful.

  • The adaptation procedure of Talk2DINO-Part, including its training objectives and learnable parameters, is not sufficiently described. Given the substantial performance gap between Talk2DINO-Part and Absolute + Linear (12.38 mIoU), despite using the same backbone and supervision, further clarification would help establish a more transparent basis for comparison.

Questions for authors

I would appreciate it if the authors could address the concerns raised in the weaknesses above.

Reviewer_Ck2kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies weakly-supervised part semantic segmentation using only image-level object categories and per-object part-presence labels, without part-level spatial annotations. The proposed approach builds upon the pretrained Talk2DINO framework and introduces two components. First, relative competition among parts belonging to the same object is used to identify part-specific visual regions and construct image-specific visual prototypes. Second, a shared orthogonal transformation adapts part-text representations toward these prototypes while preserving their pairwise cosine similarities.

Critical strengths
  1. Meaningful weak-supervision setting. The paper studies part segmentation using only image-level object and part-presence labels, reducing the need for spatial annotations.

  2. Effective and interpretable design. The method achieves an improvement on PP-116 (26.85 to 32.60 on mIoU). The ablations and prototype analyses provide useful evidence for the effectiveness of relative competition and orthogonal adaptation.

Critical weaknesses
  1. Questionable optimality of preserving the original relations. Fig. 4(c) shows that linear adaptation initially achieves substantially higher Text-GT relational correlation (0.6121) than orthogonal adaptation (0.5314). This suggests that the pretrained relations are not necessarily optimal for visual part correspondence.

  2. Unclear source of the orthogonal constraint's benefit. The deterioration of linear adaptation in later epochs may indicate overfitting to noisy prototypes rather than the necessity of preserving the original geometry. The current analysis does not distinguish these explanations.

  3. Limited spatial accuracy of induced prototypes. Despite improvements, the final support purity reaches only 0.3711 (Fig. 3). This raises questions about whether the method recovers accurate part correspondence or merely selects discriminative features.

  4. Training–inference competition mismatch. Eq. (5) constructs prototypes using competition among image-present parts within one object, whereas Eq. (13) predicts over the entire part vocabulary. The impact of this mismatch is not analyzed.

  5. Inconsistent improvements on ADE-234. Although mIoU improves from 5.33 to 6.38, mACC decreases from 20.06 to 15.82. This trade-off is insufficiently explained and weakens the claimed general effectiveness.

  6. Incomplete validation of the relational prior. The paper demonstrates that orthogonality preserves pairwise similarities, but provides limited evidence that the preserved pretrained relations are themselves reliable visual-semantic priors across different object categories.

Questions for authors
  1. Early-stage performance: Fig. 4(c) shows that linear adaptation initially achieves higher Text-GT relational correlation. What are the validation mIoU values at these epochs? Can early-stopped linear adaptation match orthogonal adaptation?

  2. Structural prior: How well do the original pretrained part relations agree with ground-truth visual relations across different object categories? Are the improvements related to the quality of this initial agreement?

  3. Prototype reliability: Given the final support purity of 0.3711, how much of the segmentation improvement comes from accurate spatial correspondence versus discriminative but spatially incorrect support patches?

  4. ADE-234 behavior: Which categories account for the decrease in mACC, and is this related to the difference between object-conditioned competition during training and full-vocabulary competition during inference?

Reviewer_eNj8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies weakly supervised part segmentation, in which only the object categories in the image and the corresponding part lists are known, with no dense spatial part annotations. Such part taxonomies are already defined in part segmentation benchmarks, where the presence of parts does not require precise annotations of part locations or boundaries. The challenging part of this problem is that a given region in the image may be highly similar to several closely related part concepts. Having reliable part correspondences requires resolving competition between several candidates for each part. Leveraging this competition, the authors claim that a region in the image is more informative when its similarity to a given part concept exceeds its similarity to other concepts for the same object. Subsequently, they break the problem down into two subproblems: locating part-specific visual targets and updating their representation without losing the semantic relationship with existing part concepts. They start with Talk2DINO's frozen CLIP-to-DINOv2 projector, which learns a single matrix W applied to the part text embeddings within Talk2DINO's object masks. Each patch is scored by its margin over the strongest competitor for that part. Then the top patches are averaged into a visual prototype, and the same W is used to align the part embedding with its prototype. SVD keeps W orthogonal, which preserves pairwise cosine similarities, and at inference time each patch is assigned a part label by taking the argmax over the full part vocabulary.

Critical strengths
  1. Given that Talk2Dino exists, I think this is a very meaningful approach to solving the problem in the presence of weak part annotations in images. The math is fairly easy to follow, though the presentation and finesse could have been better.

  2. The method is cheap, as it trains only a single matrix W.

Critical weaknesses
  1. Different components in the paper are already established techniques. For example, the loop of inducing correspondences between patches and part concepts, fitting an orthogonal map to them, and repeating is observed in the self-learning procedure of this ICLR 2018 paper (Word Translation Without Parallel Data). Relations between parts are already used as a prior in KPS at ICCV 2025. The paper cites it but doesn't discuss its similarity.

  2. The paper's own numbers weaken its premise that pre-trained part-to-concept relationships are reliable, as the text correlates with the ground-truth visuals at only ρ=0.53\rho = 0.53 in Figure 4c.

  3. On ADE-234, the method gains only about 0.18 mIoU over SimZSS-Part and loses 4.2 mACC to Talk2DINO-Part. This contradicts the clear improvements claimed in the abstract and conclusion. Also, Partimagenet is not evaluated, even though it's cited.

  4. Baselines are a bit thin. For example, part-specific methods like PartClipSeg and PartCatSeg are cited but not compared. PBAPS is another method from NeurIPS 2025 that is training-free, builds dynamic two-part prototypes, and reports 46.35 mIoU on PP-116. A very simple baseline that could be compared against clusters DinoV2 features or uses proposals from SAM, then names them by text similarity to the label parts.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.