acceptodds
Withdrawn from ICLR 2027

Hierarchical Prototype Learning for 3D Scene Graph Generation from RGB-D Sequences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Scene Graph Generation (3DSGG) aims to construct structured scene representations by recognizing object nodes and predicting relational edges from 3D environments. Existing RGB-D sequence-based methods have made promising progress by reconstructing 3D geometry and enhancing node and edge features with multi-modal inputs. However, they still struggle to address the severe inter-class similarity and intra-class diversity of both objects and relations in complex 3D scenes. In this work, we propose HPSSG, a hierarchical prototype learning framework for RGB-D sequence-based 3DSGG, inspired by the hierarchical perception process of human scene understanding. Specifically, semantic-level prototypes serve as stable semantic anchors to improve the separability between similar classes, while mode-level prototypes capture multiple latent patterns within each category to model diverse object instances and relation configurations. To maintain hierarchical consistency, we further introduce hierachy-aware prototype entailment learning, which constrains mode-level prototypes with their corresponding category semantics. Moreover, prior-guided compositional prototype learning uses subject-object compositional priors to select and refine mode-level edge prototypes, improving the structural consistency of relation prediction. Experiments on the 3DSSG dataset demonstrate that our method outperforms various leading methods without any additional inference cost. Our code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Graph, Prototype Learning, RGB-D Sequence

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_zEQCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HPSSG, a hierarchical prototype learning framework for RGB-D sequence-based 3D scene graph generation. HPSSG models nodes and edges with semantic-level and mode-level prototypes to improve inter-class separability while capturing diverse intra-class patterns. The framework further introduces two strategies: Hierarchy-aware Prototype Entailment Learning, which maintains hierarchical consistency by constraining mode-level prototypes within the semantic cones defined by their corresponding semantic-level prototypes, and Prior-guided Compositional Prototype Learning, which leverages subject-object mode-level prototypes to guide the selection and refinement of edge mode-level prototypes. Experiments on the 3DSSG dataset demonstrate that HPSSG outperforms previous methods without additional inference cost.

Critical strengths
  1. The paper presents a well-motivated hierarchical prototype learning framework to address inter-class similarity and intra-class diversity in 3D scene graph generation. The overall framework is intuitive and clearly structured.

  2. The proposed method achieves notable improvements on the 3DSSG benchmark under both reconstructed dense and ground-truth point cloud settings, surpassing the best reported baselines in the corresponding comparison tables by 4.3% and 4.5% respectively. These improvements are achieved without increasing inference parameters or latency, demonstrating the effectiveness and efficiency of the proposed approach.

Critical weaknesses
  1. The paper does not adequately discuss or compare against GMPSSG (ICML 2026), which addresses the same RGB-D-based 3D scene graph generation task and also tackles inter-class similarity and intra-class diversity through prototype learning. The authors should clarify the key technical differences and provide a direct experimental comparison under consistent settings. In addition, in Appendix D, the authors explicitly write "our method GMPSSG" when referring to their own method, despite GMPSSG being entirely absent from the main discussion. This naming error suggests that the authors were aware of GMPSSG.

  2. The method assigns the same number of mode-level prototypes (K=5) to every object and relation category, despite their potentially different levels of intra-class diversity. For example, an object category such as "chair" may naturally exhibit multiple distinct modes due to variations in shape, size, and structure. However, for a relation category such as "attached to" or "standing on", it is less clear whether its different instances exhibit a comparable degree of diversity that would justify five distinct prototypes. Although Table 4(a) shows that K=5 achieves the best overall performance, these results do not explain why the same number of modes is appropriate for both object and relation categories.

References:

  1. Learning Gaussian Mixture-distributed Prototypes for 3D Scene Graph Generation from RGB-D Sequences. Rongxing Ding et al., ICML 2026.
Questions for authors
  1. How does HPSSG differ from GMPSSG (ICML 2026) in terms of its core technical contributions? Could the authors provide a quantitative comparison with GMPSSG under consistent experimental settings? Such a comparison would be important for assessing the novelty and effectiveness of the proposed method.

  2. Could the authors clarify why K=5 is used for both object and relation categories? Given that relation categories may have different levels of intra-class diversity compared with object categories, what evidence supports assigning the same number of prototypes to both? A concrete analysis of the learned relation prototypes would help clarify whether all five prototypes capture meaningful and distinct patterns.

Reviewer_WUCrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HPSSG, a hierarchical prototype learning framework for RGB-D sequence-based 3DSSG. By integrating category-level semantic prototypes with various mode-level prototypes, the method addresses challenges related to inter-class similarity and intra-class diversity. The method further introduce hierarchy-aware prototype to constrains the semantic consistency between semantic-level and mode-level prototypes. Experimental results on the 3DSSG dataset demonstrate performance improvements.

Critical strengths
  1. This paper addresses two relevant challenges in 3D scene graph generation, i.e., inter-class similarity and intra-class diversity. The proposed framework introduces semantic-level prototypes and mode-level prototypes to represent semantic categories and different patterns within each category. This design is intuitive and logical, with the respective roles of these two types of prototypes clearly articulated.

  2. The proposed method achieves consistently outperforms the compared baselines on 3DSSG, whether using reconstructed or ground-truth point clouds. These results show the practical value of the proposed strategy. Furthermore, since the prototype modules are only utilized during the training phase and can be removed at inference, the method achieves performance gains without incurring additional inference cost.

Critical weaknesses
  1. Although this paper mainly involves inter-class similarity and intra-class diversity, the main comparison and component ablation studies are conducted on a simplied label space (only consists of 20 object categories and 7 valid relationship categories). The experimental results in this settings are not sufficient to demonstrate that the method effectively addresses fine-grained category confusion or models intra-class structural variations. Furthermore, in the main experiments, both door and picture, which are cited in the motivation to illustrate inter-class similarity, show decreased recall relative to MonoSSG. Among the relationship categories, “hanging on” improves while “attached to” declines. The “chair” category, used to illustrate intra-class diversity, also exhibits decreased recall. These results fail to verify your motivation and thus make this method unconvincing.

  2. The compositional learning objective does not sufficiently support the claimed node-edge compatibility modeling. In Eq. (12), the highest-scoring edge prototype is selected as the positive target and then used to enhance based on that same score. This self-reinforcing mechanism encourages the model to produce high-confidence assignments, yet it fails to verify whether these assignments actually capture meaningful compositional patterns. Moreover, it only contrasts prototypes within the ground-truth edge category, and does not provide direct supervision for distinguishing compatible from incompatible relation categories. Therefore, the authors' claim of node-edge compatibility modeling lacks a solid foundation. The reported performance improvements are likely attributed to the increased confidence of the assignment results.

  3. The authors only conduct experiments on a single dataset 3DSSG. I think it is insufficient. The authors should conduct extensive experiments on various datasets such as RelationScanNet. It is necessary to conduct evaluations on RelationScanNet to determine whether the proposed prototype learning framework can generalize, as well as whether it is not restricted to the specific class distributions and relational patterns unique to 3DSSG.

  4. This manuscript does not follow the required ICLR in-text citation format. The paper uses numerical citations throughout the main text instead of the author-year format specified in the official ICLR template. Numerical citations are typically more compact and can leave additional space for main text content under the same page limit. This raises a fairness concern compared to submissions that follow the prescribed format.

Questions for authors
  1. How do you explain the decreased recall for door, picture, and chair, which are used to motivate your method? What evidence demonstrates that the proposed prototypes effectively address inter-class similarity and intra-class diversity?

  2. What evidence shows that Eq. (12) learns meaningful node-edge compatibility, rather than merely increasing the confidence of prototype assignments?

  3. Can you provide results on another dataset, such as RelationScanNet, to assess whether the proposed framework generalizes beyond 3DSSG?

Reviewer_XSitOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HPSSG, a hierarchical prototype‑learning framework for RGB‑D sequence‑based 3D scene graph generation (3DSGG). To mitigate inter‑class similarity and intra‑class diversity for both object nodes and relational edges, the authors design two levels of prototypes: semantic‑level prototypes as stable category anchors and mode‑level prototypes to capture diverse intra‑class latent patterns. Two auxiliary losses are further introduced: hierarchy‑aware prototype entailment learning for hierarchical consistency and prior‑guided compositional prototype learning to model subject‑object compatibility for edge prediction.

Overall, the paper is clearly written and experiments are well‑organized and presented. Given that I am not a deep expert in the 3D scene‑graph sub‑field, my assessment is mainly built upon my general computer‑vision background.

Critical strengths
  1. Effective training‑only hierarchical prototype design with no inference cost. The semantic‑mode dual prototype paradigm well targets two core pain points in 3DSGG: inter‑class confusion and intra‑class variation. Since prototype modules are discarded at inference time, the method brings performance gains without extra inference parameters or latency, which is practically valuable for real‑world RGB‑D perception systems.
  2. Comprehensive ablation analysis. The paper validates each key module, hyper‑parameter choices and different prototype branches. It also provides both quantitative metrics and qualitative visual comparisons to demonstrate improvements on node classification and relation prediction.
Critical weaknesses
  1. Missing comparison against recent state‑of‑the‑art 2026 baselines. The manuscript does not compare against NoPA (CVPR 2026) and GMPSSG (Ding et al., ICML 2026), the strong recent RGB‑D 3DSGG methods. Without experimental comparison or in‑depth qualitative discussion against this competitive work, the paper’s claimed state‑of‑the‑art performance cannot be sufficiently justified.
  2. Limited conceptual novelty of core insights. The key motivating observations including inter‑class similarity / intra‑class diversity in 3D scene graphs are well‑recognized long‑standing challenges in this domain. Moreover, prototype‑based representation learning itself is an established paradigm widely explored in prior vision literature. The paper mainly combines and adapts existing ideas for the 3DSGG task, and it fails to deliver substantially new conceptual insights beyond domain‑specific engineering.

Minor Concerns

There exist several typographical and capitalization issues throughout the manuscript. For example, at lines 95 and 98: “Second, We design” and “Third, We introduce prior‑guided composition” incorrectly capitalize the pronoun We after commas; we should be lowercase. In addition, please double‑check consistent hyphenation and spelling for the term “hierarchy‑aware prototype” across the full text, and perform a full proofread to resolve other similar grammatical or formatting typos.

Questions for authors

see the weakness.

1 comment

Thank the authors for the author’s response, especially the comparison with the 2026 method; the additional experiments provided by the author address my concerns regarding the paper’s effectiveness. However, the explanation of the paper’s novelty only partially resolves my concerns. The paper’s novelty remains incremental; thus, I maintain my previous recommendation of a weak acceptance.

Reviewer_toUaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HPSSG, a hierarchical prototype learning method for 3D scene graph generation from RGB-D sequences. It addresses two key challenges: inter-class similarity, which causes confusion between visually similar objects, and intra-class diversity, which arises from variations in appearance within the same category. The method uses semantic-level prototypes as category anchors and mode-level prototypes to capture diverse latent patterns within each category. The work offers a useful contribution to the research community.

Critical strengths

1.The hierarchical prototype design is well motivated by the twin problems of inter-class similarity and intra-class diversity, which are genuinely difficult in 3D scene graph generation. The idea of separating stable semantic anchors from diverse intra-class modes is intuitive and aligns with how humans perceive categories.

2.The ablation study is comprehensive, with analyses of individual components, the number of mode-level prototypes, the momentum update coefficient, and the contribution of node versus edge branches.

3.The method adds no inference-time overhead, since prototypes are only used during training as auxiliary supervision. This is a practical advantage that makes the approach easy to adopt.

Critical weaknesses

1.The generalizability of the prototype approach to other domains is unclear, and the single-dataset evaluation limits the strength of the claims.

2.The paper does not analyze which specific categories or edge types drive the improvement versus which remain difficult, making it hard to assess whether the method genuinely addresses the core problem of inter-class confusion.

3.The prototype learning framework is somewhat complex, with five trivial loss terms and multiple hyperparameters, yet the paper does not provide sensitivity analysis for most of these weights, making it unclear how robust the results are to tuning choices. It is also unclear whether the gains come from the prototype structure itself or simply from the additional auxiliary supervision.

Questions for authors

1.The ablation shows that K=8 slightly degrades performance compared to K=5, but the reason for this degradation is not explored.

2.How would it behave in a zero-shot or open-vocabulary setting where some categories have no training data is not discussed, and whether semantic-level prototypes transfer to unseen categories is unknown.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.