acceptodds
Under review as a conference paper at ICLR 2027

PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding

Abstract

Recent advances in 3D multimodal large language models (3D-MLLMs) have enabled unified solutions for 3D scene understanding tasks, including visual question answering, captioning, and referring segmentation. However, existing 3D-MLLMs remain largely object-centric, limiting their ability to model fine-grained part structures that are essential for embodied interaction with 3D environments. In this work, we present PAR3D, a unified part-aware 3D-MLLM framework that enables models to understand, reason about, and ground both objects and their parts in 3D scenes. To enable training and evaluation of part-aware 3D scene understanding, we introduce ScenePart, a synthetic 3D scene dataset with part-level annotations and language instructions. We further develop Part-Aware 3D Representation Learning to enrich 3D visual representations with fine-grained part-level semantics, and propose Hierarchical Segmentation Query Generation to ground part targets via hierarchical object–part queries. Extensive experiments show that our method substantially improves part-level question answering and referring segmentation, while also achieving strong performance across object-level vision-language tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Understanding, 3D Vision-Language, Part-Aware Representation, Referring Segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bwv7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAR3D extends 3D-LLaVA toward part-level scene understanding. The authors introduce ScenePart, a synthetic dataset with object-part annotations and language instructions, and combine part-aware contrastive learning, representation-preserving regularization, and object-part segmentation queries. The model supports question answering and referring segmentation, with improvements on synthetic part benchmarks, a real-world ScanNet part test set, and object-level segmentation tasks.

Critical strengths
  1. ScenePart provides 800 scenes, approximately 44,000 part masks, and 273,000 language-task annotations for training and evaluating scene-level part understanding.
  2. On real-world part segmentation, PAR3D achieves 39.4 mIoU versus 33.2 against the strongest baseline, providing initial evidence of transfer from synthetic part supervision.
Critical weaknesses
  1. Section 3.4 uses separate object and part tokens and also adds supervision for the host-object mask. It would be useful to keep both mask targets but use two identical [SEG] tokens. Without this comparison, it is hard to tell how much the separate token types matter.
  2. In Table 4, adding ScenePart-200K to 3D-LLaVA’s training raises ScenePart-Seg mIoU from 11.1 to 51.8. I would like to see this version evaluated on part-level QA and real-world scenes to understand how much the full model adds when the training data are the same.
Questions for authors
  1. How do two general [SEG] tokens compare with [OBJ] and [PART] under identical object and part mask supervision?
  2. Reason3D also supports language-conditioned point-level mask prediction. Could the authors clarify why 3D-LLaVA was chosen as the base framework and whether the proposed components could be applied to other 3D-MLLMs?
7 comments
AuthorsOpenReview ↗

Response to Reviewer bwv7

Thank you for your careful review and constructive suggestions. We appreciate your recognition of ScenePart’s contribution to scene-level part understanding and PAR3D’s performance on real-world part segmentation. We address your questions and comments below.

Q1: Comparison with Two [SEG] Tokens under Identical Mask Supervision.

Thank you for this insightful question. We trained a baseline with two identical [SEG] tokens, keeping object- and part-mask supervision and all other training settings unchanged. PAR3D achieves higher overall mIoU, with improvements on both coarse and fine parts. These results support using distinct object and part tokens to reduce granularity conflicts and improve part grounding.

MethodAll mIoU ↑Coarse mIoU ↑Fine mIoU ↑
Two [SEG]60.160.145.7
Ours60.760.946.0

Q2: Choice of 3D-LLaVA and Applicability to Other 3D-MLLMs.

Thank you for highlighting Reason3D as an alternative framework. We clarify our framework choice and the applicability of our designs below.

  1. Choice of 3D-LLaVA. For each target task, Reason3D pretrains on data from other tasks before task-specific fine-tuning. In contrast, 3D-LLaVA unifies segmentation, QA, and captioning in one model. This matches our goal of extending unified multi-task understanding from objects to parts.

  2. Generalizability. Part-aware representation learning focuses on learning point-level visual features, while hierarchical queries extend LLM tokens for multi-granularity segmentation. Both designs can be applied to other 3D-MLLMs.

W2: Extended Ablations on Part-Level QA and Real-World Segmentation.

We appreciate your suggestion and have extended the Table 4 ablation to QA and real-world part segmentation, including 3D-LLaVA trained with ScenePart and the Utonia-matched baseline. With the same training data and Utonia encoder, our modules consistently improve performance across multiple benchmarks:

  1. Segmentation. Gains on ScanRefer, real ScanNet part segmentation, and all three ScenePart-Seg granularities.
  2. QA. Improvements in all four metrics on both ScenePart-QA and ScanQA.

Segmentation Ablation (mIoU).

MethodsScanReferReal ScanNet PartScenePart-Seg ObjectCoarseFineAll
3D-LLaVA43.331.229.09.04.211.1
+ ScenePart44.434.478.452.837.651.8
+ ScenePart + Utonia47.436.384.254.041.154.9
PAR3D (Full)49.939.489.660.946.060.7

Question Answering Ablation. C: CIDEr; B-4: BLEU-4; M: METEOR; R-L: ROUGE-L.

MethodsScenePart-QA CB-4MR-LScanQA CB-4MR-L
3D-LLaVA39.60.110.032.192.617.118.443.1
+ ScenePart177.245.643.780.692.116.218.243.2
+ ScenePart + Utonia183.052.344.681.992.815.618.444.1
PAR3D (Full)191.161.746.982.195.715.918.945.0
AuthorsOpenReview ↗

Follow-up on Author Responses and Request for Feedback

Thank you again for your insightful and constructive feedback. We have provided detailed responses and additional experiments addressing your concerns. For ease of review, we briefly summarize the key clarifications and findings below.

  1. Controlled two-token comparison (Q1). Under identical paired-mask supervision and training settings, [OBJ][PART] improves overall mIoU over two [SEG] tokens (60.1 → 60.7), with gains on both coarse and fine parts.

  2. Gains beyond ScenePart supervision (W2). With the same training data and Utonia encoder, our modules provide consistent gains across segmentation and QA benchmarks. Real ScanNet part mIoU improves from 36.3 to 39.4, and all four QA metrics improve on both ScenePart-QA and ScanQA.

  3. Applicability to other 3D-MLLMs (Q2). Part-aware representation learning targets point-level visual features, while hierarchical queries extend LLM tokens for multi-granularity segmentation. Both designs can be adapted to other 3D-MLLMs.

We would greatly appreciate your feedback on whether these comparisons address your concerns about the proposed designs. Please let us know if there are any remaining points we can clarify.

Thanks for the rebuttal. Some of my concerns are addressed. But the contribution of different [obj] and [part] tokens are limited, as it only brings a small miou improvement for both all miou and fine-part miou.

Reviewer_uboJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAR3D extends 3D-LLaVA for part-aware scene QA and referring segmentation, introducing ScenePart with 800 synthetic scenes, ~21K object masks, 44K part masks, and 273K language annotations. Using a frozen pretrained Utonia encoder, it adapts its query decoder through instance segmentation, part-aware contrastive learning, and feature-preservation regularization. During instruction tuning, OBJ and PART tokens predict host-object and part masks. Reported improvements span ScenePart, established object-level benchmarks, and a small manually annotated ScanNet part-segmentation test set.

Critical strengths
  1. Its paired object–part masks and language annotations address a scene-level supervision gap, raising 3D-LLaVA’s ScenePart-Seg mIoU from 11.1 to 51.8 (Table 4).
  2. PAR3D improves ScanRefer/Multi3DRefer mIoU from 43.3/42.7 to 49.9/53.4 and achieves 39.4 on the real ScanNet part test, versus 33.2 for the strongest baseline. These results support the overall training recipe but do not isolate individual component contributions.
Critical weaknesses
  1. Part-level baselines use released checkpoints, while PAR3D receives ScenePart supervision and a stronger encoder (Appendix F). In Table 4’s cumulative ablation, data contributes 40.7 of the 49.6-point improvement (~82%), while the final contrastive and hierarchical-query additions yield conditional gains of 0.7 and 1.3 points. Data- and backbone-matched comparisons are missing for ScenePart-QA and real-world part grounding. Although autoregressive tokens provide object context, separate mask queries share scene features (Eq. 7); a two-generic-token control would help distinguish hierarchical conditioning from extra capacity or paired supervision.
  2. Language-overlap metrics do not establish factual QA correctness, especially for counting and existence, and the paper lacks question-type semantic accuracy and a language-only control. The scene-level split is appropriate but does not establish asset- or template-disjoint evaluation, leaving generalization to unseen geometries and linguistic compositions unmeasured. Real-world evaluation covers only segmentation across 114 referring expressions in 52 scenes, with QA illustrated through selected examples, providing preliminary evidence rather than strong validation of robust part-aware reasoning and transfer.
Questions for authors
  1. Please compare PAR3D with 3D-LLaVA + ScenePart and a matched Utonia-based baseline on ScenePart-QA and the real ScanNet part test, fixing instruction mixtures and training budgets to isolate gains from the proposed losses and queries.
  2. Please clarify host-object conditioning beyond autoregressive context and compare [OBJ] → [PART] with two generic segmentation tokens under identical paired-mask supervision, including rates of predicted part points outside the host.
  3. Please report train/test scene and unique-asset counts, any overlap in meshes, floor plans, or language templates, and whether test scenes were excluded from both task-specific training stages.
  4. Please report ScenePart-QA semantic accuracy by question type with a language-only baseline, and paired confidence intervals for real-world segmentation gains over a data- and backbone-matched baseline.
3 comments
AuthorsOpenReview ↗

Response to Reviewer uboJ (Part 1/2)

We are grateful for your positive rating. Thank you for your detailed review and for recognizing ScenePart’s contribution to scene-level part supervision and PAR3D’s improvements on object-level benchmarks and real-world part segmentation. We address your questions below.

Q1: Data- and Backbone-Matched Comparisons on QA and Real-World Part Segmentation.

Thank you for suggesting these data- and backbone-matched comparisons. We have extended the Table 4 ablation to include QA, real-world part segmentation, and results at different granularities. With the same training data, Utonia encoder, instruction mixture and training budget, our modules consistently improve performance across multiple benchmarks:

  1. Segmentation. Gains on ScanRefer, real ScanNet part segmentation, and all three ScenePart-Seg granularities.
  2. QA. Improvements in all four metrics on both ScenePart-QA and ScanQA.

Segmentation Ablation (mIoU).

MethodsScanReferReal ScanNet PartScenePart-Seg ObjectCoarseFineAll
3D-LLaVA43.331.229.09.04.211.1
+ ScenePart44.434.478.452.837.651.8
+ ScenePart + Utonia47.436.384.254.041.154.9
PAR3D (Full)49.939.489.660.946.060.7

Question Answering Ablation. C: CIDEr; B-4: BLEU-4; M: METEOR; R-L: ROUGE-L.

MethodsScenePart-QA CB-4MR-LScanQA CB-4MR-L
3D-LLaVA39.60.110.032.192.617.118.443.1
+ ScenePart177.245.643.780.692.116.218.243.2
+ ScenePart + Utonia183.052.344.681.992.815.618.444.1
PAR3D (Full)191.161.746.982.195.715.918.945.0

Q2: Host-Object Conditioning and Comparison with Two [SEG] Tokens.

We appreciate this insightful question. We clarify the conditioning mechanism and compare the two token designs under identical paired-mask supervision.

Host-object Conditioning. The LLM conditions the [PART] query on the preceding [OBJ] through autoregressive context. We impose no additional spatial constraints and encourage the model to learn the object–part hierarchy from data.

Controlled Two-Token Comparison. We trained a baseline with two identical [SEG] tokens, keeping object- and part-mask supervision and all other training settings unchanged. PAR3D achieves higher overall mIoU, with improvements on both coarse and fine parts.

We report the out-of-parent rate (P): the mean per-sample percentage of predicted part points outside the predicted parent-object mask, measuring containment consistency between the two predictions. PAR3D reduces this rate from 7.57% to 3.64%.

Both models use the same paired supervision. Distinct token types further improve part grounding and object–part consistency.

MethodAll mIoU ↑Coarse mIoU ↑Fine mIoU ↑All OOP (P) ↓Coarse OOP (P) ↓Fine OOP (P) ↓
Two [SEG]60.160.145.77.578.406.75
Ours60.760.946.03.643.843.44
AuthorsOpenReview ↗

Response to Reviewer uboJ (Part 2/2)

Q3: ScenePart Dataset Splits and Train–Test Separation.

Thank you for raising this important question about ScenePart’s train–test separation.

  1. Scenes and assets. ScenePart contains 700 training scenes and 100 test scenes, with 2,356 and 331 unique 3D assets, respectively. Both the 3D-CoMPaT and 3D-FUTURE asset libraries are split into disjoint training and test pools. No object meshes or floor plans are shared between splits.
  2. Training. Test scenes are excluded from both training stages and from ScenePart-200K instruction generation.
  3. Language templates. Training and test instructions in ScenePart share the same question templates. We apply LLM-based refinement to diversify expressions and reduce reliance on fixed template wording.

Q4: Question-Type QA Evaluation and Language-Only Baseline.

We appreciate this constructive suggestion. We extend our evaluation with question-type metrics and a language-only baseline to better assess factual QA correctness.

  1. Existence and counting. We normalize Boolean and numerical answers and report accuracy.
  2. Color, spatial, and cross-object questions. Model-generated responses vary in wording and detail, making consistent mapping to canonical answers difficult. We report refined exact match (EM-R) as a complementary metric, following LEO [1] and Real-3DQA [2].

We also evaluate LLaVA-1.5-7B as a language-only baseline, without 3D inputs or additional ScenePart training. All scores below are reported as percentages.

MethodExist. ↑Count. ↑Color ↑Spatial ↑Cross-obj. ↑
3D-LLaVA76.9044.2110.4019.4512.30
Chat-Scene72.7542.5413.6011.7014.00
Grounded 3D-LLM77.7546.1913.3510.5011.70
LLaVA-1.5 (Text-only)76.8044.8421.3517.9013.90
PAR3D (Ours)98.1597.6090.1065.4561.95

The results highlight two points:

  1. Performance across question types. PAR3D achieves 98.15% and 97.60% accuracy on existence and counting, respectively, and substantially higher EM-R than all baselines on the remaining three types.
  2. Language-only comparison. The language-only baseline performs comparably to existing 3D-MLLMs on existence and counting and outperforms them on color. This suggests limited part-level understanding in these 3D-MLLMs.

For real-world segmentation, we report the expanded data- and backbone-matched comparisons in Q1.

[1] Huang et al. An Embodied Generalist Agent in 3D World. ICML 2024.

[2] Ma et al. Do 3D Large Language Models Really Understand 3D Spatial Relationships? ICLR 2026.

W2: Real-World Part-Aware Evaluation.

We appreciate your emphasis on real-world validation. Given the annotation cost, we prioritized manually annotated part masks for direct quantitative evaluation with mIoU. QA evaluation requires additional question–answer annotations and reliable assessment of answer correctness. Quantitative real-world QA evaluation remains future work.

AuthorsOpenReview ↗

Follow-up on Author Responses and Request for Feedback

Thank you again for your insightful and constructive feedback. We have provided detailed responses and additional experiments addressing your concerns. For ease of review, we briefly summarize the key clarifications and findings below.

  1. Data- and backbone-matched comparisons (Q1). With training data, encoder, instruction mixture, and training budget fixed, our modules provide consistent gains across segmentation and QA benchmarks. For example, PAR3D improves ScenePart-QA CIDEr from 183.0 to 191.1 and real ScanNet part mIoU from 36.3 to 39.4.

  2. Controlled two-token comparison (Q2). Under identical paired-mask supervision and training settings, [OBJ][PART] improves overall mIoU over two [SEG] tokens (60.1 → 60.7) and reduces the out-of-parent rate relative to predicted parent masks (7.57% → 3.64%).

  3. Question-type QA evaluation (Q4). PAR3D outperforms all compared 3D-MLLMs and the language-only baseline across all five question types. It achieves 98.15% existence accuracy and 97.60% counting accuracy, with higher EM-R on color, spatial, and cross-object questions.

We would appreciate your assessment of whether these additional evaluations address your main concerns. We would be happy to discuss any remaining questions.

Reviewer_he2MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly studies object and part understanding in 3D indoor scenes. Building on 3D-LLaVA, the authors propose PAR3D, aiming to use a unified framework for language-guided segmentation at both the object and part levels, while also supporting tasks such as question answering and description generation. To provide part-level supervision, the authors construct ScenePart, a synthetic dataset containing object–part correspondences and language instructions. In terms of the method, the authors adopt part-aware 3D representation learning and use “OBJ” and “PART” tokens to jointly predict the masks of the target part and the object it belongs to. Experimental results show that PAR3D outperforms the compared methods on part understanding tasks and is competitive on object-level benchmarks.

Critical strengths
  1. Putting object and part understanding in the same framework has practical value. Especially in real-world interactions, models need to understand not only objects but also their functional components. PAR3D can simultaneously handle tasks such as object segmentation, part segmentation, and question answering. When segmenting a part, it also predicts the object to which the part belongs, thereby taking the relationship between the two into account. This allows the model to understand and handle requests involving the local structures of objects.

  2. The results in real-world scenes reported in the paper provide preliminary support for training models with synthetic part supervision. This can be seen in Table 3: in the real ScanNet part evaluation, PAR3D achieves an mIoU of 39.4, higher than the best result of 33.2 among the compared methods. Detailed part annotation in real 3D scenes is relatively time-consuming, and such annotations are also very difficult to obtain at scale. If existing assets with part annotations can be used to construct synthetic scenes, and the learned capabilities can be transferred to real scan data, this could reduce reliance on manual annotation of real scenes.

Critical weaknesses
  1. Compared with existing work, this paper’s new contributions in terms of data and methods are still not clear enough. There has already been some research on scene-level part understanding. For example, OpenPart3D[1] has already combined objects with part annotations into scenes and provided language queries to study scene-level part segmentation; Search3D[2] also considers hierarchical representations of objects and parts. I do not seem to have found a discussion of these works in the paper. In terms of the method, part contrastive learning and dual-token queries also build on existing ideas. For example, PartField[3] uses contrastive learning to construct part features, while Reason3D[4] uses dual tokens to perform hierarchical segmentation from coarse regions to target objects. Although the specific settings of these works are not exactly the same as those of PAR3D, they are directly related to this paper’s core designs. I acknowledge the practical value of integrating object and part understanding into a unified 3D-MLLM. However, the paper still needs to explain more clearly what important content ScenePart adds compared with existing data, and what specific problems that have not yet been adequately addressed are solved by combining these designs. At present, my impression is that this work is closer to a combination and extension of existing ideas in the same field, so I still have reservations about its new contributions.

  2. The extent to which the proposed designs help with fine-grained part understanding is still not clear enough. In Table 4, adding ScenePart increases overall mIoU from 11.1 to 51.8, which is the largest single improvement in the ablation study. Meanwhile, the baselines in Table 2 lack equivalent part supervision, so the relatively large performance gaps may partly come from differences in training data. Although Table 4 includes 3D-LLaVA trained on ScenePart, it does not separately report its coarse-part, fine-part, and question-answering results. It is therefore still difficult to judge the specific benefits of the proposed designs for fine-grained understanding under the same supervision.

[1] https://arxiv.org/abs/2506.19331
[2] https://arxiv.org/abs/2409.18431
[3] https://arxiv.org/abs/2504.11451
[4] https://arxiv.org/abs/2405.17427

Questions for authors
  1. Please clarify the questions about new contributions raised in Weakness 1. Compared with works such as OpenPart3D and Search3D, what are the key new contributions of ScenePart and PAR3D? What specific gaps in data content or model capabilities do they fill? If my understanding of existing work or this paper’s contributions is incorrect, please also point this out. This question will directly affect my judgment; if the response can clearly explain the new contributions and provide support for them, I am willing to reconsider my score.

  2. Could the strong baselines in Table 2 be trained on the same ScenePart data, with results reported for coarse- and fine-part segmentation, and evaluated on real-world part segmentation tasks? Please also provide the breakdown of results and the part question-answering results for the 3D-LLaVA in Table 4 that uses the same data and encoder, to assess how much of the performance gap comes from differences in data and what gains the proposed designs bring under the same supervision.

  3. ScenePart contains 21K objects and 44K parts. How many of these objects have part annotations? For these annotated objects, could you provide the distribution of the number of parts per object, as well as statistics on part size and hierarchy depth? I would like to use this information to understand whether the data sufficiently cover complex structures and small parts.

  4. How exactly does the information from [OBJ] affect the mask prediction of [PART]? With parent-object supervision and other training settings kept unchanged, how does using two [SEG] tokens of the same type to predict the parent object and the part, respectively, compare with using [OBJ] / [PART]? Could you provide statistics on parts being localized to the wrong objects or predicted part masks extending beyond the ground-truth parent-object masks, to support the role of the hierarchical query design?

5 comments
AuthorsOpenReview ↗

Response to Reviewer he2M (Part 1/2)

Thank you for your careful review and constructive feedback. We appreciate your recognition of the practical value of unified object- and part-level understanding and the potential of synthetic part supervision for real-world transfer. Below, we clarify our contributions and provide additional experiments and dataset statistics to address your questions.

Q1: New Contributions Compared with Related Work.

Thank you for highlighting these relevant works. Our work differs fundamentally in its goal of unified, multi-task object- and part-level understanding within a single 3D-MLLM. We clarify the key distinctions below.

ScenePart (ours) vs. OpenPart3D.

  1. Task scope. The 3D-PU dataset from OpenPart3D targets language-guided segmentation only. ScenePart provides both segmentation and QA supervision for multi-task, multi-granularity scene understanding.
  2. Data construction. To support a broad range of questions, ScenePart adopts a richer data construction pipeline, including scene graphs to preserve spatial information and LLM-assisted question refinement.
  3. Availability. We could not find a public release of the 3D-PU dataset or its construction code, limiting further empirical investigation. We will publicly release the data construction code for ScenePart.

PAR3D (ours) vs. Search3D.

  1. Task scope. Search3D focuses on open-vocabulary segmentation. PAR3D is a unified, multi-task 3D-MLLM supporting both object- and part-level scene understanding.
  2. Reasoning capability. Search3D uses text-feature matching to retrieve and segment relevant regions. PAR3D can interpret complex language instructions and reason over scene content to generate textual answers and descriptions.

Our Contrastive Learning vs. PartField.

Contrastive learning is a common approach to learning discriminative part features, used in object-level 3D part perception methods such as PartField. In PAR3D, it serves a specific role as a proxy objective for introducing part-level supervision into Stage 1 visual backbone training. This helps alleviate the backbone’s object-centric bias and develop part-aware perception.

Our Hierarchical Queries vs. Reason3D.

  1. Design objective. Reason3D uses coarse-to-fine localization to find small objects in large scenes. Our queries address granularity conflicts between object- and part-level targets within a 3D-MLLM.
  2. Adaptive output. Reason3D adopts task-specific fine-tuning and output templates, with the output token predefined for each task. PAR3D learns to determine the target granularity and adaptively generate [OBJ] or [OBJ][PART] within a single jointly trained model.

We will expand the related-work discussion and clarify these distinctions in the revised manuscript.

Q2: Extended Ablation Studies across Tasks and Datasets.

We appreciate your suggestion and have extended the Table 4 ablation to include QA, real-world part segmentation, and results at different granularities. Given the computational cost, we cannot retrain all Table 2 baselines with ScenePart. We therefore focus on 3D-LLaVA, the base framework of PAR3D, to isolate the contributions of data, encoder, and model design. With the same training data and Utonia encoder, our modules consistently improve performance across multiple benchmarks:

  1. Segmentation. Gains on ScanRefer, real ScanNet part segmentation, and all three ScenePart-Seg granularities.
  2. QA. Improvements in all four metrics on both ScenePart-QA and ScanQA.

Segmentation Ablation (mIoU).

MethodsScanReferReal ScanNet PartScenePart-Seg ObjectCoarseFineAll
3D-LLaVA43.331.229.09.04.211.1
+ ScenePart44.434.478.452.837.651.8
+ ScenePart + Utonia47.436.384.254.041.154.9
PAR3D (Full)49.939.489.660.946.060.7

Question Answering Ablation. C: CIDEr; B-4: BLEU-4; M: METEOR; R-L: ROUGE-L.

MethodsScenePart-QA CB-4MR-LScanQA CB-4MR-L
3D-LLaVA39.60.110.032.192.617.118.443.1
+ ScenePart177.245.643.780.692.116.218.243.2
+ ScenePart + Utonia183.052.344.681.992.815.618.444.1
PAR3D (Full)191.161.746.982.195.715.918.945.0
AuthorsOpenReview ↗

Response to Reviewer he2M (Part 2/2)

Q3: Part Annotation Coverage and Structural Complexity in ScenePart.

This is an important question about ScenePart’s coverage of complex structures and small parts. 9477 of ScenePart’s 21K object instances have part annotations, comprising 44553 part masks across coarse and fine granularities. The table below reports part counts per annotated object, and part sizes measured by 3D bounding-box diagonals in meters.

StatisticCoarseFineTotal
Part masks207492380444553
Parts per object (min)202
Parts per object (max)51113
Parts per object (mean)2.192.514.70
BBox diagonal min (m)0.2260.054--
BBox diagonal max (m)3.2833.283--
BBox diagonal mean (m)1.3651.181--
  • Fine-part sampling. We randomly subsample fine-part masks to balance question counts across granularities. Some objects therefore retain no fine-part annotations.

  • Hierarchy depth. The original 3D-CoMPaT annotations contain nested part structures. For training, we flatten the part-to-part hierarchy, so both coarse and fine parts are treated as separate supervision targets directly associated with their host objects.

Q4: Hierarchical Query Design and Comparison with Two [SEG] Tokens.

Thank you for this insightful question. We clarify how [OBJ] informs [PART] and evaluate the benefit of distinct token types under identical paired-mask supervision.

Hierarchical Query Design. Our design can be understood through three aspects:

  1. Granularity-aware tokens: [OBJ] and [PART] explicitly distinguish object- and part-level queries.
  2. Autoregressive conditioning: The LLM conditions the [PART] query on the preceding [OBJ] through causal attention.
  3. Paired supervision: The two queries receive corresponding object- and part-mask supervision.

Controlled Two-Token Comparison. We trained a baseline with two identical [SEG] tokens, keeping object- and part-mask supervision and all other training settings unchanged. PAR3D achieves higher overall mIoU, with improvements on both coarse and fine parts.

As requested, we report the out-of-parent rate (GT): the mean per-sample percentage of predicted part points outside the ground-truth parent-object mask. PAR3D reduces this rate from 3.93% to 3.35%.

These results support using distinct object and part tokens to reduce granularity conflicts and improve part grounding.

MethodAll mIoU ↑Coarse mIoU ↑Fine mIoU ↑All OOP (GT) ↓Coarse OOP (GT) ↓Fine OOP (GT) ↓
Two [SEG]60.160.145.73.934.223.65
Ours60.760.946.03.353.613.08
AuthorsOpenReview ↗

Follow-up on Author Responses and Request for Feedback

Thank you again for your insightful and constructive feedback. We have provided detailed responses and additional experiments addressing your concerns. For ease of review, we briefly summarize the key clarifications and findings below.

  1. Contributions beyond related work (Q1). We clarified our distinct goal of unified, multi-task object- and part-level understanding within a single 3D-MLLM. ScenePart provides supervision for diverse question types, and PAR3D supports multi-granularity perception and reasoning.

  2. Gains beyond data and encoder (Q2). With the same training data and Utonia encoder, our modules provide consistent gains across multiple segmentation and QA benchmarks. For example, mIoU improves from 54.9 to 60.7 on ScenePart-Seg and from 36.3 to 39.4 on real ScanNet part segmentation.

  3. Controlled two-token comparison (Q4). Under identical paired-mask supervision and training settings, [OBJ][PART] improves overall mIoU over two [SEG] tokens (60.1 → 60.7) and reduces the out-of-parent rate relative to ground-truth parent masks (3.93% → 3.35%).

We would greatly appreciate your feedback on whether these clarifications and additional results address your main concerns. Please let us know if any points require further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.