acceptodds
Under review as a conference paper at ICLR 2027

Partita: Multi-Granularity Promptable 3D Segmentation from Hierarchical Supervision

Abstract

Promptable 3D segmentation involves targets at different granularities, especially when proposing masks from a single click. However, single-granularity annotations limit candidate coverage and target diversity for interactive segmentation. We introduce Partita, a promptable 3D segmentation model that learns multi-granularity prediction and interactive refinement from hierarchical supervision. To provide this supervision at scale, we develop an automatic annotation pipeline that uses a vision-language model (VLM) to organize existing part annotations into nested hierarchies through recursive grouping and correction, yielding PartNeXt-XL, a dataset with 4.20M region masks across about 168K objects. Partita fuses low-resolution prompt-aware features with high-resolution geometric features to resolve finer parts and support efficient interactive refinement. Whereas prior promptable 3D methods supervise a single target mask per prompt, Partita uses multi-granularity matching to jointly supervise predictions across multiple levels. Experiments demonstrate improvements in interactive segmentation and multi-granularity prediction, as well as competitive automatic proposal generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point Clouds, Multi-Granularity Segmentation, 3D Data Annotation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Y1LkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Partita addresses single-click, multi-granularity 3D part segmentation. It constructs PartNeXt-XL by recursively grouping existing part annotations with a VLM, trains multiple mask candidates against the nested targets containing a click, and combines low-resolution prompt interaction with a high-resolution geometry branch.

Critical strengths
  1. The data and training contributions are supported by useful controlled ablations. Table 3 tests hierarchical annotations under both fixed-mask and fixed-object settings, and Table 11 in the Appendix separates set matching from single-target supervision on the same data. These results strengthen the case that the improvement is not simply due to adding more training objects.
  2. The geometry branch offers a concrete quality–memory trade-off: Table 4 reports 11.18 versus 20.25 GiB for full-resolution interaction, with broadly comparable overall IoU.
Critical weaknesses

The semantic quality of the new supervision remains insufficiently quantified. Appendix A.1 verifies structural properties, but disjointness, coverage, and connectivity do not establish that a grouping corresponds to a meaningful part. Appendix E explicitly acknowledges this limitation. The manually reviewed Partita200 benchmark is initialized by the same annotation pipeline (Appendix C.1), while no quantitative human audit of the training hierarchies is reported. Because scalable meaningful hierarchy construction is a central contribution, the distinction between structurally valid and semantically valid annotations affects the interpretation of the dataset and the Partita200 gains

Questions for authors

What fraction of a randomly sampled set of training hierarchy nodes is judged semantically valid by independent human inspection? Please describe any existing audit, its sampling procedure, and how often manual review changed Partita200's initial hierarchy.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Y1Lk

We thank the reviewer for recognizing our controlled ablations and the geometry branch’s quality–memory trade-off, and for the helpful suggestions on evaluating annotation quality.


W, Q1. Semantic quality of the generated training hierarchies.

We agree that structural validity does not guarantee semantic validity. As acknowledged in our Discussion and Limitations, semantic verification within the automatic annotation pipeline relies on the VLM’s capabilities and therefore cannot strictly guarantee that every generated grouping is semantically sound. We therefore conducted a human audit during the rebuttal period, randomly sampling 1,000 generated nodes (merged masks) for manual inspection. Of these, 92.6% were judged semantically valid. This result supports the use of the generated annotations as training supervision. We describe the sampling and review protocol below.

We uniformly sampled 1,000 newly generated non-leaf nodes from our training set and divided them equally between two independent reviewers. The interface showed each target region in the context of the complete object and allowed reviewers to adjust viewpoints and inspect the original constituent parts. A region was judged semantically valid if it formed a meaningful part or component whose constituents had reasonable functional, geometric, or compositional relationships. The two batches yielded validity rates of 93.4% (467/500) and 91.8% (459/500).

We further examined the causes of semantically invalid nodes and identified three categories: (1) semantically inappropriate groupings produced by our annotation pipeline, where the grouped parts do not form a meaningful part or component; (2) semantically incoherent whole objects, appearing as arbitrary assemblies of disorganized mesh fragments; and (3) semantically inappropriate atomic parts in the original annotations, arising from HY3D-Bench-Part's automatic merging of small-area components into adjacent larger components. The statistics are summarized below:

AssessmentPercentage
Semantically valid92.6%
Invalid: semantically inappropriate groupings3.8%
Invalid: semantically incoherent whole objects2.0%
Invalid: semantically inappropriate atomic parts1.6%

Q2. Construction and manual review of Partita200.

Appendix C.1 provides only a brief description of this process, so we clarify it here. Partita200 is a curated subset of automatically annotated objects. Specifically, one author visually inspected the candidate objects and their generated hierarchies, retaining objects with reasonable groupings and leaving their annotations unchanged. Objects with questionable groupings were discarded. Selection was stratified by maximum hierarchy depth, with 50 objects each at depths 2, 3, 4, and 5. We retained only textured assets spanning diverse categories, such as furniture, household items, machinery, vehicles, characters, and buildings. All selected objects were excluded from the training set by asset ID.


W. Additional evidence for the effectiveness of our annotations.

We further assess the effectiveness of our annotations through experiments on PartNetM. As described in the paper, PartNetM is a subset of PartNet whose hierarchies originate from human annotations. Following the fixed-mask-budget and fixed-object settings of Table 3, we show that our multi-granularity annotations improve Match mIoU and AP on this human-annotated multi-granularity benchmark, while AR remains comparable, as shown below:

Training annotationsObjectsMasksMatch mIoUAPAR
Original annotations40.47k0.74M60.4116.6535.75
+ Multi-granularity40.47k1.00M67.6819.1035.66
Original annotations54.65k1.00M61.2416.7535.78

Please let us know if any points remain unclear or if you have further questions. We would be happy to provide additional clarification.

Dear Reviewer Y1Lk, as the discussion period ends today, we would be grateful if you could share any further feedback on our rebuttal, if time permits. Please let us know if any concerns remain or further clarification would be helpful. Thank you for your time and consideration.

Reviewer_DMgDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the task of promotable 3D segmentation on single 3D objects. To improve performance on this task vs. the existing baselines, the authors introduce PartNeXt-XL, a large scale multi-granularity dataset. To annotate the dataset, a VLM is leveraged to recursively groups connected-component parts into nested hierarchies using Set-of-Mark renderings and adjacency information. Their proposed model is then trained on the new dataset, achieving state-of-the-art results across various tasks and benchmarks.

Critical strengths
  • The quantitative performance of the proposed method in good, outperforming competitive methods with clear margins
  • The paper contributes a substantial dataset which is a substantial part of training foundation models, making it a valuable contribution
  • The paper contains an extensive and broad evaluation with experiments across 3 tasks with 4 baselines and thorough ablations
Critical weaknesses
  • An important claim of the paper is that scaling the data drives the gains vs. the baseline methods. However, this claim is not fully validated, since the baselines are mostly compared by using their released weights. With no data-fair and architecture-fair comparison, this claim does not fully hold up.
  • The paper lacks a thorough data scaling analysis: At what point does scaling the data no longer help improving the results? How much data achieves significant improvements? What's the minimal amount of data to outperform the baselines?
  • The architectural novelty is limited, since the paper mostly combines existing components (SAM decoder, DETR-style set matching, U-Net branch). While not a blocking factor, it reduces the novelty to the data scaling
Questions for authors
  • How does a scaling curve showing performance vs. number of training objects look like?
  • What results do controlled comparisons with a fixed architecture and swapped training data, in both directions, yield?

It would be appreciated to see thorough analyses for these two points.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DMgD (Part 1/2)

We thank the reviewer for the constructive feedback and for recognizing the value of our dataset, the quantitative results, and the breadth of our evaluation. Below, we provide data-fair and architecture-fair comparisons, along with performance results across different training data scales.


W1, Q2. Data-fair and architecture-fair comparisons.

We would like to clarify that we do not attribute our gains to the number of training objects. Our own training set contains about 190K objects, whereas PartSAM is trained on about 500K and P³-SAM on 3.7M. Note that our encoder is initialized from P³-SAM weights (Sec. 4.1); with Sonata initialization instead, IoU@1 on PartObjaverse-Tiny changes from 73.46 to 71.83 at 100k updates (Table 12).

Our emphasis is on expanding the diversity of segmentation granularities per object. The performance gains from our multi-granularity annotations, set-to-set supervision, and decoder design are demonstrated by the ablation studies in Tables 3, 11, and 4, which were also acknowledged by Reviewer Y1Lk. To further address your concerns about fair comparisons, we provide data-fair and architecture-fair comparisons below.

1. Data-fair comparisons

Fixed data, different supervision. Table 11 compares single-target and set-to-set training on the same multi-granularity data. Set-to-set supervision improves Partita200 Match mIoU from 68.97 to 76.74 and AP from 33.28 to 39.38, demonstrating the benefit of set-to-set supervision.

Fixed data, different decoder architectures. Our decoder uses a SAM-style two-way interaction design similar to PartSAM’s, but adds a dedicated high-resolution geometric branch. Table 4 provides complementary evidence on decoder design through comparisons with fixed data: our design achieves an overall IoU of 73.46, compared with 68.17 without the geometric branch.

Fine-tuning PartSAM and S²AM3D on our data. We fine-tuned both methods on our data for 50k steps, starting from their respective released checkpoints, and evaluated them on Partita200.

We first consider PartSAM, which, like our method, predicts multiple masks at different granularities from a point click. We compare two settings: fine-tuning on our data using its native single-target supervision, and fine-tuning on the same data using our training recipe. Compared with the released weights, the former improves AP and AR but reduces Match mIoU, while the latter yields clear improvements across all three metrics.

PartSAM settingMatch mIoUAPAR
Released weights51.4822.6236.57
Our data47.1324.6540.63
Our data + training recipe67.2926.4342.08

These results highlight the importance of both multi-granularity training data and a suitable training recipe for effectively exploiting these annotations.

We also evaluate S²AM3D, which controls segmentation granularity through an explicit scale parameter: given a point click and a specified scale, it predicts a single mask. We retain this original setup during fine-tuning. Fine-tuning on our data yields a modest improvement in Match mIoU. AP and AR are not reported because S²AM3D does not provide confidence scores for its predicted masks.

S²AM3D settingMatch mIoU
Released weights54.19
Our data56.40

2. Architecture-fair comparisons

Training our model on the baselines’ original training data is constrained by two factors:

  1. Multi-granularity annotations. Our hierarchy-aware training recipe requires nested masks at multiple granularities for the same click. However, the baselines’ training data generally do not provide such annotations.
  2. Data availability. We do not have access to the complete training datasets and annotations used by the baselines.

We therefore conduct controlled experiments on our data, keeping the architecture fixed while varying the training annotations.

Fixed architecture, different annotations. Table 3 compares original and multi-granularity annotations under fixed-object and fixed-mask budgets. We additionally evaluate these models on PartNetM, whose hierarchies are human-annotated. Multi-granularity annotations improve Match mIoU and AP on both benchmarks under both controls.

Training annotationsObjectsMasksPartita200 MatchPartita200 APPartNetM MatchPartNetM AP
Original annotations40.47k0.74M60.9032.3360.4116.65
+ Multi-granularity40.47k1.00M76.7439.3867.6819.10
Original annotations54.65k1.00M62.4531.2261.2416.75
AuthorsOpenReview ↗

Response to Reviewer DMgD (Part 2/2)

W2, Q1. Performance versus the number of training objects.

We evaluate how Partita’s performance varies with training set size using 10%, 20%, 40%, and 100% of our data, with all models trained for 50k steps under the same training settings.

Training subsetObjectsPartObj-Tiny IoU@1Partita200 Match mIoU
10%19.02k68.2371.50
20%38.05k69.4574.10
40%76.10k70.7575.30
100%190.24k71.4475.89

Most of the observed improvement is achieved using 40% of the training data, and performance continues to improve as more data is added.


W3. Novelty and contributions.

Our focus extends beyond architectural novelty to expanding granularity diversity and enabling effective multi-granularity 3D segmentation. We pursue this through three complementary contributions: automated hierarchical annotation, hierarchy-aware training and efficient decoding, and multi-granularity evaluation datasets and protocols. The comparisons above demonstrate the gains from our annotations, supervision strategy, and decoder design.


Please let us know if any points remain unclear or if you have further questions. We would be happy to provide additional clarification.

Reviewer_DMgDOpenReview ↗

Response to Rebuttal

I thank the authors for their detailed rebuttal. The fine-tuning experiments and scaling study address my main concerns, supporting the claim that granularity diversity rather than raw data scale drives the gains. I therefore raise my score and support acceptance.

Thank you for your thoughtful review and for considering our responses. We appreciate your time and constructive feedback, and will incorporate the relevant clarifications and findings into the final version.

Reviewer_xZcfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Partita addresses the ambiguity of segmentation granularity in interactive 3D segmentation by predicting multiple masks at different granularities in a single forward pass. To support this, the authors introduce a VLM-based hierarchical annotation pipeline that automatically organizes existing fine-grained part annotations into coarse-to-fine hierarchies, resulting in the PartNeXt-XL dataset with 168K 3D objects and 4.2M hierarchical masks. Combined with a hierarchy-aware training strategy, this approach enables effective multi-granularity segmentation and achieves state-of-the-art (SOTA) performance.

Critical strengths

The key contribution of this paper is its hierarchy-aware training recipe, which addresses granularity ambiguity in interactive segmentation by enabling the model to predict five candidate masks for a single click. During training, all ground-truth masks containing the clicked point are collected across different hierarchical levels. A randomly sampled mask serves as the primary target and is assigned to the prediction with the lowest mask loss, while the remaining predictions are matched to other ground-truth masks using hungarian matching. This strategy encourages the model to generate diverse candidate masks at different levels of granularity, enabling multiple plausible segmentation results from a single user click. As a result, the proposed method achieves state-of-the-art (SOTA) performance on interactive 3D segmentation benchmarks.

Critical weaknesses

One major weakness of this paper is the lack of sufficient implementation and training details, which makes it difficult to fully understand and reproduce the proposed method. In particular, the description of the mask head is rather limited. Although the authors mention that the projected granularity queries generate candidate masks through pointwise dot products with the fused features, the exact mask generation process remains unclear. For example, how are the resulting dot-product scores converted into the final segmentation masks? Is a fixed threshold applied to obtain binary masks, or is another mechanism used? The authors should provide a more detailed explanation of this process. Furthermore, the paper provides insufficient details about the training procedure and hyperparameter settings, making it challenging to reproduce the reported results. I encourage the authors to provide a more comprehensive description of the mask prediction mechanism, along with detailed training configurations and implementation specifics, to improve the clarity and reproducibility of the work.

Questions for authors
  1. Correspondence between granularity queries and mask granularity: The authors introduce multiple granularity queries to predict masks at different levels of granularity. Is there a consistent correspondence between individual queries and specific granularity levels? For example, does a particular query consistently predict finer-grained masks, while another predicts coarser-grained masks?

  2. Explicit hierarchical relationships among queries: As illustrated in the supervision component of Figure 3, there appears to be no explicit constraint enforcing hierarchical containment relationships among the masks predicted by different queries. Have the authors explored imposing such constraints to encourage a consistent fine-to-coarse hierarchy among the predicted masks? If so, how would this affect segmentation performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xZcf

We thank the reviewer for the constructive feedback and for recognizing the effectiveness of our hierarchy-aware training strategy. Below, we clarify the implementation and training details and provide additional experimental evidence on query specialization and hierarchical containment.


W1. Implementation and training details.

We would like to kindly point out that, due to space limitations in the main text, the training and implementation details are provided in Appendix B: architecture (B.1), supervision and losses (B.2), training data (B.3), and training setup, hyperparameters, augmentation, and interaction rollout (B.4).

For the mask head, the upsampled prompt-conditioned features and geometric features are concatenated and fused by a two-layer MLP (608→512→512608\rightarrow512\rightarrow512) to obtain point features fif_i. Each granularity query is projected by its own three-layer MLP into a 512-dimensional mask vector wkw_k. The pointwise dot product zki=wk⊤fiz_{ki}=w_k^\top f_i gives the mask logits. Each binary mask is then obtained using a fixed threshold: points with zki>0z_{ki}>0 (equivalently, sigmoid probability >0.5>0.5) are foreground.


Q1. Query–mask granularity correspondence.

Our set-to-set matching does not preassign queries to granularity levels, allowing their roles to emerge through multi-granularity supervision. In practice, the predicted masks exhibit a consistent correspondence between query identity and relative granularity.

Here, we analyzed all 30,549 test clicks on PartNetM and Partita200, ranking the five masks by foreground point count from 1 (smallest) to 5 (largest).

QueryPartNetM mean rankPartita200 mean rank
Q14.524.78
Q22.012.00
Q31.021.01
Q44.384.19
Q53.083.03

The distributions are strongly concentrated: Q3 predicts the smallest mask in 98.5% / 98.9% of cases, Q2 the second-smallest in 96.3% / 97.9%, and Q5 the middle-sized mask in 88.2% / 96.0% on PartNetM / Partita200, respectively. Q1 and Q4 favor coarser masks.


Q2. Explicit hierarchical containment constraints.

We observe that our model naturally learns containment relationships from nested ground-truth masks without explicit hierarchical containment constraints, achieving mean containment coverage of 94.44% / 96.63% on PartNetM / Partita200. We compute coverage as ∣A∩B∣/min⁡(∣A∣,∣B∣)|A \cap B| / \min(|A|, |B|) for all pairs of predicted masks from each click and report the mean.

We also explored a nesting regularizer in early experiments, which penalizes parts of a smaller predicted mask that lie outside a larger one, thereby encouraging containment. We evaluated the early models without / with this regularizer using the same test clicks:

DatasetCoverage w/o nesting (%)Coverage w/ nesting (%)Match mIoU w/o nesting (%)Match mIoU w/ nesting (%)
PartNetM96.8997.20 (+0.31)62.1061.87 (−0.23)
Partita20098.6298.92 (+0.30)58.8859.51 (+0.63)

These early experiments show slightly higher containment coverage but no consistent segmentation benefit from the nesting regularizer, so we retain the simpler training objective.


Please let us know if any points remain unclear or if you have further questions. We would be happy to provide additional clarification.

Thank you for the clarification. After reading the rebuttal, I believe my primary concerns have been adequately addressed, and I am inclined to support acceptance.

Thank you for your thoughtful review and for considering our responses. We appreciate your time and constructive feedback, and will incorporate the relevant clarifications and findings into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.