acceptodds
Under review as a conference paper at ICLR 2027

Partner: Learning to Coordinate Parts for Promptable 3D Holistic Segmentation

Abstract

As 3D assets possess inherent hierarchical structures, semantic granularities vary across subjective user intents. Although existing interactive methods refine individual part masks, extending them to holistic segmentation remains challenging due to boundary overlaps or semantic missing. To address this gap, we propose Partner, a learning-based framework for real-time, interactive holistic segmentation. In each round, Partner updates the entire partition based on user prompts for a single part. Through its Semantic Coordinator and adaptive Split-Fill Predictor, Partner achieves unified, instruction-guided global adjustments, outperforming heuristic baselines in both refinement quality and interaction efficiency while serving as a versatile plug-and-play module.

Primary area:
generative models
Keywords:
promptable segmentation, 3D part segmentation, 3D asset creation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_863pOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Partner, a method for interactive repartitioning of 3D point clouds. It takes as input an initial segmentation representing object parts and provides an updated part segmentation given user feedback. It allows the user to provide positive/negative prompts to refine a selected part. The user can only provide feedback to one part segmentation at a time, but the method jointly readjusts all part masks. The resulting masks are mutually exclusive and combined cover the entire 3D shape. The approach follows a 2-stage pipeline. First, an interactive model refines a single mask, and then a holistic module refines the part masks given the initial segmentation and the refined single mask. Additional point-wise features are used from a PointTransformer V3 backbone to produce the final result.

Critical strengths
  • Interactive segmentation of 3D data is an important problem for 3D annotation and 3D asset editing.
  • The ability to refine the entire segmentation through corrections to individual parts can be useful, but most interactive object segmentation methods have this attribute inherently. They update multiple object masks jointly, as in AGILE3D or object/background masks. In fact, the challenge, especially for editing, is often the opposite: preventing unintended changes to other parts when correcting a single part.
Critical weaknesses
  • Comparison with relevant interactive 3D segmentation baselines, such as AGILE3D and NPISeg3D, is missing. The paper argues that methods such as AGILE3D require clicks on every object part, but differences in the interaction protocol do not justify excluding them. What matters is the segmentation quality (IoU) as a function of the number of user interventions. AGILE3D and NPISeg3D support joint updates of multiple object masks. After all, the proposed method also reports the number of parts refined. These methods could also be adapted to use automatically generated initial segmentations or sampled clicks from SAM, as in the proposed approach. Such comparisons are important given the relatively weak results on ScanNet, the most challenging dataset in the evaluation.
  • Evaluation protocol. The metric in Eq. 6 is not the standard in the community that evaluates IOU on a stricter 1-1 matching. It leaves over-segmentation of parts without penalty and uses only the best match. It also allows the same predicted part to be matched in multiple ground truths.
  • ScanNet performance. The performance on ScanNet is rather weak despite the pre-trained backbone on ScanNet, with only 42.5mIoU after refining 10 parts with 5 corrections per part. The IoU@T, where T is the number of parts, is confusing given that the standard in the community is to report IoU@UserFeedback
  • Dependence on initial partition.
  • The methodology is difficult to follow, and methodological choices are not sufficiently motivated.
  • The protocol is rather restrictive, the user is allowed to focus on one part at a time and hope the corrections will influence remaining segmented parts positively.

Liu, J., Zhou, P., Xiao, Z., Shen, J., Yin, W., Sonke, J.-J., & Gavves, E. (2025). Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes (NPISeg3D). Proceedings of the 42nd International Conference on Machine Learning (ICML 2025)

Questions for authors
  • Comparison with additional baselines is important to conclude which methods are more effective and should be preferred for interactive 3D segmentation and editing tasks.
  • Clarification of the chosen evaluation metric and evaluation under stricter metrics to account for incorrect part assignments and the number of predicted parts.
1 comment
AuthorsOpenReview ↗

Reply to reviewer 863p

Thank you for recognizing the importance of interactive 3D segmentation for annotation and asset editing, and the potential usefulness of refining the entire segmentation through corrections to individual parts. We address your specific concerns below, referring to the shared response where appropriate.

(W1-Q1) Regarding your assessment that joint mask prediction is not new and your question about method effectiveness: As clarified in C1, Partner's contribution is interactive editing within a holistic segmentation context, rather than segmentation from scratch or joint mask prediction itself. The reported same-source pipeline improves mIoU@10 by 5.5–18.2 points across four datasets. We will describe Partner's contribution more precisely in terms of this editing task.

(W1-Q1) Regarding missing AGILE3D/NPISeg3D comparisons and adaptation through automatic holistic segmentation results or sampled clicks: C6 details why conversion of holistic segmentation results to initial automatic prompts, background handling, and dynamic-target management require explicit rules. We believe that the measured performance differences would depend on several adaptation parameters, whose effects may also vary across datasets. This makes a fair comparison difficult.

(W2-Q2) Regarding matching direction, repeated matches, and over-segmentation penalties: C2 gives the GT-to-prediction definition and explains why fragmentation can lower best-match IoU while repeated matches remain allowed. Best-overlap and one-to-one metrics share the overall evaluation objective and provide complementary details. We will consider adding clearly defined stricter matching and part-count evaluation.

(W3.1) Regarding your assessment of limited ScanNet performance: Removing Stage 1 reduces mIoU@10 from 74.6 to 63.5 on PartObj-Tiny and from 81.7 to 75.0 on PartNeXt, supporting primary-mask refinement. These ablations do not isolate the cause of the ScanNet result. As shown in C5, using PointSAM initialization improves Partner's ScanNet result to 50.0, compared with 45.9 for PointSAM with heuristic refinement from the same initial value of 39.6. This demonstrates benefit from a stronger initial holistic segmentation result produced by another method, not a tested replacement of Stage 1.

(W3.2) Regarding reporting refinement rounds rather than actual user feedback: C3 clarifies that TT counts selection/refinement rounds, including repeated selections, rather than clicks or distinct parts. The current protocol compares refinement efficiency through the rounds needed to reach the same quality. We will consider adding cumulative actual-click distributions for each method at each reported checkpoint to make feedback costs more informative and intuitive.

(W4) Regarding dependence on an initial holistic segmentation result: As explained in C1, an existing holistic segmentation result is the task's starting point, not an incidental requirement. Compatibility with initial holistic segmentation results from other methods lets Partner benefit from strong automatic segmentation, as illustrated in C5. This compatibility allows the same editing framework to use initial holistic segmentation results from different sources and should not be viewed as a weakness of the method.

(W5) Regarding your assessment that the method is difficult to follow and insufficiently motivated: Thank you for this feedback. We would appreciate clarification of the specific passages or design choices you find unclear, so that we can address them directly and improve the presentation in the next version.

(W6) Regarding the single-part interaction protocol: Selecting one primary part makes the meaning of positive and negative prompts explicit and facilitates evaluation and comparison. In a multi-label interface, each click also specifies one part label, rather than the desired holistic segmentation result. Partner uses such local feedback to jointly update the holistic segmentation result. Multi-label methods can accumulate feedback for several parts within a round, so the protocols are not identical. Unintended changes to non-target parts are a separate evaluation issue addressed in C2.

Reviewer_A2H1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents PARTNER for promptable 3D holistic segmentation. Starting from a complete part partition, the user selects one part and provides positive and negative point prompts. A single-mask stage first updates the selected part. A Semantic Coordinator and a Split-Fill Predictor then update the full partition through merge, split, and boundary refinement. Training interactions are synthesized from static annotations. The method is evaluated on four datasets with several initial segmentation methods.

Critical strengths
  1. The problem is practical and clearly defined. Many 3D editing tasks need a complete, non-overlapping partition rather than one isolated mask. PARTNER connects local user input with global partition updates in a coherent way.

  2. The main controlled comparison is strong. PARTNER and the single-mask baseline use the same backbone, first stage, and initialization. PARTNER improves mIoU@10 by 5.5 to 18.2 points across four datasets. This gives solid evidence that the learned second stage is useful.

Critical weaknesses
  1. The value of the Split-Fill decision is not isolated. The paper does not compare the decision gate with the same split query always enabled, and it does not report separate results for merge, split, and boundary refinement. On PartNeXt, the variant without the stage-2 prompt scores 72.0, below the heuristic single-mask baseline at 74.6. This shows that the gain cannot be assigned to the Split-Fill decision alone; stage-2 prompt reinjection may also be an important factor. Simple feature-redistribution and matched learned multi-object baselines are still missing.

  2. The practical evaluation is still limited. The experiments use GT-guided simulated interactions, and the actual number of prompts used after early stopping is not reported. The main metric also does not directly measure part-count errors or damage to non-target parts. Some protocol details, including the default initialization, test assets, PartNeXt hierarchy, and overlap checks, remain unclear.

Questions for authors
  1. What method produces the initial partition for PARTNER in Table 2? Please explain how the reported ScanNet initializations of 24.2 mIoU for PARTNER and 39.6 for PointSAM were produced, and provide a matched-initialization comparison if possible. Please also clarify the test assets, PartNeXt hierarchy, and overlap checks.

  2. How does the same trained model perform when the split query is always enabled and the explicit decision gate is removed? Please report separate results for merge, split, and boundary-refinement cases, including part-count errors.

  3. How does each interaction affect non-target parts? Please report label changes, IoU loss, or cross-round regression, together with the actual number of prompts used after early stopping.

  4. Can you add a no-training feature-redistribution baseline using the same features and prompts? If a method such as AGILE3D can be adapted to this setting without extra supervision, how does it perform under the same initialization and prompt budget?

2 comments
AuthorsOpenReview ↗

Reply to reviewer A2H1

Thank you for recognizing the task's practical value. We address the requested ablations, protocol details, and additional evaluations below.

(W1.1-W1.3-Q2.1-Q4.1) Regarding the unisolated decision contribution, the always-on test, and missing redistribution baselines: C4 describes the planned removal of the complete decision mechanism and retraining.

(W1.2) Regarding your assessment of Stage-2 prompt reinjection and gain attribution: The paper's ablations already identify Stage-2 prompt reinjection as an important design. C4 reproduces the ablation table, including 72.0 without Stage-2 prompts versus 74.6 for the heuristic counterpart on PartNeXt. These results support using prompts directly for holistic refinement rather than attributing the full gain to the decision alone.

(W2.1-Q3.2) Regarding unreported actual prompts and the limits of simulated interaction: C3 clarifies round-based reporting and the cumulative actual-click distributions under consideration for each method at each reported mIoU@TT checkpoint, including executed rounds and stopping events.

(W2.2-Q2.2-Q3.2) Regarding part-count errors, separate merge/split/boundary results, and non-target damage or regression: C2 explains that mIoU covers all GT parts but does not separately measure these errors. We will consider dedicated part-count comparisons in the supplementary material.

(W2.3-Q1.1-Q1.2-Q1.3) Regarding default initialization, matched ScanNet initialization, test assets, hierarchy, and overlap checks: C5 explains how the default holistic segmentation result is produced and distinguishes the ScanNet initial values of 24.2 and 39.6. The matched-source PointSAM comparison is already reported there. For PartNeXt, the manuscript uses instance and coarser hierarchy annotations; the inspected preprocessing forms hierarchy targets by merging valid sibling masks under the same immediate parent.

(Q3.1) Regarding how prompts affect non-target parts: All prompts refer to the selected primary part. A positive prompt in a region currently assigned to another part indicates that the region should belong to the primary part, guiding its reassignment and reducing the competing part's region. A negative prompt reinforces that the region should not be covered by the primary part; Stage 2 determines its assignment among the non-target parts using the current holistic segmentation result and point features. Thus, prompts targeting one part can also guide updates to other parts.

(W1.3-Q4.2) Regarding missing AGILE3D comparisons and adaptation through automatic holistic segmentation results or sampled clicks: C6 details why conversion of holistic segmentation results to initial automatic prompts, background handling, and dynamic-target management require explicit rules. We believe that the measured performance differences would depend on several adaptation parameters, whose effects may also vary across datasets. This makes a fair comparison difficult.

Reviewer_A2H1OpenReview ↗

Follow-up on the Rebuttal and Rating Update

Thank you for clarifying how the initial partitions were generated. The ScanNet comparison from the same PointSAM initialization, where Partner reaches 50.0 mIoU versus 45.9 for heuristic refinement, helps establish the benefit of the full refinement pipeline.

Three questions remain open. First, the planned retraining ablation would be informative, but the rebuttal does not report the requested test on the same trained model with the split query always enabled and the decision gate removed. Second, the explanation of how prompts affect non-target parts is helpful, but the extent of those changes remains unmeasured. Third, the clarified stopping rules explain how rounds are counted, but the actual number of clicks each method needs to reach comparable segmentation quality remains unreported.

These results matter for assessing the independent contribution of Split-Fill and the practical cost of interaction. Given the remaining gaps, I am lowering my rating from 3 (Weak Accept) to 2 (Weak Reject).

Reviewer_yzTROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Partner addresses interactive holistic 3D segmentation by updating an entire partition from prompts targeting one part. It combines a single-mask predictor, a Semantic Coordinator, and a Split-Fill Predictor, trained with dynamically transformed annotations. Experiments on four datasets show improvements over single-mask models with heuristic post-processing.

Critical strengths
  1. The framework addresses a practical problem: coordinating local edits with the surrounding partition while supporting merging, splitting, and boundary refinement.

  2. Controlled comparisons using the same backbone, first-stage predictor, and initialization show consistent gains across four datasets, supporting the effectiveness of learned holistic refinement.

Critical weaknesses
  1. The evaluation metric does not enforce one-to-one matching or separately penalize unmatched predicted parts, limiting its assessment of partition quality. Ground-truth-guided prompts also provide limited evidence of real-user efficiency.

  2. The Split-Fill Predictor is retained in all architectural ablations. Its benefit beyond the split query’s existing empty-target supervision remains unclear.

Questions for authors
  1. Do the gains persist with one-to-one matching and part-count error metrics?

  2. How does performance change when the split query is always enabled and the decision head is removed?

  3. How are early stopping and repeated part selections counted in mIoU@T, and do the gains hold at equal actual click counts?

1 comment
AuthorsOpenReview ↗

Reply to reviewer yzTR

Thank you for recognizing the practical editing problem and the consistent gains in the reported same-source comparisons. We address the remaining evaluation and ablation questions below.

(W1.1-Q1) Regarding one-to-one matching, unmatched parts, and whether gains persist with count-aware metrics: C2 explains the current metric's role and the complementary information provided by stricter matching. Fixed GT averaging does not imply a correct predicted count. We will consider clearly defined stricter metrics and part-count comparisons.

(W1.2) Regarding real-user efficiency and GT-guided prompts: As discussed in C3, the simulated protocol applies shared rules and supports comparison of rounds required to reach the same segmentation quality. Actual-click distributions will provide a richer and more intuitive view of feedback costs. We will also consider a suitable realistic-use evaluation in the next version.

(W2-Q2) Regarding the role of the Split-Fill decision: As described in C4, we will design an ablation that removes the complete Split-Fill decision mechanism and retrain the network. In this variant, candidate masks will compete directly through feature matching, without an explicit split decision. We will add the results to the ablation table in the final version.

(Q3) Regarding how early stopping and repeated selections enter mIoU@TT, and gains at equal clicks: C3 describes both stopping levels, repeated refinement from the latest holistic segmentation result, and score padding without extra clicks. Every executed repeat selection counts as another round. We will consider reporting cumulative actual-click distributions at each reported checkpoint; equal-click gains are not established by the current round-based tables.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to shared concerns

Thank all reviewers for the constructive reviews. We address shared concerns below.

C1. Contribution

Partner provides interactive editing within a holistic segmentation context, rather than segmentation from scratch or joint mask prediction itself. We will clarify its contribution: prompts target part editing of an existing holistic segmentation result through merge, split, and boundary refinement, maintaining exhaustive coverage and mutual exclusivity. The same-source single-mask-plus-heuristic comparison shows mIoU@10 gains of 5.5-18.2 points across four datasets, supporting the full editing pipeline.

C2. Evaluation metric

Each GT part is matched independently to the prediction with maximum IoU, then averaged over all GT parts. This reflects merge, split, and boundary errors: fragmentation can leave the best prediction missing GT area and reduce IoU. The metric permits repeated matches and has no separate unmatched-prediction or count penalty. One-to-one evaluation emphasizes independent instance correspondence; unmatched handling and averaging also affect its score. Both metrics assess overall segmentation with complementary detail. We will consider stricter matching and part-count metrics in the appendix.

C3. Interaction efficiency

In mIoU@TT, TT counts selection/refinement rounds, including repeated selections, not clicks or distinct parts. Each round permits up to five prompts. Partner stops prompting at selected-part IoU 0.95 and stops the whole procedure when the worst-matched part reaches 0.9. Repeated refinement starts from the latest holistic segmentation result; later checkpoints after stopping retain the score without extra clicks.

Within the reported checkpoints, rounds needed to reach the same quality already compare refinement efficiency. Stronger refinement may also need fewer actual clicks. We will consider cumulative actual-click distributions for each method and checkpoint, including executed rounds, repeats, and stops, to enrich this comparison and make costs more intuitive. GT-guided simulation applies shared correction rules to reduce human variability, although trajectories can differ. We will also consider realistic-use efficiency evaluation.

C4. Decision and Stage-2 prompts

Current ablations do not isolate the Split-Fill decision beyond empty-target supervision. We will remove the complete decision mechanism, retrain the network with candidate masks competing through feature matching, and add this ablation to the final version. Retraining differs from same-checkpoint always-on testing and no-training redistribution.

Existing ablations establish Stage-2 prompt reinjection as important:

DatasetDefault @10w/o Stage-2 prompt @10
PartObj-Tiny74.664.8
PartNeXt81.772.0

On PartNeXt, 72.0 is also below the heuristic result of 74.6. These results support direct prompt conditioning of holistic refinement, not attributing all gains to the decision.

C5. Initialization

P³-SAM-based initialization generates candidate masks from initial automatic prompts, then filters and aggregates them into a holistic segmentation result. Table 2 pairs PointSAM/Partner (PointSAM init) and PartSAM/Partner (PartSAM init): sources differ between pairs but are shared within each pair.

On ScanNet, default Partner improves 24.2 to 42.5 mIoU. From PointSAM initialization at 39.6, Partner reaches 50.0 versus 45.9 with heuristic refinement (+4.1). Thus, 24.2 and 39.6 are different initializations; 50.0 and 45.9 share the PointSAM source. This uses another initial holistic segmentation result, not a Stage-1 replacement.

C6. Related methods and adaptation

AspectPartnerAGILE3DAdaptation needed
Starting contextExisting holistic segmentation resultAccumulated region-labeled clicksInitial automatic prompts: count/placement/order; sparse clicks may lose mask information.
Click labelPositive/negative for one primary partRegion IDAccount for automatic/user clicks; delete/relabel old clicks; manage merge/split IDs.
Negative feedback / backgroundNot primary; no background classImplicit negatives; background ID 0Adapting to our background-free protocol omits AGILE3D's native background component.

Count GT-based changes to initial automatic prompts or new targets as extra information.

Adaptation parameters may affect performance differently across datasets, making multi-dataset validation and fair comparison difficult. Partner edits existing holistic segmentation results; its contribution does not conflict with AGILE3D or NPISeg3D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.