acceptodds
Under review as a conference paper at ICLR 2027

PartIMap: Online Open-Vocabulary 3D Panoptic Mapping as Partition Inference

Abstract

Online open-vocabulary 3D panoptic mapping is a fundamental task in scene understanding. The central difficulty is fusing partial and noisy 2D masks from many views into consistent 3D instances. Existing online systems fuse with hand-designed rules that lack an explicit objective, which discard the uncertainty of partial observations, lead to potentially sub-optimal merging, and require special treatment if estimated poses change. We present PartIMap, which treats mask fusion as probabilistic inference. PartIMap maintains a probabilistic voxel map in which semantics are von Mises-Fisher distributions on the descriptor sphere, kept per instance and per voxel, and instance geometry is a per-voxel Beta occupancy state. Deciding which masks belong to the same object is cast as maximum a posteriori inference of a partition on a mask graph. This partition model retains probabilistic data association as its first-order relation while capturing higher-order relations such as transitivity, and yields soft membership weights from the same posterior. Every fused quantity is an additive sum over its constituent observations, and the map can be updated effectively and efficiently by re-merging when a live SLAM system closes a loop. PartIMap is zero-shot and requires no supervised training. On ScanNetV2 and Replica it outperforms both online and offline methods, improving the strongest online baseline by 3.2 AP and 3.9 PRQ(T) on ScanNetV2 at comparable speed, degrades the least when poses come from live SLAM, and exposes per-voxel uncertainty. Codes will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scene understanding, online 3D panoptic mapping, open-vocabulary

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HG7DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents PartIMap, an online, zero-shot open-vocabulary 3D panoptic mapping system that takes an RGB-D stream as input. Instance semantics are stored as von Mises-Fisher distributions, together with a per-voxel language grid that is decoupled from merge decisions, and instance geometry is a per-voxel Beta occupancy weighted by membership. Mask merging is cast as MAP inference of a partition on a mask graph, i.e., correlation clustering. Edge evidence is a hand-set linear map of four cues, solved by greedy agglomeration, and membership weights are derived from the same posterior. The method achieves state-of-the-art results on ScanNetV2 and Replica.

Critical strengths
  1. The experimental results are Section 3.3 analyzes the log-posterior of a partition on the mask graph. The greedy merge score is the exact change in that objective, and the membership weights used to fuse semantics and occupancy come from the same posterior. In Table 4, replacing probabilistic data association plus the supporter rule with this partition raises ScanNet AP from 23.7 to 24.8 and PRQ(T) from 48.78 to 49.60.

  2. Section 3.4 keeps raw observations so that a loop closure re-integrates the map and runs the same partition step. Table 2 gives OnlineAnySeg this re-fusion and leaves its own merge rules unchanged. PartIMap still leads on eight of the ten segmentation metrics, with ScanNet AP 20.0 against 17.6 and PRQ(T) 49.78 against 45.02, and the frame rate stays at 6.9 FPS.

Critical weaknesses

The writing is clear and easy to follow, and I only have some minor issues:

  1. Algorithm 2 replays every depth map and recomputes the full containment matrix, so the cost grows with sequence length. The reported 6.9 FPS does not show the latency of one loop closure or the memory growth on long sequences.

  2. The OnlinePG numbers in Table 1 are taken from the original paper, and the 2D backbone is not stated.

  3. TT and sstays_{stay} are tuned under probabilistic data association and then reused for the partition model in Appendix D. How this affects rows 1 and 3 of Table 4 is not discussed.

  4. Section 3.2 says a wrong merge does not pollute the language grid. Decoding still adds the instance statistic to the voxel vector, so the final label still depends on the merge.

  5. The geometric cue uses FCGF, a supervised pretrained 3D feature, so the claim of requiring no supervised training needs a narrower statement.

  6. Decl. favors methods with low absolute accuracy. The authors note this. Absolute drops should be reported as well.

  7. The text has grammatical errors, including “uncertain observations contributes” in Section 1, “Our hyperparameters therefore transfers” in Appendix D, and “Baseline methods commonly over-segments” in Appendix F. AP is reported to one decimal and the other metrics to two.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HG7D (1/2)

We thank the reviewer for the careful reading, and for noting that the greedy merge score is the exact change in the objective, that the membership weights come from the same posterior, and that PartIMap leads OnlineAnySeg even when OnlineAnySeg is given our re-fusion. Results marked (*) are new experiments. Please also see the summary in our General Response.

W1: Latency of one loop closure and memory growth on long sequences.

Latency. A loop closure costs about 5 s on small scenes and early in large ones, and up to about 30 s late in large scenes; the median over the 27 loop-closure events in our 10 ScanNetV2 sequences is 20 s. As the reviewer notes, Algorithm 2 replays every integrated depth map and recomputes the full containment matrix, so this cost grows with sequence length. We revise the sentence in Section 4.2 that the frame rate is unchanged under SLAM and report loop-closure latency separately.

Memory. Memory grows linearly at about 1.6 MB per keyframe on average, so a 24 GB GPU holds about 14k keyframes; the longest ScanNetV2 sequences have fewer than 1k keyframes, and ScanNet++ sequences with up to 1.3k keyframes run to the end (response to Reviewer Fjmw, 3/4).

Re-integrating only the frames whose corrected pose changes beyond a tolerance, and recomputing containment only for the affected instances, would bound this cost; we list it as future work.

Revision: latency and memory are reported in the new Section 4.4; the cost growth and its remedy in Appendix G.

W2: The OnlinePG numbers in Table 1 and its 2D backbone.

We took the OnlinePG numbers from the original paper because its code is not released. OnlinePG uses LSeg, the same vision-language model as ours, and EntitySeg [1] as the class-agnostic segmenter. The Table 1 caption now states this.

W3: TT and sstays_{\mathrm{stay}} are tuned under probabilistic data association and reused for the partition model; effect on rows 1 and 3 of Table 4.

The reuse favors the PDA rows, as the hyperparameters were selected in that setting. sstays_{\mathrm{stay}} has no separate effect in the partition model: λab−θ=T(sab−τ)\lambda_{ab}-\theta=T(s_{ab}-\tau) with τ=sstay+θ/T\tau=s_{\mathrm{stay}}+\theta/T, so sstays_{\mathrm{stay}} and θ\theta act only through τ\tau, which Table 5 sweeps via θ\theta. Re-tuning TT for the partition model does not make a significant change, as shown in the Sweep of TT (*):

ScanNetV2Replica
TTAPmIoUmAccPRQ (T)PRQ (S)APmIoUmAccPRQ (T)PRQ (S)
125.0248.7069.2850.8244.8116.1355.8667.5843.4224.45
225.9649.2670.1150.9045.3116.3756.1267.8043.4224.39
4 (inherited)25.7249.0670.3351.8444.9016.4156.1067.8043.5024.45
825.0049.0570.3250.9344.7616.2456.1067.7843.4324.38

Revision: Appendix D states that sstays_{\mathrm{stay}} acts only through τ\tau; the TT sweep is added to Appendix E.

AuthorsOpenReview ↗

Response to Reviewer HG7D (2/2)

W4: A wrong merge does not pollute the language grid, but decoding adds the instance statistic, so the label still depends on the merge.

We agree. Only the stored grid (Gv,Nv)(\mathbf G_v,N_v) is independent of merges; the decoded direction norm(Gv+Rk(v))\mathrm{norm}(\mathbf G_v+\mathbf R_{k(v)}) depends on the merge through the instance resultant Rk(v)\mathbf R_{k(v)}, in which each member is weighted by its membership weight. Section 3.2 now reads: "Because the grid is written independently of grouping, an incorrect merge does not alter the stored grid; the decoder balances evidence from the grid and the instance statistics."

W5: FCGF is a supervised pretrained 3D feature, so the claim of requiring no supervised training needs a narrower statement.

PartIMap trains and fine-tunes nothing and uses only frozen pretrained models; several of them, including LSeg, CropFormer, and FCGF, were trained with supervision by their authors. In our paper, "zero-shot" means that we add no additional training or fine-tuning on target datasets, not that the pretrained components were trained without supervision. We revised the abstract, Section 1, and the Ethics statement to say this.

W6: Decl. favors methods with low absolute accuracy; report absolute drops as well.

We added the absolute drops (ground-truth poses minus live SLAM; lower is better) to Appendix E:

ScanNetV2 ↓Replica ↓
MethodAPmIoUmAccPRQ (T)PRQ (S)APmIoUmAccPRQ (T)PRQ (S)
Open3DIS1.962.243.761.111.280.992.152.081.530.79
PanoGS3.884.406.214.305.201.031.100.564.751.85
OVO4.924.111.485.671.740.670.000.450.680.99
ConceptGraphs4.862.682.991.935.250.421.960.263.730.59
OnlineAnySeg4.931.962.202.885.211.120.850.721.830.93
OVI-MAP2.113.933.912.702.431.882.344.622.472.77
PartIMap (ours)5.680.803.102.065.230.380.630.402.11-1.59

Compared with OnlineAnySeg, PartIMap drops less on six of ten metrics. Among all methods, PartIMap has the smallest drop on ScanNetV2 mIoU and on Replica AP and PRQ (S). Like Decl., absolute drops favor methods with less room to fall. Under live SLAM, PartIMap's absolute results remain the best on eight of ten metrics (Table 2).

W7: Grammatical errors; AP reported with one decimal and other metrics with two.

Thank you for the detailed reading. We corrected the three errors and proofread the full text, and we now report all metrics, including AP, with two decimals in both text and tables.

We hope these answers address your concerns and are happy to discuss further.

References

[1] L. Qi et al. High Quality Entity Segmentation. ICCV, 2023.

I appreciate the authors' response. The above explanation makes sense to me. Therefore, I tend to maintain my score.

Reviewer_FjmwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PartIMap focuses on the online open-vocabulary 3D panoptic mapping problem, treating the fusion of multi-view 2D masks into a global 3D object as a probabilistic inference problem. The inputs are live RGB-D keyframes with their poses estimated by ORB-SLAM3 and class-agnostic 2D segmentations. Then the 2D masks are lifted to a hashed-voxel map via additive sufficient statistics: vMF distributions model semantic descriptors, per-voxel language accumulators decouple writing from merging, and Beta distributions model occupancy. Instance fusion is cast as graph-partition (correlation-clustering) MAP inference over a mask graph with evidence cues, solved by greedy aggregation. The experiments show the performance of the proposed method.

Critical strengths
  1. The techniques this paper adopts, such as MAP and ORB-SLAM3, are efficient, and the overall runtime is not too high.
  2. It is a reasonable idea to cast it as graph-partition MAP inference over a mask graph.
Critical weaknesses
  1. The contribution is largely a system integration and re-formulation rather than a new algorithmic mechanism. The overall pipeline closely resembles prior online open-vocabulary mapping systems such as OnlineAnySeg and OVI-MAP. The building block is off-the-shelf: such as vMF/Beta probabilistic representation, the voxel representation, correlation-clustering / greedy aggregation. The contributions seem more incremental.
  2. Hyperparameters are too much and calibration dependence are under-analyzed. Many parameters must be set (θ = 3, s_tstay, slope T in Eq. 6, plus priors such as Beta b₀ and member-weight terms). These parameters transferability across datasets and across 2D backbones is not evaluated, so it is unclear whether re-tuning is needed on new scenes.
  3. Scaling is a problem to the proposed method. Intrinsic scalability limitation (NP-hard). Exact MAP is NP-hard and is approximated by greedy aggregation; the scaling behavior as instances/masks grow, and the worst case (whether it degenerates to an O(n²) full-graph merge), are not analyzed. Whether large scenes / long sequences remain online is unverified.
  4. No large-scene and failure-case analysis. Evaluation is confined to small indoor scenes (ScanNetV2 and Replica). No test on large scenes or long trajectories is provided, so the scene scale at which the map breaks down is unclear. Additionally, the paper reports no failure-case study: how the map degrades under occlusion, texture-less regions, large depth noise, how is the performance if the graph is failed.
Questions for authors
  1. What is the computational scaling of the partition inference as the number of instances/masks grows? Does the worst case degenerate to an O(n²) full-graph merge? Could you tested on some larger scenes?
  2. How do θ, s_tstay, and T transfer between Replica and ScanNetV2 and to unseen scene types? How sensitive are they?
  3. If masks are wrongly merged early under weak evidence, can later partition inference correct this, or is the error irreversible? Could you add failure studies?
  4. What is the real end-to-end FPS once LSeg + CropFormer are included, and at what scene scale / sequence length does online operation hold?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Fjmw (1/4)

We thank the reviewer for finding the graph-partition MAP formulation a reasonable idea and the runtime acceptable. We answer each point below; results marked (*) are new experiments, and all other results are in the submitted paper. Please also see the summary in our General Response.

W1: The contribution is largely system integration and re-formulation; the building blocks are off-the-shelf; the pipeline resembles OnlineAnySeg and OVI-MAP.

Our contribution is a fusion model that connects the building blocks under one posterior. It enables capabilities that existing online 3D panoptic segmentation methods do not possess: modeling the uncertainty of partial observations, an explicit merge objective, and the same inference at loop-closure re-fusion. These capabilities cannot come from mere system integration. The controlled comparisons show that this model, not the system around it, produces the gains.

What we reuse. We do not claim vMF or Beta distributions, correlation clustering, greedy agglomeration, or the hashed voxel volume as new. We reuse OnlineAnySeg's mask-bank organization (Appendix A) and two of our four cues, the FCGF geometric cue and the visibility-gated containment (Section 3.3). We now make this attribution explicit in Section 3.

What is new.

  1. Merging is a partition, not a matching. Appendix C shows that probabilistic data association (PDA), the standard probabilistic tool for association, cannot merge k≥3k\ge3 mutual duplicates however strong their evidence, since each pairwise weight stays below 1/(k−1)1/(k-1). This explains why OnlineAnySeg and MaskClustering add a heuristic supporter rule. The partition posterior merges such groups under the same criterion as a pair and captures transitivity. Table 4 follows this prediction: PDA (#1) barely moves the baseline, the supporter rule (#2) recovers part of the gap, and the partition (#3) improves all five metrics over both.
  2. One posterior gives the merge test, its threshold, and the membership weights. As Reviewers H6mB and HG7D note, the merge gain is the exact change in the objective, and the membership weights come from the same posterior. θ\theta is a prior log-odds rather than a free similarity threshold.
  3. A re-derivable map. Every fused quantity is an additive sum over stored observations, so loop closure reuses the ordinary partition step. Such a design facilitates loop-closure re-fusion, as recognized by Reviewers H6mB and HG7D. OVO re-merges with fixed similarity thresholds, and OnlineAnySeg and OVI-MAP have no live-SLAM mode.
OnlineAnySegOVI-MAPPartIMap
Merge decisionthresholded similarity or supporter rulehard rules on a TSDF instance mapMAP partition (Eq. 4): merge iff λˉPQ>θ\bar\lambda_{PQ}>\theta
Weak evidencemerged at full weightmerged at full weightuncertainty-weighted by β\beta (Eq. 8)
Instance geometrybinary voxel setclass-agnostic TSDF instancesBeta occupancy (Eq. 2)
Semanticsone averaged descriptordescriptor from selected viewsvMF per instance + per-voxel grid
Loop closurenot supportednot supportedsame partition step

The gains do not come from system integration.

  • In Table 2, OnlineAnySeg† runs inside our system, with the same 2D models, ORB-SLAM3, loop-closure re-fusion, and map-reset guard, and keeps only its own merge rules. PartIMap still leads on eight of ten metrics (AP 20.04 vs. 17.57, PRQ (T) 49.78 vs. 45.02), as Reviewer HG7D also noted.
  • New row (*) in Table 4: the partition model alone, on OnlineAnySeg's representation and without vMF, occupancy, or membership weights, improves over OnlineAnySeg by 1.53 AP and 1.87 PRQ (T), and outperforms vMF + PDA + supporter rule (#2) on four of five metrics.
  • In Table 3, with SAM masks and CLIP descriptors, PartIMap improves over OnlineAnySeg on all metrics.

Revision: explicit attribution in Section 3 and Appendix A; row (*) in Table 4.

AuthorsOpenReview ↗

Response to Reviewer Fjmw (2/4)

W2 & Q2: Many hyperparameters; transfer of θ\theta, sstays_{\mathrm{stay}}, and TT across datasets and backbones.

Our paper reported analysis on θ\theta, and we add a sweep on TT. sstays_{\mathrm{stay}} essentially depends on θ\theta and TT, b0=1b_0=1 is fixed by construction, while other hyperparameters are inherited. A single set of values is used for both datasets and both backbone pairs. New sweeps show that θ\theta and TT transfer across datasets and backbones.

Parameter accounting.

  • sstays_{\mathrm{stay}} is absorbed into the threshold. From Eqs. 6–7, λab−θ=T(sab−τ)\lambda_{ab}-\theta=T(s_{ab}-\tau) with τ=sstay+θ/T\tau=s_{\mathrm{stay}}+\theta/T, so a merge is accepted if and only if the mean score across the cut exceeds τ\tau. sstays_{\mathrm{stay}} and θ\theta act only through τ\tau, and the θ\theta sweep in Table 5 covers τ∈[2.25,3.25]\tau\in[2.25,3.25].
  • b0=1b_0=1 is a fixed pseudo-count, not tuned. With b0→0b_0\to0, the expected occupancy would be 1 everywhere on the support, so soccs_{\mathrm{occ}} would reduce to plain containment and largely duplicate the spatial cue.
  • Voxel size, mask-size filters, and visibility gates are inherited unchanged from OnlineAnySeg and OnlinePG (Appendix D).

Transferability comes from aligned feature space. Our hyperparameters operate in a generalizable feature space. Among the four cues used in PartIMap's pairwise evidence λ\lambda, spatial and occupancy cues measure relative spatial proximity in metric space and is naturally transferable across datasets and scene types. Our semantic cue calibration (Appendix D) makes the semantic cues transferable across backbones. Its normalization effect does not make the optimum identical, as the sweeps below show; but it keeps the default close to it, so that the benefits transfer as shown in Table 3.

Sweep of TT (*). With ground-truth poses and τ\tau held fixed, i.e., θ\theta rescaled with TT; results suggest that metrics do not vary significantly with TT.

ScanNetV2Replica
TTAPmIoUmAccPRQ (T)PRQ (S)APmIoUmAccPRQ (T)PRQ (S)
125.0248.7069.2850.8244.8116.1355.8667.5843.4224.45
225.9649.2670.1150.9045.3116.3756.1267.8043.4224.39
4 (default)25.7249.0670.3351.8444.9016.4156.1067.8043.5024.45
825.0049.0570.3250.9344.7616.2456.1067.7843.4324.38
OnlineAnySeg22.5045.3365.8547.9042.3615.9154.1366.8640.3922.16

Transfer of θ\theta (*) to Replica (CropFormer + LSeg) and to SAM + CLIP on ScanNetV2, all other values unchanged:

Replica (CropFormer + LSeg)ScanNetV2 (SAM + CLIP)
θ\thetaAPmIoUmAccPRQ (T)PRQ (S)APmIoUmAccPRQ (T)PRQ (S)
114.9755.5867.3140.7824.3121.8930.7851.3539.1026.65
215.8456.5368.1541.5525.0022.2330.4452.5839.6326.43
3 (default)16.4156.1067.8043.5024.4521.9332.9154.3339.5428.30
415.6354.9167.1141.1823.5419.8730.8151.1040.2624.44
515.9254.8766.6141.7923.5919.4329.5150.0838.8225.80
OnlineAnySeg15.9154.1366.8640.3922.1619.7329.7750.8037.9626.79

In the transferred setting, every θ\theta outperforms or is comparable to the OnlineAnySeg baseline. The default stays close to or is the optimum.

Revision: both sweeps are added to Appendix E; Appendix D gives the parameter accounting above, and its statement on transfer is narrowed to the evaluated datasets and backbones.

AuthorsOpenReview ↗

Response to Reviewer Fjmw (3/4)

W3, W4 & Q1, Q4: Scaling of partition inference with the number of masks; the O(n2)O(n^2) worst case; larger scenes and long sequences; end-to-end FPS.

Partition inference is not the bottleneck: on sequences of up to 26k frames and 690 instances, a merge round takes 22 ms (median). The mask graph decomposes into small components, each corresponding to a 3D object. The worst case requires all instances to fall into one merge component, which we never observe.

Inference decomposes. Exact MAP and greedy inference both decompose over the connected components of the graph G+G^+, which contains all pairs with positive weight λab−θ\lambda_{ab}-\theta. A cluster that spans two components can be split along a cut without positive terms, which never lowers Eq. 4, and the greedy test λˉPQ>θ\bar\lambda_{PQ}>\theta rejects such merges. λ\lambda is large only for masks that overlap in 3D and agree semantically, which in real scenes are fragments of one object, so components stay small.

Cost model (*). Fitted on all runs below, one merge round with nn instances and MM merges costs t=8.7 μs⋅n+M (190 μs+0.06 ns⋅n2)t=8.7\ \mu\mathrm{s}\cdot n+M\ (190\ \mu\mathrm{s}+0.06\ \mathrm{ns}\cdot n^2). In the worst case (all nn instances in one component, M=n−1M=n-1), the per-merge n2n^2 bookkeeping makes a round cubic in nn, but with a small constant: 258 ms at n=1,000n=1{,}000 and 1.42 s at the map capacity of 2,500 instances. Measured runs have M≈0.26nM\approx 0.26n and a largest component of almost constant size, and the n2n^2 term only matters beyond n≈1,800n\approx 1{,}800.

Larger scenes (*). We ran ScanNetV2 scene0231_00 (4,439 frames, the validation scene with the most instances entering a merge) and five ScanNet++ [1] sequences (10k–26k frames, scene scale up to 21 m across). Per merge round (times in ms):

Instances nnRoundsM/nM/nMeasuredCost modelWorst case
34–99640.427.7 (10.9)4.615.6
100–1992240.3011.6 (17.2)9.431.9
200–2993150.2616.4 (25.1)15.151.4
300–3992150.2220.3 (29.1)20.469.9
400–499540.2022.1 (38.3)26.390.2
500–690160.1222.0 (27.5)39.2135
1,000 (extrapolated)–––74258
2,500 (capacity)–––3861,420

Measured: median (p90) solver time; cost model uses M=0.26nM=0.26n; worst case: M=n−1M=n-1.

All runs stay far below the worst case. The new figure in Section 4.4 plots solver time and the size of the largest component of G+G^+ against nn. G+G^+ stays almost constant and never exceeds 18 instances even on these larger scenes, which is consistent with the real world that 3D objects have limited sizes.

Long sequences and memory. All six sequences run online to the end, up to 26k frames (1.3k keyframes at every 20th frame of the 60-fps ScanNet++ video, i.e., 3 keyframes/s). Memory grows linearly at about 1.6 MB per keyframe, so a 24 GB GPU holds about 14k keyframes, or about 78 min of video at 3 keyframes/s. The longest ScanNetV2 sequences have fewer than 1k keyframes. Building-scale scans would need submaps, which we list as future work in Appendix G.

End-to-end FPS (Q4). Including LSeg and CropFormer, PartIMap runs at 5.7 FPS with ground-truth poses and at 4.1 FPS with live ORB-SLAM3, on one RTX 3090 GPU and an i9-9900K CPU. Tables 1–2 follow OnlineAnySeg and OnlinePG in excluding the 2D models, which setting the compared online methods also follow, so that the comparison (6.9 vs. 7.0 FPS for OnlineAnySeg) isolates the fusion.

Revision: new Section 4.4, Runtime and Scaling Analysis, with the figure, the cost model, end-to-end FPS, and memory growth; limits of scale in Appendix G.

References

[1] C. Yeshwanth et al. ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes. ICCV, 2023.

Reviewer_H6mBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript presents PartIMap, which formulates online multi-view mask fusion as graph partition inference. Additive semantic and occupancy statistics support instance grouping and map reconstruction after SLAM corrections. Experiments evaluate segmentation quality, backbone sensitivity, and live loop-closure handling.

Critical strengths
  1. Explicit fusion objective: The partition formulation aggregates association evidence and represents transitive grouping relationships through a clearly specified objective and merge-gain calculation.

  2. Well-integrated system design: Additive statistics and observation provenance connect semantic fusion with geometric correction. Loop-closure re-fusion improves ScanNetV2 AP from 17.2 to 20.0.

  3. Relatively thorough comparisons: Results include matched-backbone baselines, alternative foundation models, component ablations, and live SLAM experiments with comparable loop-closure support for OnlineAnySeg.

Critical weaknesses
  1. Limited sensitivity analysis: Robustness to observation order, merge frequency, and other hyperparameters remains unclear.

  2. Incomplete clarification of contributions: The current ablations do not fully clarify the benefits of the partition objective, merge ordering, and soft membership weights.

Questions for authors
  1. Can the system split incorrectly merged instances after loop closure? Please compare against repartitioning from raw observations.

  2. What gains remain against clustering baselines using identical features, graphs, and thresholds, thereby isolating the objective, merge ordering, and soft weighting?

  3. How sensitive are results to observation order, merge frequency, and repeated multiplicative weighting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer H6mB (1/2)

We thank the reviewer for recognizing the explicit fusion objective, the integration of additive statistics with loop-closure re-fusion, and the breadth of our comparisons. Results marked (*) are new experiments; all other results are in the submitted paper. Please also see the summary in our General Response.

W1 & Q3: Sensitivity to observation order, merge frequency, and repeated multiplicative weighting.

PartIMap is not sensitive to these configurations.

Model properties that limit sensitivity.

  • Observation order. Instance statistics (Eqs. 1,2) and voxel language grid (Section 3.2) take the form of additive sum, and the partition inference estimator (Section 3.3) does not depend on node order, so PartIMap is not sensitive to observation order.
  • Merge frequency. The acceptance test λˉPQ>θ\bar\lambda_{PQ}>\theta does not depend on cluster sizes (Eq. 7), so the criterion is the same whether a partition step sees 5 or 10 new keyframes.
  • Repeated multiplicative weighting. For a member with strong evidence, βa=σ(∑b(λab−θ))\beta_a=\sigma\bigl(\sum_b(\lambda_{ab}-\theta)\bigr) is close to 1, since every supporting edge adds a positive term, so repeated multiplication changes little. The product decays only for an instance that is marginal again and again, and down-weighting repeated weak evidence is the intended behavior.

Measured sensitivity (*) on the 10 ScanNetV2 scenes with ground-truth poses:

SettingAPmIoUmAccPRQ (T)PRQ (S)
Default (merge every 5 keyframes, gain ordering)25.7249.0670.3351.8444.90
Reversed frame order (*)25.6349.4370.5852.0644.28
Merge every 10 keyframes (*)25.1548.7870.1652.1144.61
Merges ordered by λˉPQ\bar\lambda_{PQ} instead of Δ\Delta (*)25.4849.0470.3151.5144.94

All changes are small. Changing merge frequency also changes the frequency that weights are multiplied, yet the two factors together do not move performance away from the default configuration. The last row concerns merge ordering and is discussed under W2 & Q2.

Revision: all sensitivity results are added to Appendix E.

W2 & Q2: The current ablations do not fully clarify the benefits of the partition objective, merge ordering, and soft membership weights.

Under identical features, graph, and thresholds, the partition objective accounts for the largest gain, and merge ordering has little effect. Soft weighting and the probabilistic representation add further gains. In Table 4, #3 vs. #2 shows benefits of the partition model, and #2 vs. #0 shows benefits of soft weights.

Objective and soft weighting (*). To further seperate the benefits of the partition objective and the membership weights, we added a new row (*) to Table 4, where we adopt the partition model without any probabilistic representations and the membership weights. (*) vs. baseline then shows benefits of the partition objective, while #3 and #4 vs. (*) shows benefits of soft membership weights and probabilistic representations.

ModelAPmIoUmAccPRQ (T)PRQ (S)
#0OnlineAnySeg (baseline)22.5045.3365.8547.9042.36
#1vMF + PDA22.6546.0066.1447.6042.81
#2vMF + PDA + Supp.23.6646.9267.5948.7843.13
(*)Part. (baseline representation)24.0347.5368.1749.7742.72
#3vMF + Part.24.7847.8469.1149.6043.81
#4Prob. rep. + Part. (Ours)25.7249.0670.3351.8444.90

Merge ordering (*). Ordering merges by mean evidence λˉPQ\bar\lambda_{PQ}, the order of classical average linkage, instead of by the gain Δ\Delta does not change performance significantly (last row of the table under W1 & Q3).

Revision: row (*) added to Table 4 with a revised component analysis (Section 4.3); merge ordering added to Appendix E.

AuthorsOpenReview ↗

Response to Reviewer H6mB (2/2)

Q1: Can the system split incorrectly merged instances after loop closure? Comparison against repartitioning from raw observations.

PartIMap has the capability to correct wrong merges by repartitioning, but this functionality is not enabled in the submitted system, and we did not claim it's one of our contributions. We ran the requested comparison (*). Repartitioning gives gains on PRQ (T) and mixed changes on the other metrics, while adding 30 s to 4 min per sequence.

Splits under the same model. Splitting a set of observations AA off an instance CC changes the log-posterior (Eq. 4) by −∑a∈A,b∈C∖A(λab−θ)-\sum_{a\in A, b\in C\setminus A}(\lambda_{ab}-\theta). A split is therefore accepted if and only if the mean evidence across the cut falls below θ\theta, the same threshold that governs merging, so no extra parameter is needed.

Suggested repartitioning baseline (*). On the 10 ScanNetV2 scenes, with ground-truth poses and with live SLAM, we split all instances into their raw observations at the end of each sequence and re-run partition inference until convergence:

Ground-truth posesLive SLAM
MethodAPmIoUmAccPRQ (T)PRQ (S)APmIoUmAccPRQ (T)PRQ (S)
PartIMap25.7249.0670.3351.8444.9020.0648.4867.3749.5039.21
+ repartitioning (*)25.6249.4670.2852.9045.8820.5449.3667.5150.7238.74

Repartitioning improves PRQ (T) by 1.06 and 1.22 and mIoU by 0.40 and 0.88 (ground-truth poses and live SLAM). AP changes by -0.10 and +0.48 and PRQ (S) by +0.98 and -0.47. The extra latency at the end of a sequence is about 30 s on a small scene and up to 4 min on a large one.

Bottleneck is CropFormer bias. In a preliminary analysis, about 80% of PartIMap's over-merged instances contain at least one under-segmented CropFormer mask, i.e., a single 2D mask that covers two or more objects. Such under-segmentation is often consistent across views, so this is a bias problem of CropFormer. Because the partition operates on masks, no split at the observation level can separate the objects inside one mask. Under SLAM, drift mainly fragments objects, as the increase in split errors in Appendix F shows qualitatively, and the loop-closure re-fusion helps recovering such fragments (AP 17.23 → 20.04, Appendix E).

Revision: the repartitioning baseline is added to Appendix E. Appendix G discusses retro-correction as future work, e.g., repartitioning only the instances touched by a loop closure to bound its latency.

We hope these answers address your concerns and are happy to discuss further.

Dear Reviewer H6mB,

Thank you again for taking the time to review our paper and share your comments. We hope our earlier response has addressed your concerns. Since the ICLR discussion period ends soon, we would greatly appreciate your feedback on our response, and we are happy to answer any further questions.

Best regards,

The Authors

I have carefully read the rebuttal from the authors. My concerns about baseline, sensitivity, and choices of weighting have been fully addressed. Thanks for the detailed responses from the authors. I will keep my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers and ACs for their time and constructive comments. We are encouraged that the reviewers found the partition formulation principled, with "a clearly specified objective and merge-gain calculation" (H6mB) and "a reasonable idea to cast it as graph-partition MAP inference over a mask graph" (Fjmw), and noted that "the greedy merge score is the exact change in that objective, and the membership weights [...] come from the same posterior" (HG7D). Reviewers also recognized that PartIMap is well-designed, with "well-integrated system design" and "relatively thorough comparisons" (H6mB), and its superior performance, "leads on eight of ten metrics when OnlineAnySeg is given our loop-closure re-fusion" (HG7D). They also found that "the overall runtime is not too high" (Fjmw), and that "the writing is clear and easy to follow" (HG7D).

Results marked (*) are new experiments for this rebuttal; all other results are in the submitted paper. The reviews share four concerns.

  1. Sensitivity and hyperparameters (H6mB, Fjmw, HG7D). In the paper we analyzed the prior weight θ\theta. We added a sweep on TT (*) which shows PartIMap is not sensitive to TT. The indifference level sstays_{\mathrm{stay}} acts only through the merge threshold τ=sstay+θ/T\tau=s_{\mathrm{stay}}+\theta/T, and b0b_0 is fixed. Reversing the frame order or halving the merge frequency (*) changes metric by at most 0.62. θ=3\theta=3, selected on ScanNetV2 training scenes, transfers to Replica and to SAM+CLIP (*).
  2. Isolating the contributions (H6mB, Fjmw). With identical features, graph, and thresholds, replacing probabilistic data association by the partition model improves all five metrics (+2.13 AP, +2.00 PRQ (T)). A new row in Table 4 (*) applies the partition model to OnlineAnySeg's representation, without vMF, occupancy, or membership weights: it improves over OnlineAnySeg by 1.53 AP and 1.87 PRQ (T).
  3. Runtime, scaling, and long sequences (Fjmw, HG7D). On five ScanNet++ sequences of 10k–26k frames and the most instance-rich ScanNetV2 scene (*), one round of partition inference takes 22 ms (median) at 500–690 instances, and the worst case of a single merge component does not occur. End to end, including LSeg, CropFormer, and ORB-SLAM3, PartIMap runs at 4.1 FPS. We now report loop-closure latency (median 20 s) and memory growth (1.6 MB per keyframe).
  4. Correcting wrong merges and failure cases (H6mB, Fjmw). The partition model supports splits under the same threshold as merges. Repartitioning from raw observations (*) improves PRQ (T) by 1.06 (ground-truth poses) and 1.22 (live SLAM) at 30 s–4 min of extra latency. A failure analysis identifies under-segmented 2D masks as the main source of over-merging, while occlusion (*) and texture-less region cost little.

We have revised the manuscript accordingly:

  • Section 4.3 / Table 4: new row (*) and revised component analysis (H6mB W2, Q2; Fjmw W1).
  • New Section 4.4, Runtime and Scaling Analysis: scaling figure and cost model, end-to-end FPS, loop-closure latency, and memory growth (Fjmw W3, W4, Q1, Q4; HG7D W1). Section 4.2 now reports loop-closure latency separately from the frame rate (HG7D W1).
  • Appendix E, renamed Additional Quantitative Results: sensitivity to observation order, merge frequency, and merge ordering (H6mB W1, Q3); sweeps of TT and θ\theta (Fjmw W2, Q2; HG7D W3); repartitioning baseline (H6mB Q1; Fjmw Q3); occlusion analysis (Fjmw W4); absolute drops from ground-truth to SLAM poses (HG7D W6).
  • Appendix G, Limitations and Future Work: repartitioning, failure cases, and limits of scale (H6mB Q1; Fjmw W4, Q3; HG7D W1).
  • Section 3 and Appendices A, D: explicit attribution of the components we reuse from OnlineAnySeg (Fjmw W1); parameter accounting (Fjmw W2; HG7D W3).
  • Text: refined statements on the language grid (Section 3.2) and on zero-shot (abstract, Section 1, Ethics) (HG7D W4, W5); OnlinePG backbones in the Table 1 caption (HG7D W2); grammar and decimals (HG7D W7).

We are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Summary of the Rebuttal and Discussion

Dear ACs,

Thank you for handling our submission. Below we briefly summarize the reviews, the evidence we added during the rebuttal, and the outcome of the discussion.

Strengths recognized in the initial reviews.

  • Principled formulation: "a clearly specified objective and merge-gain calculation" (H6mB); "a reasonable idea to cast it as graph-partition MAP inference over a mask graph" (Fjmw); "the greedy merge score is the exact change in that objective, and the membership weights [...] come from the same posterior" (HG7D).
  • System design and evaluation: "well-integrated system design" and "relatively thorough comparisons" (H6mB); "the overall runtime is not too high" (Fjmw).
  • Performance and clarity: "The method achieves state-of-the-art results on ScanNetV2 and Replica", and when OnlineAnySeg is given our loop-closure re-fusion, PartIMap "still leads on eight of the ten segmentation metrics"; "the writing is clear and easy to follow" (HG7D).

Main concerns and how we addressed them.

  1. Contribution and isolation of its components (Fjmw, H6mB). Fjmw initially viewed the contribution as system integration of off-the-shelf blocks, and H6mB asked to isolate the partition objective, merge ordering, and soft weighting. We first clarified our contributions: a single posterior as explicit merging rule with uncertainty-aware weights, that generalizes probabilistic data association, and is reused unchanged at loop-closure re-fusion. These are properties that existing methods do not possess. A new row in Table 4 applies the partition model alone to the baseline's representation and improves over it by 1.53 AP and 1.87 PRQ (T); on top of it, the probabilistic representation and membership weights add a further 1.69 AP and 2.07 PRQ (T). Ordering merges by mean evidence instead of the gain changes little (Table 9).
  2. Hyperparameters and sensitivity (H6mB, Fjmw, HG7D). We showed that our model properties already limit sensitivity and support transferability of hyperparameters (Appendix D). New sweeps of TT (Table 10), and of θ\theta (Table 11) show that they transfer to Replica and to SAM + CLIP. The method is not sensitive to reversing the frame order or halving the merge frequency (Table 9).
  3. Scalability, runtime, and long sequences (Fjmw, HG7D). Runtime is bounded because partition inference decomposes over the connected components of the positive-evidence graph, so that the worst case never happens in reality. On five ScanNet++ sequences of 10k–26k frames and the most instance-rich ScanNetV2 scene (new Section 4.4), a merge round takes 22 ms (median) at 500–690 instances, the largest component never exceeds 18 instances, and all sequences run online to the end. End to end, including LSeg, CropFormer, and ORB-SLAM3, PartIMap runs at 4.1 FPS. We now also report loop-closure latency (median 20 s) and memory growth (1.6 MB per keyframe).
  4. Correcting wrong merges and failure cases (H6mB, Fjmw). The same posterior accepts splits under the same threshold as merges. Repartitioning from raw observations (Table 12) improves PRQ (T) by 1.06 with ground-truth poses and 1.22 with live SLAM, with mixed changes on other metrics and 30 s to 4 min of extra latency, so we report it as an option rather than a contribution. A failure analysis (Appendix G) traces about 80% of over-merged instances to under-segmented 2D masks, which partition over masks cannot split; and shows stable accuracy for objects with occlusion.
  5. Minor points (HG7D). We stated the OnlinePG backbones, quantified the effect of reusing TT and sstays_{\mathrm{stay}} from the PDA setting, narrowed the statements on the language grid and on zero-shot (no training or fine-tuning), added absolute drops under live SLAM, and fixed the grammar and decimal reporting.

Outcome of the discussion.

  • Reviewer H6mB (initial rating: weak acceptance): "My concerns about baseline, sensitivity, and choices of weighting have been fully addressed," and kept the positive rating.
  • Reviewer Fjmw (initial rating: weak rejection), the only initially negative reviewer: the "detailed analysis and experiments [...] addressed most of my concerns," and raised the rating.
  • Reviewer HG7D (initial rating: weak acceptance) raised only minor issues and no questions; we addressed every point in the revision, and they maintained the positive rating.

In short, the reviewers found the formulation principled and the results strong. The main concerns about the contribution, sensitivity, and scalability were answered with justification and backed up by new controlled experiments. All reviewers approved of our detailed responses, and the only initially negative reviewer raised their rating (2 → 3).

We thank you and the reviewers again for your time and constructive feedback.

Best regards,

Authors of Submission 45655

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.