- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents PartIMap, an online, zero-shot open-vocabulary 3D panoptic mapping system that takes an RGB-D stream as input. Instance semantics are stored as von Mises-Fisher distributions, together with a per-voxel language grid that is decoupled from merge decisions, and instance geometry is a per-voxel Beta occupancy weighted by membership. Mask merging is cast as MAP inference of a partition on a mask graph, i.e., correlation clustering. Edge evidence is a hand-set linear map of four cues, solved by greedy agglomeration, and membership weights are derived from the same posterior. The method achieves state-of-the-art results on ScanNetV2 and Replica.
Critical strengths
-
The experimental results are Section 3.3 analyzes the log-posterior of a partition on the mask graph. The greedy merge score is the exact change in that objective, and the membership weights used to fuse semantics and occupancy come from the same posterior. In Table 4, replacing probabilistic data association plus the supporter rule with this partition raises ScanNet AP from 23.7 to 24.8 and PRQ(T) from 48.78 to 49.60.
-
Section 3.4 keeps raw observations so that a loop closure re-integrates the map and runs the same partition step. Table 2 gives OnlineAnySeg this re-fusion and leaves its own merge rules unchanged. PartIMap still leads on eight of the ten segmentation metrics, with ScanNet AP 20.0 against 17.6 and PRQ(T) 49.78 against 45.02, and the frame rate stays at 6.9 FPS.
Critical weaknesses
The writing is clear and easy to follow, and I only have some minor issues:
-
Algorithm 2 replays every depth map and recomputes the full containment matrix, so the cost grows with sequence length. The reported 6.9 FPS does not show the latency of one loop closure or the memory growth on long sequences.
-
The OnlinePG numbers in Table 1 are taken from the original paper, and the 2D backbone is not stated.
-
and are tuned under probabilistic data association and then reused for the partition model in Appendix D. How this affects rows 1 and 3 of Table 4 is not discussed.
-
Section 3.2 says a wrong merge does not pollute the language grid. Decoding still adds the instance statistic to the voxel vector, so the final label still depends on the merge.
-
The geometric cue uses FCGF, a supervised pretrained 3D feature, so the claim of requiring no supervised training needs a narrower statement.
-
Decl. favors methods with low absolute accuracy. The authors note this. Absolute drops should be reported as well.
-
The text has grammatical errors, including “uncertain observations contributes” in Section 1, “Our hyperparameters therefore transfers” in Appendix D, and “Baseline methods commonly over-segments” in Appendix F. AP is reported to one decimal and the other metrics to two.
Questions for authors
None.
3 comments
Response to Reviewer HG7D (2/2)
W4: A wrong merge does not pollute the language grid, but decoding adds the instance statistic, so the label still depends on the merge.
We agree. Only the stored grid is independent of merges; the decoded direction depends on the merge through the instance resultant , in which each member is weighted by its membership weight. Section 3.2 now reads: "Because the grid is written independently of grouping, an incorrect merge does not alter the stored grid; the decoder balances evidence from the grid and the instance statistics."
W5: FCGF is a supervised pretrained 3D feature, so the claim of requiring no supervised training needs a narrower statement.
PartIMap trains and fine-tunes nothing and uses only frozen pretrained models; several of them, including LSeg, CropFormer, and FCGF, were trained with supervision by their authors. In our paper, "zero-shot" means that we add no additional training or fine-tuning on target datasets, not that the pretrained components were trained without supervision. We revised the abstract, Section 1, and the Ethics statement to say this.
W6: Decl. favors methods with low absolute accuracy; report absolute drops as well.
We added the absolute drops (ground-truth poses minus live SLAM; lower is better) to Appendix E:
| ScanNetV2 ↓ | Replica ↓ | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | AP | mIoU | mAcc | PRQ (T) | PRQ (S) | AP | mIoU | mAcc | PRQ (T) | PRQ (S) |
| Open3DIS | 1.96 | 2.24 | 3.76 | 1.11 | 1.28 | 0.99 | 2.15 | 2.08 | 1.53 | 0.79 |
| PanoGS | 3.88 | 4.40 | 6.21 | 4.30 | 5.20 | 1.03 | 1.10 | 0.56 | 4.75 | 1.85 |
| OVO | 4.92 | 4.11 | 1.48 | 5.67 | 1.74 | 0.67 | 0.00 | 0.45 | 0.68 | 0.99 |
| ConceptGraphs | 4.86 | 2.68 | 2.99 | 1.93 | 5.25 | 0.42 | 1.96 | 0.26 | 3.73 | 0.59 |
| OnlineAnySeg | 4.93 | 1.96 | 2.20 | 2.88 | 5.21 | 1.12 | 0.85 | 0.72 | 1.83 | 0.93 |
| OVI-MAP | 2.11 | 3.93 | 3.91 | 2.70 | 2.43 | 1.88 | 2.34 | 4.62 | 2.47 | 2.77 |
| PartIMap (ours) | 5.68 | 0.80 | 3.10 | 2.06 | 5.23 | 0.38 | 0.63 | 0.40 | 2.11 | -1.59 |
Compared with OnlineAnySeg, PartIMap drops less on six of ten metrics. Among all methods, PartIMap has the smallest drop on ScanNetV2 mIoU and on Replica AP and PRQ (S). Like Decl., absolute drops favor methods with less room to fall. Under live SLAM, PartIMap's absolute results remain the best on eight of ten metrics (Table 2).
W7: Grammatical errors; AP reported with one decimal and other metrics with two.
Thank you for the detailed reading. We corrected the three errors and proofread the full text, and we now report all metrics, including AP, with two decimals in both text and tables.
We hope these answers address your concerns and are happy to discuss further.
References
[1] L. Qi et al. High Quality Entity Segmentation. ICCV, 2023.
I appreciate the authors' response. The above explanation makes sense to me. Therefore, I tend to maintain my score.
Response to Reviewer HG7D (1/2)
We thank the reviewer for the careful reading, and for noting that the greedy merge score is the exact change in the objective, that the membership weights come from the same posterior, and that PartIMap leads OnlineAnySeg even when OnlineAnySeg is given our re-fusion. Results marked (*) are new experiments. Please also see the summary in our General Response.
Latency. A loop closure costs about 5 s on small scenes and early in large ones, and up to about 30 s late in large scenes; the median over the 27 loop-closure events in our 10 ScanNetV2 sequences is 20 s. As the reviewer notes, Algorithm 2 replays every integrated depth map and recomputes the full containment matrix, so this cost grows with sequence length. We revise the sentence in Section 4.2 that the frame rate is unchanged under SLAM and report loop-closure latency separately.
Memory. Memory grows linearly at about 1.6 MB per keyframe on average, so a 24 GB GPU holds about 14k keyframes; the longest ScanNetV2 sequences have fewer than 1k keyframes, and ScanNet++ sequences with up to 1.3k keyframes run to the end (response to Reviewer Fjmw, 3/4).
Re-integrating only the frames whose corrected pose changes beyond a tolerance, and recomputing containment only for the affected instances, would bound this cost; we list it as future work.
Revision: latency and memory are reported in the new Section 4.4; the cost growth and its remedy in Appendix G.
We took the OnlinePG numbers from the original paper because its code is not released. OnlinePG uses LSeg, the same vision-language model as ours, and EntitySeg [1] as the class-agnostic segmenter. The Table 1 caption now states this.
The reuse favors the PDA rows, as the hyperparameters were selected in that setting. sstay has no separate effect in the partition model: λab−θ=T(sab−τ) with τ=sstay+θ/T, so sstay and θ act only through τ, which Table 5 sweeps via θ. Re-tuning T for the partition model does not make a significant change, as shown in the Sweep of T (*):
Revision: Appendix D states that sstay acts only through τ; the T sweep is added to Appendix E.