- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SEGA processes long indoor RGB sequences by reconstructing overlapping clusters, aligning them to a global proxy geometry, associating local instances, and propagating/refining masks with SAM3. A final instance-aware bundle-adjustment stage filters and reweights sparse correspondences using the recovered identities.
Critical strengths
- Clear motivation: SEGA combines geometry-guided cross-cluster association with video mask propagation to improve instance consistency and coverage across long indoor sequences.
- Good performance: Evaluation covers both ScanNet200 and ScanNet++v2, includes oracle-geometry and predicted-proxy variants, and reports separate pose/reconstruction measurements.
Critical weaknesses
-
Limited methodological novelty is my main concern. SEGA primarily integrates established techniques for multi-view reconstruction, geometric alignment, mask propagation, and bundle adjustment. The proposed geometry-guided propagation and instance-based track filtering appear to be incremental refinements within this pipeline. Although the experiments demonstrate practical benefits, the paper does not clearly establish a substantial methodological advance beyond combining and adapting existing components. The contribution therefore appears primarily focused on system integration and engineering improvement.
-
Potential conflict between mask completion and negative prompting. Section 3.2 acknowledges that the initial 3D support may cover only part of an object. However, the description around Eq. (8) treats predictions extending beyond this support as geometrically inconsistent and samples negative prompts from these regions. This could suppress valid extensions into previously unobserved object regions. The paper should clarify whether is updated during propagation and how unsupported regions are distinguished from confirmed background.
-
The geometric feedback contribution is not precisely characterized. Sections 3.1–3.2 describe constraints beyond sparse feature tracks and refer to dense constraints for refinement. However, Section 3.3 defines as a subset of the original sparse tracks, and Eq. (9) presents a reweighted sparse reprojection objective without an additional dense or object-level residual. The paper should reconcile these descriptions and distinguish instance-guided track filtering from introducing new geometric constraints.
-
The IA-BA ablation lacks a clearly specified compute-matched baseline. The authors should compare instance-guided BA against ordinary BA, starting from the same reconstruction and using the same number of iterations, to isolate the benefit of instance guidance.
Questions for authors
Please refer to the Weakness part.
7 comments
Response to Reviewer ChJ5 (1b/4)
Transfer beyond the primary benchmarks. New Tables F1–F2 (Tables 17–18 in the revised manuscript) evaluate the same SEGA pipeline and hyperparameters on Replica without dataset-specific fine-tuning or adjustment. Table F1 shows that IA-BA improves AUC@30 from 77.63 to 78.49 and reduces Chamfer distance from 0.0618 to 0.0589. Under the separate GT-geometry segmentation setting, Table F2 reports 27.4 AP for SEGA versus 22.2 for Open3DIS across 48 categories. Figure 6 provides complementary qualitative results on an iPhone-captured indoor sequence using the same configuration without scene-specific tuning. These additions support transfer of the proposed design beyond the ScanNet benchmarks and complement the mechanism-focused ablations above.
Table F1 — Camera pose estimation and 3D point-cloud reconstruction on Replica (Table 17 in the manuscript)
| Method | RRA@30 ↑ | RTA@30 ↑ | AUC@30 ↑ | Acc ↓ | Comp ↓ | Chamfer Dist. ↓ | F-Score ↑ |
|---|---|---|---|---|---|---|---|
| LingBot-Map | 90.83 | 86.85 | 72.26 | 0.1126 | 0.0661 | 0.1787 | 0.5839 |
| FastVGGT | 100.00 | 97.46 | 75.29 | 0.0560 | 0.0268 | 0.0828 | 0.6781 |
| SEGA (w/o IA-BA) | 100.00 | 98.11 | 77.63 | 0.0415 | 0.0203 | 0.0618 | 0.7268 |
| SEGA (w/ IA-BA) | 100.00 | 98.89 | 78.49 | 0.0397 | 0.0192 | 0.0589 | 0.7313 |
Table F2 — Class-agnostic 3D instance segmentation on Replica under the GT-geometry setting, covering 48 instance categories (Table 18 in the manuscript)
| Method | GT Geometry | AP ↑ | AP50 ↑ | AP25 ↑ |
|---|---|---|---|---|
| SAM3D | ✓ | 19.4 | 29.5 | 38.5 |
| Open3DIS | ✓ | 22.2 | 34.8 | 42.3 |
| SEGA | ✓ | 27.4 | 44.3 | 61.2 |
Response to Reviewer ChJ5 (2/4)
C2. Mask completion versus negative prompting
Thank you very much for identifying this ambiguity. We agree that the wording around Eq. (8), now Eq. (11) is too broad. The reference support () anchors object identity; it does not define the object’s complete boundary. It is aggregated from the initially matched observations and remains fixed throughout each propagation pass; propagated masks do not update it.
During propagation, we compare each prediction’s associated proxy-point IDs with (). Negative correction is applied only when no shared points remain, indicating possible tracking drift. We sample negative prompts inside the suspected drifting prediction and continually update propagation from that frame. Subsequent predictions incorporate these corrective prompts.
More importantly, extending beyond () does not itself trigger correction. If the reference contains chair-seat points, a prediction covering both the seat and a newly visible backrest retains reference overlap and receives no negative correction. The newly recovered backrest itself need not overlap the reference: the check applies to the prediction as a whole. Regions outside the reference support are therefore not automatically classified as background. Zero overlap serves as a drift-detection heuristic rather than definitive evidence of background.
We have revised Eq. (11) and added Algorithm 1 to make this distinction explicit. Table 5 compares correction enabled and disabled: object-mask mIoU increases from 24.51% to 40.33%, and object-pixel recall from 50.49% to 67.09%. These metrics support improved overall mask quality and coverage, rather than separately measuring newly visible surfaces. Figure 4 provides complementary visualizations of the projected fixed support, negative-prompt locations, and predictions with correction enabled and disabled.
The Reviewer could refer to our response to the Reviewer a8CV's C1.
Response to Reviewer ChJ5 (1a/4)
We thank the reviewer for recognizing the motivation and empirical performance of SEGA. We appreciate the opportunity to clarify the methodological contribution, the negative-prompting criterion, and the geometric refinement objective.
C1. Methodological novelty beyond system integration
Thank you for raising this concern. SEGA’s central methodological contribution is geometry-anchored recovery of persistent object identities across independently reconstructed windows, with these identities guiding both mask completion and geometric refinement. The reconstruction and video-segmentation models provide the underlying predictions; our contribution concerns how their partial outputs are associated and reused for further inference when a global scene representation is not supplied in advance.
Geometric alignment alone establishes common coordinates but does not reconcile local instance IDs or recover object regions missing from the initial masks. SEGA first associates local instances through shared proxy geometry and aggregates their partial supports. These supports serve as fixed identity anchors for video propagation, allowing additional object regions to be predicted while checking for identity drift. The recovered global identities subsequently filter existing sparse feature correspondences for bundle adjustment. Geometry therefore guides identity recovery and completion, while recovered identities guide the geometric measurements retained for refinement.
We support these design choices with comparisons and ablations:
We additionally investigate repeated use of this feedback in the alternating variant in Table 15. After geometric refinement, we recompute association and propagation before another IA-BA pass. Across one to four passes, segmentation improves alongside reconstruction: AUC@30 increases from 82.42 to 83.48, while Chamfer distance decreases from 0.0403 to 0.0391. These results support revisiting geometry-dependent instance predictions, with diminishing gains and increasing runtime. They do not isolate refreshed feedback from all other effects of repeated processing.