acceptodds
Under review as a conference paper at ICLR 2027

SEGA: Multi-View 3D Instance Segmentation with Geometry-Guided Association for Indoor Environments

Abstract

We present SEGA, a geometry-guided framework for 3D instance segmentation of indoor scenes from multi-view image sequences. Existing scene reconstruction and understanding methods typically prioritize either geometric reconstruction or object-level perception, struggling to maintain both globally consistent geometry and coherent instance identities across hundreds to thousands of views. Our key insight is that these objectives are mutually beneficial: geometry provides a robust basis for cross-view instance association, while instance-level perception supplies constraints for refining geometry. SEGA decomposes a long sequence into overlapping clusters, reconstructs local geometry, and predicts 2D instance masks within each cluster. We introduce a 3D-Aware Alignment module that aligns these local predictions with a global proxy geometry and associates instances across clusters, producing temporally coherent video segmentation with globally consistent instance identities. We then apply a post processing step Instance-Aware Bundle Adjustment, which uses instance-consistent correspondences to refine the scene geometry. Finally, the predicted video masks are lifted into 3D instance proposals using the reconstructed geometry. We evaluate SEGA on ScanNet200 and ScanNet++v2 across two tasks: class-agnostic 3D instance segmentation and panoptic lifting for novel-view rendering. SEGA achieves state-of-the-art performance, improving Average Precision and Panoptic Quality over the strongest baselines. These results demonstrate the benefits of jointly modeling geometry and instance-level perception for robust indoor scene understanding from long multi-view sequences.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D segmentation, scene understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ChJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

SEGA processes long indoor RGB sequences by reconstructing overlapping clusters, aligning them to a global proxy geometry, associating local instances, and propagating/refining masks with SAM3. A final instance-aware bundle-adjustment stage filters and reweights sparse correspondences using the recovered identities.

Critical strengths
  • Clear motivation: SEGA combines geometry-guided cross-cluster association with video mask propagation to improve instance consistency and coverage across long indoor sequences.
  • Good performance: Evaluation covers both ScanNet200 and ScanNet++v2, includes oracle-geometry and predicted-proxy variants, and reports separate pose/reconstruction measurements.
Critical weaknesses
  • Limited methodological novelty is my main concern. SEGA primarily integrates established techniques for multi-view reconstruction, geometric alignment, mask propagation, and bundle adjustment. The proposed geometry-guided propagation and instance-based track filtering appear to be incremental refinements within this pipeline. Although the experiments demonstrate practical benefits, the paper does not clearly establish a substantial methodological advance beyond combining and adapting existing components. The contribution therefore appears primarily focused on system integration and engineering improvement.

  • Potential conflict between mask completion and negative prompting. Section 3.2 acknowledges that the initial 3D support QaQ_a may cover only part of an object. However, the description around Eq. (8) treats predictions extending beyond this support as geometrically inconsistent and samples negative prompts from these regions. This could suppress valid extensions into previously unobserved object regions. The paper should clarify whether QaQ_a is updated during propagation and how unsupported regions are distinguished from confirmed background.

  • The geometric feedback contribution is not precisely characterized. Sections 3.1–3.2 describe constraints beyond sparse feature tracks and refer to dense constraints for refinement. However, Section 3.3 defines TinstT_{\mathrm{inst}} as a subset of the original sparse tracks, and Eq. (9) presents a reweighted sparse reprojection objective without an additional dense or object-level residual. The paper should reconcile these descriptions and distinguish instance-guided track filtering from introducing new geometric constraints.

  • The IA-BA ablation lacks a clearly specified compute-matched baseline. The authors should compare instance-guided BA against ordinary BA, starting from the same reconstruction and using the same number of iterations, to isolate the benefit of instance guidance.

Questions for authors

Please refer to the Weakness part.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ChJ5 (1a/4)

We thank the reviewer for recognizing the motivation and empirical performance of SEGA. We appreciate the opportunity to clarify the methodological contribution, the negative-prompting criterion, and the geometric refinement objective.

C1. Methodological novelty beyond system integration

Thank you for raising this concern. SEGA’s central methodological contribution is geometry-anchored recovery of persistent object identities across independently reconstructed windows, with these identities guiding both mask completion and geometric refinement. The reconstruction and video-segmentation models provide the underlying predictions; our contribution concerns how their partial outputs are associated and reused for further inference when a global scene representation is not supplied in advance.

Geometric alignment alone establishes common coordinates but does not reconcile local instance IDs or recover object regions missing from the initial masks. SEGA first associates local instances through shared proxy geometry and aggregates their partial supports. These supports serve as fixed identity anchors for video propagation, allowing additional object regions to be predicted while checking for identity drift. The recovered global identities subsequently filter existing sparse feature correspondences for bundle adjustment. Geometry therefore guides identity recovery and completion, while recovered identities guide the geometric measurements retained for refinement.

We support these design choices with comparisons and ablations:

  1. Beyond post-hoc grouping of local predictions. Our composition comparisons, now in Table 10, augment chunked PanSt3R predictions with Open3DIS-style grouping. This strengthened baseline achieves 13.6/15.0 AP on ScanNet200/ScanNet++v2, compared with 17.1/18.1 AP for SEGA under pixel-aligned evaluation. The advantage thus persists when the baseline includes cross-cluster grouping. SEGA additionally uses the associated supports to generate further masks, extending the role of association beyond consolidation of existing predictions.
  2. Contribution of association and geometry-anchored propagation. In Table 4, removing iterative graph matching reduces AP from 18.11 to 15.97, and removing 3D-aware prompting reduces it to 15.62. These ablations support the usefulness of the mechanisms connecting cross-window association to mask recovery.
  3. Contribution of identity-guided geometric refinement. The additional comparison in Table 14 starts ordinary BA and IA-BA from the same saved reconstruction with matched optimization settings and iteration budget. Predicted-guidance IA-BA improves AUC@30 from 81.33 to 82.42 and reduces Chamfer distance from 0.0409 to 0.0403. This supports the benefit of identity-guided correspondence selection beyond additional optimization steps alone.

We additionally investigate repeated use of this feedback in the alternating variant in Table 15. After geometric refinement, we recompute association and propagation before another IA-BA pass. Across one to four passes, segmentation improves alongside reconstruction: AUC@30 increases from 82.42 to 83.48, while Chamfer distance decreases from 0.0403 to 0.0391. These results support revisiting geometry-dependent instance predictions, with diminishing gains and increasing runtime. They do not isolate refreshed feedback from all other effects of repeated processing.

AuthorsOpenReview ↗

Response to Reviewer ChJ5 (1b/4)

Transfer beyond the primary benchmarks. New Tables F1–F2 (Tables 17–18 in the revised manuscript) evaluate the same SEGA pipeline and hyperparameters on Replica without dataset-specific fine-tuning or adjustment. Table F1 shows that IA-BA improves AUC@30 from 77.63 to 78.49 and reduces Chamfer distance from 0.0618 to 0.0589. Under the separate GT-geometry segmentation setting, Table F2 reports 27.4 AP for SEGA versus 22.2 for Open3DIS across 48 categories. Figure 6 provides complementary qualitative results on an iPhone-captured indoor sequence using the same configuration without scene-specific tuning. These additions support transfer of the proposed design beyond the ScanNet benchmarks and complement the mechanism-focused ablations above.

Table F1 — Camera pose estimation and 3D point-cloud reconstruction on Replica (Table 17 in the manuscript)

MethodRRA@30 ↑RTA@30 ↑AUC@30 ↑Acc ↓Comp ↓Chamfer Dist. ↓F-Score ↑
LingBot-Map90.8386.8572.260.11260.06610.17870.5839
FastVGGT100.0097.4675.290.05600.02680.08280.6781
SEGA (w/o IA-BA)100.0098.1177.630.04150.02030.06180.7268
SEGA (w/ IA-BA)100.0098.8978.490.03970.01920.05890.7313

Table F2 — Class-agnostic 3D instance segmentation on Replica under the GT-geometry setting, covering 48 instance categories (Table 18 in the manuscript)

MethodGT GeometryAP ↑AP50 ↑AP25 ↑
SAM3D✓19.429.538.5
Open3DIS✓22.234.842.3
SEGA✓27.444.361.2
AuthorsOpenReview ↗

Response to Reviewer ChJ5 (2/4)

C2. Mask completion versus negative prompting

Thank you very much for identifying this ambiguity. We agree that the wording around Eq. (8), now Eq. (11) is too broad. The reference support (QaQ_a) anchors object identity; it does not define the object’s complete boundary. It is aggregated from the initially matched observations and remains fixed throughout each propagation pass; propagated masks do not update it.

During propagation, we compare each prediction’s associated proxy-point IDs with (QaQ_a). Negative correction is applied only when no shared points remain, indicating possible tracking drift. We sample negative prompts inside the suspected drifting prediction and continually update propagation from that frame. Subsequent predictions incorporate these corrective prompts.

More importantly, extending beyond (QaQ_a) does not itself trigger correction. If the reference contains chair-seat points, a prediction covering both the seat and a newly visible backrest retains reference overlap and receives no negative correction. The newly recovered backrest itself need not overlap the reference: the check applies to the prediction as a whole. Regions outside the reference support are therefore not automatically classified as background. Zero overlap serves as a drift-detection heuristic rather than definitive evidence of background.

We have revised Eq. (11) and added Algorithm 1 to make this distinction explicit. Table 5 compares correction enabled and disabled: object-mask mIoU increases from 24.51% to 40.33%, and object-pixel recall from 50.49% to 67.09%. These metrics support improved overall mask quality and coverage, rather than separately measuring newly visible surfaces. Figure 4 provides complementary visualizations of the projected fixed support, negative-prompt locations, and predictions with correction enabled and disabled.

The Reviewer could refer to our response to the Reviewer a8CV's C1.

Reviewer_ECyAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SEGA (Multi-view 3D Instance Segmentation with Geometry-guided Association), a framework designed to jointly perform 3D reconstruction and 3D instance segmentation for long video sequences. It first decomposes the video sequence to chunks that fit a foundation model to get 3D reconstruction parameters and 2D instance masks, then presents a 3D-aware alignment module to align local geometry to a global one with globally ID-consistent masks. In the end it performs a post-processing Instance-aware bundle adjustment to refine the scene geometry. The proposed method can handle up to thousands of frames and achieves state-of-the-art results on ScanNet200 and ScanNet++v2. It also improves downstream 3D Gaussian panoptic lifting and rendering.

Critical strengths
  1. Instead of treating 3D reconstruction and 3D instance segmentation as a sequence of work and only relying on 3D reconstruction to perform 3D instance segmentation, this method introduces a bi-directional mechanism to let the two tasks help improve each other. And the performance gain validates the effectiveness of this design.
Critical weaknesses
  1. The paper heavily relies on content in the appendix, e.g. in Line 147 the key geometric components and Line 219 the sparse feature tracker, also L215 refers to equation (12). The important concepts and algorithms should be at least briefly described in the main paper.

  2. The whole pipeline is complex consisting of several components, so it heavily relies on the success of several models, e.g. PanSt3R, SAM3 and the bundle adjustment. It's unclear if any of these fails, will the whole pipeline recovers by the following steps.

Questions for authors
  1. How sensitive is the graph matching and 3D propagation to a noisy initial geometry prediction? If two different objects were wrongly merged during graph matching, will the IA-BA identify and correct it or it reinforce the false correspondence?
  2. How much latency does using SAM3 leads to, is it a bottleneck? Do we have a ligher-weight alternative?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ECyA (1/3)

We thank the reviewer for recognizing the benefits of the bidirectional interaction between reconstruction and instance segmentation. We address the requests for clearer implementation details, characterization of error recovery, and computational analysis below.

C1. Main-paper self-containment and geometric implementation details

Thank you for this suggestion. We agree that the essential geometric operations should be understandable from the main paper. We have revised the Section 3.1 with concise Background in the main paper. We greatly value your suggestion.

C2. Pipeline dependencies and failure recovery

Thank you for raising this concern. Recovery depends on the error type and the valid evidence remaining available. SEGA includes downstream stages designed to address specific upstream errors: propagation can extend incomplete object observations, negative prompting discourages tracking drift onto distractors, and video refinement consolidates duplicate or fragmented tracks.

Our submitted ablations support the contributions of these stages. In Table 4, removing 3D-aware prompting reduces AP from 18.11 to 15.62, while removing video refinement reduces AP to 16.15. The geometric evaluations, now reported in Tables 12–13, additionally show that IA-BA improves reconstruction using imperfect predicted instance associations. These results demonstrate the usefulness of the corrective stages, rather than guaranteeing recovery from arbitrary component failures.

We further examine whether revisiting upstream predictions benefits the pipeline through the additional multi-round experiment in Table 15. After geometric refinement, we recompute instance associations and propagated masks using the refined geometry, followed by another IA-BA pass. From one to four passes, AP increases from 28.4 to 29.9, AR from 54.2 to 59.6, and AUC@30 from 82.42 to 83.48, while Chamfer distance decreases from 0.0403 to 0.0391.

These joint improvements support the benefit of revisiting association and propagation after geometric refinement. They show that imperfect intermediate predictions can be improved through subsequent processing in the evaluated setting.

AuthorsOpenReview ↗

Response to Reviewer ECyA (2/3)

C3. Sensitivity to noisy proxy geometry

Thank you for this question. Graph matching and propagation depend on the cross-view correspondences established through the proxy geometry. SEGA uses the depth-consistency check in Eq. (7) to reject geometrically inconsistent correspondences, although this filtering cannot eliminate all errors in the proxy geometry.

Our geometry-quality experiment, now Table 9 in Appendix A.3.2, examines this dependence. Replacing predicted proxy geometry with GT depth and cameras, then rerunning association and propagation, improves AP from 28.4 to 32.5 and AR from 54.2 to 65.6. The initial local masks and hyperparameters remain fixed, and both variants are evaluated using the same GT geometry. The improvement therefore reflects better instance predictions resulting from more accurate proxy geometry, rather than a change in the geometry used for evaluation.

As a complementary parameter study, Table 7(a) varies the depth-consistency threshold from 0.05 to 0.3, yielding 16.5–18.1 AP, while Table 7(b) varies the 3D-IoU matching threshold from 0.3 to 0.6, yielding 17.5–18.1 AP. Overall AP thus varies by 1.6 points across the tested depth thresholds and 0.6 points across the matching thresholds; sensitivity differs across metrics.

Together, these experiments show that proxy quality affects association and propagation.

C4. Incorrect object merges and geometric feedback

Thank you for highlighting this case. IA-BA refines geometry but does not split or reassign instance identities. Instance agreement filters existing feature correspondences; sharing an object identity does not itself create a match. Nevertheless, an incorrect merge can allow erroneous cross-object correspondences to survive filtering and bias refinement. The robust reprojection objective reduces the influence of large residuals, but cannot guarantee rejection of geometrically plausible wrong matches. Reliable correspondences from other objects can still constrain the shared camera poses.

We examine the effect of guidance quality in Table 14. With the same predicted initialization, sparse-track candidates and iteration budget, predicted-guidance IA-BA improves AUC@30 from 81.33 to 82.42 compared with ordinary BA, and reduces Chamfer distance from 0.0409 to 0.0403. Replacing predicted guidance with GT instance masks further improves these metrics to 83.56 and 0.0391, respectively. Thus, predicted guidance remains useful despite its imperfections, while more accurate mask and identity information yields further improvements.

To revisit the identities themselves, our additional multi-round experiment in Table 15 recomputes association and propagation using the refined geometry before another IA-BA pass. This allows subsequent optimization to use updated instance predictions, rather than repeatedly retaining the original assignments. The joint segmentation and reconstruction improvements support this alternating process. They establish aggregate benefits with predicted associations.

AuthorsOpenReview ↗

Response to Reviewer ECyA (3/3)

C5. SAM3 latency and a lighter-weight FastSAM alternative

Thank you for this question. We additionally measure masking-stage latency and evaluate a FastSAM-based alternative on the same 400 input frames using eight NVIDIA L40S GPUs. Table E (Table 16 in the revised manuscript) reports segmentation quality and runtime.

The SAM3-based masking stage takes 984 seconds out of 1,806 seconds, accounting for approximately 54.5% of end-to-end runtime. It is therefore a major latency bottleneck in this configuration.

The FastSAM-based variant reduces masking-stage time from 984 to 211 seconds, corresponding to a 4.66× masking-stage speedup. End-to-end runtime decreases from 1,806 to 988 seconds, giving a 1.83× overall speedup. This comes with a measurable quality trade-off: AP decreases from 18.1 to 17.5, AR from 32.3 to 28.7, and recall at IoU 0.25 from 76.7 to 68.3.

Table E — Accuracy–runtime comparison 400 frames L40s 8 GPUs. (Table 16 in the manuscript)

SEGA variantAP ↑AP50 ↑AP25 ↑AR ↑RC50 ↑RC25 ↑Masking-stage time (s) ↓End-to-end time (s) ↓Overall speedup ↑
SAM3-based18.133.847.232.357.876.79841,8061.0$\times$
FastSAM-based17.531.243.928.750.468.32119881.8$\times$

These results provide a faster alternative when reduced latency is prioritized, with a small accuracy and recall costs trade-off. SAM3 provides higher segmentation quality; FastSAM substantially reduces runtime.

Reviewer_a8CVOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEGA splits a long RGB sequence into overlapping clusters and aligns the local reconstructions to a shared proxy geometry. It uses this geometry to associate and propagate instance masks, then filters feature correspondences using consistent instance IDs for bundle adjustment. The final output is a scene-level 3D instance segmentation.

Critical strengths

Using a shared proxy geometry for local reconstruction alignment, cross-cluster instance matching, and propagation is a coherent design. Local objects from different clusters can then be associated in the same scene coordinate system, which suits the identity-consistency problem in long sequences.

Critical weaknesses

The relation between geometric constraints and object completion needs clarification. The existing 3D support is incomplete, yet propagated regions outside this support are used to generate negative prompts. It remains unclear how newly visible object regions are retained, and whether the object support is updated during propagation.

Questions for authors
  1. Clarify when negative prompts are applied and how object support is updated, and show propagation cases where new surfaces appear.
  2. I suggest directly evaluating the 3D instances on the final predicted point cloud in one representative setting. The separate geometry results are useful, but pixel-aligned AP mainly reflects mask and identity consistency.
  3. How do cluster size and overlap affect reconstruction quality, instance consistency, and computational cost?
  4. Whether IA-BA remains reliable when instance associations are incorrect still needs direct validation.
6 comments
AuthorsOpenReview ↗

Response to Reviewer a8CV (1/4)

We thank the reviewer for recognizing the coherent use of shared proxy geometry to maintain consistent object identities across local reconstruction clusters. We greatly appreciate the suggestions and now clarify object completion and strengthening the evaluation.

C1. Object completion, negative prompting, and support updates

We really appreciate the reviewer for the need of clarification of our negative prompting. The reason behind this design is occlusion changes the reconstructed visible object regions and can make propagation switch to a neighboring distractor. We therefore constrain the geometric agreement during propagation, mitigating possible drifts.

The reference object support (QaQ_a) is aggregated from the initially matched observations and remains fixed throughout the propagation pass; propagated masks do not update it. It anchors the object’s identity without defining its complete boundary.

As SAM3 produces each mask, we compare its associated proxy-point IDs with (QaQ_a). Negative correction is applied only when no shared points remain, indicating possible drift. We then sample negative prompts inside the suspected drifting prediction, add them to SAM3’s prompting state, and continually update propagation from that frame without backward repropagating. Subsequent predictions therefore incorporate the corrective prompts; earlier frames are not repropagated backward. The reference support (QaQ_a) remains unchanged.

Crucially, the correction criterion is loss of reference overlap, not growth beyond the reference boundary. If (QaQ_a) contains chair-seat points, a prediction covering both the seat and a newly revealed backrest retains those reference points and receives no negative correction—even if the backrest substantially enlarges the mask. If propagation instead switches entirely to a neighboring table and loses all chair-reference points, the zero-overlap condition activates correction.

We have clarified this rule in Eq. (11) and Appendix A.1, with Algorithm 1 specifying the fixed-support behavior and correction procedure. Figure 4 is additionally provided to show qualitative comparisons of the projected reference support, negative-prompt locations, and predictions with correction disabled and enabled.

The additional ablation in Table A shows improved overall mask quality and object-pixel coverage. Enabling correction increases AP from 10.3 to 18.1, object-mask mIoU from 24.51% to 40.33%, and object-pixel recall from 50.49% to 67.09%. These overall measurements do not separately quantify newly visible surfaces.

Table A — Effect of negative correction on object-mask quality of ScanNet++v2. (Table 5 in the manuscript)

VariantAPARObject-mask mIoUObject-pixel Recall
SEGA + negative prompt OFF10.320.424.5150.49
SEGA + negative prompt ON18.132.340.3367.09

Metric definitions: AP and AR are mentioned in the paper as 3D segmentation Average Precision and Recall. For 2D, Object-mask mIoU measures average overlap between predicted and GT object masks, accounting for missed and incorrectly included pixels. Object-pixel recall measures the proportion of GT object pixels recovered by the predictions.

AuthorsOpenReview ↗

Response to Reviewer a8CV (2/4)

C2. Evaluation on the final predicted point cloud

Thank you for this suggestion. We additionally evaluate instance segmentation using the actual predicted 3D coordinates on 400-frame ScanNet++v2 sequences. This complements pixel-aligned AP by assessing spatial instance agreement after reconstruction.

We align each predicted scene to GT using scene-level Sim(3) alignment followed by ICP. For each annotated GT mesh vertex, we find its nearest point in the aligned predicted cloud and transfer its predicted instance membership if the distance is within 0.2 m. Unmatched annotated GT vertices remain in the evaluation rather than being excluded. We report class-agnostic AP, AP50 and AP25, assigning every predicted instance confidence 1.0 and using the same evaluator and tie-handling rules across methods.

For this comparison, memory constraints require us to process IGGT and PanSt3R in chunks of at most 50 images. We associate their chunk-level instance predictions across the sequence using the grouping adaptation Open3DIS-style described in the supplementary material, and evaluate the resulting instances on SEGA’s reconstructed geometry. These are therefore adapted segmentation comparisons under shared geometry, rather than evaluations of the baselines’ native end-to-end reconstructions.

Table B — Instance segmentation evaluated using predicted 3D coordinates. (Table 11 in the manuscript)

MethodAP ↑AP50 ↑AP25 ↑
Chunked PanSt3R + SEGA reconstruction1.554.9816.52
Chunked IGGT + SEGA reconstruction0.532.179.91
SEGA without IA-BA2.186.0820.80
SEGA with IA-BA2.316.3920.93

The first three rows share SEGA’s pre-IA-BA reconstruction, allowing comparison of their instance predictions under common geometry. The final two rows retain identical video masks and differ only in geometric refinement. IA-BA increases AP/AP50/AP25 from 2.18/6.08/20.80 to 2.31/6.39/20.93. These gains show that refining the predicted geometry improves spatial instance agreement in this evaluated setting, beyond what pixel-aligned evaluation alone can establish.

The low absolute AP highlights the difficulty of evaluating instances using their actual predicted 3D coordinates. Unlike our pixel-aligned and standared GT-geometry evaluation, this protocol transfers instance memberships through spatial nearest-neighbor matching after global alignment. Residual coordinate errors can transfer labels to incorrect surfaces, while annotated GT vertices without a neighbor within 0.2 m remain unassigned and contribute to missed object coverage.

AuthorsOpenReview ↗

Response to Reviewer a8CV (3/4)

C3. Cluster size and overlap

Thank you for this question. We additionally evaluate cluster size and overlap on the same 400 input frames using 8 NVIDIA L40S GPUs, keeping the remaining settings fixed. Table C reports instance segmentation, camera-pose and reconstruction quality, cluster count, and end-to-end runtime. These results are included in the revised manuscript’s hyperparameter analysis (Table 6).

Accuracy shows limited sensitivity to cluster size within the tested feasible range. With overlap fixed at five frames, 25- and 50-frame clusters achieve comparable results: 17.9 versus 18.1 AP, 82.39 versus 82.42 AUC@30, and 0.9396 versus 0.9401 F-score. However, smaller clusters require more chunks and increase runtime from 1,806 to 2,398 seconds. Increasing cluster size to 75 frames exceeds the available GPU memory in this setting.

Sufficient overlap is important for maintaining consistency between chunks. Shared observations constrain inter-chunk alignment and provide the common geometric basis for subsequent instance association. This is consistent with the use of overlapping frames for alignment in chunk-based reconstruction methods such as VGGT-Long. Reducing overlap from five to two frames lowers AP from 18.1 to 17.2, AUC@30 from 82.42 to 80.15, and F-score from 0.9401 to 0.9033, while saving only 3.5% runtime. Increasing overlap to ten frames improves AP to 18.3 and F-score to 0.9487, at a 13.7% runtime increase.

These results support our 50-frame cluster size with 5-frame overlap as a practical accuracy–runtime balance among the tested configurations.

Table C — Cluster-size and overlap sensitivity on 400 frames using 8 L40s GPUs (Table 6 in the manuscript)

Cluster sizeOverlapAP ↑AR ↑AUC@30 ↑F-score ↑Number of clustersTotal time (s) ↓
25517.933.182.390.9396202,398
50217.229.480.150.903391,743
50518.132.382.420.940191,806
501018.333.782.590.9487102,053
755OOMOOMOOMOOM6OOM

Bold identifies the default configuration. OOM denotes an out-of-memory failure; the cluster count in that row indicates the intended partition.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their thoughtful and constructive feedback. We have revised the manuscript to clarify SEGA’s methodological contribution and implementation, strengthen the evaluation of geometric feedback, and document computational and generalization trade-offs. The revised paper adds eight experiment tables (Table 5-6, 11, 14-18), Algorithm 1, and three figures (Figure 4-6). We summarize the major additions and clarifications below.

  1. A more self-contained method and precise characterization of geometric feedback. New Section 3.1 explains projection/back-projection, robust similarity alignment, and sparse feature-track construction. Section 3.4 and Eq. (12) clarify that IA-BA uses global instance agreement to filter existing sparse correspondences while retaining the confidence-weighted robust reprojection objective. It introduces neither additional instance-dependent weights nor dense surface or object-level residuals.
  2. Clarified negative prompting, supported by an ablation and pseudocode. Eq. (11), Appendix A.1, and Algorithm 1 specify that the reference support remains fixed during each propagation pass. Negative correction is triggered by zero shared proxy-point IDs, rather than by mask growth beyond the reference support. Table 5 compares correction enabled and disabled. Figure 4 visualizes the reference support, negative-prompt locations, and predictions with correction enabled and disabled.
  3. Evaluation on the final predicted point cloud. New Appendix A.3.4 and Table 11 complement pixel-aligned evaluation with instance evaluation using actual predicted 3D coordinates. After Sim(3) alignment and ICP, instance memberships are transferred to annotated GT vertices through nearest-neighbor mapping within 0.2 m, with unmatched annotated vertices retained in scoring. Adapted baseline comparisons use common SEGA geometry, while the IA-BA comparison holds video masks fixed.
  4. Cluster-size, overlap, and computational sensitivity. New Table 6 in Appendix A.2 compares configurations on the same 400 frames using 8 L40S GPUs, reporting segmentation, pose and reconstruction quality, cluster count, runtime, and memory failures. The two feasible cluster sizes produce similar accuracy at fixed overlap, while smaller clusters increase runtime. The overlap sweep quantifies the accuracy–runtime trade-off and supports the default 50-frame clusters with 5-frame overlap among the tested configurations.
  5. An ordinary-BA control and an instance-guidance quality study. New Table 14 in Appendix A.4 compares ordinary BA and IA-BA from the same saved initialization, using the same sparse-track candidates, optimization settings, and additional iteration budget. A GT-instance-guided variant further examines guidance quality without supplying GT geometry. We also clarify how false splits and merges affect filtering and that IA-BA does not itself reassign instance identities.
  6. Alternating refinement and downstream visual evidence. New Table 15 evaluates an additional variant that recomputes association and propagation after geometric refinement, followed by another IA-BA pass Cumulative runtime is reported alongside these gains. New Figure 5 complements the Table 2(b) with downstream RGB and instance renderings with and without IA-BA under fixed video-mask supervision. These results support aggregate improvement through alternating refinement.
  7. Masking-stage latency and a FastSAM alternative. New Appendix A.5 and Table 16 show that SAM3-based masking accounts for 984 of 1,806 seconds of end-to-end runtime. The FastSAM-based variant reduces masking time and total runtime. We explicitly report the accompanying quality trade-off.
  8. Transfer to Replica and a real-world iPhone capture. New Appendix A.6 and Tables 17–18 evaluate the unchanged SEGA configuration on Replica without dataset-specific fine-tuning or hyperparameter adjustment. These experiments cover predicted camera poses and reconstruction, together with instance segmentation under GT geometry. New Figure 6 provides complementary qualitative results on an iPhone-captured indoor sequence without scene-specific tuning.

Project website. We have added a project website with interactive qualitative 3D scenes examples. We encourage the reviewers to try out these results.

These additions complement the submitted baseline comparisons, component ablations, threshold analysis, and oracle-proxy experiment. They clarify the scope of each mechanism, provide controlled evidence for instance-guided refinement, and characterize the observed benefits together with their computational costs and remaining limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.