acceptodds
Withdrawn from ICLR 2027

Persistent3D: Open-Vocabulary Persistent Object spatial Perception from Monocular Video via Cross-Frame Instance Association

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial perception for embodied intelligence requires persistently recognizing and understanding scene instances during dynamic 3D reconstruction. However, cross-frame instance association fails to maintain stable object identities under occlusion, visually similar instances, and substantial viewpoint changes. This prevents consistent aggregation of 3D information across views. To address this issue, we propose Persistent3D, an open-vocabulary framework for persistent object spatial perception from monocular video. Persistent3D first recovers 3D scene geometry and camera poses from video, and combines VLM-based open-vocabulary recognition with instance segmentation to construct per-frame object observations. Starting from frame-level instances, bidirectional temporal propagation and one-to-one mask matching are performed within local temporal windows to alleviate association failures caused by short-term occlusion and viewpoint variation. Meanwhile, a same-frame mutual exclusion constraint is introduced to maintain local cross-frame temporal associations and suppress erroneous merging between visually similar instances. For local identity fragmentation caused by long-term occlusion or viewpoint changes, instance reconnection is performed by leveraging scene-point index overlap, color distributions, and frame-set mutual exclusion constraints across different identity fragments in the 3D scene. During persistent spatial perception, a sequence-wide instance association graph is dynamically constructed to maintain stable persistent identities across frames. Building on this, Persistent3D fuses multi-view vision-language evidence to estimate functional object orientations and fits semantically oriented bounding boxes. The final unified 3D object representation integrates persistent identity, open-vocabulary semantics, 3D geometry, and functional orientation. Extensive experiments demonstrate consistent and effective improvements in cross-frame instance association, 3D information aggregation, and functional orientation estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied Intelligence, Spatial Perception, Open-World Scenes, Persistent Object Identity, Cross-Frame Instance Association

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f3DbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Persistent3D is an open-vocabulary framework for persistent object-level 3D perception from monocular video. It combines local bidirectional mask propagation, global identity constraints, and geometry-aware track reconnection to maintain object identities across occlusions and reappearances. These identities support multi-view geometry aggregation and functional orientation estimation. Experiments on ScanNet report improved overall association performance and reconstruction completeness, with ablations evaluating the main association components.

Critical strengths
  1. The paper addresses the practically relevant problem of maintaining object identities under occlusion, re-entry, and confusion among similar objects.

  2. The reported ScanNet results demonstrate better overall tracking performance and more complete instance-level reconstructions than competing methods.

  3. The ablation study examines several key design choices and provides evidence for the effectiveness of the individual components.

Critical weaknesses
  1. The novelty is unclear. While the problem is practically relevant, the paper does not clearly identify the specific bottlenecks in existing methods or explain what is new beyond combining existing techniques. In addition to the methods compared in the paper, relevant recent work includes MaskClustering [1] and Open3DIS [2] for multi-view instance aggregation, and OpenTrack3D [3] for cross-view instance consistency through visual-spatial tracking. The authors should more clearly explain how their method differs from these approaches and to what extent the reported gains stem from the proposed mechanisms rather than the use of stronger pretrained models.

  2. The evaluation is largely confined to ScanNet, limiting the evidence for generalization. The main quantitative results with clearly identified data sources are reported on ScanNet, and reconstruction is evaluated on only five scenes. Although ICL-NUIM, self-collected videos, and public online videos are listed in the dataset descriptions, no results or visualizations are explicitly attributed to them, leaving their role in the evaluation unclear. The authors should provide additional quantitative experiments on other datasets and clarify how these listed data sources were used..

  3. The computational cost and practical applicability are not evaluated. The framework combines multiple large pretrained models, bidirectional mask propagation, and multi-view VLM inference, potentially introducing substantial computational overhead. However, the paper reports neither end-to-end runtime nor a module-level timing breakdown. The authors should report the runtime along with the hardware configuration used.

[1] MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation. CVPR, 2024.

[2] Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance. CVPR, 2024.

[3] OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation. CVPR Findings, 2026.

Questions for authors
  1. How does the method handle inconsistent category labels for the same object across views, such as synonyms or different levels of specificity, given that track reconnection requires category consistency?

  2. How sensitive is track reconnection to pose drift and reconstruction errors in MASt3R-SLAM, especially when revisiting an object after a long trajectory?

Reviewer_ZbuTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Persistent3D constructs persistent object representations from monocular video by combining cross-frame association with reconstructed scene geometry. Bidirectional propagation, matching constraints, and fragment reconnection maintain object identities, which support multi-view geometry aggregation and functional-orientation estimation. Experiments evaluate tracking, reconstruction, and orientation, with component ablations supporting the association design.

Critical strengths
  1. The framework connects persistent identity tracking with object-level spatial understanding, addressing a useful problem involving occlusion, re-entry, and viewpoint changes.
  2. Component ablations support the association design, while the reconstruction results show improved completeness and F-score across five ScanNet scenes (Tables 2 and 4, pp. 8–9).
Critical weaknesses
  1. The tracking evaluation needs clarification. Appendix B, Eq. (B5), defines DetA differently from the standard HOTA formulation. Please clarify whether this is a typographical error or reflects the evaluator used for Table 1. Although Appendix B.1 and Table 5 document the data sources, the exact tracking sequences, evaluated frame counts, and the role of mask refinement remain insufficiently clear. These details are needed to assess the comparability and reproducibility of the reported tracking gains.
  2. Some of the claimed contributions lack direct empirical validation. The experiments concerning orientation rely solely on the authors' own results, with no comparisons against single-view or purely geometric baselines; consequently, it is difficult to disentangle the respective contributions of multi-view fusion, the foundation model, and category-specific rules. Similarly, the results for class-agnostic tracking do not provide a direct verification of open-vocabulary recognition capabilities.
Questions for authors
  1. Which HOTA evaluator is actually used? Does the evaluator used for Table 1 follow the standard HOTA definition, with Eq. (B5) being only a typographical error? If not, please report results under the standard definition.
  2. After fixing the basic model and category rules, how much better is multi-view orientation estimation compared to single-view, pure geometric methods?
Reviewer_V3TaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Persistent3D, an open-vocabulary framework for persistent object-level spatial perception from monocular RGB video, motivated by the difficulty of maintaining stable instance identities across frames under occlusion, visually similar objects, and substantial viewpoint changes during dynamic 3D reconstruction. It first reconstructs scene geometry and obtains per-frame instance observations through 2D open-vocabulary segmentation with Qwen and SAM2.1. For persistent identity association, it combines bidirectional temporal mask propagation and one-to-one matching with a global association graph that enforces same-frame mutual exclusion. Disconnected identity fragments are further reconnected using 3D scene-point overlap, color similarity, and frame-set exclusivity. Based on the resulting persistent identities, the framework aggregates multi-view 3D geometry, estimates functional object orientations through vision-language predictions, and constructs semantically oriented bounding boxes. Experiments show the proposed method outperforms several baseline.

Critical strengths
  1. This paper tackles an important and practical problem.
  2. The proposed method is well motivated.
  3. The paper has promising and strong empirical results over baseline methods in several benchmarks.
Critical weaknesses
  1. Limited novelty and insufficient discussion of related work. Persistent3D mainly integrates existing components, including MASt3R-SLAM, SAM2, Qwen VLM-based recognition, and geometric association. Several recent works are insufficiently discussed, including IGGT [1], MAG-3D [2], and UNITE [3]. The authors should better clarify the differences and unique contributions of Persistent3D compared with these approaches.

  2. Limited evaluation of 3D reconstruction. Table 2 evaluates only five ScanNet scenes, which is insufficient to establish robust improvements and may be sensitive to scene selection. Why not evaluate on the full benchmark or a substantially larger subset? Moreover, since geometric completeness is a claimed advantage, it would be valuable to report performance stratified by ground-truth object size, particularly for small objects, and compare against the baselines.

  3. Missing efficiency analysis. The framework involves a lengthy pipeline with multiple large pretrained models, potentially resulting in substantial computational overhead. However, inference speed and memory consumption are not reported. The authors should provide end-to-end runtime, per-module latency, and speed–accuracy comparisons with existing baselines to demonstrate the practical feasibility of the approach.

References: [1] Li et al. IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction. ICLR 2026 [2] Zheng et al. MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding. arXiv:2604.09167 [3] Koch et al. Unified Semantic Transformer for 3D Scene Understanding. TMLR 2026

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.