- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes 4RROW, which combines TimeDINO video encoding, D4RT-style point-wise queries, and instance queries in the same feed-forward model to jointly handle dynamic geometry and instance recognition. The model can extract object motion and aggregate points from different frames to a selected reference time. The experiments report strong open-vocabulary VIS and competitive depth and point-cloud results. Joint geometry and semantic prediction is also evaluated on static ScanNet++ and dynamic Waymo
Critical strengths
The model performs strongly on open-vocabulary VIS while keeping competitive depth and point-cloud accuracy. In Figures 3–4 and the supplementary demonstrations, instance semantics can separate objects moving together, keep the static parts of objects, and support object-level point-cloud aggregation. These results make the value of the unified architecture more concrete
Both ScanNet++ and Waymo experiments include geometry quality in the evaluation. The oracle semantics and oracle geometry experiments on ScanNet++ further show which aspect limits the performance. This is more helpful than only reporting 2D segmentation results
Critical weaknesses
Figures 1, 3, 4 and the supplementary material show cross-frame aggregation, but I am still not clear how accurate the aggregated shapes are. I am also not sure whether the point-cloud errors in Table 1 already cover cross-time queries. DAP combines instance masks and depth, but does not directly evaluate the aggregated shapes. In Figure 4, the two methods also use different ways to extract the foreground, so the difference in geometry quality is not easy to see clearly. This is my main concern at present, because it relates to the new capability emphasized by the paper
Questions for authors
Which source/target time combinations are used in the point-cloud evaluation of Table 1?
What training and cross-frame instance association settings are used for the Pano3D baseline in the Waymo experiments?
Answers to questions
We thank the reviewer for their careful feedback. We are happy that our main contribution, which is also identified as strength #1 by the other reviewers, was highlighted in the summary: To our knowledge, our work presents the first method to combine reconstruction, tracking, and segmentation in a single framework, which enables multi-view reconstruction of individual dynamic objects.
Cross-frame aggregation
We agree this is an important point. As explained in our submission (L294) there are no established benchmarks for 4D reconstruction and recognition on dynamic scenes. Moreover, for real-world scenes obtaining ground-truth for cross-frame aggregation under partial visibility is practically impossible. Therefore in our submission we showed this capability only visually. Nevertheless, it will be possible to establish a new protocol to measure this on a synthetic dataset and we will do so in the final version of our manuscript. Since to our knowledge (as also noted by Reviewer imRq) no other model supports this capability, this evaluation will establish a reference baseline for future work.
Point-Cloud Evaluation
Point cloud evaluation always uses the same source and target times according to established existing protocol. The reviewer is correct that it does not cover cross-time queries; Similarly, the new protocol promised above is intended to fill this gap.
Cross-frame on Waymo
Both Pano3D and 4RROW use object queries which cross attend to all tokens of all frames, giving them a global view and directly producing temporally consistent instances across the clip. In contrast, for this task the point queries reference the specific frame (t_src == t_tgt == t_cam) since the ground-truth segmentation annotations label the observed pixels per frame.
Extracting foreground for D4RT and 4RROW
4RROW predicts instances directly. D4RT is incapable of this so we can only use the inferior technique of thresholded motion prediction, which also means that with D4RT one cannot isolate static objects. Illustrating this difference, rather than comparing geometry quality, is the purpose of Figure 4; we will clarify this in the caption.