- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes to discover motion cue (dynamic object regions) from streaming 3D reconstruction method that uses TTT (test time training), ie ZipMap. Similar to Easi3R, which recovers dynamic object regions from feedforward 3D reconstruction model, the paper similarly shows how to recover dynamic object regions by probing cues from intermediate layers and gates from TTT layers.
The paper demonstrates that the extracted masks serve as moving object segmentation as a zero-shot approach. Also the masks can be used for suppressing dynamic region and improving 3D reconstruction and camera pose accuracy.
However, due to several weaknesses (unclear intuition/insights, unjustified gains, missing recent methods for comparison, unclear details), I would rate the paper as Weak Rejection.
Critical strengths
-
Adopting Easi3R-like approach for TTT-based 3D reconstruction
The paper shows that the Easi3R-like approach (discovering dynamic object mask from 3D reconstruction method) can be applicable to the TTT-based 3D reconstruction method which has its distinctive modules and behavior from it at read and write gate. In Fig. 3 and equation (2) shows that the writing operation involves with differentiating dynamics region from static region, ie .
-
Good downstream accuracy
Empirically, the method shows overall good pose, 3D reconstruction, and moving object segmentation accuracy, compared with previous methods.
Critical weaknesses
-
Lacking intuition and insights
The proposed formula to detect dynamic objects (ie, from Eq. (2) to Eq. (8)) are highly empirical. I am wondering how the equations are derived and what intuitions/insights are behind the equation. The core motivation share the same as Easi3R: discovering dynamic object mask from internal feature maps. I wonder what new lessons we can get from the paper, apart from the similar principle is applicable to TTT-based models. It will be interesting to analyze the TTT layers behavior on static vs dynamic region, eg what information is propagated via TTT layers. One could quantitatively measure the learning rate and gate values from write/read on both static and dynamic region. However those analyses are missing.
-
Good downstream accuracy is possibly inherited from the strong baseline model
The strong downstream performance for dynamic scenes is somewhat expected. ZipMap is already a very strong dynamic 3D reconstruction model as shown in its website. It already has the capability for handling dynamic scenes, by using 6 dynamic scene training datasets (BEDLAM, Dynamic Replica, Kubric, OmniWorld, PointOdyssey, and Spring). Unlike Easi3R, which demonstrates discovering dynamic region information from DUSt3R (strictly trained for 3D reconstruction), This diminishes the contribution of the paper.
Also, to be strictly fair, it's recommend to include in-depth qualitative & quantitative comparison with ZipMap as well (eg, Fig. 1, Fig. 5, and Fig. 6). According to Table 2 and 3, the accuracy improvement is marginal.
-
Missing recent results on benchmark tables (Table 2, 3)
Many published papers are missing for comparison, e.g, MapAnything, Pi-3, St4RTrack, D4RT, Any4D, V-DPM, VGGT-Omega, FastVGGT. It is recommended to include those methods for fair comparison.
-
Unclear details and intuitions for 'dynamic interference reduction'
The paragraph from Line 314 explains how the method utilizes the motion cues for improving downstream accuracy. However, its underlying assumptions and details are unclear.
-
I am curious what the motivation for suppressing dynamic interference is. ZipMap is already trained to reconstruct dynamic scenes in 3D. It's curious why we need to suppress it and in which cases it helps. Wouldn't it hurt the reconstruction accuracy on dynamic object regions?
-
What's the technical details on temporal stabilization (L323), smoothing translation, and gated rotation averaging on SO(3)? It will be great if further details are provided. It would be curious to see the performance without them. Also when applying these to the ZipMap baseline, does it improve their accuracy as well?
-
Questions for authors
-
Definition of 4D reconstruction
I wonder what the clear definition of 4D reconstruction is. Per-frame 3D reconstruction of dynamic scene can be just considered as dynamic 3D reconstruction. 4D reconstruction may imply 3D reconstruction with 3D motion estimation. It's curious how the paper interprets the meaning of 4D reconstruction. Also, it's curious if ZipMap is strictly considered a 3D reconstruction method from the paper's perspective.
-
Confusing words -- the meaning of some terms are not clear.
- L50: stateful
- L207: dominant frame-level representation
- L209: static ambiguity
-
Minor
- L213, L220: Eq. equation 1 Eq. (1)