- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the author proposed SM4RT a pipeline to reconstruct 4D scene with input of a Monocular video only. The main motivation of the method is how to present the motion of the scene. Comparing to previous method which present the motion of each 3D point separately, the proposed method group the points belong to same rigid part together and only estimate motion base for each rigid part instead of each points. The method first compute the geometry tokens of the whole input video using a pret-rained feed-forward model such as depth anything 3. Then, the geometry token are go through two separate branch. One branch is scene geometry branch which decodes the camera intrinsic and extrinsic, and output the depths. Finally, it unprojects the depth of the first frame into 3D world coordinates. The other branch is Motion Geometry Branch which compute the motion based and points assignments. At the end, it transforms the first frame 3D points to a dynamic 4D points based on the motion bases and points assignment. To validate the proposed method, the paper compare it with some SOTA baseline method on sereral bench marks such as depth, 3D tracking, 3D reconstruction and show it's strong performance.
Critical strengths
-
The method's contribution is strong and novel. Instead of presenting each 3D points separately, they assign 3D points to different rigid group and compute single motion base for each rigid group. In this way, it enforces a compact presentation of 3D motion and preserve object-level rigidness.
-
In their experiments, the shows competitive or improved performance across many benchmarks such as depth estimation, 3D tracking, and 3D reconstruction compare the SOTA baselines. It has potential broad application in the industry for area requiring accurate rigid dynamic objects reconstruction.
Critical weaknesses
-
The proposed method is better suit to rigid or articulated objects reconstruction only because it's rigid motion presentation. It could not perfectly reconstruct highly deformable objects such as clothes or if a rigid object undergoes deformation during a collision, but per point trajectory reconstruction methods don't apply such strict constraint and more flexible.
-
The validation experiments are most limited to synthetic datasets which less convincing. Synthetic could be not realistic and diverse enough. More validation experiments and comparison on real world datasets should be included.
Questions for authors
- Could author provide more result and analyze on real world datasets and scenes?
- Could author provide more quantitative result and analyze on highly deformable objects and compare with baselines?
- How's the performance of the proposed method change with different number of motion basses? Why you pick N=10?
- How does method handle a condition that the objects that are not visible in the first frame and appear in the middle?d
4 comments
Response to Reviewer (2/2)
Q3. Number of Motion Bases
We denote by N the number of motion bases. For the model, N is a hyperparameter fixed before training. We set N=10 because we consider ten bases sufficient to describe the motion of typical dynamic scenes, such as a few rigid parts, one articulated object, or several independently moving objects.
To verify this, we first estimate the number of bases that a scene actually uses from the entropy of the per-point motion assignment. We average the assignment weights over all dynamic points to obtain the share of the scene taken by each base, compute the entropy H of these shares, and take its exponential as the effective number of bases N_eff. N_eff is a soft count between 1 and N, so a scene whose dynamic points split evenly between two bases gives N_eff = 2. We measured it on three datasets:
| Dataset | mean H → N_eff | median H → N_eff | max over scenes H → N_eff | min over scenes H → N_eff |
|---|---|---|---|---|
| DAVIS | 1.333 → 3.79 | 1.373 → 3.95 | 2.073 → 7.95 | 0.119 → 1.13 |
| Kubric | 1.451 → 4.27 | 1.493 → 4.45 | 2.027 → 7.59 | 0.351 → 1.42 |
| PStudio | 1.578 → 4.85 | 1.612 → 5.01 | 2.047 → 7.74 | 0.481 → 1.62 |
The model uses roughly half of the budget on average (3.8–4.9 of 10). The simplest single-rigid scenes fall to 1.1–1.6 equivalent bases, and the most complex articulated sequences reach 7.6–8.0, so the effective count tracks scene complexity. Figure 8 of the revised manuscript visualizes the contribution of each base, with the corresponding sequences in the supplementary gifs.
Furthermore, we trained the model with N=5 and N=15 to check the budget from both sides (Table 9 of the revised manuscript). With N=5, All-APD drops by about 2 to 5 points across the four datasets, and the drop is largest on PStudio. This agrees with the entropy estimate above: PStudio has the highest average N_eff (4.85), close to 5, and its complex skeletal sequences use up to about 8 bases, so five bases are not enough to decompose them correctly. With N=15, performance does not improve further. N=10 keeps the best All-APD and the lowest Dynamic EPE on all four datasets, and N=15 is better only in Dynamic APD on DS and All-EPE on PO. This is also consistent with the entropy estimate, since even the most complex scenes use fewer than 10 bases, and the additional bases likely leave more room for a rigid part to be split across several bases. Together, these results show that 5 bases are too few and 15 bring no gain, which supports N=10 for the scenes we evaluate.
APD ↑ (All / Dyn)
| Setting | ADT | DS | PO | PStudio |
|---|---|---|---|---|
| N=5 | 86.46 / 79.23 | 79.60 / 70.28 | 79.84 / 68.81 | 75.43 / 75.43 |
| N=10 | 88.56 / 80.68 | 82.61 / 71.62 | 81.62 / 73.68 | 79.98 / 79.98 |
| N=15 | 87.70 / 80.44 | 80.11 / 72.70 | 81.51 / 71.69 | 77.62 / 77.62 |
EPE ↓ (All / Dyn)
| Setting | ADT | DS | PO | PStudio |
|---|---|---|---|---|
| N=5 | 0.1529 / 0.1972 | 0.2151 / 0.3197 | 0.2620 / 0.4152 | 0.2078 / 0.2078 |
| N=10 | 0.1339 / 0.0935 | 0.1918 / 0.2912 | 0.2578 / 0.3808 | 0.1754 / 0.1754 |
| N=15 | 0.1438 / 0.1759 | 0.2108 / 0.3003 | 0.2507 / 0.3981 | 0.1952 / 0.1953 |
Q4. Handling Late-Appearing Objects
SM4RT does not anchor tracking to the first frame. It processes the whole video jointly, predicts an assignment map for every frame and shares the motion bases across frames. The assignment is time-shared in the sense that a point keeps the assignment of its reference frame for all target frames. To track an object that appears later, we take a frame in which it is visible as the reference frame, so its points obtain their assignments from that frame.
The motion bases can be re-referenced to any frame without rerunning the model. Each base is a twist sequence relative to the first frame, so its motion from a reference frame to a target frame is obtained by composing the transform at the target frame with the inverse of the transform at the reference frame and mapping the result back to a twist. The points of the late-appearing object then blend these re-referenced bases with their assignments and follow the motion of the parts they belong to. Figure 13 in Appendix C.2 shows an example of an object that enters the scene mid-sequence.
The rebuttal has addressed most of my concerns and strengthened the experimental validation. I will maintain my original rating of 3 (Weak Accept).
Thank You
Dear Reviewer,
Thank you for your timely feedback and for taking the time to read our response. We’re glad the additional experiments and clarifications addressed most of your concerns.
Best Regards, The Authors
Response to Reviewer (1/2)
W1 & Q2. Performance on Deformable Objects
Sorry for the confusion. SoM does not require each object to be rigid. Our core assumption is that motion is structured: even when an object deforms, its motion is dominated by a few major components, and what varies across the object is how these components are mixed. A swinging arm in a loose sleeve moves mostly with the upper and lower arm, and the cloth between them follows a mixture of the two. A bending leaf rotates about its stem while its tip bends further, and a rigid object dented in a collision still moves mostly as a whole. SM4RT identifies such components as motion bases and composes them for each point through the soft assignment, so a deforming region is described by a spatially varying blend of several transforms, in the same spirit as skinning in character animation, which models the deformation of skin and clothing from a few bone motions. Figure 8 of the revised manuscript shows such compositions on skeletal and non-rigid sequences, where up to about eight bases are combined.
Three of our benchmarks consist mainly of non-rigid motion: PStudio (human bodies), DS and PO (humans and animals). SM4RT achieves the best APD on PStudio among all methods in Table 3, and the best All APD on DS and PO. Under the same backbone (Table 8), replacing the composition of bases with a point-wise displacement head lowers APD on all three datasets, by 4.08 points on PStudio. Reducing the number of bases from 10 to 5 also causes the largest drop on PStudio (79.98 to 75.43), which confirms that such motion is modeled by combining several bases.
When a deformation has no dominant components, such as a blooming petal, a few bases cannot capture all of its local detail. On such sequences, shown in Figure 11 (Appendix C.2), SM4RT performs comparably to the point-wise 4RC.
W2 & Q1. Real-world Performance
We agree that real-world evaluation is crucial, and our evaluation is not limited to synthetic datasets. We follow and combine the benchmarks used by prior work, namely TAPVid-3D [Ref. Koppula et al., 2024] and WorldTrack [Ref. Feng et al., 2025], which are also adopted by recent 4D methods [Ref. Liu et al., 2025]. They include the real-world ADT, PStudio and DriveTrack datasets, covering indoor multi-object, human-articulated and outdoor driving scenes, alongside the synthetic PO and DS datasets. This already provides broad coverage for this line of work. Prior work has not used more real-world datasets because annotating accurate 3D trajectories in real scenes is very costly, so precise metrics are difficult to compute beyond these benchmarks.
Still, we provide more qualitative video results on real-world datasets in the Supplementary Materials.