acceptodds
Under review as a conference paper at ICLR 2027

SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

Abstract

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by , rather than unstructured point-wise displacements. Building on this insight, we propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces Structure-of-Motion (SoM) to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in . Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from monocular RGB video. SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Reconstruction, Geometry of Motion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PcjsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposed SM4RT a pipeline to reconstruct 4D scene with input of a Monocular video only. The main motivation of the method is how to present the motion of the scene. Comparing to previous method which present the motion of each 3D point separately, the proposed method group the points belong to same rigid part together and only estimate motion base for each rigid part instead of each points. The method first compute the geometry tokens of the whole input video using a pret-rained feed-forward model such as depth anything 3. Then, the geometry token are go through two separate branch. One branch is scene geometry branch which decodes the camera intrinsic and extrinsic, and output the depths. Finally, it unprojects the depth of the first frame into 3D world coordinates. The other branch is Motion Geometry Branch which compute the motion based and points assignments. At the end, it transforms the first frame 3D points to a dynamic 4D points based on the motion bases and points assignment. To validate the proposed method, the paper compare it with some SOTA baseline method on sereral bench marks such as depth, 3D tracking, 3D reconstruction and show it's strong performance.

Critical strengths
  1. The method's contribution is strong and novel. Instead of presenting each 3D points separately, they assign 3D points to different rigid group and compute single motion base for each rigid group. In this way, it enforces a compact presentation of 3D motion and preserve object-level rigidness.

  2. In their experiments, the shows competitive or improved performance across many benchmarks such as depth estimation, 3D tracking, and 3D reconstruction compare the SOTA baselines. It has potential broad application in the industry for area requiring accurate rigid dynamic objects reconstruction.

Critical weaknesses
  1. The proposed method is better suit to rigid or articulated objects reconstruction only because it's rigid motion presentation. It could not perfectly reconstruct highly deformable objects such as clothes or if a rigid object undergoes deformation during a collision, but per point trajectory reconstruction methods don't apply such strict constraint and more flexible.

  2. The validation experiments are most limited to synthetic datasets which less convincing. Synthetic could be not realistic and diverse enough. More validation experiments and comparison on real world datasets should be included.

Questions for authors
  1. Could author provide more result and analyze on real world datasets and scenes?
  2. Could author provide more quantitative result and analyze on highly deformable objects and compare with baselines?
  3. How's the performance of the proposed method change with different number of motion basses? Why you pick N=10?
  4. How does method handle a condition that the objects that are not visible in the first frame and appear in the middle?d
4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

W1 & Q2. Performance on Deformable Objects

Sorry for the confusion. SoM does not require each object to be rigid. Our core assumption is that motion is structured: even when an object deforms, its motion is dominated by a few major components, and what varies across the object is how these components are mixed. A swinging arm in a loose sleeve moves mostly with the upper and lower arm, and the cloth between them follows a mixture of the two. A bending leaf rotates about its stem while its tip bends further, and a rigid object dented in a collision still moves mostly as a whole. SM4RT identifies such components as motion bases and composes them for each point through the soft assignment, so a deforming region is described by a spatially varying blend of several transforms, in the same spirit as skinning in character animation, which models the deformation of skin and clothing from a few bone motions. Figure 8 of the revised manuscript shows such compositions on skeletal and non-rigid sequences, where up to about eight bases are combined.

Three of our benchmarks consist mainly of non-rigid motion: PStudio (human bodies), DS and PO (humans and animals). SM4RT achieves the best APD on PStudio among all methods in Table 3, and the best All APD on DS and PO. Under the same backbone (Table 8), replacing the composition of bases with a point-wise displacement head lowers APD on all three datasets, by 4.08 points on PStudio. Reducing the number of bases from 10 to 5 also causes the largest drop on PStudio (79.98 to 75.43), which confirms that such motion is modeled by combining several bases.

When a deformation has no dominant components, such as a blooming petal, a few bases cannot capture all of its local detail. On such sequences, shown in Figure 11 (Appendix C.2), SM4RT performs comparably to the point-wise 4RC.

W2 & Q1. Real-world Performance

We agree that real-world evaluation is crucial, and our evaluation is not limited to synthetic datasets. We follow and combine the benchmarks used by prior work, namely TAPVid-3D [Ref. Koppula et al., 2024] and WorldTrack [Ref. Feng et al., 2025], which are also adopted by recent 4D methods [Ref. Liu et al., 2025]. They include the real-world ADT, PStudio and DriveTrack datasets, covering indoor multi-object, human-articulated and outdoor driving scenes, alongside the synthetic PO and DS datasets. This already provides broad coverage for this line of work. Prior work has not used more real-world datasets because annotating accurate 3D trajectories in real scenes is very costly, so precise metrics are difficult to compute beyond these benchmarks.

Still, we provide more qualitative video results on real-world datasets in the Supplementary Materials.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3. Number of Motion Bases

We denote by N the number of motion bases. For the model, N is a hyperparameter fixed before training. We set N=10 because we consider ten bases sufficient to describe the motion of typical dynamic scenes, such as a few rigid parts, one articulated object, or several independently moving objects.

To verify this, we first estimate the number of bases that a scene actually uses from the entropy of the per-point motion assignment. We average the assignment weights over all dynamic points to obtain the share of the scene taken by each base, compute the entropy H of these shares, and take its exponential as the effective number of bases N_eff. N_eff is a soft count between 1 and N, so a scene whose dynamic points split evenly between two bases gives N_eff = 2. We measured it on three datasets:

Datasetmean H → N_effmedian H → N_effmax over scenes H → N_effmin over scenes H → N_eff
DAVIS1.333 → 3.791.373 → 3.952.073 → 7.950.119 → 1.13
Kubric1.451 → 4.271.493 → 4.452.027 → 7.590.351 → 1.42
PStudio1.578 → 4.851.612 → 5.012.047 → 7.740.481 → 1.62

The model uses roughly half of the budget on average (3.8–4.9 of 10). The simplest single-rigid scenes fall to 1.1–1.6 equivalent bases, and the most complex articulated sequences reach 7.6–8.0, so the effective count tracks scene complexity. Figure 8 of the revised manuscript visualizes the contribution of each base, with the corresponding sequences in the supplementary gifs.

Furthermore, we trained the model with N=5 and N=15 to check the budget from both sides (Table 9 of the revised manuscript). With N=5, All-APD drops by about 2 to 5 points across the four datasets, and the drop is largest on PStudio. This agrees with the entropy estimate above: PStudio has the highest average N_eff (4.85), close to 5, and its complex skeletal sequences use up to about 8 bases, so five bases are not enough to decompose them correctly. With N=15, performance does not improve further. N=10 keeps the best All-APD and the lowest Dynamic EPE on all four datasets, and N=15 is better only in Dynamic APD on DS and All-EPE on PO. This is also consistent with the entropy estimate, since even the most complex scenes use fewer than 10 bases, and the additional bases likely leave more room for a rigid part to be split across several bases. Together, these results show that 5 bases are too few and 15 bring no gain, which supports N=10 for the scenes we evaluate.

APD ↑ (All / Dyn)

SettingADTDSPOPStudio
N=586.46 / 79.2379.60 / 70.2879.84 / 68.8175.43 / 75.43
N=1088.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
N=1587.70 / 80.4480.11 / 72.7081.51 / 71.6977.62 / 77.62

EPE ↓ (All / Dyn)

SettingADTDSPOPStudio
N=50.1529 / 0.19720.2151 / 0.31970.2620 / 0.41520.2078 / 0.2078
N=100.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
N=150.1438 / 0.17590.2108 / 0.30030.2507 / 0.39810.1952 / 0.1953

Q4. Handling Late-Appearing Objects

SM4RT does not anchor tracking to the first frame. It processes the whole video jointly, predicts an assignment map for every frame and shares the motion bases across frames. The assignment is time-shared in the sense that a point keeps the assignment of its reference frame for all target frames. To track an object that appears later, we take a frame in which it is visible as the reference frame, so its points obtain their assignments from that frame.

The motion bases can be re-referenced to any frame without rerunning the model. Each base is a twist sequence relative to the first frame, so its motion from a reference frame to a target frame is obtained by composing the transform at the target frame with the inverse of the transform at the reference frame and mapping the result back to a twist. The points of the late-appearing object then blend these re-referenced bases with their assignments and follow the motion of the parts they belong to. Figure 13 in Appendix C.2 shows an example of an object that enters the scene mid-sequence.

The rebuttal has addressed most of my concerns and strengthened the experimental validation. I will maintain my original rating of 3 (Weak Accept).

Dear Reviewer,

Thank you for your timely feedback and for taking the time to read our response. We’re glad the additional experiments and clarifications addressed most of your concerns.

Best Regards, The Authors

Reviewer_YvtrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SM4RT, a feed-forward model that jointly predicts geometry, camera parameters, and world-coordinate motion from monocular video in a single forward pass. Unlike existing 4D reconstruction methods that predict motion as independent per-point displacements, SM4RT models the fact that points on the same rigid object move together, and unlike prior structured approaches such as Shape of Motion, it does so without per-scene optimization. Its Structure-of-Motion (SoM) representation decomposes scene motion into a small set of motion bases, each a temporal sequence of se(3) twists, together with a sparse per-pixel assignment map shared across frames. Dense motion is obtained by blending the bases for each pixel and applying the resulting rigid transformation to the source-frame point, so that points with the same assignment share a rigid trajectory.

Critical strengths
  1. Feed-forward structured motion representation.

SE(3) motion-basis decompositions such as Shape of Motion have proven effective, but they require per-video optimization and rely on external depth and 2D track priors. SM4RT instead estimates time-shared bases and assignments together with geometry from monocular RGB video alone, in a single forward pass on top of the DA3 backbone, and the resulting assignment maps reveal articulated structure without part-level supervision. Unlike existing feed-forward 4D models that output only independent per-point trajectories, this provides structural information about which points move together without any per-scene optimization, which I consider the core contribution of the paper.

Critical weaknesses

Unclear whether the gains come from the SoM representation. SM4RT introduces the SoM representation together with the DA3 backbone, but there is no experiment comparing SoM against point-wise prediction on the same backbone. In Table 6, replacing only the backbone with VGGT causes a large performance drop, falling below several baselines, which makes it difficult to rule out that much of the reported improvement comes from the backbone. In addition, the ablations are limited to the backbone, injection strategy, and fine-tuning data, and the contributions of the components that make up SoM, such as the regularization losses, the number of bases, and SparseMax, are not analyzed. Since the shift in motion representation is the paper’s central claim, this validation is my main concern.

  1. Structure preservation depends heavily on post-processing.

The foreground deformation (VM) of the raw model output is at a similar level to that of point-wise methods. VM decreases substantially only when the hdbscan clustering post-processing (Adaptive) is applied, which considerably increases runtime and slightly lowers tracking accuracy. Although the raw model already shows low deformation in background regions, its effect on foreground objects appears limited relative to the claim that SoM enforces structure by construction.

Questions for authors
  1. Ablation isolating the effect of the representation Under the same DA3 backbone and training data, could the authors show results when the SoM head is replaced with a point-wise displacement head or a per-pixel SE(3) head? In Table 6, simply switching the backbone from VGGT to DA3 raises ADT APD substantially from 80.79 to 88.56, so it is important to distinguish whether the improvement over the baselines comes from the SoM representation or from the backbone and training data.

  2. Comparison with Shape of Motion Shape of Motion learns a similar representation, SE(3) motion bases with soft assignments, through per-scene optimization. Could the authors directly compare the two methods in terms of tracking accuracy, structure preservation (VM), and inference time? This would quantify both the benefits and the costs of the feed-forward formulation.

  3. Ablation on SoM components Since SparseMax already induces sparse assignments, why is entropy regularization additionally needed? Could the authors report results when the entropy, pseudo-background, and singularity losses are each removed? Also, what is the rationale for fixing N=10, and could the authors analyze how performance changes with N and how many bases are actually active in practice?

  4. Dynamic point tracking in Table 3 On DS and PO, the Dynamic APD of SM4RT is lower than that of V-DPM, and apart from ADT, it shows no clear advantage over existing methods. How do these results align with the claim that the structured representation benefits dynamic point tracking, and what explains the differences across datasets?

  5. Fair comparison for Adaptive grouping In Table 5, the large improvement in foreground VM comes mainly from the Adaptive post-processing. How would the metrics in Table 5 change if a similar post-processing step, such as trajectory-based clustering followed by per-cluster rigid fitting, were applied to the point-wise baselines? I believe a comparison with matched post-processing is needed.

  6. Quantitative evaluation of twist interpolation and extrapolation The motion interpolation and extrapolation in Figure 3 are shown only qualitatively. Could the authors provide a quantitative evaluation, such as reconstructing held-out intermediate frames or predicting future frames? Since the Introduction presents future state prediction as a capability of this representation, supporting evidence would be helpful.

9 comments
AuthorsOpenReview ↗

Response to Reviewer (1/6)

W1 & Q1. Isolating the Representation from the Backbone. We agree that isolating the representation from the backbone is important. We add two ablations in which only the motion head differs, with the same backbone, training data and schedule.

Ablation 1, head replacement on the identical DA3 backbone (Table 8 of the revised manuscript):

APD ↑ (All / Dyn)

Head (identical DA3 backbone)ADTDSPOPStudio
SoM: motion-token-weighted SE(3)88.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
Point-wise displacement85.22 / 72.0479.20 / 70.8680.64 / 72.6275.90 / 75.90
Per-point SE(3)85.10 / 72.0179.26 / 69.7175.04 / 62.9471.32 / 71.32

EPE ↓ (All / Dyn)

Head (identical DA3 backbone)ADTDSPOPStudio
SoM: motion-token-weighted SE(3)0.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
Point-wise displacement0.1740 / 0.27580.2156 / 0.30260.2559 / 0.38570.2066 / 0.2066
Per-point SE(3)0.1745 / 0.27530.2154 / 0.31210.2966 / 0.45610.2378 / 0.2378

With the DA3 backbone fixed, SoM outperforms the point-wise displacement head on every metric except PO All-EPE. This also shows where the gain over the baselines comes from. With a point-wise head, the DA3 backbone stays below the strongest baseline of Table 3 in All-APD on all four datasets (85.22 vs 85.96 for V-DPM on ADT, 79.20 vs 82.17 for 4RC on DS, 80.64 vs 81.28 for V-DPM on PO, and 75.90 vs 79.60 for D4RT on PStudio), whereas SoM is above it on all four. The backbone alone therefore does not explain the improvement over the baselines.

The per-point SE(3) head shows that SE(3) outputs alone are not the source of the gain either. Predicting a full transform for every point adds parameters that a single point correspondence cannot determine, and it does not encode that points on the same rigid part share one motion. Even with the singularity supervision, this head is close to the displacement head on ADT and DS and worse on PO (75.04 vs 80.64) and PStudio (71.32 vs 75.90). SoM instead predicts N shared transform sequences that many points jointly constrain, so the gain comes from the shared decomposition rather than from the SE(3) outputs.

Ablation 2, geometry-branch replacement on the identical VGGT backbone: Table 6 varies the backbone but keeps the original VGGT, which was not trained on motion data, so that comparison mixes backbone quality with the data it was trained on. To isolate the geometry side, we take the VGGT depth and camera branch already fine-tuned for motion by V-DPM, hold it fixed, train our motion module on it from scratch, and compare against V-DPM end-to-end.

APD ↑ (All / Dyn)

Model (VGGT backbone)ADTDSPOPStudio
SM4RT motion head + V-DPM geometry branch87.48 / 80.4476.19 / 76.0183.76 / 78.7078.52 / 78.52
V-DPM (end-to-end)85.96 / 72.1074.27 / 77.6681.28 / 81.8578.73 / 78.73

EPE ↓ (All / Dyn)

Model (VGGT backbone)ADTDSPOPStudio
SM4RT motion head + V-DPM geometry branch0.1466 / 0.19820.2387 / 0.23090.1698 / 0.20340.1797 / 0.1797
V-DPM (end-to-end)0.1666 / 0.13590.2548 / 0.21230.1899 / 0.17490.1812 / 0.1812

With the geometry branch held fixed, our motion module improves All-EPE on all four datasets and All-APD on ADT, DS and PO (+1.52 / +1.92 / +2.48), with a small decrease on PStudio (78.52 vs 78.73). On the dynamic subset, Dynamic APD rises on ADT and falls on DS and PO, which follows the trade-off between dynamic and static points analyzed in Q4.

Together, the two ablations show that the motion representation improves tracking when the backbone or the geometry branch is held fixed, and that the margin over the baselines in Table 3 requires SoM. The VGGT to DA3 gap in Table 6 reflects the geometry backbone, which is complementary to the motion representation.

AuthorsOpenReview ↗

Response to Reviewer (2/6)

W1 & Q3.1. Entropy Regularization on Top of SparseMax. SparseMax and entropy regularization act differently. SparseMax is the activation that turns the assignment logits into weights, and it sets the weights of irrelevant bases to exactly zero. It does not, however, decide among the remaining bases: with equal logits it returns equal weights, so an assignment such as [0.5, 0.5, 0, …] is sparse yet ambiguous between two motions. Near a motion boundary, such a mix of a static and a moving base produces intermediate motion rather than a clear separation. Entropy regularization is a loss that pushes the remaining weight toward fewer bases, while tracking supervision decides which base is selected. It does not impose a one-hot assignment, so bases can still be blended for non-rigid motion. Consistent with this, removing the entropy term lowers both All and Dynamic APD on all four datasets (Q3.2), and Figure 10 of the revised manuscript shows blurred motion boundaries without it.

W1 & Q3.2. Per-term Ablation of the Regularization Losses. Table 8 of the revised manuscript removes one term at a time.

APD ↑ (All / Dyn)

SettingADTDSPOPStudio
Full model88.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
w/o L_singular86.10 / 81.6382.42 / 72.0279.94 / 70.9378.85 / 78.85
w/o L_entropy87.50 / 78.2581.85 / 69.5679.04 / 69.3778.40 / 78.40
w/o L_pseudo84.91 / 78.4981.37 / 69.4878.48 / 69.1875.78 / 75.78

EPE ↓ (All / Dyn)

SettingADTDSPOPStudio
Full model0.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
w/o L_singular0.1575 / 0.09170.1941 / 0.28900.2663 / 0.39340.1810 / 0.1810
w/o L_entropy0.1441 / 0.10540.1977 / 0.30940.2735 / 0.40710.1968 / 0.1968
w/o L_pseudo0.1670 / 0.10400.2024 / 0.31010.2795 / 0.40890.2034 / 0.2034

All three terms improve tracking. The pseudo-background term has the largest effect. It penalizes the weight of the background base at pixels dominated by another base, which keeps the static background base from bleeding into dynamic regions and suppressing their motion. Removing it, like removing the entropy term discussed in Q3.1, lowers both All and Dynamic APD on all four datasets.

The twist singularity term addresses a different degeneracy. Without it, a dense SE(3) field can explain displacements through per-point translations alone and encode rotation as translations that vary across points. Such a field fits each dynamic point more freely, which is why removing the term slightly raises Dynamic APD on ADT and DS, but it loses the shared rigid motion, so All-APD drops on all four datasets. The loss of structure is clearer on Kubric-Rigid: with the metrics of Table 5, removing the term raises the clustered foreground vM from 0.0254 to 0.1048 and lowers raw AJ from 84.37 to 80.64.

MethodvM fg ↓vM fg (clustered) ↓AJ ↑APD ↑
SM4RT (Full)0.19050.025484.3790.62
SM4RT (w/o L_singular)0.19690.104880.6488.22
AuthorsOpenReview ↗

Response to Reviewer (3/6)

W1 & Q3.3. N=10 and Active Bases. We denote by N the number of motion bases. For the model, N is a hyperparameter fixed before training. We set N=10 because we consider ten bases sufficient to describe the motion of typical dynamic scenes, such as a few rigid parts, one articulated object, or several independently moving objects.

To verify this, we first estimate the number of bases that a scene actually uses from the entropy of the per-point motion assignment. We average the assignment weights over all dynamic points to obtain the share of the scene taken by each base, compute the entropy H of these shares, and take its exponential as the effective number of bases N_eff. N_eff is a soft count between 1 and N, so a scene whose dynamic points split evenly between two bases gives N_eff = 2. We measured it on three datasets:

Datasetmean H → N_effmedian H → N_effmax over scenes H → N_effmin over scenes H → N_eff
DAVIS1.333 → 3.791.373 → 3.952.073 → 7.950.119 → 1.13
Kubric1.451 → 4.271.493 → 4.452.027 → 7.590.351 → 1.42
PStudio1.578 → 4.851.612 → 5.012.047 → 7.740.481 → 1.62

The model uses roughly half of the budget on average (3.8–4.9 of 10). The simplest single-rigid scenes fall to 1.1–1.6 equivalent bases, and the most complex articulated sequences reach 7.6–8.0, so the effective count tracks scene complexity. Figure 8 of the revised manuscript visualizes the contribution of each base, with the corresponding sequences in the supplementary gifs.

Furthermore, we trained the model with N=5 and N=15 to check the budget from both sides (Table 9 of the revised manuscript). With N=5, All-APD drops by about 2 to 5 points across the four datasets, and the drop is largest on PStudio. This agrees with the entropy estimate above: PStudio has the highest average N_eff (4.85), close to 5, and its complex skeletal sequences use up to about 8 bases, so five bases are not enough to decompose them correctly. With N=15, performance does not improve further. N=10 keeps the best All-APD and the lowest Dynamic EPE on all four datasets, and N=15 is better only in Dynamic APD on DS and All-EPE on PO. This is also consistent with the entropy estimate, since even the most complex scenes use fewer than 10 bases, and the additional bases likely leave more room for a rigid part to be split across several bases. Together, these results show that 5 bases are too few and 15 bring no gain, which supports N=10 for the scenes we evaluate.

APD ↑ (All / Dyn)

SettingADTDSPOPStudio
N=586.46 / 79.2379.60 / 70.2879.84 / 68.8175.43 / 75.43
N=1088.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
N=1587.70 / 80.4480.11 / 72.7081.51 / 71.6977.62 / 77.62

EPE ↓ (All / Dyn)

SettingADTDSPOPStudio
N=50.1529 / 0.19720.2151 / 0.31970.2620 / 0.41520.2078 / 0.2078
N=100.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
N=150.1438 / 0.17590.2108 / 0.30030.2507 / 0.39810.1952 / 0.1953
Reviewer_F2VxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to constraint dense 3D point tracking with decoupled rigid-body motion. Based on an architecture similar to 4RC, it uses the motion head to predict segementaion maps of motion bases and correponding twist vectors, which are then blended to produce dense point tracking results. The model (SM4RT) is trained in three stages: the first stage on Kubric to learn the motion bases, the second stage on more dynamic datasets to further improve dense point tracking, the third on static and dynamic datasets to refine depth and poses. The final model's tracking and reconstruction performance is competitive on dynamic and static benchmarks. An additional evaluation on the introduced Kubric-Rigid-Eval benchmark also supports the effectiveness of constraining motion with N rigid-body motion to reduce shape distortion.

Critical strengths
  1. The idea of constraining motion to rigid-body motion is meaningful and underexplored. The visualized motion bases in Figure 7 look reasonably good.
  2. According to the evaluation, the tracking and reconstruction performance of SM4RT is competitive.
Critical weaknesses
  1. Althought the idea of structure-of-motion is interesting, the solution is limited in to a fixed 10 motion bases. It is not mentioned how 10 was chosen and not shown how well the motion bases prediction generalizes to realworld exmaples with clustered motion. More failure cases should be shown.

  2. There are quite some regularization terms such as the twist singularity term, entropy term, and background penalty term. Do they affect tracking performance? Or do they exist only to make the motion bases more reasonable? Qualitative and quantitative ablation study over these terms can be very informative. The loss weights λ\lambda should also be reported.

Questions for authors
  1. The visualized motion bases in Figure 7 look smooth. But are there cases when multiple motion bases capture the same rigid-body motion so that an rigid part is segemented into different parts? It is unclear whether the visualzied motion bases map in Fig. 1 and Fig. 7 are post-processed by the adaptive grouping strategy. A qualitative comparison w/ and w/o adaptive grouping will be ideal.
6 comments
AuthorsOpenReview ↗

Response to Reviewer (1/3)

W1. Rationale on motion base number selection

We denote by N the number of motion bases. For the model, N is a hyperparameter fixed before training. We set N=10 because we consider ten bases sufficient to describe the motion of typical dynamic scenes, such as a few rigid parts, one articulated object, or several independently moving objects.

To verify this, we first estimate the number of bases that a scene actually uses from the entropy of the per-point motion assignment. We average the assignment weights over all dynamic points to obtain the share of the scene taken by each base, compute the entropy H of these shares, and take its exponential as the effective number of bases N_eff. N_eff is a soft count between 1 and N, so a scene whose dynamic points split evenly between two bases gives N_eff = 2. We measured it on three datasets:

Datasetmean H → N_effmedian H → N_effmax over scenes H → N_effmin over scenes H → N_eff
DAVIS1.333 → 3.791.373 → 3.952.073 → 7.950.119 → 1.13
Kubric1.451 → 4.271.493 → 4.452.027 → 7.590.351 → 1.42
PStudio1.578 → 4.851.612 → 5.012.047 → 7.740.481 → 1.62

The model uses roughly half of the budget on average (3.8–4.9 of 10). The simplest single-rigid scenes fall to 1.1–1.6 equivalent bases, and the most complex articulated sequences reach 7.6–8.0, so the effective count tracks scene complexity.

For clustered motion, the motion bases can be combined through the soft assignment to form more complex motion, such as a group of objects moving together or a non-rigid part. We show several such examples in Figure 8 of the revised manuscript, with the corresponding sequences in the supplementary gifs.

We have added more failure cases to the revised manuscript in Figure 12. One example is over-segmentation, where a single rigid part is split across several bases, so the rigid prior no longer holds for the part as a whole. It occurs mostly on small non-dominant objects and on large foreground objects.

Furthermore, we trained the model with N=5 and N=15 to check the budget from both sides (Table 9 of the revised manuscript). With N=5, All-APD drops by about 2 to 5 points across the four datasets, and the drop is largest on PStudio. This agrees with the entropy estimate above: PStudio has the highest average N_eff (4.85), close to 5, and its complex skeletal sequences use up to about 8 bases, so five bases are not enough to decompose them correctly. With N=15, performance does not improve further. N=10 keeps the best All-APD and the lowest Dynamic EPE on all four datasets, and N=15 is better only in Dynamic APD on DS and All-EPE on PO. This is also consistent with the entropy estimate, since even the most complex scenes use fewer than 10 bases, and the additional bases likely leave more room for a rigid part to be split across several bases. Together, these results show that 5 bases are too few and 15 bring no gain, which supports N=10 for the scenes we evaluate.

APD ↑ (All / Dyn)

SettingADTDSPOPStudio
N=586.46 / 79.2379.60 / 70.2879.84 / 68.8175.43 / 75.43
N=1088.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
N=1587.70 / 80.4480.11 / 72.7081.51 / 71.6977.62 / 77.62

EPE ↓ (All / Dyn)

SettingADTDSPOPStudio
N=50.1529 / 0.19720.2151 / 0.31970.2620 / 0.41520.2078 / 0.2078
N=100.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
N=150.1438 / 0.17590.2108 / 0.30030.2507 / 0.39810.1952 / 0.1953
AuthorsOpenReview ↗

Response to Reviewer (2/3)

W2. Ablation on Regularizations

All three terms improve tracking, not only the interpretability of the bases. Removing any of them lowers All-APD on all four datasets, as shown below and in Table 8 of the revised manuscript.

APD ↑ (All / Dyn)

SettingADTDSPOPStudio
Full model88.56 / 80.6882.61 / 71.6281.62 / 73.6879.98 / 79.98
w/o L_singular86.10 / 81.6382.42 / 72.0279.94 / 70.9378.85 / 78.85
w/o L_entropy87.50 / 78.2581.85 / 69.5679.04 / 69.3778.40 / 78.40
w/o L_background84.91 / 78.4981.37 / 69.4878.48 / 69.1875.78 / 75.78

EPE ↓ (All / Dyn)

SettingADTDSPOPStudio
Full model0.1339 / 0.09350.1918 / 0.29120.2578 / 0.38080.1754 / 0.1754
w/o L_singular0.1575 / 0.09170.1941 / 0.28900.2663 / 0.39340.1810 / 0.1810
w/o L_entropy0.1441 / 0.10540.1977 / 0.30940.2735 / 0.40710.1968 / 0.1968
w/o L_background0.1670 / 0.10400.2024 / 0.31010.2795 / 0.40890.2034 / 0.2034

The background term L_background (L_pseudo in the manuscript) has the largest effect. It penalizes the weight of the background base at pixels dominated by another base, which keeps the static background base from bleeding into dynamic regions and suppressing their motion. The entropy term penalizes diffuse assignments, so that each point commits to a few bases instead of blending many. Removing either term lowers both All and Dynamic APD on all four datasets.

The twist singularity term addresses a different degeneracy. Without it, a dense SE(3) field can explain displacements through per-point translations alone and encode rotation as translations that vary across points. Such a field fits each dynamic point more freely, which is why removing the term slightly raises Dynamic APD on ADT and DS, but it loses the shared rigid motion, so All-APD drops on all four datasets. The loss of structure is clearer on Kubric-Rigid: with the metrics of Table 5, removing the term raises the clustered foreground vM from 0.0254 to 0.1048 and lowers raw AJ from 84.37 to 80.64.

MethodvM fg (raw) ↓vM fg (clustered) ↓AJ (raw) ↑APD (raw) ↑
SM4RT (Full)0.19050.025484.3790.62
SM4RT (w/o L_singular)0.19690.104880.6488.22

Figure 10 of the revised manuscript provides a qualitative comparison of the assignment maps with and without each regularization term.

The loss weights are listed below.

Loss termWeight
Point MSE1000
Twist singularity regularization200
Background regularization1
Entropy regularization0.5
AuthorsOpenReview ↗

Response to Reviewer (3/3)

Q1. Clarification on Visualizations

Yes, such cases exist. A single rigid part is sometimes split across several bases, mostly on small non-dominant objects and on large foreground objects, where keeping the assignment consistent over the whole part is harder. We show examples in Figure 12 of the revised manuscript. Since these bases describe the same rigid motion, the split mainly affects the structure read-out rather than tracking, and adaptive grouping merges them back into one group.

The assignment maps in Figure 1 and Figure 7 are direct model outputs without adaptive grouping. Figure 7 maps the soft assignment weights to color by PCA, so pixels with similar weights receive similar colors, which partly explains the smooth appearance. Adaptive grouping is applied only to the results labeled SM4RT (Adaptive), and we have stated this in the captions. Figure 6 compares the projected motion w/ and w/o adaptive grouping.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Reply

We thank all reviewers for their careful reading and constructive comments. We have added the following experiments and analyses and updated the manuscript accordingly.

  • Isolating the motion representation. On the same DA3 backbone and training data, replacing SoM with a point-wise displacement head or a per-point SE(3) head lowers APD on all four datasets (Table 8). With a point-wise head, the backbone alone does not surpass the strongest baselines, so the improvement comes from SoM. We also train our motion module on the fixed V-DPM geometry branch.
  • Ablation of SoM components. A per-term ablation of the entropy, pseudo-background and twist-singularity regularizers (Table 8 and Figure 10), the loss weights, and an explanation of why entropy regularization complements SparseMax.
  • Number of motion bases. The effective number of bases used per scene, estimated from the assignment entropy on DAVIS, Kubric and PStudio, and a comparison of N=5, 10 and 15 (Table 9 and Figure 8), which together support N=10.
  • Matched post-processing. Trajectory-based clustering and per-cluster rigid fitting applied to all point-wise baselines, including 4RC (Table 5). It brings limited structural gains to these methods, whereas SM4RT reaches a foreground deformation 4.4× lower than the best post-processed baseline.
  • Comparison with Shape-of-Motion on Kubric-Rigid in tracking accuracy, structure preservation and runtime.
  • Dynamic-point tracking. Static-side metrics that complement Dynamic APD (Table 10), and an analysis of the differences across datasets.
  • Interpolation and extrapolation. A quantitative evaluation on WorldTrack against linear baselines (Tables 13 and 14).
  • Deformable and real-world scenes. Results on non-rigid human and animal motion, background stability on real-world DAVIS videos (Table 12 and Figure 9), additional real-world visualizations, failure cases (Figures 11 and 12), and tracking of objects that appear mid-sequence.

We hope these additions address the reviewers' concerns and further support the benefits of structured motion representation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.