acceptodds
Withdrawn from ICLR 2027

Velo3R: Motion-Guided Unified Online 4D Human-Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Online 4D human–scene reconstruction aims to recover human motion, camera trajectory, and scene geometry from a monocular video as it arrives, using only the current and preceding frames and expressing every estimate in a shared world coordinate system. Maintaining coherent motion is especially challenging online, as occlusions and ambiguities between body and camera movement must be handled without access to future frames. These ambiguities can lead to inconsistent frame-to-frame estimates, producing jitter and unreliable global trajectories. We seek an explicit link between successive body states to encourage temporal consistency while allowing new observations to correct the reconstruction. To this end, we introduce Velo3R, which uses explicit motion estimates to refine the latent human representations used for current-frame mesh prediction. Conditioned on temporal tokens and previously reconstructed body states, our conditional velocity field links joint positions, velocities, and accelerations through integration and differentiation. Inspired by physics-informed neural networks, we jointly supervise these quantities and feed field-derived motion features back into human representations before mesh regression. Experiments on EMDB-2 and RICH demonstrate improvements in global motion accuracy and temporal consistency. Compared with Human3R, Velo3R reduces W-MPJPE from 267.9 to 259.6 mm on EMDB-2 and from 184.9 to 170.6mm on RICH. It also reduces world-coordinate jitter by 9.9% and 14.5%, respectively, with lower foot sliding on both datasets. These results highlight the value of explicit motion feedback for temporally coherent online 4D human–scene reconstruction. Our implementation will be made available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
online human-scene reconstruction, human mesh reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sZXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses causal, online human–scene reconstruction from monocular video by incorporating a joint-state-conditioned continuous velocity field into a recurrent visual reconstruction framework. The field connects joint positions, velocities, and accelerations through integration and differentiation along its predicted trajectory. Its motion estimates are fed back into the human representation before mesh regression, while joints extracted from the previous final mesh anchor the next transition. On EMDB-2 and RICH, the complete system improves world-space joint accuracy and several temporal metrics over the evaluated Human3R baseline.

Critical strengths
  1. Incorporating higher-order dynamics into a human–scene reconstruction model is a well-motivated idea.

  2. Velo3R performs well in terms of world-grounded human motion accuracy and temporal consistency, achieving state-of-the-art performance among online methods.

  3. The paper is well written and easy to follow, with a clear presentation.

Critical weaknesses
  1. The qualitative results are not compelling. Even upon zoom-in and careful inspection, it is still difficult to discern clear improvements over Human3R. In some cases, Velo3R appears noticeably worse than the Human3R baseline. For example, after repeatedly comparing the results for 05_twopersons_running, it is observed that Human3R produces better overall reconstruction quality, with smoother acceleration and less jitter. In 11_basketball_practice, the main difference highlighted by the authors is an identity switch in Human3R, which is not the primary focus or claimed improvement of this work. Beyond this difference, Velo3R appears comparable to Human3R in scene reconstruction quality, motion quality, and human–scene alignment, including artifacts such as feet penetrating below the ground surface, with no discernible improvement.

  2. In Table 2, Velo3R consistently underperforms OnlineHMR in local pose accuracy, without sufficient analysis or explanation.

  3. The experiments do not establish whether explicitly representing high-order motion quantities (velocity and acceleration) is more effective than learning them implicitly. For example, could comparable results be achieved by directly applying higher-order supervision to the Human3R baseline's human head during training?

  4. The improvements attributable to some higher-order supervision terms are very small. In Table 5, velocity supervision alone achieves a W-MPJPE of 259.8 mm, while full position, velocity, and acceleration supervision achieves 259.6 mm. Adding acceleration supervision without position supervision instead increases W-MPJPE to 264.2 mm and W-Accel from 43.4 to 43.5. In Table A1, adding the bone term worsens all five reported metrics, including W-MPJPE from 265.9 to 266.2 mm and W-Jitter from 210.7 to 213.3. These results provide limited evidence that each constraint contributes a consistent benefit.

  5. Despite presenting Velo3R as a unified human–scene reconstruction model, the paper does not provide results or evaluations specifically demonstrating the degree of alignment between the reconstructed humans and scenes. In light of related work such as UniSH and SHOW, it remains unclear whether Velo3R places the reconstructed scene and humans in a shared 3D space at a consistent scale, or whether the human meshes and scene point clouds are spatially aligned.

Questions for authors

See weaknesses.

Reviewer_JkoiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel method for online 4D human-scene reconstruction built upon the prior work Human3R. It introduces an additional velocity field prediction module to output joint velocities from the history joint positions and temporal features and use the resulting motion feature to help improve global human motion reconstruction. Experiment results and ablation studies show the effectiveness of the proposed method.

Critical strengths
  1. This paper highlights the importance of incorporating additional motion features for online 4D human-scene reconstruction, which I find reasonable and important for this task.
  2. The qualitative and quantitative results are strong with a very comprehensive ablation study.
  3. The supplementary videos show some qualitative results of challenging in-the-wild videos, which help demonstrates the generalization ability of the method.
Critical weaknesses
  1. Lack of novelty. The whole pipeline is an incremental improvement upon the baseline Human3R with only an additional velocity field prediction module to provide motion features, which also results in a marginal performance improvement compared Human3R.
  2. Although the velocity field prediction helps human motion reconstruction, there is no experiments showing how it will affect the scene reconstruction and the consistency between the human and the scene. The accuracy of the velocity prediction is also not evaluated.
  3. The formulation of learning this velocity field is not sound. In Tab. 4, compared to direct motion regression, its improvement is not very significant. During inference, the joint displacement is integrated from this velocity field using a simple second-order numerical approximation (Heun’s method) and could introduce errors. Also, why the input joint positions X is varying for the velocity prediction model instead of always using the initial joint positions X_0?
  4. The current velocity prediction is only dependent on the last frame's joint position, while a larger history context could provide more temporal cues and help a more smooth prediction.
Questions for authors
  1. Why the input joint positions X is varying for the velocity prediction model instead of always using the initial joint positions X_0? Will the results be different between fϕ(XE,zt,Δt,Δt)f_{\phi}(X_E, z_t, \Delta t, \Delta t), fϕ(X0,zt,Δt,Δt)f_{\phi}(X_0, z_t, \Delta t, \Delta t), and fϕ(Xtfield,zt,Δt,Δt)f_{\phi}(X_t^{field}, z_t, \Delta t, \Delta t)?
  2. Can the authors provide more evaluation results for scene reconstruction and human-scene consistency and also compare them to the baseline Human3R?
  3. How will the velocity integration methods (e.g. 1st order Euler / 4th-Order Runge-Kutta) affect the prediction performance?
  4. Can the authors validate using only the last frame's joint position is sufficient for velocity prediction?
Reviewer_pvRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Velo3R, an online framework for unified 4D human scene reconstruction from monocular video. Building upon Human3R, the authors introduce a conditional velocity field to explicitly model inter-frame human motion. The method estimates joint positions by integrating predicted velocities and obtains accelerations through differentiation of the field.

Critical strengths
  1. This paper addresses an important challenge in online human scene reconstruction, where temporal consistency and global motion estimation remain difficult.
  2. The framework is well-structured, integrating motion prediction with human mesh reconstruction into a unified architecture , and the visual comparison is quite prominent.
Critical weaknesses
  1. The novelty of the method is limited. Most of the structures in the method are frozen. The neural velocity field is more like a combination of these techniques in Human3R. In addition, the experimental results in Table1 and Table2 show that the gap in Human3R is not obvious.
  2. The experimental results in Table4 show that switching from direct regression to Human3R to the velocity field only brings about a 3.3mm improvement in W-MPJPE.
  3. Velocity supervision is not obvious. Add Acc. sup in Table5. All the subsequent errors have increased.
Questions for authors
  1. Why does the local reconstruction accuracy on EMDB-1 decrease?
  2. Can you explain in the Table A1 show Motion Feature Feedback effect most?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.