acceptodds
Under review as a conference paper at ICLR 2027

HumanWM: A World Action Model for Egocentric Human Motion Reconstruction

Abstract

Reconstructing human motion from egocentric video requires inferring a largely unseen body from viewpoint changes and interactions with the environment. We present HumanWM, a World Action Model for Egocentric Motion that adapts a pretrained world model to jointly model egocentric video and structured human actions. The model introduces Coupled Kinematic Supervision that jointly constrains spatial structure and acceleration for egocentric motion reconstruction. At inference, full-sequence optimization balances fidelity to predicted motion with rotational and root-dynamics regularization. On 3,000 sequences from 57 Nymeria test recordings, HumanWM outperforms all evaluated baselines across pose, root-trajectory, motion-dynamics, and foot-skating metrics using RGB alone. Compared with RGB-only ReViV, the RGB-only configuration reduces globally aligned and Procrustes-aligned mean per-joint position errors by 38.2% and 44.9%, respectively. Task text further improves pose and root-trajectory accuracy, yielding 78.10 mm GA-MPJPE, 46.49 mm PA-MPJPE, and 0.216 m final root trajectory error.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Motion, World Action Models, Human Motion Reconstruction, Multimodal World Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G8GLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HumanWM reconstructs full-body SMPL-X human motion from egocentric video by adapting a pretrained video world model (Cosmos 3 Edge) into a joint video-and-action model. The key move is to extend the world model's token domain with a 144-dimensional human-action channel, so that video and body action are denoised in the same sequence model rather than action being regressed from frozen video features. Two mechanisms are introduced. Coupled Kinematic Supervision (CKS) adds, on top of the flow objective, a path term and a dynamics term on the horizontal root trajectory, plus geometric terms. Second, an inference-time energy is minimized. HumanWM is validated on 3,000 sequences from 57 Nymeria test recordings, and beats all evaluated baselines across pose, root trajectory, dynamics, and foot-skating metrics.

Critical strengths

(a) The formulation is an interesting reframing, and the margins are large. Treating human action as an additional modality inside a pretrained video world model — rather than as a regression head attached to video features — is a clean idea, and the gap over ReViV (78.10 vs 137.11 GA-MPJPE; 46.49 vs 91.88 PA-MPJPE) is big enough that it is unlikely to be an artifact of tuning.

(b) The engineering is specified in enough detail to be reproducible. Loss terms, weights, optimizer, sampler settings, window/stride, blending weights and the stitching procedure (circular-mean yaw, median translation, geodesic rotation mean) are all written down.

Critical weaknesses

(a) The entire quantitative evaluation is a single dataset. Everything is Nymeria. The split is person-disjoint, which rules out the crudest form of leakage, but it does not address scene, capture-rig, activity-distribution or sensor-specific overfitting — and it leaves the central claim of the paper untested. The paper's thesis is that “world-model pretraining” supplies priors about human motion in scenes; the one thing such a prior should buy is transfer to capture conditions the fine-tuning data does not contain. With no result on EgoBody, EgoExo4D, or any other egocentric corpus, the 78.10 vs 137.11 gap is consistent with a well-fit Nymeria model and tells me nothing about whether the world-model story is the operative explanation.

(b) The two headline mechanisms are not isolated, and the one ablation that is run undercuts the framing. There is no ablation that replaces Cosmos 3 Edge with a comparably sized non-world-model video backbone (or a randomly initialized one), so "adapting a pretrained world model" — the title claim — is never tested against the obvious alternative.

Questions for authors

(1) Can you report cross-dataset results — train on Nymeria, evaluate zero-shot on at least one other egocentric body-motion benchmark?

(2) Please ablate the backbone: Cosmos 3 Edge (pretrained) vs the same architecture randomly initialized vs a non-world-model video backbone of similar capacity, with the rest of the pipeline fixed.

(3) What is the runtime of the full-sequence L-BFGS, and how does the system behave in a streaming/online setting without future frames?

1 comment
AuthorsOpenReview ↗

Response to Reviewer G8GL: Q1–Q3

Q1

Thank you for this suggestion. We have added zero-shot motion reconstruction from Nymeria to Ego-Exo4D. The available validation subset, with both video and motion annotations, comprises 1,419 clips from 387 takes. We directly reconstruct motion from Ego-Exo4D egocentric RGB, without target-domain fine-tuning, text, or ground-truth head/camera trajectories. HumanWM achieves 93.27 mm GA-MPJPE, 56.39 mm PA-MPJPE, and 96.62 mm pelvis-aligned MPJPE. Mean and final root position errors are 0.1262 m and 0.2023 m, respectively.

We also report ReViV, EgoAllo, and EgoEgo Stage2 with their input conditions:

MethodAuxiliary inputsGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
EgoEgo Stage2GT Cam. Traj.141.79121.270.21100.3654
EgoAlloGT Cam. Traj.128.11105.300.18980.2968
ReViVNone162.28105.960.18770.3138
HumanWMNone93.2756.390.12620.2023

Q2

Thank you for this suggestion. We compare pretrained Cosmos 3 Edge, the same architecture with random initialization, DINOv3 with a temporal decoder, and the InternVideo2-Stage2_6B video backbone. The two Cosmos variants share the same architecture and differ only in initialization. The DINOv3 baseline uses a frozen ViT-H+/16 image backbone (approximately 0.84B parameters) and a six-layer temporal action decoder, while InternVideo2 provides a larger pretrained video-encoder comparison. All four configurations use the same Nymeria training split and training budget, action representation, motion supervision, and postprocessing, and are evaluated with RGB-only inputs on the same 3,000 test sequences.

MethodGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Pelvis-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
Frozen DINOv3 + temporal decoder357.6981.56220.420.5981.228
InternVideo2-Stage2_6B232.54106.40233.410.4280.820
Cosmos 3 Edge: random initialization419.8480.53180.780.7091.283
Cosmos 3 Edge: pretrained84.7850.6491.280.1170.232

With architecture held fixed, the pretrained model outperforms random initialization on all five metrics, supporting a contribution from pretraining beyond network size alone. InternVideo2 improves GA-MPJPE and both root-trajectory metrics over DINOv3. HumanWM achieves lower errors than both encoder baselines across all five reported metrics.

Q3

The table reports HumanWM's core model prediction and L-BFGS refinement times, measured over three runs per video after model loading and warm-up and summarized by the median. Core model prediction uses 30-step UniPC sampling on an NVIDIA A100, summed over all windows, excluding separate visual encoding, motion decoding, and file I/O. Refinement runs once on the complete predicted sequence using the CPU.

Video duration (s)HumanWM: core model prediction, 30-step UniPC (s)HumanWM: L-BFGS refinement (s)
3.004.580.30
5.009.150.33
6.2711.420.36

Refinement takes 0.30–0.36 s on these three videos. The current method performs offline reconstruction: motion estimates use later observations within each video window, and full-sequence refinement waits for the complete predictions. It therefore cannot be used unchanged for framewise online reconstruction without future frames.

Reviewer_52WpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HUMANWM, a World Action Model for reconstructing 3D human motion and generating future video from monocular egocentric RGB inputs. Built upon the pretrained Cosmos 3 Edge world model, it maps human motion—including camera transitions, root states in a hybrid grounded gravity-view representation, and 21 SMPL-X body joint rotations—into a 144-dimensional native action interface.

To mitigate trajectory drift and erratic dynamics caused by continuous egocentric viewpoint changes, the paper introduces Coupled Kinematic Supervision (CKS), which jointly constrains low-pass-filtered path structure and second-order local acceleration. The method also uses auxiliary geometric losses and test-time L-BFGS full-sequence inference optimization. Evaluated on a person-disjoint split of the Nymeria dataset with 3,000 test sequences, HUMANWM reports substantial quantitative improvements over existing baselines in both pose/trajectory reconstruction and video generation metrics.

The approach is promising, particularly in its integration of a pretrained world model with structured human kinematics. However, the evaluation leaves important questions about whether the model recovers motion from visual evidence or relies on dataset-specific priors, whether the results generalize beyond Nymeria, and whether baseline comparisons are fair. The reported single-task versus multi-task ablation also confounds joint video-action training with differences in text conditioning. Further clarification is needed regarding the episode-level support-plane assumption and the inference cost of combining UniPC sampling with L-BFGS optimization.

Critical strengths
  1. Novel Adaptation of Pretrained World Models: Adapting a pretrained video world model’s native action interface to jointly learn egocentric visual dynamics and structured human kinematics is well-motivated and effectively bridges visual world modeling with egocentric motion estimation.

  2. Principled Kinematic Representation and Supervision: The hybrid grounded gravity-view representation, which decouples canonical ground-frame root positions from time-varying gravity-view root rotations, combined with CKS, effectively addresses the trade-off between global trajectory drift and local high-frequency jitter.

  3. Strong Empirical Performance on Nymeria: The method achieves impressive numerical gains on the evaluated Nymeria benchmark without requiring external SLAM head trajectories at inference time, outperforming both SLAM-conditioned and RGB-only baselines.

Critical weaknesses
  1. Generalization and the Reliability of Reconstructed Motion Remain Insufficiently Established. All training and evaluation are conducted on a custom split of Nymeria. In qualitative visualizations, some input videos appear to show little or no directly observable arm movement, while the predicted arm configurations nevertheless look plausible and apparently accurate. This raises the question of whether the model actually recovers the underlying motion or relies on dataset-specific priors. Cross-dataset evaluation and more challenging cases with unobservable or atypical arm motions are needed to establish generalization.

  2. Baseline Comparisons and Ablations Do Not Fully Isolate the Sources of Improvement. Baselines such as EgoEgo, EgoAllo, and UniEgoMotion perform substantially worse despite being provided with ground-truth camera trajectories, but the paper does not clearly establish whether they were retrained fairly or evaluated using checkpoints and conventions from other datasets. The single-task versus multi-task ablation also changes the text-conditioning setup, confounding the effect of multi-task video-action training. In addition, the inference-time support-plane assumption and the cost of UniPC sampling plus L-BFGS optimization need clarification.

Questions for authors
  1. Arm-Motion Reconstruction from Weak or Missing Visual Evidence and Potential Dataset Overfitting. In the qualitative visualizations, some input videos appear to contain little or no directly observable arm movement, while the reconstructed body poses nevertheless show plausible and apparently accurate arm configurations. Recovering unobserved body motion is one of the key challenges of egocentric reconstruction, but plausible-looking predictions do not necessarily establish that the actual arm motion has been recovered from the input. When the arms are not visible, the model may rely heavily on correlations learned from the training distribution, such as typical arm configurations associated with particular activities or whole-body movements. Such priors can be useful but may also produce convincing yet incorrect motions for atypical or out-of-distribution actions. Could the authors include challenging examples where the arms are completely or almost completely unobservable, with quantitative comparisons against ground truth? Could they also test whether inferred arm motion remains accurate for atypical movements or under misleading task descriptions? These experiments would help distinguish recovery from indirect visual cues from plausible motion completion driven primarily by learned priors.

  2. Lack of Cross-Dataset Generalization. All training and evaluation are conducted exclusively on a custom split of the Nymeria dataset. Given the high capacity of the Cosmos 3 backbone and the overfitting concerns above, a single-dataset evaluation is insufficient to establish generalizable egocentric motion understanding. Can the authors provide zero-shot cross-dataset evaluations on other standard egocentric benchmarks, such as EgoBody, Ego-Exo4D, or ADT, to demonstrate out-of-distribution generalization?

  3. Insufficient Clarity on Baseline Implementations and Performance Gaps. In Tables 1 and 2, baselines such as EgoEgo, EgoAllo, and UniEgoMotion perform substantially worse despite being provided with ground-truth camera trajectories. For example, UniEgoMotion yields 319.50 mm GA-MPJPE and 0.543 m mean root error, compared with HUMANWM’s 84.78 mm and 0.117 m. The paper does not clearly explain how these baselines were adapted. Were they retrained from scratch on exactly the same 500 Nymeria training recordings, or evaluated zero-shot using checkpoints trained on different datasets or coordinate conventions? Please clarify the specific architectural formulations and evaluation protocols used for these baselines, explain why they perform so poorly even with ground-truth camera trajectories, and provide fair cross-dataset comparisons on the benchmarks where the baselines were originally developed.

  4. Confounded Ablation on Multi-Task Training and Text Conditioning. In Section 4.4 (Table 4) and Appendix A (lines 676–678), the single-task (Recon.) model uses a fixed training instruction, whereas the multi-task model uses episode-specific task descriptions. The reported improvement—GA-MPJPE decreasing from 114.42 mm to 103.79 mm—therefore confounds the benefit of joint video generation/forecasting with the benefit of language supervision during training. Could the authors isolate the contribution of joint video-action training by comparing Recon. and Multi-task under identical text-conditioning settings?

  5. Oracle Episode-Level Ground-Plane Support (gsupg_{\text{sup}}). In Section 3.2 (line 199), the vertical root coordinate is encoded as zt−gsupz_t-g_{\text{sup}}, where gsupg_{\text{sup}} is an episode-level support reference estimated from contacted toe and heel segments. How is gsupg_{\text{sup}} obtained during inference? How does the representation handle multi-elevation locomotion, such as walking up or down stairs or on slopes, where a single episode-level ground-height assumption may break down?

  6. Inference Latency and Post-Processing Overhead. The full-sequence inference optimization runs L-BFGS for up to 80 iterations over fused 33-frame windows and accounts for a 67.6% reduction in root acceleration error (Table 4). What is the wall-clock runtime breakdown, in FPS or seconds per sequence, between the 30-step UniPC diffusion sampling and the L-BFGS optimization? How does the total inference cost compare with feed-forward baselines such as ReViV or UniEgoMotion?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 52Wp: Q1–Q3

Q1

HumanWM jointly models egocentric video and wearer motion so that visible body features, viewpoint changes, and temporal context can inform the reconstruction when an arm is out of view. However, these indirect cues may be consistent with several arm motions, and a task description may favor a plausible movement that differs from the wearer's actual motion.

To evaluate reconstruction fidelity, we compare ReViV and HumanWM on the same 24 videos and fixed invisible-arm intervals, measuring pelvis-relative elbow and wrist errors against the recorded motion. Both ReViV and HumanWM are trained on Nymeria. ReViV uses RGB-only inputs; for HumanWM, we additionally compare no text, original text, and completely incorrect action descriptions while holding the video inputs and evaluation intervals fixed.

MethodText conditionElbow error (mm) ↓Wrist error (mm) ↓
ReViVNo text208.64251.22
HumanWMNo text117.86178.06
HumanWMOriginal text108.58163.40
HumanWMCompletely incorrect action description127.83214.56

Under matched RGB-only inputs, HumanWM reduces elbow and wrist errors by 43.5% and 29.1%, respectively, relative to ReViV. Both paired 95% confidence intervals for the error differences lie below zero. Thus, the improvement on these invisible-arm intervals is already present without text conditioning.

Original text yields lower elbow and wrist errors than no text. Completely incorrect action descriptions increase these errors by 17.7% and 31.3%, respectively, relative to original text, with both paired confidence intervals excluding zero. These results show that textual semantics influence invisible-arm reconstruction and that incorrect descriptions can pull predictions away from the actual motion; the model is not robust to misleading text in this evaluation.

Q2

Thank you for this suggestion. We have added zero-shot motion reconstruction from Nymeria to Ego-Exo4D. The available validation subset, with both video and motion annotations, comprises 1,419 clips from 387 takes. We directly reconstruct motion from Ego-Exo4D egocentric RGB, without target-domain fine-tuning, text, or ground-truth head/camera trajectories. HumanWM achieves 93.27 mm GA-MPJPE, 56.39 mm PA-MPJPE, and 96.62 mm pelvis-aligned MPJPE. Mean and final root position errors are 0.1262 m and 0.2023 m, respectively.

We also report ReViV, EgoAllo, and EgoEgo Stage2 with their input conditions:

MethodAuxiliary inputsGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
EgoEgo Stage2GT Cam. Traj.141.79121.270.21100.3654
EgoAlloGT Cam. Traj.128.11105.300.18980.2968
ReViVNone162.28105.960.18770.3138
HumanWMNone93.2756.390.12620.2023

Q3

Thank you for requesting clarification of the baseline implementations. We fine-tuned UniEgoMotion on the same 500 Nymeria training recordings used for HumanWM. When revisiting the original experiments, we identified an error in the format of the camera trajectories used during UniEgoMotion fine-tuning. We corrected the input format and repeated fine-tuning on the same recordings. ReViV and EgoPriMo are also trained on Nymeria and evaluated on the same test split as HumanWM. EgoEgo and EgoAllo use official checkpoints without fine-tuning on our training split.

We approximate the head-mounted camera trajectory from ground-truth body poses and provide it as the trajectory input to EgoEgo, EgoAllo, and UniEgoMotion. All methods are evaluated on the same 3,000 Nymeria test sequences using the common 21 joints at 15 Hz. GA-MPJPE uses sequence-level alignment, PA-MPJPE uses framewise alignment, and root errors align only the initial translation. The revised results are:

MethodAuxiliary inputsGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Pelvis-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
UniEgoMotion (revised)GT Cam. Traj.155.2392.66130.700.4990.893
ReViVNone137.1191.88207.950.1800.322
HumanWMNone84.7850.6491.280.1170.232

Ground-truth head trajectories constrain motion but do not directly determine the remaining body joints. Correcting and repeating UniEgoMotion fine-tuning substantially reduces its errors, while HumanWM retains higher RGB-only reconstruction accuracy.

AuthorsOpenReview ↗

Response to Reviewer 52Wp: Q4–Q6

Q4

Thank you for pointing this out. We compared reconstruction-only and multi-task training with the same text conditioning, pretrained backbone initialization, and training budget (number of updates and global batch size), and evaluated both using the same reconstruction inputs and inference settings. Table 4 shows that adding the generation tasks reduces GA-MPJPE from 114.42 to 103.79 mm, a reduction of 10.63 mm (9.3%). This improvement therefore does not arise from different text conditioning between the two training configurations.

The statement in Appendix A that the two configurations use different text conditioning was an inadvertent wording error. We will correct it and present this comparison more clearly in the revision. In fact, both settings use the same text prompt for each video clip during training. Reconstruction-only training predicts motion from the complete RGB video. Multi-task training retains this task and additionally masks part of the RGB observations, retaining only the initial frame or a video prefix. Together with the same text prompt, these observations condition the joint prediction of the complete motion sequence and the unobserved subsequent video frames. This comparison measures the effect of adding the generation tasks as a group; it does not isolate the contribution of the video-generation loss alone.

Q5

HumanWM does not require ground-truth foot contacts or a pre-estimated support height at inference. In §3.2, gsupg_{\mathrm{sup}} is used to construct training labels: contacted feet in the annotations define an episode-level height reference, and the vertical root coordinate is encoded as ht=zt−gsuph_t=z_t-g_{\mathrm{sup}}. At inference, the model predicts hth_t directly from video, and the decoder reads this relative height without accessing test-time contact labels or adding back gsupg_{\mathrm{sup}}. The offset is needed only to recover absolute height in the original motion-capture world coordinates. For the root-trajectory errors in Q3, which align only the initial translation, a constant height offset cancels out, so scoring also does not require recovering absolute height in the original motion-capture coordinates. We will clarify its role in training-label normalization in §3.2.

The fixed quantity is the coordinate origin, not the ground height beneath the feet at every frame. Subtracting the same constant preserves vertical displacement: ht+1−ht=zt+1−zth_{t+1}-h_t=z_{t+1}-z_t. Ascending or descending stairs and slopes can therefore be represented without requiring motion on a single plane. This establishes that the representation permits changes in elevation, rather than demonstrating prediction accuracy in every such setting.

Q6

We measured HumanWM and ReViV on GPUs of the same NVIDIA A100 model with RGB-only inputs, reporting median runtimes over three runs per video after model loading and warm-up. Core model prediction times sum over all windows and exclude separate visual encoding, motion decoding, and file I/O. HumanWM uses 30-step UniPC sampling; full-sequence L-BFGS refinement runs once on the CPU and is reported separately.

Video duration (s)HumanWM: core model prediction, 30-step UniPC (s)HumanWM: L-BFGS refinement (s)ReViV: core model prediction (s)
3.004.580.302.50
5.009.150.335.07
6.2711.420.367.59

L-BFGS refinement takes 0.30–0.36 s on these videos.

Thank you for your response. It addresses part of my concerns, and I will take these additional experimental results into account in my final assessment.

Reviewer_G5W8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The manuscript proposes HumanWM for egocentric human motion reconstruction from monocular RGB videos. The method adapts the pretrained Cosmos-3 world model to jointly predict camera motion, root trajectory, and body poses and also additionally supports video prediction. The manuscript addresses an important problem of egocentric human motion reconstruction using RGB videos alone, without requiring externally provided camera trajectories.. The authors propose Coupled Kinematic Supervision (CKS), which combines a hybrid gravity-view representation with trajectory and acceleration losses. The model is trained on Nymeria and evaluated on 3,000 sequences from 57 test recordings. It achieves 84.78 mm GA-MPJPE using RGB alone, compared to 137.11 mm for ReViV.

Critical strengths

S1. The paper addresses an important problem of egocentric human motion reconstruction using RGB videos alone, without needing GT camera trajectories.

Critical weaknesses

W1. Motivation of using world model is not clear: The paper motivates using a pretrained world model to capture visual dynamics and interactions with the surrounding environment. However, the proposed method predicts camera motion, and body poses without explicitly modeling objects or human-object interactions. The manuscript does not explain why a strong pretrained image/video encoder would not be sufficient for this task. Moreover, the paper jointly trains video generation and motion reconstruction, but the benefit of this joint training is not shown. Tab. 4 shows that multi-task training reduces GA-MPJPE, but the comparison also changes text conditioning (Appendix A). Therefore, it is unclear whether the improvement comes from video-generation supervision, text conditioning, or the pretrained backbone.

W2. Concerns with evaluation and baselines: The manuscript reports huge improvements over ReViV using RGB alone (Tab. 1). However, HumanWM is trained on 500 Nymeria recordings, while it is not clear whether ReViV and the other baselines are fine-tuned on the same training split. Differences in training data, and training strategy could contribute to the reported improvement.

Questions for authors

Q1. Benefit of joint video-motion modeling. Does jointly training video generation and motion reconstruction improve reconstruction accuracy (see W1) ? Can the authors compare reconstruction-only and multi-task training while keeping the conditioning, backbone and training budget same?

Q2. Role of scene information: The manuscript motivates world models through visual dynamics and interactions with the surrounding environment. Can the authors show that the proposed method uses scene information to improve reconstruction of the unseen body? It would be informative to see a comparison with reduced or masked scene information (see W1). Additionally, the manuscript does not establish whether a strong pretrained image/video encoder under similar conditions would be sufficient for this task.

Q3. Baseline training conditions: Were ReViV and the other baselines fine-tuned on the same Nymeria training split (see W2)?

Q4. Contribution of CKS: Tab. 4 shows that adding the coupled trajectory losses reduces root acceleration error only from 1.030 to 1.001, whereas inference optimization reduces it from 3.088 to 1.001. Can the authors ablate both the path and acceleration losses to highlight their contributions? Is the improvement primarily due to inference optimization?

4 comments
AuthorsOpenReview ↗

Response to Reviewer G5W8: Q1 and Q2

Q1

Thank you for pointing this out. With the same reconstruction conditioning, pretrained backbone initialization, and training budget (number of updates and global batch size), multi-task training reduces GA-MPJPE from 114.42 to 103.79 mm compared with reconstruction-only training, a reduction of 10.63 mm (9.3%) (Table 4). We did not sufficiently highlight this comparison or explain its implications in the original manuscript. We will make the result more prominent and expand the accompanying analysis in the revision.

Due to an inadvertent wording error in Appendix A, single-task and multi-task training were described as using different text conditioning. We will correct this in the revision. In fact, both settings use the same text prompt for each video clip during training. The reconstruction-only setting conditions on the complete RGB video to reconstruct the corresponding human motion. The multi-task setting retains this reconstruction task and additionally constructs generation tasks by masking part of the RGB observations. Only the initial frame or a video prefix is retained as visual conditioning; together with the same text prompt, these observations condition the joint prediction of the complete motion sequence and the unobserved subsequent video frames. Both configurations are evaluated using the same reconstruction inputs and inference settings. This comparison measures the effect of adding the generation tasks as a group; it does not isolate the contribution of the video-generation loss alone.

Q2

Thank you for this suggestion. Changes in viewpoint and visual dynamics in the scene may provide indirect cues about body motion outside the field of view. We choose a world model to use its pretrained ability to predict future observations, jointly modeling visual dynamics and wearer motion to constrain reconstruction from incomplete body observations.

Following the reviewer's suggestion, we randomly mask 12.5%, 25%, or 50% of the pixels outside the wearer mask in 228 video clips from 57 recordings, leaving pixels inside the mask unchanged. We compare reconstruction before and after masking using the same model without text conditioning.

Scene masking ratioGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓
0%86.2551.55
12.5%103.5654.89
25%115.2660.34
50%146.8174.20

Reconstruction errors increase with the masking ratio. At 50% masking, GA-MPJPE and PA-MPJPE increase by 70.2% and 43.9%, respectively. For all three masking ratios, the paired bootstrap 95% confidence intervals for the error increases over the original inputs lie above zero. Since visible body information is preserved, the increased errors show that scene cues outside the body contribute to HumanWM's motion reconstruction.

To assess whether a strong image encoder is sufficient, we add a DINOv3 baseline with a frozen ViT-H+/16 backbone (approximately 0.84B parameters) and a six-layer temporal action decoder predicting camera motion, root translation, and body poses. It uses HumanWM's action representation, motion supervision, and postprocessing, with the same Nymeria training split and training budget, but no text conditioning or video-generation supervision. Both methods are evaluated with RGB-only inputs on the same 3,000 test sequences.

MethodGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Pelvis-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
Frozen DINOv3 + temporal decoder357.6981.56220.420.5981.228
HumanWM, RGB-only84.7850.6491.280.1170.232

Under the same RGB-only test conditions, the evaluated DINOv3-plus-temporal-decoder baseline does not match HumanWM's reconstruction accuracy, particularly for sequence-level pose and root-trajectory recovery.

AuthorsOpenReview ↗

Response to Reviewer G5W8: Q3 and Q4

Q3

Thank you for asking about the baseline training settings. ReViV, UniEgoMotion, and EgoPriMo are trained on Nymeria. They are evaluated on the same 3,000 test sequences. We fine-tuned UniEgoMotion on the same 500 Nymeria training recordings used for HumanWM. When revisiting the experiments, we identified an error in the format of the camera trajectories used during UniEgoMotion fine-tuning. We corrected the format and repeated fine-tuning. For EgoEgo and EgoAllo, we evaluated their public checkpoints without fine-tuning on our training split.

The revised results for Table 1 are shown below:

Method / versionAuxiliary inputsGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Pelvis-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
UniEgoMotion (revised)GT Cam. Traj.155.2392.66130.700.4990.893
EgoPriMoText176.0075.38197.010.4260.810
ReViVNone137.1191.88207.950.1800.322
HumanWMNone84.7850.6491.280.1170.232
HumanWMText78.1046.4984.730.1100.216

We additionally evaluated HumanWM's zero-shot transfer from Nymeria to Ego-Exo4D, without fine-tuning any compared method on Ego-Exo4D. The evaluation uses the available validation subset with both video and motion annotations, comprising 1,419 clips from 387 takes:

MethodAuxiliary inputsGA-MPJPE (mm) ↓PA-MPJPE (mm) ↓Root mean (m) ↓Root final (m) ↓
EgoEgo Stage2GT Cam. Traj.141.79121.270.21100.3654
EgoAlloGT Cam. Traj.128.11105.300.18980.2968
ReViVNone162.28105.960.18770.3138
HumanWMNone93.2756.390.12620.2023

Using RGB alone, HumanWM achieves the lowest errors across all four reported metrics, providing quantitative evidence of its cross-dataset generalization.

Q4

Thank you for this suggestion. On the same 3,000 test sequences, we compare models trained with both trajectory losses jointly removed or retained, evaluating the same raw predictions before and after refinement for each model. With refinement disabled, retaining both losses reduces low-pass horizontal path error from 0.11176 to 0.09852 m (11.8%) and 3D root acceleration error from 3.322 to 3.088 m/s² (7.0%). Thus, training supervision already improves path accuracy and motion dynamics.

Both trajectory lossesInference refinementHorizontal path error (m) ↓3D root acceleration error (m/s²) ↓
RemovedOff0.111763.322
RemovedOn0.111751.030
RetainedOff0.098523.088
RetainedOn0.098511.001

Frame-to-frame jitter can amplify acceleration error. Inference refinement regularizes second- and third-order temporal changes in root position while limiting deviations from the original predictions, suppressing residual jitter and improving temporal smoothness. For the full model, refinement further reduces 3D root acceleration error to 1.001 m/s² (67.6%), while leaving horizontal path error nearly unchanged. The two trajectory losses primarily improve the accuracy of the reconstructed motion path, while inference refinement further suppresses residual jitter.

Thank you for the detailed rebuttal and additional experiments. The controlled multi-task comparison, scene-masking and backbone experiments, and cross-dataset evaluation address my original concerns.

I have no further questions and will take the additional results into account in my final assessment.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for reviewing our rebuttal and additional experiments. We are glad that the new results address your original concerns. Your constructive feedback has helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.