- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HumanWM reconstructs full-body SMPL-X human motion from egocentric video by adapting a pretrained video world model (Cosmos 3 Edge) into a joint video-and-action model. The key move is to extend the world model's token domain with a 144-dimensional human-action channel, so that video and body action are denoised in the same sequence model rather than action being regressed from frozen video features. Two mechanisms are introduced. Coupled Kinematic Supervision (CKS) adds, on top of the flow objective, a path term and a dynamics term on the horizontal root trajectory, plus geometric terms. Second, an inference-time energy is minimized. HumanWM is validated on 3,000 sequences from 57 Nymeria test recordings, and beats all evaluated baselines across pose, root trajectory, dynamics, and foot-skating metrics.
Critical strengths
(a) The formulation is an interesting reframing, and the margins are large. Treating human action as an additional modality inside a pretrained video world model — rather than as a regression head attached to video features — is a clean idea, and the gap over ReViV (78.10 vs 137.11 GA-MPJPE; 46.49 vs 91.88 PA-MPJPE) is big enough that it is unlikely to be an artifact of tuning.
(b) The engineering is specified in enough detail to be reproducible. Loss terms, weights, optimizer, sampler settings, window/stride, blending weights and the stitching procedure (circular-mean yaw, median translation, geodesic rotation mean) are all written down.
Critical weaknesses
(a) The entire quantitative evaluation is a single dataset. Everything is Nymeria. The split is person-disjoint, which rules out the crudest form of leakage, but it does not address scene, capture-rig, activity-distribution or sensor-specific overfitting — and it leaves the central claim of the paper untested. The paper's thesis is that “world-model pretraining” supplies priors about human motion in scenes; the one thing such a prior should buy is transfer to capture conditions the fine-tuning data does not contain. With no result on EgoBody, EgoExo4D, or any other egocentric corpus, the 78.10 vs 137.11 gap is consistent with a well-fit Nymeria model and tells me nothing about whether the world-model story is the operative explanation.
(b) The two headline mechanisms are not isolated, and the one ablation that is run undercuts the framing. There is no ablation that replaces Cosmos 3 Edge with a comparably sized non-world-model video backbone (or a randomly initialized one), so "adapting a pretrained world model" — the title claim — is never tested against the obvious alternative.
Questions for authors
(1) Can you report cross-dataset results — train on Nymeria, evaluate zero-shot on at least one other egocentric body-motion benchmark?
(2) Please ablate the backbone: Cosmos 3 Edge (pretrained) vs the same architecture randomly initialized vs a non-world-model video backbone of similar capacity, with the rest of the pipeline fixed.
(3) What is the runtime of the full-sequence L-BFGS, and how does the system behave in a streaming/online setting without future frames?
Response to Reviewer G8GL: Q1–Q3
Q1
Thank you for this suggestion. We have added zero-shot motion reconstruction from Nymeria to Ego-Exo4D. The available validation subset, with both video and motion annotations, comprises 1,419 clips from 387 takes. We directly reconstruct motion from Ego-Exo4D egocentric RGB, without target-domain fine-tuning, text, or ground-truth head/camera trajectories. HumanWM achieves 93.27 mm GA-MPJPE, 56.39 mm PA-MPJPE, and 96.62 mm pelvis-aligned MPJPE. Mean and final root position errors are 0.1262 m and 0.2023 m, respectively.
We also report ReViV, EgoAllo, and EgoEgo Stage2 with their input conditions:
Q2
Thank you for this suggestion. We compare pretrained Cosmos 3 Edge, the same architecture with random initialization, DINOv3 with a temporal decoder, and the InternVideo2-Stage2_6B video backbone. The two Cosmos variants share the same architecture and differ only in initialization. The DINOv3 baseline uses a frozen ViT-H+/16 image backbone (approximately 0.84B parameters) and a six-layer temporal action decoder, while InternVideo2 provides a larger pretrained video-encoder comparison. All four configurations use the same Nymeria training split and training budget, action representation, motion supervision, and postprocessing, and are evaluated with RGB-only inputs on the same 3,000 test sequences.
With architecture held fixed, the pretrained model outperforms random initialization on all five metrics, supporting a contribution from pretraining beyond network size alone. InternVideo2 improves GA-MPJPE and both root-trajectory metrics over DINOv3. HumanWM achieves lower errors than both encoder baselines across all five reported metrics.
Q3
The table reports HumanWM's core model prediction and L-BFGS refinement times, measured over three runs per video after model loading and warm-up and summarized by the median. Core model prediction uses 30-step UniPC sampling on an NVIDIA A100, summed over all windows, excluding separate visual encoding, motion decoding, and file I/O. Refinement runs once on the complete predicted sequence using the CPU.
Refinement takes 0.30–0.36 s on these three videos. The current method performs offline reconstruction: motion estimates use later observations within each video window, and full-sequence refinement waits for the complete predictions. It therefore cannot be used unchanged for framewise online reconstruction without future frames.