- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an offline optimization framework called PixelBody for recovering pixel-aligned human meshes from monocular videos. Starting from the predictions of SAM 3D Body, the method first adapts a pre-trained motion VAE decoder to reconstruct an initial pose trajectory. It then jointly optimizes the decoder and pose latent variables in two stages: keypoint-based registration, followed by dense refinement using silhouette, surface normals, and relative depth. The framework integrates newly fine-tuned Sapiens2 and Wan2.1 predictors and optimizes human parameters across video sequences.
The authors report competitive camera-space reconstruction performance on the EMDB dataset and achieve superior world-space accuracy when real camera trajectories are provided. Experiments also investigate issues such as decoder adaptation, dense fitting constraints, and temporal smoothness.
Critical strengths
-
Meaningful research on the adaptive motion decoder during test time. This paper explores an interesting approach that differs from the traditional use of fixed motion priors for optimization. Specifically, adaptive adjustment of the decoder is significantly superior to merely optimizing the latent space, and a comparison experiment with direct pose optimization indicates an improvement in the PA-MPJPE metric. The corresponding subject clustering bootstrap confidence interval further enhances the credibility of this specific result.
-
A well designed optimization framework, accompanied by practical experimental analysis. The two-stage fitting process combines complementary geometric observations with temporal consistency constraints. Supplementary materials provide detailed implementation details, predictor comparisons, controlled ablation experiments, and negative results. The authors also explain several important evaluation limitations.
Critical weaknesses
-
The necessity of the full joint-optimization formulation and its cross-dataset robustness are not sufficiently established. Table 3(c) shows that adapting only the decoder slightly outperforms jointly adapting the decoder and latents. Therefore, although the evidence supports decoder adaptation over frozen-latent optimization, it does not establish that joint adaptation is necessary. Similarly, Table 3(d) indicates that most of the dense-refinement improvement comes from surface normals. Adding depth and silhouettes provides limited additional geometry improvement. This raises the question of whether the full multimodal pipeline offers a compelling accuracy-complexity trade-off over a simpler formulation.
-
The results on 3DPW are also concerning: the reported PixelBody configurations perform worse than the corresponding strong SAM 3D Body initializer setting, although the compared preprocessing pipelines are not fully identical.The explaination for the failure on this dataset is required.
Questions for authors
-
As I said in the weakness-1, the joint-optimization doesn't work well all the time. These result show that some components looks redundant. Why we need the joint-optimization and how to explian the trivial improvement for the joint-optimization?
-
How to explain the disadvantage on the 3DPW dataset(Table 6 in Appendix)? Do the preset hyperparameters or the pretrained model fail to generalize to unseen dataset?
-
What are the specific differences between the optimization process of this method and the preprocessing processes of Neuman( arXiv:2203.12575 ), HUGS(CVPR,2024,arXiv:2311.17910), and SkiningGS(arXiv:2506.21632)? It looks like these three methods utilize the a similar data preprocess to extract the SMPL from the video.
-
Since this paper claims it as a HMR system(Line 80), the robustness and timing are important. You only provide the timing of your method. Could you please also provide the runtime of other optimization-based methods?