acceptodds
Withdrawn from ICLR 2027

PixelBody: Multimodal Priors for Monocular Human Mesh Recovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward human mesh recovery predicts 3D bodies from images and video in one shot but does not fit them to the input pixels, so the results can disagree with image cues such as landmarks, silhouettes, and surface normals. PixelBody is an offline method that starts from the body trajectory of a feed-forward method and iteratively fits it to such cues. During the fitting process, it utilizes a pose autoencoder pretrained on motion capture data. However, instead of applying it as a regularization, PixelBody adapts the decoder to each person: its weights are first fine-tuned to reproduce the initial trajectory from the encoded pose latents, and then the decoder and latents are fitted together to the video cues. Our ablations show that fine-tuning the decoder prevents the fit from being limited to the motion-capture prior, while still benefiting from it, and achieves lower 3D error than both direct pose optimization and a frozen decoder. Fitting runs in two stages: registration to skeleton and surface landmarks, then refinement that adds silhouettes, body normals, and relative depth. The landmarks, normals, and depth come from predictors we fine-tuned from Sapiens2 and Wan 2.1. PixelBody has the lowest camera-space error on EMDB1 among published methods that recover a body mesh, and, with ground-truth cameras, the lowest world-space error on EMDB2 among the compared methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human mesh recovery, keypoints estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VGozOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an offline optimization framework called PixelBody for recovering pixel-aligned human meshes from monocular videos. Starting from the predictions of SAM 3D Body, the method first adapts a pre-trained motion VAE decoder to reconstruct an initial pose trajectory. It then jointly optimizes the decoder and pose latent variables in two stages: keypoint-based registration, followed by dense refinement using silhouette, surface normals, and relative depth. The framework integrates newly fine-tuned Sapiens2 and Wan2.1 predictors and optimizes human parameters across video sequences.

The authors report competitive camera-space reconstruction performance on the EMDB dataset and achieve superior world-space accuracy when real camera trajectories are provided. Experiments also investigate issues such as decoder adaptation, dense fitting constraints, and temporal smoothness.

Critical strengths
  1. Meaningful research on the adaptive motion decoder during test time. This paper explores an interesting approach that differs from the traditional use of fixed motion priors for optimization. Specifically, adaptive adjustment of the decoder is significantly superior to merely optimizing the latent space, and a comparison experiment with direct pose optimization indicates an improvement in the PA-MPJPE metric. The corresponding subject clustering bootstrap confidence interval further enhances the credibility of this specific result.

  2. A well designed optimization framework, accompanied by practical experimental analysis. The two-stage fitting process combines complementary geometric observations with temporal consistency constraints. Supplementary materials provide detailed implementation details, predictor comparisons, controlled ablation experiments, and negative results. The authors also explain several important evaluation limitations.

Critical weaknesses
  1. The necessity of the full joint-optimization formulation and its cross-dataset robustness are not sufficiently established. Table 3(c) shows that adapting only the decoder slightly outperforms jointly adapting the decoder and latents. Therefore, although the evidence supports decoder adaptation over frozen-latent optimization, it does not establish that joint adaptation is necessary. Similarly, Table 3(d) indicates that most of the dense-refinement improvement comes from surface normals. Adding depth and silhouettes provides limited additional geometry improvement. This raises the question of whether the full multimodal pipeline offers a compelling accuracy-complexity trade-off over a simpler formulation.

  2. The results on 3DPW are also concerning: the reported PixelBody configurations perform worse than the corresponding strong SAM 3D Body initializer setting, although the compared preprocessing pipelines are not fully identical.The explaination for the failure on this dataset is required.

Questions for authors
  1. As I said in the weakness-1, the joint-optimization doesn't work well all the time. These result show that some components looks redundant. Why we need the joint-optimization and how to explian the trivial improvement for the joint-optimization?

  2. How to explain the disadvantage on the 3DPW dataset(Table 6 in Appendix)? Do the preset hyperparameters or the pretrained model fail to generalize to unseen dataset?

  3. What are the specific differences between the optimization process of this method and the preprocessing processes of Neuman( arXiv:2203.12575 ), HUGS(CVPR,2024,arXiv:2311.17910), and SkiningGS(arXiv:2506.21632)? It looks like these three methods utilize the a similar data preprocess to extract the SMPL from the video.

  4. Since this paper claims it as a HMR system(Line 80), the robustness and timing are important. You only provide the timing of your method. Could you please also provide the runtime of other optimization-based methods?

Reviewer_gthSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PixelBody is an offline method for human mesh recovery from monocular video. It starts from the output of a feed-forward model (SAM 3D Body) and refines it so the body better matches the image, using predicted landmarks, silhouettes, normals and depth.

The main idea is in how the motion prior is used. Instead of keeping a pretrained motion model fixed as a regularizer, the authors fine-tune its decoder for each person similar to exemlar fine tuning. The decoder is first adjusted to reproduce the initial trajectory, and then the decoder and its latent codes are optimized together against the image cues. Fitting is done in two stages, first to landmarks and then to the dense cues. The authors also train their own predictors for these cues, starting from Sapiens2 and Wan 2.1.

The decoder adaptation is close in spirit to exemplar fine-tuning and pivotal tuning, which the paper acknowledges, and to STRIDE, which also fine-tunes a motion prior per video. In my understanding the novelty with respect to those is mainly in fitting it to pixel-aligned cues.

The method is evaluated on EMDB, RICH and 3DPW. It reports the lowest camera-space error on EMDB among published mesh recovery methods, and the lowest world-space error on EMDB2 when ground-truth cameras are used. Results on RICH are mixed, and on 3DPW it is worse than its starting point. The ablations show that adapting the decoder works better than optimizing poses directly or keeping the decoder frozen.

Critical strengths

The fine tuning idea by updating only the decoder is an interesting idea and is shown to be better than direct pose optimization in ablations.

The paper provides clear visuals as supplementary material that showcases the quality of the fits, along with visualisations of side views.

The paper proposes a unified system and strong predictors as "expert" components for the optimisation. Sapiens2 body, reaches better results wrt MammaNet for instance. Along with Wan2.1 normals and depth component for the video.

The paper provides good ablation studies for many components and settings, makes it easy to justify for many proposed individual components.

Critical weaknesses

One issue that I find concerning is the use of GT boxes and geometry, the authors indicate this in the conclusion: "Its evaluation crops, boxes, and track prompts derive from ground truth. Table 7 and 8 Its limitations include reliance on the initializer and pre-computed person tracks.", but it is not very clear why the authors dont report results as they would be on the deployment? I understand that some methods do report like this and use GT boxes, but main comparisons such as Sam3D Body if the non-bbox version is used in headline tables it is concerning. Looking at Tab7 made me think this. Also it appears that when non-GT cameras are used the system is performing worse with respect to many of the more recent baselines, this is not presented in the main paper nor discussed.

The gains also do not carry over well beyond EMDB. On 3DPW, PixelBody is worse than its own starting point, SAM 3D Body, on all three metrics in every configuration, even on the same samples and ground truth, and the paper does not explain why. On RICH, all PixelBody versions have higher MPJPE and PVE than the published SAM 3D Body and Duomo. In world space, the advantage only holds with ground-truth cameras. Note that JOSH and HumanBA do not use ground-truth intrinsics either.

Together this makes the in-the-wild framing hard to support with respect to recent baselines.

Questions for authors

To me this is a borderline paper, on the one hand method is very clearly described and results are presented in many settings and contributions are well ablated, on the other hand gains are very mixed despite the heavily engineered system, and especially weakness increases for a true in-the-wild, deployment-level setting of the experiment.

With regards to the 3DPW, why was this results moved to appendix and do the authors have any comments on why there is performance regression?

The main Sam3D body results reported in the main paper, are GT bboxes used to obtain the results? Would the authors consider reporting results that do not rely on groundtruth tracks?

Table 3c shows that latents only optmisation makes the results worse than the starting position, this does not look right to me, would the authors want to comment on this?

Reviewer_XCheOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PixelBody, an offline optimization-based framework for recovering pixel-aligned 3D human body meshes from monocular videos. The proposed method initializes human body trajectories using SAM 3D Body and subsequently refines them using multiple image-based cues, including skeleton and surface landmarks, silhouettes, surface normals, and relative depth.

The main technical contribution is a test-time pose decoder adaptation strategy. Instead of directly optimizing body poses or restricting the optimization to the latent space of a frozen motion prior, the method first fine-tunes a pretrained motion-VAE decoder to reconstruct the initial trajectory and then jointly optimizes the decoder and its latent representations to fit the observed video. The optimization consists of two stages: landmark-based registration followed by dense refinement. The framework also incorporates two fine-tuned prediction models, Sapiens2 Body and Wan2.1 Body, to provide the necessary pixel-aligned cues.

Experiments on EMDB, RICH, and 3DPW demonstrate competitive performance, with particularly strong results on EMDB. The ablation studies further show that adapting the pose decoder improves reconstruction accuracy compared with direct pose optimization and optimization using a frozen decoder.

Critical strengths
  • The proposed framework effectively combines multiple complementary image cues with a pretrained motion model. Unlike conventional feed-forward HMR approaches, which may produce geometrically plausible but imperfectly aligned reconstructions, PixelBody explicitly optimizes body meshes to better match the observed video. The combination of sparse landmarks and dense geometric cues is well motivated and provides a practical approach to improving reconstruction accuracy.
  • The test-time adaptation of the pretrained pose decoder is an interesting aspect of the proposed method. Rather than constraining the reconstructed motion to the latent space of a fixed motion prior, the method adapts the decoder to the target sequence, potentially improving its flexibility while retaining useful initialization from motion-capture data. The ablation studies in Table 3(b) and (c) provide empirical evidence supporting the effectiveness of this design.
  • The proposed method achieves strong results on EMDB, outperforming the compared mesh-recovery methods in camera-space reconstruction accuracy. In particular, the full model achieves 35.2 mm PA-MPJPE, 59.0 mm MPJPE, and 70.5 mm PVE on EMDB1. It also achieves the best reported world-space reconstruction accuracy among the compared methods on EMDB2 when ground-truth camera information is available. The paper provides useful ablation studies investigating the contributions of the optimization stages, pose decoder adaptation, and different dense cues.
Critical weaknesses
  • Although the proposed pose decoder adaptation strategy is interesting, the overall framework primarily integrates several established techniques, including pretrained HMR initialization, test-time optimization, decoder fine-tuning, and fitting to pixel-aligned geometric cues.
  • The main novelty appears to lie in applying decoder adaptation to motion-VAE-based human mesh fitting rather than introducing a fundamentally new optimization formulation or learning mechanism. The paper would benefit from a clearer explanation of the methodological differences from closely related test-time adaptation approaches and why the proposed combination constitutes a substantial technical contribution.
  • The ablation results in Table 3 raise questions about the necessity of several design choices. First, Table 3(c) shows that adapting only the decoder achieves slightly better reconstruction accuracy than jointly adapting the decoder and latent representations. This suggests that the proposed joint optimization may not be essential. Second, Table 3(d) indicates that most of the improvements from dense cues come from surface normals, while adding depth and silhouettes provides only marginal or inconsistent improvements. For example, the PA-MPJPE remains nearly unchanged when depth and silhouette losses are added to the normal-based configuration. These results suggest that the full framework may be unnecessarily complex relative to the actual contribution of its individual components. A more detailed analysis would help justify the design choices.
  • PixelBody requires multiple pretrained prediction models and a computationally expensive iterative optimization procedure. The reported runtime is approximately 2.9-3.4 seconds per frame for the profiled components, excluding some preprocessing operations such as tracking and depth prediction. Consequently, the complete pipeline is considerably more expensive than feed-forward HMR methods. Although offline optimization is an explicit design choice, the paper does not sufficiently establish whether the accuracy improvements justify the additional computational cost. A more systematic accuracy-versus-runtime comparison would strengthen the practical significance of the proposed approach.
Questions for authors
  • The full model uses a decoder pretrained on a larger private motion-capture corpus, while the main ablation studies use the AMASS-pretrained decoder. Could the authors clarify the differences between these training datasets and provide additional evidence separating the effects of decoder pretraining from those of the proposed adaptation strategy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.