acceptodds
Withdrawn from ICLR 2027

Biomechanically Grounded 3D Human Mesh Reconstruction via Spatio-Temporal Adapters

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recovering 3D human pose and shape from monocular input is central to computer vision, yet for anatomy-aware applications such as clinical motion analysis and sports biomechanics, geometric accuracy alone is insufficient: reconstructions must also be biomechanically valid. Most methods build on SMPL, whose joints are unconstrained ball-and-sockets and often produce physically impossible rotations. The SKEL model instead provides a physiologically grounded skeleton that is valid by construction, but learning to predict SKEL has remained limited by the scarcity of SKEL annotations. We introduce BioHMR, a SKEL-based framework that is simultaneously scalable, temporally coherent, and efficient. BioHMR reconstructs in an explicit skeletal latent space: a compact set of learnable joint tokens, one per SKEL joint, gathers evidence from image patches, reasons over the SKEL kinematic tree, and tracks each joint through time within a Vision Transformer, giving temporally coherent reconstruction without dense spatio-temporal attention. To overcome the annotation scarcity, a lightweight SMPL-to-SKEL mapper, trained by mesh consistency, generates pseudo-SKEL supervision from abundant SMPL annotations roughly faster than prior conversion, making large-scale SKEL training practical. The same joint-token attention drives an optional pose-aware patch pruning for a tunable compute-accuracy trade-off at inference. Across COCO, LSP-Extended, 3DPW, and MOYO, BioHMR attains state-of-the-art accuracy, efficiency, and temporal coherence among SKEL-based methods, with the lowest MPJPE on both 3DPW and MOYO and 2D accuracy on par with the strongest SMPL-based baselines. Moreover, its reconstructions stay biomechanically valid by construction, cutting per-joint violation rates on MOYO from 44–56% for SMPL-based HMR2.0 to 0–4%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human mesh recovery, efficient transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yRHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BioHMR presents a SKEL-based framework for 3D human mesh recovery from images and video. Its main components are a learned SMPL-to-SKEL mapper for generating pseudo-SKEL supervision, the Biomechanics-Aware Spatio-Temporal Adapter (BASTA) with joint tokens, graph attention, and per-joint temporal attention, and an optional pose-aware patch selection mechanism for reducing inference cost. The paper reports strong results on COCO, LSP-Extended, 3DPW, and MOYO, together with lower joint-limit violation rates than SMPL-based baselines.

Critical strengths
  • SKEL gives the paper a representation with joint-specific degrees of freedom, making joint-limit violations an explicit part of the evaluation rather than an unmeasured property of the recovered mesh.
  • BASTA combines joint-token cross-attention, reasoning over the SKEL kinematic graph, and per-joint temporal attention within the ViT backbone.
  • The evaluation spans image and video benchmarks and reports joint, surface, temporal, and joint-limit metrics, with ablations for the main components.
Critical weaknesses
  1. The mapping from the 6D rotation output to the SKEL pose representation is not specified. Section 3.3.3 describes 24 joint rotations in a 144-dimensional 6D representation, conversion to Euler angles, and selection of 46 physiologically valid axes. It does not define the Euler convention (including rotation order and angle range), joint coordinate frames, or the exact conversion to the 46-dimensional SKEL pose vector. Appendix B.4 further writes the mapper as M:R157→R147M:\mathbb{R}^{157}\to\mathbb{R}^{147}, with a 144-dimensional pose output, and feeds q^\hat{q} directly to SSKELS_{\mathrm{SKEL}} in Eq. (5), while Section 3.6 defines q∈R46q\in\mathbb{R}^{46}. The conversion from the 6D representation to the 46-dimensional SKEL pose is therefore under-specified, and the paper does not provide enough detail to reproduce this part of the pose representation.

  2. The paper uses ``valid by construction'' in two different senses. SKEL does remove forbidden rotation axes and therefore rules out gross out-of-plane motions at the structural level. It does not, by itself, keep every scalar joint angle inside its physiological range. The paper acknowledges this in Section 4.3, which states that a regressor can still hyperextend a joint, and Table 3 reports nonzero joint-limit violations at the strict >0∘>0^\circ and >5∘>5^\circ thresholds. The evidence supports a narrower claim about structural anatomical validity than a guarantee of complete joint-limit compliance.

  3. The mapper objective does not establish that the SKEL pose is uniquely identifiable from mesh consistency. Equations (5)--(7) supervise vertices, joints, and a Laplacian residual, while Appendix B.4 states that the mapper uses no optimization-derived SKEL pose labels. Symmetries, pose and shape compensation, or pose and translation compensation can therefore leave multiple parameter settings with the same or nearly the same mesh and joint losses. Selective joint masking down-weights the joint loss in regions where SMPL and SKEL differ systematically, while dense vertex supervision remains. The paper does not show that these ambiguities occur for its SKEL implementation, so the precise issue is an unproved identifiability property rather than a demonstrated failure. The resulting pseudo-labels may produce the same mesh without being uniquely determined by the stated objective.

  4. The temporal loss encourages smooth predictions, but it does not establish physical consistency. Section 3.6 penalizes adjacent-frame differences in rotation matrices and shape parameters. A constant but incorrect pose and shape can still make these temporal consistency terms zero, and the objective contains no velocity or acceleration limits, contact constraints, dynamics, or temporal regularization for camera and translation. The reported ACCEL reduction supports smoother predictions in the tested setting, while the ablation in Table 7 supports an empirical contribution from temporal attention. Neither result establishes physical consistency. The loss compares predictions at adjacent frames and places no direct constraint on the motion between them.

  5. Equation (8) uses cross-attention weights as a proxy for patch importance, but the paper gives no result connecting this score to a patch's effect on the final prediction. Attention weights alone do not measure the magnitude or direction of the value carried by a patch, so a low-weight patch can still affect the output substantially. The paper does not provide an error bound or sensitivity analysis for this selection rule, and it does not show a concrete failure case.

Questions for authors

See weaknesses

Reviewer_t7pxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BioHMR, a biomechanically informed framework for monocular human mesh reconstruction using the SKEL body model. It introduces a learned SMPL-to-SKEL mapper for scalable pseudo-label generation, a joint-token-based spatio-temporal adapter (BASTA), and pose-aware patch pruning. Experiments report competitive reconstruction accuracy, improved temporal modeling, and low joint-limit violation rates.

Critical strengths
  1. Practical supervision strategy. The learned SMPL-to-SKEL mapper substantially reduces annotation conversion costs while maintaining good mesh alignment and biomechanical validity.

  2. Promising empirical results. The method achieves strong results on 3DPW and MOYO, with useful component ablations and detailed joint-limit evaluations.

Critical weaknesses
  1. Limited architectural novelty. BASTA primarily combines existing cross-attention, graph attention, and temporal attention mechanisms. The paper does not sufficiently establish what is fundamentally new beyond their integration with SKEL joint tokens.

  2. Incomplete comparisons with recent HMR methods. The evaluation lacks several recent monocular and video-based HMR approaches. Comparisons with stronger contemporary baselines under matched training and evaluation protocols are necessary to substantiate the claimed competitiveness.

  3. Insufficient validation of biomechanical accuracy. Low joint-limit violation rates largely reflect the constraints of SKEL itself. Although label validity is evaluated, the paper does not sufficiently establish whether the predicted joint angles are anatomically accurate relative to independent ground truth.

  4. Limited comparison with biomechanical reconstruction methods. BioPose (2025) is discussed but not directly compared on shared reconstruction metrics. This weakens the evidence that the proposed framework advances beyond existing biomechanically informed video reconstruction approaches.

  5. Insufficient isolation of key design choices. The paper lacks comparisons between BASTA and simpler joint-token architectures with comparable parameter budgets. Similarly, pose-aware patch selection is not compared against standard token-pruning strategies.

  6. Unclear robustness of pseudo-label transfer. The mapper is trained on AMASS and evaluated on several datasets, but its reliability on unusual poses and challenging in-the-wild annotations is not systematically analyzed.

Questions for authors
  1. What distinguishes BASTA from existing combinations of joint-token attention, graph reasoning, and temporal modeling?

  2. Can the authors provide comparisons with more recent HMR methods and BioPose under compatible evaluation protocols?

  3. Can the authors evaluate anatomical joint-angle accuracy against independent ground truth, beyond joint-limit violations?

  4. How does BASTA compare with simpler joint-token baselines, and how does pose-aware pruning compare with standard pruning methods?

  5. How robust is the SMPL-to-SKEL mapper to unusual poses and out-of-distribution SMPL annotations?

Reviewer_gbboOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript proposes BioHMR for 3D human mesh reconstruction from images and videos using the biomechanical SKEL model. The authors claim BASTA as a novel component in the ViT architecture that incorporates joint-token cross-attention, kinematic-tree graph reasoning, and temporal attention. A learned SMPL-to-SKEL mapper generates pseudo-labels, followed by refinement. The proposed conversion pipeline is approximately 300× faster than the previous optimization-based approach. The method is trained on standard 2D/3D human pose datasets, and evaluated on 3DPW, and MOYO, COCO abd LSP-Extended.

Critical strengths

S1. The proposed SMPL-to-SKEL conversion pipeline is useful for scaling supervision, as it reduces the conversion time from 180 to 0.58 seconds per frame (Tab. 4).

Critical weaknesses

W1. Missing comparison: The manuscript does not cite or discuss an important work, SKEL-CF (ECCV 2026, first appeared on arXiv in November 2025). SKEL-CF reports 61.5/38.7 mm MPJPE/PA-MPJPE on 3DPW, compared to 65.3/39.9 mm for BioHMR. This questions the claim of state-of-the-art accuracy among SKEL-based methods.

W2. Source of improvement is not clear: Removing BASTA or temporal attention from the proposed method still achieves 70.5 and 70.9 mm MPJPE, respectively (Tab. 7), outperforming HSMR by more than 10 mm. This suggests that a large part of the improvement might come from other factors, such as pseudo-label generation or the training strategy. Additionally, Tab. 7 only ablates the entire BASTA module, which does not show the individual contribution of kinematic-tree graph reasoning, one of the claimed contributions (L114).

Questions for authors

Q1. Comparison to baselines. It is important to cite and discuss SKEL-CF, ECCV 2026 (see W1) and explain how the proposed method compares to it? SKEL-CF seems to report better results than the proposed method, but are there differences in the training or evaluation settings that explain this?

Q2. Source of improvement. It would be informative to see how BioHMR performs when trained with the pseudo-ground-truth annotations from HSMR and/or SKEL-CF (ECCV 2026). This is important because the source of improvement is still not clear even when important components are removed (see W2). Is the improvement primarily due to the proposed architecture or the better pseudo-ground-truth annotations?

Q3. Contribution of kinematic-tree graph reasoning. The paper uses graph attention over the SKEL kinematic tree. However, Tab. 7 only ablates the entire BASTA module (see W2). It would be interesting to see an ablation without graph attention or with standard self-attention across joint tokens.

Q4. Contribution of the learned SMPL-to-SKEL mapper. In Tab. 4, the proposed mapper is combined with offline optimization and a modified objective. A comparison between direct optimization and mapper-initialized optimization using the same objective would be important to show the source of improvement.

Q5. Inference efficiency. The manuscript shows a 24% reduction in inference latency for a 15% relative drop in COCO [email protected]. This seems like a substantial accuracy drop. It would be informative to see a comparison with random or uniform patch pruning at a similar computational cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.