acceptodds
Under review as a conference paper at ICLR 2027

Ray-Flow HMR: Probabilistic Human Mesh Recovery with Accurate, Consistent, and Diverse Samples

Abstract

Monocular human mesh recovery is an inherently ambiguous task. This motivates a probabilistic approach. Recent works learn the distribution of body meshes conditioned on the image. However, these methods often produce samples that lack accuracy, consistency, diversity, or some combination of these. We argue that learning image-consistent samples in body-model parameter space is difficult, since the mapping that goes through body geometry and perspective projection is complex and non-linear. In this work, we propose to enforce keypoint consistency at model construction, by parameterising 3D keypoints as depths along camera rays formed by 2D keypoints. Our model, Ray-Flow HMR, generates body samples in three stages: first, we sample 2D keypoints using an image-conditioned normalising flow; then, we sample their depths along the corresponding rays using another conditional normalising flow; finally, we regress body meshes via a learned inverse kinematics module. This construction ensures that sampled 3D keypoints project to the conditioning 2D keypoints, allowing for inverse kinematics to better preserve image-space consistency for the mesh samples. On in-the-wild datasets, Ray-Flow HMR outperforms probabilistic baselines in accuracy, consistency, and diversity, while remaining competitive with deterministic methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Probabilistic, human mesh recovery, sample consistency, sample diversity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cGPwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Ray-Flow HMR models monocular human mesh recovery through a flow over 2D keypoints, a conditional flow over depths along camera rays, and a deterministic IK decoder producing SMPL parameters. Reprojection consistency is exact for intermediate joints and approximate for final meshes. Training combines BEDLAM with selective real-image supervision. Experiments on 3DPW and EMDB1 assess accuracy, reprojection consistency, and diversity, alongside a supervision ablation and uncertainty-guided view selection.

Critical strengths
  1. A useful geometric formulation. Separating 2D localization from depth ambiguity guarantees intermediate reprojection consistency. The sampled-IK objective encourages the final meshes to preserve this property while retaining variation.

  2. Promising empirical results. On EMDB1 (Table 1), the huge variant improves over HuMR in point/best-of-100 PA-MPJPE (51.33/37.09 versus 56.10/40.90 mm) and sample reprojection error (4.80 versus 5.68 pixels). It also improves consistency on 3DPW, while view selection illustrates a practical use of the predicted uncertainty.

Critical weaknesses
  1. The benefit of ray-depth sampling is not isolated experimentally. Table 2 studies real-data supervision, but does not compare ray-depth sampling against direct 3D-joint prediction under otherwise matched conditions, including the backbone, training data, supervision, camera inputs, and IK decoder. The reported results support the overall system, but do not sufficiently isolate the contribution of its central parameterization.

  2. Preservation of the geometric constraint through IK is not directly quantified. The exact reprojection guarantee holds for intermediate joints. The authors acknowledge that decoded SMPL joints can leave their rays and state that the deviation is small, but do not quantify it. Final reprojection errors are encouraging, yet do not isolate the displacement from the sampled 2D keypoints or changes in sample diversity caused by IK. This leaves a central link between the method's geometric motivation and its final outputs insufficiently validated.

Questions for authors
  1. What is the specific advance over prior work? Chen et al. (CVPR 2019), Section 3.1 and Equation (2) and ElePose (CVPR 2022), Equation (2) already reconstruct joints through depth prediction and perspective unprojection, while HuMR (WACV 2025) uses 2D uncertainty for probabilistic mesh recovery. Please clarify what your two-stage conditional distributions and IK integration add beyond these approaches, distinguishing the new probabilistic formulation from the established ray geometry.

  2. Can you isolate the benefit of ray-depth sampling? Please provide a controlled comparison against direct 3D-joint prediction with the same backbone, training data, supervision, camera inputs, and IK decoder. Reporting final-mesh accuracy, reprojection error, and diversity would help determine whether the proposed parameterization explains the gains.

  3. How much does IK decoding affect ray consistency and sample diversity? The exact reprojection guarantee applies only to the intermediate 3D joints. Please quantify the final SMPL joints' reprojection deviation from the sampled 2D keypoints and compare 3D sample diversity before and after IK, particularly for occluded joints. This would clarify how much of the geometric consistency and predicted uncertainty is preserved in the final meshes.

3 comments
AuthorsOpenReview ↗

Response to reviewer cGPw

We thank the reviewer for the review, and for finding the geometric formulation useful and the results promising. Concerns shared with other reviewers are answered in the general response; we refer to it below and address the remaining points directly.

 


W1, Q2: Isolating ray-depth sampling

See General Response G3.

We add an ablation with Cartesian flow models. Ray-depth parameterisation outperforms Cartesian models.

 


W2, Q3: Effect of IK on consistency and diversity

See General Response G4.

We analyse the effect of IK. IK preserves most of the consistency and diversity of the ray samples.

  • Consistency: IK adds only 0.21–0.54 px of reprojection error.
  • Diversity: IK keeps 89–92% of visible and 86–91% of invisible joint diversity.

 


Q1: Advance over prior work

We appreciate the comment. We agree that previous works already use the well-established ray geometry. We use it to explicitly constrain a 3D-joint distribution, which a learned IK decodes into meshes with little loss.

Compared to the specific works:

  • Chen et al. and ElePose are deterministic and train 2D-to-3D lifting networks that stay in 3D-joint space.
  • HuMR enforces consistency implicitly as a loss function. It predicts meshes by sampling SMPL parameters directly, which gives up the ray constraint.

Enforcing consistency is more difficult when a model is:

  • learning a distribution, and
  • working in the SMPL parameter space.

Our formulation addresses both:

  • explicit consistency from ray parameterisation makes the 3D-joint distribution easier to learn (ablated in G3), and
  • each keypoint sample conditions the IK to produce high-quality mesh samples that retain most of the consistency and diversity (analysed in G4).

Together, these give a better distribution than probabilistic baselines. We tighten our wording in the revised manuscript: lines 113–115, 132–136; line 92.

Thanks for running the additional experiments! The 2D + Cartesian flow baseline in G3 addresses my ablation concern by retaining the same 2D conditioning while testing the effect of the ray constraint. G4 also answers my question by measuring how far the final joint projections move from the sampled 2D keypoints and how much sample diversity is retained after IK. So I am raising my rating to 3 (Weak acceptance).

We thank the reviewer for engaging with our response and for raising their rating. We are encouraged by the feedback, and grateful for the time spent reviewing our work.

Reviewer_WHUUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Ray-Flow-HMR models image-conditioned human meshes through a cascade of a 2D keypoint flow, a conditional ray-depth flow, and a learned inverse-kinematics decoder. Parameterizing lifted joints along camera rays preserves their sampled image projections. The paper evaluates point accuracy, best-of-100 accuracy, re-projection consistency and diversity, and demonstrates an uncertainty-guided view-selection application.

Critical strengths
  1. The ray-based representation provides a useful inductive bias for an ambiguous inverse problem. Separating image-plane uncertainty from depth uncertainty is well motivated.
  2. Results on 3DPW and EMDB show improved reprojection consistency.
  3. The view-selection experiment provides encouraging evidence of downstream utility.
Critical weaknesses
  1. The central architectural claim lacks a controlled ablation. An unconstrained XYZ variant with the same encoder, flows, IK decoder, and losses is needed to isolate the benefit of the ray-based representation.

  2. Image-space reprojection consistency is an established prior in HMR. For example, GVHMR [1] uses keypoint constraints, while PEAR [2] uses dense pixel constraints. The novelty therefore needs to be distinguished more clearly from existing approaches. Moreover, exact reprojection consistency holds for the lifted joints, but not necessarily for the final decoded mesh.

  3. The evaluation provides limited evidence of probabilistic quality. A direct analysis of coverage, calibration, or sample validity on existing data would strengthen the claims.

  4. The method requires camera intrinsics, which are often unavailable or estimated inaccurately in the wild. Its sensitivity to errors in these inputs remains unclear.

  5. The first contribution reads more as a motivation than an established finding. The geometric formulation is useful, but the paper does not isolate the lack of an explicit consistency representation as the source of the observed limitations.

  6. The plausibility of the sampled variation needs closer examination. Anatomical constraints and image cues can rule out some poses even when their projections are consistent. For example, in the teaser, the person’s right arm appears to lie behind the torso, whereas the penultimate sample appears to assign it an inconsistent depth. This raises the question of whether the diversity reflects plausible ambiguity or includes geometrically inconsistent reconstructions.


[1] World-Grounded Human Motion Recovery via Gravity-View Coordinates

[2] PEAR :Pixel-aligned Expressive humAn mesh Recovery

Questions for authors
  1. With the architecture and training recipe otherwise fixed, how does ray-depth sampling compare with unconstrained XYZ sampling?

  2. How large are the reprojection and 3D joint residuals introduced by the IK decoder, particularly in the tails and for visible versus occluded joints?

2 comments
AuthorsOpenReview ↗

Response to reviewer WHUU

We thank the reviewer for the review, and for finding the ray-based representation a useful inductive bias and the view-selection experiment encouraging. Concerns shared with other reviewers are answered in the general response; we refer to it below and address the remaining points directly.

 


W1, Q1: Ray depth vs. unconstrained XYZ

See General Response G3.

We add an ablation with Cartesian flow models. Ray-depth parameterisation outperforms Cartesian models.

 


W2a: Novelty over GVHMR and PEAR

We appreciate the comment. We agree that reprojection consistency is an established prior in HMR. We use camera rays to explicitly constrain a 3D-joint distribution, which a learned IK decodes into meshes with little loss.

Compared to the specific works:

  • GVHMR's keypoint constraints and PEAR's dense pixel constraints enforce consistency implicitly as training losses, which the output is free to violate. GVHMR also takes 2D keypoints as input tokens, but these do not constrain the output.
  • Both are deterministic and regress a single estimate.

Enforcing consistency is more difficult when a model is learning a distribution in the SMPL parameter space.

Our formulation addresses both:

  • explicit consistency from ray parameterisation makes the 3D-joint distribution easier to learn (ablated in G3), and
  • each keypoint sample conditions the IK to produce high-quality mesh samples that retain most of the consistency and diversity (analysed in G4, and in W2b below).

Together, these give a better distribution than probabilistic baselines. We cite GVHMR and PEAR and tighten our wording in the revised manuscript: lines 115–116, 132–136.

 


W2b, Q2: Final-mesh consistency; IK residuals and tails

See General Response G4.

We analyse the effect of IK. IK preserves most of the consistency and diversity of the ray samples.

 


W3: Probabilistic quality

See General Response G2.

We evaluate additional distribution quality metrics. Ray-Flow HMR learns a better distribution than HuMR, with more accurate typical samples and a better PA-ES.

 


W4: Sensitivity to camera intrinsics

In the revised manuscript: Appendix E and Table 9 (lines 888–908).

We add a sensitivity analysis to the camera intrinsics.

All 3DPW results in the paper already use FLNet's estimated focal length. On EMDB1, where the calibration is known, we replace the calibrated K with FLNet's estimate and scale the calibrated focal length by fixed factors.

Focal lengthf / f_calPA-MPJPE (mm) ↓Consist. (px) ↓Div. (mm) ↑
Point / Min.Point / SampleVis. / Invis.
Calibrated148.50 / 35.534.20 / 4.9651.85 / 96.05
FLNetmedian 1.1348.77 / 35.774.23 / 5.0051.85 / 95.91
Calibrated × 0.50.567.56 / 49.885.28 / 6.1549.20 / 96.46
Calibrated × 0.750.7552.19 / 37.874.33 / 5.0751.62 / 96.84
Calibrated × 1.251.2548.61 / 35.714.25 / 5.0251.85 / 95.91
Calibrated × 1.51.549.65 / 36.464.35 / 5.1352.11 / 96.29
Calibrated × 2252.00 / 38.114.56 / 5.3553.16 / 97.80

Ours (huge) on EMDB1, S = 100.

  • FLNet's estimate changes the results only marginally.
  • Over-estimating the focal length costs little, while under-estimating it degrades accuracy more.
  • Sample diversity is stable across all settings.

Existing focal length estimators are accurate enough for our model to achieve results similar to those with known intrinsics.

 


W5: Contribution 1 reads as a motivation

In the revised manuscript: line 92.

We agree. With the added Cartesian ablation in G3, we now state contribution 1 as a finding it supports:

We show that explicitly modelling the image consistency set helps the model learn the image-conditioned predictive human mesh distribution more faithfully.

 


W6: Plausibility of the samples

We agree that our method does not guarantee plausibility. We also agree about the teaser: judging from the image, the right arm should probably lie further behind the torso than in the penultimate sample.

In practice, plausibility is learned from data. The depth distribution is conditioned on the full image features, not only on the 2D keypoints, so the model learns from the image how uncertain each joint should be. Cues such as shading, occlusion and scene layout are therefore available to it, and training on the likelihood of the ground-truth depth penalises placing samples at depths those cues rule out.

Ray-Flow HMR samples are more plausible than those of the baselines. The probabilistic quality metrics in General Response G2 reflect plausibility: typical-sample accuracy and PA-ES score every sample against the ground truth, so an incorrect sample worsens both. We score better on both.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer,

As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.

Reviewer_VvhSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Ray-Flow HMR factorizes probabilistic monocular human mesh recovery into a flow over 2D keypoints, a conditional flow over their depths along camera rays, and a learned inverse-kinematics decoder producing SMPL parameters. The ray parameterization guarantees that intermediate 3D joints project to their sampled 2D keypoints. The model uses image and camera features throughout, synthetic supervision from BEDLAM, and restricted real-image supervision intended to preserve diversity. Evaluation on 3DPW and EMDB compares point/sample accuracy, 2D consistency, and 3D diversity, with an additional uncertainty-guided view-selection example.

Critical strengths
  1. The factorization has a clear geometric rationale. Separating image-plane uncertainty from depth along rays directly represents an important source of monocular ambiguity. The paper correctly distinguishes the exact intermediate reprojection constraint from approximate learned IK decoding. This offers an interpretable approach to improving consistency without eliminating depth variation.
  2. The reported results show substantial practical promise. The base model improves EMDB minimum PA-MPJPE from HuManiFlow's 59.87 to 41.02 mm and sample consistency from 9.41 to 5.78 pixels (Table 1). Shared visibility masks, sample-count curves, the real-data supervision ablation, and uncertainty-guided view selection make the evidence broader than point accuracy alone. These results are the main reason I regard the paper as close to the acceptance boundary.
Critical weaknesses
  1. The baseline comparison and attribution evidence are incomplete. The paper discusses SAM 3D Body: Robust Full-Body Human Mesh Recovery (CVPR 2026) in its limitations, but does not provide a quantitative comparison. I acknowledge the authors' stated data/compute constraints and that their primary contribution is probabilistic HMR. Nevertheless, reporting SAM 3D Body's point-estimate accuracy and 2D reprojection consistency on the shared benchmarks would help establish the practical accuracy gap and contextualize the proposed method's benefits. Such a comparison should use compatible joint definitions and alignment, account for the MHR/SMPL representation difference, and match the available input information, including auxiliary prompts. This would be an accuracy reference, not a substitute for the probabilistic evaluation.

    The contribution of ray parameterization is also not fully isolated: Appendix B varies real-data supervision, but does not compare ray-depth and Cartesian-joint sampling with the same encoder, data, supervision, and IK decoder.

  2. Validation under severe occlusion and complex multi-person scenes is insufficient. Although the paper reports visible/invisible-joint diversity and qualitative examples, it does not provide a dedicated evaluation stratified by occlusion severity or focused on crowded scenes with inter-person occlusion. Aggregate results on 3DPW and EMDB do not establish robustness in these particularly ambiguous cases. Since representing uncertainty in occluded body parts is a central motivation, targeted evidence is important for assessing whether plausible diversity and image consistency are retained when the target person is heavily occluded or overlaps with others.

    Final-output validation also remains incomplete: the learned IK decoder's effect on reprojection consistency is not quantified, and best-of-100 error and dispersion alone do not establish calibrated uncertainty or typical-sample quality. The huge model's lower invisible-joint diversity than HuMR further motivates examining these trade-offs in difficult visibility conditions.

Questions for authors
  1. Targeted validation: Under the same base encoder and training policy, how does a Cartesian-joint flow followed by the same IK decoder compare with the ray-depth flow? This would isolate the main architectural contribution.
  2. Clarification and measurement: What are the mean and tail reprojection deviations relative to sampled 2D joints before and after IK, and how much diversity is lost in the conversion? Please also specify how positive depth is enforced.
  3. Validation using existing predictions: Can you report typical-sample accuracy and a distribution-sensitive measure such as energy score or empirical coverage, with subject/sequence-aware uncertainty? Please interpret the lower invisible-joint diversity relative to HuMR as part of that trade-off.
  4. Challenging-scene evaluation: Can you report accuracy, reprojection consistency, and sample diversity by occlusion severity, including target-person recovery in crowded scenes with substantial inter-person occlusion? This would test the proposed uncertainty modeling in the situations that most directly motivate it.
5 comments
AuthorsOpenReview ↗

Response to reviewer VvhS

We thank the reviewer for the detailed review, and for finding the factorisation geometrically well motivated and the results promising. Concerns shared with other reviewers are answered in the general response; we refer to it below and address the remaining points directly.

 


W1a: SAM 3D Body as an accuracy reference

In the revised manuscript: Table 1 (line 379); line 375.

We add SAM 3D Body as a deterministic reference in the main results. An excerpt of the updated table:

3DPWEMDB1Harmony4D
MethodPA-MPJPE (mm) ↓Consist. (px) ↓PA-MPJPE (mm) ↓Consist. (px) ↓PA-MPJPE (mm) ↓Consist. (px) ↓
CameraHMR40.893.5346.614.2750.084.68
SAM 3D Body34.664.1837.962.9834.374.47
Ours (huge)40.103.9048.504.2050.814.76

Point estimates.

SAM 3D Body meshes are converted to SMPL with the MHR conversion tool. It is the more accurate deterministic method, trained with much more data and compute.

 


W1b, Q1: Ray depth vs. Cartesian joints

See General Response G3.

We add an ablation with Cartesian flow models. Ray-depth parameterisation outperforms Cartesian models.

 


W2a, Q4: Severe and inter-person occlusion

See General Response G1.

We add Harmony4D, and stratify results by partner occlusion. Ray-Flow HMR outperforms probabilistic baselines on Harmony4D.

 


W2b, Q2a: Effect of IK

See General Response G4.

We analyse the effect of IK. IK preserves most of the consistency and diversity of the ray samples.

 


Q2b: Enforcing positive depth

The depth flow samples log-depths, so every depth is positive by construction.

We clarify this in the revised manuscript: Equation 8 (line 291).

 


W2b, Q3a: Typical-sample quality and energy score

See General Response G2.

We evaluate additional distribution quality metrics. Ray-Flow HMR learns a better distribution than HuMR, with more accurate typical samples and a better PA-ES.

 


W2b, Q3b: Uncertainty across sequences; invisible-joint diversity

With the updated model in G0, our invisible-joint diversity is now higher than HuMR's on all three datasets.

Invisible-joint diversity (mm) ↑3DPWEMDB1Harmony4D
HuMR75.41†88.2753.28
Ours (huge)79.9896.0565.36

†: HuMR fine-tuned on 3DPW-train.

The improvements over HuMR hold across sequences: every 95% confidence interval excludes zero.

We compute the intervals with a bootstrap over sequences:

  • redraw the set of test sequences with replacement, 2,000 times;
  • recompute the gap between the two models on each draw;
  • take the middle 95% of the gaps, reported as intervals in brackets.
Improvement over HuMR (mm) ↑EMDB1Harmony4D
Typical-sample PA-MPJPE4.7 [2.4, 6.9]4.7 [4.1, 5.4]
PA-ES3.9 [2.0, 5.7]5.2 [4.8, 5.8]
Sequences where Ours has lower PA-ES15 / 1739 / 39

The higher diversity does not harm typical-sample accuracy or overall distribution quality (PA-ES). On invisible joints, both are on par with HuMR on EMDB1 and better on Harmony4D:

Invisible joints: improvement over HuMR (mm) ↑EMDB1Harmony4D
Typical-sample PA-MPJPE2.4 [−4.2, 9.2]1.7 [0.8, 2.8]
PA-ES0.8 [−3.5, 4.8]3.1 [2.2, 4.0]

95% confidence intervals in brackets.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer,

As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.

Thank you for the additional experiments. The SAM 3D Body comparison, Cartesian ablations, and IK analysis substantially address my earlier concerns, and I am raising my rating from 2 to 3 (Weak acceptance). My remaining concern is the exclusion of 457 Harmony4D instances involving target-person confusion. Please clarify whether the exclusion criterion and excluded set are identical across methods, and report results accounting for these failures.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank the reviewers for their comments. These have prompted several new experiments to address the key concerns raised by more than one reviewer. We discuss these in the general responses below. Most of these results are now part of the revised manuscript, which is marked with blue text.

 


G0. Updated main results

In the revised manuscript: Table 1 (line 379).

Since the submission, we have improved training on real data: we mask real-data losses to visible joints only, and add an L1 loss on the depth of the zero-noise sample. The new model now gives more accurate point estimates and more diverse samples, with similar consistency. We have updated the main manuscript and figures accordingly.

3DPWEMDB1
MethodPA-MPJPE (mm) ↓Consist. (px) ↓Div. (mm) ↑PA-MPJPE (mm) ↓Consist. (px) ↓Div. (mm) ↑
Point / Min.Point / SampleVis. / Invis.Point / Min.Point / SampleVis. / Invis.
Ours (huge)40.50 / 29.523.76 / 4.1840.43 / 66.9251.33 / 37.094.21 / 4.8045.08 / 79.34
Ours (huge, new)40.10 / 29.853.90 / 4.3845.21 / 79.9848.50 / 35.534.20 / 4.9651.85 / 96.05

 


G1. Harmony4D as additional evaluation dataset

In the revised manuscript: Table 1 (line 379); lines 423–424; Appendix D and Table 8 (lines 836–863).

We add Harmony4D, a dataset of close two-person interactions, as a third evaluation set, and stratify results by occlusion severity.

  • Harmony4D is now a column in Tables 1 and 2 of the revised manuscript.
  • To test behaviour under heavy occlusion, we stratify by partner occlusion: the share of the target's silhouette hidden by the other person.

Visibility is computed by projecting the ground-truth meshes of both people. Under heavy occlusion a model may reconstruct the partner instead of the target; we exclude these instances (457 of 11,642).

OcclusionInstancesMethodPA-MPJPE (mm) ↓Consist. (px) ↓Div. (mm) ↑PA-ES (mm) ↓
Point / Min.Point / SampleVis. / Invis.Vis. / Invis.
< 10%9,896HuMR55.69 / 39.815.13 / 5.9539.75 / 46.2433.60 / 41.88
Ours47.89 / 35.064.55 / 5.2154.72 / 59.9628.09 / 39.72
10–30%651HuMR80.39 / 55.437.13 / 8.4551.37 / 73.0645.55 / 63.89
Ours66.13 / 47.156.06 / 7.1759.86 / 78.9137.60 / 56.67
30–50%316HuMR89.22 / 61.177.81 / 9.5064.68 / 80.7451.63 / 64.75
Ours75.90 / 53.446.89 / 8.6673.18 / 88.2642.39 / 59.96
≥ 50%322HuMR96.79 / 66.628.01 / 9.8278.71 / 84.9757.40 / 64.22
Ours84.98 / 58.476.32 / 8.2488.09 / 90.0048.27 / 57.68
Overall11,185HuMR59.25 / 42.105.41 / 6.3141.52 / 53.2835.00 / 46.64
Ours50.81 / 36.964.76 / 5.5155.92 / 65.3629.22 / 43.58

Min.: best of 100 samples. Sample: averaged over the 100 samples. Vis. / Invis.: joints visible or occluded in the ground truth.

Ray-Flow HMR outperforms HuMR on every metric at every occlusion level. We provide qualitative comparisons in Appendix D of the revised manuscript.

AuthorsOpenReview ↗

General Response (2/3)


G2. Quality of the learned distribution

In the revised manuscript: Table 2 (line 391); lines 426–450.

We add probabilistic quality metrics and show that Ray-Flow HMR learns a better distribution.

We report the following:

  • Typical-sample PA-MPJPE: the PA-MPJPE averaged over all samples, i.e. the average sample quality.
  • PA-ES: energy score 𝔼‖x − y‖ − ½ 𝔼‖x − x′‖, where y is the ground-truth joint and x, x′ are independent Procrustes-aligned samples. This jointly scores accuracy and spread; lower is better.
  • 90% coverage: the share of ground-truth coordinates inside the samples' central 90% interval.
DatasetMethodTypical PA-MPJPE (mm) ↓PA-ES (mm) ↓90% cov. ↑
All / Vis. / Invis.All / Vis. / Invis.Vis. / Invis.
EMDB1HuMR55.79 / 53.74 / 108.7235.60 / 34.38 / 67.080.67 / 0.68
Ours (huge)51.10 / 48.96 / 106.2831.69 / 30.35 / 66.320.67 / 0.66
Harmony4DHuMR60.05 / 56.02 / 75.4937.41 / 35.00 / 46.640.69 / 0.70
Ours (huge)55.36 / 50.56 / 73.7832.19 / 29.22 / 43.580.74 / 0.74

S = 100, on COCO-17 joints. Vis. / Invis.: the HuMR visibility split on EMDB1; ground-truth visibility on Harmony4D.

We exclude 3DPW from this analysis due to the biases learned by HuMR fine-tuning on 3DPW-train.

Our model produces:

  • more accurate typical samples, and
  • better probabilistic distributions in terms of energy score.

Coverage is similar to HuMR's; it measures only the calibration of the spread, so it should be read with the other metrics.

 


G3. Ablation with Cartesian normalising flow

In the revised manuscript: Table 3 (line 433); lines 452–460.

Ray-depth parameterisation outperforms Cartesian flow models.

We ablate the ray-depth parameterisation with two Cartesian flow models:

  • Cartesian flow: a single 3D flow that samples joints directly in 3D space.
  • 2D + Cartesian flow: keeps our 2D flow, but its second stage samples each joint in Cartesian coordinates around the root, conditioned on the 2D sample, instead of along that joint's ray.

All other conditions are held fixed with the main model.

DatasetMethodPA-MPJPE (mm) ↓Consist. (px) ↓Div. (mm) ↑PA-ES (mm) ↓
Point / Min.Point / SampleVis. / Invis.Vis. / Invis.
3DPWCartesian flow41.25 / 31.214.21 / 4.5736.59 / 63.5431.32 / 58.61
2D + Cartesian flow40.91 / 31.104.22 / 4.9746.78 / 78.7029.25 / 56.30
Ours (huge)40.10 / 29.853.90 / 4.3845.21 / 79.9828.96 / 55.41
EMDB1Cartesian flow50.02 / 37.955.20 / 5.7139.09 / 72.4633.82 / 71.68
2D + Cartesian flow53.42 / 39.164.82 / 5.8749.54 / 95.4733.18 / 73.57
Ours (huge)48.50 / 35.534.20 / 4.9651.85 / 96.0530.35 / 66.32
Harmony4DCartesian flow53.95 / 39.827.13 / 7.5741.94 / 49.1133.53 / 47.71
2D + Cartesian flow53.14 / 40.035.04 / 6.1957.37 / 65.9230.51 / 45.20
Ours (huge)50.81 / 36.964.76 / 5.5155.92 / 65.3629.22 / 43.58

S = 100; all models ViT-Huge, evaluated with the protocol of Table 1.

  • The Cartesian flow is worse than ours on every metric, on all three datasets.
  • Loosely conditioning 3D on 2D gains a little diversity on some joint groups, but at a large cost in consistency and sample accuracy, and hence a worse PA-ES.

Only with explicit ray parameterisation does a model sample accurately, consistently and diversely at the same time.

AuthorsOpenReview ↗

General Response (3/3)


G4. Effect of IK on consistency and diversity

In the revised manuscript: Tables 4–6; lines 462–480.

IK preserves most of the consistency and diversity of the ray samples.

Before IK, every lifted joint projects exactly onto its sampled 2D keypoint, so any change in consistency comes from IK alone. We measure it on all three datasets:

  • Consistency: IK adds only 0.21–0.54 px of reprojection error.
  • Diversity: IK keeps 89–92% of visible and 86–91% of invisible joint diversity.
  • Joint shift: on average 2.0–3.0 px in 2D; 90% of joints move at most 6.2 px.
Change due to IKConsist. (px) ↓Div. Vis. (mm) ↑Div. Invis. (mm) ↑
Before → After IKBefore → After IKBefore → After IK
3DPW4.02 → 4.2849.18 → 45.2188.36 → 79.98
EMDB13.61 → 4.1558.12 → 51.85111.52 → 96.05
Harmony4D4.33 → 4.5461.08 → 55.9274.75 → 65.36

Consistency here uses visible COCO-17 joints.

 

2D joint shift due to IK (px)MeanVis.Invis.p90p95p99
3DPW2.01.93.54.25.810.5
EMDB13.02.65.66.29.623.3
Harmony4D2.52.23.14.86.916.3

 

3D joint shift due to IK (mm)MeanVis.Invis.p90p95p99
3DPW222034435694
EMDB138326071106269
Harmony4D3127365878155

Ours (huge), S = 100.

Invisible joints shift more than visible ones, as expected without image evidence. The p99 tails mainly come from images where the model fails to reconstruct the person.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.