- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Mask-HMR is a one-stage multi-person mesh recovery model whose DETR queries encode implicit instance and body-part masks through generated point-wise prediction heads. Sparse mask supervision regularizes the queries, and a final part-aware pooling module refines joint rotations. The model reports strong reconstruction accuracy on standard and close-interaction benchmarks.
Critical strengths
-
The model offers a favorable accuracy–latency trade-off for multi-person mesh recovery.
-
Integrating point-wise mask prediction with person queries avoids a separate segmentation model at inference. The part-aware refinement is anatomically motivated.
-
The component ablations, point-budget analysis, and segmentation evaluation provide useful supporting evidence.
Critical weaknesses
-
The architectural novelty appears limited, as query-based mask prediction with intermediate decoder supervision is widely used in other tasks. Before final refinement, masks influence the queries through auxiliary losses rather than explicit mask-gated attention. The experiments do not clearly distinguish the benefit of the implicit representation from that of additional mask supervision.
-
Accurate masks do not necessarily ensure image-aligned meshes. Some examples show masks that follow the person’s silhouette while the projected mesh remains misaligned. This weakens the claim that the mask representation effectively constrains mesh recovery.
-
The main comparisons do not isolate architectural improvements from stronger training supervision. The final model uses substantial SAM 3D Body pseudo-labeled data, while the architectural ablations use downsampled synthetic datasets. A plain-DETR baseline trained on the full data mixture, or real-benchmark evaluation of the existing ablation models, would clarify the source of the gains.
-
Detection and reconstruction quality should be evaluated jointly. Recall is not consistently best in Table 2, and some gains in inter-person reconstruction are modest, particularly on Harmony4D.
Questions for authors
-
How much of the improvement remains when plain DETR is trained with the same full data mixture and mesh pseudo-labels?
-
Can Table 2 include detection-aware reconstruction metrics, with consistent matching rules and confidence thresholds across methods?
3 comments
Continue of Response to Reviewer aCzr
W4. Detection and reconstruction quality should be evaluated jointly. Recall is not consistently best in Table 2, and some gains in inter-person reconstruction are modest, particularly on Harmony4D.
Thank you for the suggestion. We have included the metric NMVE of Hi4D to the Table 2 as a metric evaluating both the detection and reconstruction quality. For other datasets, 3DPW, EMDB, and Harmony4D the metric is not reported as these datasets contain background persons that are not annotated. As a result, the precision of the models do not represent their detection ability and the metric NMVE is not valid.
Recall measures the ability of a model to detect all human instances in the image, and it mainly depends on the input resolution and the training data. For example, AiOS and SAT-HMR have the highest input resolutions and achieve the highest recall as they are able to detect smaller human instances. In addition, most of the training data of SAT-HMR are synthetic images in which all human instances are labeled, while for Mask-HMR, we used more real-world data, in which not all human instances are labeled. As a result, Mask-HMR is more conservative in detection at the same confidence threshold.
We agree that the inter-person reconstruction gains on Harmony4D are modest. Compared with Hi4D, Harmony4D contains smaller human instances, the average height of a person relative to the image height is 0.30 in Harmony4D, compared with 0.51 in Hi4D. As a result, the performance on Harmony4D can favor methods with higher input resolutions.
We have added a discussion to the limitations and future directions section (Appendix H, page 21) on the trade-off between the input resolution and the detection of small human instances, as well as on the limitations of the current training data.
Q1. How much of the improvement remains when plain DETR is trained with the same full data mixture and mesh pseudo-labels?
Thank you for the question. We have trained a plain DETR baseline from scratch on the same full data mixture with the same mesh pseudo-labels (Table 4, page 10). With identical training data, Mask-HMR remains more accurate on all four benchmarks, and MPJPE is lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. These results demonstrate the effectiveness of the proposed model design regardless of the training data. The gain is largest on Hi4D, the benchmark with the closest contact, which is consistent with the role of the implicit masks in separating closely interacting people.
Q2. Can Table 2 include detection-aware reconstruction metrics, with consistent matching rules and confidence thresholds across methods?
Thank you for the suggestion. We have added the detection-aware reconstruction metric, normalized mean vertex error (NMVE), together with the detection metrics recall and precision, to Table 2 (page 8). NMVE is not calculated for 3DPW, EMDB, and Harmony4D, as these datasets contains unannotated people in the background, so the precision does not reflect the detection ability of the models. On Hi4D, Mask-HMR achieves the best NMVE among all methods, 17.2 mm lower than the second-best method (Multi-HMR), demonstrating its strong performance in both detection and reconstruction.
Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?
Response to Reviewer aCzr
Thank you for the comment. We agree that query-based mask prediction is widely used in segmentation, where the mask is the output of the model. However, the novelty of Mask-HMR comes not only from the integration of these components, but also from the idea of embedding the masks directly into the human queries, which makes segmentation and mesh recovery one integrated task. In Mask-HMR, each query carries the mask of its own person through the point-wise mask training and uses it as a prompt for mesh recovery, without an external segmentation model or dense mask decoding at inference. This idea can extend beyond masks, e.g., future work could embed other prompts in the queries, such as contact information to model the interaction between people, or per-person depth to improve their placement in the scene.
The effect of the implicit masks is more than that of an auxiliary loss, as encoding the masks in the queries changes where the queries attend during cross attention. On the BEDLAM validation set, we compare (i) the model without masks with (ii) the model with implicit masks but without the per-joint refinement. The cross-attention mass that a query places on its own person increases from 33.2% to 64.5% averaged over the decoder layers. This suggest the effects of the implicit mask embedding beyond the auxiliary loss.
We do not apply mask-gated attention. Compared with segmentation, accurate human mesh recovery also requires information from the surrounding environment, which mask-gated attention would block.
Regarding the benefit of the implicit representation versus the mask supervision, the two are integrated by design in Mask-HMR, as the supervision is how each query learns its implicit masks. We therefore evaluate their gain as a single component.
Thank you for the comment, and we apologize for the confusion caused by the description. We agree that, in the current paradigm, the masks do not directly "constrain" the human mesh. First, masks can be ambiguous in certain situations, for example, when a person wears loose clothing that hides the underlying body. In addition, to maintain real-time performance, the current model does not project the predicted human mesh onto the image plane, compare it with the masks, and use the discrepancy as feedback for mesh recovery. We have revised the introduction to avoid this confusion (page 1).
"An instance mask provides a 2D silhouette guidance for body mesh recovery and can serve as a prompt for distinguishing human instances, even under occlusion or overlap."
We have also added a limitations and future directions section to the appendix (Appendix H, page 21) that discusses this issue and the future extension of using the masks as explicit constraints.
"Although masks serve as prompts for human mesh recovery, they do not strictly constrain the predicted human mesh. The underlying human mesh could still be ambiguous with the predicted masks, e.g., when a person wears loose clothing that hides the underlying body. In addition, to maintain real-time performance, the current model does not back project the predicted human mesh and compare it with the masks. Future studies that prioritize accuracy over real-time speed could incorporate the back projected mesh as a feedback, iteratively refining the mesh based on the discrepancy between its projection and the predicted masks."
Thank you for the suggestion. Following it, we trained a plain DETR baseline from scratch on the same full data mixture with the same mesh pseudo-labels (Table 4, page 10). With identical training data, Mask-HMR remains more accurate on all four benchmarks, with MPJPE lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. The largest gain is on Hi4D, the benchmark with the closest contact, where MPJPE drops by 4.4 mm and PA-MPJPE by 3.6 mm. This suggests that the implicit masks can be helpful with the reconstruction when person closely interact. We have added this comparison and its discussion to the revised paper (Section 4.4.2, page 10).