acceptodds
Under review as a conference paper at ICLR 2027

MASK-HMR: Real-Time Multi-Person Human Mesh Recovery with Implicit Mask Prompts

Abstract

Masks provide informative visual prompts for human mesh recovery (HMR). However, existing methods using mask prompts for HMR typically rely on an upstream segmentation model, increasing computational cost and inference latency. We propose Mask-HMR, a one-stage multi-person human mesh recovery framework with implicit mask prompts. Mask-HMR follows a DEtection TRansformer (DETR) pipeline, in which a fixed set of learnable human queries is initialized and decoded into mesh parameters. In Mask-HMR, each human query learns to encode implicit instance and body-part masks of the associated human instance. A learned generator maps the query to a pointwise mask prediction head, which predicts instance and body-part masks of the associated human at a given image point. By encoding mask information directly in the human queries, Mask-HMR benefits from mask prompts without dense mask prediction. In addition, instead of full mask supervision, the implicit masks and the generator can be trained with a small set of sampled points. Mask-HMR thus gains the benefit of mask prompting with limited inference and training costs. We also design a per-joint pose refinement module at the final layer, where the pointwise mask prediction head of each query is evaluated in the region of interest to extract joint-relevant features for joint pose refinement. Experiments show that Mask-HMR achieves state-of-the-art performance among one-stage multi-person HMR methods on both standard and close-interaction benchmarks while retaining real-time inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human mesh recovery, Human instances segmentation, Implicit masks, Detection Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aCzrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Mask-HMR is a one-stage multi-person mesh recovery model whose DETR queries encode implicit instance and body-part masks through generated point-wise prediction heads. Sparse mask supervision regularizes the queries, and a final part-aware pooling module refines joint rotations. The model reports strong reconstruction accuracy on standard and close-interaction benchmarks.

Critical strengths
  1. The model offers a favorable accuracy–latency trade-off for multi-person mesh recovery.

  2. Integrating point-wise mask prediction with person queries avoids a separate segmentation model at inference. The part-aware refinement is anatomically motivated.

  3. The component ablations, point-budget analysis, and segmentation evaluation provide useful supporting evidence.

Critical weaknesses
  1. The architectural novelty appears limited, as query-based mask prediction with intermediate decoder supervision is widely used in other tasks. Before final refinement, masks influence the queries through auxiliary losses rather than explicit mask-gated attention. The experiments do not clearly distinguish the benefit of the implicit representation from that of additional mask supervision.

  2. Accurate masks do not necessarily ensure image-aligned meshes. Some examples show masks that follow the person’s silhouette while the projected mesh remains misaligned. This weakens the claim that the mask representation effectively constrains mesh recovery.

  3. The main comparisons do not isolate architectural improvements from stronger training supervision. The final model uses substantial SAM 3D Body pseudo-labeled data, while the architectural ablations use downsampled synthetic datasets. A plain-DETR baseline trained on the full data mixture, or real-benchmark evaluation of the existing ablation models, would clarify the source of the gains.

  4. Detection and reconstruction quality should be evaluated jointly. Recall is not consistently best in Table 2, and some gains in inter-person reconstruction are modest, particularly on Harmony4D.

Questions for authors
  1. How much of the improvement remains when plain DETR is trained with the same full data mixture and mesh pseudo-labels?

  2. Can Table 2 include detection-aware reconstruction metrics, with consistent matching rules and confidence thresholds across methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aCzr

W1. The architectural novelty appears limited, as query-based mask prediction with intermediate decoder supervision is widely used in other tasks. Before final refinement, masks influence the queries through auxiliary losses rather than explicit mask-gated attention. The experiments do not clearly distinguish the benefit of the implicit representation from that of additional mask supervision.

Thank you for the comment. We agree that query-based mask prediction is widely used in segmentation, where the mask is the output of the model. However, the novelty of Mask-HMR comes not only from the integration of these components, but also from the idea of embedding the masks directly into the human queries, which makes segmentation and mesh recovery one integrated task. In Mask-HMR, each query carries the mask of its own person through the point-wise mask training and uses it as a prompt for mesh recovery, without an external segmentation model or dense mask decoding at inference. This idea can extend beyond masks, e.g., future work could embed other prompts in the queries, such as contact information to model the interaction between people, or per-person depth to improve their placement in the scene.

The effect of the implicit masks is more than that of an auxiliary loss, as encoding the masks in the queries changes where the queries attend during cross attention. On the BEDLAM validation set, we compare (i) the model without masks with (ii) the model with implicit masks but without the per-joint refinement. The cross-attention mass that a query places on its own person increases from 33.2% to 64.5% averaged over the decoder layers. This suggest the effects of the implicit mask embedding beyond the auxiliary loss.

We do not apply mask-gated attention. Compared with segmentation, accurate human mesh recovery also requires information from the surrounding environment, which mask-gated attention would block.

Regarding the benefit of the implicit representation versus the mask supervision, the two are integrated by design in Mask-HMR, as the supervision is how each query learns its implicit masks. We therefore evaluate their gain as a single component.

W2. Accurate masks do not necessarily ensure image-aligned meshes. This weakens the claim that the mask representation effectively constrains mesh recovery.

Thank you for the comment, and we apologize for the confusion caused by the description. We agree that, in the current paradigm, the masks do not directly "constrain" the human mesh. First, masks can be ambiguous in certain situations, for example, when a person wears loose clothing that hides the underlying body. In addition, to maintain real-time performance, the current model does not project the predicted human mesh onto the image plane, compare it with the masks, and use the discrepancy as feedback for mesh recovery. We have revised the introduction to avoid this confusion (page 1).

"An instance mask provides a 2D silhouette guidance for body mesh recovery and can serve as a prompt for distinguishing human instances, even under occlusion or overlap."

We have also added a limitations and future directions section to the appendix (Appendix H, page 21) that discusses this issue and the future extension of using the masks as explicit constraints.

"Although masks serve as prompts for human mesh recovery, they do not strictly constrain the predicted human mesh. The underlying human mesh could still be ambiguous with the predicted masks, e.g., when a person wears loose clothing that hides the underlying body. In addition, to maintain real-time performance, the current model does not back project the predicted human mesh and compare it with the masks. Future studies that prioritize accuracy over real-time speed could incorporate the back projected mesh as a feedback, iteratively refining the mesh based on the discrepancy between its projection and the predicted masks."

W3. The main comparisons do not isolate architectural improvements from stronger training supervision. A plain-DETR baseline trained on the full data mixture, or real-benchmark evaluation of the existing ablation models, would clarify the source of the gains.

Thank you for the suggestion. Following it, we trained a plain DETR baseline from scratch on the same full data mixture with the same mesh pseudo-labels (Table 4, page 10). With identical training data, Mask-HMR remains more accurate on all four benchmarks, with MPJPE lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. The largest gain is on Hi4D, the benchmark with the closest contact, where MPJPE drops by 4.4 mm and PA-MPJPE by 3.6 mm. This suggests that the implicit masks can be helpful with the reconstruction when person closely interact. We have added this comparison and its discussion to the revised paper (Section 4.4.2, page 10).

AuthorsOpenReview ↗

Continue of Response to Reviewer aCzr

W4. Detection and reconstruction quality should be evaluated jointly. Recall is not consistently best in Table 2, and some gains in inter-person reconstruction are modest, particularly on Harmony4D.

Thank you for the suggestion. We have included the metric NMVE of Hi4D to the Table 2 as a metric evaluating both the detection and reconstruction quality. For other datasets, 3DPW, EMDB, and Harmony4D the metric is not reported as these datasets contain background persons that are not annotated. As a result, the precision of the models do not represent their detection ability and the metric NMVE is not valid.

Recall measures the ability of a model to detect all human instances in the image, and it mainly depends on the input resolution and the training data. For example, AiOS and SAT-HMR have the highest input resolutions and achieve the highest recall as they are able to detect smaller human instances. In addition, most of the training data of SAT-HMR are synthetic images in which all human instances are labeled, while for Mask-HMR, we used more real-world data, in which not all human instances are labeled. As a result, Mask-HMR is more conservative in detection at the same confidence threshold.

We agree that the inter-person reconstruction gains on Harmony4D are modest. Compared with Hi4D, Harmony4D contains smaller human instances, the average height of a person relative to the image height is 0.30 in Harmony4D, compared with 0.51 in Hi4D. As a result, the performance on Harmony4D can favor methods with higher input resolutions.

We have added a discussion to the limitations and future directions section (Appendix H, page 21) on the trade-off between the input resolution and the detection of small human instances, as well as on the limitations of the current training data.

Q1. How much of the improvement remains when plain DETR is trained with the same full data mixture and mesh pseudo-labels?

Thank you for the question. We have trained a plain DETR baseline from scratch on the same full data mixture with the same mesh pseudo-labels (Table 4, page 10). With identical training data, Mask-HMR remains more accurate on all four benchmarks, and MPJPE is lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. These results demonstrate the effectiveness of the proposed model design regardless of the training data. The gain is largest on Hi4D, the benchmark with the closest contact, which is consistent with the role of the implicit masks in separating closely interacting people.

Q2. Can Table 2 include detection-aware reconstruction metrics, with consistent matching rules and confidence thresholds across methods?

Thank you for the suggestion. We have added the detection-aware reconstruction metric, normalized mean vertex error (NMVE), together with the detection metrics recall and precision, to Table 2 (page 8). NMVE is not calculated for 3DPW, EMDB, and Harmony4D, as these datasets contains unannotated people in the background, so the precision does not reflect the detection ability of the models. On Hi4D, Mask-HMR achieves the best NMVE among all methods, 17.2 mm lower than the second-best method (Multi-HMR), demonstrating its strong performance in both detection and reconstruction.

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Reviewer_v22gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Propsoes a single stage multi person mesh reconstruction from a single image. The method reports improved reconstruction accuracy over existing single stage reconstruction approaches.

Critical strengths
  1. Well motivated integration with segmentation and reconstruction.
  2. consistent improves in the benchmarks demonstrated in the paper.
  3. Competitive inference time is reported for the propsoed solution.
Critical weaknesses
  1. Detection and reconstruction improvements are not clearly disentangled.
  2. Novelty is primarily in integrating existing ideas. All the three cores- single stage multi person reconstruction, implicit masks and body part guided pose estimation are all established approaches. The contribution rests in its integration.
Questions for authors
  1. Can we report mesh and pose errors on the intersection of people successfully detected by both methods, alongside separate detection precision, recall, and F1? This will help disentangle both the detection and reconstruction quality clearly
  2. If person proposals are held fixed, how much of an improvement will come from implicit mask supervision/joint refinement alone?
  3. How much do the extra pseudo-labels aid in performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer v22g

W1. Detection and reconstruction improvements are not clearly disentangled.

Thank you for the comment. On 3DPW and EMDB, all models achieve a recall close to 100%, so the improvements come from reconstruction. On Hi4D, to disentangle the improvements, we areport the detection metrics precision and recall, together with NMVE, a metric that combines detection and reconstruction (Table 2, page 8). Mask-HMR achieves the highest precision and the lowest NMVE.

We also added Table 8 (page 18), where on the Hi4D dataset we report the sepereate detection metrics as well as the mesh and pose errors on the intersection of frames with all human instances are detected by all the five methods. This analysis seperates the improvements completely and Mask-HMR achieves the highest precision and a comparable F1 score. In addition, when all methods are scored on the same people, Mask-HMR achieves the lowest error on every reconstruction metric, showing the improvement of the model on reconstruction.

As shown in Table 2 (page 8), Mask-HMR is more conservative in detection, i.e., it achieves the best precision but not the recall. While the reconstruction improvement comes from the architecture of the model, the detection performance mainly depends on the input resolution and the training data, i.e., whether all persons in the training images are labeled. SAT-HMR achieves the highest recall because its adaptive resolution scales up to 1288 pixels, compared with 896 pixels in Mask-HMR. In addition, about 80% of the 400k images that SAT-HMR uses in each epoch are synthetic images in which all persons are labeled, compared with about 40% for Mask-HMR. Since the mask supervision treats these unannotated people as background, the model may learn to suppress some the detections.

W2. Novelty is primarily in integrating existing ideas.

Thank you for the comment. We agree that the individual components are established approaches. However, the novelty of Mask-HMR comes not only from the integration of these components, but also from the idea of embedding the masks directly into the human queries, which makes segmentation and mesh recovery one integrated task.

Previous mask-prompted HMR methods first detect all human instances with a detection model, extract the associated masks with a segmentation model, and then attach each mask to a person that has already been detected. This is not directly applicable to one-stage DETR models, where the assignment between queries and the person is only determined during decoding. In Mask-HMR, each query carries the mask of its own person through the point-wise mask training and uses it as a prompt for mesh recovery, without external detection and segmentation models or dense mask decoding at inference.

This idea brings a new perspective to prompted HMR and can extend beyond masks, e.g., future work could embed other prompts in the queries, such as contact information to model the interaction between people, or per-person depth to improve their placement in the scene. We have revised the introduction (page 2) and related work (page 3) to state this contribution more clearly, and added this future direction to the appendix (Appendix H, page 21).

AuthorsOpenReview ↗

Continue of Response to Reviewer v22g

Q1. Can we report mesh and pose errors on the intersection of people successfully detected by both methods, alongside separate detection precision, recall, and F1?

Thank you for the suggestion. We added Table 8 (page 18), where on the Hi4D dataset we report the sepereate detection metrics as well as the mesh and pose errors on the intersection of frames with all human instances are detected by all the five methods. Mask-HMR is able to produce the best precision score, as well as the construction performance on the frames where the persons are detected by all methods.

MethodPrec. ↑Recall ↑F1 ↑PA-MPJPE ↓MPJPE ↓MVE ↓Pair-PA ↓
AiOS56.098.971.555.279.396.1297.3
BEV99.597.098.263.397.9119.9110.1
SAT-HMR99.399.499.360.492.9109.393.9
Multi-HMR99.096.597.847.664.680.678.9
Mask-HMR100.098.499.237.854.266.578.2

We do not report on 3DPW, EMDB, and Harmony4D, because these datasets contain other people visible in the background, while only the actors are annotated. As a result, all models have a low precision on these datasets, and the value does not reflect their ability to identify human instances accurately.

Q2. If person proposals are held fixed, how much of an improvement will come from implicit mask supervision/joint refinement alone?

Thank you for the question. To isolate the effect of the per-joint refinement with person proposals held fixed, we evaluate the model with and without the final refinement. With identical proposals, the refinement reduces PA-MPJPE by 0.4-1.5 mm on all four benchmarks, with the largest gains on EMDB and Hi4D, as shown in the table below (Table 7, page 17).

DatasetVariantPA-MPJPEMPJPEMVEPair-PA
3DPWw/o refinement38.761.072.7–
Mask-HMR38.3 (↓0.4)60.5 (↓0.5)71.8 (↓0.9)–
EMDBw/o refinement44.571.485.3–
Mask-HMR43.0 (↓1.5)70.0 (↓1.4)83.5 (↓1.8)–
Hi4Dw/o refinement40.157.269.981.5
Mask-HMR38.9 (↓1.2)56.0 (↓1.2)68.4 (↓1.5)80.8 (↓0.7)
Harmony4Dw/o refinement55.178.895.4173.4
Mask-HMR54.5 (↓0.6)78.6 (↓0.2)95.1 (↓0.3)173.4 (0.0)

As an estimate, subtracting the refinement gain from the total gain of Mask-HMR comparing to the plain DETR baseline leaves 0.8, 1.7, 3.2, and 3.3 mm in MPJPE on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. In general the implicit mask supervision provides a greater improvements comparing to the refinement module.

Q3. How much do the extra pseudo-labels aid in performance?

Thank you for the question. To separate the contribution of the pseudo-labels from that of the proposed design, we trained a plain DETR baseline from scratch on the same full data mixture with the same mesh pseudo-labels (Table 4, page 10). With identical training data, Mask-HMR remains more accurate on all four benchmarks, with MPJPE lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively. These results demonstrate the effectiveness of the model design regardless of the training data. The largest gain is on Hi4D, the benchmark with the closest contact, where MPJPE drops by 4.4 mm and PA-MPJPE by 3.6 mm. This suggests that the implicit masks can be helpful with the reconstruction when person closely interact.

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Reviewer_7RgpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Mask-HMR, a real-time, single-stage framework for multi-person human mesh recovery (HMR) using implicit mask prompts. The proposed method builds upon a DETR-style architecture, where a fixed set of learnable human queries is decoded into SMPL parameters and camera translations.

The key idea is to implicitly encode instance-level and body-part segmentation information within each human query, rather than relying on an external segmentation model. A learned generator transforms each query into a pointwise mask prediction head, which is supervised using a small number of sampled image points. This design enables the model to incorporate segmentation-related spatial and anatomical information without requiring dense mask prediction. Additionally, the authors introduce a per-joint pose refinement module that uses predicted instance and body-part masks to pool joint-relevant image features and refine the estimated joint rotations.

Experiments on 3DPW, EMDB, Hi4D, and Harmony4D demonstrate that Mask-HMR substantially outperforms existing single-stage multi-person HMR methods in reconstruction accuracy while maintaining real-time inference speed.

Critical strengths
  • The proposed framework integrates instance-level and body-part segmentation information directly into DETR-style human queries, eliminating the need for an external segmentation model during inference. This is a practically meaningful design that addresses the computational overhead associated with conventional mask-guided HMR approaches.
  • Mask-HMR achieves substantial improvements over existing single-stage methods on standard HMR benchmarks. For example, on 3DPW, it achieves a PA-MPJPE of 38.3 mm, compared with 46.9 mm for Multi-HMR and 52.7 mm for SAT-HMR. The method also achieves competitive performance on close-interaction benchmarks while maintaining an inference time of 18.3 ms per image on an RTX 4090 GPU.
  • The per-joint pose refinement module uses body-part-aware feature pooling to extract features relevant to individual joints. The design is intuitively reasonable, as different joints require information from different anatomical regions. The incorporation of an abstention mechanism for occluded body parts is also a useful design choice.
Critical weaknesses
  • The proposed framework combines several established ideas, including DETR-based multi-person HMR, implicit pointwise mask prediction, and body-part-aware feature pooling. In particular, implicit mask prediction is closely related to Implicit PointRend, while the use of body-part-specific features for pose estimation resembles the motivation behind PARE. Although integrating these techniques into a unified real-time multi-person HMR framework is valuable, the paper could more clearly articulate its methodological contributions beyond this integration.
  • The evaluated methods differ in backbone architectures, input resolutions, and potentially training datasets and annotation sources. For example, Mask-HMR uses a ViT-B backbone at a resolution of 896 pixels, whereas SAT-HMR uses an adaptive-resolution configuration and Multi-HMR uses ViT-L. Furthermore, Mask-HMR is trained using approximately 400k images from multiple datasets, including pseudo-annotations generated by SAM 3D Body. It is therefore difficult to determine how much of the reported improvement comes from the proposed architecture rather than the training data and configuration.
  • One of the central motivations of the paper is that mask information helps distinguish closely interacting or overlapping people. While the experiments on Hi4D and Harmony4D provide relevant evidence, the evaluation does not explicitly quantify performance as a function of occlusion severity, overlap ratio, or visibility of individual body parts. Consequently, it remains unclear how much of the observed improvement is specifically attributable to better handling of occlusion, rather than to general improvements in reconstruction accuracy.
  • Although the overall inference time is reported, the paper provides limited analysis of how computational cost scales with the number of people, the number of human queries, and the spatial resolution used for mask evaluation. Since the per-joint refinement module explicitly evaluates the implicit masks over person-centric regions, understanding the computational overhead of this operation would be useful, particularly for crowded scenes.
Questions for authors

Please see the weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Rgp

W1. Although integrating these techniques into a unified real-time multi-person HMR framework is valuable, the paper could more clearly articulate its methodological contributions beyond this integration.

Thank you for the comment. We agree that the individual components have architectural precedents. The novelty of Mask-HMR comes not only from the integration of these components, but also from the idea of embedding the masks directly into the human queries, which makes segmentation and mesh recovery one integrated task.

Previous mask-prompted HMR methods first detect all human instances with a detection model, extract the associated masks with a segmentation model, and then attach each mask to a person that has already been detected. This is not directly applicable to one-stage DETR models, where the assignment between queries and the person is only determined during decoding. In Mask-HMR, each query carries the mask of its own person through the point-wise mask training and uses it as a prompt for mesh recovery, without external detection and segmentation models or dense mask decoding at inference.

This idea brings a new perspective to prompted HMR and can extend beyond masks, e.g., future work could embed other prompts in the queries, such as contact information to model the interaction between people, or per-person depth to improve their placement in the scene. We believe Mask-HMR can inspire future work on prompted human mesh recovery and on broader applications where such prompts are helpful, such as 3D scene reconstruction. We have added a future direction section to the appendix that discusses the broader implications of embedding prompts into the queries (Appendix H, page 21).

W2. The evaluated methods differ in backbone architectures, input resolutions, and potentially training datasets and annotation sources. It is therefore difficult to determine how much of the reported improvement comes from the proposed architecture rather than the training data and configuration.

Thank you for the comment. Regarding the backbone and resolution, the differences do not favor Mask-HMR. SAT-HMR uses the same ViT-B backbone, and its resolution scales up to 1288 pixels for small human instances, which is higher than the 896 pixels used by Mask-HMR. Multi-HMR uses the same 896-pixel resolution with a larger ViT-L backbone. Compared with these methods, Mask-HMR achieves higher accuracy with a lighter configuration, and the improvement should not come from the backbone or the input resolution.

Regarding the training data, to isolate its effect, we trained a plain DETR baseline from scratch on the same data mixture as Mask-HMR. With identical training data, Mask-HMR remains more accurate on all four benchmarks, with MPJPE lower by 1.3, 3.1, 4.4, and 3.5 mm on 3DPW, EMDB, Hi4D, and Harmony4D, respectively (Table 4, page 10). These results demonstrate the effectiveness of the proposed model design regardless of the training data. The gain is largest on Hi4D, the benchmark with the closest contact, which is consistent with the role of the implicit masks in separating closely interacting people.

AuthorsOpenReview ↗

Continue of Response to Reviewer 7Rgp

W3. It remains unclear how much of the observed improvement is specifically attributable to better handling of occlusion, rather than to general improvements in reconstruction accuracy.

Thank you for the suggestion. First, in our comparison with the plain DETR baseline trained on the same data (Table 4, page 10), the gains of Mask-HMR are largest on the close-interaction benchmarks. The MPJPE of Mask-HMR is lower than plain DETR by 4.4 mm on Hi4D and 3.5 mm on Harmony4D, compared with 1.3 mm on 3DPW and 3.1 mm on EMDB. This demonstrates the benefit of the implicit mask prompts in close-interaction scenes.

To quantify the effect of occlusion more directly, we have also added an analysis stratified by occlusion on the Hi4D (Table 9, page 18). For each person, the occlusion is computed as the fraction of the body hidden by the other person from the ground-truth meshes. Errors are computed on the frames in which both models detect both people.

OcclusionPlain DETR MPJPEMask-HMR MPJPEPlain DETR PA-MPJPEMask-HMR PA-MPJPE
0-10%53.549.9 (-3.6)37.034.1 (-2.9)
10-30%66.463.1 (-3.3)46.844.1 (-2.7)
30-50%77.170.9 (-6.1)56.951.6 (-5.2)
>50%89.386.1 (-3.2)65.461.7 (-3.7)
All59.655.8 (-3.8)41.938.7 (-3.2)

As shown in the table, Mask-HMR is more accurate than the baseline across occlusion levels. The gain is largest at 30-50% occlusion, where MPJPE is lower by 6.1 mm and PA-MPJPE by 5.2 mm, indicating an additional benefit when people substantially overlap. Above 50% occlusion, the gain decreases. This might suggest that when the occlusion is this severe, few reliable image features remain for the reconstruction, so the mask prompts provide less benefit.

W4. The paper provides limited analysis of how computational cost scales with the number of people, the number of human queries, and the spatial resolution used for mask evaluation.

Thank you for the suggestion. We agree that an explicit analysis of the computational cost clarifies the efficiency of our model. Mask-HMR follows the DETR architecture, which always processes a fixed number of queries (30). The computational cost is independent of the number of people in the image.

For each query, the per-joint refinement module evaluates the implicit masks on a single G×GG\times G person-centric grid with G=28G=28, which results in 23,520 point evaluations per image. This operation adds 5.0 GFLOPs and 1.8 ms per image. We have added a computational cost section (Section 4.3.3, page 8) with Table 3 (page 9) to the revised paper, reporting the cost of each component.

ConfigurationGFLOPsLatency (ms)FPS
Mask-HMR32518.354.6
w/o refinement32016.560.6 (↑6.0)
+ dense masks (112×112112\times112)34723.742.2 (↓12.4)
+ dense masks (448×448448\times448)64497.110.3 (↓44.3)

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chairs and all reviewers for their time and thoughtful feedback. In this paper, we propose Mask-HMR, a real-time, single-stage multi-person human mesh recovery method. Mask-HMR embeds an implicit mask representation into each human query. The design makes the segmentation and mesh recovery an integrated task and mesh recovery benefits from mask prompts without additional detection and segmentation models. Experiments show that this design improves both training efficiency and reconstruction accuracy at a small computational cost.

To facilitate the discussion, we briefly summarize the reviews and our responses below.

Strengths:

  • Efficient model design: Mask-HMR has a well-motivated integration of segmentation and reconstruction, which gives a favorable accuracy-latency trade-off. (7Rgp, v22g, aCzr)
  • Improvements: Mask-HMR outperforms existing single-stage multi-person methods on various benchmarks. (7Rgp, v22g)
  • Well-motivated per-joint refinement: The per-joint pose refinement is anatomically motivated, with useful design choices such as the abstention mechanism. (7Rgp, aCzr)
  • Supportive results and analysis: The component ablations and the reported analysis support the design choices. (aCzr, v22g)

Concerns and Our Response:

  • Novelty lies in the integration: We agree that the individual components have architectural precedents. However, the novelty of Mask-HMR comes not only from the integration of these components, but also from the idea of embedding the masks directly into the human queries, which makes segmentation and mesh recovery one integrated task. This idea can extend beyond masks, e.g., future work could also embed other prompts in the queries, such as contact information to model the interaction between people, or per-person depth to improve their placement in the scene. We believe Mask-HMR can inspire future work on prompted human mesh recovery and on broader applications where prompts are helpful, such as 3D scene reconstruction [1]. We have revised the introduction (page 2) and related work (page 3) to emphasize the novelty. We have added a future direction section to the appendix that discusses the broader implications of embedding prompts into the queries (Appendix H, page 21). (7Rgp, v22g, aCzr)
  • Source of the performance gains: We have clarified the backbone and resolution differences among the methods. We also added an ablation that analyzes the gains from the training data (Section 4.4.2 and Table 4, page 10). (7Rgp, v22g, aCzr)
  • Performance gain of the per-joint refinement: We have added an analysis of the gain from the per-joint refinement module to the appendix (Table 7, page 17). (v22g)
  • Disentangling detection and reconstruction: We have added detection precision and recall, together with the detection-aware reconstruction metric NMVE, to disentangle the improvements (Table 2, page 8, and Table 8, page 18). (v22g, aCzr)
  • Occlusion analysis: We have added a performance analysis stratified by occlusion (Table 9, page 18). (7Rgp)
  • Computational cost: We have added a computational cost analysis (Section 4.3.3, page 8). (7Rgp)
  • Masks and meshes do not perfectly align: We have revised the introduction to describe masks as guidance rather than constraints, and added a limitations and future work section that discusses using masks as constraints for mesh recovery (introduction, page 1, and Appendix H, page 21). (aCzr)

We believe the revision has strengthened the completeness and clarity of our manuscript.

[1] Chen, Xingyu, et al. "SAM 3D: 3Dfy Anything in Images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.