MASK-HMR: Real-Time Multi-Person Human Mesh Recovery with Implicit Mask Prompts

Masks provide informative visual prompts for human mesh recovery (HMR). However, existing methods using mask prompts for HMR typically rely on an upstream segmentation model, increasing computational cost and inference latency. We propose Mask-HMR, a one-stage multi-person human mesh recovery framework with implicit mask prompts. Mask-HMR follows a DEtection TRansformer (DETR) pipeline, in which a fixed set of learnable human queries is initialized and decoded into mesh parameters. In Mask-HMR, each human query learns to encode implicit instance and body-part masks of the associated human instance. A learned generator maps the query to a pointwise mask prediction head, which predicts instance and body-part masks of the associated human at a given image point. By encoding mask information directly in the human queries, Mask-HMR benefits from mask prompts without dense mask prediction. In addition, instead of full mask supervision, the implicit masks and the generator can be trained with a small set of sampled points. Mask-HMR thus gains the benefit of mask prompting with limited inference and training costs. We also design a per-joint pose refinement module at the final layer, where the pointwise mask prediction head of each query is evaluated in the region of interest to extract joint-relevant features for joint pose refinement. Experiments show that Mask-HMR achieves state-of-the-art performance among one-stage multi-person HMR methods on both standard and close-interaction benchmarks while retaining real-time inference.