acceptodds
Under review as a conference paper at ICLR 2027

HR-ViP: Leveraging High-resolution Vision Transformers for Whole-Body Human Pose Estimation

Abstract

Whole-body human pose estimation requires accurate localization of dense body, face, hand, and foot landmarks while maintaining efficiency as the number of people increases. Existing approaches face a fundamental trade-off. Top-down methods apply high-resolution Vision Transformers to individual person crops, providing detailed spatial representations but incurring computation for each detected person. In contrast, shared-feature multi-person methods process all instances within a common image representation but struggle to maintain both accuracy and speed for dense whole-body landmarks. We present HR-ViP, a top-down framework that efficiently leverages high-resolution Vision Transformers for fine-grained whole-body localization and multi-person inference. HR-ViP introduces Train-Time Random Token Subsampling (TRTS), an efficient strategy for adapting ViTs to high-resolution inputs, together with feature-map distillation to transfer their spatial representations to smaller student models. Combined with batched person-crop processing, the distilled models enable efficient multi-person whole-body pose estimation while retaining the benefits of high-resolution representations. In the single-person setting on COCO-WholeBody, HR-ViP achieves 72.1 WB AP at a resolution of . At , an 86M-parameter DINOv3-B student matches the 66.8 WB AP of a 300M-parameter DINOv3-L model while increasing throughput from 91 to 136 FPS. In the multi-person setting, HR-ViP achieves 68.1 WB AP at 50 FPS with ten detected people, substantially outperforming the full-image methods in whole-body accuracy while retaining real-time throughput. These results demonstrate that HR-ViP achieves a strong accuracy–throughput trade-off for both fine-grained whole-body localization and multi-person inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human pose, transformer, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Axe8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents HR-ViP, a top-down framework for whole-body human pose estimation that leverages high-resolution Vision Transformers together with Train-Time Random Token Subsampling (TRTS), feature-map distillation, and batched person-crop processing. The goal is to achieve a strong accuracy–throughput trade-off in both single-person and multi-person settings. Experiments are conducted on COCO-WholeBody v1.0.

Critical strengths
  1. Comprehensive single-person comparison. Table 1 covers a wide range of baselines, including CNN-based, CSPNeXt-based, and Transformer-based models, and includes strong recent methods such as RTMPose, DWPose, RTMW, and CIGPose. The comparison is thorough and the results are competitive.

  2. Practical motivation. The trade-off between top-down accuracy and shared-feature efficiency in multi-person whole-body pose estimation is a real and important problem, especially for real-time applications.

  3. Detailed runtime reporting in Table 1. The authors specify the hardware (Nvidia H200), batch size, mixed precision, and the absence of ONNX/TensorRT acceleration, which improves reproducibility.

Critical weaknesses
  1. The multi-person comparison does not match the rigor of the single-person comparison (major) Table 1 provides a strong SOTA comparison for the single-person setting. However, Table 2, which addresses the multi-person setting, does not include the strongest top-down methods that already appear in Table 1, such as RTMW, DWPose, and CIGPose. Instead, Table 2 compares HR-ViP against weaker or older full-image baselines such as CID, YOLOPose-L, and RTMO-L. No results on the CrowdPose dataset.

As a result, the abstract’s claim that HR-ViP “substantially outperforming the full-image methods in whole-body accuracy while retaining real-time throughput” is not directly supported. The reader cannot tell whether HR-ViP remains competitive against RTMW or CIGPose when multiple people are present.

  1. Missing multi-person scaling analysis Table 2 only reports FPS@1, FPS@10, and FPS@15. There is no scaling curve showing how throughput and accuracy degrade as the number of detected people increases. Since the central motivation of the paper is efficiency in multi-person scenarios, a scaling analysis is necessary to substantiate the claim of “retaining real-time throughput.”

  2. Novelty of TRTS is not sufficiently differentiated TRTS shares conceptual similarities with existing token compression and sparsification methods such as DynamicViT and Token Merging. Does TRTS differ fundamentally from these approaches, There is no ablation study for this.

  3. Distillation recipe lacks differentiation from prior work Feature-map distillation has been widely used in pose estimation, including in DWPose and RTMW. The paper does not explain how the proposed distillation scheme differs from or improves upon these existing recipes.

  4. Unclear relationship between detector quality and final WB AP In Table 2, HR-ViP-B + KD + DARK with Sparse R-CNN (DINOv3-L) has a lower Det. AP (58.4) than the YOLOX-L variant (60.7), yet achieves a higher WB AP (68.1 vs. 68.5) and much higher FPS@10 (49.8 vs. 32.6). This seems counterintuitive and needs explanation.

  5. Runtime definition in Table 2 The runtime values for prior full-image methods “may use different hardware and are reported for reference.” This makes the FPS comparison in Table 2 potentially "meaningless".

  6. batched person-crop processing seems not be tested in Ablation study.

Questions for authors
  1. Can you provide multi-person results for RTMW, DWPose, and CIGPose on COCO-WholeBody, or explain why such results are unavailable?

  2. How does TRTS differ from DynamicViT and Token Merging, both conceptually and empirically?

  3. How does your distillation recipe differ from the two-stage distillation in DWPose and the distillation strategy in RTMW?

  4. Can you provide a scaling curve of FPS and WB AP as the number of people increases from 1 to 30? or 10 is the up-limit of the paper?

  5. Why does the Sparse R-CNN (DINOv3-L) detector yield higher WB AP and FPS despite lower Det. AP?

  6. How much does the improvement comes from a stronger baseline: DINOv3?

  7. What is the configuration of HR-ViP (ResizeConv, Top-k) in the scatter plot, and is it a single-person or multi-person result?

Reviewer_P1WLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work aims to solve whole-body pose estimation with an accuracy–throughput trade-off. They propose a top-down method based on efficient high-resolution backbone adaptation and feature-map distillation. Additional strategies used include batched multi-person processing pipeline. Experiments on the benchmark COCO WholeBody show gains and also show a trade-off between accuracy and efficiency.

Critical strengths

--For whole-body pose estimation, the idea of using high-resolution backbone adaptation and distillation is reasonable and the proposed HR-ViP is elaborate.

It achieves an accuracy–throughput trade-off.

It carries out abundant ablation study about distillation strategies, decoder designs and detector selections.

Many details and additional experiments are provided in Appendix.

Critical weaknesses

In the visualization section, the qualitative results could be explained in more detail. For example, in Figure 3, whether the false positive detection of background objects at the right up corner which appears in YoloPose result also exists in HR-ViP? This would make the qualitative analysis more informative and easier to interpret. About feature visualization. The paper aims to solve trade-offs for both fine-grained whole-body localization and multi-person inference. It is better to provide keypoint heatmaps to support the model can focus on fine-grained regions even in overlapping multiple persons.

--It is more like a technique report, but of limit methodological novelty. One of the contributions - the task-specific feature distillation uses popular pose supervision and feature MSE (and compares with spatial KL distillation). Besides, DWPose and RTMW also adopt knowledge distillation for efficient whole-body estimation. Another contribution just compares different decoder variants.

--In Fig.1(b), some methods are not specially designed for whole-body pose estimation, such as CID and RTMO. If they are to verify “full-image methods exhibit weaker dependence on person count”, it is reasonable.

--About fair comparison. Performance of whole-body pose estimation is highly related to input image resolutions. In table 1, denoting HR-ViP-L+DARK as the best one is unfair, since it is at 1024768, a much higher resolution than other compared methods at 384288. It is suggested to select the best results under the same image resolution. In fact, under 384*288 input setting, the proposed HR-ViP is less competitive than DWPose and LTMW on several metrics.

-- no teacher performance as the ceiling are reported. In the knowledge distillation analysis, it only report effect of teacher capacity and student input resolution on the compact student model, as in Table 7.

Questions for authors

The most important factor driving my rating is the discussion of its novelty and some explanations about experiment comparisons.

Reviewer_xxH7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The work presents HR-ViP, a transformer framework for high resolution whole body HPE. The paper introduces TRTS to reduce the cost of finetuning larger ViTs, introduces feature distillation and batched person crop processing. The paper additionally explores different heatmap decoders.

Critical strengths

1, The experimetns support the claim that high resolution ViTs can be finetuned efficiently without reduction in accuracy. TRTS demonstrates reduction in training memory and time. 2. The student models and batched inference demonstrate improvements and are useful for practical deployement of whole body pose estimators

Critical weaknesses
  1. The entire methodology seems incremental and novelty is limited: The proposed components of the paper are closely established techniques. The TRTS component as a random patch removal has been explored in [1], ViTPose already explores the advantages of distillation [2], DWPose[3] uses the intermediate feature supervision for whole body estimation. HR-ViP largely a combination of above techniques with a pretrained DiNOv3 and the batched inference seems more of an implementation trick. The paper needs to establish perhaps a methodological insight/analysis neyond the combination of existing techniques.

  2. The experiments are limited to single dataset (COC-wholebody), an additional evaluation on other dataset say Halpe (MMPose supports this) can help strengthen the generalization claims. Additionally, for Table 6, 7 it would be helpful to see results across various seeds as differences are minor.

  3. MMPose inference’s implementations supports collation of person samples and contribution of batched pose seems limited. A stage-wise latency breakdown can help understand where the gains are coming from beyond batching. In Table 8, the batched method does show improvement compared to sequential, Can the authors show the improvement more isolated as crop, backbone inference, DARK refinement and coordinate restoration.

  4. Can the authors specify source and evaluation protocol of multiperson baselines in Table 2. Few of the baselines are trained originally on regular body keypoints. This can help with the reproducibility. Also, since its already mentioned that prior art could be run on different hardware, it would be better to report at least couple of competing methods across common hardware.

  5. Can the authors specify which teacher model and resolution was used in Table 1 distillation results? Additionally it would be helpful to compare students distilled from full token as well as TRTS trained teacher model under identical setup. This can help understand whether training efficiency gained from TRTS also retains the effective knowledge.

[1] Liu, Yue, et al. "Patchdropout: Economizing vision transformers using patch dropout." 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023.

[2] Xu, Yufei, et al. "Vitpose: Simple vision transformer baselines for human pose estimation." Advances in neural information processing systems 35 (2022): 38571-38584.

[3] Yang, Zhendong, et al. "Effective whole-body pose estimation with two-stages distillation." 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, 2023.

Questions for authors

All questions/points raised in weaknesses from 2-5 are important and they should be answered.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.