- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the task of text‑driven category‑agnostic pose estimation. To address the limitation of prior graph‑based methods, which can only model pairwise relationships between keypoints and cannot explicitly capture high‑order group‑wise relations formed jointly by multiple keypoints, this paper introduces learnable latent hypergraphs for high‑order keypoint modeling. Unlike previous approaches that employ two separate structures for feature reasoning and geometric refinement, this work leverages the same hypergraph topology to jointly perform feature reasoning and spatial coordinate updates. Experiments are conducted on the MP‑100 benchmark dataset, and improvements over prior state‑of‑the‑art methods are achieved in terms of PCK and mPCK metrics.
Critical strengths
This method introduces learnable latent hypergraphs to address the limitation of pairwise graphs, which cannot explicitly model high‑order group‑wise relations among keypoints.
Critical weaknesses
- The experiments are insufficient: comparative evaluations are conducted on only one dataset, and the ablation studies are also relatively limited.
- The total number of hyperedges is fixed, without adequate explanation of its suitability for diverse inputs.
Questions for authors
- The performance is imbalanced across different object categories. Why does the mPCK drop on the insect category compared with the baseline?
- Will the latent‑hypergraph grouping fail when keypoints are occluded?
- Why is cross‑dataset validation absent in this work?
- The balancing loss only enforces utilization of hyperedges, but does not guarantee semantically meaningful group‑wise structures. Are there any quantitative metrics to evaluate the quality of learned hyperedge grouping?
- No runtime or GPU memory overhead is reported. What is the practical inference cost introduced by hypergraph aggregation and geometric refinement?
- The noise‑augmented gating module is only activated during training and disabled at inference time. How does this design affect robustness against OOD test samples?
3 comments
Response to reviewer X5n2 (2/3)
Q2) Robustness to occlusion.
A) In CAPE, occluded query keypoints remain as prediction targets and participate in structural reasoning, although MP-100 evaluates localization only on visible keypoints. In HyperCape, these keypoints remain in the hypergraph and can exchange information with other keypoints through their shared hyperedges.
Robustness to natural occlusion. We group MP-100 Split 1 test episodes by the proportion of visible query keypoints. HyperCape outperforms CapeX across all four visibility levels, with all 95% confidence intervals excluding zero. The largest improvement is observed under severe occlusion (+4.64 mPCK).
| Occlusion level | Visible keypoints | Δ mPCK (Ours − CapeX) |
|---|---|---|
| None | 100% | +1.71 |
| Mild | 75–<100% | +1.92 |
| Moderate | 50–<75% | +1.69 |
| Severe | <50% | +4.64 |
Robustness to synthetic occlusion. To directly assess the stability of learned hyperedge assignments, we additionally evaluate 7,500 MP-100 Split 1 episodes. We randomly select 30% of query keypoints and mask each with a square whose side length is 10% of the bounding box's maximum side length. We then compare hyperedge assignments before and after masking.
| Metric | Result |
|---|---|
| Masked keypoints with unchanged memberships | 95.3% |
| Instance-level assignment Jaccard similarity | 0.959 |
| Occluded mPCK (HyperCape / CapeX) | 75.70 / 74.65 |
The learned hypergraph remains largely consistent under synthetic occlusion, with most masked keypoints retaining their original memberships. HyperCape also maintains higher absolute localization accuracy than CapeX after masking. Together, these results demonstrate that the learned grouping is largely stable under occlusion and continues to support accurate keypoint localization.
Manuscript revision. We have added Appendix B.7 with quantitative evaluations of localization performance under natural occlusion and hyperedge assignment stability under synthetic occlusion.
Q4) Quality of learned hyperedge grouping.
A) We agree that the balancing loss alone does not guarantee meaningful hyperedge grouping; it only encourages balanced utilization. However, hyperedge assignments are also learned end-to-end through the localization objective.
To evaluate grouping quality, we compare HyperCape against randomized assignments that preserve hyperedge cardinalities and the distribution of keypoint membership counts within each instance, controlling for hyperedge size and membership sparsity. We assess three properties on 400 test instances from 20 unseen MP-100 Split 1 categories:
| Metric | HyperCape | Matched random |
|---|---|---|
| Skeleton-connected hyperedges ↑ | 33.2% | 11.2% |
| Relative spatial distance ↓ | 0.78 | 1.00 |
| Cross-instance membership Jaccard ↑ | 0.868 | 0.222 |
These results are consistent across three independent samples of 400 instances. Moreover, 51.6% of co-member keypoint pairs are separated by at least three hops on the ground-truth skeleton, showing that the learned groups are not restricted to directly connected skeletal neighbors. Together, these findings provide quantitative evidence of coherent structural grouping beyond balanced hyperedge utilization.
Manuscript revision. We have added a quantitative analysis of learned hyperedge grouping quality, including comparisons with matched random assignments, in Appendix B.3.
Q5) Runtime and GPU memory overhead.
A) We measured inference latency and peak GPU memory on the same 300 MP-100 test episodes, with three paired repetitions under identical conditions.
| Model | Trainable Params. | Peak GPU Memory | Wall-clock Latency vs. CapeX |
|---|---|---|---|
| CapeX | 37.36M | 738.1 MB | Baseline |
| HyperCape () | 37.91M (+1.47%) | 732.8 MB | +2.8% (median) |
HyperCape incurs only 2.8% median wall-clock latency overhead, with comparable GPU memory usage. These results indicate that hypergraph aggregation and geometric refinement add minimal overhead to the overall inference pipeline.
Manuscript revision. We have extended the Parameter Complexity analysis in the Appendix C to include measured inference latency and peak GPU memory usage.
Response to reviewer X5n2 (3/3)
Q6) Inference-time noise and OOD robustness.
A) Noise-augmented gating serves as a training-time regularizer by exposing the model to perturbed hyperedge assignments. At inference, noise is disabled to ensure deterministic hyperedge assignments, while retaining the representations learned under these perturbations.
MP-100 evaluates generalization to unseen object categories through category-disjoint splits. To examine whether disabling noise affects this generalization, we evaluated both inference settings on Split 1. The results were nearly identical (79.44 mPCK in both settings, with differences below 0.03 percentage points at every PCK threshold). This indicates that disabling inference-time noise does not degrade performance on unseen categories in our evaluation, while ensuring deterministic predictions.
Response to reviewer X5n2 (1/3)
We sincerely thank the reviewer for the constructive feedback and for recognizing the contribution of learned latent hypergraphs to modeling higher-order group-wise relationships among keypoints. We have conducted additional experiments and analyses to address the reviewer's concerns and provide detailed responses below.
A) MP-100 is the established benchmark for category-agnostic pose estimation, and the major prior methods, including POMNet, CapeFormer, GraphCape, CapeX, CapeNext, and GenCape, follow its evaluation protocol without reporting separate cross-dataset transfer experiments. MP-100 itself comprises 100 object categories drawn from 13 source datasets, including COCO, AP-10K, CUB-200, and DeepFashion2, and evaluates generalization through five category-disjoint splits. Our evaluation therefore follows the established CAPE protocol and provides direct comparisons with prior methods under the same benchmark.
Although EdgeCape additionally reports results on AP-10K, it trains and evaluates within AP-10K rather than performing cross-dataset transfer from MP-100. Moreover, directly evaluating an MP-100-trained model on AP-10K risks train–test leakage, as AP-10K is one of MP-100's source datasets.
To further examine generalization across distinct object domains, we are conducting a cross-super-category evaluation, as discussed in our response to Reviewer sd3Q (W2). This experiment holds out entire super-categories during training, providing a stronger distribution shift than the standard category-disjoint splits. We will report the results during the discussion period and would be grateful if the reviewer could take a look.
To address the concern about limited experimental analysis, we have substantially expanded the revised manuscript with:
These additions provide a more comprehensive evaluation of HyperCape's structural modeling, robustness, and computational efficiency.
A) The fixed number M specifies the maximum available hyperedge slots, not the number of groups that must be used for every input. Hyperedge memberships are predicted independently for each instance, allowing the effective structure to adapt to different keypoint configurations.
To verify this, we analyzed hyperedges containing at least two keypoints on MP-100 Split 1 test instances. With M=8, the number of such hyperedges varies from 3 to 8 per instance, averaging 6.92 for categories with 9–15 keypoints and 8.00 for those with at least 25. With M=16, it varies more widely (3–13 per instance, with averages of 6.51–12.07 across the same keypoint ranges). These results show that a fixed hyperedge capacity does not impose a fixed number of structural groups across diverse inputs.
A) As shown in Appendix B.6 (Table 14), HyperCape outperforms CapeX in eight of nine object groups. The only exception is Insect, comprising fly and locust (2 of 100 categories; 6,000 of 300,000 queries), with a small decrease of −0.45 mPCK.
Both categories contain low-resolution grayscale images (160–192 pixels per side, versus the MP-100 median of 1,024) and densely packed keypoints. Of their 32–35 keypoints, 24 belong to six legs with highly repetitive textual descriptions. Our analysis shows that HyperCape substantially outperforms CapeX on fly body keypoints (38.1 vs. 30.0 mPCK), while the overall decrease is concentrated in the legs. For 84–95% of leg keypoints, predictions from both methods are closer to a ground-truth keypoint on another leg than to the correct one, indicating substantial ambiguity between similar legs. Since the learned hyperedges often group joints within individual legs, such confusion may also affect other joints through hyperedge aggregation, potentially contributing to the small performance decrease.
Manuscript revision. We have added a brief analysis of the Insect performance drop in Appendix B.6.