- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work formulates Exemplar-Free Cross Category Incremental Keypoint Detection (CCIKD), a new continual learning setting for keypoint detection. In CCIKD, models sequentially learn new object categories with heterogeneous keypoint schemas (varying keypoint count, order, and semantics) without access to old category training data. The authors identify Output Identity Collision (OIC) as a core failure mode for the widely used Shared baseline: forcing semantically incompatible keypoints to share output heads via local index mapping introduces semantic ambiguity and amplifies catastrophic forgetting. To mitigate OIC, this paper proposes POISE, a prototype-guided persistent output-identity space expansion framework. POISE builds visual/geometric prototypes from training data and maintains an append-only identity bank. It supports three routing behaviors: reusing compatible existing identities (Historical Align), initializing new identities from related historical parameters (Transfer), or full random initialization (Expand). POISE decouples output identity organization from knowledge preservation objectives and is plug-and-play with diverse continual learning strategies, including EWC, LwF, and PODNet, as well as parameter-efficient methods L2P and InfLoRA. The authors construct benchmark IKDD and reconstruct MP100 into CCIKD-compliant incremental protocols across multiple category orders. Extensive experiments on ViTPose and HRNet backbones show consistent improvements over Shared. Larger gains appear on longer, highly heterogeneous streams: for the MP100-8 stage with EWC, POISE lifts FinalPCK from 45.51 to 70.66. Comprehensive ablations validate prototype design choices, routing variants, encoder sensitivity, and capacity accuracy tradeoffs. The paper also discusses limitations such as prototype-dependent compatibility estimation, append-only output growth, and the top-down setting assumption.
Critical strengths
-
The CCIKD setting fills a gap in incremental keypoint detection, where new categories arrive with fully heterogeneous keypoint schemas. The paper quantitatively diagnoses Output Identity Collision caused by naive local index output mapping, which explains why existing baselines suffer severe forgetting on heterogeneous streams. This analysis itself provides meaningful understanding for the community.
-
POISE decouples output identity routing from continual learning preservation losses. It works off the shelf with regularization-based, distillation-based, and PEFT continual learning approaches and generalizes across two different pose backbones. Comprehensive benchmarks, multi-order protocols, and rich ablations convincingly demonstrate effectiveness, especially for long, highly heterogeneous category streams.
Critical weaknesses
-
Cross-keypoint compatibility is purely estimated via visual-geometric prototypes, without any explicit semantic correspondence signal. When training instances for certain keypoints are scarce, prototype representations can be noisy and lead to suboptimal routing decisions. There is little analysis reporting routing failure cases.
-
Although the authors report memory statistics and compare against the full expanded upper bound, POISE cannot prune obsolete identities. For extremely long real-world continual sequences, readout-layer parameter expansion may become a practical deployment barrier, and the paper does not propose mitigation for this limitation.
Questions for authors
-
How does POISE behave when some category keypoints have very few training instances? Please provide supplementary analysis on the routing success rate under low-sample keypoint cases.
-
In ablation, all expand yield the highest accuracy but huge parameter overhead. Have you explored simple post-hoc identity pruning strategies after routing to constrain output size, and what performance tradeoff did you observe?
-
The evaluation assumes top-down input with known bounding boxes and category labels. Could you briefly discuss major obstacles if we want to migrate POISE to an end-to-end incremental keypoint detection pipeline without given category labels?
-
In Table 31, the test multiple pre-training encoders for prototype construction. Is there any case where a better prototype encoder hurts downstream keypoint final performance?
5 comments
Response to Reviewer aYLb (2/2)
W2+Q2. Output Capacity Growth and Identity Pruning
POISE controls output growth through compatibility-guided selective identity reuse rather than post-hoc pruning. As described in Appendix B, old category–keypoint mappings remain fixed to preserve output correspondences, while their prediction parameters remain trainable. Thus, persistence constrains identity assignments, not parameter optimization or output capacity. The original Table 5 (Table 6 of the revised paper) demonstrates a favorable accuracy–capacity trade-off: All-Expand achieves 82.05 AIAPCK with 1565 identities, whereas POISE achieves 81.78 with only 825, representing 47.3% fewer output identities and readout-layer parameters for just 0.27 points lower accuracy.
Since all old categories remain in the evaluation, directly pruning identities still referenced by old keypoints would disrupt established mappings. Safe pruning or merging requires identity reassignment while preserving old prediction functions, particularly challenging without old images. As an initial systematic study of the challenging CCIKD setting, our focus is on limiting unnecessary output expansion while preserving old correspondences. We appreciate the reviewer's suggestion, which aligns with Appendix E.1. Future work will explore resource-constrained identity consolidation, safe pruning, and identity migration to balance long-term capacity efficiency and old knowledge retention.
Q3. Extending POISE to End-to-End Incremental Keypoint Detection
Our evaluation follows the established top-down setting used by HRNet, ViTPose, and CAPE methods such as POMNet, CapeFormer, and SCAPE. CCIKD thus assumes known bounding boxes and category labels, where category information determines the keypoint schema and persistent output-identity mapping, allowing us to focus on heterogeneous incremental keypoint learning and old knowledge retention.
Extending POISE to a fully end-to-end incremental pipeline without given category labels presents three main challenges:
- Category recognition and schema selection. The model must jointly recognize categories and determine their keypoint numbers, semantics, and output routes. Incorrect category predictions may lead to inappropriate schema selection.
- Joint old knowledge preservation. Incrementally learning object detection, classification, and keypoint prediction without old images introduces forgetting across multiple components, which persistent output routing alone cannot address.
- Detection–keypoint coupling. Predicted bounding boxes introduce localization errors and scale variations that affect keypoint features and geometric relationships, requiring reliable associations between instances and their heterogeneous keypoint outputs.
To the best of our knowledge, this is the first systematic study of CCIKD, focusing on heterogeneous cross-category incremental keypoint learning and old retention. We appreciate the reviewer's suggestion, which aligns with the potential extension discussed in Appendix E.3. A promising direction is to integrate POISE with class-incremental object detection, enabling category-aware schema selection from predicted instances while jointly preserving old detection and keypoint prediction capabilities.
Q4. Impact of Prototype Encoder Quality on Detection Performance
We examined whether POISE's gains primarily stem from DINOv3 and how prototype encoder quality affects detection. To investigate this, we evaluated three prototype encoders—MAE, CLIP, and DINOv3—in original Table 31 (Table 33 of the revised paper). DINOv3 achieves the highest FinalPCK on both representative protocols, outperforming MAE and CLIP, with consistent trends in AIAPCK and BWTPCK.
POISE outperforms Shared-Kmax with all three encoders, showing that its gains are not solely attributable to DINOv3. Frozen prototypes provide visual–geometric compatibility evidence for output routing rather than directly participating in heatmap prediction. Routing quality depends on visual representations, geometric evidence, member-level compatibility, matching margins, and consistency constraints. Therefore, stronger general-purpose representations do not necessarily guarantee better routing or downstream detection performance. These results support the effectiveness and encoder adaptability of POISE's persistent output-identity organization.
We have incorporated the above additional experimental analyses into Appendix D.10 of the revised manuscript. We sincerely thank the reviewer for the constructive feedback and hope these revisions adequately address the concerns raised. If the reviewer has any further questions or concerns, we are willing to provide additional details. Thank you again for your time and thoughtful feedback.
Dear Reviewer aYLb,
Thank you again for your thoughtful review and constructive feedback. Last week, we provided point-by-point responses to your concerns and questions, together with additional experiments on prototype-sampling sensitivity and routing stability, analyses of accuracy–capacity trade-offs and encoder adaptability, and further clarification of the task assumptions and potential extensions.
We would greatly appreciate your feedback on whether our responses have adequately addressed your concerns, or whether any questions remain. As the discussion deadline is approaching, we would be grateful if you could share any further questions or suggestions at your earliest convenience, allowing us to address them before the discussion period ends.
Your insightful suggestions have helped us evaluate POISE more comprehensively under different prototype availability and learning conditions, better characterize its robustness and capacity trade-offs, and clarify the scope and potential extensions of our work.
We sincerely appreciate your time and consideration.
Best regards,
The Authors
Response to Reviewer aYLb (1/2)
Dear Reviewer aYLb,
Thank you for your thoughtful review and constructive feedback. We particularly appreciate your recognition that CCIKD “fills a gap” in incremental keypoint detection, that our analysis “provides meaningful understanding for the community,” and that our comprehensive experiments “convincingly demonstrate effectiveness.” In response to your valuable suggestions, we have conducted additional analyses of prototype routing and detection performance under limited training observations, further clarified the accuracy–capacity trade-off and limitations of output identity expansion, discussed the challenges of end-to-end incremental keypoint detection, and elaborated on the influence of different prototype encoders.
W1+Q1. Robustness to Limited Prototype Observations
Following your suggestion, we conducted a prototype-sampling sensitivity analysis across 14 IKDD and MP100 incremental protocols. Prototypes are constructed using 1, 5, 20, or all available training images per category, with the same frozen DINOv3 encoder and routing configuration. Each setting independently constructs its routing history, while the detector is trained on the complete current-stage data.
For IKDD-6stage and MP100-10stage orders, we evaluate routing consistency relative to Full using Action Agreement (Align/Transfer/Expand decisions), Sharing Precision/Recall/F1 (cross-category identity-sharing relationships), and Old Member Overlap (overlap of old identity members when both settings select Align). We also report identity allocation, additional sharing absent in Full, and downstream performance using ViTPose with LwF under identical training settings.
These results validate an important design consideration of POISE: conservative identity allocation under unreliable prototype evidence. Through compatibility thresholds, matching margins, and consistency checks, POISE favors Expand over potentially inappropriate historical sharing when evidence is insufficient, mitigating the risk of incorrect parameter coupling at the cost of increased output capacity. As prototype observations increase, routing consistency and historical identity reuse improve while expansion decreases, with consistent capacity-reduction trends across all 14 protocols. Nevertheless, occasional sharing decisions absent in Full indicate that limited observations can still affect routing reliability.
Importantly, more prototype observations do not necessarily yield better detection performance. On MP100, 1-shot achieves higher AIAPCK and FinalPCK than Full but requires 1557 versus 825 identities, approaching All-Expand behavior. While performance declines from 1 to 20 observations, Full achieves stronger accuracy than 20-shot with fewer identities. These findings highlight POISE's ability to balance conservative independent adaptation and compatibility-guided knowledge reuse, rather than indiscriminately increasing parameter sharing.
We sincerely thank the reviewer for this insightful suggestion. The additional experiments have enabled us to evaluate POISE more comprehensively under varying prototype availability and incremental learning conditions, providing further insights into its routing behavior, robustness, and accuracy–capacity trade-offs. These analyses have strengthened our experimental validation and deepened our understanding of the method's effectiveness across different learning scenarios.