acceptodds
Under review as a conference paper at ICLR 2027

Exemplar-free Cross-Category Incremental Keypoint Detection: When New Categories Bring Heterogeneous Schemas

Abstract

Object categories often have heterogeneous keypoint schemas differing in keypoint number, order, and semantics, yet keypoint detectors are typically trained for fixed categories with predefined schemas, making continual expansion to new categories challenging. Under the standard top-down setting, we study exemplar-free Cross-Category Incremental Keypoint Detection (CCIKD). At each incremental stage, the model learns only from new categories while preserving previous keypoint detection performance. This setting exposes a fundamental output-space challenge: assigning a separate identity to every category-keypoint causes unbounded output growth, while shared heads reuse identities through fixed local indices, causing Output-Identity Collision (OIC), where incompatible keypoints are forced to share parameters. During continual updates, such collisions cause semantic ambiguity, leading to greater forgetting of previously learned knowledge. CCIKD therefore requires preserving old knowledge while organizing an evolving output space under heterogeneous schemas. To address these challenges, we propose POISE, prototype-guided Persistent Output Identity Space Expansion, which builds visual–geometric prototypes and an append-only identity bank. POISE selectively reuses compatible identities, transfers related output parameters for initialization, and expands the bank for unmatched keypoints, while decoupling output-identity organization from preservation objectives. Experiments on IKDD and MP100 across diverse orders, protocols, and continual-learning strategies demonstrate improved performance and reduced forgetting, with increasingly larger gains on longer and more heterogeneous streams. In particular, with EWC on MP100-8stage, POISE boosts FinalPCK from 45.51 to 70.66.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Incremental Learning, Keypoint Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aYLbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work formulates Exemplar-Free Cross Category Incremental Keypoint Detection (CCIKD), a new continual learning setting for keypoint detection. In CCIKD, models sequentially learn new object categories with heterogeneous keypoint schemas (varying keypoint count, order, and semantics) without access to old category training data. The authors identify Output Identity Collision (OIC) as a core failure mode for the widely used Shared KmaxK_{max} baseline: forcing semantically incompatible keypoints to share output heads via local index mapping introduces semantic ambiguity and amplifies catastrophic forgetting. To mitigate OIC, this paper proposes POISE, a prototype-guided persistent output-identity space expansion framework. POISE builds visual/geometric prototypes from training data and maintains an append-only identity bank. It supports three routing behaviors: reusing compatible existing identities (Historical Align), initializing new identities from related historical parameters (Transfer), or full random initialization (Expand). POISE decouples output identity organization from knowledge preservation objectives and is plug-and-play with diverse continual learning strategies, including EWC, LwF, and PODNet, as well as parameter-efficient methods L2P and InfLoRA. The authors construct benchmark IKDD and reconstruct MP100 into CCIKD-compliant incremental protocols across multiple category orders. Extensive experiments on ViTPose and HRNet backbones show consistent improvements over SharedKmaxK_{max}. Larger gains appear on longer, highly heterogeneous streams: for the MP100-8 stage with EWC, POISE lifts FinalPCK from 45.51 to 70.66. Comprehensive ablations validate prototype design choices, routing variants, encoder sensitivity, and capacity accuracy tradeoffs. The paper also discusses limitations such as prototype-dependent compatibility estimation, append-only output growth, and the top-down setting assumption.

Critical strengths
  1. The CCIKD setting fills a gap in incremental keypoint detection, where new categories arrive with fully heterogeneous keypoint schemas. The paper quantitatively diagnoses Output Identity Collision caused by naive local index output mapping, which explains why existing baselines suffer severe forgetting on heterogeneous streams. This analysis itself provides meaningful understanding for the community.

  2. POISE decouples output identity routing from continual learning preservation losses. It works off the shelf with regularization-based, distillation-based, and PEFT continual learning approaches and generalizes across two different pose backbones. Comprehensive benchmarks, multi-order protocols, and rich ablations convincingly demonstrate effectiveness, especially for long, highly heterogeneous category streams.

Critical weaknesses
  1. Cross-keypoint compatibility is purely estimated via visual-geometric prototypes, without any explicit semantic correspondence signal. When training instances for certain keypoints are scarce, prototype representations can be noisy and lead to suboptimal routing decisions. There is little analysis reporting routing failure cases.

  2. Although the authors report memory statistics and compare against the full expanded upper bound, POISE cannot prune obsolete identities. For extremely long real-world continual sequences, readout-layer parameter expansion may become a practical deployment barrier, and the paper does not propose mitigation for this limitation.

Questions for authors
  1. How does POISE behave when some category keypoints have very few training instances? Please provide supplementary analysis on the routing success rate under low-sample keypoint cases.

  2. In ablation, all expand yield the highest accuracy but huge parameter overhead. Have you explored simple post-hoc identity pruning strategies after routing to constrain output size, and what performance tradeoff did you observe?

  3. The evaluation assumes top-down input with known bounding boxes and category labels. Could you briefly discuss major obstacles if we want to migrate POISE to an end-to-end incremental keypoint detection pipeline without given category labels?

  4. In Table 31, the test multiple pre-training encoders for prototype construction. Is there any case where a better prototype encoder hurts downstream keypoint final performance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aYLb (1/2)

Dear Reviewer aYLb,

Thank you for your thoughtful review and constructive feedback. We particularly appreciate your recognition that CCIKD “fills a gap” in incremental keypoint detection, that our analysis “provides meaningful understanding for the community,” and that our comprehensive experiments “convincingly demonstrate effectiveness.” In response to your valuable suggestions, we have conducted additional analyses of prototype routing and detection performance under limited training observations, further clarified the accuracy–capacity trade-off and limitations of output identity expansion, discussed the challenges of end-to-end incremental keypoint detection, and elaborated on the influence of different prototype encoders.

W1+Q1. Robustness to Limited Prototype Observations

Following your suggestion, we conducted a prototype-sampling sensitivity analysis across 14 IKDD and MP100 incremental protocols. Prototypes are constructed using 1, 5, 20, or all available training images per category, with the same frozen DINOv3 encoder and routing configuration. Each setting independently constructs its routing history, while the detector is trained on the complete current-stage data.

For IKDD-6stage and MP100-10stage orders, we evaluate routing consistency relative to Full using Action Agreement (Align/Transfer/Expand decisions), Sharing Precision/Recall/F1 (cross-category identity-sharing relationships), and Old Member Overlap (overlap of old identity members when both settings select Align). We also report identity allocation, additional sharing absent in Full, and downstream performance using ViTPose with LwF under identical training settings.

ProtocolSamplesAction agreementSharing P/R/F1Old OverlapOld AlignTransfer identitiesExpand identitiesAdditional sharing absent in FullFinal identitiesAIAPCKFinalPCKBWTPCK
IKDD144.71N/A/0/0N/A00102N/A(0 shared pairs)10287.4483.31−6.43
549.4150.00/6.12/10.9150.00613833/6(50.00%)9687.6084.70−5.89
2061.1876.47/26.53/39.3964.291525624/17(23.53%)8787.5984.52−6.33
Full100.00100/100/100100.0039855—6387.8385.26−5.35
MP100-10stage128.6725.00/0.13/0.25N/A2815496/8(75.00%)155781.5174.14−18.81
544.1725.84/7.43/11.5434.8897267946333/449(74.16%)121379.6673.75−20.87
2056.7739.74/29.39/33.7963.73321333591696/1155(60.26%)92478.3270.46−21.05
Full100.00100 /100/100100380362463—82580.6373.23−19.27

These results validate an important design consideration of POISE: conservative identity allocation under unreliable prototype evidence. Through compatibility thresholds, matching margins, and consistency checks, POISE favors Expand over potentially inappropriate historical sharing when evidence is insufficient, mitigating the risk of incorrect parameter coupling at the cost of increased output capacity. As prototype observations increase, routing consistency and historical identity reuse improve while expansion decreases, with consistent capacity-reduction trends across all 14 protocols. Nevertheless, occasional sharing decisions absent in Full indicate that limited observations can still affect routing reliability.

Importantly, more prototype observations do not necessarily yield better detection performance. On MP100, 1-shot achieves higher AIAPCK and FinalPCK than Full but requires 1557 versus 825 identities, approaching All-Expand behavior. While performance declines from 1 to 20 observations, Full achieves stronger accuracy than 20-shot with fewer identities. These findings highlight POISE's ability to balance conservative independent adaptation and compatibility-guided knowledge reuse, rather than indiscriminately increasing parameter sharing.

We sincerely thank the reviewer for this insightful suggestion. The additional experiments have enabled us to evaluate POISE more comprehensively under varying prototype availability and incremental learning conditions, providing further insights into its routing behavior, robustness, and accuracy–capacity trade-offs. These analyses have strengthened our experimental validation and deepened our understanding of the method's effectiveness across different learning scenarios.

AuthorsOpenReview ↗

Response to Reviewer aYLb (2/2)

W2+Q2. Output Capacity Growth and Identity Pruning

POISE controls output growth through compatibility-guided selective identity reuse rather than post-hoc pruning. As described in Appendix B, old category–keypoint mappings remain fixed to preserve output correspondences, while their prediction parameters remain trainable. Thus, persistence constrains identity assignments, not parameter optimization or output capacity. The original Table 5 (Table 6 of the revised paper) demonstrates a favorable accuracy–capacity trade-off: All-Expand achieves 82.05 AIAPCK with 1565 identities, whereas POISE achieves 81.78 with only 825, representing 47.3% fewer output identities and readout-layer parameters for just 0.27 points lower accuracy.

Since all old categories remain in the evaluation, directly pruning identities still referenced by old keypoints would disrupt established mappings. Safe pruning or merging requires identity reassignment while preserving old prediction functions, particularly challenging without old images. As an initial systematic study of the challenging CCIKD setting, our focus is on limiting unnecessary output expansion while preserving old correspondences. We appreciate the reviewer's suggestion, which aligns with Appendix E.1. Future work will explore resource-constrained identity consolidation, safe pruning, and identity migration to balance long-term capacity efficiency and old knowledge retention.

Q3. Extending POISE to End-to-End Incremental Keypoint Detection

Our evaluation follows the established top-down setting used by HRNet, ViTPose, and CAPE methods such as POMNet, CapeFormer, and SCAPE. CCIKD thus assumes known bounding boxes and category labels, where category information determines the keypoint schema and persistent output-identity mapping, allowing us to focus on heterogeneous incremental keypoint learning and old knowledge retention.

Extending POISE to a fully end-to-end incremental pipeline without given category labels presents three main challenges:

  1. Category recognition and schema selection. The model must jointly recognize categories and determine their keypoint numbers, semantics, and output routes. Incorrect category predictions may lead to inappropriate schema selection.
  2. Joint old knowledge preservation. Incrementally learning object detection, classification, and keypoint prediction without old images introduces forgetting across multiple components, which persistent output routing alone cannot address.
  3. Detection–keypoint coupling. Predicted bounding boxes introduce localization errors and scale variations that affect keypoint features and geometric relationships, requiring reliable associations between instances and their heterogeneous keypoint outputs.

To the best of our knowledge, this is the first systematic study of CCIKD, focusing on heterogeneous cross-category incremental keypoint learning and old retention. We appreciate the reviewer's suggestion, which aligns with the potential extension discussed in Appendix E.3. A promising direction is to integrate POISE with class-incremental object detection, enabling category-aware schema selection from predicted instances while jointly preserving old detection and keypoint prediction capabilities.

Q4. Impact of Prototype Encoder Quality on Detection Performance

We examined whether POISE's gains primarily stem from DINOv3 and how prototype encoder quality affects detection. To investigate this, we evaluated three prototype encoders—MAE, CLIP, and DINOv3—in original Table 31 (Table 33 of the revised paper). DINOv3 achieves the highest FinalPCK on both representative protocols, outperforming MAE and CLIP, with consistent trends in AIAPCK and BWTPCK.

POISE outperforms Shared-Kmax with all three encoders, showing that its gains are not solely attributable to DINOv3. Frozen prototypes provide visual–geometric compatibility evidence for output routing rather than directly participating in heatmap prediction. Routing quality depends on visual representations, geometric evidence, member-level compatibility, matching margins, and consistency constraints. Therefore, stronger general-purpose representations do not necessarily guarantee better routing or downstream detection performance. These results support the effectiveness and encoder adaptability of POISE's persistent output-identity organization.

We have incorporated the above additional experimental analyses into Appendix D.10 of the revised manuscript. We sincerely thank the reviewer for the constructive feedback and hope these revisions adequately address the concerns raised. If the reviewer has any further questions or concerns, we are willing to provide additional details. Thank you again for your time and thoughtful feedback.

Dear Reviewer aYLb,

Thank you again for your thoughtful review and constructive feedback. Last week, we provided point-by-point responses to your concerns and questions, together with additional experiments on prototype-sampling sensitivity and routing stability, analyses of accuracy–capacity trade-offs and encoder adaptability, and further clarification of the task assumptions and potential extensions.

We would greatly appreciate your feedback on whether our responses have adequately addressed your concerns, or whether any questions remain. As the discussion deadline is approaching, we would be grateful if you could share any further questions or suggestions at your earliest convenience, allowing us to address them before the discussion period ends.

Your insightful suggestions have helped us evaluate POISE more comprehensively under different prototype availability and learning conditions, better characterize its robustness and capacity trade-offs, and clarify the scope and potential extensions of our work.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_54dJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes POISE for exemplar-free cross-category incremental keypoint detection, where a model progressively learns new categories with different keypoint counts, semantics, and orderings while retaining its detection capabilities for previously learned categories. The authors identify a challenge arising from heterogeneous keypoint schemas: sharing output channels according to within-category indices may force semantically different keypoints to share parameters, increasing interference during incremental learning. POISE assesses cross-category keypoint compatibility using visual and geometric prototypes and employs three operations-Align, Transfer, and Expand-to selectively reuse output identities, initialize new identities through parameter transfer, or expand the output identity space. The method decouples output-space organization from knowledge preservation and can be combined with EWC, LwF, PODNet, and parameter-efficient continual learning methods. Experiments on the IKDD and MP100 incremental settings demonstrate the effectiveness of POISE.

Critical strengths
  • The paper highlights an important issue in heterogeneous keypoint learning: within-category keypoint indices do not represent consistent semantics across categories, and fixed-index sharing may introduce additional learning interference. This offers a valuable perspective on the problem.

  • Using prototype compatibility, POISE employs Align, Transfer, and Expand to support identity sharing, transfer-based initialization, and independent learning, respectively. It enables output-space expansion while preserving existing mappings, providing a flexible mechanism for cross-category knowledge reuse and semantic separation.

  • As a plug-and-play module, POISE can be combined with regularization-based, knowledge-distillation-based, and parameter-efficient continual learning strategies. Its effectiveness is demonstrated across multiple incremental protocols.

Critical weaknesses
  • The paper frames the use of the same local indices for keypoints from different categories as output-identity collision. However, sharing output parameters across semantically different keypoints does not necessarily create conflicting prediction tasks. A clearer distinction is needed between index-level semantic mismatch, parameter interference during optimization, and catastrophic forgetting more generally.

  • POISE determines whether to share output identities using local visual prototypes from a frozen DINOv3 encoder and geometric statistics, whereas actual predictions rely on continually updated ViTPose or HRNet features. Similar local appearance and spatial distributions do not necessarily imply that two keypoints should share the same heatmap prediction parameters. Conversely, differences between prototypes do not necessarily imply that sharing will cause negative transfer.

  • Historical Align has a gap in its consistency checks among newly added members, and fixed routing may preserve incorrect sharing decisions over subsequent stages. According to the algorithm described in Appendix B.4.2, keypoints from different new categories within the same stage can independently pass the checks and be assigned to the same historical identity. All decisions are based on a frozen historical snapshot, and members accepted during the current stage are not incorporated into subsequent compatibility evaluations. Therefore, two new keypoints being individually compatible with a historical identity does not imply that they are compatible with each other.

  • The main experiments primarily evaluate POISE with classic general-purpose continual learning methods. The results provide substantial support for its improvement over a fixed-index shared output head, but comparisons with recent keypoint learning and heterogeneous output modeling methods under matched incremental protocols remain insufficient.

Questions for authors
  • The current incompatibility load is primarily defined through prototype similarity and does not directly measure actual training interference. Please verify whether output-identity mismatch independently increases forgetting on old categories by holding model capacity and the knowledge-preservation strategy fixed and changing only the correspondence between keypoints and output channels.

  • POISE selects Align or Transfer based on visual and geometric prototype compatibility, but prototype similarity does not necessarily imply that sharing prediction parameters is more beneficial. Please provide an analysis of the routing decisions to verify whether higher-compatibility matches indeed achieve better overall performance on old and new categories under Align than under Transfer.

  • As currently described, Historical Align evaluates different new categories independently against a frozen historical snapshot, allowing multiple new keypoints to match the same historical identity. Please check the mutual compatibility of these newly added members to prevent individually accepted matches from forming an internally inconsistent sharing group.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 54dJ (1/2)

Dear Reviewer 54dJ,

Thank you for your thoughtful review. We appreciate your recognition that heterogeneous keypoint learning presents a “valuable perspective,” that POISE provides a “flexible mechanism” for knowledge reuse. In response, we have clarified the relationship between output mismatch, parameter interference, and forgetting; conducted controlled correspondence-mismatch and routing-action interventions; audited old identity consistency; and expanded comparisons with recent methods.

W1+Q1. Clarifying Output-Identity Collision

As suggest, we conducted controlled correspondence-mismatch experiments on IKDD-6stage and MP100-10stage, using ViTPose with LwF. We compare POISE with Fixed-Template Mismatch (perturbs old Align targets while preserving the original routing-action structure and stage-wise output capacity) and Re-routed Mismatch (allows subsequent routing to evolve from the perturbed identity memory). OldmPCK evaluates all old categories, while Affected-oldmPCK evaluates only those associated with perturbed correspondences.

RoutingIKDD AIAPCKFinalPCKBWTPCKOldmPCKAffected-oldmPCKMP100 AIAPCKFinalPCKBWTPCKOldmPCKAffected-oldmPCK
Fixed-Template Mismatch78.3372.70−20.0269.0364.81(4 classes)76.3667.07−25.4164.3661.54(74 classes)
Re-routed Mismatch78.9679.97−18.6670.6052.26(2 classes)77.3268.81−23.5066.2863.69(70 classes)
POISE87.8385.26−5.3582.9683.4680.6373.23−19.2771.1071.62

The results shows that inappropriate output correspondences can significantly aggravate old forgetting and degrade overall performance. Affected-oldmPCK is lower than overall OldmPCK, suggesting that performance degradation is particularly pronounced among old categories affected by incorrect correspondences. Fixed-Template Mismatch reduces OldmPCK, despite preserving output capacity and routing-action structure. This is consistent with the matched-capacity comparison in original Table 29 (Table 31 of the revised paper), where POISE outperforms Matched-Random.

POISE addresses potential output-correspondence mismatch through compatibility-guided identity routing. Align selectively reuses compatible old identities, and Transfer and Expand allocate independent identities when direct sharing is supported. This design follows our analysis in Sec.3: Figures 2–3 reveal the unreliability of index-based correspondence under heterogeneous schemas, and Eqs.4–5 characterize the associated structural incompatibility risk rather than directly measuring optimization interference or forgetting. Our objective is to proactively avoid potentially conflicting parameter sharing, rather than assume that all sharing inevitably causes interference. These results confirm that output-identity mismatch can increase old categories forgetting, while POISE is designed to mitigate this risk through compatibility-guided identity organization.

W2+Q2. Prototype Compatibility and the Roles of Align and Transfer

As suggest, we conducted controlled routing-action interventions on IKDD-6stage and MP100-10stage, using ViTPose with LwF. We compare POISE with Transfer→Align (forcing historical identity sharing), Align→Transfer (replacing shared identities with independently trainable, old initialized identities), and Align↔Transfer (applying both conversions).

RoutingIKDD AIAPCKFinalPCKBWTPCKId.MP100 AIAPCKFinalPCKBWTPCKId.
Transfer→Align85.7883.40-7.145578.2671.73-26.80463
Align→Transfer87.5884.46-6.2710283.7278.56-12.741205
Align↔Transfer86.5984.80-5.799479.2872.39-21.05843
POISE87.8385.26−5.356380.6373.23−19.27825

These interventions reveal the utility of compatibility-guided routing. Forcing Transfer→Align degrades performance on both benchmarks, suggesting that insufficiently supported sharing can be harmful. Align→Transfer slightly reduces performance on IKDD but improves MP100, at the cost of substantially more output identities. Align↔Transfer underperforms POISE on MP100 despite comparable output capacity. These findings reveal the importance of selectively balancing parameter sharing and independent adaptation. POISE uses prototype compatibility to guide this balance. Align reuses old identities when compatibility supports sharing, while Transfer initializes independent identities from related old parameters when sharing is less certain. The original Table 5 (Table 6 of the revised paper) supports that POISE outperforms Align-Expand and approaches All-Expand with substantially fewer identities.

The DINOv3 encoder offers a consistent cross-stage representation space, while the continually updated ViTPose/HRNet backbone learns the prediction functions. This separation avoids keypoint-model representation drift in prototype comparisons, especially when old images are unavailable for recomputing old prototypes.

AuthorsOpenReview ↗

Response to Reviewer 54dJ (2/2)

Favoring either greater parameter sharing (Align) or independent adaptation (Transfer) involves performance–capacity trade-offs, whereas compatibility-guided selective routing provides a more balanced solution. Although higher prototype compatibility does not guarantee that Align will outperform Transfer for every keypoint, it provides useful evidence for deciding when to share prediction parameters or enable independent adaptation. Our results support POISE's compatibility-guided routing as an effective mechanism for balancing historical knowledge retention, new-category adaptation, and controlled output growth.

W3+Q3. Pairwise Consistency of old Align

As suggest, we conducted a pairwise consistency audit of the original old Align routing assignments. For each old identity receiving multiple new keypoints within the same stage, we directly evaluated the compatibility of every pair of new members using their prototypes and the corresponding routing criteria. The audit covers 15 incremental category orders and 92 stages. Across the three MP100-10stage orders, all 173 examined pairs satisfy the compatibility criteria. Across all 11 MP100 protocols, only 1 of 398 pairs falls slightly below the threshold. On MegaKPT, only 9 of 3056 pairs fail the corresponding check. IKDD introduces one category per stage, so no eligible same-stage pairs arise under category-wise injectivity.

DatasetProtocolsExamined pairsIncompatible pairsConsistency rate
IKDD30——
MP100113981(0.25%)99.75%
MegaKPT130569(0.29%)99.71%

These results demonstrate that incompatible co-assignments are extremely rare and that old Align maintains high empirical consistency across the evaluated incremental streams. Although Align does not explicitly impose pairwise constraints among new members, its aggregate and member-level compatibility, coverage, matching margins, and consistency checks effectively limit incompatible sharing in practice. Together with our continual-learning results, these findings further support the practical robustness of POISE's compatibility-guided routing mechanism for historical knowledge retention and persistent output-identity organization.

W4. Comparisons with Recent Methods

Following your suggestion, we expanded our comparisons with recent keypoint detection and heterogeneous output modeling methods. We adapted GKDT to the IKDD-6stage and MP100-10stage CCIKD protocols under consistent category orders, incremental data splits, cumulative evaluation sets, and metrics. We consider GKDT-I, which independently trains each stage from the same pretrained checkpoint, and GKDT-S, which sequentially updates the previous-stage model. Both have no old exemplar replay and use prompt-conditioned inference. Our original Appendix D.1 and Table 18 (Table 19 of the revised paper) also report CapeFormer-I/S and SCAPE-I/S under the same protocols.

MethodIKDD AIAPCKFinalPCKBWTPCKMP100 AIAPCKFinalPCKBWTPCK
CapeFormer-I43.9935.75−24.6553.7246.48−27.15
CapeFormer-S56.2154.19−14.0768.2067.18−15.69
SCAPE-I57.1346.09−32.6658.0148.27−35.35
SCAPE-S68.1661.94−20.4571.4166.64−20.57
GKDT-I26.0126.98-6.6421.2422.28-26.98
GKDT-S54.2346.79-6.5057.2566.70-22.64
Shared86.0483.15−7.0277.0269.36−23.25
POISE87.9786.19−4.2281.7875.07−17.21

POISE achieves higher accuracy than GKDT, CapeFormer, SCAPE, and Shared-Kmax under both CCIKD protocols, providing additional evidence for the effectiveness of persistent output-identity organization and old knowledge retention. Importantly, CCIKD addresses a complementary challenge to general and prompt-conditioned keypoint detection. While these methods primarily emphasize open-category generalization through prompts or support examples, such capabilities do not necessarily ensure old knowledge retention during sequential updates. CCIKD instead focuses on continually learning heterogeneous new categories while preserving old capabilities, without exemplar replay or additional test-time keypoint prompts. These cross-paradigm comparisons therefore complement our controlled experiments on continual-learning strategies and output-organization mechanisms.

We have incorporated the above explanations, controlled comparisons, and additional experimental analyses into Appendices D.5, D.7, D.8, and D.9 of the revised manuscript. We sincerely thank the reviewer for the constructive feedback and hope these revisions adequately address the concerns raised. If the reviewer has any further questions or concerns, we are willing to provide additional details. Thank you again for your time and thoughtful feedback.

Dear Reviewer 54dJ,

Thank you again for your thoughtful review and constructive feedback. Last week, we provided point-by-point responses to your concerns and questions, together with additional controlled experiments on output-correspondence mismatch and routing decisions, pairwise consistency analyses, and expanded comparisons with recent methods.

We would greatly appreciate your feedback on whether our responses have adequately addressed your concerns, or whether any questions remain. As the discussion deadline is approaching, we would be grateful if you could share any further questions or suggestions at your earliest convenience, allowing us to address them before the discussion period ends.

Your valuable suggestions have helped us clarify the motivation behind output-identity organization, better understand the roles of compatibility-guided routing, and provide more comprehensive empirical evidence for old knowledge retention.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_ACJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Object categories usually have heterogeneous keypoint schemas with different numbers of keypoints and appearances. Thus, continual learning of keypoints on those objects are challenging. This paper targets at the examplar-free incremental keypoint detection, and proposes the method called POISE, prototype-guided Persistent Output Identity Space Expansion, to address the problem. The key in the method is to build visual-geometric prototypes and an append-only identity bank. The results on two benchmarks IKDD and MP100 show the effectiveness of the method.

Critical strengths
  1. The problem that this paper targets, namely, incremental keypoint detection, is interesting and has to-some-extent potential of applications.

  2. The experimental results are well organized for the presentation of the paper.

Critical weaknesses
  1. For intro, the claim that "keypoint detectors are typically trained on fixed categories..." may be outdated, as now the keypoint detectors are no longer trained on fixed categories. The general keypoint detector [a], few-shot and zero-shot keypoint detectors [b-e] are supporting open-set categories. The works of recent progress happened in keypoint detection field should be included in intro.

  2. The proposed method is based on building prototypes, while many works already use prototypes [f, g, b]. The reviewer expects more novel contribution in incremental keypoint detection. Moreover, the difference between this work and the existing continual learning works should be described to distinguish the novelty of the method.

  3. MP100 is relatively small-scale. The continual learning benchmark should be developed in a dataset with more diverse categories and larger-scale data capacity such as MegaKPT [a]. Reporting the results on MegaKPT dataset is required.

  4. In addition to examplar-free methods, the general keypoint detection methods (for example, the GKDT [a]) also support continual learning. The results of those examplar-dependent methods and zero-shot methods should be reported in Table 2 as well, to compare the performance between different types of methods.

[a] GKDT: General Keypoint Detection Transformer @ ECCV 2026

[b] Few-shot Keypoint Detection with Uncertainty Learning for Unseen Species @ CVPR 2022

[c] OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection @ ECCV 2024

[d] Detect Any Keypoints: An Efficient Light-Weight Few-Shot Keypoint Detector @ AAAI 2024

[e] Exploiting Class-agnostic Visual Prior for Few-shot Keypoint Detection @ IJCV

[f] Meta-point learning and refining for category-agnostic pose estimation @ CVPR 2024

[g] CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose @ CVPR 2023

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer ACJT (1/2)

Dear Reviewer ACJT,

Thank you for your constructive review. We particularly appreciate your recognition that incremental keypoint detection is an "interesting" problem with "potential of applications," and that our experimental results are "well organized for the presentation." In response to your valuable suggestions, we have further clarified the novelty of CCIKD, expanded our evaluation to the larger-scale MegaKPT benchmark, and added comparisons with recent methods, including GKDT.

W1. Updating the Introduction to Reflect Recent Advances

The original Introduction primarily focused on continual-learning challenges, while Related Work discusses category-agnostic methods (POMNet and CapeFormer) and visual/text-prompted approaches (X-Pose and GKDT). Following your suggestion, we have revised the opening paragraph of the Introduction in our revised manuscript to acknowledge these advances more explicitly and clarify their relationship to CCIKD.

Our original statement was intended to describe conventional keypoint detectors trained on fixed categories with predefined schemas, rather than all existing methods. Building upon architectures such as ViTPose and HRNet, CCIKD addresses a complementary challenge to open-category generalization: continually learning categories with heterogeneous keypoint schemas while preserving old knowledge without access to previous images. Beyond catastrophic forgetting, this requires maintaining persistent output correspondences as heterogeneous schemas evolve. Our focus is therefore on old knowledge retention and heterogeneous output-identity organization

Our original Appendix D.1 and Table 18 also include comparisons with support-conditioned CAPE methods under consistent CCIKD evaluation protocols. Following your suggestion, we have revised the opening paragraph of the Introduction in the revised manuscript (L37-L77) to explicitly acknowledge recent keypoint detection advances, refine the original fixed-category statement, and clarify the complementary motivation of CCIKD.

W2. Further Clarification of Our Novelty

Our primary contributions lie in introducing a new continual-learning setting—exemplar-free Cross-Category Incremental Keypoint Detection (CCIKD)—and developing a persistent output-identity organization mechanism for evolving heterogeneous keypoint schemas. Visual–geometric prototypes serve as a tool for estimating cross-keypoint compatibility. To the best of our knowledge, this is the first systematic study of jointly evolving object categories and heterogeneous keypoint schemas without old images.

  • CCIKD addresses a distinct challenge beyond catastrophic forgetting. Unlike incremental classification, detection, and segmentation, heterogeneous keypoint schemas lack predefined cross-category output correspondences. Existing incremental keypoint methods consider more restricted settings: Nayak et al. use aligned schemas and exemplar memory; IKL expands keypoints within existing domains; and PoseAdapt focuses on continual human pose adaptation. CCIKD instead requires maintaining persistent output correspondences alongside old knowledge retention as both categories and keypoint schemas evolve.
  • POISE differs from existing keypoint representation-based methods in how representations are used.
MethodInputOutputPrimary Purpose
FSKDFew-shot annotated support and query imagesLearned keypoint features and similarity/uncertainty estimatesFew-shot localization and uncertainty modeling
MetaPointImage features and annotated support keypointsLearnable meta-point embeddings and refined keypoint coordinatesSupport-conditioned keypoint prediction and refinement
CLAMPAnimal image features and textual keypoint promptsAligned visual–language representations and keypoint presence mapsCross-modal alignment and heatmap prediction
OursCurrent-stage annotated keypoints and old prototype memoryFrozen visual–geometric compatibility evidence and persistent identity mappingsStage-wise persistent output-identity routing via Align, Transfer, and Expand

Unlike representations used directly for localization or prediction, POISE uses frozen prototypes solely as compatibility evidence for persistent output routing, without participating in heatmap prediction or introducing prototype-learning objectives. The key innovation thus lies in leveraging prototype compatibility to guide persistent output-identity organization across incremental stages.

  • Experiments validate this mechanism. Tables 2–4 demonstrate improvements across continual-learning strategies and backbones. In Table 5, POISE outperforms Matched-Random at equal capacity and approaches All-Expand with far fewer identities. Table 31 further confirms improvements over Shared-Kmax using MAE, CLIP, or DINOv3 prototypes, supporting the effectiveness and encoder adaptability of persistent output-identity organization.
AuthorsOpenReview ↗

Response to Reviewer ACJT (2/2)

W3. Larger-Scale Evaluation on MegaKPT

Following your suggestion, we extended our evaluation to the MegaKPT benchmark by constructing a MegaKPT-10stage CCIKD protocol. Following the super-category order in Fig. 2 of the original GKDT paper, we sequentially introduce Clothes, Human pose, Animal pose, Vehicle, Human face, Human limbs, Animal face, Furniture, Insect pose, and Medical SC. Each stage introduces a new super-category, with training restricted to current-stage data and testing covering all seen categories. Given the substantially larger scale and the limited rebuttal time, we evaluate an order using ViTPose with LwF.

MethodAIAPCKFinalPCKBWTPCK
Shared-Kmax39.3028.11−48.86
POISE53.4240.92−39.03
Δ+14.12+12.81+9.83

POISE substantially outperforms Shared-Kmax, extending the effectiveness of persistent output-identity organization to a larger incremental learning scenario.

Regarding our original benchmark selection, we prioritized systematic and controlled evaluation of the newly formulated CCIKD setting. IKDD provides a low-heterogeneity benchmark with six categories sharing an aligned 17-keypoint schema, while MP100 covers 100 categories with heterogeneous keypoint numbers, orderings, and semantics. We reconstructed MP100 into CCIKD protocols, whose manageable scale and diverse schemas enabled comprehensive evaluations across incremental protocols, category orders, continual-learning strategies, backbones, and routing ablations, allowing us to isolate the contribution of persistent output-identity organization.

We also considered MegaKPT during benchmark development. However, its substantially larger scale and imbalanced data distributions introduce additional challenges related to stage-wise training budgets, data availability, and long-tailed learning, making controlled analysis of output organization more difficult. We therefore prioritized MP100 for detailed and systematic analysis. Following your suggestion, we also extended our evaluation to MegaKPT, with the additional results further supporting POISE's effectiveness in CCIKD.

W4. Comparisons with Recent Methods

Following your suggestion, we additionally evaluated GKDT under the IKDD-6stage and MP100-10stage CCIKD protocols, using consistent incremental protocol. We considered two variants: GKDT-I, which independently trains each stage from the same pretrained checkpoint, and GKDT-S, which sequentially trains from the previous-stage model. Both have no old image replay and use prompt-conditioned inference. Our original submission also includes comparisons with support-conditioned methods capable of handling heterogeneous keypoint schemas. Appendix D.1 and Table 18 report CapeFormer and SCAPE under the same protocols.

MethodIKDD-6 AIAPCKFinalPCKBWTPCKMP100-10 AIAPCKFinalPCKBWTPCK
CapeFormer-I43.9935.75−24.6553.7246.48−27.15
CapeFormer-S56.2154.19−14.0768.2067.18−15.69
SCAPE-I57.1346.09−32.6658.0148.27−35.35
SCAPE-S68.1661.94−20.4571.4166.64−20.57
GKDT-I26.0126.98-6.6421.2422.28-26.98
GKDT-S54.2346.79-6.5057.2566.70-22.64
Shared86.0483.15−7.0277.0269.36−23.25
POISE87.9786.19−4.2281.7875.07−17.21

These results highlight the importance of explicitly addressing old knowledge retention and heterogeneous output organization. POISE achieves higher accuracy than CapeFormer, SCAPE, GKDT, and Shared-Kmax under both CCIKD protocols, providing broader evidence for the effectiveness of persistent output-identity organization.

CCIKD addresses a complementary challenge to general and prompt-conditioned keypoint detection. While these methods primarily focus on open-category generalization through prompts or support examples, real-world incremental scenarios may introduce new categories in small groups with limited visual diversity, making continual adaptation particularly challenging without old training images. Moreover, prompt-based keypoint localization does not necessarily guarantee the preservation of previously learned capabilities during sequential updates. CCIKD specifically addresses this challenge by continually learning new categories with heterogeneous keypoint schemas while retaining old knowledge, without exemplar replay or additional test-time keypoint prompts. These cross-paradigm comparisons thus complement our controlled continual-learning experiments and further clarify the distinct contributions of POISE.

We have incorporated the above explanations, controlled comparisons, and additional experimental analyses into Appendices D.6-D.7 of the revised manuscript. We sincerely thank the reviewer for the constructive feedback and hope these revisions adequately address the concerns raised. If the reviewer has any further questions or concerns, we are willing to provide additional details. Thank you again for your time and thoughtful feedback.

Thanks for the authors' response.

For ``W2. Further Clarification of Our Novelty'', the detailed comparison with existing works such as FSKD [b,c,d,e], MetaPoint [f], CLAMP [g], is still missing in the revised paper. The proposed setting should compare with existing settings to reflect its difference. Moreover, for the experiments in main paper, the results on MegaKPT benchmark and the comparison between examplar-free and examplar-dependent methods are also missing. The revision should be made in main paper, instead of in appendix or openreview.

Comments on the paper

1 comment

Dear Authors and Reviewers (54dJ and aYLb),

I would like to commend the authors for their detailed point-by-point responses, and I thank Reviewer ACjT for engaging in the follow-up discussion.

To Reviewers 54dJ and aYLb: As we approach the end of the discussion period, the authors have provided specific responses to your questions. I kindly request that you take a moment to review their responses and leave a brief note on OpenReview.

A simple confirmation of whether your concerns have been addressed and if your score remains the same is completely sufficient. If I do not receive a follow-up before the discussion period closes, I will proceed with my evaluation based on the original reviews and the authors' written responses.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.