- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulates a new task named zero-shot skeleton temporal action segmentation and proposes a framework to address it. Specifically, it decomposes the human skeleton into different regions and phases, and computes joint motion energy to guide transferable action representation learning. It further constructs an atom codebook by incorporating learnable prototypes and a cross-attention mechanism for cross-modal alignment. Extensive experiments demonstrate that the proposed method can recognize and localize unseen actions.
Critical strengths
- This paper is the first to propose zero-shot skeleton temporal action segmentation, extending clip-level zero-shot skeleton action recognition to the frame level to recognize unseen action categories and localize their temporal boundaries. This extension fills a gap in skeleton-based action understanding, which should be appreciated.
- The experiments are comprehensive, with extensive qualitative and quantitative analyses. Meanwhile, the figures are intuitive and fruitful.
Critical weaknesses
- The core of temporal action segmentation lies in recognizing actions and identifying their temporal boundaries. However, the proposed method mainly focuses on recognizing unseen actions, overlooking how to model their temporal relationships and localize action segments, which I believe is essential for temporal action understanding tasks.
- The motivations behind several technical designs are not well explained. For example, the rationale and reasons for designing and remain unclear. Meanwhile, the compression mechanism in Section 3.2 is not sufficiently justified, raising concerns about the effectiveness of these designs. Additionally, the experimental section lacks empirical evaluations to validate them.
- The novelty of the cross-modal atom codebook is limited. In zero-shot skeleton action recognition, learnable prototypes (i.e., atoms) and cross-attention mechanisms have been widely explored. To some extent, this paper appears to directly apply similar techniques to the new task, lacking task-specific insights and designs.
- Numerous studies have explored RGB-based zero-shot temporal action detection and localization, but these works are not discussed in the paper. Additionally, comparing the proposed method only with zero-shot skeleton action recognition methods may not be sufficiently fair. A more appropriate comparison would involve existing RGB-based zero-shot temporal action detection and localization methods to skeleton inputs, thereby ensuring a fair comparison under the same input modality.
Questions for authors
- If Section 3.2 aims to guide the skeleton encoder toward learning structured motion features while avoiding shortcut learning induced by label annotations, why not adopt self-supervised learning techniques to achieve this?
- The proposed method is evaluated only on unseen actions, lacking a holistic evaluation of both seen and unseen actions, which would better align with the claimed real-world setting.
- Recent zero-shot skeleton action recognition methods (2026) are not discussed in the related work. Additionally, the calibration technique used during inference should be properly cited.