acceptodds
Withdrawn from ICLR 2027

Zero-Shot Skeleton-Based Temporal Action Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human action segmentation in continuous skeleton streams relies heavily on closed-set annotations, struggling to generalize to novel categories in the wild. While zero-shot learning (ZSL) offers a promising paradigm, existing skeleton-based ZSL methods are strictly confined to clip-level recognition. By encoding entire actions into global semantic prototypes, they inherently erase the fine-grained, localized motion primitives shared between seen and unseen categories, rendering them ineffective for dense frame-level segmentation. In this paper, we formalize and tackle the novel task of zero-shot skeleton-based temporal action segmentation. We propose SkelAtom, a framework that achieves compositional generalization by explicitly decomposing untrimmed actions into sparse combinations of shared motion atoms. To ensure these atoms capture transferable physical movements, we introduce Kinematic Anchor Grounding (KAG), which supervises the latent space with class-agnostic kinematic descriptors derived from body-part dynamics. Furthermore, a Cross-Modal Atom Codebook aligns both visual frames and textual prototypes into the same atomic basis, enabling visual knowledge learned from seen actions to transfer to unseen categories. We establish four zero-shot benchmarks on the MCFS-130 and PKU-MMD datasets. Experimental results show that SkelAtom achieves state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
skeleton-based temporal action segmentation, zero-shot learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BgWrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates a new task named zero-shot skeleton temporal action segmentation and proposes a framework to address it. Specifically, it decomposes the human skeleton into different regions and phases, and computes joint motion energy to guide transferable action representation learning. It further constructs an atom codebook by incorporating learnable prototypes and a cross-attention mechanism for cross-modal alignment. Extensive experiments demonstrate that the proposed method can recognize and localize unseen actions.

Critical strengths
  1. This paper is the first to propose zero-shot skeleton temporal action segmentation, extending clip-level zero-shot skeleton action recognition to the frame level to recognize unseen action categories and localize their temporal boundaries. This extension fills a gap in skeleton-based action understanding, which should be appreciated.
  2. The experiments are comprehensive, with extensive qualitative and quantitative analyses. Meanwhile, the figures are intuitive and fruitful.
Critical weaknesses
  1. The core of temporal action segmentation lies in recognizing actions and identifying their temporal boundaries. However, the proposed method mainly focuses on recognizing unseen actions, overlooking how to model their temporal relationships and localize action segments, which I believe is essential for temporal action understanding tasks.
  2. The motivations behind several technical designs are not well explained. For example, the rationale and reasons for designing ρi\rho_{i} and ηi\eta_{i} remain unclear. Meanwhile, the compression mechanism in Section 3.2 is not sufficiently justified, raising concerns about the effectiveness of these designs. Additionally, the experimental section lacks empirical evaluations to validate them.
  3. The novelty of the cross-modal atom codebook is limited. In zero-shot skeleton action recognition, learnable prototypes (i.e., atoms) and cross-attention mechanisms have been widely explored. To some extent, this paper appears to directly apply similar techniques to the new task, lacking task-specific insights and designs.
  4. Numerous studies have explored RGB-based zero-shot temporal action detection and localization, but these works are not discussed in the paper. Additionally, comparing the proposed method only with zero-shot skeleton action recognition methods may not be sufficiently fair. A more appropriate comparison would involve existing RGB-based zero-shot temporal action detection and localization methods to skeleton inputs, thereby ensuring a fair comparison under the same input modality.
Questions for authors
  1. If Section 3.2 aims to guide the skeleton encoder toward learning structured motion features while avoiding shortcut learning induced by label annotations, why not adopt self-supervised learning techniques to achieve this?
  2. The proposed method is evaluated only on unseen actions, lacking a holistic evaluation of both seen and unseen actions, which would better align with the claimed real-world setting.
  3. Recent zero-shot skeleton action recognition methods (2026) are not discussed in the related work. Additionally, the calibration technique used during inference should be properly cited.
Reviewer_Qer2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies zero-shot temporal action segmentation in continuous skeleton sequences, where the action categories evaluated at test time are absent from training. The proposed framework, SkelAtom, combines a skeleton encoder with frozen text embeddings and introduces two main components. Kinematic Anchor Grounding (KAG) regresses class-agnostic static and dynamic body-part descriptors from segment-averaged skeleton features. A shared cross-modal atom codebook expresses visual frames and textual class prototypes through sparse atom combinations, injects the resulting representations into the original features through residual connections, and aligns segment-averaged visual combination weights with the corresponding textual weights.

The authors construct four zero-shot evaluation settings on MCFS-130 and PKU-MMD. They report improvements over adapted recognition and localization baselines in frame accuracy, edit score, and segmental F1, together with component ablations, sensitivity analyses, qualitative visualizations, and supplementary analyses of the descriptors and codebook.

Critical strengths
  1. A useful problem formulation with evaluation across complementary settings.

Extending skeleton-based zero-shot learning to continuous, frame-level action segmentation is practically meaningful because recognizing an unseen action category and recovering its temporal extent impose different requirements. The four settings cover different numbers of unseen categories, subject changes, and viewpoint changes across two datasets. This evaluation framework is a useful contribution even if some methodological components build on existing ideas.

  1. A coherent framework with promising empirical results and relevant ablations.

The combination of segment-level kinematic supervision and shared cross-modal representations is technically motivated. The reported improvements extend across all four evaluation settings; for example, on MCFS-130 U26, AvgF1 improves from the strongest listed baseline’s 29.15 to 32.94. Tables 3–5 examine component contributions, correct versus shuffled kinematic targets, and codebook design choices. These experiments provide evidence that the components are useful within the proposed system and support further investigation of the approach.

Critical weaknesses
  1. The novelty of the shared atom-codebook mechanism is insufficiently established against close prior work.

MSCode [1], published in July 2026 before this submission, already represents human motion and text through sparse combinations of shared learnable codewords and explicitly aligns their activation distributions (§3.2, Eqs. 3–6). Earlier, FDT [2] used sparse activation and weighted aggregation over shared semantic tokens for image–text representation learning. These are close methodological precedents for a central component of SkelAtom, but the submission does not discuss them.

SkelAtom does have identifiable differences: frame-wise top-(K') softmax, segment-averaged squared-(L_2) recipe alignment, residual feature injection, KAG supervision, and the segmentation setting. However, the paper does not adequately establish which of these changes provides a substantive advance over existing shared-codebook formulations. The current component ablations establish usefulness within SkelAtom, but do not resolve this comparative question. This substantially affects my assessment of the methodological contribution, while leaving open the possibility that the task adaptation and the particular combination are valuable.

  1. The evidence for physically grounded compositional transfer remains incomplete, and a central sparsity result needs reconciliation.

KAG regresses descriptors from averaged encoder features before the codebook (Eq. 4). This can encourage the encoder to preserve kinematic information, but does not by itself establish the physical meaning of individual atoms. Moreover, the classifier retains the original features through the residual path (Eq. 6). Proposition C.9 concerns a lower bound for label-only features; it does not establish that the learned atoms become physically consistent motion primitives.

Table 4 and Figures 4 and 14 provide useful ablations and qualitative correlations. Nevertheless, they do not isolate whether KAG improves atom-level physical consistency and whether this property explains the unseen-class improvements. This distinction matters because transferable physical primitives are the paper’s central explanation of its gains.

There is also an apparent inconsistency between Table 5 and Figure 3(b). With the stated default (K=32), setting (K'=32) in Eq. 5 should remove sparsity. Figure 3(b) shows substantially smaller drops than the “w/o sparse” row in Table 5. Different configurations could explain this, but the relevant difference is not specified.

Questions for authors
  1. Closest prior work and controlled comparison — evidence relevant to the claimed contribution.

Could you explicitly identify the contribution beyond MSCode and FDT, and compare against a corresponding shared-codebook alternative adapted to the same segmentation protocol? Keeping the skeleton backbone, textual descriptions, KAG supervision, and training budget fixed would help determine whether the gains arise from the proposed codebook modifications. A comparison with and without KAG would further separate the contribution of the codebook from that of the auxiliary supervision.

  1. KAG and atom-level grounding — clarification and supporting evidence.

What quantitative evidence shows that KAG improves the physical consistency or kinematic predictability of the atom combinations, including on unseen classes? How does this compare with the original encoder features and with absent or shuffled KAG supervision?

The broader principle of learning skeleton representations by predicting motion signals already appears in MAMP [3], although its masked joint-displacement prediction differs from the proposed segment-level statistical targets. A matched auxiliary-prediction control would help establish the value of KAG’s particular descriptors.

  1. Table 5 versus Figure 3(b) — clarification of experimental results.

Please reconcile the dense-attention results. Relative to the full model, Table 5 reports decreases of 3.00 points in Acc, 2.78 in Edit, and 2.52 in F1@50 when sparsity is removed. In contrast, the (K'=32) endpoint in Figure 3(b) appears to show decreases of approximately one point or less. Was (K) fixed at 32 in that experiment? Please provide the numerical values underlying the figure and identify any differences in configuration, training, or evaluation between the two results.

  1. Scope of the task novelty — clarification of related work.

The abstract states that existing skeleton-based zero-shot methods are strictly confined to clip-level recognition. Nagano et al. [4] already study skeleton-based zero-shot spatio-temporal localization with person-frame predictions. Please clarify how the proposed task differs in its prediction targets, class-disjoint protocol, and evaluation, and narrow the blanket statement accordingly.

Since that work was released on August 26, 2026, only nine days before the stated submission date, I regard it as near-concurrent work. Its omission is not a primary reason for my recommendation; the issue is the precision of the task-level novelty claim.

Reviewer_PVDLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SkelAtom for zero-shot skeleton-based temporal action segmentation. The method consists of two main components: Kinematic Anchor Grounding , which supervises feature representations using motion energy and temporal variations across body parts, and a Cross-Modal Atom Codebook, which sparsely encodes and aligns skeleton features and textual class prototypes in a shared space. Experiments are conducted on the U13/U26 splits of MCFS-130 and the X-sub/X-view splits of PKU-MMD. The authors report improvements over the adapted baselines.

Critical strengths

KAG uses motion descriptors derived directly from skeleton coordinates, while the Cross-Modal Atom Codebook seeks to align skeleton and text representations through a shared sparse coding space. The motivations behind both components are clear, and the ablation studies provide preliminary evidence of their effectiveness.

Critical weaknesses

1.Although the paper cites related works such as LAC and Skeleton Motion Words, which also explore compositional action representations and motion primitives or tokens, it does not clearly establish what fundamentally distinguishes SkelAtom's motion atoms from these existing representations. KAG and cross-modal alignment may offer useful improvements, but their specific contributions need to be demonstrated through more direct comparisons with prior methods. 2.Appendix B.3 states that segments shorter than 12 frames are excluded from KAG supervision, and the KAG prediction heads are removed during inference. Thus, KAG directly supervises pooled features of sufficiently long segments from seen classes, rather than individual frame representations or the semantic meaning of individual atoms. The ability of a segment-averaged feature to predict aggregate motion statistics does not, by itself, establish that the underlying frame representations or sparse atoms are physically interpretable. The evidence provided therefore falls short of supporting the paper's claim that the learned atoms have meaningful physical interpretations.

Questions for authors

1.Beyond the segment-level KAG ablations, could the authors provide frame-level motion descriptor supervision or atom-level semantic analyses to quantitatively assess whether the learned atom weights consistently correspond to specific body movements? 2.Could the authors release the exact textual descriptions and generation settings used for all action categories, and evaluate the sensitivity of the results to alternative prompts and generated descriptions? In addition, could they provide more direct comparisons with LAC and Skeleton Motion Words to clarify what advantages the shared atom codebook offers beyond existing action-composition and motion-token approaches? 3.Could the authors evaluate generalized zero-shot temporal segmentation, in which seen and unseen categories coexist in the same test sequence? For the U13/U26 comparison, could they also disentangle the effects of a larger unseen label space from those of reduced training supervision, since both factors change between the two settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.