acceptodds
Withdrawn from ICLR 2027

Balancing Cross-Modal Class Alignment and Local Separation for Zero-Shot Skeleton Action Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot skeleton action recognition requires visual–semantic relations learned from seen classes to generalize to unseen actions. Existing cross-modal methods mainly reduce instance- or distribution-level modality discrepancy, but such alignment does not directly guarantee a suitable class structure. Visual and semantic representations of the same action may still form inconsistent class centers, while stronger positive alignment may compress the local separation between similar actions. We address this alignment–separation problem with a hierarchical class-geometry objective. Cross-Modal Prototype Neighborhood Modeling (CPNM) maintains visual and semantic prototype memories and uses modality-swapped anchors to form positive cross-modal class neighborhoods. Hard-Neighbor Discrepancy Amplification (HNDA) then identifies the nearest confusing class prototypes and imposes a relative margin around these neighborhoods. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate competitive performance under ZSL and GZSL protocols. Controlled ablations and geometry analyses further verify the complementary roles of cross-modal neighborhood alignment and local hard-neighbor separation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Zero-Shot Skeleton Action Recognition, Cross-Modal Representation Learning, Prototype Learning, Visual-Semantic Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SrtSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method for zero-shot skeleton action recognition. Specifically, it first employs an EMA-based prototype updating strategy to obtain class-center features for each action category. Then, it designs an energy formulation to address the cross-modal consistency issue. Moreover, it amplifies the discrepancies between hard neighboring categories to increase inter-class distances. Experimental results demonstrate competitive performance compared to prior works.

Critical strengths
  1. The paper is well written, with a clear structure and detailed presentation. The figures are intuitive and informative, and the experiments are comprehensive and thorough.
  2. The proposed method is technically sound, particularly the positive cross-modal neighborhood energy in CPNM. Meanwhile, the motivation is clear and provides an effective solution to address the cross-modal geometric consistency issue.
Critical weaknesses
  1. I believe that the cross-modal geometry issue has already been discussed in Flora [1] (CVPR 2026 Findings), particularly for generative-based architectures. Flora directly aligns the distributions of two modalities in a shared latent space, which shares similar underlying principles with the proposed method. Moreover, Flora adopts a simpler and more straightforward approach, which undermines the novelty of the proposed module. Additionally, EMA-based prototype updating has been widely explored in image-based zero-shot learning. Thus, the overall novelty and technical contributions are limited.
  2. Regarding HDNA, I also believe that its motivation and technical design are similar to those of FS-VAE and TDSM. Both methods aim to increase inter-class distances through similar loss function designs. However, the paper does not discuss these works and does not even cite TDSM.
  3. The insights regarding cross-modal consistency and inter-class separation have already been extensively discussed and explored in prior works, which diminishes the significance of the claimed contributions to the community.
  4. In the experimental section, the proposed method is not compared with several recent methods published in 2025 and 2026, such as Neuron (CVPR 2025), TDSM (ICCV 2025), and Flora (CVPR 2026 Findings), raising concerns about the effectiveness and competitiveness of the proposed method.
Questions for authors
  1. Compared to Flora, what are the advantages of Eq. (6)? Meanwhile, how does HDNA differ from the loss function designs in TDSM and FS-VAE?
  2. If the cross-modal geometry issue has been effectively addressed, can the proposed method also achieve competitive performance in low-shot training settings, as demonstrated in Flora? Alternatively, can it outperform prior methods in more challenging settings, such as the 60/60 split on NTU120?
Reviewer_WAQbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a prototype-based approach to zero-shot skeleton action recognition that combines cross-modal class alignment with local class separation. Cross-Modal Prototype Neighborhood Modeling (CPNM) aligns skeleton and text representations using same-class prototypes from the other modality. Hard-Neighbor Discrepancy Amplification (HNDA) then enforces a relative margin against the nearest prototypes of other classes. The approach is evaluated on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD under zero-shot learning (ZSL) and generalized zero-shot learning (GZSL) settings.

Critical strengths
  1. The method achieves a clear improvement on NTU-60 48/12, increasing ZSL accuracy from 57.20% with FS-VAE to 62.23%.

  2. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The motivation is not sufficiently linked to the recognition problem. The paper argues that instance-level alignment in existing methods is insufficient, but does not clearly establish how this limitation affects zero-shot skeleton action recognition. In particular, the claimed class-center mismatch and insufficient class separation are not directly linked to recognition errors on unseen actions. This makes the need for the proposed class-level constraints less convincing. The performance gains also provide limited support for the practical impact of the proposed solution. Apart from the clear gain on NTU-60 48/12, several improvements over strong existing methods are small. For example, on NTU-60 55/5, ZSL accuracy improves by only 0.05 percentage points over FS-VAE, while GZSL H decreases by 2.55 points. The results therefore do not consistently demonstrate a clear advantage over existing methods.

  2. Comparisons with recent methods and the novelty over prior prototype-based work are insufficient. The evaluation lacks comparisons with relevant recent methods, including TDSM (ICCV 2025), Neuron (CVPR 2025), Flora (CVPR 2026 Findings), and SkeletonContext (CVPR 2026). Comparisons under matched evaluation settings are needed to establish the method’s performance against the state of the art. PGFA (IEEE TIP 2025) already uses prototype-guided skeleton–text alignment. Although its use of unseen test skeletons explains the different evaluation setting, the paper lacks a detailed analysis of the added benefit of its prototype design over this prior approach. The distinction in evaluation settings alone does not establish the novelty of the proposed design.

Questions for authors
  1. Equation (15) includes the CPNM energy only through L_HNDA. Could the authors report an ablation that adds an explicit L_CPNM term to the full objective? This comparison would clarify whether an explicit alignment loss provides additional benefit beyond the alignment already included in HNDA.

  2. How were the action-level and motion-level descriptions prepared, and are they the same as those used by the compared methods? Table 8 shows that the text input affects performance. Could the authors provide the full descriptions and clarify whether differences in semantic information contribute to the reported gains over other methods?

  3. Does each class use a single fixed text feature as input for all its skeleton samples? If so, could the authors clarify what the text prototype memory adds beyond directly using the current encoded class text?

  4. Given a batch size of 64 and up to 110 seen classes, how many samples typically contribute to each class-prototype update? Could the authors clarify the sampling strategy and provide evidence that the EMA-updated prototypes remain stable despite the small number of same-class samples in each mini-batch?

Reviewer_9xmVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a class-geometry objective for zero-shot skeleton action recognition using a dual-VAE backbone. CPNM aligns representations through modality-swapped prototype memories, while HNDA imposes a relative margin against nearby negative-class prototypes. Experiments on three datasets include ZSL/GZSL comparisons, component ablations, and seen-class geometry analyses.

Critical strengths
  1. The paper addresses the trade-off between cross-modal alignment and local class separation in zero-shot skeleton action recognition. The motivation is reasonable, as stronger visual-semantic alignment does not necessarily guarantee sufficient discrimination between similar action classes.

  2. CPNM maintains visual and semantic prototype memories and uses modality-swapped anchors to improve class-level alignment. HNDA complements this by selecting nearby negative-class prototypes and imposing a relative margin. The two components are logically connected and provide a reasonable approach to balancing alignment and discrimination.

  3. The ablations provide useful analysis for examining the contributions of CPNM and HNDA. In particular, A2 uses randomly selected negative classes under the same relative-margin objective, allowing a direct comparison with hard-neighbor selection in A3, although the improvement mainly comes from seen-class accuracy. Table 6 further provides geometric evidence that CPNM improves cross-modal prototype consistency, while HNDA partially restores local class separation.

Critical weaknesses
  1. The main components largely build on established ideas in cross-modal prototype learning, hard-negative mining, and margin-based metric learning. In particular, TDSM (ICCV 2025, which is related, but not cited in this paper) already employs a triplet diffusion loss to separate correct and incorrect skeleton-text pairs through a relative margin, although it uses randomly selected negatives instead of hard-neighbor prototypes. CPNM is essentially a cross-modal center loss with EMA-updated centers. HNDA with k=1 closely resembles the nearest-negative-center margin formulation of Triplet-Center Loss (CVPR 2018) and similar prototype-based margin objectives have also been widely explored in metric learning (e.g., Proxy Anchor Loss). The novelty of HNDA therefore mainly lies in its combination with CPNM's positive neighborhood energy. Comparisons with standard hard-negative triplet would better establish its contribution.

  2. The geometry analysis in Table 6 is conducted only on seen classes, whereas the main objective is to improve generalization to unseen actions. Moreover, HNDA only partially restores the nearest-negative distance reduced by CPNM (0.146 vs. 0.182 for the original baseline). The authors should further investigate whether the proposed constraints improve class-center consistency and local separation for unseen classes, rather than demonstrating these effects only on training classes.

3.Apart from NTU-60 48/12 (+5.03 points over FS-VAE), ZSL gains over the baselines listed in Table 2 are within 0.6 points. Moreover, the comparisons omit stronger published results, including Flora on 48/12 (65.3% vs. 62.23%) and TDSM on 96/24 (65.06% vs. 62.93%). The A0 baseline also underperforms published FS-VAE results in ZSL accuracy, raising concerns about baseline strength. Additionally, no ablation is conducted on 48/12, where the largest improvement over the included baselines is reported.

Questions for authors
  1. How does the proposed objective compare with direct prototype alignment and separately weighted hard-negative margin losses under identical settings?
  2. What are the gate inputs, fusion equation, and exact validation-class partition?
  3. Was Table 7 produced after fixing the hyperparameters on validation data, or was it used to select the reported defaults?
  4. Can you report repeated-run means and standard deviations to establish the reliability of the smaller improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.