acceptodds
Withdrawn from ICLR 2027

MotionProbe: Direct LLM Probing for Zero-Shot Skeleton Action Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot skeleton action recognition commonly represents unseen actions by generating textual descriptions with an LLM and encoding them into semantic prototypes. However, we find that this two-stage process is not neutral: repeated generations under the same prompt produce different recognition outcomes, while auxiliary text encoders impose biases inherited from their own pretraining. The resulting prototypes therefore depend on both how the LLM verbalizes an action and how another model encodes it, showing that semantic construction is a critical part of zero-shot transfer rather than harmless preprocessing. Since the purpose of introducing an LLM is to access its rich action knowledge rather than the descriptions themselves, we propose MotionProbe, which directly reads prompt-conditioned hidden states from a frozen LLM, avoiding the generation variability and encoder bias introduced by the conventional pipeline. Using class-shared motion prompts, MotionProbe probes each action from complementary perspectives and organizes the resulting hidden states into a semantic constellation. These semantic views are then progressively transformed into task-ready prototypes: seen skeletons ground them in discriminative motion geometry, while unlabeled test skeletons further calibrate each candidate constellation and determine how its views should be combined. Extensive experiments on multiple benchmarks demonstrate that MotionProbe attains state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Zero-shot learning, Skeleton action recognition, Large language models, Semantic prototypes, Cross-modal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s74rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies zero-shot skeleton action recognition. It argues that generating descriptions with an LLM and encoding them with CLIP makes prototypes depend on sampling and on the encoder. MotionProbe instead reads final-token hidden states of a frozen Qwen3-4B under six fixed prompts, maps them into the skeleton space with an MLP, and calibrates them on the unlabeled test set via Sinkhorn assignment. Results are on NTU-60/120 and PKU-MMD.

Critical strengths

The diagnosis of the generate-then-encode pipeline is cleanly controlled. Regenerating descriptions from the same prompt changes Flora's accuracy by up to 8.70 and 11.61 points on the two NTU-60 splits (Appendix B), and Table 5 changes only the semantic interface: direct probing gives 75.9 versus 65.3 for generate+CLIP on 48/12. This is the most solid part of the paper.

Critical weaknesses
  1. The headline gains depend on test-time access that most baselines do not have. Target calibration uses the whole unlabeled test set, while Flora, SkeletonContext and most methods in Table 2 are inductive. Without calibration the full model reaches 62.84 on NTU-60 48/12 (Appendix Table 7), below Flora (65.3) and SkeletonContext (64.4), on exactly the split where the paper claims its largest gain (+10.6).

  2. The significance is narrow. The task is closed-set recognition on NTU and PKU-MMD with 5 to 24 unseen classes, and the core idea, using a frozen LLM's last-token hidden state as a class embedding, is already common practice for LLM sentence embeddings (e.g., PromptEOL), which is not cited. In the current LLM era, these gains say little beyond this specific benchmark setting.

Overall, I think the topic too narrow considering the current powerful MLLMs, including motion MLLMs, pose MLLMs, video MLLMs. This work is expected to be of very limited impact.

Questions for authors
  1. Please report the full model without target calibration on all eight protocols, and separate inductive and transductive methods in Tables 2 and 3.

  2. Under the same test-time access, how does MotionProbe compare with transductive methods? Skeleton-Cache is in Table 2, but PGFA is discussed and not compared.

  3. Calibration is run on the same test set that is evaluated. What happens if calibration uses a disjoint unlabeled set?

Reviewer_Xy6xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies semantic prototype construction for zero-shot skeleton action recognition. MotionProbe replaces the conventional description-generation-and-encoding pipeline with prompt-conditioned hidden states extracted from a frozen Qwen3-4B model. It constructs a semantic constellation consisting of an anchor and five motion-related views, aligns these representations with skeleton features using seen-class supervision and relation-guided geometry, and performs transductive prototype calibration using unlabeled test skeletons. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD I evaluate both conventional and generalized zero-shot recognition within each dataset. The reported results support the effectiveness of direct probing within the complete pipeline, particularly on the more difficult unknown-class splits.

Critical strengths
  1. A useful controlled comparison directly examines the central semantic-interface claim. Table 5 compares direct probing against both Generate + CLIP and Generate + same LLM while keeping the downstream components fixed. The latter comparison is particularly useful because it controls the identity of the language model and examines whether explicit description generation is beneficial. Direct probing performs better on all four NTU splits, including 75.9% versus 69.7% on NTU-60 48/12. This provides meaningful empirical support for the proposed interface choice beyond comparisons between complete systems.
  2. The evaluation covers multiple datasets and protocols, with strong reported performance. The experiments include three datasets, both ZSL and GZSL settings, and cross-subject and cross-view evaluation on PKU-MMD I. MotionProbe achieves the highest reported ZSL accuracy and GZSL harmonic mean on seven of the eight evaluated protocols. The improvements are substantial in some settings, particularly NTU-60 48/12 and PKU-MMD I 46/5. Component ablations and supplementary studies of prompt perturbations and target-class imbalance provide additional evidence beyond the main benchmark tables.
Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the additional contributions. The central semantic-interface contribution is to use a frozen LLM as a prompt-conditioned text encoder, extracting final-token hidden states instead of generating descriptions and subsequently encoding them. Table 5 supports the effectiveness of this choice within the proposed system, but effectiveness alone does not establish substantial methodological novelty. The paper does not convincingly articulate a new principle or mechanism beyond this representation substitution and its combination with multi-view alignment and prototype refinement. The additional components do not sufficiently resolve this concern because their distinctive contributions are not isolated. In particular, the ablations do not separate ordinary Fisher whitening from the proposed relation-guided modification, or prototype relocation from view-level calibration. Moreover, all semantic-interface comparisons in Table 5 include target calibration, leaving the interface advantage without target adaptation insufficiently characterized.

  2. Equation (11) is mathematically inconsistent with the view weights reported in Figure 5(c). Equation (11) defines the five view weights as a softmax over unscaled cosine similarities. Since each cosine similarity lies in [-1, 1], every weight must lie between approximately 0.0327 and 0.6488. However, Figure 5(c) reports an Identity weight of 0.80 for 'touch pocket,' together with other weights of 0.00, 0.01, and 0.02. These values cannot be produced by the stated equation, even after rounding to two decimal places. An omitted temperature parameter or another undocumented transformation could explain the discrepancy, but the manuscript does not establish which computation was actually evaluated. Because view-level calibration is a claimed component and Figure 5 is used to interpret its behavior, the current presentation does not allow verification of the mathematical specification.

  3. All zero-shot evaluations remain within a single dataset, so out-of-domain generalization is not established. The experiments evaluate transfer to unseen action labels, but not transfer to a genuinely unseen domain. Each benchmark trains and tests within NTU-60, NTU-120, or PKU-MMD I, while the cross-subject and cross-view protocols on PKU-MMD I still hold the dataset and its underlying data-collection setting fixed. These settings cover variation in subjects and viewpoints, but do not test changes in sensor characteristics, skeleton normalization, camera setup, scene distribution, or action vocabulary across datasets. This distinction is especially important for a method whose central motivation is language-based semantic transfer. The LLM prototypes are intended to describe unseen actions independently of labeled skeleton examples, so their behavior under cross-domain shifts remains an open question. The current experiments therefore establish strong performance on unseen class labels within each dataset, but do not quantify robustness to domain-level changes.

Questions for authors
  1. Clarification of the view-weighting computation. What exact computation produces the weights in Figure 5(c)? Does Equation (11) omit a temperature parameter or another transformation? Please provide the relevant implementation or a reproducible numerical example and clarify whether the same computation was used in all quantitative experiments.
  2. Clarification of the evaluation protocol. How were the GZSL calibration parameter gamma and the other hyperparameters selected without using final test labels? Which methods in Tables 2-3 have access to the same unlabeled target data? Using such data is valid in a transductive setting, but the comparison should make differences in inference-time information explicit.
  3. Evidence relevant to the novelty assessment. Can the authors isolate the benefit of the relation-guided modification beyond ordinary Fisher whitening, and of view-level weighting beyond prototype relocation? A matched comparison of the three semantic interfaces without target calibration would also help determine how much of the improvement is attributable to direct probing itself.
  4. Evidence of out-of-domain zero-shot generalization. Can the authors provide a cross-dataset transfer experiment, such as training on seen classes from one dataset and evaluating on unseen classes from another dataset with an explicitly defined class mapping? Please report the corresponding ZSL and GZSL results.
Reviewer_P1adOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses Zero-Shot Skeleton Action Recognition (ZSSAR). The authors first identify a core limitation in existing LLM-assisted ZSSAR approaches: the standard "generate-then-encode" pipeline is non-neutral. Specifically, autoregressive text generation introduces variability across repeated runs, and auxiliary text encoders (e.g., CLIP, BERT) introduce inductive pretraining biases. To overcome these issues, the authors propose MotionProbe, a direct probing framework that bypasses explicit text generation and external text encoders. MotionProbe extracts hidden states from the final layer tokens of a frozen LLM using class-shared motion prompts, forming "semantic constellations." During training, seen skeleton sequences shape these constellations via a relation-guided Fisher transformation and local hard-neighbor contrastive learning. At test time, unlabeled target skeletons calibrate both the global location of candidate constellations and the internal weights of their motion views via Sinkhorn optimal transport. Experiments across NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate state-of-the-art performance in both ZSL and GZSL settings.

Critical strengths

S1: This paper convincingly demonstrates that the standard pipeline—consisting of Large Language Model (LLM) text generation followed by CLIP encoding—introduces unnecessary instability and bias. By introducing a direct, multi-view probing method (MotionProbe), the paper proposes a concise, non-generative alternative. This approach eliminates the randomness inherent in text generation as well as the mismatch between the auxiliary encoder and the model, resulting in significant performance improvements.

S2: The empirical improvements are highly significant. Furthermore, a controlled experiment comparing "direct detection" with "re-encoding of generated text" (Table 5) provides strong substantiation for the aforementioned claims.

Critical weaknesses

W1: Although direct probing avoids the token-by-token text generation characteristic of autoregressive models, the computational overhead involved in extracting hidden states—corresponding to hundreds of action categories—from a large language model (LLM) with billions of parameters is not negligible. Providing a brief discussion or specific metrics regarding the overhead (in terms of time and memory) incurred during the feature extraction process would make the analysis more comprehensive.

W2: The phenomenon of "significant performance fluctuations due to repeated generation"—as presented by the authors in Table 1—is predicated on the use of stochastic decoding strategies (such as setting Temperature > 0 or employing Top-p sampling). In practice, however, many LLM-based approaches mitigate the randomness inherent in generation by producing multiple outputs and subsequently utilizing a "prompt ensemble" or "mean embedding" strategy. The authors do not address whether the baselines could have leveraged such ensemble or averaging techniques to neutralize this stochasticity; consequently, their analysis may have, to some extent, exaggerated the shortcomings of the traditional paradigm.

Questions for authors

Please refer to W1 and W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.