- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GestureLens, a two-stage pipeline for IMU-based gesture recognition. It pre-trains a motion representation model on unlabeled human activity data using nucleus-focused masking and motion-specific encodings, then trains a text-guided gesture classifier on the frozen representation with limited labeled data. Experiments cover different wearable devices and user populations.
Critical strengths
The paper addresses a practical problem: learning gesture recognition models with limited labeled data across devices and user populations. The evaluation includes both public and collected datasets, with gains in some settings, particularly earbud and blind-user gestures.
Critical weaknesses
-
Limited novelty. The paper reads more like a technical report than a substantial methodological advance. The motivating failure and success cases are rather basic and presented directly, without enough analysis to establish the need for the proposed design. Figures 4 and 5 mainly combine familiar components, and the methodological novelty is not clear.
-
The presentation and experimental evidence need clarification. The criteria for assigning “short motion,” “generalization,” and “real-time” capabilities in Table 1 are not clearly defined. The figures are too small to read comfortably. In Section 4, several variables in the three loss components are insufficiently explained, and important classifier details are deferred to the appendix. Although additional experiments and latency measurements are provided there, the main experimental discussion is brief. Performance gains are also uneven: GestureLens trails the strongest baselines on Sony Watch and MMG Gesture, and its accuracy improvement on UTD-MHAD is small. These results should be discussed more carefully.
Questions for authors
-
What is the main methodological advance over existing motion tokenization, masked reconstruction, and text-guided contrastive learning methods?
-
How are “short motion,” “generalization,” and “real-time” defined in Table 1? What evidence determines whether a method receives a checkmark or a cross?
-
Appendix F.8 reports 66.3 ms end-to-end latency. Does this include gesture acquisition and segmentation, or only transmission and inference after a gesture window is available?
-
Please clarify the variables and pair definitions in the semantic and contrastive losses in Section 4, and summarize how the classifier and projection head produce the representations used by these losses.
Response to Reviewer nsho
We thank the reviewer for the feedback.
Q1 – Methodological noveltyGestureLens makes two key contributions that address the challenges of recognizing short, isolated gestures.
First, nucleus-focused pre-training: Rather than treating all signal segments equally (e.g., Bio-PM), we introduce non-uniform, nucleus-focused masking that forces reconstruction of the primary gesture phase rather than the generic preparation/retraction context.
Second, text-guided contrastive learning: Rather than relying on motion similarity for contrastive learning, which fails for opposing gestures like swipe-up/down, we introduce a semantic loss driven by text embeddings that penalizes the embedding space non-uniformly.
The ablations show the contribution of both stages. Removing nucleus and significant-axis encodings reduces earbud accuracy from 85% to 65%. With the full encoder, replacing cross-entropy alone with the complete text-guided objective improves earbud accuracy from 58% to 85% and blind-user accuracy from 73% to 85% (Table 4).
W2 – Uneven gains– Why does GestureLens achieve larger improvements on some datasets than others?The improvements vary across datasets because the datasets differ in the number of training samples, gesture types, and user populations. GestureLens is designed to learn a common motion representation that works across these different settings, rather than optimizing for a specific dataset.
For example, on Sony Watch, Cosmo and Bio-PM achieve 91% accuracy compared to our 86%. However, their performance drops to 60% and 66% on earbud gestures, where GestureLens achieves 85%. Similarly, Yuan et al. achieves 79% on MMG Gesture compared to our 77%, but only 53% on earbuds. On UTD-MHAD, GestureLens achieves 79% compared to the strongest baseline at 77%.
The improvements are larger on datasets with limited labeled samples and subtle gestures. For example, the earbud dataset has only 315 samples across seven classes, compared to 3,200 samples on Sony Watch and 23,640 on MMG Gesture. Our contribution is providing a single, generalized, frozen feature representation that maintains strong performance across data heterogeneity, rather than optimizing for individual, data-rich tasks.
We will expand this discussion and improve Figs. 4 and 5.
Q2 – Table 1 criteriaWe will define the criteria explicitly:
We will revisit the checkmarks to ensure they reflect demonstrated results.
Q3 – LatencyThe reported 66.3 ms includes Bluetooth transmission (46 ms) and phone-side processing and inference (20.3 ms), averaged over 10 trials. It is measured after gesture acquisition and does not include the time taken to perform the gesture. We will clarify this in Appendix F.8.
Q4 – Loss definitionsThe frozen encoder produces motion embeddings that are passed to the classifier. The classifier is trained with three losses: cross-entropy for class prediction, supervised contrastive loss with same-class positive pairs, and semantic loss using similarities between BERT-encoded gesture descriptions.
The semantic loss projects the 128-dimensional motion features to a normalized 32-dimensional space. For each pair, we compute:
L_sem = Σ w_ij max(0, m_ij − ||s_i − s_j||)The margin
m_ijis 2.0, 1.5, or 1.0 for semantic similarity above 0.8, above 0.5, or otherwise, respectively, andw_ij = similarity² + 0.1. The contrastive loss uses temperature 0.07. The final loss is:L_total = L_CE + w_s L_sem + w_c L_conHere,
w_sandw_cincrease to 0.3 and 0.5, respectively. BERT and the training-only projection heads are not needed at inference.We will move these definitions and the classifier architecture details into §4.