acceptodds
Under review as a conference paper at ICLR 2027

GestureLens: Generalized Learning Framework for IMU-based Human Gesture Recognition

Abstract

IMU-based gesture interfaces are being increasingly adopted as efficient, accessible, and intuitive alternatives to traditional input methods, such as touchscreens and voice. However, current gesture recognition algorithms are tailored to work for specific devices (e.g., smartwatches vs. earbuds) or user populations (e.g., blind vs. sighted users), limiting their generalizability. In this paper, we design GestureLens, a generalized IMU-based gesture recognition framework that works across devices and populations with minimal training samples. To overcome the challenges and high cost of collecting large-scale labeled training data, GestureLens leverages readily available unlabeled human activity data. The GestureLens pipeline comprises two stages: (1) pre-training a motion representation model using abundant unlabeled human activity data, and (2) training a gesture classifier on top of the frozen representation using a small amount of labeled gesture data. For pre-training, we introduce a token-based strategy and embeddings that learn to identify and focus attention on the key motion signatures in the temporal data For classification, we design a text-guided classifier that can reliably differentiate between temporally or semantically similar gestures. We evaluate GestureLens across both hand gestures (captured through a smartwatch) and earbud gestures (captured through earbuds), using data collected from blind and sighted users. Across these diverse devices and user populations, GestureLens achieves an accuracy of 85%, across an average of 13 gesture classes using only 10% of labeled data for training. GestureLens significantly outperforms state-of-the-art self-supervised learning approaches and specialized gesture recognition models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Gesture Recognition, Accessibility, Inertial Sensing, Wearable Computing, Self-Supervised Learning, Sensor Time-Series

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nshoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GestureLens, a two-stage pipeline for IMU-based gesture recognition. It pre-trains a motion representation model on unlabeled human activity data using nucleus-focused masking and motion-specific encodings, then trains a text-guided gesture classifier on the frozen representation with limited labeled data. Experiments cover different wearable devices and user populations.

Critical strengths

The paper addresses a practical problem: learning gesture recognition models with limited labeled data across devices and user populations. The evaluation includes both public and collected datasets, with gains in some settings, particularly earbud and blind-user gestures.

Critical weaknesses
  1. Limited novelty. The paper reads more like a technical report than a substantial methodological advance. The motivating failure and success cases are rather basic and presented directly, without enough analysis to establish the need for the proposed design. Figures 4 and 5 mainly combine familiar components, and the methodological novelty is not clear.

  2. The presentation and experimental evidence need clarification. The criteria for assigning “short motion,” “generalization,” and “real-time” capabilities in Table 1 are not clearly defined. The figures are too small to read comfortably. In Section 4, several variables in the three loss components are insufficiently explained, and important classifier details are deferred to the appendix. Although additional experiments and latency measurements are provided there, the main experimental discussion is brief. Performance gains are also uneven: GestureLens trails the strongest baselines on Sony Watch and MMG Gesture, and its accuracy improvement on UTD-MHAD is small. These results should be discussed more carefully.

Questions for authors
  1. What is the main methodological advance over existing motion tokenization, masked reconstruction, and text-guided contrastive learning methods?

  2. How are “short motion,” “generalization,” and “real-time” defined in Table 1? What evidence determines whether a method receives a checkmark or a cross?

  3. Appendix F.8 reports 66.3 ms end-to-end latency. Does this include gesture acquisition and segmentation, or only transmission and inference after a gesture window is available?

  4. Please clarify the variables and pair definitions in the semantic and contrastive losses in Section 4, and summarize how the classifier and projection head produce the representations used by these losses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nsho

We thank the reviewer for the feedback.


Q1 – Methodological novelty

GestureLens makes two key contributions that address the challenges of recognizing short, isolated gestures.

First, nucleus-focused pre-training: Rather than treating all signal segments equally (e.g., Bio-PM), we introduce non-uniform, nucleus-focused masking that forces reconstruction of the primary gesture phase rather than the generic preparation/retraction context.

Second, text-guided contrastive learning: Rather than relying on motion similarity for contrastive learning, which fails for opposing gestures like swipe-up/down, we introduce a semantic loss driven by text embeddings that penalizes the embedding space non-uniformly.

The ablations show the contribution of both stages. Removing nucleus and significant-axis encodings reduces earbud accuracy from 85% to 65%. With the full encoder, replacing cross-entropy alone with the complete text-guided objective improves earbud accuracy from 58% to 85% and blind-user accuracy from 73% to 85% (Table 4).


W2 – Uneven gains – Why does GestureLens achieve larger improvements on some datasets than others?

The improvements vary across datasets because the datasets differ in the number of training samples, gesture types, and user populations. GestureLens is designed to learn a common motion representation that works across these different settings, rather than optimizing for a specific dataset.

For example, on Sony Watch, Cosmo and Bio-PM achieve 91% accuracy compared to our 86%. However, their performance drops to 60% and 66% on earbud gestures, where GestureLens achieves 85%. Similarly, Yuan et al. achieves 79% on MMG Gesture compared to our 77%, but only 53% on earbuds. On UTD-MHAD, GestureLens achieves 79% compared to the strongest baseline at 77%.

The improvements are larger on datasets with limited labeled samples and subtle gestures. For example, the earbud dataset has only 315 samples across seven classes, compared to 3,200 samples on Sony Watch and 23,640 on MMG Gesture. Our contribution is providing a single, generalized, frozen feature representation that maintains strong performance across data heterogeneity, rather than optimizing for individual, data-rich tasks.

We will expand this discussion and improve Figs. 4 and 5.


Q2 – Table 1 criteria

We will define the criteria explicitly:

  • Short motion: Methods designed or evaluated for isolated gestures.
  • Generalization: Demonstrated transfer across devices, datasets, or populations without retraining the representation model.
  • Real-time: Measured inference latency on the reported hardware.

We will revisit the checkmarks to ensure they reflect demonstrated results.


Q3 – Latency

The reported 66.3 ms includes Bluetooth transmission (46 ms) and phone-side processing and inference (20.3 ms), averaged over 10 trials. It is measured after gesture acquisition and does not include the time taken to perform the gesture. We will clarify this in Appendix F.8.


Q4 – Loss definitions

The frozen encoder produces motion embeddings that are passed to the classifier. The classifier is trained with three losses: cross-entropy for class prediction, supervised contrastive loss with same-class positive pairs, and semantic loss using similarities between BERT-encoded gesture descriptions.

The semantic loss projects the 128-dimensional motion features to a normalized 32-dimensional space. For each pair, we compute:

L_sem = Σ w_ij max(0, m_ij − ||s_i − s_j||)

The margin m_ij is 2.0, 1.5, or 1.0 for semantic similarity above 0.8, above 0.5, or otherwise, respectively, and w_ij = similarity² + 0.1. The contrastive loss uses temperature 0.07. The final loss is:

L_total = L_CE + w_s L_sem + w_c L_con

Here, w_s and w_c increase to 0.3 and 0.5, respectively. BERT and the training-only projection heads are not needed at inference.

We will move these definitions and the classifier architecture details into §4.

Reviewer_CJahOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GestureLens, a two-stage learning framework for IMU-based gesture recognition across different wearable devices and user populations using limited labeled data. First, the encoder is pretrained on unlabeled human-activity data. The method identifies a high-energy motion segment, termed the nucleus, concentrates masked reconstruction within this segment, and incorporates nucleus, significant-axis, and positional encodings. Second, the frozen encoder is combined with a text-guided classifier trained using 10% labeled target-domain data. The classifier represents manually defined gesture descriptions with BERT and jointly optimizes classification, semantic, and contrastive objectives.

GestureLens is evaluated on six gesture datasets covering smartwatches, earbuds, sighted users, blind users, and gesture vocabularies of different sizes. The reported accuracies across these datasets range from 77% to 88%, with approximately 85% accuracy in the primary cross-device and cross-population settings. The evaluation includes comparisons with supervised and self-supervised baselines, component and masking ablations, analyses of label availability and class count, and confusion matrices. The paper also reports 93% average accuracy on three cross-dataset human-activity-recognition benchmarks. A quantized implementation evaluated on a Samsung Galaxy S20 has a model size of 863.16 kB and an end-to-end latency of 66.3 ms.

Critical strengths

Important and practically relevant problem setting. The paper targets low-label IMU gesture recognition across different wearable form factors and user populations, including earbud interactions and gestures performed by blind users. This is consequential because collecting large labeled gesture datasets for every device and population is expensive, and accessibility-oriented populations are often underrepresented.

Critical weaknesses

The central generalization claims are not supported by a sufficiently rigorous evaluation protocol. GestureLens trains a separate classifier using 10% labeled data from every target dataset, making the setting closer to few-shot target-domain adaptation than a unified cross-device or cross-population recognition model. More importantly, the paper does not clearly establish that the training, validation, and test sets are subject-disjoint. Without strict user-level separation, the claimed cross-user and cross-population generalization may not be valid, directly undermining the paper’s primary contribution.

The deployment claims and baseline comparisons lack sufficient supporting evidence. The reported 863.16 kB value is the size of the quantized TorchScript file rather than a complete model-complexity analysis; parameter count, FLOPs, runtime memory, and power consumption are not reported. Inference is evaluated on a Galaxy S20, while the smartwatch only streams sensor data, so the experiments do not demonstrate execution on watches or earbuds themselves. Meanwhile, the baseline evaluation omits implementation versions, checkpoints, complete preprocessing, tuning ranges, and final configurations, despite adapting some methods to different modalities or settings. These omissions make it difficult to verify either the claimed wearable deployability or the fairness of the reported performance gains.

Questions for authors

Are the training, validation, and test sets strictly subject-disjoint for every dataset? Please specify the exact splitting procedure, the source of validation data, and whether samples from the same participant can appear in multiple splits. Given that a separate classifier is trained with 10% labeled data from each target dataset, please also clarify the precise sense in which the method provides cross-user or cross-population generalization.

What quantitative evidence demonstrates that the detected high-energy nucleus contains the most class-discriminative information? How does focused masking compare, under the same architecture and training budget, with uniform masking, random equal-length intervals, or alternative temporal-localization methods?

Why are all signals resampled to 20 Hz, particularly for short and impact-like gestures such as taps and double taps? Was anti-aliasing applied, and how sensitive are the results to sampling frequency, nucleus-detection parameters, and missing sensor modalities?

Which code versions, pretrained checkpoints, preprocessing pipelines, and hyperparameter-search spaces were used for each baseline? How many random seeds and independent 10% labeled-data samples were evaluated, and can the authors report variance or confidence intervals for the main results?

What are the total and trainable parameter counts, FLOPs, peak runtime memory, and power requirements of GestureLens? Since inference was evaluated on a Galaxy S20 rather than on the smartwatch or earbuds themselves, what evidence supports direct deployment on resource-constrained wearable devices?

1 comment
AuthorsOpenReview ↗

Response to Reviewer CJah

We thank the reviewer for the comments. We address the evaluation and implementation concerns below.


Q1 – Generalization – Does GestureLens generalize to unseen users?

Our generalization claim is based on using a single frozen encoder across different devices and user populations. The pre-training dataset (HHAR) is disjoint from all target gesture datasets in terms of users, devices, and classes. However, our original evaluation uses sample-level splits, which do not guarantee that users in the test set are unseen during classifier training.

To evaluate cross-user generalization, we conducted Leave-One-Subject-Out (LOSO) experiments on sighted-user hand gestures, sighted-user earbuds, and blind-user hand gestures, using 3 and 4 labeled examples per class from the held-out user. GestureLens achieves the highest accuracy on two of the three datasets. For blind-user gestures, it achieves 88.5% (K=3) and 91.6% (K=4), compared to the strongest baseline at 86.6% and 89.0%. On earbuds, GestureLens achieves 58.0% and 67.4%, outperforming the strongest baselines at 52.4% and 54.6%.

For sighted-user hand gestures, GestureLens achieves 64.2% and 67.8%, while Bio-PM performs best at 87.4% and 90.3%. This dataset contains substantially more samples (3,200) than the earbud (315) and blind-user (1,200) datasets, although further analysis is needed to explain the performance difference. These results show that GestureLens is particularly effective for unseen-user adaptation on the challenging earbud and blind-user datasets. We will include the complete results, including per-user performance, in the revised paper.


Q2 – Nucleus identification and masking

Please see our response to F3YW W1. Existing random-masking approaches such as LIMU-BERT reconstruct random motion segments, whereas GestureLens concentrates 80% of masks within the detected nucleus. With 10% labeled data, LIMU-BERT achieves 42% accuracy on earbuds and 56% on blind-user gestures, compared to 85% for GestureLens on both (Table 3). Table 5 further evaluates nucleus masking probabilities, with the highest accuracy at 80%. These results establish the benefit of the complete pre-training approach.


Q3 – Sampling frequency – Why do we resample all signals to 20 Hz, and does this affect short gestures?

We resample to 20 Hz to maintain a common temporal scale across devices and match the pre-training representation. At this frequency, the nucleus detection window of 15 samples corresponds to 750 ms. We also normalize the signal energy to [0,1], allowing the same detection threshold of 0.4 across datasets (Appendix B.2).

The evaluation includes short smartwatch and earbud gestures, achieving 86% and 85% accuracy, respectively. While these results show that 20 Hz is sufficient for the evaluated gestures, we do not claim that it is optimal for all short or impact-like gestures.


Q4 – Baseline reproducibility

All baselines use the same target-data splits and labeled-data budget. Appendix E describes the baseline implementations, and Appendix F.6 already reports variability for several encoder comparisons. For example, ContrastSense achieves 62.4% ± 3.6% on sighted-user hand gestures and 67.1% ± 3.9% on blind-user gestures using cross-entropy classification.

We will add the exact implementation versions, checkpoints, preprocessing, and hyperparameter configurations, and clarify the number of independent runs used for the reported results.


Q5 – Deployment – Can GestureLens run on resource-constrained wearable devices?

The reviewer raises the concern that our real-time evaluation is performed on a smartphone rather than directly on a wearable. Our goal is to enable real-time gesture interactions using commodity wearables, where the wearable captures IMU data and a paired smartphone performs recognition. This avoids running computationally expensive models on resource-constrained wearables.

GestureLens has 1.25M parameters at inference: a frozen encoder with 63K parameters and a classifier with 1.19M parameters (the training-only projection heads add 75K). Only the classifier is trained. Each gesture requires 58.7 MFLOPs, and the model size is 5.2 MB in FP32 and 1.8 MB with INT8 quantization.

On a Samsung Galaxy S20, recognition takes 66.3 ms after gesture acquisition, including 46 ms for Bluetooth transmission and 20.3 ms for processing and inference (Appendix F.8). These results demonstrate that GestureLens supports real-time wearable gesture interaction with smartphone-assisted inference, without requiring the model to execute directly on the wearable. We will clarify this distinction in the paper.

Reviewer_F3YWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GestureLens, an IMU-based gesture recognition framework designed to generalize across devices and user populations with limited labeled data.

Critical strengths
  • This paper has a clear motivation. Leveraging readily available unlabeled activity data for gesture recognition with limited labels is well motivated. The inclusion of earbud interactions and gestures performed by blind users broadens the practical relevance.
  • The experimental results are significant, showing a marked improvement over existing SOTA self-supervised learning approaches and specialized gesture recognition models.
  • Broad experimental analysis. The paper examines input encodings, loss components, masking probabilities, text perturbations, encoder choices, and the amount of labeled training data.
Critical weaknesses
  • The energy rules of the motion core area in the first stage have certain heuristic characteristics; even if a high-energy region is identified, evidence is still needed to confirm that it corresponds to the area with the strongest action discriminability. Larger preparatory movements may have higher energy, while subtle contact movements might have lower energy. How can one prove that high-energy regions directly correspond to "identifying the most discriminative movement areas"?
  • The layout and the writing of the paper need to be improved. For example, the text in Figures 4 and 5 is too small and the flow details are hard to read. It is recommended to enlarge the font and adjust the layout inside the figure.
  • The design of the text-guided classifier is not clearly defined. How textual relations determine the margins and weights in the semantic loss. For example, how does this commonality help the model distinguish the opposite gesture direction, given that the text semantics of sliding up and down are close to each other?
  • There is a lack of controlled experiments with "no pre-training". How much improvement does first-stage pre-training with human activity data give, compared to training from scratch with gesture labels only?
Questions for authors

Please see the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer F3YW

We thank the reviewer for the feedback and for recognizing the motivation and experimental results.


W1 – Energy-based nucleus identification – Does the energy-based nucleus capture the most discriminative movement?

The reviewer raises the concern that preparatory movements may also have high energy, while the actual gesture may contain subtle movements. However, our approach does not simply select the region with the highest energy. Instead, we identify significant changes in normalized motion energy (Eq. 1). Preparatory movements are generally gradual and do not exhibit the sharp energy changes associated with the primary gesture movement.

This observation is supported by prior work on smartwatch-based hand gestures (MobiCom'23, CHI'24). For example, MobiCom'23 (Fig. 3) shows that the nucleus of a flick gesture has a distinct motion signature across five users, while the preparation and retraction vary considerably. Further, CHI'24 shows that identifying the nucleus enables high accuracy across gestures performed by blind users.

Our experiments further show the benefit of focusing on this region. Fig. 6 shows an 8.7% reduction in reconstruction error, while Table 5 shows that concentrating 80% of masks within the nucleus achieves 85% accuracy on blind-user gestures, compared to 69% when 90% are concentrated there.


W2 – Figure readability

We will enlarge and reorganize Figs. 4 and 5.


W3 – Text-guided classifier – How does the text-guided classifier distinguish gestures with similar semantics?

The reviewer raises the question that swipe-up and swipe-down have similar textual descriptions, making it difficult to distinguish them. However, our text descriptions explicitly capture how these gestures differ. For example, swipe-up is described as "an upward swipe gesture with properties: primary type: directional, direction: up, complexity: simple" (§4), while swipe-down has the same primary type but specifies direction: down. Thus, even though both gestures belong to the same category, their descriptions capture the difference in movement direction.

We use BERT to encode these descriptions and their similarity to determine how strongly different gesture classes should be separated. Unlike standard contrastive loss, which treats all negative pairs equally, our semantic loss assigns larger margins to semantically similar but distinct gestures. For example, a similarity of 0.9 gives a margin of 2.0 and weight of 0.91, while a similarity of 0.4 gives a margin of 1.0 and weight of 0.26. This helps distinguish gestures that share similar motion patterns but represent different commands.

Our results show the benefit of this design. Adding semantic and contrastive losses improves earbud accuracy from 58% to 85% (Table 4). Further, Table 6 shows that shuffling or replacing meaningful descriptions reduces performance.


W4 – Contribution of pre-training – How much does pre-training improve gesture recognition compared to supervised learning?

To evaluate the contribution of pre-training, we compare GestureLens against two supervised models trained from scratch: (1) a Transformer classifier and (2) a Transformer with our text-guided classifier. We vary the labeled training data from 10% to 100% across six gesture datasets.

Our results show that supervised models require substantially more labeled data to achieve comparable accuracy. For example, on Sony Watch, GestureLens achieves 86% accuracy with only 10% labeled data, while supervised models require nearly 80% labeled data to reach 80% accuracy. On UMAHand and UTD-MHAD, supervised models also require substantially more labeled data to approach GestureLens.

Also, on blind-user and earbud gestures, supervised models do not reach GestureLens accuracy even with 100% labeled data. For blind-user gestures, GestureLens achieves 85–90% accuracy with 10–100% labeled data, while supervised models reach only 54–56% at 100%. Similarly, on earbuds, GestureLens reaches 84–95%, while supervised models achieve only 55–57% at 100%. These datasets contain subtle and variable gestures, for which prior work has shown the need for specialized recognition models.

These results show that pre-training not only reduces the amount of labeled gesture data required but also enables a single representation to work across devices and populations, whereas supervised models trained from scratch may require specialized designs for these challenging settings.

We will add these results in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.