acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Clustering for Weak Annotation of Human Activity Recognition

Abstract

Annotating Inertial Measurement Unit (IMU) data for Human Activity Recognition (HAR) is challenging and time-consuming, as raw inertial signals are difficult for humans to directly interpret and assign activity labels. Recent studies therefore leverage synchronized ego-centric video for weak annotation, using visual representations to cluster activity samples and requiring manual labels only for representative samples. However, visual data can raise privacy concerns and its representations may encode scene and recording context unrelated to the performed activity, influencing the resulting feature space, clustering structure, and activity labels. To incorporate additional activity-relevant information, we propose WAFER (Wearable And visual Foundation Embeddings for weak annotation of HAR), a multimodal weak annotation method that incorporates representations from a pretrained wearable foundation model alongside visual and optical-flow features. Temporally aligned feature embeddings from video, wearable IMU signals, and optical flow are independently standardised, weighted, and fused before clustering. Human annotation is required only for few representative samples, after which activity labels are propagated to the remaining samples within each cluster. The resulting weak annotations are then used to train downstream HAR models using only IMU data, such that video is not required during downstream training. Across three benchmark datasets, WAFER improves weak annotation accuracy by up to 6.7. Using these weak annotations for downstream HAR training improves accuracy by up to 6.7 and macro F1 by up to 10.5, achieving performance approaching the performance of fully supervised training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Learning, Human Activity Recognition, Wearable Sensing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TckaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WAFER, a method to reduce the annotation effort for IMU-based human activity recognition. It extends the visual clustering approach of Bock et al. by adding features from a pretrained wearable model. Video, optical-flow, and imu features are standardized, combined through weighted concatenation, and clustered separately for each subject. One representative from each cluster is labeled, and that label is propagated to the remaining samples. These annotations are then used to train HAR models using only IMU data.

The method is evaluated on WEAR, ActionSense, and Wetlab through annotation accuracy and downstream performance with TinyHAR and DeepConvLSTM under leave-one-subject-out evaluation. WAFER improves annotation accuracy over the selected CLIP+flow baseline on all three datasets, with the clearest downstream gains on WEAR. The paper also examines modality weights, visual backbones, and the tradeoff between annotation effort and performance.

Critical strengths
  1. The results on WEAR show a clear benefit from adding wearable features. Annotation accuracy improves from 83.96% to 90.64% in the main comparison. The modality ablation also supports this: within the weight search in Table 4, the best CLIP+flow configuration reaches 85.14%, while using all three modalities reaches 90.64%. IMU-only features perform much worse. This suggests that the modalities provide complementary information.
  2. its interesting that the evaluation checks whether better annotations lead to better downstream recognition. The paper evaluates two HAR architectures under leave-one-subject-out evaluation and reports the annotation effort for different cluster counts. On WEAR, DeepConvLSTM with Weak-CE improves from 73.38% to 80.12% accuracy and from 63.23 to 73.77 macro F1. Matching the percentage of retained data for the thresholded comparisons is also reasonable, since the feature scales differ between the methods.
Critical weaknesses
  1. My main concern is that the comparisons do not establish an improvement over prior work as broadly as the paper suggests. The main technical change is adding pretrained wearable embeddings to the clustering and label-propagation pipeline of Bock et al. (2024b). This is a useful extension, but its value needs to be established through clear comparisons.
  2. The main annotation comparison only uses the earlier paper’s CLIP+flow configuration. However, Table 2 of Bock et al. reports 72.70% with CLIP alone on Wetlab, which is above WAFER’s 71.60%, and 60.25% with DINOv2 alone on ActionSense, which is close to WAFER’s 60.84%. Comparing against the fixed CLIP+flow pipeline is reasonable, but it does not establish an improvement over the strongest configurations reported in that paper. Bock et al., Table 2
  3. The downstream results are also mixed. In Table 3, the best weak-training DeepConvLSTM accuracy on ActionSense is 16.26% for WAFER, compared with 17.35% for the baseline. I find the WEAR results promising, but the evidence for a broader improvement is less convincing.
  4. Another concern is the model-selection protocol. Section 4.4 selects the fusion weights using annotation accuracy on WEAR, and Appendix A.4 uses the same dataset to select the visual backbone. I could not find a description of independent validation subjects or an accounting of the ground-truth labels used for these choices. This matters because reducing annotation requirements is the main motivation of the paper. It is unclear whether the reported WEAR performance is independent of these decisions.
  5. The baseline comparison also needs clarification. It is not clear which results were taken from the earlier paper and which were rerun under the current setup. For example, Random-CE differs between the baseline and WAFER columns despite using the same stated annotation budget. This could reflect differences in sampling or training randomness, but it needs to be explained. Appendix A.6 mentions three-seed averaging under GMM clustering, without making clear whether this also applies to downstream training. Performance variability is not reported, making the smaller gains difficult to assess.
Questions for authors
  1. Which subjects and ground-truth labels were used to select the modality weights, wrist weights, and visual backbone? Were the reported WEAR results evaluated independently of these choices, including within each downstream leave-one-subject-out fold? How are labels used for model selection counted in the annotation budget?
  2. Which results in Tables 2, 3, and A3 were taken from Bock et al., and which were rerun? Were the baseline and WAFER downstream models evaluated using identical sensor inputs, preprocessing, training windows, losses, training schedules, folds, and seeds? Why does Random-CE differ between the two columns?
  3. Why is the main annotation comparison limited to CLIP+flow, given the stronger visual-only results reported by Bock et al. on Wetlab and ActionSense? Are those results directly comparable under your protocol, and how do they affect the claim of improvement across all three datasets?
1 comment

Dear reviewer,

Thank you for taking the time to assess our work and marking it clear in presentation and fixable in terms of soundness. Below we'd like to answer to your questions:

  1. We first evaluated different modality weights through subject-wise GMM clustering and weak annotation experiments, measuring annotation accuracy against ground-truth labels. We then selected the best-performing configuration across WEAR subjects and used these fixed parameters for downstream HAR training under the LOSO protocol. The weights were not optimized separately for individual subjects or downstream folds. We acknowledge that the initial configuration selection used ground-truth labels from all WEAR subjects and was not performed using an independent validation set.
  2. All baseline and WAFER experiments were rerun under the same experimental setup, including sensor inputs, preprocessing, window configurations, downstream architectures, losses, training schedules, LOSO folds, and random seeds, to ensure a fair comparison.
  3. We selected CLIP+flow as the primary baseline because Bock. et al. identified this combination as the preferred configuration for downstream HAR evaluation. To ensure a consistent comparison we follow the same baseline.
Reviewer_961COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an approach to weakly annotate wearable sensor data (specifically IMU) using synchronized video collected of the participants performing activities. The authors concatenate wearable FM, video FM, and optical flow representations, which is used for clustering with GMMs. Only the cluster ‘center’ is annotated by humans, thereby substantially reducing the annotation efforts. The authors show that training with the cluster-based annotations produces strong performance at activity recognition.

Critical strengths
  1. The paper tackles an important, practical problem. Annotation of sensor data is costly and time consuming; approaches towards reducing those are of particular practical utility.

  2. The authors utilize foundation models to obtain representations. With the development and release of more effective FMs, there is potential for the performance to improve further.

Critical weaknesses
  1. Incremental relative to existing methods: the idea of utilizing clustering for annotation is not new. As the authors acknowledge, their approach is based on Bock et al., who utilize only video features for clustering. As such, this paper concatenates the representations from sensor data and video and performs clustering. While this makes the approach directly comparable to Bock et al., the approach also seems incremental. Please clearly specify how this approach improves upon Bock et al's beyond the addition of the sensor and optical flow representations.

  2. Table 3 and other tables lack standard deviation and confidence intervals. Many of the variations in Table 3 are close in performance so it is hard to know the statistical significance of the scores.

  3. The setup for some experiments is not clear. In Table 4 and Figure 8 etc., how are the parameters tuned with a LOSO setup? What functions as the validation set?

Questions for authors
  1. For the weak annotation experiments, how is the test set constructed? Does it use human annotations or the weak annotations? It should be the former since the authors are aiming to prove the effectiveness of their weak annotation approach and doing manually annotated test set would make the comparison to fully supervised approaches consistent.

  2. The authors refer to the “fully supervised training” baseline. They mean when using ground truth labels. However, both TinyHAR and DeepConvLSTM are both end-to-end, so all of their methods are end to end albeit they can use the weak annotations. So, the authors should refer to this baseline in a different way.

  3. One of the contributions mentioned by the authors is that they perform visual context analysis, to show how representations are influenced by background and scene. This finding is already known, e.g., [1] and more, and so it does not feel like a contribution.

  4. The use of video encoders instead of CLIP or DINO-v3, which are primarily image-based could lead to performance improvements. The authors should consider trying it.

References:

[1] Zhou, Ellie, Jihoon Chung, and Olga Russakovsky. "Seeing beyond the scene: Analyzing and mitigating background bias in action recognition." arXiv preprint arXiv:2512.17953 (2025).

2 comments

Dear reviewer,

Thank you for taking the time to assess our work and marking it clear in presentation and fixable in terms of soundness. Below we'd like to answer to your questions:

  1. We indeed evaluate the weak annotation against the human-provided ground truth labels. For downstream HAR model trainings, we use LOSO, in which we leave one subject out to test train the model on the rest. This is done for for every subject.
  2. Yes, the fully supervised training means the models trained on ground truth labels. We can also refer to this as ground truth supervised training in a revision.
  3. Our analysis demonstrates how visual context affects activity-wise clustering in egocentric videos, motivating the incorporation of wearable representations. We also show how these representations reduce the effect of background bias on clustering and further improving the activity-wise clustering.
  4. While video encoders can be used, our current approach already uses both spatial and temporal information by combining visual representations from CLIP and motion representations from RAFT and I3D.

Dear authors, thank you for your response. It clarifies some of my questions but it does not tackle the weaknesses mentioned above, e.g., novelty and the lack of confidence intervals. Due to this, i will retain the current score.

Reviewer_JLKwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WAFER, a weak-annotation pipeline for wearable human activity recognition. WAFER combines CLIP features from synchronized egocentric video, OxWearables representations from IMU signals, and optical-flow features extracted using RAFT and I3D. The standardized and weighted features are concatenated and clustered using a per-subject Gaussian mixture model. A human annotator labels one representative window per cluster, and the label is propagated within the cluster. The resulting weak labels are used to train IMU-only TinyHAR and DeepConvLSTM models. Experiments on WEAR, ActionSense, and Wetlab show improved annotation accuracy over the visual and optical-flow pipeline.

Critical strengths
  1. The paper addresses the important practical problem of reducing manual annotation effort for wearable HAR datasets.
  2. Adding wearable representations to the visual weak-annotation pipeline improves annotation accuracy on all three evaluated datasets. The paper also evaluates the usefulness of the generated labels using two downstream IMU-only models.
Critical weaknesses
  1. The main contribution is incremental. The pipeline largely follows Bock et al., with the principal addition being OxWearables features and weighted concatenation. However, the paper does not compare against raw IMU features, a randomly initialized IMU encoder, or dimension-matched representations. It is therefore unclear whether the improvement comes from wearable foundation-model pretraining, simply adding IMU information, or changing the dimensionality and geometry of the clustering space.
  2. The evaluation protocol and statistical reliability require further clarification. For example, fusion weights are selected using ground-truth annotation accuracy on WEAR, but it is unclear how this selection is separated from final evaluation. The paper also reports point estimates without variation across subjects, GMM initializations, or downstream training seeds. This is important because the downstream improvements are not consistent across all configurations.
Questions for authors
  1. Is there any methodological contribution beyond Bock et al. other than adding OxWearables features and weighted concatenation? Can the authors provide all modality subsets on each dataset?

  2. How were the fusion weights (0.50, 0.25, 0.25) selected? Were ground-truth labels from any reported evaluation subjects used directly or indirectly during this selection?

  3. Can the authors compare OxWearables features against raw IMU features, handcrafted time-frequency features, and a randomly initialized encoder with matched output dimensionality?

1 comment

Dear reviewer,

Thank you for taking the time to assess our work and marking it clear in presentation and fixable in terms of soundness. Below we'd like to answer to your questions:

  1. For the methodological contribution, we'd like to clarify that the per-modality subsets can be read in Table 4 for the WEAR dataset. For instance, just the IMU modality will result in an accuracy of 65.2% (see the first column). Similarly, such results for the other two datasets can be easily produced in a revision.
  2. The fusion weights were selected for an optimal annotation accuracy, as per the last row in Table 4, depicting the weight-space search we carried out. Ground truth labels were exclusively taken to annotate the representative window of the cluster in this process. We will clarify this in a revision.
  3. For the comparison of the OxWearables features, we argue that this is beyond the scope of our paper: Our main aim here is to investigate how complementary visual, optical-flow, and wearable representations can be combined toward multimodal weak annotation of HAR datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.