acceptodds
Under review as a conference paper at ICLR 2027

HALO: LEARNING MOTION REPRESENTATIONS AS CONTROLLED DEPARTURES FROM SIGNAL STATISTICS

Abstract

Wearable sensors can record the same activity differently across people, body locations, and devices, making a model trained in one setting difficult to reuse in another. Handcrafted signal statistics often remain useful under such changes, but they are limited to a fixed set of measurements. Neural encoders are more flexible, yet they usually learn a separate feature space instead of building on this prior structure. We introduce HALO (Handcrafted Anchors, Limited Offsets), a 0.16M-parameter encoder that starts from a statistical representation and learns a bounded departure from it. HALO treats the statistics as an anchor and predicts a zero-initialized, input-conditioned offset whose distance from the anchor is limited by an interpretable radius. Varying this radius produces a family of representations from a single forward pass. To separate motion from acquisition effects, we pretrain HALO in HAR-CAUKER, a synthetic motion world with physically coupled inertial rendering and supervision from simulator-only factors, then freeze the encoder for real-data evaluation. Across 14 public datasets, HALO ranks first in aggregate under cross-subject, cross-position, and cross-dataset protocols, exceeding the strongest external comparator by 0.6, 11.0, and 9.0 macro-F1 points. It ranks first in all six cross-position directions and on 9/13 held-out datasets. Without a fitted classifier, HALO also improves Recall@1 over MantisV2 by 9.7/8.6 points across positions/datasets; shuffling its learned directions removes 15.9/20.0 points, showing that the learned departure carries transferable, input-matched structure. These results support learning bounded departures from signal statistics as a practical alternative to relearning representations for each deployment.

Primary area:
learning on time series and dynamical systems
Keywords:
wearable human activity recognition, inertial sensing, representation learning, synthetic pretraining, cross-domain generalization, handcrafted features, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QMN4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HALO studies the task of human action recognition (HAR) from wearable sensors and the generalizability (cross-subject, cross-position, cross-dataset) of the learned feature representation. HALO offers two simple yet effective designs: (1) it hypothesizes that simple hand-crafted motion statistics are more stable cross-domain representations, and encodes each handcrafted statistical anchor with a network-learned feature correction. (2) it proposes HARCauKer for neural-part pretraining, a synthetic motion environment with physically coupled inertial rendering. It is tailored for IMU HAR. It constructs a thorough cross-domain generalization benchmark, and HALO outperforms the general-purpose time-series baseline MantisV2 and various IMU-based methods.

Critical strengths

1.        The paper is well-written, well-motivated and clearly highlights the challenges.

2.        It provides a thorough generalization benchmark for cross-domain HAR and clear reproduction details.

3.        It uses constrained features based on simple hand-crafted features and designs simulation training tailored for IMU HAR. The results outperform the general-purpose time-series foundation model MantisV2, particularly in cross-position and cross-dataset settings.

Critical weaknesses

1.        A random set of 128 hand-crafted features is selected from the 10,098 candidates: Why not use all hand-crafted features? I understand this already proves that hand-crafted statistics can be more stable across domains and can reduce latency. It is similar to using a random mask on all hand-crafted features. Is it possible to adaptively find the optimal selection of hand-crafted features from all candidates? A more detailed discussion of hand-crafted features can benefit future research.

2.        The L_factor reconstructs 10 standardized simulator targets, but their definitions are not clearly shown, nor is how these simulator targets contribute to the final synthesized data. This limits the reproducibility and the justification of whether the simulator is reasonable.

3.        Since (\rho = 1) achieves the best cross-position and cross-dataset performance, it is a fixed hyperparameter rather than being adaptively determined. Why do the authors refer to this mechanism as a "Controlled Departure"? What additional benefits does this explicit norm constraint provide compared to standard feature addition?

Questions for authors

No

4 comments

Thank you for recognizing the value of our transfer tests and IMU-specific simulation. Your point that clearer feature descriptions could help future research is well taken. We have shared an anonymous code release and an anonymous Research Story, and will also make the related paper sections clearer.

Feature choice

Why not use all 10,098 candidates? The main reason is model size. Using all 10,098 statistics in our current correction network would make the final layer alone have over 100 million weights (Appendix B). So we randomly selected 128 features using seed 0 and kept the same selection across all datasets. We also wanted to check whether this choice would affect the results. We tried 64, 128, and 256 features, as well as five different random selections of 128 features. In every setting, HALO improved over its corresponding anchor (Appendix C, Tables 14 and 15).

Could we choose the 128 features adaptively? We think this is a promising direction. For this study, we wanted to understand how well a single, fixed anchor could work across different datasets, so we kept the feature selection unchanged. But we agree that some features may be more useful than others, depending on the setting.

To explore this, we ran an additional ablation looking at different groups of IMU features. We found that rotation-invariant features alone achieved 48.13 cross-position macro-F1, quite close to 49.12 with the full anchor. Meanwhile, axis-specific and pairwise-Gram features achieved 82.90 cross-subject macro-F1, compared with 83.88 for the full anchor.

This gives us a better idea of what an adaptive selector might look for. Different feature groups seem to capture complementary information, and their usefulness can depend on the type of domain shift. We could use this insight to choose a more suitable anchor while still letting HALO learn a bounded correction around it.

Thank you for raising this question. It helped us take a closer look at the feature selection, and we will include these additional results and discussion in the revision.

Simulator factors

Thank you for asking how these factors help train HALO. HAR-CauKer first generates motion and then renders the corresponding IMU signals using orientation and gravity (Section 3.3 and Appendix B, Eq. 5). From each generated example, we compute ten known properties, such as rhythm, turning, tilt, and graph depth. These serve as additional training targets for the encoder, helping it capture motion information even when sensor conditions change. Importantly, they do not affect how the synthetic IMU signals are generated.

We also checked how much this training objective helps. Removing the factor loss reduces cross-dataset macro-F1 by 0.86 points, while removing the variation in sensor conditions leads to a larger drop of 4.82 points (Table 13).

Your question helped us realize that this distinction was not clear enough in the paper. We have added Figure 2 to illustrate the process, and the anonymous Research Story gives a fuller illustrated explanation. We will add the exact formulas and scaling to Appendix B to make the process easier to reproduce.

Radius control

The radius gives us a simple way to adjust how much HALO can move away from its statistical anchor. For each IMU window, the network proposes a correction, and the radius limits its size to ρ||a(x)||₂ without changing its direction (Eq. 3). We train HALO with ρ=1, but we can also try different radii with the same frozen encoder, without retraining it.

We tested how this affects performance. For cross-subject transfer, reducing the radius to ρ=0.2 improves macro-F1 from 83.4 to 84.4. For cross-position and cross-dataset transfer, increasing it to ρ=10 gives 59.1/66.5, compared with 58.8/65.7 at ρ=1 (Table 4). We also trained a separate model without any bound, which performed 6.7/5.2 points worse on these two transfers (Table 3).

So when we say the correction is "controlled," we simply mean that we can set a clear limit on how far the representation moves from its anchor. The results also suggest that this flexibility can be useful for different types of domain shift.

Thank you again for taking the time to go through these questions with us. They helped us see where our explanation could be clearer.

Reviewer_QMN4OpenReview ↗

Official Comment of Reviewer QMN4

Thank you for the detailed responses and additional clarification and experiments. Most of my concerns are answered and I will lean to maintain my original score.

Thank you for taking the time to follow up. We’re glad our responses addressed most of your concerns, and we sincerely appreciate your thoughtful review.

Reviewer_oWn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HALO learns wearable motion representations by adding a bounded neural correction to a fixed handcrafted statistical anchor. The model is pretrained on synthetic motion signals and evaluated across subjects, sensor positions, and datasets. The results show a strong anchor-only baseline and further improvements from learned correction, particularly under position and dataset shifts.

Critical strengths
  1. The paper provides evidence that handcrafted statistical representations can serve as effective foundations for cross-dataset transfer. The strong anchor-only results and additional improvements from HALO support retaining statistical structure when learning transferable representations.

  2. The evaluation goes beyond recognition accuracy. Classifier-free retrieval, direction interventions, and comparisons with concatenation and unbounded correction provide useful evidence about the proposed representation.

Critical weaknesses
  1. The methodological novelty relative to existing hybrid approaches is insufficiently established. Combining handcrafted features with learned adaptation is an established direction, including rTsfNet, which the paper cites. The distinction rests primarily on bounded additive correction, whose advantage over closely related hybrid approaches remains insufficiently demonstrated.

  2. Since the correction uses both the anchor and learned features, it is unclear whether its benefit comes from additional signal information or simply a nonlinear transformation of the anchor.

  3. The anchor substantially outperforms CATCH22, but the source of this advantage is not isolated. The dimension and coordinate-selection studies do not explain which aspects of descriptor construction account for the gap, leaving the contributions of descriptor design and learned correction insufficiently distinguished.

Questions for authors
  1. What is the main methodological contribution beyond existing handcrafted–learned hybrid approaches: the anchor construction, bounded correction, or synthetic pretraining? Which comparisons best isolate this contribution?

  2. How does performance change if the correction is predicted from the anchor alone, without the temporal/spectral features, while keeping the training objective and bounded composition unchanged?

  3. Which aspects of the anchor construction account for its substantial advantage over CATCH22?

  4. Does HALO’s advantage persist under a random train/test split within the same dataset, without holding out subjects? This would clarify whether transfer robustness involves a trade-off with in-distribution recognition performance.

4 comments

Thank you for the careful review. You recognized the strength of the anchor and asked a useful follow-up: what does learning add beyond it? Your anchor-only suggestion led to a new control that makes this contribution directly measurable.

1. What is new relative to other hybrid methods?

Thank you for the insightful comparison with rTsfNet. It brings out HALO's central choice: a fixed statistical representation serves as the reference for a bounded learned correction. Earlier hybrids also demonstrate the value of bringing signal knowledge into learning, and HALO develops that idea in a different part of the representation.

Google's SensorLM uses statistical descriptions to support sensor-language learning. rTsfNet is a strong IMU HAR method that learns multiple input-dependent 3D rotations, recomputes handcrafted features in each resulting view, and fuses the views for recognition. Its network changes where the statistics are measured.

HALO instead computes a fixed anchor once and directly corrects that existing description in its standardized feature space. Equation (3) limits the correction, and radius zero returns the original anchor (Sections 3.1 and 3.2). The illustrated method explanation shows this mechanism. This controlled, reference-relative correction is the methodological contribution we test. In the structural controls, direct concatenation reaches 50.2/59.0 and a separately trained unbounded correction 52.1/60.5, while bounded HALO reaches 58.8/65.7 cross-position/dataset macro-F1 (Section 7.1, Table 3).

Table 1 uses one pooled transfer benchmark. rTsfNet's released configuration and input mapping are dataset-specific, and its classifier assumes a fixed layout and label set. Evaluating it on our pooled sources would require a new input mapping and source-only retraining. We therefore discuss rTsfNet as an important conceptual comparator and reserve Table 1 for methods evaluated under the specified transfer protocols.

2. Does the correction use information beyond the anchor?

Your question identifies a direct test of what the temporal and spectral branches contribute. Following your suggestion, we trained an anchor-only correction that predicts from the anchor without those branches. It uses the same synthetic corpus, three pretraining objectives, correction radius, and downstream readout as full HALO. The table reports the control's mean over pretraining seeds 1, 2, and 3:

RepresentationCross-subjectCross-positionCross-dataset
Fixed anchor83.949.159.5
Anchor-only correction, new control84.0552.8060.89
Full HALO84.956.363.4

All values are macro-F1 on the 0 to 100 scale. The fixed-anchor and full-HALO rows are the headline results in Table 1. Learning from the anchor alone adds 0.15/3.70/1.39 points. The full model adds a further 0.85/3.50/2.51 points relative to that control. These are descriptive comparisons with the Table 1 headline values. Together, they show two useful contributions under the tested architecture: an anchor-based remapping and additional benefit from the temporal and spectral pathways. We will document the control's architecture and settings in the revised manuscript and anonymous code release.

3. Why does the anchor exceed CATCH22?

Thank you for drawing attention to CATCH22, a well-established feature set. It is a valuable general-purpose reference, so we used its official per-channel implementation. Your question led us to test the source of the gap directly. The main finding is that IMU three-axis geometry matters, beyond feature count or overall scale.

Add one component to the CATCH22 reference.

Change from CATCH22Cross-subjectCross-positionCross-dataset
CATCH22 (reference)76.0839.9253.21
+ two log-scale values79.6143.2455.78
+ rotation-invariant triad features82.1549.5959.36
+ recorded-axis and Gram features84.3846.5359.72

Remove one component from our full-anchor reference.

Change from full anchorCross-subjectCross-positionCross-dataset
Full anchor (reference)83.8849.1259.48
− log-scale83.1949.0659.48
− rotation-invariant features (axis/Gram only)82.9042.1556.48
− axis/Gram features (invariant only)77.2948.1356.53
− 63 waveform coordinates per triad (equal width, five draws)82.95 ± 0.5948.67 ± 1.0059.14 ± 0.27

(All scores use the same held-out splits and source-trained 300-tree forest, with each change measured against the reference at the top of its table. The equal-width anchor matches CATCH22 at 132 features per six-channel window, and ± is the standard deviation across five draws.)

The mirror comparisons show what matters. Adding rotation-invariant features raises CATCH22's cross-position score from 39.92 to 49.59, while removing them lowers the anchor's score from 49.12 to 42.15. Across subjects, adding axis/Gram features raises CATCH22 from 76.08 to 84.38, while removing them lowers the anchor from 83.88 to 77.29. The width-matched anchor still exceeds CATCH22 across all three protocols, and scale changes have smaller effects. A label-free rotation test supports the geometry finding: matching-window retrieval is 77.7%/69.3% for the full anchor versus 20.7%/17.6% for CATCH22 on RealWorld/DSADS. We will add this full ablations and per-dataset results to the revision.

4. What about a random within-dataset split?

Your question helps distinguish seen-person recognition from transfer to new people. Randomly dividing individual windows can put recordings from the same person on both sides. Overlapping windows may share raw samples, while nearby or simultaneous non-overlapping windows can share activity cycles and session conditions. A prior HAR analysis found that this dependence can make subject-dependent scores optimistic. Our subject-held-out folds keep every window from one person on the same side, including overlapping windows and different sensor positions. Cross-position transfer also holds out both person and site (Appendix A.4, lines 940 to 949). A random window split measures an easier seen-person setting. Our claims concern the cross-subject, cross-position, and cross-dataset shifts that the reported protocols directly test.

Reviewer_oWn3OpenReview ↗

Appreciate the authors' response - well explained

Thank you for the additional experiments. The anchor-only correction control and descriptor ablations address my main empirical concerns. While I still find the methodological novelty modest, the paper presents a clear approach to refining a strong statistical representation, supported by the transfer results. I am willing to change my score.

Thank you so much for your kind words!

We’re really glad our additional experiments addressed your concerns. We truly appreciate your thoughtful feedback and encouragement!

Reviewer_5xVQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To improve the cross-domain generalizability of wearable activity recognition across different sensor placements and datasets, this paper introduces HALO, a lightweight representation framework pretrained in a physically coupled synthetic motion environment (HAR-CauKer) that represents inertial signals by adding a bounded, input-dependent neural correction to a fixed anchor of handcrafted signal statistics.

Critical strengths
  1. This paper presents an interesting idea of utilizing synthetic data and limited offsets to exploit known statistical features, thus integrating domain knowledge.

  2. The paper benchmarks against SOTA methods, MOMENT, LIMU-BERT, RelCon

Critical weaknesses
  1. The HAR-CauKer synthetic render is a key contribution (as seen in line 106) but is very underspecified. Section 2.3 states that it uses “Gaussian-process kernels within causal programs to generate structured time series”, but this is the extent of an explanation of how it works. There is a “nothing “ sentence that states “HAR-CauKer brings the two together by driving physically coupled inertial rendering with programmable latent motion and using the known motion factors during pretraining.” … but none of these details are ever expounded upon. This seems to be a key reason why the HALO methodology works, with the authors stating “Because the simulator exposes the factors that generated each observation, we can ….” and yet there is no further information of how it works. I checked the appendix and there are no further details and no existing code release to explain it either. It seems that from Line 260, the primary baselines, RelCon and LIMU-BERT were pre-trained on this source dataset as well? However, this remains unclear.

  2. Due to the missing details of the synthetic data generator, many of the subsequent methodology details in Section 3.1 do not make sense. In line 184, the authors state “Of its 10,098 candidate coordinates, we sample once with seed 0 and append log s(x).” … without details of how the synthetic renderer works, this sampling procedure leaves much to the imagination. Furthermore, Equation (1)'s definitions of scale and shape are underspecified and do not explain why these functions, such as s(x) or x tilde, were designed in this way. Once again, the entire HALO methodology is designed to be “limited offsets” from statistical features, but the design of the defined statistical features is never explained nor cited from some other source, instead it is assumed to be fact without any further context.

  3. There seems to be very little lift from the actual “limited offsets” LO component of HALO. Table 1 shows that simply utilizing the anchors generally allows for second (or near second) best performance. This implies one of the primary contributions of this paper is the anchor definitions, but…. with the underspecification of the intuition of actual anchors as detailed in weakness 2, this draws much of the paper into question as to what exactly the takeaway of the paper is. I am somewhat concerned that the anchors also do so well by themselves, easily beating out all of the other FMs and SOTA methodologies. It is worth noting that the benchmarks may be unfairly crippled, with Relcon only trained with 2000 steps, but it is unclear without a loss curve or some other comparison.

Questions for authors

see weaknesses.

2 comments

Thank you for looking closely at the synthetic generator, the anchor, and the baseline training. These details matter for judging HALO's transfer results. We added Figure 2 and shared the anonymous code and an illustrated explanation to make the full path easier to follow. HALO learns from synthetic motion, freezes the encoder, and then fits only a classifier using real source-domain labels (Sections 3.3 and 3.4).

1. Why we use HAR-CauKer

The anchor already works well. HALO keeps it and learns a bounded correction for what the anchor misses (Sections 3.1 and 3.2, Table 1). This raises a training question. When two real recordings differ, did the motion change, or did the wearer, mounting, or device change? Those effects are hard to separate in collected data. HAR-CauKer gives us two views of one motion under different recording conditions, so the encoder can learn which differences to ignore (Section 3.3, Fig. 2).

Each of 3,000 fixed causal programs has 14 nodes and produces 48 motion instances. Six selected trajectories describe angular velocity and world-frame acceleration. A shared orientation then produces related gyroscope, linear-acceleration, accelerometer-with-gravity, and magnetometer signals from that motion (Appendix B, Eq. 5). We vary sampling, mounting, calibration, and noise to make two views of each sensor triad. The contrastive task teaches the encoder that the two views came from the same motion. The program label and ten known factors ask it to keep differences between motions. No real activity labels enter pretraining (Section 3.3 and Appendix B, Eq. 6). The illustrated explanation and generator code show the full process. Removing the recording changes lowers cross-dataset macro-F1 by 4.82 points (Table 5 and Appendix B.1).

2. How we build the anchor

The same fixed feature extractor reads every synthetic and real window. ROCKET and MiniROCKET showed how useful a broad bank of fixed features can be. We use a similar rule for IMU signals. We combine a short list of signals, time scales, statistics, and time or frequency views (Section 2.1 and Appendix B):

Fixed constructionCount
11 scalar signals × 2 domains22 combinations
Per combination: 9 whole-window statistics + 5 segment counts × (9 statistics × 6 aggregations + 4 statistics × 9 amplitude encodings)459
Total22 × 459 = 10,098 candidates

The 10,098 calculations are a pool, not the size of HALO's output. We drew 128 coordinates once with seed 0, added one log-scale value, and used that same anchor for every dataset. Target labels and scores played no role in the choice (Section 3.1 and Appendix B). We also tested widths of 64, 128, and 256 and five different draws at 128. HALO beat the matching anchor throughout those tests, so the gain does not rest on one lucky draw (Appendix C, Tables 14 and 15). Equation (1) separates waveform shape from overall strength and keeps that strength in the log-scale value.

3. What learning adds to the anchor

Strong hand-built features have appeared before. On UCI HAR, rTsfNet reports 96.37% accuracy for handcrafted features plus SVM versus 97.76% for its neural model. On PAMAP2, Yuan et al. report 0.753 F1 for statistics plus random forest versus 0.789 for a model pretrained on 700,000 person-days and then fine-tuned. On UCR, Mantis reports 79.69% for Catch22+, above 77.89% for MOMENT, pretrained on 13 million series. These studies use different protocols. They explain why we take the anchor seriously.

Table 1 puts HALO anchors directly beneath HALO. With the same source-trained random forest, the anchor scores 83.9/49.1/59.5 across subjects/positions/datasets. HALO scores 84.9/56.3/63.4, adding 1.0/7.2/3.9 macro-F1 points (Table 1 and Appendix A.6). HALO leads all six position-transfer directions and improves on the anchor in 12 of 13 held-out datasets (Tables 21 and 22). The subject gain is small. The larger gains appear when the position or dataset changes. When we shuffle the learned correction across windows, classifier-free retrieval drops sharply. Simply joining fixed and learned features also falls short of bounded correction (Tables 2 and 3). The anchor is strong. HALO adds its clearest gains when the recording conditions change.

4. How we pretrained the baselines

HALO is the only method pretrained on HAR-CauKer. We used released pretrained weights for MOMENT, Mantis-8M, MantisV2, and UniMTS. For methods trained on each benchmark source, we followed the corresponding benchmark setups (Section 4 and Appendix A.6). RelCon has no released billion-segment weights, and LIMU-BERT's available checkpoints overlap the evaluation data. We trained those two methods with their own objectives on each split's unlabeled source windows. Held-out windows were excluded. Frozen-feature comparisons used a common source-trained random forest, and every method saw the same source and target windows within each split (Section 4 and Appendix A.6).

Your question about RelCon's training budget prompted a longer run. All other settings stayed the same. The mean macro-F1 scores on a 0 to 100 scale are:

RelCon runCross-subjectCross-positionCross-dataset
Submitted: 3,000 motif steps, 2,000 encoder steps59.232.543.0
New: 8,000 motif steps, 40,000 encoder steps61.13 (≈+1.9)32.72 (≈+0.2)42.23 (≈-0.8)

The new run improves cross-subject performance, while the two transfer scores stay close to the submitted results. Changes are approximate because Table 1 rounds the submitted scores to one decimal place. These numbers describe source-only RelCon training. We will add the new settings and per-unit results to the revised paper and experimental record.

Thank you again for emphasizing clarity and reproducibility. We share your view that both help others verify and build on research. We have added Figure 2 and organized the anonymous code and Research Story in response, and we will keep improving the related parts of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Discussion Summary

We sincerely thank the Area Chair and all three reviewers for the time and care they put into our paper, especially during such a busy review season. We are particularly grateful to QMN4 for recognizing the value of our transfer benchmark, IMU-specific simulation, and results, and to oWn3 for suggesting experiments that genuinely made our work stronger. We were especially happy to hear that these experiments addressed oWn3's main concerns and that oWn3 was willing to raise their score. We also thank 5xVQ for highlighting the importance of clarity and reproducibility. We know everyone has many papers to review, and it means a lot to us that you took the time to look closely at ours.

Thanks to your feedback, we added anchor-only controls and descriptor ablations, providing stronger evidence for HALO's +7.2/+3.9 macro-F1 gains over the statistical anchor across position and dataset shifts. We also added Figure 2, shared our anonymous code and illustrated Research Story, and extended RelCon's training to check the fairness of our comparisons. We believe HALO offers a useful direction for learning transferable motion representations by building on reliable statistical features. We have put a lot of care into this work, and your feedback has helped us make it better. Thank you again for taking our research seriously.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.