acceptodds
Under review as a conference paper at ICLR 2027

: Learning Multiple Time-Frequency Latents for Diverse Accelerometer Tasks

Abstract

Wearable accelerometers support many applications, but the signal characteristics relevant to each task differ greatly. Existing self-supervised learning methods often encourage invariance to selected signal characteristics or aggregate each window into a single embedding, potentially discarding information required by some tasks. We present , a self-supervised method for learning transferable representations of accelerometer signals. decomposes raw signals into frequency bands and uses iterative competitive cross-attention with learned queries to organize temporal and spectral features into multiple time-frequency latents. Two auxiliary reconstruction losses further encourage the latents to preserve local structure and high-frequency variations. To apply the learned representation to a target task, we keep the encoder fixed and train a lightweight prediction head that combines the latents. We evaluate on 11 classification and regression tasks that broaden downstream accelerometer SSL evaluation beyond standard human activity recognition. achieves the best result among the compared self-supervised methods on 8 tasks and performs comparably on the rest. Analysis of task-head weights and performance under individual-latent masking further shows that tasks rely on different combinations of the learned latents.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
accelerometer, self-supervised learning, time-series representation learning, time-frequency representations, multi-latent representations, wearable sensing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_UrykOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Tile, a self-supervised learning approach for pretraining a network to extract temporal and frequency features of accelerometer signals for multiple downstream tasks. Tiles decomposes raw accelerometer signals into three frequency bands and processes them using separate encoders. The extracted features are then aggregated into multiple time-frequency latents through iterative competitive cross-attention with learnable queries. Band-specific reconstruction losses, including MSE, SSIM, and MS-STFT, are used in the reconstruction-based pretraining phase. Task-specific heads are used for different downstream tasks. Evaluation is conducted on 11 classification and regression tasks. Results show that Tiles achieves strong performance, while ablation studies investigate the effectiveness of its key design choices.

Critical strengths

[+] The paper studies a meaningful problem in accelerometer self-supervised learning. Different downstream tasks may rely on different temporal and spectral characteristics of accelerometer signals. Learning transferable representations that preserve signal information in different frequency bands is a practical research problem. The motivation is clearly presented and aligned with the proposed Tile method.

[+] The paper provides a diverse downstream task study beyond conventional human activity recognition benchmarks. The study on diverse task types helps assess the transferability of learned representations across tasks with different signal requirements.

Critical weaknesses

[-] The methodological novelty is relatively limited. The proposed Tile framework mainly integrates existing techniques, including frequency decomposition, Slot Attention-based aggregation, and frequency-specific losses. While their combination for accelerometer representation learning is potentially useful, the paper does not introduce a fundamentally new learning mechanism.

[-] The performance gap between Tile and the supervised baseline is insufficiently discussed. Although Tile achieves competitive performance against existing SSL methods, it underperforms the supervised baseline on 7 out of 11 downstream tasks, including all four regression tasks. Some performance gaps are substantial, particularly on JumpPower and Writing. However, the paper mainly highlights the tasks where Tiles outperforms the supervised baseline, without adequately discussing the reasons for its performance on the remaining tasks. A more thorough analysis would help clarify the limitations of the proposed approach and their transferability across different tasks.

Questions for authors
  1. How much does frequency decomposition contribute compared with the band-specific reconstruction losses? Table 3 shows that removing frequency decomposition leads to only a 0.5% average performance decrease on regression tasks, whereas removing the auxiliary reconstruction losses causes a much larger decrease (33.7%). Would applying the same frequency-aware reconstruction losses to a model that directly encodes raw signals achieve comparable performance?

  2. Can the authors further explain why Tiles underperforms the supervised baseline on 7 of the 11 downstream tasks, including all four regression tasks? In particular, is this performance gap mainly caused by the limitations of the learned representations or by the frozen-encoder adaptation setting? Have the authors evaluated full fine-tuning of the pretrained encoder to distinguish these possibilities?

  3. How does Tiles compare with related multi-view and time-frequency representation learning methods mentioned in the paper, such as localized time-frequency contrastive learning (Taghanaki et al., 2023) and TSPulse (Ekambaram et al., 2026), under comparable pre-training and downstream evaluation settings?

5 comments

We thank you for your thoughtful review and valuable questions that have helped us improve our paper. We address your concerns and questions point by point below.


W1: Contribution and supporting evidence.

Tiles’ contribution is a representation that separates accelerometer signals into multiple time-frequency encodings, allowing downstream tasks to use different combinations of them. Frequency decomposition, competitive aggregation and band-specific reconstruction work together to learn this structure.

  • Multiple encodings at fixed size: Figure 7 keeps the total representation dimension fixed at 256 and retains the same frequency decomposition and band-specific reconstruction losses. Eight latents give higher average classification performance and lower average regression error than a single latent, without increasing the total representation dimension.
  • Distinct and stable roles: We analyzed 6,200 held-out windows from 31 CAPTURE-24 participants. The dominant slot accounts for 94.7% of each time–band cell’s energy on average. NMI between slot identity and its dominant frequency band and temporal region is 0.970 and 0.901, respectively. These results provide direct evidence of consistent time-frequency roles.
  • Benefits of the aggregation design: Controlled ablations retain the encoders, multiple latents, decoders, losses and downstream head. Removing competition reduces performance by an average relative change of 0.57% across the 11 tasks, while reducing aggregation from three iterations to one reduces it by 2.52%. Full Tiles has better mean performance on 6/11 and 7/11 tasks in the respective pairwise comparisons.

W2/Q2: Supervised comparison and full fine-tuning.

It is common for frozen SSL encoders to underperform end-to-end supervised models on some downstream tasks (Xu et al., 2025; Haresamudram et al., 2022). This is because supervised training adapts the entire encoder to each task, while frozen evaluation trains only the prediction head. To test how much freezing contributes to our performance gaps, we ran full fine-tuning of Tiles on all 11 downstream tasks.

TaskTiles frozenTiles fine-tunedSupervised
Basketball67.1 ± 4.374.6 ± 4.262.2 ± 3.4
Cooking55.5 ± 3.659.6 ± 4.860.6 ± 4.4
Locomotion73.6 ± 3.174.1 ± 3.872.4 ± 2.1
TennisShot82.3 ± 4.085.7 ± 2.779.3 ± 3.2
Transportation62.8 ± 4.268.9 ± 7.356.4 ± 6.9
Workouts71.0 ± 4.176.5 ± 3.873.6 ± 5.1
Writing49.0 ± 4.657.8 ± 3.755.6 ± 9.5
JumpPower6.46 ± 0.794.81 ± 0.863.88 ± 0.28
NumSteps0.69 ± 0.130.56 ± 0.090.65 ± 0.16
ThrowSpeed3.03 ± 0.422.79 ± 0.362.73 ± 0.21
WalkDistance0.29 ± 0.050.24 ± 0.050.25 ± 0.07

Fine-tuning improves Tiles’ mean performance on every task and achieves a better mean than the supervised baseline on 8/11 tasks, including Writing. Cooking, JumpPower and ThrowSpeed still favor supervised training. These results support the frozen-encoder setting as a contributor to the original gaps, while showing room for improvement on the remaining tasks.

We will include the full fine-tuning results in the appendix.

References

Xu, Maxwell A., et al. “RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data.” International Conference on Learning Representations, 2025.
Haresamudram, Harish, Irfan Essa, and Thomas Plötz. “Assessing the State of Self-Supervised Human Activity Recognition Using Wearables.” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 3, 2022, article 116, pp. 1–47.

Q1: Frequency decomposition and reconstruction losses.

The “No frequency decomposition” ablation in Table 3 already evaluates a raw-input model with MSE, SSIM and MS-STFT. It uses a single 12-block ResNet encoder and applies these losses to the full-band reconstruction. This model achieves comparable average regression performance to Tiles, while classification performance drops by 5.9% on average.


Q3: Comparisons with multi-view and time-frequency methods.

We evaluated TSPulse (Ekambaram et al., 2026), which learns temporal, spectral and semantic representations, on all 11 downstream tasks. We also included HiMAE (Lee et al., 2026) as a hierarchical representation-learning comparison.

  • HiMAE: The pretrained weights used in the HiMAE paper are not publicly available. We therefore adapted HiMAE-Base to three-axis accelerometer inputs and pretrained it using the same CAPTURE-24 split and pretraining budget as Tiles. We retained its hierarchical encoder–decoder, five-sample patches, 80% masking and per-window normalization to [-1,1]. The autoencoder has approximately 1.25M parameters, and its encoder provides a 256-dimensional representation.

  • TSPulse: We used IBM’s tspulse-block-dualhead-512-p16-r1 checkpoint, recommended for classification. It was pretrained on approximately one billion time points from Monash and LibCity, covering energy, weather, traffic and other time series. We linearly interpolated our 250-sample inputs to 512, disabled masking and retained the full temporal, FFT and register-token embeddings across all three axes. Its frozen backbone and decoder contain approximately 0.37M parameters, compared with Tiles’ 4.8M. However, its full representation contains 9,936 features, compared with Tiles’ 256, meaning that the same MLP configuration gives TSPulse a much larger task-specific MLP head.

Both models were evaluated with frozen representations and using the same folds. We have not evaluated the localized time-frequency contrastive method of Taghanaki et al. (2023).

TaskHiMAETSPulseTiles
Basketball ↑60.260.567.1
Cooking ↑53.544.355.5
Locomotion ↑69.763.173.6
TennisShot ↑78.083.182.3
Transportation ↑52.652.162.8
Workouts ↑67.848.971.0
Writing ↑26.454.449.0
JumpPower ↓7.814.866.46
NumSteps ↓0.8250.6440.69
ThrowSpeed ↓4.462.863.03
WalkDistance ↓0.4090.3470.29

Tiles outperforms HiMAE on all eleven tasks and TSPulse on six, including five of the seven classification tasks. TSPulse leads on TennisShot and Writing, but trails Tiles by 22.1 points on Workouts, 11.2 on Cooking and 10.7 on Transportation. It achieves lower RMSE than Tiles on three of the four regression tasks.

These differences may reflect pretraining domain and representation choices. TSPulse is pretrained on diverse general time series, normalizes each window and axis using RevIN, and exposes 9,936 temporal, spectral and register-token features. Its detailed features may help estimate quantities related to timing, repetition and waveform shape. Tiles instead uses a compact 256-dimensional representation learned from wearable accelerometer data, which may better capture the movement patterns needed for activity classification.

References

Lee, Simon, et al. “HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series.” International Conference on Learning Representations, 2026.
Ekambaram, Vijay, et al. “TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis.” International Conference on Learning Representations, 2026.

Thank you for your reply, which has partly addressed my concerns. I still have some questions and would appreciate a brief clarification.

Regarding Figure 4, while the low-frequency trends seem to be reasonably reconstructed, some periodic patterns, particularly the ECG-like pattern in the second subfigure, appear less well preserved. Could the authors briefly explain why this might be the case? Does this suggest that the model reconstructs different frequency bands with varying accuracy? Also, could the authors provide some quantitative frequency-domain reconstruction metrics for the different frequency bands to better assess how well these patterns are preserved? A brief comparison with the baselines would also be helpful.

Thanks again for your efforts.

Reviewer_PNsROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework that models multiple latent representations to capture diverse inductive biases in wearable accelerometer data, such as rotational variations. The method first decomposes the input signals into three components using spectral cutoffs and optimizes the model with a reconstruction loss and two additional self-supervised objectives. Experiments across 11 classification and regression tasks demonstrate that the proposed method outperforms competing approaches, supporting its effectiveness across a range of wearable sensing tasks.

Critical strengths

The paper provides a clear motivation for moving beyond invariance-based self-supervised learning objectives, such as augmentation-based methods, by highlighting that task-relevant invariances depend on the downstream task. This motivates the use of multiple latent representations to capture diverse inductive biases. Furthermore, the evaluation extends beyond conventional human activity recognition (HAR) benchmarks to a broader range of tasks.

Critical weaknesses

The main concern is the limited technical novelty of the proposed method. Most of its components are adopted from prior work with relatively minor modifications. Although combining existing techniques can constitute a valid contribution, the benefits of their integration are not sufficiently established.

Furthermore, the Introduction motivates Tiles by contrasting it with methods that compress each window into a single global embedding. However, during inference, the (K) latents are combined through a self-attention layer and flattened into a 256-dimensional vector before being passed to the MLP, matching the dimensionality of the global embeddings used by the baselines. This raises questions about whether Tiles provides a sufficiently distinct representational advantage over existing approaches and whether the proposed multi-latent design offers benefits beyond those of conventional global embeddings.

Questions for authors
  1. Could the authors clarify what the ± values in Table 3 represent? If they denote standard deviations, could the authors discuss the relatively large variability observed?

  2. Since the motivation links to several invariances such as rotation, I'm curious whether there are distinct latent patterns that capture those inductive biases, like rotation, after training.

  3. On the experiment about Figure 6, because total parameters are held fixed, per-band encoder depth falls from 12 ResBlocks (one band) to a single ResBlock (twelve bands). The authors themselves attribute part of the degradation to insufficient per-band capacity. Could they repeat the experiment with a fixed per-band encoder (e.g., the main model's four-ResBlock encoder), letting total parameters grow with the number of bands?

  4. Although HiMAE and TSPulse are cited as relevant baselines, they are not included in the main experimental comparisons. Could the authors clarify this omission or provide comparisons with these methods to strengthen the empirical evaluation?

2 comments

We thank you for your thoughtful review and valuable questions. We address your concerns and questions one by one below.


W1: Contribution and evidence for the integrated design.

Tiles’ contribution is a representation that separates accelerometer signals into multiple time-frequency encodings, allowing downstream tasks to use different combinations of them. Frequency decomposition, band-specific reconstruction and iterative competition with queries shared across inputs work together to learn this structure.

  • Integrated design: Replacing iterative competitive aggregation with a standard autoencoder, while retaining frequency decomposition and the auxiliary reconstruction losses, reduces performance by 8.2% on classification and 6.1% on regression on average (Table 3). This supports the benefit of organizing the representation into multiple latents beyond the reconstruction objectives alone.
  • Distinct and stable roles: We analyzed 6,200 held-out windows from 31 CAPTURE-24 participants. The dominant slot accounts for 94.7% of each time–band cell’s energy on average. Normalized mutual information between slot identity and its dominant frequency band or temporal region is 0.970 and 0.901, respectively. These results provide direct evidence of distinct and consistent time-frequency roles.
  • Competition and iteration: We ran two additional ablations that retain the encoders, latent count and dimensions, decoders, losses and downstream head. Removing competition by changing the attention softmax from the slot dimension to the input-frame dimension reduces performance by an average relative change of 0.57% across the 11 tasks. Reducing aggregation updates from three to one reduces it by 2.52%. Both averages give equal weight to each task. Full Tiles has better mean performance on 6/11 and 7/11 tasks in the respective pairwise comparisons, supporting a larger average benefit from repeated updates than from competition.

W2: Why multiple latents help at the same representation size.

We keep the total representation dimension at 256 to match the baselines. Tiles’ latents remain separate through downstream self-attention and are concatenated afterward, allowing the head to combine information across the learned encodings.

Figure 7 compares one and eight latents at this fixed total dimension, using the same frequency decomposition and band-specific reconstruction losses, including MSE, SSIM and MS-STFT. Eight latents give higher average classification performance and lower average regression error than one latent. This supports learning multiple encodings within the same dimensional budget.

Q1: Meaning of the standard deviations in Table 3.

The ± values are standard deviations of relative performance changes across tasks, calculated separately for the seven classification tasks and four regression tasks. The large spread reflects differences in task sensitivity to each ablation. For example, stochastic query initialization reduces macro-F1 from 49.0% to 29.4% on Writing and from 55.5% to 40.0% on Cooking, while Locomotion remains nearly unchanged at 73.6% versus 73.7%. We will clarify this in the Table 3 caption.


Q2: Latent roles and rotation invariance.

We discuss rotation to motivate preserving information that different downstream tasks may need. We have not identified a clear separation of rotation-related roles among individual latents. Testing their sensitivities to controlled rotations remains future work.


Q3: Fixed per-band encoder capacity.

With four ResBlocks per band, the six- and twelve-band variants show the following relative changes over a three-band run:

BandsEncoder blocksRelative change ± SD
624+6.73 ± 7.38%
1248+5.68 ± 4.79%

Values are mean ± SD across the 11 tasks, with higher F1 or lower RMSE counted as improvements. These runs suggest that more bands can help when per-band encoder capacity is maintained.


Q4: HiMAE and TSPulse comparisons.

We additionally evaluated HiMAE (Lee et al., 2026) and TSPulse (Ekambaram et al., 2026) on all 11 downstream tasks.

  • HiMAE: The pretrained weights used in the HiMAE paper are not publicly available. We therefore adapted HiMAE-Base to three-axis accelerometer inputs and pretrained it on CAPTURE-24. We retained its hierarchical encoder–decoder, five-sample patches, 80% masking and per-window normalization to [-1,1]. The autoencoder has approximately 1.25M parameters, and its encoder provides a 256-dimensional representation.

  • TSPulse: We used IBM’s tspulse-block-dualhead-512-p16-r1 checkpoint, recommended for classification. It was pretrained on approximately one billion time points from Monash and LibCity, covering energy, weather, traffic and other time series. We linearly interpolated our 250-sample inputs to 512, disabled masking and retained the full temporal, FFT and register-token embeddings across all three axes. Its frozen backbone and decoder contain approximately 0.37M parameters, compared with Tiles’ 4.8M. However, its full representation contains 9,936 features, compared with Tiles’ 256, meaning that the same MLP configuration gives TSPulse a much larger task-specific MLP head.

Both models were evaluated with frozen representations and using the same folds.

TaskHiMAETSPulseTiles
Basketball ↑60.260.567.1
Cooking ↑53.544.355.5
Locomotion ↑69.763.173.6
TennisShot ↑78.083.182.3
Transportation ↑52.652.162.8
Workouts ↑67.848.971.0
Writing ↑26.454.449.0
JumpPower ↓7.814.866.46
NumSteps ↓0.8250.6440.69
ThrowSpeed ↓4.462.863.03
WalkDistance ↓0.4090.3470.29

Tiles outperforms HiMAE on all eleven tasks and maintains the highest mean on seven tasks among HiMAE and the baselines in Table 2. With TSPulse also included, Tiles has the highest mean among all considered SSL baselines on four tasks.

TSPulse achieves lower RMSE than Tiles on three of the four regression tasks. Its classification performance is less consistent: it leads on TennisShot and Writing, but trails Tiles by 22.1 points on Workouts, 11.2 on Cooking and 10.7 on Transportation. The differences may reflect pretraining domain and representation choices. TSPulse is pretrained on diverse general time series, normalizes each window and axis using RevIN, and exposes 9,936 temporal, spectral and register-token features. Its detailed features may help estimate quantities related to timing, repetition and waveform shape. Tiles instead uses a compact 256-dimensional representation learned from wearable accelerometer data, which may better capture the movement patterns needed for activity classification.

References

Lee, Simon, et al. “HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series.” International Conference on Learning Representations, 2026.
Ekambaram, Vijay, et al. “TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis.” International Conference on Learning Representations, 2026.
Reviewer_7HQWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Tiles is a self-supervised method for learning transferable accelerometer representations across tasks with different temporal, spectral, and magnitude requirements. It decomposes signals into three frequency bands, processes them with separate encoders, and aggregates their features into multiple latents using iterative competitive attention initialized with learned queries. Pretraining uses a band-specific combination of MSE, SSIM, and multi-scale STFT reconstruction losses. Downstream tasks use a frozen representation and a lightweight prediction head.

The model is pretrained on approximately 2,500 hours of CAPTURE-24 data and evaluated on 11 classification and regression tasks. Under common pretraining data and controlled model and representation sizes, Tiles achieves the best or tied-best result among the compared self-supervised methods on eight tasks. Component ablations and latent-masking analyses examine the design and task-dependent use of the representation.

Critical strengths

The evaluation addresses a meaningful gap in accelerometer representation learning. The 11 tasks extend beyond standard activity recognition and depend on different signal properties. Tiles achieves the best or tied-best SSL result on eight tasks, with substantial improvements on some tasks: Basketball macro-F1 increases from 59.6 to 67.1, and JumpPower RMSE decreases from 8.08 to 6.46 relative to the strongest compared SSL baseline. Shared pretraining data and controlled encoder, representation, and head sizes make these comparisons informative.

The integrated design is coherent and supported by relevant ablations. Frequency decomposition, reconstruction losses, learned queries, aggregation, and the downstream head receive experimental examination. The global-autoencoder ablation retains the frequency decomposition and auxiliary losses, providing a useful control. Latent masking also supports the claim that tasks rely on different parts of the representation. Together, these results substantiate the practical value of the complete approach.

Critical weaknesses

The distinctive contribution of competitive, iterative aggregation remains incompletely isolated. The matched-loss global-autoencoder ablation is valuable, but it changes both the aggregation mechanism and the multi-latent representation structure. Consequently, it does not determine whether competition and iteration improve on a simpler deterministic multi-query attention or pooling method that also retains multiple latents. This limits the evidence for the most distinctive architectural component, even though the complete model performs well.

The robustness and scope of the transfer findings require qualification. Several downstream datasets contain relatively few participants. Transportation and ThrowSpeed use segment- and trial-level folds rather than participant-disjoint evaluation, limiting conclusions about new-user generalization on those tasks. The reported standard deviations quantify downstream-fold variability; independent pretraining-seed repetitions are not reported. The larger gains are encouraging, but the stability of smaller differences and of the pretrained representation remains less clear.

Questions for authors

Can you compare iterative competitive aggregation with a deterministic noncompetitive multi-latent alternative, while keeping frequency decomposition, reconstruction losses, representation dimension, downstream head, and model capacity approximately fixed? This would clarify whether competition and iteration contribute beyond retaining multiple outputs.

How stable are the main results across independent pretraining seeds? Please distinguish pretraining variability from downstream-fold and prediction-head variability. Paired fold-level differences would also help assess the smaller performance margins.

For Transportation and ThrowSpeed, what grouping metadata is available to prevent closely related segments or trials from appearing across folds? Please clarify the intended generalization setting and whether a stricter grouping protocol changes the conclusions, where such a protocol is feasible.

2 comments

We thank you for your thoughtful review and valuable questions. Your feedback has helped us improve the paper. We address your concerns and questions point by point below.


W1/Q1: Isolating competition and iteration.

We agree that the global-autoencoder replacement changes both aggregation and representation structure. We therefore ran two additional ablations that retain multiple latents.

  • Competition: We changed the attention softmax from the slot dimension to the input-frame dimension, allowing each slot to attend independently. The subsequent normalization over frames was unchanged. This retains the learned queries and multiple outputs while removing competitive normalization.
  • Iteration: We reduced the number of aggregation updates from three to one, while retaining competition.

Both variants retained the frequency decomposition, encoders, latent dimensions, decoders, reconstruction losses and downstream head. They used the same pretraining configuration and downstream evaluation protocol.

SettingAvg. changeTasks won
Full TilesReference5/11
No competition−0.57%3/11
One iteration−2.52%3/11

Average change is calculated relative to full Tiles, with equal weight for each task and higher F1 or lower RMSE counted as improvements. Tasks won counts the highest mean among the three settings. In pairwise comparisons, full Tiles outperformed no competition on 6/11 tasks and one iteration on 7/11 tasks.

The effects depend on the task. Writing macro-F1 was 49.0 for full Tiles, compared with 46.0 without competition and 41.3 with one iteration. ThrowSpeed RMSE was 3.03, compared with 3.14 and 3.51, respectively. However, removing competition improved TennisShot from 82.3 to 83.8, and one iteration improved Basketball from 67.1 to 69.0.

These comparisons favor the full configuration overall, with a larger average benefit from repeated updates than from competition.

W2/Q2: Training variability and paired comparisons.

The results use one pretrained checkpoint per SSL method, reused across all five downstream folds, with one downstream initialization per fold. Reusing a pretrained backbone across downstream evaluations is consistent with related work, including Yuan et al. (2024), RelCon (Xu et al., 2025), and TSPulse (Ekambaram et al., 2026). The reported standard deviations are across folds.

We repeated downstream training with five head seeds, 42–46, per fold while keeping each pretrained checkpoint fixed. Tiles achieves the highest five-seed mean in the SSL comparison on 7/11 tasks. It ranks first on every seed for Basketball, Locomotion, TennisShot, Workouts and Writing, and on four of five seeds for JumpPower. Independent pretraining-seed repetitions have not been performed.

We also ran paired statistical comparisons using Table 2 predictions. All methods use identical folds. For the nine participant-disjoint tasks, we ran 10,000 paired participant-bootstrap resamples, using the same sampled participants for both methods and recomputing the fold scores.

Positive gains favor Tiles. Classification gains are macro-F1 percentage points, and regression gains are reductions in RMSE. Gains are mean ± SD across folds. Fold p-values are from uncorrected paired t-tests. Holm correction uses the 11-task comparison family.

TaskBaselineGain ± SDFolds wonFold pBootstrap 95% CIBootstrap pHolm p
BasketballAE+7.6 ± 6.25/50.053[4.6, 10.4]<0.0010.002
CookingSimCLR+2.2 ± 8.83/50.61[−4.3, 10.2]0.541
LocomotionMAE+1.8 ± 5.42/50.51[−2.5, 6.7]0.481
TennisShotAE+3.7 ± 2.35/50.024[0.6, 7.6]0.0150.12
TransportationAugPred−0.7 ± 3.72/50.68———
WorkoutsMAE+1.6 ± 1.94/50.14[−0.7, 4.3]0.201
WritingAE+3.5 ± 2.24/50.025[1.3, 6.0]<0.0010.002
JumpPowerMAE+1.6 ± 1.85/50.11[0.9, 2.4]<0.0010.002
NumStepsMAE−0.05 ± 0.183/50.57[−0.13, 0.03]0.261
ThrowSpeedSimCLR0.00 ± 0.372/50.99———
WalkDistanceRelCon−0.01 ± 0.063/50.66[−0.05, 0.02]0.441

The five-seed rankings and the paired analysis use different sets of runs. In the paired analysis, improvements on Basketball, Writing and JumpPower survive Holm correction. The remaining comparisons are inconclusive.

Transportation and ThrowSpeed use segment- and trial-level folds, so we report only fold-level tests for these tasks. We will include the paired-comparison table and analysis procedure in the appendix, with a brief summary in the main text.

References

Ekambaram, Vijay, et al. “TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis.” International Conference on Learning Representations, 2026.
Xu, Maxwell A., et al. “RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data.” International Conference on Learning Representations, 2025.
Yuan, Hang, et al. “Self-Supervised Learning for Human Activity Recognition Using 700,000 Person-Days of Wearable Data.” npj Digital Medicine, vol. 7, 2024, article 91.

W2/Q3: Grouping and participant-disjoint evaluation.

For Transportation, we keep all windows from each contiguous segment with the same transportation label in one fold. Tiles and all baselines use identical folds, so the results compare methods on held-out segments under the same protocol.

We also evaluated Transportation using three leave-one-participant-out folds, holding out all recordings from one participant and using separate recordings from the other two for training and validation. We compared Tiles with Augmentation Prediction, the strongest baseline in Table 2.

MethodSegment-levelParticipant-disjoint
Tiles62.8 ± 4.221.0 ± 10.0
Augmentation Prediction63.5 ± 7.021.1 ± 5.3

Both methods fall to around 21% under participant-disjoint evaluation, close to the 16.7% chance reference for a balanced six-class problem. Their mean scores remain close under both protocols.

For ThrowSpeed, all windows from a trial stay in the same fold, but the released data do not provide participant IDs for individual trials, preventing a participant-disjoint re-evaluation with the available metadata.

Nine of the eleven downstream tasks already use participant-disjoint folds. We will clarify the split protocols in Table 2, identifying Transportation and ThrowSpeed as segment-level and trial-level evaluations, respectively.

Reviewer_thmFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles a representation learning problem for wrist worn accelerometer data from wearable sensors such as wrist watches. The self-supervised learning is framed hand-crafted features extraction (a form of wavelet decomposition) with an encorder and decoder architecture with a form of attention adapted from slot attention in Locatello et al., 2020 to produce a learned key-query matrix. The idea is to preserve heterogeneous motion information using multiple latent vectors (time-frequency) rather than a single global embedding.

The architecture reconstructs the frequency-decomposed input during pretraining. supplements pointwise MSE reconstruction with band-specific Structural Similarity Index Measure (SSIM), and Multi-Scale Short-Time Fourier Transform (MS-STFT) losses to better preserve local and high-frequency structure.

The pretrained encoder is frozen for downstream evaluation where a lightweight a single multi-head self-attention layer and an MLP head is trained and evaluated. The model is pretrained on a151 participant, CAPTURE-24. It is evaluated 7 classification and 4 regression tasks.

Critical strengths
  • The representation learning for wrist-worn accelerometer data is important. For example, a representation suitable for walking recognition may not preserve the same information required for handwriting or physical-quantity regression

  • The use of 11 downstream tasks. The inclusion of regression is useful because it tests whether learned representations retain continuous magnitude and fine-grained physical information, rather than only class-separable features. Generally, compared to the presented baselines, the results are, overall, positive; tiles produces the best or tied-best SSL performance on eight of the 11 tasks.

Critical weaknesses
  • The evidence for latent specialization is not sufficiently strong. The central conceptual claim is that competitive aggregation organizes information into latents that downstream tasks use differently. However, the main evidence consists of: i) task-head weight magnitudes, ii) performance changes after zeroing one latent, iii) and t-SNE plots of selected latents. These analyses show that the heads are sensitive to different latent coordinates, but they do not establish that the latents have learned distinct, stable, or interpretable time-frequency roles. Notably the authors acknowledge that the analysis does not establish fixed semantic meanings for the latents.

  • The ablation study is useful but it seems to conflate a lot of design changes that range from explicit frequency filtering, three separate encoders, iterative competitive attention, multiple latent vectors, band-specific decoders, SSIM and MS-STFT objectives, and a self-attention downstream head. The study does not provide a strong factorial comparison to isolate benefits and the ablation does not cleanly establish that competition between learned queries is necessary because the “no competitive aggregation” condition described as replacement with a standard autoencoder, may alter more than the aggregation rule.

  • Not evaluated with more recent hierarchical, multi-view, and wearable foundation-model approaches.

  • Comparison fairness requires better reporting, as it stands this is not clear. It is unclear whether the same tuning budget was available to all baselines, were the encoder in the baseline methods swapped for the encorders in Tiles, are the parameters in Table 5 for one of three encoders in Tiles? Then, more generally, the paper reports fold means and standard deviations but no paired significance tests, confidence intervals for method differences. Several comparable results differ only by a small amount relative to fold variability. It seems, for Transportation and ThrowSpeed, participant-disjoint evaluation is not possible, so segment- or trial-level folds are used. This creates a materially easier transfer setting and may allow subject-specific characteristics to occur in both training and test folds. As a result, these numbers should not be treated as equivalent evidence of cross-subject generalization.

  • The model is pretrained only on CAPTURE-24, consisting of wrist-worn data only and using only accelerometer data, while accelerometers and gyroscopes co-exist in a single form factor quiet often with wearables but the paper does not compare pretraining on another source, combining multiple pretraining datasets, or transferring across sensor placements in a controlled manner.

Questions for authors
  1. Are Butterworth filters applied to continuous recordings before windowing or independently to each five-second window? What padding and initial-condition procedure is used to control boundary artifacts?

  2. How are physical frequency cutoffs implemented for TennisShot at 120 Hz and Writing at 100 Hz? Are these inputs filtered with the same cutoffs in Hz, and were frequency-response differences examined?

  3. Are the same folds paired across methods, and if so, can the authors report paired tests or bootstrap confidence intervals for the differences between Tiles and the strongest baseline on each task?

  4. Does each reported fold use one pretraining run and one downstream initialization, or are results averaged across multiple training seeds? The latter seems important given the reported variability and the stochasticity of latent formation.

  5. What was the exact pretraining hyperparameter-search budget for Tiles and for each baseline? Were choices made using only the CAPTURE-24 validation loss, or was downstream validation performance consulted?

  6. Given that only a few (e.g. three) participants are available and segments from a participant can appear across folds, how much performance is attributable to participant or device identification rather than transportation recognition?

  7. Does Tiles retain its advantage with a strictly linear head or a parameter-matched MLP without self-attention?

  8. Can the authors provide improved/stronger evidence for latent specialization?

  9. Can the authors provide stronger ablation study to isolate performance, such as a single-latent model using exactly the same reconstruction losses, a multi-token reconstruction model without competitive normalization, for example?

  10. what would be the performance to a LoRA type tuning of a foundation model with a similar parameter cost?

6 comments

We thank you for your thoughtful review and valuable questions. Your feedback has helped us improve the paper. We address your concerns and questions point by point below.


W1/Q8: Evidence for distinct and stable latent roles.

We agree that task-head weights and latent masking alone do not establish time–frequency specialization. We therefore analyzed 6,200 windows from the 31 held-out participants in CAPTURE-24.

  • Distinctness: For each slot, we measured the energy of its decoder output multiplied by its mask, across time and the three frequency bands. Active slots had a mean time×band overlap of 0.039 (0 means no overlap), and the largest slot contribution accounted for 94.7% of each cell’s energy on average. This shows that different slots contribute to largely separate time–frequency regions within a window.
  • Stability: We measured normalized mutual information (NMI) between slot identity and its dominant frequency band or temporal region. NMI was 0.970 for frequency band and 0.901 for temporal region (p=0.001 for both). For seven of eight slots, at least 87% of participants agreed with the slot’s overall dominant band and region. This shows that slot roles remain consistent across windows and participants.
  • Control: Random slot initialization produced similarly low overlap (0.035) but near-zero association between slot identity and frequency or time. Thus, both models separate contributions within a window, but Tiles also maintains consistent roles for individual slot IDs across windows.

These results provide evidence that Tiles’ slots have distinct and stable time–frequency roles, complementing the original analyses showing that downstream tasks use different combinations of slots.


W2/Q9: Separating the effects of multiple latents, competition and iteration.

We ran two additional controlled ablations to test competition and iteration separately while retaining multiple latents.

  • Single versus multiple latents: Figure 7 already includes a single-latent model trained with the same frequency decomposition and band-specific reconstruction losses as full Tiles, including MSE, SSIM and MS-STFT. We vary the number of latents and adjust their individual dimensions to keep the total representation dimension fixed at 256. The single-latent model gives lower average classification performance and higher average regression error than the eight-latent configuration.
  • Competition: We retained multiple latents and changed the attention softmax from the slot dimension to the input-frame dimension, allowing each slot to attend independently. The subsequent normalization over frames was unchanged.
  • Iteration: We reduced the number of aggregation updates from three to one, while retaining competition.

All other model components, reconstruction losses and training and evaluation settings were unchanged.

SettingAvg. changeTasks won
Full TilesReference5/11
No competition−0.57%3/11
One iteration−2.52%3/11

Average change is calculated relative to full Tiles, with equal weight for each task and higher F1 or lower RMSE counted as improvements. Tasks won counts the highest mean among the three settings. In pairwise comparisons, full Tiles outperformed no competition on 6/11 tasks and one iteration on 7/11 tasks.

  • Task-dependent effects: On Writing, macro-F1 was 49.0 for full Tiles, compared with 46.0 without competition and 41.3 with one iteration. On ThrowSpeed, RMSE was 3.03, compared with 3.14 and 3.51, respectively. However, simpler variants sometimes performed better: removing competition improved TennisShot from 82.3 to 83.8, and one iteration improved Basketball from 67.1 to 69.0.

These comparisons favor the full configuration overall, with a larger average benefit from repeated updates than from competition.

W3: Comparisons with HiMAE and TSPulse.

Our evaluation includes RelCon (Xu et al., 2025), a recent foundation-model approach for wearable motion. We additionally evaluated HiMAE (Lee et al., 2026) and TSPulse (Ekambaram et al., 2026) on all 11 downstream tasks.

  • HiMAE: The pretrained weights used in the HiMAE paper are not publicly available. We therefore adapted HiMAE-Base to three-axis accelerometer inputs and pretrained it on CAPTURE-24. We retained its hierarchical encoder–decoder, five-sample patches, 80% masking and per-window normalization to [-1,1]. The autoencoder has approximately 1.25M parameters, and its encoder provides a 256-dimensional representation.
  • TSPulse: We used IBM’s tspulse-block-dualhead-512-p16-r1 checkpoint, recommended for classification. It was pretrained on approximately one billion time points from Monash and LibCity, covering energy, weather, traffic and other time series. We linearly interpolated our 250-sample inputs to 512, disabled masking and retained the full temporal, FFT and register-token embeddings across all three axes. Its frozen backbone and decoder contain approximately 0.37M parameters, compared with Tiles’ 4.8M. However, its full representation contains 9,936 features, compared with Tiles’ 256, meaning that the same MLP configuration gives TSPulse a much larger task-specific MLP head.

Both models were evaluated with frozen representations and using the same folds.

TaskHiMAETSPulseTiles
Basketball ↑60.260.567.1
Cooking ↑53.544.355.5
Locomotion ↑69.763.173.6
TennisShot ↑78.083.182.3
Transportation ↑52.652.162.8
Workouts ↑67.848.971.0
Writing ↑26.454.449.0
JumpPower ↓7.814.866.46
NumSteps ↓0.8250.6440.69
ThrowSpeed ↓4.462.863.03
WalkDistance ↓0.4090.3470.29

Tiles outperforms HiMAE on all eleven tasks and maintains the highest mean on seven tasks among HiMAE and the baselines in Table 2. With TSPulse also included, Tiles has the highest mean among all considered SSL baselines on four tasks.

TSPulse achieves lower RMSE than Tiles on three of the four regression tasks. Its classification performance is less consistent: it leads on TennisShot and Writing, but trails Tiles by 22.1 points on Workouts, 11.2 on Cooking and 10.7 on Transportation. The differences may reflect pretraining domain and representation choices. TSPulse is pretrained on diverse general time series, normalizes each window and axis using RevIN, and exposes 9,936 temporal, spectral and register-token features. Its detailed features may help estimate quantities related to timing, repetition and waveform shape. Tiles instead uses a compact 256-dimensional representation learned from wearable accelerometer data, which may better capture the movement patterns needed for activity classification.

References

Lee, Simon, et al. “HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series.” International Conference on Learning Representations, 2026.
Ekambaram, Vijay, et al. “TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis.” International Conference on Learning Representations, 2026.

W4/Q3/Q5: Comparison setup, pretraining tuning and paired evaluation.

  • Pretraining and tuning: We did not sweep general pretraining hyperparameters, such as learning rate, optimizer or regularization settings, for any method. For RelCon and Augmentation Prediction, we used the official implementations and their method-specific settings, adapting the encoder and batch size to our shared comparison setup. Therefore, there was no additional method-specific search. AE uses a fixed encoder–decoder architecture and MSE objective, with no additional method-specific pretraining choices in this setup. For MAE, we tested masking ratios of 20%, 50% and 80%, and segment lengths of 10, 25 and 50 frames. For SimCLR, we tested all pairs of the eight available augmentations. These MAE and SimCLR choices were selected using downstream validation results.

  • Architecture and parameter counts: Tiles uses three four-block ResNet encoders, one per frequency band. Each SSL baseline uses a single 12-block encoder built from the same ResNet architecture, giving both approaches 12 residual blocks in total. All produce 256-dimensional representations and use the same-sized MLP head. The 4.8M encoder parameters reported for Tiles in Table 5 cover all three band-specific encoders and aggregation together, not each band separately.

  • Paired comparisons: We ran paired statistical comparisons using the Table 2 predictions. All methods use identical folds, allowing us to compare Tiles with the strongest SSL baseline through paired differences across the five folds. We also ran 10,000 paired participant-bootstrap resamples for the nine participant-disjoint tasks, using the same sampled participants for both methods and recomputing the fold scores.

Positive gains favor Tiles. Classification gains are macro-F1 percentage points, and regression gains are reductions in RMSE. Gains are mean ± SD across folds. Fold p-values are from uncorrected paired t-tests. Holm correction uses the 11-task comparison family.

TaskBaselineGain ± SDFolds wonFold pBootstrap 95% CIBootstrap pHolm p
BasketballAE+7.6 ± 6.25/50.053[4.6, 10.4]<0.0010.002
CookingSimCLR+2.2 ± 8.83/50.61[−4.3, 10.2]0.541
LocomotionMAE+1.8 ± 5.42/50.51[−2.5, 6.7]0.481
TennisShotAE+3.7 ± 2.35/50.024[0.6, 7.6]0.0150.12
TransportationAugPred−0.7 ± 3.72/50.68———
WorkoutsMAE+1.6 ± 1.94/50.14[−0.7, 4.3]0.201
WritingAE+3.5 ± 2.24/50.025[1.3, 6.0]<0.0010.002
JumpPowerMAE+1.6 ± 1.85/50.11[0.9, 2.4]<0.0010.002
NumStepsMAE−0.05 ± 0.183/50.57[−0.13, 0.03]0.261
ThrowSpeedSimCLR0.00 ± 0.372/50.99———
WalkDistanceRelCon−0.01 ± 0.063/50.66[−0.05, 0.02]0.441
  • Tiles achieves the highest five-seed mean among SSL methods on 7/11 tasks. Improvements on Basketball, Writing and JumpPower survive Holm correction. The remaining comparisons are inconclusive.

  • Participant overlap: Transportation and ThrowSpeed use segment- and trial-level folds, so we report only fold-level tests for these tasks. ThrowSpeed’s released data do not provide participant IDs for individual trials. Additional Transportation results are reported under Q6.

We will include the paired-comparison table and analysis procedure in the appendix and clarify the pretraining searches and parameter-count definitions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Findings and Paper Revisions

We thank the reviewers for their feedback and follow-up questions. The discussion helped us strengthen the evidence for latent specialization and clarify where Tiles’ downstream gains hold. We summarize the main findings and paper revisions below.

Rebuttal discussion

  • Representational advantage. We clarified that Tiles keeps its latents separate until the downstream head combines them for each task. Figure 7 already compares one and eight latents with the same reconstruction losses and 256 total dimensions, with eight latents yielding higher average classification macro-F1 and lower average regression RMSE.
  • Latent specialization. Across 6,200 held-out windows, individual latents contributed to largely separate time–frequency regions. Seven of eight showed consistent frequency and time preferences across participants. With stochastic query initialization, within-window overlap remained low, but individual latents largely lost their consistent roles across windows.
  • Competition and iteration. We tested the two mechanisms separately. Removing competition reduced performance by 0.57% on average across tasks, while reducing aggregation from three updates to one lowered it by 2.52%.
  • Reliability and scope of the gains. Across five downstream head seeds with pretrained checkpoints fixed, Tiles ranked first among the original SSL methods on 7 of 11 tasks. In a separate paired analysis of Table 2, gains over the strongest SSL baseline on Basketball, Writing, and JumpPower remained significant after Holm correction. On Transportation, holding out entire participants reduced performance for both methods, limiting what the segment-level results establish about transfer to new users.
  • Comparison with alternative methods. In frozen evaluations, Tiles outperformed our adapted HiMAE-Base on all 11 tasks and TSPulse on six. Fine-tuning Tiles improved mean performance on all 11 tasks and surpassed supervised training from scratch on eight, suggesting that freezing the encoder explains part of the original gap.

Paper revisions

  • Downstream performance (§5.1). Appendices A.6.1 and A.6.2 report the paired comparisons and full fine-tuning results.
  • Latent specialization (§5.2). We added the held-out analysis and the stochastic-initialization control (Appendix A.5.5).
  • Aggregation (§5.3 and Table 3). We renamed the single-representation ablation “No multi-latent aggregation” and added separate tests of competition and iteration (Appendix A.5.6).
  • Reconstruction (Figure 4). We added the per-band comparison. Tiles recovered 32% of the original energy at 1–4 Hz and 33% at 4–25 Hz, compared with 13% and 4% for MSE-only (Appendix A.5.7).
  • Evaluation details (Tables 2 and 3). We identified the fold type for each task and clarified how the across-task standard deviation is calculated.
  • Implementation details (Appendix A.3.5 and Table 5). We documented filtering and effective cutoffs and clarified what the 4.8M encoder parameter count includes.

We hope the reviewers and area chair will consider these additional experiments and revisions in their final assessments. We welcome any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.