acceptodds
Published as a conference paper at NeurIPS 2026

Breaking feature collapse in self-supervised time series encoder

Özgün Turgut, Daniel Rueckert

Abstract

General-purpose time series encoders such as OTIS and MOMENT promise a single backbone that transfers across domains, but they do not scale and rely on manually tagged domain labels at tokenisation. We trace both symptoms to a single pathology: under masked-data-modelling supervision the encoder's output patch tokens partially collapse, with mean off-diagonal pair-wise cosine similarity stabilising at in OTIS and the encoder's effective rank capped accordingly. This is the natural consequence of supervising the encoder only in data space, which leaves no constraint against redundant token directions in the representation space. We introduce OTISv2, which suppresses the collapse by complementing the masked-reconstruction objective with two representation-space terms — a self-distillation loss against an exponential-moving-average teacher and a KoLeo regulariser — and replaces OTIS's per-domain variate embeddings with register tokens to remove the label dependency. The resulting encoder pulls mean patch-to-patch cosine similarity from to , scales monotonically from M to M parameters where its predecessor plateaus, and dominates general-purpose baselines on uni- and multi-variate datasets from the UEA and UCR archives from a single label-free checkpoint. We release code and pre-trained weights upon acceptance.

Related papers

See more related submissions when the market opens.