Ladder Transformer: Unified Self-Supervised Learning for Continuous Time Multimodal Streams

We present the Ladder Transformer, an encoder-decoder architecture capable of representation learning and autoregressive generation across continuous and discrete input spaces. We treat modalities like text, video, audio, and proprioception as a single token stream, and our model supports heterogeneous sampling rates natively. The model is trained using only input-space losses, with no EMA teacher, auxiliary losses, or anti-collapse regulariser. On frozen-feature video classification, our learned representations show stronger temporal understanding than existing masked modelling methods. We demonstrate the versatility of our model in a variety of autoregressive generation tasks.