acceptodds
Under review as a conference paper at ICLR 2027

Ladder Transformer: Unified Self-Supervised Learning for Continuous Time Multimodal Streams

Abstract

We present the Ladder Transformer, an encoder-decoder architecture capable of representation learning and autoregressive generation across continuous and discrete input spaces. We treat modalities like text, video, audio, and proprioception as a single token stream, and our model supports heterogeneous sampling rates natively. The model is trained using only input-space losses, with no EMA teacher, auxiliary losses, or anti-collapse regulariser. On frozen-feature video classification, our learned representations show stronger temporal understanding than existing masked modelling methods. We demonstrate the versatility of our model in a variety of autoregressive generation tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, multimodal, masked modelling, masked autoencoders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HQ1nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an encoder-decoder architecture, Ladder Transformer, which simultaneously performs representation learning and autoregressive generation over both continuous and discrete input spaces. It employs mirrored hierarchical cross-attention, allowing low-level details to reach the reconstruction space without passing through the deepest layer. This architecture demonstrates performance advantages in frozen evaluation for Motion Understanding, as well as in the fine-tuning settings for Text-to-Motion​ and Text-to-Speech.

Critical strengths
  1. The motivation is clear, and the ablation studies on the architecture validate this motivation.
Critical weaknesses
  1. An important aspect of self-supervised learning is whether it can scale in terms of data volume and model size; this paper lacks empirical studies on scaling.
  2. The paper claims that the proposed method works "with no EMA teacher, auxiliary losses, or anti-collapse regulariser," yet there is no ablation study or evidence demonstrating the connection between these components and the proposed method.
Questions for authors
  1. In the Text-to-Motion and Text-to-Speech experiments, are the training or testing protocols/standard consistent with those of the compared methods?
  2. Only motion tasks are compared. It is necessary to evaluate appearance tasks as well.
2 comments

We thank reviewer xF2K for the questions about scaling, training components, and evaluation. Below, we first correct one point in the summary, then address the weaknesses and questions. Edits in the paper PDF are marked with blue color.

Clarification: T2M and T2S model training. The text-to-motion and text-to-speech model are trained from scratch on their own data, but share the same identical transformer architecture (with the addition of small flow matching heads for generation). The word “fine-tune” in Sec. 4.3 refers to the third training stage of the T2M model (HumanML3D after motion-capture pre-training), not to transfer from the video encoder.

W1 - Scaling

We agree that model and data scaling is an important aspect of any multimodal SSL model. Our video pre-training mix is 12,100 h (Table A10), about six passes over each source at 300k steps, and the held-out loss shows no sign of overfitting, so we do not expect data to be the binding constraint at this scale. We trained video SSL models at ViT-B and ViT-L scale, and observed that increasing model size at this regime consistently resulted in improved performance. Specifically, with 300k steps of pre-training we observe the following improvement with scale:

ScaleD48SSv2Jester
ViT-B84.063.197.1
ViT-L90.367.997.7

We have also trained the Ladder Transformer as a language model up to 1.2B parameters, using the FineWeb-Edu dataset. No new instability or collapse was observed going from 300M to 1.2B parameters, but we did end up adding qk-norm and a final RMSNorm following the contemporary LLM literature, and found that it improved late-training consistency. We observe that perplexity consistently decreases with size and training data volume:

ParamsLayersHidden dimTraining tokensPPL
117 M6+67689.9 B18.27
331 M12+12102420.6 B13.78
1.2 B12+12204826.0 B11.21

W2 - EMA teacher, auxiliary losses, and collapse prevention. The statement about training "with no EMA teacher, auxiliary losses, or anti-collapse regulariser" is in contrast to leading latent prediction SSL methods that cannot be trained without these components, which introduce additional training hyperparameters, and constrain the structure of the latent space. JEPA, DINO, and SALT require an EMA teacher to prevent collapse, and LeJEPA uses an explicit Gaussian regularization of the latent space. Other pixel-reconstruction methods (MAE, RVM) also do not need these, since using the input space as a target prevents collapse.

Q1 - T2M and T2S training protocol

We follow standard evaluation protocols in both text-to-motion and text-to-speech tasks.

Text-to-motion: All rows are scored with the released evaluator of Guo et al. (2022) on the standard 263-D features of the HumanML3D test split. The one protocol difference is training data: our model is pre-trained on 276 h of motion capture (AMASS, BABEL, BONES-SEED, HumanML3D train) before the HumanML3D stage, whereas the compared models train on HumanML3D alone. The HumanML3D test split is excluded from every stage. To show that this is not what drives the result, a control in which the autoencoder and all three stages use HumanML3D with the following results:

Training dataFID ↓R@1 ↑
HumanML3D only0.0640.525
Broader pretraining corpus0.0630.528

Text-to-speech: We follow the continuation protocol of VALL-E (3 s prompt plus transcript, 4–10 s LibriSpeech utterances), with WER from HuBERT-Large CTC and speaker similarity from WavLM-TDNN. Simple-TTS is re-run in our harness on the same utterances. The published rows were obtained on a different utterance cohort and scoring pipeline, which is why Table t2s separates the blocks and reports similarity relative to each harness’s ground truth; only that ratio is comparable across blocks. Our model is trained on 40kh, less than every other row (45–250kh).

Q2 - Evaluation selection

Since we position Ladder Transformer as a SSL method for streaming continuous time data, we deliberaterly chose evaluation tasks where success depends on both appearance and motion understanding. SSv2, Diving-48, and Jester are the three benchmarks that the current state-of-the-art (V-JEPA 2) also uses to measure motion understanding under frozen evaluation.

With the discussion period drawing to a close, we haven't yet heard back and would appreciate your assessment of the scaling and evaluation controls. As a summary: ViT-B to ViT-L improves all three frozen video probes, language model training remains stable when scaled up to 1.2B parameters, and HumanML3D-only training retains essentially the same motion FID (0.064 vs. 0.063). Our rebuttal also clarifies the questions related to the baselines and benchmarks and clarifies that the reported video SSL results use pixel reconstruction alone.

Reviewer_6RkDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Ladder Transformer, a causal encoder–decoder architecture for continuous-time multimodal streams. It uses mirrored hierarchical cross-attention to support both masked representation learning and autoregressive generation within a shared timestamp-based framework. The model is evaluated on video SSL, language modeling, text-to-motion, and text-to-speech.

Critical strengths
  1. Clear motivation. The paper targets the tension between preserving low-level predictive detail and learning abstract representations.
  2. Broad evaluation. The method is tested across both representation learning and autoregressive generation.
  3. Simple SSL objective. Strong video representations are learned using only input-space reconstruction, without EMA teachers or extra anti-collapse losses.
Critical weaknesses
  1. Limited architectural novelty.
    The mirrored encoder–decoder topology is closely related to prior ladder/U-shaped architectures, including Ladder Networks and DiffMAE. The main novelty appears to lie in its adaptation to causal continuous-time modeling rather than the connectivity pattern itself.
  2. The proposed mechanism is not directly validated.
    The topology ablation convincingly shows that connectivity matters, but it does not establish the proposed explanation that shallow connections carry low-level details and thereby make deeper representations more abstract and invariant. Alternative explanations, such as improved optimization, gradient routing, or access to multiple temporal receptive fields, remain plausible.
  3. The ablations do not fully isolate what is special about the mirrored ordering.
    The video experiments compare mirrored connectivity only against apex and equal-depth/reverse wiring. Stronger alternatives such as all-level feature aggregation or learned routing could help determine whether the specific mirrored ordering is essential or whether access to intermediate encoder features is sufficient.
Questions for authors
  1. Can the authors provide layer-wise analyses showing that shallow and deep encoder layers exhibit the proposed low-level versus abstract/invariant specialization?
  2. Can the authors distinguish the proposed abstraction mechanism from alternative explanations such as improved optimization, gradient flow, or multi-scale temporal receptive fields?
  3. Have the authors considered stronger multi-level fusion baselines, such as aggregating all encoder levels or learning the encoder-to-decoder routing?
  4. Appendix A.2 validates mirrored connectivity for autoregressive language modeling. Does the same advantage hold for the continuous autoregressive generation tasks, particularly text-to-motion and text-to-speech?
6 comments

We thank reviewer 6RkD for recognizing the motivation, breadth, and simple objective. We agree that topology ablations alone do not identify the mechanism. We therefore conducted layer-wise measurements, targeted interventions, a gradient-routing control, and a learned aggregation comparison. These experiments sharpen the interpretation: mirrored routing supports depth specialization, and direct reconstruction gradients through intermediate reads appear important to that specialization. Updates to the paper are marked in blue in the manuscript.

Architectural novelty. Our primary novelty claim is combining mirrored-depth encoder-decoder cross-attention with hybrid learned-rotary positional encoding, and showing that this design gives strong performance in both discrete and continous-time streaming SSL. We agree that mirrored encoder–decoder connectivity is not new by itself (Ladder Networks, DiffMAE, Hi-End-MAE, etc.). One of our central reseach goals is also to re-evaluate the use of input-space prediction in video SSL, where joint-embedding methods have been the leading approach in frozen probes since V-JEPA from Feb 2024. To the best of our knowledge, our method along with RVM, are the first methods to effectively close the gap with purely input-space losses, although the specific approaches are very different. We have updated the Introduction and Related Work sections for clarity.

Validating the hypothesized mechanisms. The theoretical argument for the mirrored wiring relies on the hypothesis that it allows information to stratify in the encoder; lower layers can pass noisy, low-level signals to the decoder via cross-attention, allowing deep layers to focus on semantic features. To test this, we conducted the following experiments using the three alternative connectivities of Section 4.2 of the paper:

(i) Layer-wise content: In the mirrored model, linear decodability of a token’s own patch falls with depth (R² = 0.75 at the deepest level), whereas it stays high (0.91) for both alternatives. The semantic read-out behaves in the opposite way: probing every level, the mirrored model improves up to the deepest levels on SSv2 and Diving-48, while the reverse model peaks at level 10 on Diving-48 (34.9%) and declines to 28.8% at level 12.

(ii) Causal lesions: We erased the 32 most pixel-predictive directions of one encoder level from the decoder’s input and measured the reconstruction loss. At the deepest level this raises the loss by 0.4% in the mirrored model, but by 109 % (reverse) and 417 % (apex). The mirrored decoder instead depends on the lower levels (levels 4–6 in the paper’s model) and on nothing from level 9 upwards. This holds across seeds, and at 25k, 200k, and 300k training steps. A more detailed description and analysis has been added to the manuscript.

We also tested the three alternative explanations raised in the review:

(iii) Improved optimisation: The mirrored model does reach a 5–8% lower reconstruction loss at 300k steps. However, at 25k steps, where its loss is worse than that of both alternatives at 300k, it already shows the same lesion profile (+0.1% at the deepest level). The stratification is therefore not a by-product of a lower loss.

(iv) Gradient routing: We trained a model with the mirrored reads but with the gradient stopped through the reads of levels 1–11, so that the encoder is trained only through the deepest level, as in apex. Its decoder falls back on the deepest level (erasure +75 %; removing the reads of levels 1–6 costs 3%), and it loses the advantage in frozen evaluation at 25k steps (SSv2 28.7% against 34.0 %, Diving-48 19.3% against 26.8%). We therefore agree that gradient routing is at work, as the means by which the stratification arises: the gradient from the late decoder blocks shapes the lower levels into what the decoder reads for pixel detail, which frees the deep levels from carrying it. We have revised the text to say this.

(v) Multiple temporal receptive fields: The reverse model’s decoder reads the same twelve levels, with the same temporal spans, as the mirrored one, and is 5.8 pp worse on SSv2 and 22.6 pp on Diving-48. Access to several temporal spans therefore does not explain the gain; only the order of the reads differs.

Full details and results have been added to the manuscript.

Learned routing. We also trained the suggested baseline, in which each decoder block reads a learned softmax-weighted mixture of all twelve encoder levels. It thus combines all-level aggregation with learned routing. The table compares it with the mirrored model trained with the same code, using SSv2 attentive-probe accuracy at the deepest encoder level:

Training stepsMirroredLearned aggregation
25k34.0%32.0%
50k35.8%34.8%
100k38.9%37.8%

Access to intermediate levels recovers most of the gap but not all of it. Learned aggregation trails the mirrored model by 1–2 points at every checkpoint, and needs 20% more training time per step. Its learned weights also move towards the mirrored order. At 100k steps, where training ended, the first six decoder blocks read mostly deep levels and the last six read shallower ones, with a rank correlation of 0.73 to the mirrored assignment. Three of the first six blocks put more than half of their weight on the deepest level, and two of the later blocks put three quarters of theirs on levels 1–7. The routing nevertheless remains soft: no block puts more than 54% of its weight on a single level. Access alone is not sufficient, however: the reverse model also reads all levels and performs no better than apex.

Topology ablations for autoregressive generation. We repeated the cross-attention topology ablations for the cross-modal AR generation tasks. In text-to-speech, the 'mirror' cross-attention topology is significantly better than either alternative:

Text-to-speech (T2S)

TopologyWER-H ↓WER-W ↓SIM ↑
Mirrored2.51 ± .023.31 ± .030.525
Apex5.27 ± .055.93 ± .060.515
Reverse11.80 ± .1312.24 ± .020.522

In text-to-motion, the 'mirror' has a small but consistent advantage compared to 'reverse', and 'apex' collapses during training:

Text-to-motion (T2M)

TopologyFID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓
Mirrored0.063 ± .0020.5280.7210.8162.860 ± .007
Reverse0.068 ± .0030.5240.7180.8152.869 ± .006
Apexcollapsed

These results show that the mirror topology is superior for both representation learning and cross-modal autoregressive generation. These numbers and their interpretations have been added to the manuscript.

Reviewer_xF2KOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the ladder transformer, an encoder-decoder architecture designed for learning SSL representations from multimodal streams, using input-space loss only. The bottom-up encoder and top-down decoder interact through mirrored cross-attention. The paper claims that lateral connections between encoder and decoder enable hierarchical information flow (such as lower-level details reaching the reconstruction without passing through the deepest layers of the encoder). The model supports autoregressive generation for a variety of modalities (text, video, audio, proprioception) through token interleaving (continous or discrete). The model uses rotary time embeddings to support sampling rate heterogeneity.

Experiments being conducted on different tasks and modalities show the versatility of the model (although my understanding is that a different model is trained separately for each task).

However I think the paper is missing the essence of its core contribution to maximize its impact for the ICLR community. The ablation in the Figure 6 shows impressive performance gains from the proposed approach, but I am expecting more clarity regarding the design choices and corresponding ablations, and ablations across the different modalities to support that the design choice is robust and not task or domain-specific.

Overall between the lack of clarity, supporting evidence, positioning compared to prior works (RVM seems to occupy a special place in the related works yet it is not clear how it relates to the paper), and only partially supported claims, I recommend a weak rejection.

Critical strengths

The main defensible claim of the paper is the ladder transformer design with superior fine-grained motion understanding compared to other SSL methods while achieving sota performance in streaming text-to-motion.

Critical weaknesses

Other claims such as hybrid spatiotemporal positional encoding, joint multimodal masked prediction and autoregressive generation through rotary time positional encoding (for handling timestamped tokens) were already shown in prior works.

The related works section does not explain how the proposed design differ from prior "similar network architectures". That section is also missing prior works on joint masked and autoregressive modeling (such as "Enabling autoregressive models to fill in masked tokens").

Diving into the details of the proposed design, the closest prior design is that of DiffMAE with the two following differences:

  1. "a single attention block performs both self-attention and cross-attention"
  2. "the decoder does not have it’s own key/value projection for the encoder side embeddings, instead using the same projection weights as the encoder self-attention"

Yet the paper is not showing ablations on those exact two core claims and it is not clear whether the "reverse" ablation is one of them.

Questions for authors

I thank the authors for submitting a well written manuscript.

  • clarification requests

Where are the lateral branches emanating from the encoder exactly? Are they post K/V projection? How does that differ from DiffMAE?

A supporting figure showing the connection between the internals of an encoder block and a decoder block would be useful to avoid confusion between designs (e.g. comparing pathways for diffMAE, "reverse" and proposed). The Figure 3 only shows the decoder and does not mention whether concatenation of K/V is along the time or channel dimension.

  • suggestions for future work

To support the claim that "lateral connections between encoder and decoder enable hierarchical information flow", the paper is missing a qualitative visualization of the representations from different blocks of the model. For example one would expect to observe fine grained reconstruction supporting features in the first layers and semantic representations in the last layers.

Clearer ablation design including design differences from diffMAE, across the different tasks/modalities.

6 comments

We thank reviewer xF2K for an insightful review, and the concrete suggestions on attribution and qualitative analysis. Following their suggestion, we have conducted additional experiments and analyses into the mechanisms behind the superior representation learning performance of the mirror topology, and have added the results into the manuscript. All updates are marked in blue in the manuscript. Below we address the concerns and questions raised.

Details of cross-attention connectivity. We have improved Figure 3 of the paper, as well as the rest of the methods section to better describe how the encoder and decoder blocks connect, and the keys and values are read by the decoder blocks. In the shared self-cross attention block, they are concatenated along the sequence (time) dimension.

Contribution and relationship to prior work. We build on the hierarchical encoder–decoder principle of Ladder Networks and subsequent transformer designs. Our primary novel contribution is combining mirrored-depth encoder-decoder cross-attention with hybrid learned-rotary positional encoding, and showing that this design gives strong performance in both discrete and continous-time streaming SSL. We agree that DiffMAE's cross-self decoder is the closest prior design, and that mirrored encoder–decoder connectivity is not new in itself (Ladder Networks, DiffMAE, Hi-End-MAE, etc.). We have edited the Introduction and Related Work sections to better reflect this.

Differences to DiffMAE. DiffMAE differs from our method in that it uses a smaller decoder (24-layer encoder, 8-layer decoder), that is trained using a denoising diffusion objective, and is iterated during generation. Our encoder and decoder are equal-depth and trained using a plain regression loss. The decoder stack is evaluated only once, and starts from a blank token. A separate small flow matching head can be used for iterative sampling.

The modifications we make to the decoder block (fused attention block for self-attention and cross-attention, and shared K/V projections between the encoder and decoder) are primarily for improved efficiency. Applying both results in 11% fewer model parameters and 18% lower GPU memory use. Sharing the projector also allows us to roughly halve the size of the kv-cache, since we only need to store a single set of key-value projections, and the decoder self-attention window is small. We have added these details to the Methods section.

Demonstrating the hypothesized mechanism. The theoretical argument for the mirrored wiring relies on the hypothesis that it allows information to stratify in the encoder; lower layers can pass noisy, low-level signals to the decoder via cross-attention, allowing deep layers to focus on semantic features. To test this, we conducted the following experiments using the three alternative connectivities of Sec. 4.2 of the paper:

(i) Layer-wise content: In the mirrored model, linear decodability of a token's own patch falls with depth (R² = 0.75 at the deepest level), whereas it stays high (0.91) for both alternatives. The semantic read-out behaves in the opposite way: probing every level, the mirrored model improves up to the deepest levels on SSv2 and Diving-48, while the reverse model peaks at level 10 on Diving-48 (34.9%) and declines to 28.8% at level 12.

(ii) Causal lesions: We erased the 32 most pixel-predictive directions of one encoder level from the decoder’s input and measured the reconstruction loss. At the deepest level this raises the loss by 0.4% in the mirrored model, but by 109 % (reverse) and 417 % (apex). The mirrored decoder instead depends on the lower levels (levels 4–6 in the paper’s model) and on nothing from level 9 upwards. This holds across seeds, and at 25k, 200k and 300k training steps. The full details and results are included in the manuscript.

Clarification about model training. We do train different models for text-to-motion, text-to-speech, and autoregressive language modelling. This is to demonstrate that our model/architecture and training procedure can be adapted to a variety of continuous and discrete sequential data. The benchmark T2M and T2S architectures we list include a variety of task-specific adaptations, whereas our design requires minimal changes in comparison, all while achieving strong performance.

Significance of RVM. We highlight RVM in the related work section, because it is another recent method that revisits input-space prediction for video representation learning. Joint-embedding methods have been leading video SSL frozen benchmarks since V-JEPA (early 2024), most clearly on motion-centric ones such as SSv2, Diving-48 and Jester. RVM is, to our knowledge, the first model trained with a pixel loss alone (no EMA teacher, distillation or regulariser) to close this gap at matched scale. RVM is therefore evidence that the input-space objective is not in itself the barrier, which is also the premise of our work. Their model architecture and training procedure are very different from ours.

Final ablation numbers We have completed controlled ablations of both the cross-attention topology, and our decoder block modifications (re-using K/V projections and using a combined cross-attention and self-attention block). The results confirm that the decoder block changes significantly improve frozen eval performance in addition to increased efficiency:

SSv2Diving-48
Control40.6250.15
Own K/V38.27 (−1.8%)45.33 (−3.2%)
Sep. attn.28.14 (−12.0%)17.87 (−30.7%)

In text-to-speech, the 'mirror' cross-attention topology is significantly better than either alternative:

Text-to-speech (T2S)

TopologyWER-H ↓WER-W ↓SIM ↑
Mirrored2.51 ± .023.31 ± .030.525
Apex5.27 ± .055.93 ± .060.515
Reverse11.80 ± .1312.24 ± .020.522

In text-to-motion, the 'mirror' has a small but consistent advantage compared to 'reverse', and 'apex' collapses during training:

Text-to-motion (T2M)

TopologyFID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓
Mirrored0.063 ± .0020.5280.7210.8162.860 ± .007
Reverse0.068 ± .0030.5240.7180.8152.869 ± .006
Apexcollapsed

These results show that both the decoder block changes and the cross-attention topology improve representation learning performance, and that the mirror topology is superior for both representation learning and cross-modal autoregressive generation. The results and their interpretations have been added to the manuscript.

Comments on the paper

2 comments

Dear Reviewers, thank you for your time and effort in reviewing this submission. As the discussion period is ongoing, I would appreciate it if you could join the discussion and engage with the authors' rebuttal at your earliest convenience.

Thank you for inviting further discussion. The revised paper and rebuttal directly address the main concerns in the reviews.

  • Both decoder changes improve frozen accuracy (xF2K). We clarified what the architecture draws from DiffMAE and ablated our two decoder changes separately. Diving-48 accuracy drops from 50.15% to 45.33% with dedicated K/V projections and to 17.87% with separate self- and cross-attention (Figs. 3 and 6).
  • Mirrored ordering matters, and the new interventions show why (6RkD, xF2K). Blocking reconstruction gradients through shallow connections makes the decoder rely on the deepest layer again and removes its early frozen-probe advantage. Even learned routing with access to all layers moves toward the mirrored order and still trails it on SSv2. Mirrored routing also gives substantially lower speech error and a smaller gain on motion generation (Fig. 7, App. A.3–A.4).
  • The video results improve with model size, and the motion result holds with matched training data (HQ1n). ViT-L improves all three frozen video probes over ViT-B, and HumanML3D-only training nearly reproduces the full-data result (0.064 vs. 0.063 FID). Our rebuttal also reports stable language model training to 1.2B parameters and clarifies the speech comparisons.

We ask you all to take these results into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.