- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an encoder-decoder architecture, Ladder Transformer, which simultaneously performs representation learning and autoregressive generation over both continuous and discrete input spaces. It employs mirrored hierarchical cross-attention, allowing low-level details to reach the reconstruction space without passing through the deepest layer. This architecture demonstrates performance advantages in frozen evaluation for Motion Understanding, as well as in the fine-tuning settings for Text-to-Motion and Text-to-Speech.
Critical strengths
- The motivation is clear, and the ablation studies on the architecture validate this motivation.
Critical weaknesses
- An important aspect of self-supervised learning is whether it can scale in terms of data volume and model size; this paper lacks empirical studies on scaling.
- The paper claims that the proposed method works "with no EMA teacher, auxiliary losses, or anti-collapse regulariser," yet there is no ablation study or evidence demonstrating the connection between these components and the proposed method.
Questions for authors
- In the Text-to-Motion and Text-to-Speech experiments, are the training or testing protocols/standard consistent with those of the compared methods?
- Only motion tasks are compared. It is necessary to evaluate appearance tasks as well.
2 comments
With the discussion period drawing to a close, we haven't yet heard back and would appreciate your assessment of the scaling and evaluation controls. As a summary: ViT-B to ViT-L improves all three frozen video probes, language model training remains stable when scaled up to 1.2B parameters, and HumanML3D-only training retains essentially the same motion FID (0.064 vs. 0.063). Our rebuttal also clarifies the questions related to the baselines and benchmarks and clarifies that the reported video SSL results use pixel reconstruction alone.
We thank reviewer xF2K for the questions about scaling, training components, and evaluation. Below, we first correct one point in the summary, then address the weaknesses and questions. Edits in the paper PDF are marked with blue color.
Clarification: T2M and T2S model training. The text-to-motion and text-to-speech model are trained from scratch on their own data, but share the same identical transformer architecture (with the addition of small flow matching heads for generation). The word “fine-tune” in Sec. 4.3 refers to the third training stage of the T2M model (HumanML3D after motion-capture pre-training), not to transfer from the video encoder.
W1 - Scaling
We agree that model and data scaling is an important aspect of any multimodal SSL model. Our video pre-training mix is 12,100 h (Table A10), about six passes over each source at 300k steps, and the held-out loss shows no sign of overfitting, so we do not expect data to be the binding constraint at this scale. We trained video SSL models at ViT-B and ViT-L scale, and observed that increasing model size at this regime consistently resulted in improved performance. Specifically, with 300k steps of pre-training we observe the following improvement with scale:
We have also trained the Ladder Transformer as a language model up to 1.2B parameters, using the FineWeb-Edu dataset. No new instability or collapse was observed going from 300M to 1.2B parameters, but we did end up adding qk-norm and a final RMSNorm following the contemporary LLM literature, and found that it improved late-training consistency. We observe that perplexity consistently decreases with size and training data volume:
W2 - EMA teacher, auxiliary losses, and collapse prevention. The statement about training "with no EMA teacher, auxiliary losses, or anti-collapse regulariser" is in contrast to leading latent prediction SSL methods that cannot be trained without these components, which introduce additional training hyperparameters, and constrain the structure of the latent space. JEPA, DINO, and SALT require an EMA teacher to prevent collapse, and LeJEPA uses an explicit Gaussian regularization of the latent space. Other pixel-reconstruction methods (MAE, RVM) also do not need these, since using the input space as a target prevents collapse.
Q1 - T2M and T2S training protocol
We follow standard evaluation protocols in both text-to-motion and text-to-speech tasks.
Text-to-motion: All rows are scored with the released evaluator of Guo et al. (2022) on the standard 263-D features of the HumanML3D test split. The one protocol difference is training data: our model is pre-trained on 276 h of motion capture (AMASS, BABEL, BONES-SEED, HumanML3D train) before the HumanML3D stage, whereas the compared models train on HumanML3D alone. The HumanML3D test split is excluded from every stage. To show that this is not what drives the result, a control in which the autoencoder and all three stages use HumanML3D with the following results:
Text-to-speech: We follow the continuation protocol of VALL-E (3 s prompt plus transcript, 4–10 s LibriSpeech utterances), with WER from HuBERT-Large CTC and speaker similarity from WavLM-TDNN. Simple-TTS is re-run in our harness on the same utterances. The published rows were obtained on a different utterance cohort and scoring pipeline, which is why Table t2s separates the blocks and reports similarity relative to each harness’s ground truth; only that ratio is comparable across blocks. Our model is trained on 40kh, less than every other row (45–250kh).
Q2 - Evaluation selection
Since we position Ladder Transformer as a SSL method for streaming continuous time data, we deliberaterly chose evaluation tasks where success depends on both appearance and motion understanding. SSv2, Diving-48, and Jester are the three benchmarks that the current state-of-the-art (V-JEPA 2) also uses to measure motion understanding under frozen evaluation.