- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the representational geometry of pretrained generative backbones relates to the generalization performance of world action models. The authors introduce nuisance orthogonality, which measures the fraction of feature displacement induced by visual perturbations that lies outside the subspace of clean-scene representations, and nuisance suppression, which characterizes how representations change after policy adaptation. Across nine generative backbones, they report a positive correlation between this metric and downstream robustness on the LIBERO-Plus benchmark.
Critical strengths
I like the motivation of this paper. The question is well chosen, the metric is simple and intuitive, and the mean-level correlation is decent. I like the authors' definition of nuisance orthogonality and the way they use it to predict generalization. It is very interesting, and these points are the main reason I see value in this work.
Critical weaknesses
From my perspective, the biggest weakness of this paper is that the analysis remains too shallow. Given that the core measurement tool, quantifying feature displacement relative to a PCA subspace, is standard, and that predicting robustness from pretrained representations has been explored in prior work, the paper’s contribution must lie in the depth of its insights. The current version falls short in this regard. While the paper shows that backbones differ in nuisance orthogonality and that these differences correlate with robustness, it does not explain what drives these differences. Section 3.1.2 identifies tokenizers, architecture, initialization, language injection, and pretraining data as potential factors, yet none is systematically isolated or ablated. The scratch FLUX.2 result raises precisely this question but leaves it unanswered. Section 5 is similarly descriptive. Ultimately, the paper establishes that models differ but offers little insight into the underlying causes or actionable guidance on how practitioners might obtain better priors.
Questions for authors
-
What exactly is scratch FLUX.2? Which components are randomly initialized (e.g., VAE, text encoder, transformer), and which are kept from pretraining?
-
What does the metric capture if a random backbone scores high? If the FLUX.2 transformer is randomly initialized, its orthogonality (0.724) exceeds that of pretrained Wan2.2 (0.594). This suggests that a large part of the metric does not come from learned knowledge. Have the authors analyzed where it comes from, and how does this fit the claim that orthogonality reflects a useful representational prior?
-
Can the contributions of the VAE, architecture, and pretraining be separated? For example, by reporting orthogonality at the VAE latent level and for a randomly initialized Wan2.2.
-
Can the authors further analyze the controlled comparison already presented in the paper? FlashMotion and Kiwi-Edit are both adapted from Wan2.2-TI2V-5B, yet their orthogonality scores differ substantially (0.645 and 0.786 vs. 0.594). Since they share the same architecture and tokenizer, this comparison could help isolate the effect of the fine-tuning objective. What insights can the authors draw from this comparison, and how do the findings relate to the claim in Section 3.1.2?
2 comments
Q3. Can the contributions of the VAE, architecture, and pretraining be separated?
The contribution of pretraining is explicitly visible through increased orthogonality of pretrained FLUX.2 and decreased orthogonality of pretrained Wan2.2. The editing objective of FLUX.2 significantly improves the orthogonality through learning to control the representations of visual variations while the video prediction objective of Wan2.2 could not. This supports our claim that the pretraining objective contributes to representational priors and their geometry relative to clean scene variations.
While completely isolating VAE and architecture to their effect on orthogonality and generalization performance since they are not completely interoperable across pretrained models due to their distinct designs, we provide VAE output orthogonality and WAM performances under VAE randomized settings.
One observation is that performance increases as each component uses a pretrained parameter, which supports semantic knowledge from pretraining definitely helps. Considering pretrained VAE as a combination of its architecture and pretraining, we can say this result is a joint effect of all components.
Another observation is that despite FLUX.2’s superior performance with pretrained modules, it has significant performance drop in the fully randomized setting. With the observation that random FLUX.2 VAE has high orthogonality from randomness and low generalization performance, its architectural property when given random initialization significantly differs from that of Wan2.2 in a way that prohibits generalization. Despite this VAE architectural disadvantage, FLUX.2 pretraining with the editing objective has reshaped the representation of VAE.
Q4. Can the authors further analyze the controlled comparison already presented in the paper? Isolating fine-tuning objectives of post-training models (Wan2.2, FlashMotion and Kiwi-Edit). What insights can the authors draw from this comparison, and how do the findings relate to the claim in Section 3.1.2?
Kiwi-Edit post-trained with video editing significantly improved orthogonalization, while the FlashMotion with video prediction has less improved. This is consistent with the pretraining result above and with our motivation in Section 3.1.1 that this observation is not limited to image models. Also, from our claim in Section 3.1.2 and experiments as an answer to Q3 that orthogonality is a joint effect of all design choices, this post-training experiments can be framed as isolated post-training objective experiments with fixed VAE and architecture. The results show that the representational structure of the transformer can be reshaped to favor separation of nuisance displacement via editing objective and provides future direction of representational post-training for policy generalization.
We added these analysis as a Section 4.2.2, 4.2.3 and Appendix B.3 of the revised paper.
We thank the reviewer for the thoughtful evaluation of our work and for the constructive comments, which helped us strengthen our evidence and deepen our analysis of what drives the orthogonality isolating each contribution of VAE, architecture, pre-training and post-training.
Edited: Following up on our previous response, we have completed the additional policy experiments, evaluations and VAE output orthogonality, then reorganized the answers based on each question and revised the paper.
W1. Requires deeper insight.
We agree that deeper insight into what drives the observed representational differences beyond establishing a correlation would help evidence of our diagnostics. We thank the reviewer for this constructive comment. We would like to reinforce our claim and provide deeper insights by answering the following questions with additional analysis and WAM training.
Q1. Clarification of Scratch FLUX.2 setting
First, to clarify our Scratch FLUX.2 setting, we only re-initialized the pretrained diffusion transformer of FLUX.2, while keeping the vision and language modules frozen. We added a brief clarification to Section 3.1.2 of the revised paper.
To answer each question, we additionally executed extensive analysis on the orthogonality random initialization and trained parameters of the VAE and the transformer of FLUX.2, Wan2.2, FlashMotion and Kiwi-Edit to present the metrics with isolation under each component, pretraining and post-training. Results over three random seeds are as below.
Table R7
Q2. Why does random backbone have high orthogonality? Where orthogonality comes from? And how does this fit the claim that orthogonality reflects a useful representational prior?
The orthogonality of the transformer are composed as a combination of nuisance separation from each pretrained component and randomness of high dimensional vectors in randomized components. Comparing pretrained VAE and random VAE under random transformer settings, increased orthogonality of pretrained VAEs lead to higher orthogonality of transformer also with increased policy performance.
Also, an untrained random transformer of Wan2.2 exhibits substantial orthogonality without useful task representations. This high orthogonality of random transformers is due to high dimensional random variations being biased towards orthogonality, even without semantic representation and separation of nuisance information. WAM trained from this Wan2.2 with random transformer reaches 39.9% average LIBERO-Plus success, below the pretrained Wan2.2 (46.5%) despite its higher orthogonality. To our analysis, this policy adaptation from random initialization also shows less nuisance suppression.
While this interpretation makes orthogonality’s association with policy generalization limited to backbones that already provide useful representations, comparing fully pretrained models with the semantic knowledge that serve as world model backbone candidates, orthogonality reliably estimates how displacements from nuisance information are separated. At the same time, we can again use this property to audit pretraining or post-training before policy adaptation to check if training objective is beneficial for further policy generalization performance, evidenced by substantial performance improvement gaps of FLUX.2 and Wan2.2.