- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies a practically important limitation of robotic world models: visually similar observations and identical actions can lead to different outcomes because of hidden environmental factors such as friction, mass, stiffness, mass distribution, or interaction rules. T2-Z augments an action-conditioned video world model with a compact 32-D environment latent. During training, the method alternates between world-model optimization with fixed latents and environment-latent optimization with the world model frozen. At test time, the world model remains frozen and only a newly initialized environment latent is optimized from one or more interaction trajectories using the same flow-matching objective. The evaluation covers seven simulated settings and four real-world robotic tasks. T2-Z consistently improves physical prediction and action selection over a standard world model, LoRA-based test-time adaptation, an amortized DINOv2 context encoder, and a TTT-KQV baseline. The learned latent space also exhibits structured PCA patterns that correlate with underlying factors, including a two-dimensional organization when mass and friction vary jointly.
Critical strengths
-
A meaningful problem formulation. The paper focuses on hidden environmental variation that is not directly observable from images but can substantially change action-conditioned dynamics. This is an important failure mode for world models intended for real-world robotic use.
-
A simple and effective solution with broad validation. T2-Z adapts only a low-dimensional latent instead of updating the full world model, making the method conceptually clean and computationally lightweight. The gains are consistent across seven simulation settings and four real-world tasks, and the emergent latent structure provides additional evidence that the representation captures meaningful environment variation.
Critical weaknesses
-
The methodological novelty is moderate. Direct optimization of latent codes through prediction error, together with alternating model/latent optimization, is conceptually related to classical latent-variable optimization, analysis-by-synthesis, and system-identification methods. The main contribution therefore appears to be the problem formulation and its effective instantiation in a modern robotic video world model rather than a fundamentally new optimization principle. The paper would benefit from a more explicit discussion of this novelty boundary.
-
The current formulation has several practical assumptions and the latent analysis could be stronger. Training requires trajectories to be grouped by environment, while test-time adaptation requires one or more interaction trajectories before prediction. This may be limiting when interactions are costly, unsafe, or when hidden factors change within an episode. In addition, the PCA visualizations are suggestive but do not quantitatively establish factor recovery or disentanglement; linear probing, correlation analysis, or an oracle physical-parameter baseline would strengthen this claim.
Questions for authors
-
Can the authors clarify the conceptual novelty of T2-Z relative to prior direct latent optimization and system-identification approaches, and explain what is specifically enabled by the proposed formulation beyond applying latent inversion to a robotic world model?
-
Can the authors provide a quantitative factor-recovery analysis (e.g., linear probing, correlation/R² with the ground-truth factors) in addition to PCA visualizations? An oracle baseline conditioned on true physical factors in simulation would also help contextualize the remaining gap.
-
How sensitive is performance to the type and informativeness of the test-time interaction trajectories? The current evaluation uses randomly sampled actions; what happens when the interaction is weakly informative or when exploratory actions are constrained?
-
How does the method behave when hidden environmental factors change within an episode? Is a single stationary environment latent sufficient, or would online/temporal latent updates be required?
8 comments
Response to Reviewer FEqY (2/7)
Our compact environment representation and alternating training recipe make this environment-level learning and adaptation effective in a generative video world model. Under the same backbone and 32-D representation, Table 3 of the original submission shows that alternating training reduces mean physical prediction error from 7.03 with joint training to 3.85 and increases action success from 68% to 72%. Directly optimizing a 3072-D conditioning token gives an error of 94.50 and action success of 24%. These ablations establish the practical contribution of the representation and training design to learning environmental information from high-dimensional videos and using it for adaptation.
2. Comparison with analysis-by-synthesis
Analysis-by-synthesis methods explain observations by optimizing a code through a generative model. GLO (Bojanowski et al., ICML 2018) learns per-image codes and a generator through reconstruction; DeepSDF (Park et al., CVPR 2019) learns shape codes and a signed-distance decoder; Image2StyleGAN (Abdal et al., ICCV 2019) optimizes a code in a pretrained generator to reconstruct an input image. These methods use the code to represent and reconstruct observed visual content. T2-z focuses on adapting the environment represented by a robotic world model. Its latent is inferred from observed interactions and then reused to predict the consequences of different, untried actions. Thus, the representation explains the observed interaction while also supplying environmental information for future action-conditioned video prediction.
3. Relationship to system identification
Many learning-based system-identification methods use observations to infer system parameters or environmental features within an already established model or latent representation. For example, RMA (Kumar et al., RSS 2021) first learns an environment embedding and policy from privileged physical factors, then trains an adaptation module to predict that embedding from state-action history. T2-z learns the environment representation itself from grouped raw videos and actions through world prediction. The model discovers the environmental information needed to explain the observations and predict new actions, without specifying the physical factors or supervising their values.
We also examine ISE (Ko et al., 2025), which uses video generation and latent inversion to explain observed interactions. The distinction concerns how the representation being optimized is constructed and supervised. T2-z learns that environment-conditioning space directly from grouped raw interaction data, rather than constructing it from task-defined successful interactions.
The resulting latent space captures a form of environmental understanding grounded in raw physical-world interactions. The model learns which regularities are shared across interactions in an environment and uses them to explain its responses to different actions. We see this as a step toward building models that can learn regularities directly from raw, high-dimensional interactions with the physical world and put those regularities to use. Our formulation enables the following capabilities.
-
Learning environmental features with environment grouping alone. Raw interaction videos, their associated actions, and shared-environment membership are sufficient to learn T2-z's environment representation, without additional physical-parameter, task-goal, or success/failure labels. ISE's main implementation obtains video features from DINOv2 and selects a task-labeled successful interaction's encoding as each object's representative state embedding. Its representation construction therefore uses a task definition and a corresponding definition of successful interaction. T2-z instead learns which environmental features explain the interactions through video prediction itself. This enables environmental regularities to emerge without defining them through task success.
-
Recovering physical information from the learned representation. We evaluate whether the representation learned through video prediction contains physical information that can be decoded quantitatively. The factor-recovery experiment under Q2 below gives its setup, metrics, full training-latent and held-out test-latent tables, and rule-classification results.
These capabilities also suggest an exciting future direction in policy adaptation. A large robot fleet may share one policy while individual robots differ slightly in their physical properties. We could infer a latent from each robot's interactions and condition the policy on that latent, allowing the shared policy to adapt its actions to those differences. This would extend the environmental information already learned for world prediction to downstream robot control.
Response to Reviewer FEqY (3/7)
We have expanded Section 2 (Related Works) of the revised manuscript with a Latent optimization paragraph covering GLO, DeepSDF, and Image2StyleGAN for reconstruction-oriented analysis-by-synthesis, and CAVIA, CoDA, and FOCA for task/context adaptation and dynamics modeling. The paragraph distinguishes observed-content encoding from environmental-information identification and highlights environment identification through action-conditioned generation in raw, high-dimensional video pixel space. Section 4.5 and Appendix F.1 of the revised manuscript provide the quantitative factor-recovery evidence. The original paper describes the grouped learning setup and alternating optimization in Section 3 and reports the design ablations in Table 3; Appendix B.1 of the revised manuscript provides the training details.
W2: Training requires trajectories to be grouped by environment, while test-time adaptation requires one or more interaction trajectories before prediction. This may be limiting when interactions are costly, unsafe, or when hidden factors change within an episode.
Thank you for raising this practical question! We agree that costly or unsafe interactions can constrain methods that adapt through interaction. However, T2-z does not require extensive observations from every new environment: our experiments show useful adaptation from one or a few test-time interaction trajectories. Learning the environment's features from a small amount of interaction evidence reduces the amount of new data needed for adaptation.
Our data organization also offers a practical route to scaling. Our key observation is that interaction data often already have the grouping structure T2-z needs. When a person or robot interacts with the same object on the same surface over a period of time, mass, mass distribution, and surface friction often remain stable. Episodes from that session can naturally form a training group, even when their physical values are unknown. This allows the model to learn shared environmental information from naturally grouped interactions.
For example, displacement after different pushes provides evidence about the same surface friction. Group membership can be recorded from the collection session, object, and setup without measuring physical parameters or deciding in advance which factors the latent should represent. Additional interactions and object-surface setups can therefore expand the training data. Q3 below examines interaction informativeness, and Q4 describes online latent updates when friction changes during one continuous long task. We have explained natural interaction grouping and its scalability in Appendix B.1 of the revised manuscript.
Q2: Can the authors provide a quantitative factor-recovery analysis (e.g., linear probing, correlation/R² with the ground-truth factors) in addition to PCA visualizations? An oracle baseline conditioned on true physical factors in simulation would also help contextualize the remaining gap.
Quantitative factor recovery
Thank you for these useful suggestions! We pair each learned environment latent with the ground-truth physical quantity of its corresponding environment and fit a function relating the two. For the test-time analysis, we use the latents adapted from a small number of test-time interaction trajectories following the main-paper procedure. We measure how accurately the fitted functions recover the physical quantities from these latents. We also evaluate whether the latents identify the four Light Switch interaction rules.
To analyze the representation learned during training, we fit and evaluate the functions on the complete collection of training latents (Table A). For the test-time latents adapted from a small number of test-time interaction trajectories, we fit the functions using 80% of the physical environments and evaluate on the remaining 20% held out (Table B). Each single-factor task and Diverse Backgrounds uses 50 optimized test latents (40 for fitting, 10 for evaluation); Mass x Friction uses 90 (72 for fitting, 18 for evaluation). Test-time latents within each task share the same initial vector for structure analysis. The corresponding ground-truth physical quantities are used only as targets for fitting the analysis functions; T2-z learns and adapts its latents through prediction. In these tables, n is the complete training-latent count or the held-out test-latent count. We use the following metrics (results shown below).
Response to Reviewer FEqY (1/7)
Thank you for your thorough review and positive assessment! We appreciate your recognition of hidden environmental variation as an important problem for robotic world models, and of the effectiveness of compact latent adaptation. Your questions help us explain the contribution more clearly and examine how the method behaves in practical settings. We are pleased to share the additional analyses and experiments below.
Thank you for this interesting question! T2-z differs fundamentally in its formulation and objective from classical latent-variable optimization, reconstruction-oriented analysis-by-synthesis, and conventional system-identification methods. However, we agree that our central claim is an environment-adaptable robotic world model, as stated in the paper. Our conceptual innovation is to bring autonomous environment-latent learning and optimization to raw pixel observations of the physical world, enabling the model to learn environmental features from interaction videos and robot actions and use them to predict untried action outcomes. We have added the latent-optimization discussion and references to Section 2 (Related Works) of the revised manuscript. We hope these additions make the contribution of our work clearer. We explain the distinctions in these three areas below.
1. Comparison with direct latent optimization
Prediction targets and the role of the latent. CAVIA (Zintgraf et al., 2019) uses task contexts for regression, classification, and reinforcement-learning adaptation. CoDA (Kirchmeyer et al., 2022) uses context vectors to adapt ODE/PDE dynamics models from system trajectories. FOCA (Park et al., 2023) optimizes system-specific contexts for polynomial regression and ODE/PDE state-trajectory prediction. T2-z addresses a substantially different prediction problem: generating high-dimensional future RGB videos conditioned on robot actions, rather than forecasting explicitly represented dynamical-system states or physical fields. Its latent learns environmental information directly from raw physical-world interaction videos and robot actions, without a predefined physical-factor representation. At test time, the latent adapts from interaction trajectories and determines how the environment changes the generated visual outcomes of different actions, including actions not used during adaptation.
Model and latent behavior during training. CAVIA and FOCA do not jointly train a persistent set of per-environment latent parameters alongside the shared model. They reinitialize and adapt a context for each sampled training task, then use the adapted context to learn the shared model. CAVIA performs an outer-loop meta-update; FOCA infers the context with an EMA target model. In T2-z, the world model and the environment latents are both retained and learned throughout training. Different action trajectories from the same environment share one latent. With the latents fixed, the model learns shared action-conditioned predictive dynamics; with the model frozen, the environment latents are updated to explain environment-specific responses. Both updates use the same flow-matching objective on raw interaction videos and robot actions. CoDA jointly learns dynamics parameters and context vectors for system modeling; T2-z develops its alternating model-latent training for generative video prediction and subsequent adaptation from physical-world observations.
The fixed-input experiments in Appendix F.3 of the revised manuscript illustrate this predictive role. We keep the initial frame and action unchanged and vary only the inferred environment latent; Table 18 (simulated Friction) and Figure 12 (real-world Ball Friction) in that appendix show systematic changes in predicted motion. The same inferred latent also transfers across visual backgrounds without further optimization, as evaluated in the Diverse Backgrounds experiment in Table 1 of the original submission. These results show that the environment representation provides a reusable condition for predicting new interactions from raw visual observations.