acceptodds
Under review as a conference paper at ICLR 2027

T2-z: Test-Time Optimization of Environment Latents for Robotic World Models

12 views

Abstract

Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take. During training, T2-z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations. Across 7 simulated settings and 4 real-world robotic tasks, T2-z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-z can efficiently infer environment latents from as little as a single test-time interaction trajectory.

Primary area:
generative models
Keywords:
World Model, Test Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FEqYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a practically important limitation of robotic world models: visually similar observations and identical actions can lead to different outcomes because of hidden environmental factors such as friction, mass, stiffness, mass distribution, or interaction rules. T2-Z augments an action-conditioned video world model with a compact 32-D environment latent. During training, the method alternates between world-model optimization with fixed latents and environment-latent optimization with the world model frozen. At test time, the world model remains frozen and only a newly initialized environment latent is optimized from one or more interaction trajectories using the same flow-matching objective. The evaluation covers seven simulated settings and four real-world robotic tasks. T2-Z consistently improves physical prediction and action selection over a standard world model, LoRA-based test-time adaptation, an amortized DINOv2 context encoder, and a TTT-KQV baseline. The learned latent space also exhibits structured PCA patterns that correlate with underlying factors, including a two-dimensional organization when mass and friction vary jointly.

Critical strengths
  1. A meaningful problem formulation. The paper focuses on hidden environmental variation that is not directly observable from images but can substantially change action-conditioned dynamics. This is an important failure mode for world models intended for real-world robotic use.

  2. A simple and effective solution with broad validation. T2-Z adapts only a low-dimensional latent instead of updating the full world model, making the method conceptually clean and computationally lightweight. The gains are consistent across seven simulation settings and four real-world tasks, and the emergent latent structure provides additional evidence that the representation captures meaningful environment variation.

Critical weaknesses
  1. The methodological novelty is moderate. Direct optimization of latent codes through prediction error, together with alternating model/latent optimization, is conceptually related to classical latent-variable optimization, analysis-by-synthesis, and system-identification methods. The main contribution therefore appears to be the problem formulation and its effective instantiation in a modern robotic video world model rather than a fundamentally new optimization principle. The paper would benefit from a more explicit discussion of this novelty boundary.

  2. The current formulation has several practical assumptions and the latent analysis could be stronger. Training requires trajectories to be grouped by environment, while test-time adaptation requires one or more interaction trajectories before prediction. This may be limiting when interactions are costly, unsafe, or when hidden factors change within an episode. In addition, the PCA visualizations are suggestive but do not quantitatively establish factor recovery or disentanglement; linear probing, correlation analysis, or an oracle physical-parameter baseline would strengthen this claim.

Questions for authors
  1. Can the authors clarify the conceptual novelty of T2-Z relative to prior direct latent optimization and system-identification approaches, and explain what is specifically enabled by the proposed formulation beyond applying latent inversion to a robotic world model?

  2. Can the authors provide a quantitative factor-recovery analysis (e.g., linear probing, correlation/R² with the ground-truth factors) in addition to PCA visualizations? An oracle baseline conditioned on true physical factors in simulation would also help contextualize the remaining gap.

  3. How sensitive is performance to the type and informativeness of the test-time interaction trajectories? The current evaluation uses randomly sampled actions; what happens when the interaction is weakly informative or when exploratory actions are constrained?

  4. How does the method behave when hidden environmental factors change within an episode? Is a single stationary environment latent sufficient, or would online/temporal latent updates be required?

8 comments
AuthorsOpenReview ↗

Response to Reviewer FEqY (1/7)

Thank you for your thorough review and positive assessment! We appreciate your recognition of hidden environmental variation as an important problem for robotic world models, and of the effectiveness of compact latent adaptation. Your questions help us explain the contribution more clearly and examine how the method behaves in practical settings. We are pleased to share the additional analyses and experiments below.

W1 and Q1: Can the authors clarify the conceptual novelty of T2-Z relative to prior direct latent optimization and system-identification approaches, and explain what is specifically enabled by the proposed formulation beyond applying latent inversion to a robotic world model?

Thank you for this interesting question! T2-z differs fundamentally in its formulation and objective from classical latent-variable optimization, reconstruction-oriented analysis-by-synthesis, and conventional system-identification methods. However, we agree that our central claim is an environment-adaptable robotic world model, as stated in the paper. Our conceptual innovation is to bring autonomous environment-latent learning and optimization to raw pixel observations of the physical world, enabling the model to learn environmental features from interaction videos and robot actions and use them to predict untried action outcomes. We have added the latent-optimization discussion and references to Section 2 (Related Works) of the revised manuscript. We hope these additions make the contribution of our work clearer. We explain the distinctions in these three areas below.

1. Comparison with direct latent optimization

  • Prediction targets and the role of the latent. CAVIA (Zintgraf et al., 2019) uses task contexts for regression, classification, and reinforcement-learning adaptation. CoDA (Kirchmeyer et al., 2022) uses context vectors to adapt ODE/PDE dynamics models from system trajectories. FOCA (Park et al., 2023) optimizes system-specific contexts for polynomial regression and ODE/PDE state-trajectory prediction. T2-z addresses a substantially different prediction problem: generating high-dimensional future RGB videos conditioned on robot actions, rather than forecasting explicitly represented dynamical-system states or physical fields. Its latent learns environmental information directly from raw physical-world interaction videos and robot actions, without a predefined physical-factor representation. At test time, the latent adapts from interaction trajectories and determines how the environment changes the generated visual outcomes of different actions, including actions not used during adaptation.

  • Model and latent behavior during training. CAVIA and FOCA do not jointly train a persistent set of per-environment latent parameters alongside the shared model. They reinitialize and adapt a context for each sampled training task, then use the adapted context to learn the shared model. CAVIA performs an outer-loop meta-update; FOCA infers the context with an EMA target model. In T2-z, the world model and the environment latents are both retained and learned throughout training. Different action trajectories from the same environment share one latent. With the latents fixed, the model learns shared action-conditioned predictive dynamics; with the model frozen, the environment latents are updated to explain environment-specific responses. Both updates use the same flow-matching objective on raw interaction videos and robot actions. CoDA jointly learns dynamics parameters and context vectors for system modeling; T2-z develops its alternating model-latent training for generative video prediction and subsequent adaptation from physical-world observations.

The fixed-input experiments in Appendix F.3 of the revised manuscript illustrate this predictive role. We keep the initial frame and action unchanged and vary only the inferred environment latent; Table 18 (simulated Friction) and Figure 12 (real-world Ball Friction) in that appendix show systematic changes in predicted motion. The same inferred latent also transfers across visual backgrounds without further optimization, as evaluated in the Diverse Backgrounds experiment in Table 1 of the original submission. These results show that the environment representation provides a reusable condition for predicting new interactions from raw visual observations.

AuthorsOpenReview ↗

Response to Reviewer FEqY (2/7)

Our compact environment representation and alternating training recipe make this environment-level learning and adaptation effective in a generative video world model. Under the same backbone and 32-D representation, Table 3 of the original submission shows that alternating training reduces mean physical prediction error from 7.03 with joint training to 3.85 and increases action success from 68% to 72%. Directly optimizing a 3072-D conditioning token gives an error of 94.50 and action success of 24%. These ablations establish the practical contribution of the representation and training design to learning environmental information from high-dimensional videos and using it for adaptation.

2. Comparison with analysis-by-synthesis

Analysis-by-synthesis methods explain observations by optimizing a code through a generative model. GLO (Bojanowski et al., ICML 2018) learns per-image codes and a generator through reconstruction; DeepSDF (Park et al., CVPR 2019) learns shape codes and a signed-distance decoder; Image2StyleGAN (Abdal et al., ICCV 2019) optimizes a code in a pretrained generator to reconstruct an input image. These methods use the code to represent and reconstruct observed visual content. T2-z focuses on adapting the environment represented by a robotic world model. Its latent is inferred from observed interactions and then reused to predict the consequences of different, untried actions. Thus, the representation explains the observed interaction while also supplying environmental information for future action-conditioned video prediction.

3. Relationship to system identification

Many learning-based system-identification methods use observations to infer system parameters or environmental features within an already established model or latent representation. For example, RMA (Kumar et al., RSS 2021) first learns an environment embedding and policy from privileged physical factors, then trains an adaptation module to predict that embedding from state-action history. T2-z learns the environment representation itself from grouped raw videos and actions through world prediction. The model discovers the environmental information needed to explain the observations and predict new actions, without specifying the physical factors or supervising their values.

We also examine ISE (Ko et al., 2025), which uses video generation and latent inversion to explain observed interactions. The distinction concerns how the representation being optimized is constructed and supervised. T2-z learns that environment-conditioning space directly from grouped raw interaction data, rather than constructing it from task-defined successful interactions.

The resulting latent space captures a form of environmental understanding grounded in raw physical-world interactions. The model learns which regularities are shared across interactions in an environment and uses them to explain its responses to different actions. We see this as a step toward building models that can learn regularities directly from raw, high-dimensional interactions with the physical world and put those regularities to use. Our formulation enables the following capabilities.

  • Learning environmental features with environment grouping alone. Raw interaction videos, their associated actions, and shared-environment membership are sufficient to learn T2-z's environment representation, without additional physical-parameter, task-goal, or success/failure labels. ISE's main implementation obtains video features from DINOv2 and selects a task-labeled successful interaction's encoding as each object's representative state embedding. Its representation construction therefore uses a task definition and a corresponding definition of successful interaction. T2-z instead learns which environmental features explain the interactions through video prediction itself. This enables environmental regularities to emerge without defining them through task success.

  • Recovering physical information from the learned representation. We evaluate whether the representation learned through video prediction contains physical information that can be decoded quantitatively. The factor-recovery experiment under Q2 below gives its setup, metrics, full training-latent and held-out test-latent tables, and rule-classification results.

These capabilities also suggest an exciting future direction in policy adaptation. A large robot fleet may share one policy while individual robots differ slightly in their physical properties. We could infer a latent from each robot's interactions and condition the policy on that latent, allowing the shared policy to adapt its actions to those differences. This would extend the environmental information already learned for world prediction to downstream robot control.

AuthorsOpenReview ↗

Response to Reviewer FEqY (3/7)

We have expanded Section 2 (Related Works) of the revised manuscript with a Latent optimization paragraph covering GLO, DeepSDF, and Image2StyleGAN for reconstruction-oriented analysis-by-synthesis, and CAVIA, CoDA, and FOCA for task/context adaptation and dynamics modeling. The paragraph distinguishes observed-content encoding from environmental-information identification and highlights environment identification through action-conditioned generation in raw, high-dimensional video pixel space. Section 4.5 and Appendix F.1 of the revised manuscript provide the quantitative factor-recovery evidence. The original paper describes the grouped learning setup and alternating optimization in Section 3 and reports the design ablations in Table 3; Appendix B.1 of the revised manuscript provides the training details.

W2: Training requires trajectories to be grouped by environment, while test-time adaptation requires one or more interaction trajectories before prediction. This may be limiting when interactions are costly, unsafe, or when hidden factors change within an episode.

Thank you for raising this practical question! We agree that costly or unsafe interactions can constrain methods that adapt through interaction. However, T2-z does not require extensive observations from every new environment: our experiments show useful adaptation from one or a few test-time interaction trajectories. Learning the environment's features from a small amount of interaction evidence reduces the amount of new data needed for adaptation.

Our data organization also offers a practical route to scaling. Our key observation is that interaction data often already have the grouping structure T2-z needs. When a person or robot interacts with the same object on the same surface over a period of time, mass, mass distribution, and surface friction often remain stable. Episodes from that session can naturally form a training group, even when their physical values are unknown. This allows the model to learn shared environmental information from naturally grouped interactions.

For example, displacement after different pushes provides evidence about the same surface friction. Group membership can be recorded from the collection session, object, and setup without measuring physical parameters or deciding in advance which factors the latent should represent. Additional interactions and object-surface setups can therefore expand the training data. Q3 below examines interaction informativeness, and Q4 describes online latent updates when friction changes during one continuous long task. We have explained natural interaction grouping and its scalability in Appendix B.1 of the revised manuscript.

Q2: Can the authors provide a quantitative factor-recovery analysis (e.g., linear probing, correlation/R² with the ground-truth factors) in addition to PCA visualizations? An oracle baseline conditioned on true physical factors in simulation would also help contextualize the remaining gap.

Quantitative factor recovery

Thank you for these useful suggestions! We pair each learned environment latent with the ground-truth physical quantity of its corresponding environment and fit a function relating the two. For the test-time analysis, we use the latents adapted from a small number of test-time interaction trajectories following the main-paper procedure. We measure how accurately the fitted functions recover the physical quantities from these latents. We also evaluate whether the latents identify the four Light Switch interaction rules.

To analyze the representation learned during training, we fit and evaluate the functions on the complete collection of training latents (Table A). For the test-time latents adapted from a small number of test-time interaction trajectories, we fit the functions using 80% of the physical environments and evaluate on the remaining 20% held out (Table B). Each single-factor task and Diverse Backgrounds uses 50 optimized test latents (40 for fitting, 10 for evaluation); Mass x Friction uses 90 (72 for fitting, 18 for evaluation). Test-time latents within each task share the same initial vector for structure analysis. The corresponding ground-truth physical quantities are used only as targets for fitting the analysis functions; T2-z learns and adapts its latents through prediction. In these tables, n is the complete training-latent count or the held-out test-latent count. We use the following metrics (results shown below).

Reviewer_qXeXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes adapting robotic world models by optimizing a compact environment latent from test-time interaction trajectories while keeping the world model frozen. Training alternates between model updates and environment-latent updates. Experiments cover seven simulated settings and four real-world tasks, evaluating prediction accuracy and action selection.

Critical strengths

The method is conceptually clear and avoids updating the full world model during test-time adaptation. The experiments show promising improvements in task-relevant prediction and downstream action selection. The appendix includes useful analyses of interaction count, latent dimensionality, and computational cost.

Critical weaknesses

The methodological advance of alternating model and latent optimization needs sharper positioning relative to prediction-based latent inference and system identification. The present comparisons establish practical benefits more clearly than conceptual novelty. Claims that the latents discover underlying factors and disentangle dynamics from appearance rely mainly on PCA visualizations. These plots do not establish factor identifiability or independence from appearance. The simulation table averages physical errors measured in incompatible units, including pixels, degrees, and lamp scores. The resulting aggregate error and its improvement lack a clear interpretation.

Questions for authors

Can the authors quantitatively test latent transfer across appearances while holding dynamics fixed, and sensitivity to interventions on individual factors? Can the authors report normalized task-level error aggregation and variability across training runs and latent initializations?

7 comments
AuthorsOpenReview ↗

Response to Reviewer qXeX (1/6)

Thank you for your thoughtful review! We appreciate your recognition of the clear latent-only adaptation mechanism and the improvements in prediction and action selection. Your questions about novelty, physical structure, and appearance transfer are especially helpful for explaining what T2-z enables. We are glad to address them with the experiments and clarifications below.

W1: The methodological advance of alternating model and latent optimization needs sharper positioning relative to prediction-based latent inference and system identification. The present comparisons establish practical benefits more clearly than conceptual novelty.

Thank you for asking us to clarify the role and contribution of alternating optimization! T2-z uses alternating model-latent optimization to learn environmental information directly from raw, high-dimensional videos of physical-world interactions and their associated robot actions. The model-update blocks learn shared predictive dynamics with the latents fixed, while the latent-update blocks fit environment-specific responses with the model frozen. This learns a video predictor together with an environment-conditioning space that can subsequently be adapted through prediction. We compare this learning procedure with prediction-based latent inference and system identification below.

1. Compared with prediction-based latent inference

In T2-z, alternating model and latent updates connect environment-representation learning, environment identification, and new-action prediction within one action-conditioned generative video world model. The contribution concerns how this alternating procedure learns from raw physical-world interactions and makes the resulting environment representation useful for video prediction and adaptation. In our examination of related literature, we consider CAVIA (Zintgraf et al., 2019), CoDA (Kirchmeyer et al., 2022), and FOCA (Park et al., 2023). These works provide context-adaptation foundations. We specify T2-z's alternating-learning formulation from the following two perspectives.

  • Prediction targets and the role of the latent. CAVIA uses task contexts for regression, classification, and reinforcement learning; CoDA uses contexts to adapt ODE/PDE dynamics models; FOCA optimizes system-specific contexts for polynomial regression and ODE/PDE state-trajectory prediction. T2-z addresses a substantially different prediction problem: generating high-dimensional future RGB videos conditioned on robot actions, rather than forecasting explicitly represented dynamical-system states or physical fields. Its environment latent is learned from raw physical-world interaction videos and determines how hidden environmental conditions change the generated visual outcomes of different actions. During alternating training, trajectories with different actions in the same environment jointly constrain the shared predictor and that environment latent. At test time, the latent adapts directly from RGB videos and robot actions in test-time interaction trajectories and conditions the frozen model to predict untried actions. The Friction intervention in Table E below and the Ball Friction intervention in Figure 12 of the revised manuscript (Appendix F.3) show this predictive role: changing the environment latent systematically changes predicted motion under the same initial frame and action.

  • Model and latent behavior during training. CAVIA and FOCA do not jointly train a persistent set of per-environment latent parameters alongside the shared model. They reinitialize and adapt a context for each sampled training task, then use the adapted context to learn the shared model. CAVIA performs an outer-loop meta-update; FOCA infers the context with an EMA target model. In T2-z, the world model and the environment latents are both retained and learned throughout training. Different action trajectories from the same environment share one latent. With the latents fixed, the model learns shared action-conditioned predictive dynamics; with the model frozen, the environment latents are updated to explain environment-specific responses. Both updates use the same flow-matching objective on raw interaction videos and robot actions. CoDA jointly learns dynamics parameters and context vectors for system modeling; T2-z develops its alternating model-latent training for generative video prediction and subsequent adaptation from physical-world observations.

AuthorsOpenReview ↗

Response to Reviewer qXeX (2/6)

Our alternating optimization and compact latent design help build a model that learns environmental information directly from high-dimensional videos of physical-world interactions. Table 3 of the original submission compares joint and alternating training with the same world-model backbone and 32-D environment representation. Alternating training reduces mean physical error from 7.03 to 3.85 and improves action success from 68% to 72%. Direct optimization of a 3072-D conditioning token gives an error of 94.50 and action success of 24%. These ablations establish the engineering contribution of the compact representation and alternating training procedure: together, they make environment inference from raw video interactions effective within a generative world model.

2. Compared with system identification

Many learning-based system-identification methods frame adaptation as using observations to infer system parameters or environmental features within an already established model or latent representation. For example, RMA (Kumar et al., RSS 2021) first trains an environmental-factor encoder together with a control policy using privileged factors such as friction and payload. An adaptation module then learns to predict that embedding from state-action history. T2-z instead alternates world-model and environment-latent updates to learn the environmental representation itself from grouped action-conditioned videos. The model discovers which environmental information is useful for prediction, without specifying physical-factor identities or supplying numerical factor values. At test time, it infers the latent directly through the frozen world model's prediction objective and reuses it to predict new action outcomes.

The original paper describes the alternating world-model and environment-latent updates and latent-only inference in Section 3, and reports the training and latent-parameterization ablations in Table 3. We have added a Latent optimization paragraph to Section 2 (Related Works) of the revised manuscript. It discusses CAVIA, CoDA, and FOCA and clarifies the distinction between task adaptation/dynamics modeling and environment identification through action-conditioned video generation.

W2: Claims that the latents discover underlying factors and disentangle dynamics from appearance rely mainly on PCA visualizations. These plots do not establish factor identifiability or independence from appearance.

Q1: Can the authors quantitatively test latent transfer across appearances while holding dynamics fixed, and sensitivity to interventions on individual factors?

Thank you for these insightful suggestions! We examine the representation from three complementary perspectives: physical-factor recovery, reuse across appearances with dynamics fixed, and changes in predictions when the latent is varied under fixed visual inputs and actions. We describe the setup, metrics, and results of each experiment below.

Quantitative physical-factor recovery

We pair each learned environment latent with the ground-truth physical quantity of its corresponding environment and fit a function relating the two. For the test-time analysis, we use the latents adapted from a small number of test-time interaction trajectories following the main-paper procedure. We measure how accurately the fitted functions recover the physical quantities from these latents. We also evaluate whether the latents identify the four Light Switch interaction rules.

To analyze the representation learned during training, we fit and evaluate the functions on the complete collection of training latents (Table A). For the test-time latents adapted from a small number of test-time interaction trajectories, we fit the functions using 80% of the physical environments and evaluate on the remaining 20% held out (Table B). Each single-factor task and Diverse Backgrounds uses 50 optimized test latents (40 for fitting, 10 for evaluation); Mass x Friction uses 90 (72 for fitting, 18 for evaluation). Test-time latents within each task share the same initial vector for structure analysis. The corresponding ground-truth physical quantities are used only as targets for fitting the analysis functions; T2-z learns and adapts its latents through prediction. In these tables, n is the complete training-latent count or the held-out test-latent count. We use the following metrics (results shown below).

AuthorsOpenReview ↗

Response to Reviewer qXeX (3/6)

  • R-squared: recovery fit quality relative to predicting the mean physical value.
  • Spearman rho: rank correlation between estimated and true physical values.
  • MAE: mean absolute error in the physical factor's original units.
  • MAPE: average relative error. For each sample, relative error is the absolute difference between the estimated and true physical values divided by the magnitude of the true physical value, expressed as a percentage.
  • 10%/20% recovery: fraction of physical-parameter estimates whose relative error is at most 10% or 20%; equivalently, the absolute estimation error is within 10% or 20% of the magnitude of the true physical value.
  • Ordering accuracy: pairwise ordering accuracy across different physical values; a correct order scores 1, a predicted tie 0.5, and an incorrect order 0.
  • Rule-classification accuracy: fraction of environments whose complete Light Switch rule is identified correctly.
  • AUROC: area under the receiver-operating-characteristic curve for each button's control prediction.

Table A. Full-collection training-latent fits and metrics

FactorFitting methodnR-squaredSpearman rhoMAEMAPE<=20%
Friction muRational function350.950690.98100.0089414.97%80.00%
Gravity gRBF regression200.997661.00000.204124.14%100.00%
Mass Collision mRBF regression200.999961.00000.001932.39%100.00%
Mass Balance ratioRBF regression200.946930.99250.295908.60%95.00%
Mass x Friction mThin-plate spline840.985400.99420.024829.73%91.67%
Mass x Friction muThin-plate spline840.850290.96300.0068813.74%72.62%
Diverse Backgrounds muCubic polynomial150.974570.99290.0065115.40%86.67%

Table A shows that the training latents contain recoverable physical information. Mass Collision and Gravity reach R-squared values of 0.99996 and 0.99766. For Mass x Friction, the training fits recover mass and friction with MAPE of 9.73% and 13.74%; both factors are within 20% relative error for 58 of 84 latents (69.05%). Friction uses a rational function; Gravity, Mass Collision, and Mass Balance use RBF regression; Mass x Friction uses a thin-plate spline; and Diverse Backgrounds uses a cubic polynomial. RBF refers to radial basis functions.

We next evaluate how well the latents adapted at test time from a small number of test-time interaction trajectories recover their environments' physical quantities. Table B reports the same recovery metrics on the held-out physical environments (results shown below).

Table B. Physical-factor recovery on held-out physical environments

FactorFitting methodHeld-out nR-squaredSpearman rhoMAEMAPE<=10%<=20%Ordering
Friction muRBF kernel ridge100.98690.92680.0061538.82%60.00%100.00%90.70%
Gravity gPrototype regression100.97590.96340.73644512.99%60.00%80.00%93.33%
Mass Collision mMatern kernel ridge100.96240.99390.06129513.79%40.00%60.00%100.00%
Mass Balance ratioNormalized Gaussian kernel regression100.99630.97560.083936.51%80.00%90.00%97.67%
Mass x Friction mRBF kernel ridge180.96890.92400.02371512.95%55.56%83.33%94.33%
Mass x Friction muRBF kernel ridge180.94050.99640.00549913.86%50.00%83.33%100.00%
Diverse Backgrounds muAffine regression100.639360.87040.0199314.33%10.00%90.00%100.00%

Table B shows accurate recovery on held-out physical environments. Friction, Gravity, Mass Collision, and Mass Balance achieve R-squared of 0.9869, 0.9759, 0.9624, and 0.9963, with MAPE of 8.82%, 12.99%, 13.79%, and 6.51%. In Mass x Friction, mass/friction R-squared is 0.96890/0.94050 and MAPE is 12.95%/13.86%. Both factors are recovered within 20% relative error for 13 of 18 held-out latents (72.22%), and within 10% for 8 of 18 (44.44%); their mean relative error is 13.40%. Diverse Backgrounds gives R-squared 0.63936, Spearman rho 0.8704, MAPE 14.33%, and recovery within 20% of 90.00%.

Reviewer_bqYcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method for adapting action-conditioned world models to unobserved environmental factors. The method introduces a learnable environmental latent that is optimized separately from the world model. At inference, the latent is updated based on observation data. Experiments across several simulated and real-world settings show improvements in object prediction and action success.

Critical strengths
  1. The hidden environmental dynamics are directly optimized at test-time through interaction data without needing to predefine environmental factors or learn an environmental encoder.
  2. The method is evaluated extensively across simulated and real-world environments.
Critical weaknesses
  1. Most experiments evaluate held-out values or configurations of environmental factors already represented during training. Would be interested to see whether it can adapt to completely unseen environmental factors / larger distribution shifts.
  2. Results on emergent structure of environmental latents is largely qualitative.
  3. The results would benefit from more statistical analysis. Mainly Tables 1 and 2.
  4. The paper requires substantial clean-up in presentation. Figures and tables are densely packed and difficult to read. Formatting can be improved. Writting towards the end could use more polishing.
Questions for authors

Refer to weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bqYc (1/4)

Thank you for your careful review and constructive suggestions! We appreciate your recognition that T2-z adapts to hidden environmental dynamics directly from interaction data, without predefined factors or a learned environment encoder. Your questions help us clarify this capability and quantify the learned representation. We address each point with the experiments and clarifications below.

W1: Most experiments evaluate held-out values or configurations of environmental factors already represented during training. Would be interested to see whether it can adapt to completely unseen environmental factors / larger distribution shifts.

Thank you for this interesting question! We test adaptation to a physical factor absent from training by introducing tabletop inclination. A world model trained only on horizontal-table Friction must account for the added tangential gravity component. We optimize only the environment latent from one test-time interaction trajectory, with the world model frozen and neither friction nor inclination values provided.

We evaluate 180 query videos from 16 inclined environments and 4 horizontal controls under the main-paper protocol, comparing T2-z with and without test-time latent optimization. ADE is the mean object-centroid prediction error across video frames; FDE is the object-centroid error at the final frame. Both are measured in pixels (results shown below).

Table A1. Adaptation to tabletop inclination absent from training

EvaluationT2-z without test-time latent optimizationT2-z
All 20 environments10.24 / 21.723.08 / 5.96
16 inclined environments10.56 / 22.383.06 / 6.02
4 horizontal controls8.97 / 19.093.15 / 5.72

Latent optimization reduces ADE/FDE from 10.24/21.72 to 3.08/5.96 pixels across all 20 environments. The improvement appears in both the inclined environments and the horizontal controls. This demonstrates that T2-z can explain changes caused by an added physical factor using visual interaction alone.

We also test whether the adapted latent captures how inclination changes predicted motion. For each base friction, we compare the same query action on the horizontal table and at four inclinations. Action amplitude is 0.21 at friction 0.02, 0.14 at 0.06, 0.17 at 0.10, and 0.21 at 0.15. Within each comparison, predictions and GT use the same displacement-measurement interval; the lowest-friction group is measured before the block leaves view. Positive inclination denotes downhill motion, and Delta Pred is the predicted displacement change relative to the matched horizontal control (results shown below).

Table A2. Matched-action displacement relative to horizontal controls

Base muInclination (deg)GT (px)Pred (px)Delta Pred (px)
0.02+0.9255.3850.97-0.89
0.02+0.4653.2852.13+0.27
0.020.0051.1751.860.00
0.02-0.4647.4946.69-5.18
0.02-0.9244.7745.04-6.82
0.06+0.8619.0118.02+6.26
0.060.0014.8911.760.00
0.06-0.8611.8710.63-1.13
0.06-1.729.119.64-2.12
0.06-2.588.618.93-2.84
0.10+4.5779.5876.01+58.14
0.10+2.2928.8933.07+15.20
0.100.0018.6717.870.00
0.10-2.3013.3613.86-4.02
0.10-4.6111.8811.31-6.57
0.15+1.7223.6723.12+1.00
0.15+0.8621.5022.110.00
0.150.0019.7322.110.00
0.15-1.4419.0220.39-1.72
0.15-2.8815.0920.45-1.66

The predictions capture the expected effect of inclination. At base friction 0.06, predicted travel increases from 11.76 pixels on the horizontal table to 18.02 pixels downhill, and decreases to 8.93-10.63 pixels uphill. At friction 0.10, it increases from 17.87 to 33.07-76.01 pixels downhill and decreases to 11.31-13.86 pixels uphill. Across the four comparisons, the predicted direction of change agrees with GT in 14 of 16 inclined configurations. These results show a capacity to adapt to an added physical factor, even with the action and world model fixed. We have included the experiment in Section 4.5 and Appendix F.4 (Tables 19 and 20) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer bqYc (2/4)

W2: Results on emergent structure of environmental latents is largely qualitative.

Thank you for this suggestion! We pair each learned environment latent with the ground-truth physical quantity of its corresponding environment and fit a function relating the two. For the test-time analysis, we use the latents adapted from a small number of test-time interaction trajectories following the main-paper procedure. We measure how accurately the fitted functions recover the physical quantities from these latents. We also evaluate whether the latents identify the four Light Switch interaction rules.

To analyze the representation learned during training, we fit and evaluate the functions on the complete collection of training latents (Table B). For the test-time latents adapted from a small number of test-time interaction trajectories, we fit the functions using 80% of the physical environments and evaluate on the remaining 20% held out (Table C). Each single-factor task and Diverse Backgrounds uses 50 optimized test latents (40 for fitting, 10 for evaluation); Mass x Friction uses 90 (72 for fitting, 18 for evaluation). Test-time latents within each task share the same initial vector for structure analysis. The corresponding ground-truth physical quantities are used only as targets for fitting the analysis functions; T2-z learns and adapts its latents through prediction. In these tables, n is the complete training-latent count or the held-out test-latent count. We use the following metrics (results shown below).

  • R-squared: recovery fit quality relative to predicting the mean physical value.
  • Spearman rho: rank correlation between estimated and true physical values.
  • MAE: mean absolute error in the physical factor's original units.
  • MAPE: average relative error. For each sample, relative error is the absolute difference between the estimated and true physical values divided by the magnitude of the true physical value, expressed as a percentage.
  • 10%/20% recovery: fraction of physical-parameter estimates whose relative error is at most 10% or 20%; equivalently, the absolute estimation error is within 10% or 20% of the magnitude of the true physical value.
  • Ordering accuracy: pairwise ordering accuracy across different physical values; a correct order scores 1, a predicted tie 0.5, and an incorrect order 0.
  • Rule-classification accuracy: fraction of environments whose complete Light Switch rule is identified correctly.
  • AUROC: area under the receiver-operating-characteristic curve for each button's control prediction.

Table B. Full-collection training-latent fits and metrics

FactorFitting methodnR-squaredSpearman rhoMAEMAPE<=20%
Friction muRational function350.950690.98100.0089414.97%80.00%
Gravity gRBF regression200.997661.00000.204124.14%100.00%
Mass Collision mRBF regression200.999961.00000.001932.39%100.00%
Mass Balance ratioRBF regression200.946930.99250.295908.60%95.00%
Mass x Friction mThin-plate spline840.985400.99420.024829.73%91.67%
Mass x Friction muThin-plate spline840.850290.96300.0068813.74%72.62%
Diverse Backgrounds muCubic polynomial150.974570.99290.0065115.40%86.67%

Table B shows that the training latents contain recoverable physical information. Mass Collision and Gravity reach R-squared values of 0.99996 and 0.99766. For Mass x Friction, the training fits recover mass and friction with MAPE of 9.73% and 13.74%; both factors are within 20% relative error for 58 of 84 latents (69.05%). Friction uses a rational function; Gravity, Mass Collision, and Mass Balance use RBF regression; Mass x Friction uses a thin-plate spline; and Diverse Backgrounds uses a cubic polynomial. RBF refers to radial basis functions.

AuthorsOpenReview ↗

Response to Reviewer bqYc (3/4)

We next evaluate how well the latents adapted at test time from a small number of test-time interaction trajectories recover their environments' physical quantities. Table C reports the same recovery metrics on the held-out physical environments (results shown below).

Table C. Physical-factor recovery on held-out physical environments

FactorFitting methodHeld-out nR-squaredSpearman rhoMAEMAPE<=10%<=20%Ordering
Friction muRBF kernel ridge100.98690.92680.0061538.82%60.00%100.00%90.70%
Gravity gPrototype regression100.97590.96340.73644512.99%60.00%80.00%93.33%
Mass Collision mMatern kernel ridge100.96240.99390.06129513.79%40.00%60.00%100.00%
Mass Balance ratioNormalized Gaussian kernel regression100.99630.97560.083936.51%80.00%90.00%97.67%
Mass x Friction mRBF kernel ridge180.96890.92400.02371512.95%55.56%83.33%94.33%
Mass x Friction muRBF kernel ridge180.94050.99640.00549913.86%50.00%83.33%100.00%
Diverse Backgrounds muAffine regression100.639360.87040.0199314.33%10.00%90.00%100.00%

Table C shows accurate recovery on held-out physical environments. Friction, Gravity, Mass Collision, and Mass Balance achieve R-squared of 0.9869, 0.9759, 0.9624, and 0.9963, with MAPE of 8.82%, 12.99%, 13.79%, and 6.51%. In Mass x Friction, mass/friction R-squared is 0.96890/0.94050 and MAPE is 12.95%/13.86%. Both factors are recovered within 20% relative error for 13 of 18 held-out latents (72.22%), and within 10% for 8 of 18 (44.44%); their mean relative error is 13.40%. Diverse Backgrounds gives R-squared 0.63936, Spearman rho 0.8704, MAPE 14.33%, and recovery within 20% of 90.00%.

For these test-latent results, Friction and Mass x Friction use RBF kernel ridge regression; Mass Collision uses Matern kernel ridge regression; Mass Balance uses normalized Gaussian kernel regression; Gravity uses prototype regression; and Diverse Backgrounds uses affine regression. The physical quantities are therefore recoverable from latents learned through video prediction, including when mass and friction vary together.

We also test whether the same representation captures discrete interaction rules. Light Switch has four rules: neither button controls the lamp, red only, blue only, or both. Each prediction contains one binary label per button, and joint rule accuracy requires both labels to be correct (results shown below in Table D).

Table D. Light Switch rule-classification accuracy

Latent collectionClassifiernJoint rule accuracy
Training (full collection)Linear softmax12100.00%
Test (held out)Two-output RBF10100.00%

Table D gives 100.00% joint rule accuracy for both training and held-out test latents. For the held-out analysis, each binary head reaches 100.00% accuracy and AUROC 1.0000; red-only versus blue-only discrimination is also 100.00%. We use a linear softmax classifier for training latents and an RBF regressor with two outputs for held-out test latents. These results extend the recovered structure from continuous physical factors to human-defined interaction rules.

Across the seven continuous task-factor pairs, held-out recovery gives macro-averaged R-squared 0.9243, Spearman rho 0.9501, MAPE 11.89%, and 83.81% recovery within 20% relative error. Together, these analyses quantitatively connect the learned representation to underlying physical factors and interaction rules. We have added the summary to Section 4.5 and the full protocol, functions, and results to Appendix F.1 (Tables 13-15) of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful reading and constructive suggestions! In addition to our responses to individual reviewers, we would like to highlight our contributions and the new experiments we have added in this rebuttal.

[Our Contributions]

We are glad that the reviewers recognize the following contributions.

  • A practical formulation for adapting robotic world models to hidden environmental variation [FEqY].
  • Direct adaptation from test-time interactions without predefined environmental factors or a learned environment encoder [bqYc].
  • Lightweight latent adaptation and gains in prediction and action selection across simulated and real-world tasks [qXeX, FEqY].

[Additional Experiments]

In this rebuttal, we have added supporting experiments and analyses to address the reviewers' concerns.

  • Quantitative latent analysis covering fits on the full collection of training latents (Table 13) and physical-factor recovery from test-time latents on held-out environments (Table 14), both in Appendix F.1 of the revised manuscript [bqYc, qXeX, FEqY].
  • Statistical analysis of results from Tables 1 and 2 of the original submission, including task-normalized error aggregation (Appendix F.2) [bqYc, qXeX].
  • Sensitivity of model predictions to changes in the environment latent in a single-factor Friction intervention (Table 18, Appendix F.3) [qXeX].
  • Adaptation to an unseen physical factor in the tabletop inclination experiment (Appendix F.4) [bqYc].
  • Comparison with oracles conditioned on true physical parameters in simulation (Appendix F.5) [FEqY].
  • Robustness of test-time adaptation when a push barely moves the block or sends it out of view early (Appendix F.6) [FEqY].
  • Online latent adaptation to a change in friction within a long task (Appendix F.7) [FEqY].
  • Variability across training runs and test-time latent initializations in Friction (Appendix F.8) [qXeX].

[Qualitative examples]

  • In real-world Ball Friction, changing only the environment latent changes the predicted ball travel distance under the same initial frame and action (Figure 12 of the revised manuscript, Appendix F.3) [qXeX].

[Clarifications and Presentation]

  • Expanded the discussion of latent optimization, analysis-by-synthesis, and system identification (Section 2) [qXeX, FEqY].
  • Clarified how the original Diverse Backgrounds evaluation tests latent transfer across appearances at fixed friction by reusing a latent optimized on one background without further optimization (Section 4.2 and Appendix F.3) [qXeX].
  • Improved spacing and readability of figures and tables and polished the writing [bqYc].

Thank you again for your thoughtful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.