- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on actions in world models in two ways. Firstly authors propose a method to identify a temporal shift between actions and visual using small perturbations. Secondly, authors show that under certain assumptions, the action effect estimated by a model too simple can be worse than no effect at all, and provide a numerical constraint as a condition to ensure that action effect is more accurate than no effect. Authors run experiments in simple environments such as pendulum and cart-pole with a synthetic, constant, action lag using different model architectures, as well as a larger experiment on the Walker/Walk simulator. Results show that the action lag is identified correctly and that while most models’ prediction beat copying the next frame, they struggle to model the difference between actions accurately.
Critical strengths
- World models typically focus their evaluation more heavily on visuals. It is nice to see work that focuses on actions and how action modelling impacts world models beyond visual quality.
- The work comprises a lot of experiments that are very thorough in the setup considered. For example, experiments comprise multiple seeds and multiple architectures.
- The work is theoretically grounded in toy scenarios.
Critical weaknesses
- My main concern is the presentation of this work. The paper is extremely difficult to read and understand. The main paper is full of jargon and equation variables which are not introduced or explained. For example, variables \pi, and U in equation (1) are never introduced. Variable q is in equation (1), yet its purpose is never discussed, only the challenges associated with using q are discussed in section 3.3. Most of the methodology has to be inferred from context. While the appendix is more readable to some extent and provides some clarifications, this is far from enough to make the paper readable and understandable. The abstract and Figure 1 are great examples of this: they cannot be understood until the full paper has been read.
- The action lag experiments are sensible but don’t seem broadly applicable beyond a toy environment. It would require access to the data collection procedure to add perturbation and assume action offsets are constant throughout the trajectories. A concrete example of how this would happen and be used in practice would be useful.
- The action fidelity experiments have very narrow findings. There is no evidence that the models evaluated here have been trained sufficiently. It is uncertain whether this is a problem of undertraining, lack of capacity, insufficient data, or if action fidelity is poor, despite the model having the ability to generate high quality visuals.
- The core message of the paper seems to be that fixing action timing does not guarantee action fidelity, which doesn’t seem like a groundbreaking finding? Temporally aligned data does not guarantee that complex dynamics can be learned accurately.
Questions for authors
- Can authors reframe their contributions and core findings in simpler, understandable terms and provide intuition?
- How could the action lag experiment be used in practice on real data?
- Is the main objective of action fidelity experiments to show that actions can be inaccurate while visuals are good quality (thus highlighting the important of evaluating action quality), or are authors aiming for a different message?