- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The method improves VLA generalization to unseen viewpoints for robot arms by incorporating system-identification trajectories into its context. Specifically, the VLA's context is extended to include a representation of exploratory motions of the robot arm in the same viewpoint. This allows the VLA to implicitly correspond the visual motions with its internal understanding of the dynamics.
The method is evaluated over simulated LIBERO benchmark and 4 real world pick and place tasks with different viewpoints, showing that it outperforms the conventional, non-context conditioned VLA in viewpoint generalization.
Critical strengths
- The method is very simple and easy to implement
- the ablation experiments are nice and offer insight into why the context representation is helpful
Critical weaknesses
- The novelty in method is somewhat light, since many prior works have (1) used exploratory trajectories / system identification latent variables to condition a policy in non-VLA settings (e.g. RMA) and (2) within the VLA setting, prior work have explored adding history or relevant demonstrations to improve the context (e.g. RICL). The main novelty of this paper would be the combination of these two ingredients and the careful study of this enriched context for viewpoint generalization.
- The experiments are limited to pick and place tasks. It doesn't even seem to require change in gripper pose, just positions. The exploration trajectories seem to be in pure cartesian position space. This makes me question if this can scale up to more complex VLA tasks.
- Misleading claims, the authors use the term "world model" in the title and paper, yet the method itself does not actually do world modeling - most people would associate "world models" with learning some sort of explicit dynamics prediction mechanism. the authors should revise their claims and title, at best they are doing implicit world modeling - and even this is tenuous, a more accurate definition would be some implicit system identification.
Moderate
- the theory and formulation seem to make assumptions that are questionable or incorrect. For example, the formulation claims that "without the system configuration, training forces the model to marginalize over all configurations in the dataset". This is not true - the proprioception, camera viewpoint, and task prompt already reveal information about the system configuration and allow the VLA to handle different viewpoints. The theory formalizing the usefulness of additional exploratory interaction trajectories for system identification in section 3.2 seems largely decorative and also well known in prior work in system identification. I would suggest moving this to appendix, or highlighting any particular novelty of this theoretical formulation, or tying it to the experiments.
Questions for authors
- Can the authors provide more details on how the hidden state representation is formed?
- Why is the method called in-context world modeling when there is no actual explicit world modeling mechanism? at best, you can claim that by doing behavior cloning, the model may be learning an internal world modeling, but this claim is very tenuous (see work from Vafa et. al, Evaluating the world model implicit in a generative model and What has a foundation model found? using inductive bias to probe for world models). in fact there is an entire line of work in LLM pre-training / multi-token prediction objectives that aim to find better objectives to learn internal world models beyond next token prediction (which is analogous to behavior cloning which is what this method relies on). So I am pretty doubtful of this world modeling claim.