acceptodds
Under review as a conference paper at ICLR 2027

In-Context World Modeling for Robot Control

Abstract

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.

Primary area:
applications to robotics, autonomy, planning
Keywords:
In-Context Learning, VLA, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UV5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method improves VLA generalization to unseen viewpoints for robot arms by incorporating system-identification trajectories into its context. Specifically, the VLA's context is extended to include a representation of exploratory motions of the robot arm in the same viewpoint. This allows the VLA to implicitly correspond the visual motions with its internal understanding of the dynamics.

The method is evaluated over simulated LIBERO benchmark and 4 real world pick and place tasks with different viewpoints, showing that it outperforms the conventional, non-context conditioned VLA in viewpoint generalization.

Critical strengths
  • The method is very simple and easy to implement
  • the ablation experiments are nice and offer insight into why the context representation is helpful
Critical weaknesses
  • The novelty in method is somewhat light, since many prior works have (1) used exploratory trajectories / system identification latent variables to condition a policy in non-VLA settings (e.g. RMA) and (2) within the VLA setting, prior work have explored adding history or relevant demonstrations to improve the context (e.g. RICL). The main novelty of this paper would be the combination of these two ingredients and the careful study of this enriched context for viewpoint generalization.
  • The experiments are limited to pick and place tasks. It doesn't even seem to require change in gripper pose, just positions. The exploration trajectories seem to be in pure cartesian position space. This makes me question if this can scale up to more complex VLA tasks.
  • Misleading claims, the authors use the term "world model" in the title and paper, yet the method itself does not actually do world modeling - most people would associate "world models" with learning some sort of explicit dynamics prediction mechanism. the authors should revise their claims and title, at best they are doing implicit world modeling - and even this is tenuous, a more accurate definition would be some implicit system identification.

Moderate

  • the theory and formulation seem to make assumptions that are questionable or incorrect. For example, the formulation claims that "without the system configuration, training forces the model to marginalize over all configurations in the dataset". This is not true - the proprioception, camera viewpoint, and task prompt already reveal information about the system configuration and allow the VLA to handle different viewpoints. The theory formalizing the usefulness of additional exploratory interaction trajectories for system identification in section 3.2 seems largely decorative and also well known in prior work in system identification. I would suggest moving this to appendix, or highlighting any particular novelty of this theoretical formulation, or tying it to the experiments.
Questions for authors
  • Can the authors provide more details on how the hidden state representation is formed?
  • Why is the method called in-context world modeling when there is no actual explicit world modeling mechanism? at best, you can claim that by doing behavior cloning, the model may be learning an internal world modeling, but this claim is very tenuous (see work from Vafa et. al, Evaluating the world model implicit in a generative model and What has a foundation model found? using inductive bias to probe for world models). in fact there is an entire line of work in LLM pre-training / multi-token prediction objectives that aim to find better objectives to learn internal world models beyond next token prediction (which is analogous to behavior cloning which is what this method relies on). So I am pretty doubtful of this world modeling claim.
Reviewer_1q5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces In-Context World Modeling (ICWM), which conditions a VLA policy on task-agnostic observation–action–observation transitions collected before execution. Training prefixes are sampled from trajectory slices bucketed by viewpoint, while deployment uses autonomous probing. The method shares the VLA backbone and requires no test-time parameter updates. Experiments evaluate viewpoint generalization on LIBERO and real robots, with additional tests involving visual perturbations and morphological changes.

Critical strengths
  1. The paper addresses an important and practically relevant problem. Enabling VLA policies to adapt to changes in camera viewpoints and robot configurations without task-specific demonstrations or parameter updates at deployment. Reducing the need for configuration-specific data collection and fine-tuning would be valuable for deploying generalist robot policies.
  2. The deployment setting is useful. A robot collects its own adaptation context without requiring task-specific demonstrations under the new configuration. The implementation is simple and compatible with sequence-based VLA training.
  3. The paper provides a broad empirical evaluation. It includes several relevant analyses, including context ablations, mismatched contexts, probing strategies, context length and inference latency. Appendix B also includes RMA-style, PAD-style, memory-style and demonstration-context comparisons.
Critical weaknesses
  1. The claim of implicit world modeling is insufficiently substantiated. World modeling involves learning how an environment evolves in response to actions. The experiments demonstrate that probing context improves adaptation, but do not sufficiently establish that the model captures action-dependent dynamics. Mismatched-context degradation and viewpoint clustering in t-SNE could also reflect sensitivity to static visual configuration. Matched controls that preserve visual cues while disrupting action–observation correspondence would help support the proposed interpretation.

  2. Generalization to combined configuration shifts remains unverified. The paper evaluates camera viewpoint changes and morphological variations largely separately. In particular, the gripper-offset experiments use in-distribution camera viewpoints, leaving it unclear whether ICWM can handle simultaneous changes in viewpoint and morphology. Evaluating the existing gripper offsets under held-out viewpoints would provide stronger evidence for the breadth of its configuration adaptation.

  3. The adaptation baselines lack sufficient implementation detail. Appendix B controls the backbone, task data, and evaluation protocol, but does not explain how the RMA-style configuration embedding is learned and incorporated, how PAD-style updates are performed, or how execution history is represented in the memory-style baseline. These omissions limit reproducibility and make the reported advantage difficult to assess.

Questions for authors
  1. Could you compare correctly paired probing contexts against contexts with identical images but shuffled action labels, using matched training protocols? This would help establish whether adaptation depends on action–observation correspondence beyond visual context alone.

  2. Could you evaluate the existing gripper offsets under held-out camera viewpoints without additional fine-tuning, comparing ICWM with MV? Reporting performance under individual and combined shifts would clarify whether adaptation generalizes to simultaneous viewpoint and morphology changes.

  3. Could you specify the RMA-style embedding’s inputs, training objective, and integration into the policy; the PAD-style adaptation loss, updated parameters, learning rate, and update steps; and the memory-style history contents, window length, and encoding? Please also clarify which test-time transitions each baseline receives and how many.

  4. What is the key methodological novelty of ICWM relative to existing history-conditioned policies and context-based adaptation methods?

Reviewer_KRRDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ICWM, which lets a robot adapt to a new setup by making a few exploratory movements before performing a task. The VLA is trained to use short observation–action clips as context, helping it infer the relationship between its actions and what it sees. At test time, the robot collects this context itself, without new task demonstrations or parameter updates. Experiments in simulation and on real robots show improvements under unseen camera viewpoints, with further tests on visual distractions and changes to robot morphology.

Critical strengths
  1. The paper is well written and easy to follow. The motivation, method, and experiments connect logically, making it clear what the authors propose and how they test it.

  2. The method addresses a practical deployment problem with a simple solution. The robot collects its own interaction context when the setup changes, avoiding new task demonstrations or parameter updates at deployment. This could reduce the effort needed to adapt a trained policy to a new camera placement without introducing a separate system-identification module.

  3. The experiments provide reasonable evidence that the context helps. The main comparison uses the same backbone and task-training data, and improvements appear in both simulation and real-robot experiments. On LIBERO-Long, unseen-view success increases from 19.8% to 25.0%. These controlled comparisons and real-world results make the method’s practical value more convincing.

Critical weaknesses
  1. The experiments do not clearly establish whether ICWM learns the action–observation relationship or mainly uses the additional images to recognize the viewpoint and infer spatial relationships. The action-removal ablation suggests that actions are useful, but it also changes the input format and does not isolate the importance of correctly pairing actions with their visual outcomes. This leaves the paper’s central system-identification claim insufficiently supported.
Questions for authors
  1. How much of the improvement depends on the correct action–outcome correspondence? Could the authors compare correctly paired context with context in which actions are shuffled across transitions, while keeping the images, action set, and input format unchanged?

  2. The paper attributes the gains under morphology changes to inferring the altered kinematics from probing context. Could the authors test this by replacing context collected under the new morphology with context from the original morphology, while keeping the policy, test robot, camera viewpoint, and task fixed? Does correctly matched context provide a clear advantage over mismatched or no context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.