acceptodds
Withdrawn from ICLR 2027

World-as-Prompt: Rewriting Robot Observations for Robust Policy Execution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action and world-action policies can acquire useful manipulation from a modest dataset in one visual domain, yet illumination, sensor noise, object color, and workspace texture turn the same task into an out-of-distribution (OOD) observation. Collecting action-labeled data for each such factor is slow and hard to scale, and wrapping the policy with an extra agent or corrector still leaves it reading the shifted image. We introduce **World-as-Prompt (WAP)**, in which an in-domain reference image and a structured appearance description form a prompt for the world the frozen policy should see. We instantiate that prompt as a world-to-world (W2W) translator. At each control step, the translator conditions on the depth of the current OOD observation, that reference, and the description, changing nuisance appearance while retaining geometry and interaction state. The module is attached before an existing vision-language-action model (VLA) or world-action model (WAM), and requires neither new action-labeled robot data nor policy fine-tuning at deployment. We study WAP on RoboTwin2.0 Clean2Random, LIBERO-Plus, and real-robot cloth folding. Across simulation and real-robot evaluations, WAP improves every evaluated VLA and WAM by to percentage points in OOD success, demonstrating that it restores much of the competence that frozen policies lose under visual shift and offers a plug-and-play way to deploy existing VLAs and WAMs in new visual conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Generate video

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ERPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

World-as-Prompt translates current observations into the visual appearance familiar to a frozen robot policy, using estimated depth, a reference image, and an appearance description. The method is evaluated on two simulation benchmarks and real-world cloth folding with VLA and WAM policies.

Critical strengths
  1. The method improves task success across multiple frozen VLA and WAM policies.

  2. The paper provides useful error analysis, including visual artifacts and cases where translation turns successful executions into failures.

  3. The authors report runtime costs and acknowledge the method’s limitations.

Critical weaknesses
  1. The method does not address the underlying cause of poor visual generalization. It instead relies on repeatedly editing observations to compensate for the policy’s lack of robustness.

  2. Since paired simulation data can already be generated, why not use the same visual variations to train a more robust policy directly? This comparison is essential to justify the proposed approach.

  3. The method is too heavy for responsive robot control, adding roughly 7–9 seconds per policy query. Such delays undermine smooth execution and are particularly problematic for tasks requiring timely feedback, including dynamic and deformable-object manipulation.

Questions for authors

Please see the above weaknesses.

Reviewer_d7kVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the method World-as-Prompt (WAP), which translates visually OOD robot observations into the visual domain of a frozen policy. The approach is simple and effective, and it improves several VLA/WAM policies without finetuning. However, the current evaluation gives limited evidence for generalization to completely unseen scenes, and the method depends heavily on a large 38B translator and monocular depth quality. The real-robot evaluation is small, and stronger ablations and larger real-world tests are needed to better support the paper’s claims.

Critical strengths
  1. The paper introduces a new idea of World-as-Prompt, where visually OOD observations are translated back into the policy’s familiar visual domain. This is a useful direction for handling shifts that mainly change appearance while keeping the task the same.
  2. The method is plug-and-play with current VLA policies. WAP does not require changing or fine-tuning the downstream VLA/WAM.
  3. The result section shows that the gains are consistent across multiple settings. The paper also shows improvements on RoboTwin2.0, LIBERO-Plus, and a real-robot cloth-folding setup, which gives evidence that the idea is not limited to a single benchmark.
Critical weaknesses
  1. Limited generalization to truly unseen visual domains. On LIBERO-Plus, the translator is trained using data from the same benchmark; this does not fully test whether WAP can handle completely new visual domains. A stronger test would evaluate scenes, textures, lighting, and objects that neither the policy nor the translator has seen in the training set.
  2. The translator is very large and expensive. WAP uses a 38B generative model and a large paired visual dataset just to change the policy input. The paper should compare this cost against simpler options such as image augmentation, a smaller translator, or directly improving the policy when action data is available.
  3. The test-time cost is very high. Every policy query must wait for depth estimation and image generation. For example, on LIBERO-Plus with π0, average episode time increases from 73.0 s to 330.8 s with WAP, which makes the method much less practical for real-time control.
  4. It is hard to separate visual appearance from task-relevant information. Color, texture, or object identity may be unimportant for one task but very important for another. For example, if the instruction is “pick up the blue cup,” changing the cup color can directly change the meaning of the task.
  5. The method depends heavily on monocular depth quality. The original RGB is only passed to the translator through DA3 depth, so depth errors can affect object boundaries, contact, and small objects. The paper itself reports depth blur, contact collapse etc. in generated images.
  6. The real-robot evaluation is very limited. The real-world test only uses one cloth-folding task and 15 OOD trials in total. The result is promising, but more tasks, objects, environments, and trials are needed to support strong real-world claims.
  7. Important ablations are missing. The paper should compare depth only, depth + reference, depth + text, and the full model. It would also be useful to compare ground-truth depth with DA3 depth to measure how much performance depends on depth quality.
Questions for authors
  1. Could the authors provide a larger real-robot evaluation? I would like to see substantially more trials across at least several tasks, with changes in camera position and distance rather than keeping the camera setup fixed.
  2. Why is the raw RGB image completely removed from the WAP input? Giving RGB together with depth, reference, and text may help preserve small objects, boundaries, and contact information. An ablation with RGB + depth versus depth only would be useful.
  3. How does WAP perform on completely unseen OOD scenes? I would like to see examples where the translator has not seen the new background, lighting, objects, textures, or their combination during training.
  4. How much of the failure comes from the depth estimator? Please compare ground-truth simulator depth with DA3 depth. This would separate errors caused by WAP from errors already introduced by monocular depth estimation.
  5. How important is each input to WAP? Please compare depth only, depth + reference, depth + text, and depth + reference + text.
Reviewer_NhKvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces World-as-Prompt (WAP), an adaptation framework used for improving frozen robot policies under visual distribution shifts. Specifically, a world-to-world translator in WAP generates an in-domain RGB observation from estimated depth, a reference image, and a structured appearance description. The translator is trained on state-matched observation pairs without action supervision. The paper evaluates the approach with VLA and WAM policies on RoboTwin2.0, LIBERO-Plus, and real-robot cloth folding.

Critical strengths
  1. The paper aims to solve a practical problem: recovering the capabilities of an existing policy under visual shifts without collecting additional action-labeled demonstrations or updating model’s weights.

  2. The LIBERO-Plus results provide substantial evidence that the complete system is useful. With the policy checkpoint and nonvisual inputs held fixed, average success increases from 82.6% to 92.9% for π0 and from 71.4% to 92.9% for Fast-WAM.

  3. The paper explicitly acknowledges important limitations, including transductive adaptation on LIBERO-Plus and also translation latency.

Critical weaknesses
  1. Missing ablation studies: The evaluation part primarily compares raw observations with WAP-translated observations. Comparisons against simpler preprocessing and an appropriately conditioned pretrained translator, together with depth-only, depth-plus-reference, and depth-plus-text ablations, would clarify what the proposed formulation adds.

  2. The training and testing dataset’s relationship isn’t clear. The paper appropriately identifies LIBERO-Plus as transductive adaptation, but the dataset description does not make the relationship between training samples, held-out samples, evaluation tasks, and perturbation configurations sufficiently clear.

Questions for authors
  1. How much does each conditioning input contribute? Please provide matched component ablations.

  2. An interesting question could be: How does the translator preserve the correspondence between current objects and appearance-based identities when depth alone cannot distinguish them? For example, a red cup and a blue cup share exactly the appearance in the depth map.

  3. Please specify the exact train/evaluation split procedure, including separation by trajectory, task, appearance configuration, and reference image.

The success improvements are promising, but the current evidence does not sufficiently support the main idea of this paper. Stronger component comparisons and clearer generalization experiments could substantially improve my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.