acceptodds
Withdrawn from ICLR 2027

Actions as Rays: Repurposing Video Diffusion Models for Robot Control with Visual Action Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained video diffusion models have learned rich priors over visual dynamics and physical interactions, yet adapting them to robot manipulation remains challenging, as existing approaches often rely on dedicated action heads, specialized tokenizers, or auxiliary policy networks to translate between visual observations and low-dimensional action signals. In this work, we introduce RayWAM, a World-Action Model that addresses this challenge through Action Raymap, a video-aligned and overcomplete action representation that transforms robot actions into structured RGB maps. By interpreting each end-effector action as the pose of a virtual camera, Action Raymap represents translation, rotation, and gripper states as dense spatial fields, while maintaining a deterministic mapping back to executable robot commands. Based on this formulation, observations and actions are represented in a unified RGB space and encoded using the same frozen pretrained video VAE, projecting robot actions directly into the latent space of pretrained video diffusion models. Unlike prior approaches that adapt video models through additional action-specific modules, RayWAM instead adapts the action representation itself to match the native modality of the pretrained model, enabling more effective exploitation of transferable representations and physical dynamics priors learned from large-scale video data. We evaluate RayWAM on LIBERO, LIBERO-Plus, and VLABench, together with real-world robotic experiments, demonstrating consistent improvements in task success rates, robustness to environmental variations, and real-world generalization across diverse manipulation scenarios.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, world-action models, visual action representations, robot control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_TTGpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RayWAM, a World-Action Model that adapts pretrained video diffusion models for robot control by representing robot actions as visual Action Raymaps. The paper evaluates RayWAM on 3 simulation benchmarks and real world. The reported results show competitive performance using a 1.3B backbone without additional embodied pretraining.

Critical strengths
  1. Strong engineering integration and comprehensive experiments
Critical weaknesses

The core novelty of Action Raymap is unclear because it closely follows existing ray-based visual conditioning paradigms.

A key concern is that the core novelty of Action Raymap remains unclear, as it closely mirrors existing ray-based visual conditioning paradigms. Though presented as a novel action representation, it bears strong resemblance to established ray-based camera representations in diffusion models: camera trajectory control methods already use ray or Plücker-like encodings to provide dense spatial conditioning compatible with visual architectures. The primary distinction appears to be replacing camera poses with robot end-effector poses. The paper does not adequately justify why this constitutes a fundamental WAM contribution rather than a straightforward domain transfer.

The paper does not compare against the most relevant visual action conditioning baselines.

The most relevant comparison is not necessarily other robot policies. It should compare against methods that already perform: visual action prompts, camera/ray conditioning, trajectory image representations...... Without such comparisons, it is difficult to determine whether the improvement comes from: better representation or better architecture or simply stronger visual conditioning.

Questions for authors

Comparison with visual action prompt methods

Have the authors compared Action Raymap against:

  • action images,
  • optical flow actions,
  • trajectory heatmaps,
  • embodiment masks,
  • other visual action conditioning approaches?

This comparison is important to establish superiority.

Reviewer_c7TgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RayWAM, which encodes robot states and actions as structured RGB images called Action Raymaps. A shared pretrained video VAE and diffusion backbone jointly predict future observations and action maps, which are converted into executable actions through deterministic geometric recovery.

Critical strengths
  1. Action Raymap is a clean geometric representation that supports deterministic action recovery without real-camera calibration or an additional learned action decoder.

  2. The evaluation covers several benchmarks and real-world tasks. The VAE round-trip experiments are particularly useful for identifying information loss in different action representations.

  3. The method achieves competitive performance with a 1.3B video backbone, making it a promising interface for adapting video models to robot control.

Critical weaknesses
  1. The experiments show VAE compatibility, not transfer of physical priors. Flatten-Resize and Shuffled Raymap have large reconstruction errors and zero policy success. This shows that these encodings fail at the VAE bottleneck. It does not show that Raymap uses physical knowledge learned from video pretraining.

  2. The attention ablation does not explain what the visual branch contributes. Removing visual–action interaction reduces success from 93% to 78%, showing that future visual features help. But these features could simply provide target locations or desired end states. The experiment does not distinguish these benefits from physical knowledge transfer.

  3. The paper lacks a strong action expert baseline on the same backbone. Latent-Repeat achieves 87.8% success versus 93% for Raymap, but repeating numerical values is a weak baseline. The relevant comparison is a dedicated numerical action expert with the same backbone, data, visual supervision, and training budget. Without it, the benefit of turning actions into images remains unclear.

  4. The performance gains are small, and inference speed is a concern. RayWAM improves over Fast-WAM by only 2.4 points on VLABench and 3.3 points on real-world tasks. It requires 20 denoising steps and VAE decoding, but the paper reports no inference latency or control frequency. It is unclear whether these gains justify the computational cost or whether the policy can react quickly enough during real-world execution.

Questions for authors
  1. What limitation of existing action experts does converting actions into images actually address?
  2. What evidence shows that RayWAM transfers physical knowledge from video pretraining, rather than simply providing a VAE-compatible action representation?
  3. Does joint visual–action prediction improve action learning beyond what the same backbone with a strong numerical action expert can achieve?
  4. How does Action Raymap help ground language instructions in the correct actions, beyond changing how actions are encoded?
  5. What is the main advantage of RayWAM over existing approaches: better performance, improved data efficiency, stronger generalization, or a simpler architecture?
  6. Are the performance gains substantial enough to justify the computational cost of generating actions through a video diffusion model?
Reviewer_hskqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RayWAM, a world action model that represents robot actions as RGB action raymaps, replacing the end-effector pose representation. This design choice helps with reusing the video model backbone as it is with no changes, and without adding any extra action experts. The results show that RayWAM achieves a better performance than FastWAM in LIBERO, LIBERO-Plus, VLA-Bench, and Real-world experiments.

Critical strengths
  • Representing robot actions as raymaps has that the benefit of reusing the video model with no extra action experts, which reduce the size and computational requirements.
  • The generalization results under different distribution-shift settings, especially in camera, layout, spatial settings in LIBERO-Plus and Real-world experiments. RayWAM is mostly best in such settings, so it might also be good to investigate more on this if the different way of action representation has any effect of performance under different spatial and geometric settings.
Critical weaknesses
  • The authors don't provide a controlled comparison between their action raymap method and having an action expert head to their base video model and then provide detailed results for both on performance, number of parameters, computational efficiency. Because normally, an action is represented by a 7D vector. while an action raymap uses 224x224x3 ~= 150k pixel values to represent the same action. So we need to understand more about the gains we get out of this action representations.
  • The authors don't provide a version of their method that has an additional embodied pretraining, similar to all other methods they compare with (OpenWAM, π0.5), and these methods clearly outperforms RayWAM in all benchmarks. So the comparison here is not fair. Also, the authors didn't provide OpenWAM results on the VLABench, OpenWAM paper shows much better results than π0.5 on VLABench.
Questions for authors

Requests:

  • Could you provide a controlled experiment where you compare RayWAM with a version with an action expert and compare between them on LIBERO-Plus or VLABench?
  • Could you also provide a version of RayWAM that is pretrained on embodiment data so that it could be comparable with other methods?
  • Could you provide results on VLABench under different distribution shift settings? like camera, spatial, layout?

Questions:

  • Could you explain more about how this action representation could be if we want to represent different robot embodiments, like a bi-manual robot or a whole-body control. Do we need to add extra channels. or could we still use the same 3 channels?
  • How is the performance sensitive to the raymap resolution? Is it possible to use smaller resolutions?
  • What is the encoding/decoding latency for converting between robot actions and action raymaps?
Reviewer_YbwsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper encodes end-effector poses and gripper commands as RGB raymaps, so a pretrained video diffusion model can jointly generate future observations and executable actions. Experiments cover three simulation benchmarks and real-robot manipulation.

Critical strengths

Encoding actions as raymaps is an interesting way to reuse video models for control and avoids real-camera calibration for action recovery. The controlled comparison against the Cosmos-style Latent-Repeat baseline is informative: with the original VAE, raymaps raise LIBERO-Goal success from 87.8% to 93.0%. The broader simulation and real-robot results also support the representation's usefulness.

Critical weaknesses
  1. The practical trade-off is unclear. Low-dimensional actions become full spatial latent grids, and every query requires 20 joint denoising steps plus VAE decoding. The paper gives no matched comparison of latency or memory against a compact action head. UVA shows that joint video-action modeling can bypass video generation when decoding actions. Without a comparison of this kind, it is hard to judge whether the representation is worth its cost.

  2. The evidence for the specific geometric design is incomplete. Flatten-Resize and shuffled raymaps both incur severe VAE reconstruction errors, so their failure conflates VAE compatibility with ray geometry. The advantage over another smooth, decodable encoding remains unresolved. The paper already cites Action Images; a comparison accounting for its camera-calibration requirements would better establish what the new representation adds.


[1] Unified Video Action Model
[2] Action Images: End-to-End Policy Learning via Multiview Video Generation

Questions for authors
  1. With matched data and training budgets, how do success, end-to-end latency, and peak memory compare with a compact numerical action head on the same backbone?

  2. Does the advantage remain against a smooth non-ray encoding with the same pose conventions and comparable VAE reconstruction accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.