acceptodds
Withdrawn from ICLR 2027

RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent world action models draw on video generation priors to jointly predict future observations and robot actions. Adapting these priors to control requires determining how future predictions should guide policy learning. RAP-WAM combines predictive Rehearsal and Action Perturbations (RAP) with future-supervised queries and Coupled Action Projection for policy learning. Predictive rehearsal trains on model-generated inputs while retaining recorded targets, and accumulates recurring errors into a spatial prior. We perturb actions and measure changes in video predictions. Action-response-based spatiotemporal credit assignment combines these responses, prediction error, and the rehearsal prior to allocate visual supervision; spatially aggregating the responses determines supervision intensity across future action steps. No separate scoring network or deployment-time probe is required. The base learns from a video-pretrained backbone without intermediate pretraining on external embodied data. We evaluate the fixed base and its agent-refined, task-conditioned system. On 42 RoboDojo tasks, their average success rates are 12.85% and 15.62%, with Scores of 17.86 and 20.59. Across 15 real-robot tasks, the complete system achieves 89.00% task-macro-average success, exceeding the strongest of five baselines by 37.67 percentage points. Across 315 fixed-output simulation windows, 45.21% of response-top20 positions fall outside error-top20, revealing information beyond prediction error.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Predictive Rehearsal, Spatiotemporal Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jvUGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

RAP-WAM is a world-action model that uses predictive rehearsal and action perturbations to guide training. During rehearsal, it predicts the next chunk from its own previous predictions and accumulates errors into a spatial prior. This prior, prediction errors, and video responses to action perturbations determine the spatial weights for the video loss; the responses also weight the action loss over time. An agent-guided post-training process builds task-specific policies, which are selected by task ID at inference.

Critical strengths
  • Predictive rehearsal adapts self-forcing to help the policy handle its own prediction errors using existing demonstrations.
  • Action-response probing complements prediction error by measuring how actions affect the predicted video.
  • The base model is competitive with pretrained VLAs on RoboDojo without additional embodied pretraining.
  • The real-world evaluation covers diverse tasks and five baselines.
Critical weaknesses
  • Missing ablations. The method combines several training mechanisms and an agent loop without matched-budget retraining ablations. The sensitivity plots describe loss weights without showing their effect on policy performance. Predictive learning and checkpoint accumulation are also loosely connected, leaving their individual contributions unclear.
  • A stronger open-source baseline is omitted. DM0.5 (Dexmal), an open-source VLA listed on the RoboDojo leaderboard on 24 August 2026 with 24.90 Score / 19.34% success, is absent from Table 1 despite outperforming both RAP-WAM (20.59 / 15.62%) and the reported strongest baseline G0.5 (20.23 / 14.88%), and despite predating the paper's 23 September 2026 cutoff by a month.
  • Real-world baseline fairness is unclear. Real-world baseline fine-tuning details are missing. The baselines' poor performance on simple tasks needs explanation.
  • The spatial prior may not transfer across views. A single global 14x14 map averages errors across samples and training. Moving wrist cameras and different task layouts may place relevant regions in different locations, weakening this prior.
  • Spatial credit lacks justification. Measuring how actions affect video does not establish which visual regions matter for action quality. Perturbing visual inputs and measuring action changes may provide a more relevant signal.
  • Temporal credit may misidentify important steps. Free-space motion can produce large visual changes while requiring little precision, whereas contact-rich actions can require high precision with little visual change.
  • Scalability and generality are limited. The method accumulates checkpoints without merging or pruning them, and generalization to unseen tasks is unproven. Real-robot policies take no language input, and all six RoboDojo Open-category tasks have zero success.
  • Paper writing needs improvement. The introduction does not clearly motivate the components, and the method is difficult to follow. A clearer pipeline figure and fewer derivations in the main text would help.
Questions for authors
  • Why is per-task policy selection needed instead of a single multitask policy? How does it compare with the base model on the real robot?
  • Can you provide matched-budget ablations of rehearsal, the prior, response credit, query alignment, attention masking, and CAP?
  • How were the RoboDojo baselines selected? In particular, why is the open-source DM0.5 (listed 24 August 2026, 24.90 / 19.34%) excluded from Table 1?
  • How many real-robot trials were run per task, and were all baselines trained with comparable data and compute?
  • Have you tested visual perturbations for credit assignment? How does the spatial prior behave with moving wrist cameras and different task layouts?
Reviewer_iD15OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RAP-WAM, a world action model for robotic manipulation. It adds future-aligned query tokens, “predictive rehearsal” (training on the model’s own predicted frames), and a loss-reweighting scheme based on how predicted video changes when the action input is perturbed. An LLM-guided post-training stage then selects a checkpoint per task. Experiments cover 42 RoboDojo simulation tasks and 15 real-robot tasks.

Critical strengths
  1. The problem this paper works on, adaptive training emphasis, sounds like an interesting direction to explore: which visual locations and future action steps deserve more supervision is therefore a natural question given that only a small portion of pixels are strongly correlated with actions.
  2. This paper leverages a similar concept from the Self Forcing paper [1]. Similarly, when training WAM, I think it is a reasonable way to use the policy's own prediction as condition to train the model, which can potentially make the policy more robust at inference time.

[1] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

Critical weaknesses
  1. The paper is hard to follow, and the motivation is not clear. I can hardly feel the connection between sentences in the introduction. For example, the first paragraph surveys existing policies and world action models (π0, UWM, DreamZero, X-WAM) without naming any limitation. The second paragraph then asks “which visual locations and future action steps should receive additional training emphasis” without explaining why this question follows or why it matters for control. Many open-ended terms are difficult to understand, such as “predictive rehearsal,” “action perturbations,” and “policy relevance.” For example, predictive rehearsal is mentioned in the abstract, the introduction, and the contribution list, but its actual procedure is only described in Section 3.2. I suggest the authors better motivate the method, clearly introduce the overall task it tackles, and explain how the proposed method addresses that problem.
  2. Weak experiments. The paper proposes several designs, including query alignment, a directional attention mask, CAP, rehearsal, and response-based credit, but provides no ablation studies of these components (as the authors acknowledge in Appendix C.5). This makes it hard to understand the contribution of each design.
Questions for authors
  1. I noticed that in Table 1, though the proposed method achieve the best results in Precision and Long-horizon tasks, its performance is particularly low on Gen-Rand and Open, could you illustrate why?
  2. For comparison in table 1, what is the training or inference budget of the proposed method? How much compute overhead will credit assignment introduce?
Reviewer_eukrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RAP-WAM, which adapts a pretrained video model to jointly predict future observations and robot actions. It uses future-supervised queries to guide action generation and predictive rehearsal to train on model-generated observations while retaining demonstration targets. They also proposed a mechanism that measures video responses to action perturbations, combining them with current and historical prediction errors to weight future visual supervision and using them to weight action steps. Execution-guided post-training with task-specific policy retention further retains complementary policy improvements. The paper evaluates the approach on 42 RoboDojo tasks and 15 real-robot tasks, reporting improved success rates and analyzing how action responses redistribute supervision across visual regions and future action steps

Critical strengths

I would appreciate the following novelties:

  • The paper focuses on an important problem: how to make future prediction useful for robot control. Learning to predict future images does not necessarily preserve the information needed to choose actions. The paper highlights the need to connect future prediction with action learning and to consider which visual details and future steps matter for control.

  • The experiments cover a wide range of robot tasks. The paper evaluates the method on 42 simulation tasks and 15 real-robot tasks, including tasks requiring precise movements and multiple steps. The reported real-robot gains are promising. The paper also examines how the supervision weights change across regions, future steps, and training progress, helping explain how the method works.

  • The method uses existing demonstrations without extra labels. It does not require region labels or a separate scoring model. Rehearsal and action perturbations are used only during training.

Critical weaknesses

However, there are still several concerns about the rigorous and precise of methodology:

A. Action representation and training consistency

  • Blocking direct current visual access may weaken the current-state information needed for control. Action tokens cannot directly attend to video tokens, while predictive queries are supervised using temporally pooled future features. The design provides no explicit mechanism to preserve the current visual details needed for control, such as precise object positions and contact states. The policy may instead rely on demonstration correlations to predict actions. Controlled comparisons with direct video-to-action attention and combined current/future visual features are needed to justify this restriction.

  • Predictive rehearsal can introduce physically inconsistent supervision when predicted inputs change the interaction state but the future targets are still from raw videos. Rehearsal replaces the conditioning observation with a model-generated image while keeping recorded future observations and action targets. This can be reasonable for appearance errors, but becomes problematic when predictions change contact or object configuration. For example, an image showing an unsuccessful grasp may still be paired with a lifting action and a future in which the object rises. Such supervision could encourage the model to overlook physically important state differences rather than learn corrective behavior.

B. Reliability of adaptive supervision. Action-response weights depend on a model whose action–effect relationships are still being learned. The response map is computed from the same WAM being optimized by perturbing noisy demonstration actions and measuring changes in video-denoising outputs. Particularly early in training, this sensitivity may reflect spurious model dependencies rather than physically meaningful action relevance, potentially assigning additional supervision to irrelevant regions.

There are some other minor concerns about the presentation:

  • The format of the numbers in Table 1 is very dense and is hard for readers to parse the key information. The authors should consider organizing this to be more concise.

  • Figure 1 needs to be redesigned for better visualization, as there is a lot of empty space in the figure while the main text and illustrations are small to read.

Questions for authors

Please discuss my main concerns about the technical issues above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.