acceptodds
Withdrawn from ICLR 2027

PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV—while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action–video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Model, Embodied AI, Video Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLRtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on two important challenges in action-conditioned world models: (1) satisfying three key objectives, i.e., Physical Plausibility, Action Adherence, and Visual Fidelity (PAV), and (2) enhancing the model’s generalization to out-of-distribution (OOD) actions.

To tackle the first challenge, the authors introduce a reinforcement learning post-training framework based on DiffusionNFT, guiding the optimization with specifically tailored reward signals corresponding to the PAV criteria. Specifically, the authors evaluate Physical Plausibility of rollouts using VJEPA-2, leveraging its superior physical world understanding. Action Adherence is quantified by extracting embodiment-focused frames from the rollouts and feeding them into VJEPA-2 to measure the alignment between the robot's motion and the conditioning actions. Visual Fidelity is assessed via a composite of frame-level and temporal-level no-reference quality metrics.

To address the second challenge, the authors synthesize OOD actions by applying three distinct perturbations, i.e., temporal shift, structured magnitude bias, and smooth temporal noise, to non-static ground-truth actions. Then, an Out-of-Distribution Reward Function is introduced during the post-training stage to explicitly improve robustness and generalization across unseen actions.

Extensive experiments are conducted on Wan2.2-5B across multiple datasets. The results demonstrate that the proposed method outperforms existing action-conditioned world models. In addition, ablation studies validate the effectiveness of each individual design component, and downstream experiments further showcase the model’s superiority when serving as a policy evaluator.

Critical strengths

This paper makes a notable contribution by identifying and formalizing three key objectives, i.e., Physical Plausibility, Action Adherence, and Visual Fidelity (PAV), which are essential for action-conditioned world models in embodied AI. To optimize for these criteria, the authors introduce a post-training framework with carefully tailored reward signals. In particular, by leveraging the rich physical priors encoded in the latent space of VJEPA-2, the paper propose the VJEPA Reward alongside an Embodiment-focused VJEPA Reward. Comprehensive evaluations demonstrate the efficacy of this approach, offering valuable insights that could inspire subsequent work in this domain.

Critical weaknesses
  1. The conceptual motivation behind enhancing OOD action generalization via post-training remains unconvincing. Post-training is typically effective at aligning and reinforcing capabilities that the base model has already acquired during pre-training, but rarely imparts entirely unseen, out-of-support capabilities. If the pre-trained base model was never exposed to these OOD actions, it is inherently difficult for post-training alone to fundamentally improve OOD generalization. Moreover, the experiments does not support the authors' claims. The reported experiments appear to be conducted on ID actions. The paper lacks dedicated evaluations or benchmarks specifically under unseen OOD actions.

  2. The framework relies on a combination of multiple reward components. As evidenced by Tables 5 and 6, the final performance appears heavily dependent on delicate hyperparameter tuning. Could the authors provide a principled heuristic for determining these reward hyperparameters? Furthermore, in multi-objective reinforcement learning, conflicting gradient signals among different rewards are a common failure mode. Have the authors observed any interference between the reward objectives? To alleviate this concern, it would be highly informative if the authors could provide training curves illustrating how each individual reward evolves across training steps.

Questions for authors
  1. The authors need to further clarify the meaning of Figure 4, as it is currently difficult to understand. For instance, the text claims that the proposed method mitigates the overestimation of success rate. However, it is unclear how this conclusion is directly supported by the figure.

  2. For action coherence, the authors should provide a more thorough discussion and comparison between VJEPA-2 and IDMs. It remains unclear why VJEPA-2 is preferred over a standard IDM for action adherence, and how much benefit it brings.

3 comments
AuthorsOpenReview ↗

We thank the reviewer for recognizing the PAV formulation’s value and raising questions on post-training limits, reward interactions, and policy evaluation.

W1. Why could post-training improve OOD behavior? The objection assumes that improving OOD action responses requires entirely new physical capabilities. Our setting is narrower: expert demonstrations cover a narrow action distribution, while plausible non-expert deviations can involve familiar motions and interactions. Perturbed actions are explicitly presented to the generator during post-training. They are OOD relative to expert demonstrations but are RL training inputs. Their recorded future videos are missing; our action-conditioned latent predictor supplies rewards for these samples.

10% of samples use this branch; the remainder retain gt supervision. This adapts responses beyond demonstrated actions while retaining expert-trajectory supervision. Tables 1 and 2 report prediction and ablation gains; Section 4.3 tests real-world non-expert policy executions; and the larger-shift experiment tests broader training exploration in response to Reviewer zveU, W3/Q1. These support the targeted setting. Extrapolation to much more distant actions still depends on the action-conditioned predictor's reliability—a boundary of the approach, rather than a barrier to improvement beyond the expert distribution.

W2. Coefficient selection and reward interference. The coefficients follow a concrete selection procedure, with empirical checks beyond a single tuned run. Appendix A.6 screens coefficients by whether a reward ranks recorded futures above 11 alternatives, including temporal corruptions. Table 5 reports RL-based sensitivity to VJEPA weighting and OOD sampling; AgiBot and DROID share the selected configuration. Table 2 shows that removing static regularization, embodiment-focused scoring, or visual-fidelity scoring reduces performance. Thus, the paper provides a practical heuristic and evidence for the components' complementary contributions.

Static regularization counters latent prediction scoring's preference for static outputs (Section 3.2); removing it raises predictor loss from 0.373 to 0.409 (Table 2). The ablations establish the combined objective's benefit at the tested configuration; they do not diagnose gradient alignment throughout training. Figure 7 reports PSNR; individual reward trajectories are currently unavailable. These curves will complement existing sensitivity and ablation studies with training-dynamics analysis and be included in the revised PDF.

Q1. How should Figure 4 be interpreted? The horizontal axis is average real-world task progression; the vertical axis is progression measured in world-model rollouts. Section 4.3 defines progression by completion of ordered task stages, not binary task success. Each point represents a policy-task combination; colors distinguish tasks and marker shapes distinguish the two policies. The dashed diagonal indicates perfect agreement; points above it indicate progression overestimation.

Figure 4 supports improved correspondence with real-world policy performance: correlation increases from 0.863 for Ctrl-World to 0.969 for our model. Overestimation is distinct, measured by positive deviations above the diagonal rather than correlation. We will clarify this distinction, consistently use task progression, and restrict the figure's quantitative conclusion to the reported association. Comparing absolute and signed progression errors would substantiate the calibration claim. We will also clarify rollout counts and define MMRV in the caption.

Q2. Why VJEPA-2 rather than an IDM, and what is the measured benefit? Our choice is supported by the reward interface and diagnostic below. The action-conditioned predictor forecasts latent futures from the initial observation and conditioning actions, providing a reference for scoring generated futures without recorded future videos. Embodiment-focused frames emphasize robot motion, while full frames retain scene interactions. This provides a common latent-prediction framework for physical and action-focused reward streams. The relevant distinction is the scoring design and measured performance; an IDM's inverse direction is not itself a disadvantage.

We also compared evaluators on DROID using the same 12-candidate GT-selection diagnostic:

EvaluatorGT-selection accuracy
UWM, pretrained60%
UWM, co-trained70%
DreamGen IDM, pretrained82%
AC-VJEPA88%

AC-VJEPA achieves 88% GT-selection accuracy, six percentage points above the strongest tested IDM. This quantitatively supports the evaluator choice alongside its compatibility with action-conditioned latent prediction. The diagnostic tests discrimination among recorded, generated, and corrupted futures, including temporal action mismatches. Its scope is evaluator selection under this protocol; isolating downstream RL gains from an IDM-to-VJEPA reward substitution requires a separate controlled training comparison. We will provide evaluator settings and diagnostic sample counts.

Thanks for the authors' response. My concerns have been resolved, and I will keep my score. Please make sure to update Figure 4 by marking the signed progression errors in the figure, as this will better substantiate the paper's claims.

AuthorsOpenReview ↗

Figure 4 improvement.

Thank you for suggesting improvements to Figure 4’s clarity. We will update the PDF after consolidating feedback from all reviewers.

Reviewer_joGbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PAVXploreRL, which is an RL post-training framework for world action models. It explicitly optimizes Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V). For ID data, the rewards combine VJEPA-2 encoder similarity on raw and robot-segmented frames, pixel-level similarity (MSE, SSIM, LPIPS), and a reference-free Visual Fidelity Reward. For OOD actions without paired videos, the framework replaces the ground-truth-dependent rewards with a VJEPA-2-AC predictor-based reward. The base model used is Wan2.2-TI2V-5B, with SFT followed by RL post-training using DiffusionNFT. Experiments on the AgiBot and DROID datasets show improvements over several baselines.

Critical strengths
  • The source code is provided in the supplementary materials, which could help reproducibility.
  • Qualitative comparisons between real-world and generated rollouts are informative.
Critical weaknesses
  • Limited evaluation across model backbones. PAVXploreRL is presented as a model-agnostic RL post-training framework, yet the proposed RL stage is evaluated only on a Wan2.2-based generator. It remains unclear whether the proposed pipeline generalizes to other model architectures, training stages, or checkpoints, including other action-conditioned models such as Cosmos-Predict2.5 Action-Cond. Evaluating the same RL pipeline on at least one additional action-conditioned backbone, along with different SFT checkpoints of the current model, would help establish whether the improvements generalize beyond the chosen training setup.
  • Limited evaluation of OOD benchmarks. The evaluation primarily relies on visual quality and loss-based metrics. These metrics do not necessarily reflect whether the generated dynamics are sufficiently accurate for downstream robotic tasks. Section 4.3 provides policy evaluation, but covers only two policies and five tasks, and does not include the authors' SFT-only model or a variant without OOD exploration. Evaluation on common simulation benchmarks like LIBERO, comparing predicted and actual task outcomes under perturbed actions, would provide stronger evidence for OOD generalization. LIBERO-Plus could further evaluate robustness under broader environment shifts.
  • Insufficient comparison with prior RL methods and reward designs. The current baselines do not isolate the benefits of the proposed RL framework relative to existing post-training techniques. In particular, PersistWorld [1] also applies DiffusionNFT-based RL to robotic world models and evaluates on DROID, but is not included in the comparison. Furthermore, comparing against the VJEPA-2-based rewards used in MIND-V [2] and WMReward [3] would help clarify the contribution of the proposed reward design. The proposed framework largely combines existing DiffusionNFT optimization with VJEPA-2-based rewards, while the main additional contribution appears to be OOD action perturbation and reward combination. Without stronger comparisons, the methodological novelty and benefits over existing approaches remain unclear.

[1] Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

[2] MIND-V: Hierarchical Video Generation for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

[3] Inference-time Physics Alignment of Video Generative Models with Latent World Models

Questions for authors

See the "Critical Weaknesses section".

Other minor comments:

  • The writing could be improved in several places. For example, Section 3.2 could more clearly specify which reward components correspond to P, A, and V, and whether each component is used in the ID setting, the OOD setting, or both.
  • Figure 1 text size is a bit too small.
  • Figure 2 resolution is too low. Consider using SVG or PDF format with vector graphics support.
1 comment
AuthorsOpenReview ↗

We thank the reviewer for the thoughtful feedback and helpful experimental suggestions.

W1. Generality across backbones and checkpoints. We also apply PAVXploreRL to Ctrl-World on DROID. Pixel-only RL and full ID/OOD RL start from the same reproduced Ctrl-World checkpoint and use 5K steps on 8 H200 GPUs under matched settings:

Ctrl-World configurationPSNR upSSIM upLPIPS downEPE downCOS upEnc downPred down
Starting checkpoint21.300.8270.0970.2220.1620.5090.370
Pixel-only RL21.580.8360.0930.2220.1740.5100.369
Full ID/OOD RL21.740.8380.0910.2230.1950.5080.367

These results address whether improvements extend beyond Wan2.2: the full reward improves PSNR, SSIM, LPIPS, COS, and both VJEPA losses over pixel-only RL on Ctrl-World. The evidence supports transfer to a second backbone, with a small trade-off in one metric. Here, model-agnostic describes the framework's interface to an action-conditioned generator and its rollouts; demonstrated applicability now includes two backbones. An SFT checkpoint sweep would further characterize training-stage sensitivity. The comparison matches initialization, updates, and hardware; reward-computation cost is additional.

W2. OOD benchmarks and downstream controls. Our evaluation includes a direct downstream test alongside visual metrics: Section 4.3 compares model-based task progression with actual robot executions for two policies across five tasks, including failures. This evaluates the world model's intended use as a policy evaluator in a real robotic setting. Table 2 separately examines the reward branches and shows gains from the combined ID/OOD configuration. These experiments address complementary questions about downstream utility and training design.

Adding SFT-only and no-OOD variants to the downstream evaluation would connect these analyses more directly and isolate exploration's contribution to policy-evaluation accuracy. That comparison remains open. LIBERO and LIBERO-Plus would extend coverage to controlled action and environment shifts, respectively; they complement the existing real-robot evidence. Our claims concern the tested robotic settings. The larger-shift experiment in our response to Reviewer zveU further tests the training perturbation design, without substituting for an independent OOD outcome benchmark.

W3. Prior RL methods and reward designs. The distinction from prior work is the supervision available during action exploration. PersistWorld addresses autoregressive rollout degradation using GT-based visual rewards [1]. MIND-V uses VJEPA-based physical alignment within hierarchical manipulation-video generation [2]. WMReward uses latent prediction consistency for inference-time alignment [3]. PAVXploreRL uses an action-conditioned predictor to score explored robot actions without paired future videos and combines this supervision with paired ID training. Sharing an optimizer or a pretrained representation does not remove this distinction in training capability. We will sharpen this comparison in related work.

We evaluated released PersistWorld weights under our DROID evaluation protocol:

ModelPSNR upSSIM upLPIPS downEPE downCOS upEnc downPred down
PersistWorld, evaluated by us21.490.8320.0930.2340.1730.5110.369
PAVXploreRL, Wan2.2 backbone21.710.8280.0920.2270.1970.5090.369

This adds the requested direct comparison with the closest RL baseline. Under the common protocol, our model achieves better overall. The Ctrl-World experiment complements this model-level comparison by fixing the starting checkpoint when comparing pixel-only and full-reward RL. Direct substitutions of MIND-V and WMReward rewards would further separate reward choices from other training differences; the present evidence supports our combined framework rather than a ranking of all VJEPA reward formulations.

Presentation questions. We will add the following mapping to Section 3.2 and improve the readability of Figures 1 and 2:

Intended emphasisRewardID useOOD use
Physical plausibilityFull-frame VJEPACompare generated and GT encodingsCompare generated encoding and action-conditioned latent prediction
Action adherenceEmbodiment-focused VJEPAEncoder comparison on robot-focused framesPredictor comparison on robot-focused frames
Visual fidelitySpatial and temporal metrics without an external GT videoYesYes
Reconstruction fidelityMSE, SSIM, LPIPS against GTYesNo

The components have complementary emphases, with raw and embodiment-focused streams sharing the same ID/OOD supervision structure. We will also distinguish interframe SSIM from SSIM against GT, enlarge Figure 1 labels, and replace Figure 2 with a vector graphic.

Reviewer_zveUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a RL post-training method for WM learning. To facilitate (P)hysical plausibility, (A)ction adherence and (V)isual fidelity not only in ID area but also in OOD areas, authors design a PAVXpolre framework where OOD trajectories acquired with control noises, with no corresponding GT videos, are evaluated through rewards calculated in VJEPA-2 latent space. Empirical results show improvement on prediction comparing to other WM baselines on both Agibot and Droid datasets. Ablations have demonstrated the effectiveness of individual reward/regularization terms together with their hyperparameters.

Critical strengths
  1. It's important to have world model behave well in OOD areas. This paper proposes an RL framework where reward functions help world model learning in both ID and OOD areas through V-JEPA latent space.

  2. This paper presents a comprehensive assessment on the RL framework presented, including reward scales. It also provides a reward coefficient selection method for reference.

Critical weaknesses
  1. The core of this paper lies in the reinforcement learning reward shaping for world model learning. Five reward components are proposed, which raise my concern for hyperparameter tuning. There are more than five reward scales on account of the regularization terms, which makes the method harder to reproduce and tune. Though A.6 provides a coefficient selection method, the video variants are basically hand-crafted and seems to care little on embodiment objectives such as action adherence.

  2. The framework of adding action noise to strengthen OOD capability through RL itself presents novelty, but the ID reward functions basically follow the common practice in WM post-training literature, while the OOD rewards seem to depend heavily on V-JEPA2, with rVJEPA−Pr_{VJEPA-P} basically a distillation from its feature space.

  3. The evaluation results on the 1% validation set is basically still ID evaluation, which share similar distribution with training dataset. Under the framework proposed, OOD capability is restricted to hand-crafted OOD action distribution, following the construction method outlined in L211-215.

Questions for authors
  1. Related to W3, what about OOD actions with more domain differences? Does the framework pushes those samples to higher errors?

  2. Is there any ablation on the action noise permutation types? What if removing one of the three permutations?

1 comment
AuthorsOpenReview ↗

We thank the reviewer for recognizing the value of OOD action exploration and suggesting larger action distribution shifts.

W1. Reward complexity and coefficient selection. The reward has two supervision branches, rather than five independent rewards applied to every sample. For ID samples, the pixel, VJEPA, and visual-fidelity weights satisfy lambda_pixel = 1 - lambda_VJEPA - lambda_VF; the OOD branch replaces GT-based supervision with an action-conditioned predictor reward. Embodiment weighting and static regularization modify the VJEPA reward, while alpha_OOD controls branch sampling. AgiBot and DROID share the same coefficients.

The concern that Appendix A.6’s GT-based reward selection underemphasizes action adherence overlooks its temporal diagnostic: world-model-generated videos exhibit subtle deviations from conditioning actions, while truncation, slowdown, and static repetition introduce larger mismatches. These alternatives probe temporal action consistency and visual quality. The diagnostic screens coefficients before expensive RL training, while RL-based sensitivity experiments (Table 5) and component ablations (Table 2) support the selected configuration. Together, these provide a reproducible selection procedure and empirical checks beyond the handcrafted diagnostic.

W2. Dependence on VJEPA-2 and contribution. Knowledge transfer from VJEPA-2 is an intentional mechanism of the method. Describing Eq. 8 as latent distillation identifies that mechanism, but does not account for the capability it enables: post-training an action-conditioned video generator on explored actions without recorded future videos. The methodological contribution lies in making these samples trainable alongside GT-based ID samples, using action-conditioned latent prediction, embodiment-focused scoring, static regularization, and visual-fidelity rewards. This contribution concerns the training signal and action exploration; DiffusionNFT supplies the optimizer.

The gains extend beyond the teacher's feature-space loss. In Table 2 on Agibot, the full ID/OOD configuration achieves LPIPS 0.126, compared with 0.135 for Pixel + ID after 1K RL steps. Appendix A.4 reports a blinded evaluation of 100 AgiBot trajectories: RL is preferred over PT for physical plausibility in 60.5% of ratings and action adherence in 65.0%, with ties reported separately. These independent human judgments support improvements in generated trajectories beyond matching the VJEPA score. Our evaluator comparison with IDMs is detailed in our response to Reviewer eLRt, Q2.

W3/Q1. What is established about OOD actions and larger shifts? The evaluation has two distinct roles. Held-out expert action-video pairs measure prediction quality under a common protocol for baseline comparison. Section 4.3 evaluates policy-generated behavior, including unsuccessful real executions, across two policies and five tasks. The latter goes beyond expert-trajectory reconstruction and should inform assessment of non-expert behavior. Our operational definition of OOD is deviation from the recorded expert action distribution; unseen embodiments and unrestricted environment changes are broader generalization settings.

We additionally trained a larger-shift variant that replaces a window's endpoint pose with a pose sampled one window length, 3.2 seconds, away and smoothly adjusts the trajectory. Both configurations use 5K AgiBot RL steps on eight H200 GPUs, with other settings fixed:

Training perturbationPSNR upSSIM upLPIPS downEPE downCOS upEnc downPred down
Original21.200.8120.1270.3060.2310.5210.373
Larger shift21.250.8150.1260.3050.2310.5210.370

Increasing the training shift preserves or improves every reported metric at the shown precision. Thus, the tested increase in exploration does not degrade evaluation performance. This directly addresses sensitivity to a broader training perturbation. Predictive accuracy under independently collected, more distant OOD actions is a separate question: it depends on the reward predictor remaining informative and requires paired outcomes for those actions. We will distinguish these questions explicitly.

Q2. Ablation of the three perturbations. The three perturbations target different deviations: temporal misalignment, systematic amplitude bias, and smooth drift. Temporal misalignment is our primary perturbation; removing it effectively deactivates action exploration. Our existing ablations establish the value of the reward branches and auxiliary components; the additional experiment above tests exploration strength. Individual perturbation-removal runs are not included. A component-level comparison should report the resulting action deviation, because removing a perturbation changes both the error type and total perturbation strength. This is a useful extension of the current ablations, rather than a prerequisite for the demonstrated ability to train on actions without paired future videos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.