acceptodds
Under review as a conference paper at ICLR 2027

ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences

Abstract

Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reward Model, Action-conditioned World Model, Inverse Dynamics Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CDW1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduce AVM-Eval to evaluate action following and find that IDM performance differs between real and generated videos. They further construct AVM-Pref and propose ACWM-RM, showing improvements in visual quality and action following over the evaluated reward baselines in downstream RL experiments.

Critical strengths
  1. The paper proposes AVM-Eval to assess the reliability of IDM-based action following evaluation on real and generated videos, providing a benchmark for analyzing existing IDM rewards.
  2. The paper constructs AVM-Pref with action-mismatch and video-mismatch preferences to supervise action following and visual quality, respectively. Ablations show their complementarity and improved ranking performance through joint training.
  3. ACWM-RM combines dynamics pretraining with preference learning to evaluate visual quality and action following without reference videos. RL experiments on DreamDojo-2B show improvements over compared reward baselines in both aspects.
Critical weaknesses
  1. When there is a conflict between AF and VQ, which type of video does ACWM-RM tend to prefer? I'd like to know whether this preference will change as the quality gap widens.
  2. The authors used PSNR/LPIPS to construct video preference pairs during training, and the evaluation in Table 5 also includes these metrics. Although SSIM and FVD are also reported, visual quality assessments independent of these training metrics still need to be strengthened. Could the authors provide additional evaluations that do not rely on these metrics?
  3. Could the authors report the time required for ACWM-RM to compute rewards for each video clip and compare it to the time taken for reward computation by IDM-, metric-, and VLM-based methods? This would help readers assess the actual computational overhead of ACWM-RM in fine-tuning.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to reviewer CDW1 (1/3)

Dear Reviewer CDW1,

We appreciate your time and constructive feedback, which has been invaluable in improving our work. We also express our sincere gratitude for your positive assessment of our paper and for recognizing the value of our contributions.

W1. When AF and VQ conflict, which does ACWM-RM prioritize, and how does this preference change as the quality gap widens?

Thank you for this insightful question. Our new experiment shows that when AF and VQ conflict, ACWM-RM tends to prefer the higher-VQ video. This preference strengthens as the VQ gap widens and generally weakens as the AF gap widens.

Specifically, we build the test set from the pairs excluded when constructing the video-mismatch pairs of AVM-Eval, which require the positive sample to have a higher AF score than the negative sample and similar VQ scores. Among them, we keep the conflicting pairs, in which one video has a higher AF score, and the other has a higher VQ score. Following Appendix I, the AF gap is the nDTW difference (Eq. (51), line 1868), and the VQ gap is the root mean square of the differences in the four VQ metrics, each normalized by its corresponding standard deviation (Eq. (52), line 1800). We split each gap into small, medium, and large levels by its tertiles (Table 1), and randomly sample 250 pairs from each of the resulting nine cells, yielding 2,250 pairs.

Table 1. Gap thresholds.

GapSmallMediumLarge
AF(0, 11.65](11.65, 44.50]> 44.50
VQ(0, 0.31](0.31, 0.53]> 0.53

Table 2. ACWM-RM’s preference rate (%) for the higher-VQ video.

AF gap / VQ gapSmallMediumLarge
Small82.888.891.2
Medium76.487.291.6
Large69.684.088.8

As shown in Table 2, preference rates of ACWM-RM for the higher-VQ video range from 69.6% to 91.6% across all cells. This preference strengthens as the VQ gap widens (e.g., 69.6% → 88.8% under a large AF gap) and generally weakens as the AF gap widens (e.g., 82.8% → 69.6% under a small VQ gap), suggesting that ACWM-RM remains sensitive to AF.

We attribute this preference mainly to our training data. Video-mismatch pairs, which provide explicit VQ supervision, outnumber action-mismatch pairs, which provide explicit AF supervision. As a result, ACWM-RM may rely more on VQ cues when AF and VQ conflict.

Related revisions: We have added this experiment to Appendix J (lines 2018–2037; Tables 24 and 25). All revisions are highlighted in red in the manuscript.

AuthorsOpenReview ↗

Response to reviewer CDW1 (2/3)

W2. Visual quality evaluation needs stronger evidence from metrics independent of the PSNR and LPIPS used to construct training preference pairs.

Thank you for this suggestion. We have added two metrics that are not used to construct our training preference pairs: FID and L2$\text{Latent}.AsshowninTable3,ACWM−RMachievesthelowestFIDandL2. As shown in Table 3, ACWM-RM achieves the lowest FID and L2\text{Latent}$, outperforming the strongest baseline, metric-based, on both metrics. This indicates that the gains of ACWM-RM are not limited to the metrics used for preference supervision.

Both metrics are used to evaluate world models in IRASim [1] and OSCAR [2]. They assess visual quality from perspectives complementary to PSNR and LPIPS: FID measures distributional similarity between generated and real frames in Inception feature space, while L2$\text{Latent}$ measures per-sample fidelity in a VAE latent space. Following IRASim, we compute L2$\text{Latent}$ using the pretrained SDXL VAE as the latent encoder.

Table 3. Evaluation on metrics not used for preference supervision. L2$_\text{Latent}$ is reported ×100.

MethodFID ↓L2$_\text{Latent}$ ↓
Original5.01 ± 0.0525.12 ± 0.03
IDM-based5.39 ± 0.0925.87 ± 0.14
VLM-based5.47 ± 0.1125.26 ± 0.07
Metric-based4.30 ± 0.0425.04 ± 0.06
Ours4.25 ± 0.0724.90 ± 0.06

[1] IRASim: A Fine-Grained World Model for Robot Manipulation. ICCV 2025.

[2] OSCAR: Omni-Embodiment Skeleton-Conditioned World Action Model for Robotics. arXiv 2026.

Related revisions: We have added FID and L2$_\text{Latent}$ results to Table 5 and updated the evaluation protocol in Section 5.1 (lines 364–367) and the visual-quality discussion in Section 5.3 (lines 420–427). All revisions are highlighted in red in the manuscript.

AuthorsOpenReview ↗

Response to reviewer CDW1 (3/3)

W3. How does ACWM-RM’s per-clip reward computation time compare with IDM-, metric-, and VLM-based methods, and what computational overhead does it add during fine-tuning?

We thank the reviewer for this valuable suggestion. In our experiments, ACWM-RM adds little overhead: it takes 0.54s to score a clip, which is only 7.42% of the total per-clip time. Its reward scoring is about 10× faster than that of the VLM-based method and 20× faster than that of the metric-based method, and only 0.44 s slower than that of the lightweight IDM-based method.

To measure the cost of reward scoring during fine-tuning, we record the per-clip time for rollout generation, reward scoring, and their sum for ACWM-RM and the IDM-, VLM-, and metric-based methods. We run all measurements on 100 DreamDojo-2B rollout clips (13 frames each), using the same dedicated H800 GPU with batch size 1.

We warm up the DreamDojo and each reward signal separately, then process the clips one at a time. Generation time is the wall-clock time DreamDojo-2B takes to produce one rollout clip. Reward-scoring time runs from when the generated clip is passed to the rewards until the score is returned. Reward share is reward-scoring time divided by total time.

Table 4. Per-clip generation and reward-scoring latency.

MethodGeneration (s)Reward-scoring (s) ↓Total (s) ↓Reward share ↓
IDM-based6.720.106.831.52%
VLM-based6.725.3612.0944.38%
Metric-based6.7210.8717.6061.80%
Ours6.720.547.267.42%

As shown in Table 4, generation time is identical across methods, so the differences come entirely from reward scoring. Reward scoring takes 44.38% and 61.80% of the per-clip time for the VLM-based and metric-based methods, making their total time 1.7× and 2.4× that of ACWM-RM. The IDM-based method is faster owing to its lightweight ResNet backbone, but it saves only about 6% of the total time and yields notably inferior performance.

Related revisions: We have added these experimental results to Appendix J (lines 1981–1991; Table 22). All revisions are highlighted in red in the manuscript.

Reviewer_KuXPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACWM-RM learns a reference-free reward for action-conditioned robot videos through dynamics pretraining, action-mismatch preferences, and generated-video quality preferences. The paper introduces AVM-Eval/AVM-Pref, evaluates several inverse-dynamics rewards, and improves visual metrics and trajectory alignment when used to fine-tune DreamDojo-2B with Flow-GRPO.

Critical strengths
  • The complementary supervision is well motivated. Ablations show that both preference types, dynamics pretraining, and predictor adaptation contribute substantially to generated-video ranking accuracy (Table 6).
  • The JEPA-IDM comparison controls for the backbone, training data, and two-stage training procedure. The 14.9-point advantage on generated-video pairs provides useful evidence for the proposed prediction-and-scoring design (Sec. 5.4).
Critical weaknesses
  • The real/generated comparison changes both the ranking task and label construction: action perturbations on real videos versus nDTW-ranked generated videos (Sec. 4.1). The accuracy drop therefore does not isolate distribution shift. Moreover, generated-video AF is validated only through 2D trajectories, leaving rotation and gripper fidelity unverified.
  • Downstream comparisons omit a combined IDM-plus-visual-metric reward and the preference-trained JEPA-IDM control. Tables 4-5 therefore do not establish whether the learned joint reward is preferable to a simpler combination of the two supervision sources.
  • RL evidence uses one model, one dataset, and one optimization seed. Reported deviations reflect five video-generation seeds rather than independent training runs (Appendix G), leaving optimization robustness and broader transfer uncertain.
Questions for authors
  1. Can you validate the real/generated performance gap using matched ranking protocols or independent action annotations, including rotation and gripper behavior?
  2. How do a combined metric-plus-IDM reward and JEPA-IDM perform under the same RL budget, and do the conclusions persist across optimization seeds?
4 comments
AuthorsOpenReview ↗

Response to reviewer KuXP (1/3)

Dear Reviewer KuXP,

We appreciate your time and constructive feedback, which has been invaluable in improving our work. We also express our sincere gratitude for your positive assessment of our paper and for recognizing the value of our contributions.

W1 & Q1. The real/generated accuracy gap does not isolate distribution shift because the ranking protocols and label construction differ.

We thank the reviewer for pointing this out. We agree that the real/generated accuracy gap does not isolate the effect of distribution shift, since the two protocols differ in both ranking task and label construction.

A controlled comparison would require applying the same protocol to both video domains, but neither protocol transfers reliably to the other domain (detailed below). We have therefore stated in the results (line 409) that the accuracy gap does not isolate the effect of distribution shift because the protocols differ in both ranking task and label construction. Importantly, our conclusion does not depend on the cause of the gap: high accuracy on real videos does not ensure reliable AF evaluation on generated videos, so an IDM should be evaluated directly on generated videos before being used as a reward for ACWM training (line 410).

Below, we explain why neither protocol transfers to the other domain:

  • Action perturbation on generated videos. As discussed in Sec. 4.1.2 (line 215), action perturbation assumes that the original action chunk is the best match for the video. This assumption holds for real videos, which are produced by executing that action chunk, but not for generated videos: an ACWM may fail to follow its conditioning action, so a perturbed action chunk may match the generated video better than the original one, making the preference labels unreliable.

  • Video ranking on real videos. Ranking different real videos by their consistency with a fixed action chunk, as in our video-mismatch protocol, would require resetting the robot and scene to identical initial conditions, executing the reference and perturbed action chunks, and recording the resulting videos. Such precise resets are difficult to achieve in practice, especially once objects have been displaced during manipulation.

Related revisions: Sec. 5.2 (Lines 409–410) now clarifies that protocol differences prevent attributing the accuracy gap solely to distribution shift. Revisions are highlighted in red.

W1 & Q1. Generated-video AF evaluation relies only on 2D trajectories, leaving end-effector rotation and gripper fidelity unverified.

We agree that our current generated-video evaluation, which relies on 2D trajectories (Sec. 5.1, L368–369), does not independently validate end-effector rotation or gripper operation. We considered two existing approaches for evaluating these aspects, but found that neither was directly applicable to our setting:

  • IDM-based method. Prior studies [1,2] use inverse dynamics models (IDMs) to infer actions from generated videos and compare them with reference actions. In these studies, IDM action-recovery accuracy is evaluated on reference videos. In our setting, without additional validation on generated videos, the resulting evaluation may be confounded by IDM errors, particularly for end-effector rotation and gripper operation.

  • IDM-free method. To our knowledge, KineBench [3] is the only IDM-free approach that estimates both end-effector pose and gripper state from generated videos. However, it cannot be applied directly to our setting for two reasons. (1) KineBench needs a 3D model of the gripper to estimate end-effector rotation, but the gripper model it uses differs from ours, and we found no published work applying its pose estimation pipeline to our gripper. (2) Its gripper-state classifier evaluates only binary open/close states, and we could not find publicly available weights for it. Applying KineBench in our setting would therefore require additional adaptation and validation of its pose-estimation and gripper-state components.

[1] UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras. arXiv 2026.

[2] Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning. arXiv 2026.

[3] KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding. ECCV 2026.

AuthorsOpenReview ↗

Response to reviewer KuXP (2/3)

W2 & Q2. The advantage over metric+IDM and JEPA-IDM rewards remains unverified under matched RL budgets and multiple training seeds.

We sincerely thank the reviewer for this helpful suggestion. We agree that combining metric- and IDM-based rewards is a natural baseline, and that JEPA-IDM should also be evaluated as an RL reward. So, we have conducted both experiments. The results show that adding the IDM reward to the metric reward (metric+IDM) brings no benefit: Metric+IDM has worse visual quality than metric-based and almost the same action-following fidelity. JEPA-IDM improves action following but not visual quality, despite also being trained on video-mismatch pairs that supervise visual quality, possibly because its reward captures visual-quality differences through action-prediction errors. ACWM-RM still achieves the best visual quality and action-following fidelity among all rewards.

Experimental Setup. All methods fine-tune DreamDojo-2B with the same RL configuration and are evaluated under the same protocol. Due to limited computational resources, we train a single seed per method and will include multi-seed results in the final version. For each fine-tuned policy, we generate videos with 5 sampling seeds and report mean ± std.

  • Metric+IDM. We combine EVAI\mathrm{EVA}_{\text{I}}, the best-performing IDM on AVM-Eval, with the four VQ metrics, assigning equal weight to the two signals. The IDM advantage AIDMA_{\mathrm{IDM}} is obtained by group-standardizing the negative normalized action-prediction error of EVAI\mathrm{EVA}_{\text{I}}, and the four VQ metrics are aggregated into a group-standardized advantage AmetricsA_{\mathrm{metrics}} following the same fusion procedure as the metric-based baseline. We then average the two advantages and re-standardize them to obtain the final GRPO advantage: Amix=Norm(0.5AIDM+0.5Ametrics)A_{\mathrm{mix}}=\mathrm{Norm}(0.5A_{\mathrm{IDM}}+0.5A_{\mathrm{metrics}}).

  • JEPA-IDM. As in the IDM-based reward, we use the negative normalized error between actions from generated videos and the corresponding reference actions as the reward.

Table 1. Visual quality of DreamDojo on Agi$_\text{C}$ with different rewards.

MethodSSIM↑MSE↓FVD↓FID↓L2latent\mathrm{L2}_{\mathrm{latent}}↓
Original88.67±0.020.49±0.018.31±1.095.01±0.0525.12±0.03
IDM-based88.04±0.110.58±0.031.31±0.075.39±0.0925.87±0.14
VLM-based88.41±0.280.49±0.018.20±1.085.47±0.1125.26±0.07
Metric-based89.62±0.190.48±0.017.56±0.634.30±0.0425.04±0.06
Metric+IDM88.14±0.020.53±0.024.77±1.115.35±0.0325.39±0.09
JEPA-IDM87.98±0.090.56±0.021.17±0.905.17±0.0325.62±0.11
Ours89.86±0.130.46±0.017.10±0.564.25±0.0724.90±0.06

Table 2. Action-following fidelity of DreamDojo on Agi$_\text{C}$ with different rewards.

MethodnDTW↑
Original14.84±0.24
IDM-based13.21±0.38
VLM-based14.00±0.41
Metric-based14.05±0.33
Metric+IDM14.02±0.49
JEPA-IDM15.27±0.32
Ours16.41±0.22

Table 3. Physical consistency of DreamDojo on Agi$_\text{C}$ with different rewards.

MethodOC↑MC↑AC↑Overall↑
Original4.25±0.104.31±0.293.43±0.064.00±0.10
IDM-based4.19±0.174.16±0.373.42±0.093.92±0.14
VLM-based4.12±0.174.07±0.323.38±0.063.86±0.12
Metric-based4.11±0.224.07±0.363.34±0.123.84±0.15
Metric+IDM4.16±0.133.98±0.163.40±0.033.85±0.12
JEPA-IDM4.14±0.113.94±0.173.41±0.093.83±0.11
Ours4.20±0.124.17±0.223.42±0.053.93±0.08

As shown in Tables 1 to 2, ACWM-RM outperforms both Metric+IDM and JEPA-IDM in visual quality and action-following fidelity:

  • Metric+IDM. Combining the two signals brings no benefit. Compared with Metric-based, it degrades visual quality (FVD 24.77 vs. 17.56) and leaves action following almost unchanged. The IDM-based reward alone already degrades both VQ and AF, so we attribute this to the unreliability of the IDM signal, which dilutes the metric reward rather than adding useful action guidance.  
  • JEPA-IDM. JEPA-IDM improves AF over Origin but remains below ACWM-RM, and it degrades all five VQ metrics. This VQ degradation may stem from its reward formulation: although JEPA-IDM receives VQ supervision from video-mismatch pairs just as ACWM-RM does, its reward remains defined by action-prediction error. Visual-quality differences must therefore be expressed through differences in action-prediction error, potentially limiting the effective use of VQ supervision.

 Related revisions: We have added the Metric+IDM and JEPA-IDM results to Table 4 (lines 346–347) and Table 5 (lines 387–388), discussed their performance in Sections 5.3 (lines 425–431) and 5.4 (lines 514–517), respectively, and detailed the Metric+IDM fusion procedure in Appendix H (lines 1545–1552). We also acknowledge the single-training-seed limitation in Section 6 (lines 537–539). These revisions are highlighted in red in the manuscript.

AuthorsOpenReview ↗

Response to reviewer KuXP (3/3)

W3. RL results rely on one model, one dataset, and one training seed, leaving optimization robustness and broader transfer uncertain.

We thank the reviewer for these concerns. For unseen ACWMs, we have added RL experiments on EnerVerse-AC, an ACWM unseen during ACWM-RM training. Among the evaluated rewards, ACWM-RM is the only one that improves nDTW (9.30 → 10.24), and it outperforms all baselines on VQ. For dataset generalization, cross-dataset generalization remains an open challenge in our fidelity. Our results on AVM-Eval (Table 3 of the manuscript) confirm this: ACWM-RM and the evaluated IDMs show limited AF performance beyond AgiBotWorld. We have therefore restricted our RL claims to AgiBotWorld in the abstract, introduction, and limitations, and leave cross-dataset generalization to future work. For training stability, multi-seed RL training could not be completed within the rebuttal period because each run requires 6 days, and we will include multi-seed results in the final version.

Unseen ACWMs. We train EnerVerse-AC [1] using ACWM-RM and three baselines as the reward signal. EnerVerse-AC is unseen during ACWM-RM training, and differs from DreamDojo in both backbone and RL algorithm: as a DDIM-based diffusion model, we follow DDPO and set η=1\eta=1 to make DDIM sampling stochastic (DDPO-style GRPO [2]), treating each denoising step as a Gaussian policy optimized with GRPO.

For training, all four rewards share the same configuration: 100 outer iterations on 2 H800 GPUs per reward, with 64 rollouts and 4 inner policy-optimization epochs per iteration (full details in Appendix G). For evaluation, we test the trained policies on Agi$_\text{C}$ in terms of visual quality (VQ) and action-following fidelity (AF). Each policy is sampled 5 times, and we report the mean ± std over these sampling runs. Results are reported in Tables 4 and 5.

Table 4. Visual quality of EnerVerse-AC on Agi$_\text{C}$ with different rewards.

MethodSSIM ↑MSE ↓FVD ↓FID ↓L2latent\mathrm{L2}_{\mathrm{latent}} ↓
Original80.99 ± 0.150.79 ± 0.0118.30 ± 2.6415.06 ± 0.0138.68 ± 0.11
IDM-based80.04 ± 0.170.82 ± 0.0139.49 ± 3.3215.81 ± 0.0939.36 ± 0.12
VLM-based81.14 ± 0.110.78 ± 0.0120.30 ± 4.1016.80 ± 0.1738.91 ± 0.05
Metric-based81.64 ± 0.160.75 ± 0.0115.60 ± 1.5814.58 ± 0.0937.92 ± 0.10
Ours81.82 ± 0.100.74 ± 0.0114.30 ± 3.1914.06 ± 0.1337.84 ± 0.03

Table 5. Action-following fidelity of EnerVerse-AC on Agi$_\text{C}$ with different rewards.

MethodnDTW ↑
Original9.30 ± 0.04
IDM-based7.30 ± 0.15
VLM-based8.86 ± 0.13
Metric-based8.88 ± 0.06
Ours10.24 ± 0.13

ACWM-RM is the only reward that improves nDTW over the original model, while the IDM-based reward, though designed for action following, causes the largest drop. ACWM-RM also achieves the best results on all five VQ metrics.

Unseen datasets. Cross-dataset generalization remains an open challenge in our fidelity. For example, EVA [3] and ACID [4] are both trained and evaluated within the same dataset. Our results on AVM-Eval (Table 3 of the manuscript) are consistent with this: ACWM-RM and the evaluated IDMs show limited AF performance beyond AgiBotWorld. We have therefore restricted our claims to AgiBotWorld in the abstract, introduction, and limitations, and leave cross-dataset generalization to future work.

Optimization seeds. We agree that multi-seed training is the most direct test of stability. A single RL run takes about 6 days, so repeated training runs could not be completed within the rebuttal period. Nevertheless, ACWM-RM improves both VQ and AF on DreamDojo with Flow-GRPO and EnerVerse-AC with DDIM-based GRPO. These results provide complementary evidence across two backbones and RL formulations, but do not replace a multi-seed stability analysis. We have noted this in the limitations and will include multi-seed results in the final version.

[1] EnerVerse-AC: Envisioning Embodied Environments with Action Condition. arXiv 2025.

[2] Training diffusion models with reinforcement learning. ICLR 2024.

[3] Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards. arXiv 2026.

[4] ACID: Action Consistency via Inverse Dynamics for Planning with World Models. arXiv 2026.

Related revisions: We have added RL results on the unseen EnerVerse-AC to Section 5.3 (lines 422–424; Table 6), with method and training details in Section 3 (lines 144–152) and Appendix G (lines 1526–1537). We also limit our RL claims to AgiBotWorld and state in the limitations (line 538) that we used a single training seed per RL setting. We highlighted these results, scope clarifications, and limitations in red.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer KuXP,

Thank you again for your valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our rebuttal and additional experiments have addressed your concerns or if any further clarification is needed.

Thank you for your time and consideration.

Reviewer_h2q2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: A Useful Reward, but Evidence Is Limited to the Same Generators and Automatic Metrics.

This paper studies reinforcement-learning rewards for action-conditioned world models (ACWMs). Full-reference visual-quality (VQ) metrics require ground-truth future videos and do not directly assess action-following fidelity (AF). Inverse dynamics model (IDM) rewards do not require reference videos, but IDMs trained on real videos may be unreliable on generated videos containing artifacts and physical inconsistencies. The paper introduces AVM-Eval with 67.2k pairs, AVM-Pref with 506.3k pairs, and ACWM-RM for jointly scoring VQ and AF.

AVM-Pref action-mismatch pairs hold a real video fixed and compare its recorded action with numerical or semantic perturbations. Video-mismatch pairs hold the action fixed and use global/local PSNR and LPIPS to select the better generated video. ACWM-RM predicts the next V-JEPA feature from video history, states, and actions, then trains the predictor and a patch-weighted score head on preference pairs. Reward computation requires no ground-truth future video.

Critical strengths
  1. The paper addresses a real gap between full-reference VQ rewards and reference-free IDM rewards. Dynamics pretraining, generated-video preferences, and patch-weighted comparison form a practical reward that does not require ground-truth future videos during RL.
  2. Fresh/Frozen Predictor, JEPA-IDM, preference-removal, and score-head ablations test the main design choices. With Flow-GRPO, ACWM-RM improves nDTW from 14.84 to 16.41 and gives the best values on all five VQ metrics. SSIM and FVD improve although they are not used to build AVM-Pref. Relative to RLVR-World, Flow-GRPO, and VideoReward, combining generated-video VQ with numerical robot actions is a clear contribution [1–3].

[1] Jialong Wu et al., “RLVR-World: Training World Models with Reinforcement Learning,” NeurIPS 2025, DOI.
[2] Jie Liu et al., “Flow-GRPO: Training Flow Matching Models via Online RL,” NeurIPS 2025, DOI.
[3] Jie Liu et al., “Improving Video Generation with Human Feedback,” NeurIPS 2025, DOI.

Critical weaknesses
  1. The core generated-video evaluation does not include an unseen generator. The same four ACWMs produce AVM-Pref and AVM-Eval videos, and downstream RL is demonstrated only on DreamDojo-2B, one of those four models. The paper itself lists evaluation on more ACWMs as future work. The claim should therefore be limited to the tested generators and AgiBotWorld.
  2. The two training targets may disagree. Appendix C keeps a VQ pair only when one candidate has higher global/local PSNR and lower global/local LPIPS, but it does not control AF with nDTW. Videos generated from the same action need not follow it equally well. The paper does not report how often VQ preference disagrees with nDTW or human AF judgment. Physical consistency is judged only by GPT-5.6 sol, and ACWM-RM scores 3.93 versus 4.00 for the original model.
Questions for authors
  1. What is the video-mismatch accuracy on an ACWM not used to construct AVM-Pref?
  2. How often do the PSNR/LPIPS preferences in AVM-Pref disagree with nDTW or human judgments of VQ and AF?
  3. Are the nDTW and VQ gains stable across multiple Flow-GRPO training seeds and another ACWM backbone?
5 comments
AuthorsOpenReview ↗

Response to reviewer h2q2 (1/4)

Dear Reviewer h2q2,

We appreciate your time and constructive feedback, which has been invaluable in improving our work. We have taken your concerns very seriously and have made every effort to address your questions.

W1 & Q1: Video-mismatch accuracy of ACWM-RM on videos from unseen ACWMs not used to construct AVM-Pref.

We thank the reviewer for this question. Our new experiments show that ACWM-RM remains effective on unseen ACWMs: on two held-out ACWMs, it achieves 81.80% (intra-model pairs) and 76.81% (cross-model pairs) accuracy, outperforming the strongest baseline by 22.28 and 13.39 percentage points, respectively.

We evaluate on two additional ACWMs, EnerVerse-AC [1] and DreamTrue [2], neither of which is used to construct AVM-Pref. Evaluation pairs are built from Agi$_\text{B}$, whose tasks are also unseen during AVM-Pref construction, so this setting tests generalization to both unseen ACWMs and unseen tasks. 

Following the same protocol as for the other ACWMs, we generate six videos per conditioning input for each model by varying the random seed. We then construct intra-model pairs and cross-model pairs, where cross-model pairs are formed only between EnerVerse-AC and DreamTrue. Labels are annotated following the AVM-Eval protocol, yielding 1.7k intra-model pairs and 1.8k cross-model pairs.

Table 1. Video-mismatch accuracy (%) on unseen ACWMs .

MethodIntra-model pairsCross-model pairs
EVA$_{\text{I}}$59.5263.42
GRT$_{\text{I}}$50.2353.12
TAP$_{\text{I}}$52.4757.37
J$_{\text{I}}$54.4654.12
Ours81.8076.81

As shown in Table 1, ACWM-RM outperforms all baselines on both intra-model and cross-model pairs. These results indicate that ACWM-RM's evaluation ability transfers to ACWMs not used to construct AVM-Pref.

[1] EnerVerse-AC: Envisioning Embodied Environments with Action Condition. arXiv, 2025.

[2] DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training. arXiv, 2026.

Related revisions: We have added these results to Appendix J, under “Video-mismatch Evaluation on Unseen ACWMs” (lines 2039–2051; Table 26). These additions are highlighted in red in the manuscript.

AuthorsOpenReview ↗

Response to reviewer h2q2 (2/4)

W1 & Q3: Generalization of ACWM-RM as an RL reward to unseen ACWMs.

We thank the reviewer for these concerns. **For unseen ACWMs **, we have added RL experiments on EnerVerse-AC, an ACWM unseen during ACWM-RM training. Among the evaluated rewards, ACWM-RM is the only one that improves nDTW (9.30 → 10.24), and it outperforms all baselines on VQ. For dataset generalization, cross-dataset generalization remains an open challenge in our fidelity. Our results on AVM-Eval (Table 3 of the manuscript) confirm this: ACWM-RM and the evaluated IDMs show limited AF performance beyond AgiBotWorld. We have therefore restricted our RL claims to AgiBotWorld in the abstract, introduction, and limitations, and leave cross-dataset generalization to future work. For training stability, multi-seed RL training could not be completed within the rebuttal period because each run requires 6 days, and we will include multi-seed results in the final version.

Unseen ACWM. We train EnerVerse-AC [1] using ACWM-RM and three baselines as the reward signal. EnerVerse-AC is unseen during ACWM-RM training, and differs from DreamDojo in both backbone and RL algorithm: as a DDIM-based diffusion model, we follow DDPO and set η=1\eta=1 to make DDIM sampling stochastic (DDPO-style GRPO [2]), treating each denoising step as a Gaussian policy optimized with GRPO.

For training, all four rewards share the same configuration: 100 outer iterations on 2 H800 GPUs per reward, with 64 rollouts and 4 inner policy-optimization epochs per iteration (full details in Appendix G). For evaluation, we test the trained policies on Agi$_\text{C}$ in terms of visual quality (VQ) and action-following fidelity (AF). Each policy is sampled 5 times, and we report the mean ± std over these sampling runs. Results are reported in Tables 2 and 3.

Table 2. Visual quality of EnerVerse-AC on Agi$_\text{C}$ with different rewards.

MethodSSIM ↑MSE ↓FVD ↓FID ↓L2latent\mathrm{L2}_{\mathrm{latent}} ↓
Original80.99 ± 0.150.79 ± 0.0118.30 ± 2.6415.06 ± 0.0138.68 ± 0.11
IDM-based80.04 ± 0.170.82 ± 0.0139.49 ± 3.3215.81 ± 0.0939.36 ± 0.12
VLM-based81.14 ± 0.110.78 ± 0.0120.30 ± 4.1016.80 ± 0.1738.91 ± 0.05
Metric-based81.64 ± 0.160.75 ± 0.0115.60 ± 1.5814.58 ± 0.0937.92 ± 0.10
Ours81.82 ± 0.100.74 ± 0.0114.30 ± 3.1914.06 ± 0.1337.84 ± 0.03

Table 3. Action-following fidelity of EnerVerse-AC on Agi$_\text{C}$ with different rewards.

MethodnDTW ↑
Original9.30 ± 0.04
IDM-based7.30 ± 0.15
VLM-based8.86 ± 0.13
Metric-based8.88 ± 0.06
Ours10.24 ± 0.13

ACWM-RM is the only reward that improves nDTW over the original model, while the IDM-based reward, though designed for action following, causes the largest drop. ACWM-RM also achieves the best results on all five VQ metrics.

Unseen datasets. Cross-dataset generalization remains an open challenge in our fidelity. For example, EVA [3] and ACID [4] are both trained and evaluated within the same dataset. Our results on AVM-Eval (Table 3 of the manuscript) are consistent with this: ACWM-RM and the evaluated IDMs show limited AF performance beyond AgiBotWorld. We have therefore restricted our claims to AgiBotWorld in the abstract, introduction, and limitations, and leave cross-dataset generalization to future work.

Optimization seeds. We agree that multi-seed training is the most direct test of stability. A single RL run takes about 6 days, so repeated training runs could not be completed within the rebuttal period. Nevertheless, ACWM-RM improves both VQ and AF on DreamDojo with Flow-GRPO and EnerVerse-AC with DDIM-based GRPO. These results provide complementary evidence across two backbones and RL formulations, but do not replace a multi-seed stability analysis. We have noted this in the limitations and will include multi-seed results in the final version.

[1] EnerVerse-AC: Envisioning Embodied Environments with Action Condition. arXiv 2025.

[2] Training diffusion models with reinforcement learning. ICLR 2024.

[3] Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards. arXiv 2026.

[4] ACID: Action Consistency via Inverse Dynamics for Planning with World Models. arXiv 2026.

Related revisions: We have added RL results on the unseen EnerVerse-AC to Section 5.3 (lines 422–424; Table 6), with method and training details in Section 3 (lines 144–152) and Appendix G (lines 1526–1537). We also limit our RL claims to AgiBotWorld and state in the limitations (line 538) that we used a single training seed per RL setting. We highlighted these results, scope clarifications, and limitations in red.

AuthorsOpenReview ↗

Response to reviewer h2q2 (3/4)

W2: GPT-5.6 sol alone judges physical consistency (PC) and yields lower PC scores than Origin.

We thank the reviewer for raising this concern. To check whether our PC results depend on GPT-5.6 sol, we additionally score the same samples with Opus 5. Across methods, the mean Spearman correlations between the two evaluators over the five seeds range from 0.811 to 0.837. DreamDojo optimized with ACWM-RM receives lower mean PC scores than Origin under both evaluators (Opus 5: 3.75 ± 0.12 vs. 3.77 ± 0.13; GPT-5.6 sol: 3.93 ± 0.08 vs. 4.00 ± 0.10).

Specifically, we use Claude Opus 5 to re-score the same videos evaluated by GPT-5.6 sol (5 sampling seeds, 1,499 videos per seed), using the same prompt and criteria definitions. For each method and seed, we concatenate the OC, MC, and AC scores of all videos into a single vector per evaluator, compute the Spearman correlation between the two vectors, and report the mean ± std over the 5 seeds.

We agree that our mean PC score is lower than Origin under both evaluators. As noted in the manuscript (line 464), ACWM-RM is trained without explicit PC supervision and targets VQ and AF. We report PC to assess potential degradation and acknowledge these lower PC scores as a limitation of the current method. Adding explicit PC signals to the reward is a promising direction for future work.

Table 4. PC scores evaluated by Opus 5. ρ: Spearman correlation between Opus 5 and GPT-5.6 sol.

MethodOC ↑MC ↑AC ↑Overall ↑ρ (Opus vs. GPT)
Original3.88 ± 0.204.06 ± 0.233.36 ± 0.063.77 ± 0.130.837 ± 0.017
IDM-based3.88 ± 0.154.03 ± 0.313.35 ± 0.053.75 ± 0.150.836 ± 0.024
VLM-based3.80 ± 0.173.98 ± 0.243.32 ± 0.073.70 ± 0.100.811 ± 0.026
Metric-based3.82 ± 0.214.01 ± 0.253.34 ± 0.093.72 ± 0.110.837 ± 0.030
Ours3.85 ± 0.184.04 ± 0.253.35 ± 0.063.75 ± 0.120.828 ± 0.024

As shown in the last column of Table 4, the scores are rank-correlated between Opus 5 and GPT-5.6 sol for all methods. The Opus 5 evaluation also reproduces the direction of the mean PC difference between our method and Origin observed with GPT-5.6 sol.

Related revisions: We have added the Opus 5 results and evaluator correlations to Appendix J (lines 1970–1980; Table 21) and noted them in the main text (lines 469–470), all highlighted in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.