acceptodds
Under review as a conference paper at ICLR 2027

Reward as an Agent for Embodied World Models

Abstract

RL for world models largely relies on conservative rollouts near the training distribution, limiting exploration and behavioral diversity. In this work, we challenge this conservative paradigm. We argue that the core limitation is not exploration itself, but the lack of reliable verification strategies to support broader exploration. Without reliable verification, policies can exploit proxy rewards through convincing videos that lack evidence of genuine task progress. On the verification side, we introduce Reward as an Agent, which centers evaluation on evidence-grounded task verification. It separates observed behavior from fixed task requirements, actively inspects disputed evidence, and revises judgments through tool-assisted reflection and requirement checks. Reward gates preserve supported progress while withholding credit for failure or unverifiable essential content. The framework is extensible: additional reward strategies can be integrated as tools to enrich verification and support further performance gains. On the exploration side, we introduce Dynamic-Aware Rollout Diversification through DynDiff-GRPO, which allocates stochastic exploration to dynamically salient regions to diversify trajectories. Together, these designs couple broader exploration with evidence-based reward evaluation. Training experiments demonstrate benchmark improvements across three open-source world models, while complementary offline evaluations show that Reward as an Agent improves reward reasonableness and reduces erroneous positive rewards on failure and unobservable clips.

Primary area:
reinforcement learning
Keywords:
RL, Robotics, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kP68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reinforcement-learning post-training for embodied world models under unreliable reward signals. It proposes Reward as an Agent, an evidence-grounded reward workflow that separates task-blind observation from task requirements and uses adaptive inspection, motion evidence, model-based reflection, scope checking, and reward gates; and DynDiff-GRPO, a dynamics-aware rollout strategy that allocates more stochasticity to dynamically salient regions while accounting for the resulting policy-dependent Gaussian scale in the GRPO likelihood. Experiments on three embodied world models report consistent gains on the robotics subset of PAI-Bench, while offline reward analyses on ordinary and hacking/control videos report improved reward reasonableness and fewer erroneous positive rewards on failure and unobservable clips.

Critical strengths

Important problem and compelling motivation. The paper targets a practically important bottleneck in RL post-training of embodied world models: unreliable rewards can encourage reward hacking as rollout diversity increases. This is consequential because stronger optimization alone cannot solve failures caused by an unreliable objective.

Sensible design choices with useful technical care. Separating task-blind observation from task requirements is a reasonable way to reduce confirmation bias, and Table 2 provides direct ablation support for this component. I also view the policy-dependent Gaussian-scale treatment in DynDiff-GRPO as technically careful: once the dynamic mask changes with the policy, the current and old transition distributions need not share the same scale, so retaining the corresponding terms in the Gaussian likelihood is appropriate.

Critical weaknesses

Empirical attribution is insufficient relative to the complexity of the system. The method contains many plausible but hand-designed components and hyperparameters, while the current ablations do not fully establish which choices drive the gains or how robust the overall recipe is. For DynDiff-GRPO, the CPS comparison and shifted-mask control are useful, and the latter provides evidence that correct spatial alignment matters. However, the current experiments do not establish whether the specific D/B-based mask construction is preferable to spatially uniform stochasticity under a matched effective noise/variance budget. Representative choices such as the D/B construction and its mixing rule would also benefit from further ablation. The same concern applies more strongly to the reward construction: the categorical T/P/V mappings, 0.70/0.20/0.10 weights, multiplicative physics factor, and target-mismatch cap directly determine the optimized reward, yet I did not find a theoretical derivation, established standard, or dedicated experimental validation of these specific choices. Finally, the paper lacks a clean online 2x2 decomposition of baseline reward vs. Reward Agent crossed with baseline/CPS rollout vs. DynDiff-GRPO, making it difficult to attribute the final gains or assess whether verification and exploration interact beyond independent improvements. This is particularly relevant to the paper’s statement that they should “scale in tandem.”

The central reward-quality evaluation lacks sufficiently independent held-out validation. The authors explicitly acknowledge that the development-exposed samples do not establish held-out generalization. This is an important limitation rather than merely a caveat: because the reward workflow was iteratively refined on these samples, the current evidence cannot distinguish a broadly transferable design from one partially overfit to the development set. A held-out evaluation is therefore important to establish whether the proposed reward design generalizes beyond the data used to construct it; without such evidence, it is difficult to assess to what extent the method provides generalizable insights. In addition, the strongest Reward Agent uses Doubao 2.1 Pro as its backbone and is also judged by Doubao 2.1 Pro, creating a same-family/self-preference confound; prior work has documented such biases in LLM-as-a-judge evaluation (Zheng et al., 2023; Pombal et al., 2026). Since the paper already evaluates a Qwen-based Reward Agent with the Doubao judge, a reciprocal cross-family evaluation of the Doubao Agent would be a relatively straightforward but important control. Finally, the main reward-quality claim has no human-labeled anchor; even a randomly sampled subset labeled for task progress/failure and reward reasonableness would substantially strengthen the evaluation.

References

  • Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023, Datasets and Benchmarks Track.
  • José Pombal, Ricardo Rei, André F. T. Martins. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models. COLM 2026.
Questions for authors
  1. Can the authors report results on a fully held-out set that was never used during the 16-version development process to revise prompts, tools, gates, or scoring rules?

  2. Can the authors evaluate the Doubao-based Reward Agent with an independent cross-family judge, ideally using a reciprocal judge matrix? If possible, can the authors also provide human annotations for a randomly sampled subset?

  3. Can the authors provide a matched online 2x2 ablation: baseline reward vs. Reward Agent crossed with baseline/CPS rollout vs. DynDiff-GRPO?

  4. What is the basis for the categorical T/P/V mappings, the 0.70/0.20/0.10 weights, the multiplicative physics factor, and the target-mismatch cap? Were they inherited, calibrated, or chosen heuristically? For DynDiff-GRPO, can the authors also compare with spatially uniform stochasticity under a matched effective noise/variance budget?

  5. Can the authors clarify whether they plan to release the code, prompts, and implementation details for Reward as an Agent and DynDiff-GRPO to support reproducibility?

Reviewer_DRV4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors aim to address the problem of conservative exploration for embodied video world models with a verification and an exploration component. For the verification, they propose to extend the WMReward and CoTracker3 approaches in a "Reward as an Agent" approach. For the exploration, they introduce Dynamic-Aware Rollout Diversification that is aimed at allocate stochastic SDE exploration to dynamically salient regions to support higher diversity in the rollouts.

Critical strengths

Having the evaluator to first describe the video without seeing the prompt is a clever and reasonable way to avoid instruction-driven hallucination. Similarly, treating CoTracker3 and WMReward as diagnostic evidence tools is a great addition to stop high visual quality from compensating for incomplete tasks.

The spatiotemporal exploration in flow-matching GRPO is mathematically sound and empirically supported.

Critical weaknesses

The 180 clips used to benchmark the Reward Agent in Table 2 are the same development samples used over 16 iterations to design and tune the pipeline, meaning that the generalisation capabilities are not properly tested. Along the same lines, the reasonableness metric uses Doubao 2.1 Pro to judge its own outputs but there is no human ground-truth labels. Evaluating the frozen agent on a truly held-out test set or on external benchmarks would make the offline validation much more convincing.

Questions for authors

The Introduction motivates the problem via policy optimization, interaction learning, and world-action models (WAMs) that "couple visual prediction with action modeling [...] making reliable evaluation of predicted interactions and task outcomes essential for action learning". However, the experiments only evaluate open-loop image-to-video generation without predicting robot actions or evaluating closed-loop policy execution. This is acknowledged in Section 5 as "video benchmarks alone do not establish closed-loop robot-control gains". Can you explain why has this evaluation not carried out?

Reviewer_tFEzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper contends that RL post-training of video world models stays conservative for lack of reliable verification, not because exploration is inherently harmful: without it, policies learn to produce convincing clips that satisfy proxy rewards without real task progress. Reward as an Agent recasts scoring as evidence-based verification, separating observed behaviour from fixed task requirements, inspecting disputed evidence, revising judgements through tool-assisted reflection, and gating credit so that failures and unverifiable essentials score nothing; extra reward strategies can be added as tools. DynDiff-GRPO directs exploration noise to dynamically salient regions to diversify rollouts. Together they yield benchmark gains on three open-source world models, and offline evaluations show more reasonable rewards with fewer false positives on failure and unobservable clips.

Critical strengths

Reward hacking in RL fine-tuning of video models is a real problem that has drawn less attention than it deserves, and the failure-mode taxonomy in Figure 1 is a useful contribution in its own right. The engineering is careful: evidence hierarchies, retry policies, pinned tool versions, and the decision to fix the contract before examining any evidence are all sound choices. The shifted-mask control is the right ablation for DynDiff-GRPO, and the paper is precise about what it does and does not establish. The limitations paragraph is forthright about the development-exposed data and the same-family judge, which I appreciated.

Critical weaknesses

W1: The evaluation of the reward is largely circular. “Reward reasonableness” is judged by Doubao 2.1 Pro, the same model that drives the reward agent; the 180 diagnostic clips went through 16 rounds of development before the reported numbers were taken, and the paper itself says they do not establish held-out generalisation; PAI-Bench is scored by yet another VLM; and there is no human rating anywhere. The authors are to be credited for stating all of this, but as it stands the central claim, that the agentic reward is reasonable and hard to hack, rests on a model grading itself on data it was tuned on.

W2: Table 1 reports single runs, with no seeds and no intervals. Are the results substantial?

W3: The cost of the reward is not reported: a commercial API call, two tool invocations, and up to three repair rounds per rollout, on every rollout of every GRPO iteration. Serving cost is relegated to the list of open items. For a method whose appeal is partly practical, wall-clock and API-call counts per iteration seem essential.

The evaluation in general seems weak - I realize that this is a difficult task, but still the onus is on the authors to make it work.

Questions for authors
  1. Is a modest human study within reach, something like 60 clips rated by three people, restricted to clips that never entered the development loop?

  2. What are the costs of the reward agent compared with a standalone reward model?

  3. Can you repeat the experiments with 10 seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.