- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reinforcement-learning post-training for embodied world models under unreliable reward signals. It proposes Reward as an Agent, an evidence-grounded reward workflow that separates task-blind observation from task requirements and uses adaptive inspection, motion evidence, model-based reflection, scope checking, and reward gates; and DynDiff-GRPO, a dynamics-aware rollout strategy that allocates more stochasticity to dynamically salient regions while accounting for the resulting policy-dependent Gaussian scale in the GRPO likelihood. Experiments on three embodied world models report consistent gains on the robotics subset of PAI-Bench, while offline reward analyses on ordinary and hacking/control videos report improved reward reasonableness and fewer erroneous positive rewards on failure and unobservable clips.
Critical strengths
Important problem and compelling motivation. The paper targets a practically important bottleneck in RL post-training of embodied world models: unreliable rewards can encourage reward hacking as rollout diversity increases. This is consequential because stronger optimization alone cannot solve failures caused by an unreliable objective.
Sensible design choices with useful technical care. Separating task-blind observation from task requirements is a reasonable way to reduce confirmation bias, and Table 2 provides direct ablation support for this component. I also view the policy-dependent Gaussian-scale treatment in DynDiff-GRPO as technically careful: once the dynamic mask changes with the policy, the current and old transition distributions need not share the same scale, so retaining the corresponding terms in the Gaussian likelihood is appropriate.
Critical weaknesses
Empirical attribution is insufficient relative to the complexity of the system. The method contains many plausible but hand-designed components and hyperparameters, while the current ablations do not fully establish which choices drive the gains or how robust the overall recipe is. For DynDiff-GRPO, the CPS comparison and shifted-mask control are useful, and the latter provides evidence that correct spatial alignment matters. However, the current experiments do not establish whether the specific D/B-based mask construction is preferable to spatially uniform stochasticity under a matched effective noise/variance budget. Representative choices such as the D/B construction and its mixing rule would also benefit from further ablation. The same concern applies more strongly to the reward construction: the categorical T/P/V mappings, 0.70/0.20/0.10 weights, multiplicative physics factor, and target-mismatch cap directly determine the optimized reward, yet I did not find a theoretical derivation, established standard, or dedicated experimental validation of these specific choices. Finally, the paper lacks a clean online 2x2 decomposition of baseline reward vs. Reward Agent crossed with baseline/CPS rollout vs. DynDiff-GRPO, making it difficult to attribute the final gains or assess whether verification and exploration interact beyond independent improvements. This is particularly relevant to the paper’s statement that they should “scale in tandem.”
The central reward-quality evaluation lacks sufficiently independent held-out validation. The authors explicitly acknowledge that the development-exposed samples do not establish held-out generalization. This is an important limitation rather than merely a caveat: because the reward workflow was iteratively refined on these samples, the current evidence cannot distinguish a broadly transferable design from one partially overfit to the development set. A held-out evaluation is therefore important to establish whether the proposed reward design generalizes beyond the data used to construct it; without such evidence, it is difficult to assess to what extent the method provides generalizable insights. In addition, the strongest Reward Agent uses Doubao 2.1 Pro as its backbone and is also judged by Doubao 2.1 Pro, creating a same-family/self-preference confound; prior work has documented such biases in LLM-as-a-judge evaluation (Zheng et al., 2023; Pombal et al., 2026). Since the paper already evaluates a Qwen-based Reward Agent with the Doubao judge, a reciprocal cross-family evaluation of the Doubao Agent would be a relatively straightforward but important control. Finally, the main reward-quality claim has no human-labeled anchor; even a randomly sampled subset labeled for task progress/failure and reward reasonableness would substantially strengthen the evaluation.
References
- Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023, Datasets and Benchmarks Track.
- José Pombal, Ricardo Rei, André F. T. Martins. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models. COLM 2026.
Questions for authors
-
Can the authors report results on a fully held-out set that was never used during the 16-version development process to revise prompts, tools, gates, or scoring rules?
-
Can the authors evaluate the Doubao-based Reward Agent with an independent cross-family judge, ideally using a reciprocal judge matrix? If possible, can the authors also provide human annotations for a randomly sampled subset?
-
Can the authors provide a matched online 2x2 ablation: baseline reward vs. Reward Agent crossed with baseline/CPS rollout vs. DynDiff-GRPO?
-
What is the basis for the categorical
T/P/Vmappings, the0.70/0.20/0.10weights, the multiplicative physics factor, and the target-mismatch cap? Were they inherited, calibrated, or chosen heuristically? For DynDiff-GRPO, can the authors also compare with spatially uniform stochasticity under a matched effective noise/variance budget? -
Can the authors clarify whether they plan to release the code, prompts, and implementation details for Reward as an Agent and DynDiff-GRPO to support reproducibility?