acceptodds
Under review as a conference paper at ICLR 2027

HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

Abstract

Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in complex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hallucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Policy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of generated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable action chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.

Primary area:
reinforcement learning
Keywords:
Reinforcement Training, World Model, Generalist Robot Policy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_11KcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hallucination-aware World Model-based Policy Optimization (HaWMPO), which first trains a Hallucination Aware Model (HAM) that predicts the intensity of hallucination for a generated chunk of video frames, and uses this model to soft-deduct reward during RL training inside of the world model. The method is shown to perform better than world model-based RL approaches without addressing world model hallucination in simulation and the real world.

Critical strengths
  • The problem is well motivated and is a main bottleneck for policy optimization in a world model.

  • The presentation on the method is clear, and the ablations provides valuable empirical study of how to deal with world model hallucinating.

Critical weaknesses
  • The design and evaluation of HaWMPO remains purely empirical, and some choices are not well justified. For example, why do you choose the four metrics and their corresponding weights when composing the hallucination score? In A.4, the conclusion is that the Image Quality metric (MUSIQ) has negative correlation with the other metrics, so why adding it to the mixture is not obvious.

  • The experiments lack close baselines and real world task coverage. All experiments are compared against methods without estimating hallucination, while existing methods like MOPO (Yu et al., 2020) reduces rewards by estimated model uncertainty, and McConachie et al. (2020) learn where a dynamics model can be trusted, and some concurrent work estimates video world-model hallucination directly (Hansen & Wang, 2026) (comparing to the concurrent work is not expected, but I put it here for discussion). Without comparing to those methods, the novelty and contribution of this paper can be undermined. The real world experiments also lack rigor. There are only 20 rollouts per task without error bars, and only one baseline approach is compared. It is also not clear whether the tested tasks appear in world model/RL training or not. More trials and tasks are needed for statistical significance.

Questions for authors
  • What data is the world model trained on? What tasks are used in RL training for the LIBERO and real world policy?

  • If I understand correctly, the training data for HAM is the same as the world model. In order to make RL benefit from it, it needs to generalize better than the world model itself; i.e. when the world model hallucinates on an OOD observation, it should be able to detect it. Have you tried to test this on a held-out set to validate HAM's performance?

Reviewer_PGUBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses hallucinations in world models by combining visual hallucination detection with world-model-based policy optimization to mitigate the problem. The authors validate their approach through experiments in both simulation and on a real robot.

Critical strengths
  1. The paper targets an important and timely problem — hallucinations in world models used for policy optimization (WMPO). Introducing hallucination detection to mitigate this issue is a meaningful contribution.

  2. The real-robot experiments, though small in scale, provide preliminary evidence beyond simulation.

Critical weaknesses

1.The paper does not describe the lifecycle of HAM in the main text. Equation (2) only presents the loss function, but it is never explained when the training data is constructed, which dataset is used, or how HAM's training is ordered relative to the world model's training. While the paper clearly states that the world model is trained and frozen before policy optimization, the equally critical information about HAM is scattered across the conclusion and the appendix. As a result, Figure 1's pipeline diagram is incomplete: it shows HAM's position at inference time but never indicates where the training phase fits in the overall pipeline, making it impossible for readers to obtain a full picture of HAM from the main text.

2.The third contribution claims to integrate the hallucination score into the GRPO optimization pipeline, but the actual implementation merely applies scalar modulation to the reward before advantage computation; GRPO's sampling, advantage estimation, and objective function remain untouched. This modulation would equally apply to any return-based policy gradient method, so describing it as an integration with GRPO is an overstatement. The actual contribution should be attributed to reward shaping.

3.The experiments are overly limited, covering only three LIBERO suites. Moreover, the results in Figure 7 are discouraging: performance on the Goal suite is consistently below the baseline at every checkpoint, which raises genuine concern about whether the method is truly effective. Furthermore, the data source used to construct HAM's supervision is not disclosed, so it is impossible to verify whether HAM training, policy optimization, and evaluation share the same data distribution. Consequently, the reported gains provide no evidence that the method generalizes beyond this benchmark setting, and may simply reflect overfitting to the three LIBERO suites.

Questions for authors

1.How exactly is the HAM hallucination detector trained? What data is used for its training?

2.The title claims applicability to generalist robot policies, yet the evaluation covers three LIBERO suites (the Long suite is omitted) and two real-world tasks. The gains in Fig. 7 are driven by Spatial, while Goal is consistently negative. What evidence supports the generality implied by the title?

Reviewer_crh3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HaWMPO, a hallucination-aware world model-based reinforcement learning framework for Vision-Language-Action (VLA) policy optimization. The method introduces an action-conditioned Hallucination-Aware Model (HAM) to estimate the reliability of generated video chunks and incorporates the predicted hallucination scores into GRPO through a Reward-Soft mechanism. The objective is to reduce the influence of unreliable imagined trajectories during policy optimization.Experiments on three LIBERO task suites show an average success rate of 63.7%, outperforming the reproduced WoVR* baseline by 2.8 percentage points. Additional real-world evaluations on two manipulation tasks demonstrate promising improvements. The paper also evaluates HAM against human-annotated hallucination labels.

Critical strengths
  1. The paper addresses an important challenge in world model-based reinforcement learning: hallucinated visual predictions can introduce misleading learning signals and negatively affect policy optimization. Estimating rollout reliability provides a relevant direction for improving VLA post-training.
  2. The proposed HAM and Reward-Soft mechanism can be incorporated into the existing world model-based policy optimization framework without modifying the world model during policy training. The continuous reliability scores provide an interpretable mechanism for adjusting rewards based on predicted hallucination severity.
  3. HaWMPO improves the average success rate over WoVR* on LIBERO, with a particularly noticeable improvement on Spatial tasks (+8.0 percentage points). The real-world experiments also show positive results. Furthermore, HAM is evaluated against human-annotated hallucination labels, achieving an AUROC of 0.9375 and an AP of 0.8952. This provides useful evidence beyond agreement with the constructed training targets.
Critical weaknesses
  1. Although HaWMPO achieves the highest average success rate in Table 1, its improvement over WoVR* is relatively modest (+2.8 percentage points) and varies substantially across task suites. The improvement is mainly driven by Spatial (+8.0 points), while Object shows a smaller gain (+2.6 points) and Goal experiences a performance decrease (-2.4 points).

    Moreover, the WoVR* baseline excludes the PACE co-evolution mechanism of the original WoVR method. While this provides a controlled comparison under a frozen-world-model setting, it does not establish how HaWMPO compares with the complete WoVR framework. The real-world evaluation is also limited to two tasks with 20 trials per method per task, where the improvement over WoVR* corresponds to only one and two additional successful trials, respectively.

  2. The current ablation study removes HAM and Reward-Soft simultaneously (Section 4.5), demonstrating the effect of the combined pathway but not isolating the contributions of its key design choices.In particular, the paper emphasizes action-conditioned hallucination estimation, yet does not investigate whether action information provides a measurable advantage over visual-only hallucination estimation. Similarly, the current results do not establish whether downstream improvements specifically depend on assigning accurate hallucination scores to the corresponding generated chunks, rather than more general changes to the reward distribution. A control using shuffled hallucination scores would help clarify this distinction.Furthermore, although the human-annotated evaluation in Table 3 is encouraging, the paper does not sufficiently describe the separation between HAM training data and the annotated evaluation trajectories. It also remains unclear whether HAM maintains its detection performance as the policy distribution changes during reinforcement learning, particularly because HAM and the world model remain frozen.

Questions for authors
  1. Performance stability: Could the authors report results across multiple independent training seeds, including variability or confidence intervals for the LIBERO success rates? This is particularly important given the relatively small average improvement over WoVR* and the inconsistent gains across task suites.
  2. Comparison with the complete WoVR framework: How would HaWMPO compare with WoVR including PACE? If a direct comparison is computationally expensive, could the authors clarify the relative training and interaction budgets, and discuss whether HAM-based reward modulation offers complementary benefits to PACE?
  3. Contribution of action conditioning and score reliability: Could the authors provide an ablation of HAM without action conditioning? Additionally, would the policy improvement persist if hallucination scores were randomly shuffled among chunks while preserving their overall distribution? These experiments would help establish whether the gains arise from correctly identifying unreliable imagined transitions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.