- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hallucination-aware World Model-based Policy Optimization (HaWMPO), which first trains a Hallucination Aware Model (HAM) that predicts the intensity of hallucination for a generated chunk of video frames, and uses this model to soft-deduct reward during RL training inside of the world model. The method is shown to perform better than world model-based RL approaches without addressing world model hallucination in simulation and the real world.
Critical strengths
-
The problem is well motivated and is a main bottleneck for policy optimization in a world model.
-
The presentation on the method is clear, and the ablations provides valuable empirical study of how to deal with world model hallucinating.
Critical weaknesses
-
The design and evaluation of HaWMPO remains purely empirical, and some choices are not well justified. For example, why do you choose the four metrics and their corresponding weights when composing the hallucination score? In A.4, the conclusion is that the Image Quality metric (MUSIQ) has negative correlation with the other metrics, so why adding it to the mixture is not obvious.
-
The experiments lack close baselines and real world task coverage. All experiments are compared against methods without estimating hallucination, while existing methods like MOPO (Yu et al., 2020) reduces rewards by estimated model uncertainty, and McConachie et al. (2020) learn where a dynamics model can be trusted, and some concurrent work estimates video world-model hallucination directly (Hansen & Wang, 2026) (comparing to the concurrent work is not expected, but I put it here for discussion). Without comparing to those methods, the novelty and contribution of this paper can be undermined. The real world experiments also lack rigor. There are only 20 rollouts per task without error bars, and only one baseline approach is compared. It is also not clear whether the tested tasks appear in world model/RL training or not. More trials and tasks are needed for statistical significance.
Questions for authors
-
What data is the world model trained on? What tasks are used in RL training for the LIBERO and real world policy?
-
If I understand correctly, the training data for HAM is the same as the world model. In order to make RL benefit from it, it needs to generalize better than the world model itself; i.e. when the world model hallucinates on an OOD observation, it should be able to detect it. Have you tried to test this on a held-out set to validate HAM's performance?