Grounded in Hallucination: Belief-Guided Embodied Task Planning in Dynamic, Uncertain Environments

A robot in a household faces heterogeneous uncertainty: unknown object locations, unstated arrangement preferences, and a housemate who moves things by unclear habits. Such settings require planning that is grounded in imagination: driven not only by the observed state, but also by beliefs about task-relevant unknowns. This leads to two challenges: identifying and representing the task-relevant unknowns among the unbounded possibilities of an open world, and revising them from new observations. Existing works pre-specify both the state space and the update rules, and so neither scale nor transfer. We propose Grounded in Hallucination, an LLM-based planning framework that identifies and represents task-relevant beliefs as free-form questions with probability-weighted answers, a representation that generalizes beyond a fixed set. To facilitate belief updates, it pairs each hypothesis with a test action that would settle it and the outcome expected if it holds, compares them against observations to reweight hypotheses. To systematically test planning under heterogeneous uncertainty, we introduce our benchmark, built on BEHAVIOR-1K with eight household task families, where the initial configuration, the goal interpretation, or an external process is uncertain. Grounded in Hallucination achieves the highest overall success rate at the fewest rounds and the lowest interaction cost. It also transfers to two-agent Communicative Watch-and-Help, matching task completion with about 60% fewer LLM calls.