- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper mainly focuses on embodied task planning under heterogeneous uncertainty. The authors propose BeliefAct, which identify task-relevant keypoints and maintain probability-weighted candidate answers. To evaluate agents' ability to handle tasks with uncertainty, the paper also introduces DUET-Bench.
Critical strengths
- Uncertainty in household environments concerns not only unobserved physical states, but also unspecified goal rules and other actors’ behavior. Addressing these factors through a shared planning interface is practically relevant.
- DUET-Bench covers several sources of uncertainty, with rule-governed external changes that can make inference useful for planning.
Critical weaknesses
- The paper does not sufficiently establish its targeted bottleneck, and the improvements over Naive are limited. In Table 1, compared to baseline, BeliefAct's overall success improves from 84.6% to 86.7%.
- The paper lacks empirical comparisons with the closest methods. It does not directly evaluate representative question-based belief, persistent language-belief, or multi-hypothesis memory methods discussed in its related work.
- The ablations do not isolate the central components, and the belief mechanism lacks direct validation.
- Since Question-based beliefs, multiple hypotheses, confidence-annotated language memory, and observation-driven revision all have established precedents, the methodological novelty is incremental. In addition, the paper does not provide sufficient ablation studies or in-depth analysis.
Questions for authors
Please see Critical Weaknesses above.
6 comments
Response to Reviewer 4Jv9 (Part 2/3)
W2. Comparison with the closest methods
The paper lacks empirical comparisons with the closest methods. It does not directly evaluate representative question-based belief, persistent language-belief, or multi-hypothesis memory methods discussed in its related work.
We added HUME (Xiao et al., RSS 2026), the closest recent method that runs zero-shot. HUME keeps LLM-generated hypotheses about missing objects, attributes, and action effects in a symbolic planning model (PDDL) and plans over them with a classical planner. Since DUET-Bench has no hand-written domain, our adaptation, HUME-DUET, has the LLM write this model from the robot's observations, and Fast Downward (Helmert, 2006), a standard off-the-shelf PDDL planner, computes the plans. Like every method in Table 1, it runs on Gemini 3.8 Flash with the same partial observations. Failed episodes are assigned 200 rounds (SR (%) / Rounds / Cost (m)):
| Method | Object search () | Static rule tasks () | Dynamic tasks () | Overall () |
|---|---|---|---|---|
| HUME-DUET | 83.3 / 76.0 / 141.1 | 68.3 / 108.5 / 441.5 | 70.8 / 79.6 / 134.6 | 73.3 / 85.9 / 213.0 |
| BeliefAct (Ours) | 85.0 / 78.4 / 153.5 | 78.3 / 79.1 / 274.6 | 91.7 / 44.1 / 125.3 | 86.7 / 61.4 / 169.7 |
Result analysis: HUME-DUET matches BeliefAct on object search, where rooms and containers map directly onto symbolic predicates, but solves far fewer rule and dynamic episodes, where the LLM must also formalize unstated rules and other people's behavior; in 15 of its 64 failures, no valid model or plan was produced.
The other methods in our related work need resources that a zero-shot benchmark does not provide: BeliefMem builds its memory from 3,000 expert ALFWorld trajectories, Agent-BRACE trains its belief and policy with reinforcement learning, and Tru-POMDP requires a programmatic state space, observation likelihoods, and transition model. Within our framework, the closest question-based design is the fixed questions ablation in W3. We will add HUME-DUET and this discussion to Sections 2 and 5.2.
W3. Isolating ablations and direct validation of the belief
The ablations do not isolate the central components, and the belief mechanism lacks direct validation.
We added two single-component ablations and a direct comparison of the belief with the simulator's hidden state. Two of the original ablations changed multiple components at once: single answer removed both the alternative answers and their probabilities, and direct transition removed tests, expected observations, statuses, and execution records together. The new 240-episode variants separate them: multiple answers (w/o prob.) keeps several ranked answers but drops the probabilities, and w/o expected observation removes only the predicted observation of each test. All variants share the episodes, context, memory note, parser, model, and horizon of the full method (SR (%) / Rounds / Cost (m)):
| Variant | Object search () | Static rule tasks () | Dynamic tasks () | Overall () |
|---|---|---|---|---|
| fixed questions | 83.3 / 75.9 / 148.8 | 81.7 / 79.8 / 288.3 | 90.0 / 52.7 / 146.6 | 86.3 / 65.3 / 182.6 |
| single answer | 78.3 / 77.4 / 146.3 | 78.3 / 81.5 / 289.9 | 90.8 / 46.6 / 135.9 | 84.6 / 63.0 / 177.0 |
| multiple answers (w/o prob.) (new) | 78.3 / 77.0 / 149.8 | 83.3 / 70.5 / 261.5 | 88.3 / 49.8 / 129.7 | 84.6 / 61.8 / 167.7 |
| no transition | 80.0 / 76.4 / 142.3 | 78.3 / 84.6 / 307.7 | 89.2 / 50.4 / 143.9 | 84.2 / 65.5 / 184.5 |
| direct transition | 80.0 / 76.1 / 137.8 | 81.7 / 86.0 / 282.0 | 90.8 / 48.3 / 140.3 | 85.8 / 64.7 / 175.1 |
| w/o expected observation (new) | 81.7 / 72.2 / 138.3 | 83.3 / 74.8 / 271.1 | 89.2 / 49.0 / 134.8 | 85.8 / 61.2 / 169.8 |
| BeliefAct (full) | 85.0 / 78.4 / 153.5 | 78.3 / 79.1 / 274.6 | 91.7 / 44.1 / 125.3 | 86.7 / 61.4 / 169.7 |
Result analysis: on dynamic tasks, every ablated variant is worse than full BeliefAct in success, rounds, and cost, including the two new ones (49.8 and 49.0 rounds vs. 44.1). The clearest effect comes from fixing the questions: in trigger rule, fixed templates raise the rounds from 35.0 to 56.3 (95% CI of the increase +9.1 to +35.6), even though the template already asks which event sets off the housemate. What the template removes is control over the question set: full BeliefAct adds a question the template lacks in 14 of 30 episodes, such as where the destination table is. We will also correct Section 5.5, which overstates the ablation effects on object search and static rule tasks.
Response to Reviewer 4Jv9 (Part 3/3)
W3. Isolating ablations and direct validation of the belief (continued)
In object search, BeliefAct retains the true room among its hypotheses more often than BSG. We compared the belief with the simulator's hidden state in every round of the 240 episodes whose answers could be mapped to ground-truth categories, using lexical rules fixed before any answer was inspected. Before the target is seen, the true room is a hypothesis in 80.6% to 88.4% of rounds, against 37.5% to 50.0% for BSG.
In DUET-Bench's unstable storage task, where one of three storage spots makes items disappear, the top spot is correct in 89% of rounds after an item disappears. For open-ended rules the confidence scores are not calibrated, and we will describe them as uncalibrated confidence scores; removing them nevertheless raises the dynamic-task rounds from 44.1 to 49.8. We will add this comparison to Section 5.
W4. Novelty and depth of analysis
Since Question-based beliefs, multiple hypotheses, confidence-annotated language memory, and observation-driven revision all have established precedents, the methodological novelty is incremental. In addition, the paper does not provide sufficient ablation studies or in-depth analysis.
BeliefAct's novelty lies in what the LLM is allowed to decide. The question-based and multi-hypothesis methods we cite in Section 2 fix in advance which unknown the belief tracks, and they either regenerate their hypotheses at every step or revise them with a predetermined rule. In BeliefAct, the model poses its own questions and adds new ones when the episode reveals a new unknown. It also pairs each hypothesis with a test action and the observation the hypothesis predicts, so it writes how its own belief is revised.
Beyond the method, this work shows three things about explicit beliefs in embodied LLM planning:
- Which unknowns to track is itself a decision the planner must keep making. Fixed templates that already ask about the trigger still require 21 additional rounds on average in trigger rule (W3).
- An explicit belief pays off mainly as earlier completion, with fewer detours once the robot starts acting on its hypotheses (W1).
- Keeping hypotheses in language avoids formalizing rules and other agents online, a step that our HUME adaptation must take and where it falls behind on rule and dynamic tasks (W2).
DUET-Bench is our second contribution. Existing benchmarks each focus on one source of uncertainty: object locations (ALFWorld), goal requirements (Housekeep, TidyBot, PARSEC), or changes during execution, possibly by another agent (Watch-and-Help, CoELA, PARTNR). DUET-Bench places all three in BEHAVIOR-1K scenes, with eight tasks of 30 fixed episodes each and success judged by deterministic simulator predicates, so one method can be evaluated across heterogeneous uncertainty. Reviewer CSmV likewise notes that DUET-Bench "addresses a blind spot in existing embodied AI benchmarks". All of the new results above will be included in the revised paper.
We hope these new results address your concerns about soundness and informativeness, and we would be grateful if you would reconsider your assessment in light of them.
Response to Reviewer 4Jv9 (Part 1/3)
Thank you for your review. Below, we carefully address each of your concerns with new experiments and analyses.
W1. Targeted bottleneck and the gain over baselines
BeliefAct's main gain is earlier task completion: its advantage in success rate over the baselines is larger under tighter round budgets. Within 60 rounds, BeliefAct completes 75.0% of the 240 episodes, against 65.0% to 67.5% for Naive, BSG, and PCE.
The bottleneck BeliefAct targets, stated in Section 1, is that existing LLM-based belief representations fix in advance which question the belief answers, and either regenerate their hypotheses at each step or revise them with predetermined rules. This matters when the relevant unknown differs across tasks and can change within a task.
In DUET-Bench, the object search task has a known unknown, the location of a hidden object. In the trigger rule task, a housemate is set off by an unstated event, such as the robot opening a container, and then brings items to the destination table if one of the items is already on it but hides them otherwise; the robot must find out which event sets off the housemate and keep revising its answer as the housemate reacts. If BeliefAct addresses this bottleneck, its gain should be largest on such rule and dynamic tasks, which the analysis below confirms.
In most DUET-Bench tasks, the agent can recover from a wrong guess: a misplaced item can be picked up and placed again while rounds remain. In trigger rule, for example, all four methods solve all 30 episodes, but BeliefAct needs 35.0 rounds against 45.6 to 50.3. BeliefAct's final success (86.7%) is close to that of the strongest baselines (85.4% for BSG and 84.6% for Naive), and we will update accordingly in the paper, replacing "highest overall success rate" with "success comparable to the strongest baseline". To measure the difference in efficiency, we re-scored the recorded runs at smaller round budgets H′, counting an episode as solved only if it succeeded within its first H′ rounds:
Result analysis: within 60 rounds, BeliefAct completes significantly more episodes than all three baselines (+7.5 pp over Naive, 95% CI +2.9 to +12.1; the intervals for BSG and PCE also exclude zero). The gain is largest on the rule and dynamic tasks (143 vs. 123 to 129 of 180 episodes), while on object search BeliefAct matches or exceeds every baseline (37 vs. 32 to 37 of 60). In the 137 rule and dynamic episodes that all four methods solve, the saving comes after the first grasp or placement, when the robot acts on its hypotheses: BeliefAct then needs 22.4 rounds against 26.8 to 31.6, mostly through less navigation and re-scanning. The ablations in W3 link this gain to the belief design. We will add this table and analysis to Section 5.