acceptodds
Under review as a conference paper at ICLR 2027

Grounded in Hallucination: Belief-Guided Embodied Task Planning in Dynamic, Uncertain Environments

Abstract

A robot in a household faces heterogeneous uncertainty: unknown object locations, unstated arrangement preferences, and a housemate who moves things by unclear habits. Such settings require planning that is grounded in imagination: driven not only by the observed state, but also by beliefs about task-relevant unknowns. This leads to two challenges: identifying and representing the task-relevant unknowns among the unbounded possibilities of an open world, and revising them from new observations. Existing works pre-specify both the state space and the update rules, and so neither scale nor transfer. We propose Grounded in Hallucination, an LLM-based planning framework that identifies and represents task-relevant beliefs as free-form questions with probability-weighted answers, a representation that generalizes beyond a fixed set. To facilitate belief updates, it pairs each hypothesis with a test action that would settle it and the outcome expected if it holds, compares them against observations to reweight hypotheses. To systematically test planning under heterogeneous uncertainty, we introduce our benchmark, built on BEHAVIOR-1K with eight household task families, where the initial configuration, the goal interpretation, or an external process is uncertain. Grounded in Hallucination achieves the highest overall success rate at the fewest rounds and the lowest interaction cost. It also transfers to two-agent Communicative Watch-and-Help, matching task completion with about 60% fewer LLM calls.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agents, Long-Horizon Planning, Belief Representation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_4Jv9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly focuses on embodied task planning under heterogeneous uncertainty. The authors propose BeliefAct, which identify task-relevant keypoints and maintain probability-weighted candidate answers. To evaluate agents' ability to handle tasks with uncertainty, the paper also introduces DUET-Bench.

Critical strengths
  1. Uncertainty in household environments concerns not only unobserved physical states, but also unspecified goal rules and other actors’ behavior. Addressing these factors through a shared planning interface is practically relevant.
  2. DUET-Bench covers several sources of uncertainty, with rule-governed external changes that can make inference useful for planning.
Critical weaknesses
  1. The paper does not sufficiently establish its targeted bottleneck, and the improvements over Naive are limited. In Table 1, compared to baseline, BeliefAct's overall success improves from 84.6% to 86.7%.
  2. The paper lacks empirical comparisons with the closest methods. It does not directly evaluate representative question-based belief, persistent language-belief, or multi-hypothesis memory methods discussed in its related work.
  3. The ablations do not isolate the central components, and the belief mechanism lacks direct validation.
  4. Since Question-based beliefs, multiple hypotheses, confidence-annotated language memory, and observation-driven revision all have established precedents, the methodological novelty is incremental. In addition, the paper does not provide sufficient ablation studies or in-depth analysis.
Questions for authors

Please see Critical Weaknesses above.

6 comments

Response to Reviewer 4Jv9 (Part 1/3)

Thank you for your review. Below, we carefully address each of your concerns with new experiments and analyses.

W1. Targeted bottleneck and the gain over baselines

The paper does not sufficiently establish its targeted bottleneck, and the improvements over Naive are limited. In Table 1, compared to baseline, BeliefAct's overall success improves from 84.6% to 86.7%.

BeliefAct's main gain is earlier task completion: its advantage in success rate over the baselines is larger under tighter round budgets. Within 60 rounds, BeliefAct completes 75.0% of the 240 episodes, against 65.0% to 67.5% for Naive, BSG, and PCE.

The bottleneck BeliefAct targets, stated in Section 1, is that existing LLM-based belief representations fix in advance which question the belief answers, and either regenerate their hypotheses at each step or revise them with predetermined rules. This matters when the relevant unknown differs across tasks and can change within a task.

In DUET-Bench, the object search task has a known unknown, the location of a hidden object. In the trigger rule task, a housemate is set off by an unstated event, such as the robot opening a container, and then brings items to the destination table if one of the items is already on it but hides them otherwise; the robot must find out which event sets off the housemate and keep revising its answer as the housemate reacts. If BeliefAct addresses this bottleneck, its gain should be largest on such rule and dynamic tasks, which the analysis below confirms.

In most DUET-Bench tasks, the agent can recover from a wrong guess: a misplaced item can be picked up and placed again while rounds remain. In trigger rule, for example, all four methods solve all 30 episodes, but BeliefAct needs 35.0 rounds against 45.6 to 50.3. BeliefAct's final success (86.7%) is close to that of the strongest baselines (85.4% for BSG and 84.6% for Naive), and we will update accordingly in the paper, replacing "highest overall success rate" with "success comparable to the strongest baseline". To measure the difference in efficiency, we re-scored the recorded runs at smaller round budgets H′H', counting an episode as solved only if it succeeded within its first H′H' rounds:

Episodes completed within H′H' rounds (all 240)NaiveBSGPCEBeliefAct
H′=40H' = 4049.6%50.4%45.8%57.1%
H′=60H' = 6067.5%67.1%65.0%75.0%
H′=100H' = 10077.9%77.5%76.2%81.7%
H′=200H' = 200 (Table 1)84.6%85.4%82.5%86.7%

Result analysis: within 60 rounds, BeliefAct completes significantly more episodes than all three baselines (+7.5 pp over Naive, 95% CI +2.9 to +12.1; the intervals for BSG and PCE also exclude zero). The gain is largest on the rule and dynamic tasks (143 vs. 123 to 129 of 180 episodes), while on object search BeliefAct matches or exceeds every baseline (37 vs. 32 to 37 of 60). In the 137 rule and dynamic episodes that all four methods solve, the saving comes after the first grasp or placement, when the robot acts on its hypotheses: BeliefAct then needs 22.4 rounds against 26.8 to 31.6, mostly through less navigation and re-scanning. The ablations in W3 link this gain to the belief design. We will add this table and analysis to Section 5.

Response to Reviewer 4Jv9 (Part 2/3)

W2. Comparison with the closest methods

The paper lacks empirical comparisons with the closest methods. It does not directly evaluate representative question-based belief, persistent language-belief, or multi-hypothesis memory methods discussed in its related work.

We added HUME (Xiao et al., RSS 2026), the closest recent method that runs zero-shot. HUME keeps LLM-generated hypotheses about missing objects, attributes, and action effects in a symbolic planning model (PDDL) and plans over them with a classical planner. Since DUET-Bench has no hand-written domain, our adaptation, HUME-DUET, has the LLM write this model from the robot's observations, and Fast Downward (Helmert, 2006), a standard off-the-shelf PDDL planner, computes the plans. Like every method in Table 1, it runs on Gemini 3.8 Flash with the same partial observations. Failed episodes are assigned 200 rounds (SR (%) / Rounds / Cost (m)):

MethodObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
HUME-DUET83.3 / 76.0 / 141.168.3 / 108.5 / 441.570.8 / 79.6 / 134.673.3 / 85.9 / 213.0
BeliefAct (Ours)85.0 / 78.4 / 153.578.3 / 79.1 / 274.691.7 / 44.1 / 125.386.7 / 61.4 / 169.7

Result analysis: HUME-DUET matches BeliefAct on object search, where rooms and containers map directly onto symbolic predicates, but solves far fewer rule and dynamic episodes, where the LLM must also formalize unstated rules and other people's behavior; in 15 of its 64 failures, no valid model or plan was produced.

The other methods in our related work need resources that a zero-shot benchmark does not provide: BeliefMem builds its memory from 3,000 expert ALFWorld trajectories, Agent-BRACE trains its belief and policy with reinforcement learning, and Tru-POMDP requires a programmatic state space, observation likelihoods, and transition model. Within our framework, the closest question-based design is the fixed questions ablation in W3. We will add HUME-DUET and this discussion to Sections 2 and 5.2.

W3. Isolating ablations and direct validation of the belief

The ablations do not isolate the central components, and the belief mechanism lacks direct validation.

We added two single-component ablations and a direct comparison of the belief with the simulator's hidden state. Two of the original ablations changed multiple components at once: single answer removed both the alternative answers and their probabilities, and direct transition removed tests, expected observations, statuses, and execution records together. The new 240-episode variants separate them: multiple answers (w/o prob.) keeps several ranked answers but drops the probabilities, and w/o expected observation removes only the predicted observation of each test. All variants share the episodes, context, memory note, parser, model, and horizon of the full method (SR (%) / Rounds / Cost (m)):

VariantObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
fixed questions83.3 / 75.9 / 148.881.7 / 79.8 / 288.390.0 / 52.7 / 146.686.3 / 65.3 / 182.6
single answer78.3 / 77.4 / 146.378.3 / 81.5 / 289.990.8 / 46.6 / 135.984.6 / 63.0 / 177.0
multiple answers (w/o prob.) (new)78.3 / 77.0 / 149.883.3 / 70.5 / 261.588.3 / 49.8 / 129.784.6 / 61.8 / 167.7
no transition80.0 / 76.4 / 142.378.3 / 84.6 / 307.789.2 / 50.4 / 143.984.2 / 65.5 / 184.5
direct transition80.0 / 76.1 / 137.881.7 / 86.0 / 282.090.8 / 48.3 / 140.385.8 / 64.7 / 175.1
w/o expected observation (new)81.7 / 72.2 / 138.383.3 / 74.8 / 271.189.2 / 49.0 / 134.885.8 / 61.2 / 169.8
BeliefAct (full)85.0 / 78.4 / 153.578.3 / 79.1 / 274.691.7 / 44.1 / 125.386.7 / 61.4 / 169.7

Result analysis: on dynamic tasks, every ablated variant is worse than full BeliefAct in success, rounds, and cost, including the two new ones (49.8 and 49.0 rounds vs. 44.1). The clearest effect comes from fixing the questions: in trigger rule, fixed templates raise the rounds from 35.0 to 56.3 (95% CI of the increase +9.1 to +35.6), even though the template already asks which event sets off the housemate. What the template removes is control over the question set: full BeliefAct adds a question the template lacks in 14 of 30 episodes, such as where the destination table is. We will also correct Section 5.5, which overstates the ablation effects on object search and static rule tasks.

Response to Reviewer 4Jv9 (Part 3/3)

W3. Isolating ablations and direct validation of the belief (continued)

In object search, BeliefAct retains the true room among its hypotheses more often than BSG. We compared the belief with the simulator's hidden state in every round of the 240 episodes whose answers could be mapped to ground-truth categories, using lexical rules fixed before any answer was inspected. Before the target is seen, the true room is a hypothesis in 80.6% to 88.4% of rounds, against 37.5% to 50.0% for BSG.

In DUET-Bench's unstable storage task, where one of three storage spots makes items disappear, the top spot is correct in 89% of rounds after an item disappears. For open-ended rules the confidence scores are not calibrated, and we will describe them as uncalibrated confidence scores; removing them nevertheless raises the dynamic-task rounds from 44.1 to 49.8. We will add this comparison to Section 5.

W4. Novelty and depth of analysis

Since Question-based beliefs, multiple hypotheses, confidence-annotated language memory, and observation-driven revision all have established precedents, the methodological novelty is incremental. In addition, the paper does not provide sufficient ablation studies or in-depth analysis.

BeliefAct's novelty lies in what the LLM is allowed to decide. The question-based and multi-hypothesis methods we cite in Section 2 fix in advance which unknown the belief tracks, and they either regenerate their hypotheses at every step or revise them with a predetermined rule. In BeliefAct, the model poses its own questions and adds new ones when the episode reveals a new unknown. It also pairs each hypothesis with a test action and the observation the hypothesis predicts, so it writes how its own belief is revised.

Beyond the method, this work shows three things about explicit beliefs in embodied LLM planning:

  1. Which unknowns to track is itself a decision the planner must keep making. Fixed templates that already ask about the trigger still require 21 additional rounds on average in trigger rule (W3).
  2. An explicit belief pays off mainly as earlier completion, with fewer detours once the robot starts acting on its hypotheses (W1).
  3. Keeping hypotheses in language avoids formalizing rules and other agents online, a step that our HUME adaptation must take and where it falls behind on rule and dynamic tasks (W2).

DUET-Bench is our second contribution. Existing benchmarks each focus on one source of uncertainty: object locations (ALFWorld), goal requirements (Housekeep, TidyBot, PARSEC), or changes during execution, possibly by another agent (Watch-and-Help, CoELA, PARTNR). DUET-Bench places all three in BEHAVIOR-1K scenes, with eight tasks of 30 fixed episodes each and success judged by deterministic simulator predicates, so one method can be evaluated across heterogeneous uncertainty. Reviewer CSmV likewise notes that DUET-Bench "addresses a blind spot in existing embodied AI benchmarks". All of the new results above will be included in the revised paper.

We hope these new results address your concerns about soundness and informativeness, and we would be grateful if you would reconsider your assessment in light of them.

Reviewer_uAhNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents BeliefAct, an LLM-based embodied task planning framework for partially observable, dynamic household environments with heterogeneous uncertainties. It represents task-relevant uncertainty as open-ended natural-language questions paired with weighted hypotheses, each attached to a test action and expected observation, and updates belief weights by comparing predicted and real observations during interaction. The authors build DUET‑Bench, a benchmark covering hidden object states, latent goal rules, and scripted external dynamics, and evaluate BeliefAct against three baselines on 240 simulation episodes plus a Communicative Watch-and-Help transfer setting. Results show higher success rates and lower interaction cost, with ablations validating the value of open question generation, multiple competing hypotheses, and structured belief transition.

Critical strengths

(1) A key strength is that the paper clearly identifies the practical challenge of heterogeneous uncertainty in embodied planning and proposes a clean, principled LLM-native belief representation: open-ended questions paired with weighted hypotheses coupled with test actions and expected observations. This design naturally accommodates different types of unobserved factors without predefining state spaces or update rules, which is a meaningful departure from prior fixed-form belief planners.

(2) The accompanying DUET-Bench systematically combines state, goal, and dynamic uncertainty, and the empirical results together with ablations provide consistent evidence that this belief mechanism improves task efficiency and success.

Critical weaknesses

(1) First, the hypothesis weights are heuristic LLM-generated estimates without formal calibration. The work lacks direct comparison between the maintained belief distribution and ground-truth latent states, so we cannot verify how well the model’s beliefs track the true underlying uncertainty. (2) Second, all evaluations rely exclusively on Gemini 3.8 Flash; the absence of cross-LLM robustness tests leaves unclear whether the gains of BeliefAct generalize to other foundation models, limiting the reliability of the core conclusions.

Questions for authors

Please refer to the above weaknesses part.

2 comments

Response to Reviewer uAhN (Part 1/2)

Thank you for your positive review and for recognizing both the belief representation and DUET-Bench. Below, we carefully address your two concerns with new analyses.

W1. Calibration and comparison with ground-truth latent states

First, the hypothesis weights are heuristic LLM-generated estimates without formal calibration. The work lacks direct comparison between the maintained belief distribution and ground-truth latent states, so we cannot verify how well the model's beliefs track the true underlying uncertainty.

We compared BeliefAct's belief with the simulator's hidden state across the 240 episodes, in every round whose answers could be mapped to ground-truth categories. In object search, BeliefAct retains the true location more often than BSG. Its confidence scores are not generally calibrated, yet removing them makes execution less efficient on dynamic tasks. Free-text answers were mapped to ground-truth categories with lexical rules fixed before any answer was inspected, and answers that match several categories get no credit. Each episode was split at the first observation relevant to the answer ("the evidence" below), so rounds before any evidence are not counted as errors.

In object search, before the target is seen, the true room has non-zero probability in 80.6% (single object) and 88.4% (multiple objects) of rounds, against 37.5% and 50.0% for the location-belief baseline BSG, which assigns zero probability to unvisited rooms. Each of DUET-Bench's six rule and dynamic tasks hides one variable, given in parentheses:

Task (hidden variable)Episodes mappedTrue answer among hypotheses before the evidenceTop answer correct after the evidence (correct / rounds)Top answer wrong when held at p≥0.80p \ge 0.80 (wrong / rounds)Brier score (uniform-guess baseline)
food splitting rule (attribute that splits six foods between two countertops)2692.3%41.3% (109/264)56.5% (113/200)1.09 (0.80)
meal composition (which two foods form the intended meal)28100.0%79.6% (797/1,001)23.5% (180/767)0.37 (0.86)
division of labor (attribute of the foods an unseen helper delivers)2926.1%28.7% (37/129)65.9% (54/82)1.22 (0.91)
unstable storage (which of three storage spots removes items)2999.5%88.9% (169/190)0.0% (0/134)0.56 (0.67)
visitor intent (the pattern by which a visitor moves items)1148.4%31.3% (55/176)65.2% (45/69)0.91 (0.75)
trigger rule (which event, such as opening a container or entering a room, sets off a housemate who moves items)3080.5%47.7% (158/331)32.2% (66/205)0.90 (0.75)

Coverage is high when there are few candidate answers and drops when the rule space is large (in division of labor, the agent lists only three or four candidate attributes). High-confidence answers are reliable only for the three-way question in unstable storage (0 of 134 wrong); for open-ended rules, 32% to 66% are wrong and Brier scores are worse than a uniform guess. We will therefore describe the weights in the paper as uncalibrated confidence scores.

The new variant multiple answers (w/o prob.) keeps several ranked answers per question but drops the scores. On dynamic tasks it solves 4 fewer episodes and needs 5.7 more rounds (49.8 vs. 44.1), mostly in trigger rule (46.3 vs. 35.0 rounds) and unstable storage (17 vs. 20 solved), where several hypotheses stay plausible for many rounds; on static rule tasks the ranked list solves 50 episodes versus 47 for full BeliefAct. This suggests that the scores help the robot weigh hypotheses that remain open during action, even though they are not calibrated. We will add the ground-truth comparison, its mapping rules, and the ablation to Section 5 and the appendix.

Response to Reviewer uAhN (Part 2/2)

W2. Generalization to other foundation models

Second, all evaluations rely exclusively on Gemini 3.8 Flash; the absence of cross-LLM robustness tests leaves unclear whether the gains of BeliefAct generalize to other foundation models, limiting the reliability of the core conclusions.

We ran all 240 episodes with GPT-6.1 Terra at xhigh reasoning effort, a stronger model from a different family. Failed episodes are assigned 200 rounds (SR (%) / Rounds / Cost (m)):

Backbone · MethodObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
GPT-6.1 Terra (xhigh) · Naive70.0 / 83.5 / 170.486.7 / 62.4 / 236.195.8 / 38.0 / 132.987.1 / 55.5 / 168.1
· BSG68.3 / 85.8 / 162.888.3 / 72.9 / 237.196.7 / 38.0 / 124.787.5 / 58.7 / 162.3
· PCE73.3 / 90.6 / 163.681.7 / 85.4 / 274.889.2 / 50.6 / 122.183.3 / 69.3 / 170.7
· BeliefAct68.3 / 86.2 / 198.991.7 / 61.6 / 238.295.8 / 38.4 / 128.387.9 / 56.1 / 173.4

With Terra, BeliefAct still solves the most static rule episodes (55 of 60, against 49 to 53), and it is the only belief-based method that keeps pace with Naive there (61.6 vs. 62.4 rounds), needing significantly fewer rounds than BSG (72.9) and PCE (85.4). On dynamic tasks the efficiency gain disappears: Naive reaches 95.8% success in 38.0 rounds, and BeliefAct and BSG match it. Overall success is at parity (211, 209, and 210 episodes for BeliefAct, Naive, and BSG; 200 for PCE).

We hope these analyses address your concerns.

Reviewer_CSmVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework called beliefact that helps embodied AI agents plan tasks in messy home environments where key information is missing or changing. Instead of using rigid templates or guessing just one answer, beliefact prompts an LLM to ask itself open-ended questions about critical unknowns. The agent THEN assigns probabilities to competing answers, pairs each hypothesis with an expected observation if tested, and updates its beliefs as it takes actions and observes outcomes. To evaluate this, the authors also introduce a simulation benchmark from eight household task families with static, goal, and dynamic uncertainties.

Critical strengths
  • The proposed belief representation is flexible and intuitive. Instead of using traditional Bayesina state emstimators or symbolic planners that restrict belief to prefixed schemas, representing belief state as NL question-answer pairs with probabilities is a flexible design that allows the model to capture heterogeneous unknowns.
  • The proposed benchmark is valuable and addresses a blind spot in existing embodied AI benchmarks by evaluating heterogeneous, concurrent uncertainties such as initial state opacity, underspecified semantic goals, and latent external dynamics.The details about the construction of the benchmark is also comprehensive in Appendix C.
  • The ablations are reasonably designed and the paper is honest about cases where their proposed method is not the winner. Separating representation (fixed questions, single answer) from transition (none, direct) appears as the right decomposition to me. The authors report where there proposed approach loses at object-search cost, and BSG solving one more episode in two blocks, which is a proper academic practice.
Critical weaknesses
  • The effect sizes are not large and there is no uncertainty quantification. Overall SR is 208/240 versus 205 for BSG and 203 for Naive, which is a gap of 3 to 5 episodes. Every method has one run per episode at temperature 1.0, with no seeds, confidence intervals, or significance tests. Ablation SR gaps are 1 to 6 episodes. Given these results, I find the claim “Highest success rate” as stated by the paper a misleading one.
  • The baselines do not seem to be strongest possible methods to compare. BSG tracks only object-location beliefs, so it cannot represent the rule and dynamic uncertainty by design. PCE has a smaller memory cap, no plan in memory, and four food-splitting episodes ended in parse failures that count as failures. The closest methods discussed in the related work are Tru-POMDP, Agent-BRACE, and Belief Memory but are not compared against. If these methods could not be compared to the proposed approach, the paper should have properly argued them away.
  • The abstract says “about 60% fewer LLM calls,” but against Naive it is 50% and the 60% claim seems to only apply to PCE. The appendix explains that the gap is because BeliefAct merges belief and action into one call, while Naive uses two. That is an engineering difference, not an evidence that beliefs help. I find this as another misleading claim about the efficacy of the approach.
  • There is not sufficient evidence for generalization of the results. All results come from Gemini 3.8 Flash at low thinking. There is no test of a stronger or weaker model, or whether the gains persist as the model’s native reasoning improves, which is the main threat to this kind of scaffold.
Questions for authors

Please find the weaknesses stated above. Also, I didn't see any analysis of failures or cost. When your method fails, what is the reason? wrong hypotheses, bad questions, loops, parse errors? At what rate?

4 comments

Response to Reviewer CSmV (Part 1/3)

Thank you for your detailed review and for recognizing the flexibility of the belief representation and the value of DUET-Bench. Below, we carefully address each of your concerns.

W1. Effect size and uncertainty quantification

The effect sizes are not large and there is no uncertainty quantification. Overall SR is 208/240 versus 205 for BSG and 203 for Naive ... Ablation SR gaps are 1 to 6 episodes. Given these results, I find the claim "Highest success rate" as stated by the paper a misleading one.

At the full 200-round budget, BeliefAct solves 208 of 240 episodes, compared with 203 for Naive, 205 for BSG, and 198 for PCE. We will revise the Abstract, Section 1, and Section 5.3 to describe final success as comparable to the strongest baselines and emphasize execution efficiency. Each episode was run once at temperature 1.0; the uncertainty estimates for the budgeted completion rates below use a paired bootstrap over episodes. In the DUET-Bench results, failed episodes are assigned 200 rounds when computing average rounds.

In most DUET-Bench tasks, a wrong guess does not end the episode: an item placed on the wrong surface can be picked up and placed again, so a method that guesses wrong still succeeds if rounds remain. This leaves room for methods with similar final success rates to differ in how many rounds they need. We re-scored the recorded runs at smaller round budgets H′H', counting an episode as solved only if it succeeded within its first H′H' rounds:

Episodes completed within H′H' rounds (all 240)NaiveBSGPCEBeliefAct
H′=40H' = 4049.6%50.4%45.8%57.1%
H′=60H' = 6067.5%67.1%65.0%75.0%
H′=100H' = 10077.9%77.5%76.2%81.7%
H′=200H' = 200 (Table 1)84.6%85.4%82.5%86.7%

Within 60 rounds, BeliefAct completes significantly more episodes than every baseline: +7.5 pp over Naive (95% CI +2.9 to +12.1), +7.9 pp over BSG (+2.9 to +12.9), and +10.0 pp over PCE (+5.0 to +15.0). On the 180 rule and dynamic episodes it also needs significantly fewer rounds (55.8 vs. 62.9 to 70.3). The ablations show the same pattern: small success gaps, but all six ablated variants need more rounds on dynamic tasks (46.6 to 52.7 vs. 44.1).

Response to Reviewer CSmV (Part 2/3)

W2. Strength of the baselines

BSG tracks only object-location beliefs, so it cannot represent the rule and dynamic uncertainty by design. PCE has a smaller memory cap, no plan in memory, and four food-splitting episodes ended in parse failures that count as failures. The closest methods discussed in the related work are Tru-POMDP, Agent-BRACE, and Belief Memory but are not compared against.

We agree that BSG cannot represent rule or actor uncertainty, so Table 1 compares complete systems. To compare at matched scope, our fixed questions ablation replaces BeliefAct's model-posed questions with task-specific templates that already cover each task's hidden rule, the other actor, and target locations. On dynamic tasks it needs 52.7 rounds instead of 44.1.

PCE's parse failures cannot account for the success gap. Even if the four food-splitting episodes that ended in parse failures counted as successes, PCE would solve at most 202 of 240 episodes against BeliefAct's 208, and on the 189 episodes both methods solve, BeliefAct needs significantly fewer rounds (37.4 vs. 42.3). We will list the interface differences you point out in Section 5.2. Because these differences remain, we use PCE as a system-level comparison and rest the claims about BeliefAct's components on the ablations, which share the full method's interface.

Tru-POMDP, Agent-BRACE, and BeliefMem each need resources that a zero-shot benchmark does not provide: Tru-POMDP requires a programmatic state space, observation likelihoods, and transition model; Agent-BRACE trains its belief and policy with reinforcement learning; and BeliefMem builds its memory from 3,000 expert ALFWorld trajectories. We will explain this in Sections 2 and 5.2.

As a closely related method that does run zero-shot, we added HUME (Xiao et al., RSS 2026), which keeps LLM-generated hypotheses inside a symbolic planning model (PDDL) and plans over them with a classical planner. Our adaptation, HUME-DUET, has the LLM write this model from the robot's observations and computes plans with Fast Downward (Helmert, 2006), a standard off-the-shelf PDDL planner. Like every method in Table 1, it runs on Gemini 3.8 Flash and receives the same partial observations (SR (%) / Rounds / Cost (m)):

MethodObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
HUME-DUET83.3 / 76.0 / 141.168.3 / 108.5 / 441.570.8 / 79.6 / 134.673.3 / 85.9 / 213.0
BeliefAct (Ours)85.0 / 78.4 / 153.578.3 / 79.1 / 274.691.7 / 44.1 / 125.386.7 / 61.4 / 169.7

Result analysis: HUME-DUET matches BeliefAct on object search, where rooms and containers map directly onto symbolic predicates, but solves far fewer rule and dynamic episodes, where the LLM must also formalize unstated rules and other people's behavior; in 15 of its 64 failures, no valid model or plan could be produced.

W3. LLM-call savings on C-WAH

The abstract says "about 60% fewer LLM calls," but against Naive it is 50% and the 60% claim seems to only apply to PCE. ... That is an engineering difference, not an evidence that beliefs help.

We agree. To confirm this, we ran Naive with a single LLM call per step on all 50 C-WAH tasks, with the same prompt and environment:

Method (Gemini 3.8 Flash)Calls / decisionStepsDist. (m)CommLLM calls
Naive (submitted, 2 calls)239.833.70.7448.0
Naive (1 call)139.333.40.5123.9
BSG243.337.30.5150.1
PCE2 + gated verify/eval41.734.41.7260.5
BeliefAct (Ours)139.233.20.5824.0

Single-call Naive matches BeliefAct in calls, steps, and distance, so the saving comes from the single-call design. We will no longer present it as evidence for the belief, and we will report each comparison separately (50% fewer calls than two-call Naive, 60% fewer than PCE). On DUET-Bench, Naive already makes one call per round (Appendix A.1), so none of the DUET-Bench results depend on this difference.

Response to Reviewer CSmV (Part 3/3)

W4. Generalization to other foundation models

All results come from Gemini 3.8 Flash at low thinking. There is no test of a stronger or weaker model, or whether the gains persist as the model's native reasoning improves, which is the main threat to this kind of scaffold.

We ran all 240 episodes with GPT-6.1 Terra at xhigh reasoning effort, a stronger model from a different family (SR (%) / Rounds / Cost (m)):

Backbone · MethodObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
GPT-6.1 Terra (xhigh) · Naive70.0 / 83.5 / 170.486.7 / 62.4 / 236.195.8 / 38.0 / 132.987.1 / 55.5 / 168.1
· BSG68.3 / 85.8 / 162.888.3 / 72.9 / 237.196.7 / 38.0 / 124.787.5 / 58.7 / 162.3
· PCE73.3 / 90.6 / 163.681.7 / 85.4 / 274.889.2 / 50.6 / 122.183.3 / 69.3 / 170.7
· BeliefAct68.3 / 86.2 / 198.991.7 / 61.6 / 238.295.8 / 38.4 / 128.387.9 / 56.1 / 173.4

With Terra, BeliefAct still solves the most static rule episodes (55 of 60, against 49 to 53), and it is the only belief-based method that keeps pace with Naive there (61.6 vs. 62.4 rounds), needing significantly fewer rounds than BSG (72.9) and PCE (85.4). On dynamic tasks the efficiency gain disappears: Naive, BSG, and BeliefAct all reach 95.8% to 96.7% success in about 38 rounds. This is the risk you raised, and we will report it in Section 5. Overall success is at parity (211, 209, and 210 episodes for BeliefAct, Naive, and BSG; 200 for PCE).

Q1. Failure analysis

When your method fails, what is the reason? wrong hypotheses, bad questions, loops, parse errors? At what rate?

BeliefAct fails 32 of 240 episodes (Naive 37, BSG 35, PCE 42), and none of its failures are parse errors: 22 exhaust the 200-round budget and 10 end with a premature declare_done. The causes differ by task category:

Task categoryBeliefAct failuresMain cause
Object search9All in one scene, where navigation cannot pass a narrow doorway; 7 of these episodes fail for every method
Static rule tasks13The true rule never enters the hypotheses, or a mistake gives no feedback. In food splitting rule (split six foods between two countertops by an unstated attribute), BeliefAct solves only 2 of 5 episodes whose attribute is origin and 3 of 5 for bulk. In meal composition (bring the two foods of an unstated meal to a table), moving a distractor food gives no signal, and every method fails 5 to 7 episodes
Dynamic tasks10All in unstable storage, described below

In unstable storage, one of three storage spots removes items while the robot is not looking, and the robot must call declare_done once no removal is pending. In 9 of BeliefAct's 10 failures, no spot had probability above 0.5 when it called declare_done, and in 5 of these it had lowered the true spot after watching items stay there, although removals happen only out of view. Belief uncertainty therefore does not reliably prevent premature stopping. Naive, BSG, and PCE fail this task in 10, 7, and 8 episodes, respectively.

The failures also explain why the margin differs across categories. On the 181 episodes that all four methods solve, BeliefAct's margin is largest where the hidden rule must be inferred from the environment's responses: it needs 7 to 15 fewer rounds than the baselines on static rule tasks and 3 to 7 fewer on dynamic tasks, and is close to them on object search (2 to 3 more rounds). We will add this breakdown to Section 5.3 and list the ungated declare_done as a limitation.

We hope these results and clarifications address your concerns, and we would be grateful if you would take them into account in your final assessment.

I appreciate the authors’ solid rebuttal. They've addressed most of my conerns. The discussion on why TruPOMDP, Agent-BRACE, and BeliefMem are incompatible with a zero-shot benchmark is convincing, and the new results for the HUME-DUET baseline are helpful. The new experiments with GPT-6.1 Terra also addressed my concern about generalization to other foundation models. I've modified my score accordingly.

Reviewer_1K8ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for LLM-based task planning for robotic systems in uncertain settings. The agent's uncertainty about the task specification, the state of the world, and its evolution over time is represented through an automatically maintained list of open-form questions, generated by the LLM, each with a credence-weighted list of candidate answers, also generated by the LLM and reassessed over time as new information becomes available through the robot's observations. Analogous to classical POMDP agents, at each decision cycle, the LLM receives a new observation, updates its belief, and outputs a chosen action, which in this case consists in invoking a pre-defined robot observation, locomotion, or manipulation primitive. To evaluate the proposed method, the paper extends an existing embodied AI simulation benchmark, introducing instances of the above-mentioned types of uncertainty (8 task types, with 30 scenarios each). Evaluation is against three LLM-based planner baselines: naive decision-making (direct LLM action selection from context), a Bayesian scene graph representation with only object location probabilities, and a decision tree encoding multiple hypotheses from which the agent determines the most promising path to a leaf action node. The results show competitive (albeit comparable) performance across tasks and metrics (success rate, decision cycles required, and cost).

Critical strengths

The paper is well written, and its overall narrative and technical exposition are easy to follow (although more rigorous formalization in Section 3 would be beneficial).

A major strength of the proposed method is its straightforward implementation by allowing the LLM to pose its own questions and automatically maintain a measure of confidence over candidate hypotheses, without requiring as much external scaffolding as other methods. This is in itself compelling and interesting and, together with the experimental evaluation, contributes a valuable data point to the broader open investigation in the field about how much of embodied AI agents' decision-making should rely directly on the "base" LLM versus lean on additional structure.

Critical weaknesses

Novelty (major). The paper's contribution, which I do believe is novel and interesting, seems to need more fine-grained qualification, because some recently published work has already showed the success of similar ideas in a comparable range of embodied decision problems. In particular, the HUME paper by David Hsu's group (published in RSS 2026 last summer) showed that an LLM-maintained belief, represented as a probabilistic factor graph, can enable open-world POMDP-like planning. Based on my understanding of the paper, it appears like the BeliefAct framework proposed here can be understood as a "lightweight" variant of HUME in which (1) the factor graph is "flat" (with the answers to different questions treated as mutually independent) and (2) the planning is left implicit, directly asking the LLM to choose the next action without explicitly constructing a tree of possible future discoveries and associated conditional plans as HUME does. Given this, HUME appears like the proposed method's "closest relative", and a thorough comparison (in the exposition and in the results) is needed.

Results (major). Related to the above point, the choice of experimental baselines should be clarified. In the case of the Bayesian scene graph baseline, the paper does not seem to compare to the cited LLM-MCTS method exactly and on the other hand chooses to only include object location in the agent's uncertain belief. Scientifically, this creates a confounder: are differences in observed performance due to how the belief is represented and maintained (most relevant to this paper's contribution) or to what the belief is allowed to be about (much less interesting, and a bit of a straw-man, since nothing fundamental seems to prevent all of the representation methods considered from maintaining a belief over the same types of unknowns). A detailed ablation would resolve this question—and it is quite important to understand what the underlying takeaway should be for the community, beyond "this method does better on this benchmark".

Scholarship (minor). In the Introduction and Related Work sections, I found that some claims were not well substantiated. This is especially important when critiquing prior work in the field. For example:

  1. The "fixed update formula" in the cited Tang et al. (2025) paper, which the paper claims is "not a generalizable solution" got me curious, so I went through the paper and was surprised to find that the fixed formula is actually Bayes' rule! I am unable to find an instance of Bayes' rule not being applicable to any of the settings handled by BeliefAct. In fact, the above-mentioned HUME paper's results seem to suggest that applying a Bayesian update based on the LLM's hypothesis-conditional observation likelihoods produces better results than directly re-querying the LLM's belief after incorporating the new observation into its context. Please revise claims about the limitations of prior work to ensure they are both accurate and well backed in the text.
  2. The statement that "existing embodied task planning approaches primarily detect and react to change after it occurs, rather than anticipating it beforehand" is incorrect—the embodied task planning literature is extremely rich and many methods over the past years and decades have successfully anticipated change (mainly in the POMDP and TAMP literatures, including work by the groups of Mykel Kochenderfer, David Hsu, Leslie Kaelbling, Tom Silver, among others). The problem seems to be that this section is only engaging the last ~3 years of LLM-based task planning, but it's making statements that appear to refer to the entire embodied task planning literature. While it is natural to focus primarily on the concrete area of LLM-based planning, the broader literature in embodied planning under uncertainty remains fully relevant here (so much so that the proposed method is essentially using the LLM to efficiently implement various components of a POMDP planner, as presented in Section 3). The paper should both nuance its statements about what prior work in the field has/hasn't considered and provide some brief broader context, to avoid giving the wrong impression that the entire discipline of embodied decision-making under uncertainty started in the current decade!

Technical rigor (minor). The mathematical formulation presented in Section 3 makes intuitive sense but contains some instances of ambiguity and inconsistency in notation. For example, LLM is used as a function in Equations 1, 3, and 4 (3 out of the 4 equations in the paper), but it has a different "function signature" in each of these cases! In addition, the decision of the LLM appears to be produced by two alternative formulas: one from Equation 1, the other from Equation 4, leading to two competing accounts. In addition, the paper uses the term "rollout," suggesting that there is an internal simulation of future actions as part of the planning process, but this doesn't seem to actually be the case—in this context, the intended meaning appears to be action "selection" or "execution".

Questions for authors
  1. Can you please clarify if there are substantial innovations in your framework with respect to HUME (Anxing Xiao et al., "Hypothesis-driven Model Expansion under Uncertainty for Open-World Robot Planning", RSS 2026), separate from a more streamlined implementation with the two differences in my interpretation?

  2. How does BeliefAct compare against HUME on your benchmark? If performance is similar, does BeliefAct offer significant computational savings?

  3. Can you offer some insight about where the benefits and differences in BeliefAct's performance come from relative to baselines? In particular, if you allow the MCTS-style planner and decision-tree-based planner baselines to maintain a belief over any task-relevant hypotheses (as generated by the LLM), what are the advantages and tradeoffs between the free-form QA-based representation and the baselines' graph-based alternatives.

  4. Can you clarify how you assess the completion conditions externally (benchmark "ground truth" completion) and internally?

  5. Section 3 states that, for the planner, "the objective is to complete the task within H rounds while limiting the cumulative cost", but it is unclear from the exposition how the planner's decisions account for this objective: is the LLM internally engaging in predictive assessments (e.g., via chain-of-thought reasoning) about how many rounds remain and how many steps (and how costly) it may still need to execute?

5 comments

Response to Reviewer 1K8R (Part 1/3)

Thank you for your careful review and constructive suggestions, in particular for pointing us to HUME. Below, we carefully address each of your concerns and questions.

W1 and Q1. Relation to HUME

... it appears like the BeliefAct framework proposed here can be understood as a "lightweight" variant of HUME in which (1) the factor graph is "flat" (with the answers to different questions treated as mutually independent) and (2) the planning is left implicit ...

Can you please clarify if there are substantial innovations in your framework with respect to HUME ..., separate from a more streamlined implementation with the two differences in my interpretation?

Your reading of the factorization and implicit planning is accurate, and we will discuss HUME (Xiao et al., RSS 2026) as the closest related work in Sections 2 and 3. In BeliefAct, probabilities are normalized within each question and no joint distribution across questions is formed. The model reads all open questions together when revising one, allowing evidence relevant to one question to inform another. In the Section 5.4 example from DUET-Bench's visitor intent task, where an unseen visitor moves items according to an undisclosed pattern, seeing an already-placed waffle while searching for another item leads the agent to rewrite its answer about the visitor's behavior. BeliefAct also has no explicit lookahead, since each test action and its expected observation is a one-step prediction.

Two further design choices distinguish BeliefAct from a streamlined HUME. HUME expands a hand-written PDDL domain with LLM-generated hypotheses about missing objects, facts, attributes, and action effects, and plans with a classical planner. BeliefAct has no symbolic domain:

  1. The model decides which unknowns to track. It adds questions during execution, with no question types fixed in advance; in 114 of 240 episodes it opens new questions after the first round.
  2. The model specifies what feedback would support each hypothesis. Each answer carries an expected observation written before acting. The next round compares this expectation with the actual observation to revise the belief.

Both choices matter on dynamic tasks: fixed questions raise the rounds from 44.1 to 52.7, and removing only the expected observation raises them to 49.0.

Q2. HUME on DUET-Bench

How does BeliefAct compare against HUME on your benchmark? If performance is similar, does BeliefAct offer significant computational savings?

We implemented HUME-DUET and ran it on all 240 episodes. Because DUET-Bench has no hand-written domain, the LLM writes grounded STRIPS models with hypotheses about objects, attributes, and action effects, and the classical planner Fast Downward (Helmert, 2006) plans over them with astar(hmax()), using HUME's normal, uncertain, and verification action costs. Like every method in Table 1, HUME-DUET runs on Gemini 3.8 Flash and receives the same partial observations. Failed episodes are assigned 200 rounds (SR (%) / Rounds / Cost (m)):

MethodObject search (n=60n=60)Static rule tasks (n=60n=60)Dynamic tasks (n=120n=120)Overall (n=240n=240)
Naive83.3 / 77.1 / 137.276.7 / 86.5 / 275.589.2 / 51.1 / 141.684.6 / 66.5 / 174.0
BSG76.7 / 81.3 / 144.480.0 / 91.6 / 287.192.5 / 48.6 / 138.785.4 / 67.5 / 177.2
PCE83.3 / 76.1 / 149.571.7 / 98.7 / 283.487.5 / 56.0 / 151.182.5 / 71.7 / 183.8
HUME-DUET83.3 / 76.0 / 141.168.3 / 108.5 / 441.570.8 / 79.6 / 134.673.3 / 85.9 / 213.0
BeliefAct (Ours)85.0 / 78.4 / 153.578.3 / 79.1 / 274.691.7 / 44.1 / 125.386.7 / 61.4 / 169.7

Result analysis: HUME-DUET runs HUME's full pipeline of hypothesis generation, verification, and symbolic planning. It is as successful as the baselines on object search, where rooms and containers map directly onto predicates, but solves far fewer rule and dynamic episodes (68.3% and 70.8%, against 78.3% and 91.7% for BeliefAct). In 15 of its 64 failures, no valid model or plan was produced. Unlike HUME's original setting, our DUET adaptation requires the LLM to write goals, operators, and effects for unstated rules and other people's behavior during the episode, and this is where HUME-DUET falls behind. When the unknowns include rules and other agents, keeping the hypotheses in free-form language spares BeliefAct this formalization step.

Response to Reviewer 1K8R (Part 2/3)

Q2. HUME on DUET-Bench (continued)

Overall, BeliefAct solves 86.7% of the episodes, compared with 73.3% for HUME-DUET. BeliefAct also needs less machinery per decision: it makes one LLM call per round and runs no planner, whereas HUME-DUET also has the LLM write the planning model, rewrite it when a plan fails, and run Fast Downward on it. Against the other baselines, whose final success is close to BeliefAct's, the difference lies in planning efficiency. Re-scoring the recorded runs at smaller round budgets H′H', counting an episode as solved only if it succeeded within its first H′H' rounds, gives:

Episodes completed within H′H' rounds (all 240)NaiveBSGPCEBeliefAct
H′=40H' = 4049.6%50.4%45.8%57.1%
H′=60H' = 6067.5%67.1%65.0%75.0%
H′=100H' = 10077.9%77.5%76.2%81.7%
H′=200H' = 200 (Table 1)84.6%85.4%82.5%86.7%

Within 60 rounds, BeliefAct completes 75.0% of episodes, significantly more than Naive (67.5%), BSG (67.1%), and PCE (65.0%).

W2 and Q3. BSG baseline and the QA representation

In the case of the Bayesian scene graph baseline, the paper does not seem to compare to the cited LLM-MCTS method exactly and on the other hand chooses to only include object location in the agent's uncertain belief. Scientifically, this creates a confounder ...

... if you allow the MCTS-style planner and decision-tree-based planner baselines to maintain a belief over any task-relevant hypotheses (as generated by the LLM), what are the advantages and tradeoffs between the free-form QA-based representation and the baselines' graph-based alternatives.

We agree that restricting BSG to locations conflates representation with scope in Table 1. BSG is a simplified Bayesian scene-graph baseline: in the same LLM call that selects the action, the model keeps a probability table over candidate target locations. We cited LLM-MCTS because it also keeps location beliefs on a scene graph, but no MCTS is run, and we will state this in Section 5.2.

At matched scope, letting the model choose and refine its own questions still reduces rounds. The fixed questions ablation replaces the model's questions with task-specific templates that already cover each task's hidden rule, the other actor, and target locations. On dynamic tasks it needs 52.7 rounds instead of 44.1. In DUET-Bench's trigger rule task, where a housemate is set off by an unstated event such as the robot opening a container, it needs 56.3 instead of 35.0, even though the template already asks which event it is. Full BeliefAct instead adds a question the template lacks in 14 of 30 episodes, such as where the destination table is.

We have not built a graph- or tree-based planner over BeliefAct's hypotheses, but two results indicate the tradeoff. On object search, where BSG and BeliefAct track the same unknown, BeliefAct keeps the true room among its hypotheses in 80.6% to 88.4% of rounds before the target is seen, against 37.5% to 50.0% for BSG, which assigns zero probability to unvisited rooms; once the target is found, BSG's automatic scene-graph update keeps its table cleaner.

HUME-DUET, a structured model that can hold hypotheses of any type, does well when the domain is easy to formalize and fails more often when rules and other agents must be formalized online. Graphs and trees thus provide consistency checks and multi-step value backup that BeliefAct lacks, while free-form questions let the agent add a new kind of unknown mid-episode without first extending a schema. We will add this discussion to Section 5.5.

Response to Reviewer 1K8R (Part 3/3)

W3. Scholarship

The "fixed update formula" in the cited Tang et al. (2025) paper, which the paper claims is "not a generalizable solution" ... I went through the paper and was surprised to find that the fixed formula is actually Bayes' rule!

The statement that "existing embodied task planning approaches primarily detect and react to change after it occurs, rather than anticipating it beforehand" is incorrect ...

Thank you for checking this. Our characterization of Tang et al. (2025) was wrong: their update is Bayes' rule, which applies to every setting we study. The practical difficulty in open-world tasks lies in specifying the state abstraction, observation likelihoods, and transition model for rules or actors that were not foreseen, and BeliefAct leaves this specification to the LLM. We will correct the sentence and discuss Bayesian updating with LLM-provided likelihoods, which HUME found to outperform re-querying the LLM's belief, as an alternative design.

Our statement about anticipating change was meant to refer to recent LLM-based planners, and as written it is wrong for the field. POMDP and TAMP work, including that of the Kaelbling, Lozano-Pérez, Hsu, Kochenderfer, and Silver groups, has long modeled latent dynamics and gathered information actively. We will restrict the claim to LLM-based planners, add this broader context, and describe BeliefAct as using an LLM to implement components of a POMDP-style planner.

W4. Notation and the term "rollout"

... LLM is used as a function in Equations 1, 3, and 4 ..., but it has a different "function signature" in each of these cases! ... In addition, the paper uses the term "rollout," suggesting that there is an internal simulation of future actions ...

BeliefAct makes one LLM call per round (Appendix B.1), and we will write all three equations with a single signature. Given the context hth_t, whose message mt−1m_{t-1} contains the carried belief Bt−1\mathcal{B}_{t-1} and the execution records Rt\mathcal{R}_t, the call returns belief revisions, the action, and a new message, and the framework merges the revisions:

(ΔBt,at,mt)=LLM⁡(ht)(\Delta\mathcal{B}_t, a_t, m_t) = \operatorname{LLM}(h_t)

Bt=Merge⁡(Bt−1,ΔBt)\mathcal{B}_t = \operatorname{Merge}(\mathcal{B}_{t-1}, \Delta\mathcal{B}_t).

Equations 3 and 4 described the belief part and the action part of this one response, which is why they read as two competing accounts of the decision. We will replace Equations 1, 3, and 4 with this signature and use "action selection" instead of "rollout" in Section 3.3, since no future actions are simulated.

Q4. External and internal completion

Can you clarify how you assess the completion conditions externally (benchmark "ground truth" completion) and internally?

Externally, DUET-Bench judges success only by deterministic simulator predicates defined for each task (Appendix C), never by an LLM or by the agent's own report. The agent sees only visibility-filtered observations and has no access to hidden rules or event schedules. In each round, the simulator executes the action, adds its cost, applies any scheduled relocation, and then checks completion. In visitor intent, completion is checked before the visitor's next move. In unstable storage, where one of three storage spots removes items while the robot is not looking, completion is checked only when the agent calls declare_done.

Internally, BeliefAct decides from its own observations when the task is complete, and in unstable storage it must signal this with declare_done.

Q5. Cost and horizon in planning

... it is unclear from the exposition how the planner's decisions account for this objective: is the LLM internally engaging in predictive assessments (e.g., via chain-of-thought reasoning) about how many rounds remain and how many steps (and how costly) it may still need to execute?

BeliefAct gives the LLM the information needed for such assessments but does not plan against the objective in Equation 1 explicitly. The prompt lists each primitive's cost (navigation: route metres; door crossing and panoramic scan: 2 m; grasp and place: approach distance plus 3 m; wait: 1 m) and the 200-round horizon. The ego state reports the current round and the cumulative cost, and the scene graph gives distances. Any assessment of the remaining rounds or cost takes place within the LLM's reasoning in that round. We will state this in Section 3.1.

We hope these responses address your concerns, and we thank you again for the detailed and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.