acceptodds
Under review as a conference paper at ICLR 2027

Belief-Centric Reinforcement Learning for LLM Agents in POMDP Tasks

Abstract

Decision making under uncertainty is a central challenge for large language model (LLM) agents (e.g., robotics). Partially Observable Markov Decision Processes (POMDPs) provide a principled framework for modeling decision-making from incomplete observations. In POMDP tasks, agent’s performance depends on two mutually coupled capabilities: how to estimate the state-relevant belief information, and how to disentangle the estimation of failure beliefs and assign the accurate credit to actions. To address this challenge, this paper proposes BeliefRL, a belief-centric RL framework that jointly adapts belief-state estimation and policy learning. BeliefRL first maintains a structured belief state over established, ruled-out, and unresolved information, continually adapts the belief-state model using simulator-grounded supervision. These state-reference discrepancies are further measured as a reliability signal to modulate policy updates. BeliefRL is validated on four partially observable tasks in VirtualHome and Overcooked. Compared with POAD, the strongest overall baseline, BeliefRL improves averaged return by 44.1% and normalized training AUC by 34.8%. Ablations support the complementary benefits of training-time belief adaptation and reliability-aware policy optimization.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language model agents, partially observable Markov decision processes, belief state estimation, credit assignment, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DjNjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BeliefRL for language agents in partially observable environments. A belief state model recursively maintains task-relevant predicates with True, False, and Unknown values. The model is initialized with simulator-grounded supervision and further adapted on policy-generated trajectories. Agreement between predicted and reference beliefs is then used to weight policy updates. Experiments on VirtualHome and Overcooked show improved returns over language-agent RL baselines, with additional evaluations of component contributions, deployment performance, generalization, and inference cost.

Critical strengths

S1. The framework addresses a practical problem with a clear separation between state estimation and action selection. The explicit Unknown value preserves unresolved information, and the training procedure keeps simulator references outside deployment.

S2. Several performance improvements are substantial. Table 5 reports returns of 0.447 versus 0.175 for POAD on Entertainment, and 0.703 versus 0.490 on Tomato Salad, averaged over five runs.

S3. The evaluation includes separate ablations of continued belief adaptation and reliability weighting. Recursive belief evaluation, frozen-policy testing, environment shifts, and measured inference costs provide useful additional evidence.

Critical weaknesses

W1. BeliefRL pretrains its belief model on additional simulator-labelled trajectories. A simpler memory model trained on the same data would provide a useful comparison, helping separate the benefit of the structured belief interface from that of extra supervision.

W2. Reliability weighting changes the strength of the learning signal, but keeps its sign. A useful earlier action can therefore still be penalized when a later belief error causes failure. This specific case needs to be tested directly to support the claim of better credit assignment.

W3. Most main results come from training returns, and frozen-policy testing covers only one task. The reported confidence intervals also compare against the average of five baselines. A stronger evaluation should test fixed policies across all tasks and training seeds, with uncertainty reported against the strongest individual baselines.

Questions for authors

Q1. How does BeliefRL compare with Agent-BRACE or another close belief-learning method under the same backbone and comparable supervision resources?

Q2. Does the agreement score predict belief-induced advantage errors on policy-generated trajectories? Can a controlled experiment show how weighting affects useful earlier actions when a later belief error causes failure?

Q3. How were checkpoints selected for the frozen-policy evaluation, and are the gains consistent across independently trained checkpoints and the other main tasks?

6 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/3)

Thank you for your thoughtful and constructive feedback. Your comments have helped us strengthen the experimental rigor and clarify the scope of BeliefRL's contributions. To address your concerns, we have conducted several additional experiments and analyses: (1) Baseline fairness: We introduce a supervision-matched GRU-Sup baseline using the same labeled trajectories and demonstrate higher training AUC for BeliefRL across all four tasks. We also add a direct comparison with Agent-BRACE, achieving a 14.0% relative AUC improvement on Food Preparation. (2) Credit assignment: We conduct on-policy belief–advantage sensitivity analyses and controlled experiments on delayed belief errors, demonstrating that belief agreement is informative about advantage discrepancies while clarifying the precise role of reliability weighting. (3) Evaluation rigor: We extend frozen-policy evaluation to all four tasks using independently trained checkpoints and report paired-bootstrap confidence intervals against the strongest individual baselines, providing more comprehensive evidence of performance and uncertainty. We believe these additions substantially address your concerns and strengthen the empirical foundation of BeliefRL. We would greatly appreciate your reconsideration of the assessment in light of these new results and clarifications.

W1 & Q1. Baseline Fairness and Comparison with Belief-Learning Methods

Thank you for suggesting a supervision-matched memory baseline and a direct comparison with closely related belief-learning methods. To address your concerns, we have added two targeted comparisons to distinguish the benefits of BeliefRL's belief-centric mechanisms from those of additional supervision.

(1) GRU-Sup under matched supervision (W1). We introduce GRU-Sup, a recurrent-memory baseline using the same LLaMA-3.1-8B backbone and 275 labeled episodes used for BeliefRL's BSM initialization. Its GRU memory is trained to predict the same True/False/Unknown predicate labels and continues to receive supervised updates alongside PPO training. This provides a controlled comparison between explicit belief modeling and recurrent memory under matched supervision resources (Appendix E.4).

We evaluate both methods across five training seeds on all four tasks. The normalized training AUC results (mean ± SD) are:

TaskGRU-SupBeliefRL
Food Preparation0.146 ± 0.0250.435 ± 0.051
Entertainment0.095 ± 0.0350.235 ± 0.069
Tomato Salad0.150 ± 0.0520.382 ± 0.104
Tomato Lettuce Salad0.125 ± 0.0100.320 ± 0.095

BeliefRL achieves higher mean training AUC on all four tasks, despite both methods benefiting from the same labeled supervision source. These findings support the value of BeliefRL's explicit belief-centric learning framework beyond simply providing additional labeled trajectories.

(2) Direct comparison with Agent-BRACE (Q1). We have additionally conducted a direct comparison with Agent-BRACE on Food Preparation (Appendix B.7, Fig. 10 and Table 11), obtaining the following normalized training AUC:

MethodTraining AUC (mean ± SD)
Agent-BRACE0.4133 ± 0.0371
BeliefRL0.4710 ± 0.0311

BeliefRL achieves an absolute AUC gain of +0.0577 (+14.0% relative) in this comparison. Methodologically, Agent-BRACE jointly optimizes belief and policy models through separate reward objectives, whereas BeliefRL uses shared observation-consistent reference beliefs to supervise online BSM adaptation and explicitly weight token-level actor advantages according to belief reliability.

Together, these experiments provide direct empirical evidence against both a supervision-matched recurrent baseline and a closely related belief-learning method, strengthening the experimental support for BeliefRL's belief-centric training approach.

AuthorsOpenReview ↗

Rebuttal (Part 2/3)

W2 & Q2. Reliability Weighting and Belief-Induced Credit Assignment Errors

Thank you for highlighting the distinction between modulating actor updates and correcting erroneous advantage signs. We have clarified the mechanism and added two targeted empirical analyses to examine belief-induced advantage errors and their propagation to earlier actions.

(1) Does belief agreement predict advantage errors? (Q2) We conducted an on-policy belief–advantage sensitivity analysis using 100 episodes per task across five training seeds (Appendix B.2). Holding the actor, BSM, critic, actions, and rewards fixed, we recomputed token-level advantages under predicted versus reference beliefs. The Spearman correlations between predicate agreement and advantage-discrepancy magnitude are:

TaskSpearman ρ\rho (q<1q<1)95% CI
Food Preparation-0.313[-0.407, -0.193]
Entertainment-0.403[-0.475, -0.317]
Tomato Salad-0.164[-0.226, -0.093]
Tomato Lettuce Salad-0.176[-0.199, -0.151]

All 95% CIs exclude zero, demonstrating a consistent negative association: higher belief agreement corresponds to smaller belief-induced advantage discrepancies. This provides empirical evidence that predicate agreement serves as an informative reliability proxy for policy optimization.

(2) Controlled analysis of delayed credit-assignment errors. To directly examine whether later belief errors can penalize useful earlier actions, we analyzed naturally occurring belief discrepancies and controlled future critic-value replacements on Food Preparation (Appendix B.3). We measured cases where an earlier action token has a positive reference-conditioned advantage but a negative advantage under erroneous belief conditioning:

ConditionSign inversionsRate [95% CI]
Natural belief discrepancies86/2,2093.89% [2.78, 7.50]
Controlled critic-tail replacement21/3066.86% [3.95, 10.25]

Furthermore, 10 matched environment-replay cases showed positive discounted continuation margins for the selected earlier actions relative to alternatives under reference beliefs. These findings illustrate how subsequent belief-dependent value errors can propagate backward and distort credit assignment.

(3) Role of reliability weighting. We agree that positive reliability weights do not reverse advantage signs. Instead, BeliefRL adjusts the relative contribution of actor updates according to belief agreement, assigning lower relative weight to less reliable decisions. The controlled experiment characterizes advantage sensitivity rather than demonstrating policy-level correction of sign inversions. Our component ablation (Sec. 4.3) separately evaluates the learning contribution of reliability weighting. Thus, our claim is reliability-aware modulation of policy updates, rather than guaranteed correction of every delayed credit-assignment error.

AuthorsOpenReview ↗

Rebuttal (Part 3/3)

W3 & Q3. Frozen-Policy Evaluation, Checkpoint Selection, and Statistical Comparisons

Thank you for requesting broader frozen-policy evaluation, clearer checkpoint selection, and uncertainty estimates against the strongest individual baselines. We have substantially strengthened the evaluation to address these concerns.

(1) Frozen-policy evaluation across all four tasks (Q3). We extended frozen-policy evaluation from one task to all four benchmarks, using three independently trained final checkpoints per method, with greedy action selection on 30 shared held-out environment seeds per checkpoint. Both the policy and BSM (where applicable) remain frozen during evaluation. The resulting frozen-policy returns (mean ± SD) are:

MethodFood Prep.EntertainmentTomato SaladTomato Lettuce Salad
GLAM0.164 ± 0.2850.000 ± 0.0000.150 ± 0.0100.467 ± 0.318
TWOSOME0.734 ± 0.0000.000 ± 0.0000.391 ± 0.4090.288 ± 0.000
ARCHER0.217 ± 0.1880.000 ± 0.0000.098 ± 0.0990.192 ± 0.084
NTPO0.535 ± 0.2040.000 ± 0.0000.391 ± 0.4090.208 ± 0.076
POAD0.535 ± 0.0000.000 ± 0.0000.391 ± 0.4090.402 ± 0.354
BeliefRL0.756 ± 0.0000.192 ± 0.3320.627 ± 0.4090.608 ± 0.392

(2) Uncertainty against the strongest individual baseline. We further strengthened the statistical analysis by comparing BeliefRL against the strongest individual baseline on each task, rather than only the average of five baselines. Using five matched training seeds, we report paired-bootstrap 95% confidence intervals for final-window training returns (Appendix B.1, Table 4):

TaskStrongest baselineΔ Return [95% CI]
Food PreparationTWOSOME+0.078 [0.021, 0.151]
EntertainmentPOAD+0.273 [0.100, 0.408]
Tomato SaladPOAD+0.213 [0.102, 0.309]
Tomato Lettuce SaladGLAM+0.063 [-0.008, 0.196]

Together, these additional evaluations provide stronger evidence of BeliefRL's performance across independently trained checkpoints, all four tasks, and held-out environments.

 

Thanks again for your valuable suggestions. We have made brief revisions to the manuscript accordingly and will continue to work to incorporate the above discussions and improvements into the final version. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and looking forward to further discussions with you.

Best regards,

All Authors

Reviewer_ZvJSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BeliefRL, a reinforcement learning framework for LLM agents operating in partially observable environments. The method combines an explicit Belief State Model (BSM), which recursively maintains structured beliefs using True, False, and Unknown predicates, with belief-conditioned policy optimization. The BSM is initialized through supervised fine-tuning and subsequently adapted during RL training using simulator-grounded reference beliefs. The agreement between predicted and reference beliefs is also used as a reliability signal to weight policy advantages during optimization. The framework is evaluated on four partially observable tasks from VirtualHome and Overcooked, against several LLM-based RL baselines. The reported results show that BeliefRL achieves the highest mean final-window returns and normalized training AUC across the four tasks. The paper also presents component ablations and additional experiments investigating belief estimation accuracy, robustness, generalization, and language ability retention.

Critical strengths
  • Relevance to the conference: the problem tackled by this paper is relevant to the ICLR community.
  • Explicit and interpretable belief representation: The paper tackles partial observability in LLM-agent RL with a structured belief state (True/False/Unknown predicates) that separates ruled-out from unresolved information. The ablation shows that removing the BSM lowers training AUC on all four tasks, and the OOD experiments suggest that the explicit belief interface contributes to robustness. Using belief reliability to modulate advantages is also an interesting idea, even if its empirical application is less clear.
  • Consistent gains over relevant baselines, with additional experimental analyses: BeliefRL achieves higher mean final-window returns and training AUC than the evaluated baselines across four tasks in two environments. The authors also report uncertainty estimates, deployment costs, language-ability retention, and results on an additional harder task, providing useful complementary evidence about the method's behavior.
Critical weaknesses

• Novelty/originality: The introduction does not highlight the originality of the work. It mentions two contributions (see the last paragraph of the introduction) but their originality appears weak. Explicit, recursively updated belief-state models for LLM agents, with structured claims and uncertainty labels, have already been explored in Agent-BRACE (Singh et al., 2026), which also jointly optimizes the belief model and policy through RL. ReBel (Tang et al., 2026a) likewise uses structured beliefs and belief-consistency signals to guide policy learning and advantage estimation. BeliefRL’s potentially distinctive element is its particular combination of simulator-grounded BSM supervision and reference-agreement-based advantage weighting; the paper should distinguish this more clearly from prior work and provide direct comparisons. As presented, these choices seem incremental and do not support a strong originality claim. I notice also that the related work section is just a list citations with very short explanation of the related methodologies. In my opinion, this section should provide a clear analysis of the differences between the proposed contribution and the 2/3 most similar works in the literature. • Reproducibility: The appendices provide useful implementation details, including training budgets, model configuration, and data-collection procedures. However, I did not find a code or dataset release, and some details needed to reproduce the results, such as prompts and fuller baseline configurations, are not specified. The paper should provide these artifacts or enough detail to reconstruct them. I notice that the paper does not provide the full prompt template, concrete per-task predicate schemas, or a worked input/output example of the actual BSM prompt. Since these choices can materially affect belief predictions - and thus the policy inputs and reported results - this as a meaningful reproducibility and methodology issue. The notation also shifts from Prompt in Eq. 3 to \Pi in Appendix D.3 without explicitly stating they are the same template. • Theoretical analysis: The theoretical analysis is not explained from an high level viewpoint. I needed som etime to understand what the authors actually want to prove and why. This should be instead clearly explained before reporting the theoretical definitions. The theoretical analysis does not provide a guarantee on BSM accuracy or BeliefRL’s task performance. Theorem 1 bounds only a specified belief-induced component of the actor-gradient discrepancy, and only under a conditional calibration assumption linking predicate-set agreement to gradient error; this relationship is neither derived nor empirically validated. The propositions on belief stagnation and negative-tail GAE are likewise conditional analyses that are not connected to the behavior of the prompted LLM BSM in Eq. 3. Thus, the theory motivates the proposed weighting scheme but does not establish that the method’s belief updates are reliable or that its policy updates improve performance. This point should be made clearer in the paper. Eq. 3 basically computes the new belief by "trusting" an LLM provided by "some" prompt. But there is no guarantee the beliefs provided in this way are correct. The method trusts the LLM fine-tuned on the simulator. This is a key point of the paper which is never made explicit in my opinion. • Experimental evaluation quality: The main baselines include recent methods such as POAD, TWOSOME, and NTPO, which is a strength. Still, the evaluation lacks a recurrent-memory baseline and a fully observable or oracle-state reference that could help interpret the remaining performance gap. Also, the final-window return gap against the aggregate baseline on Tomato Lettuce Salad has a 95% interval that crosses zero (Table 1), so the claim of consistent improvement should be qualified. The BSM accuracy test uses 20 episodes with uniformly sampled actions; reporting accuracy on policy-generated rollouts would better show whether the estimator works on the histories that matter for control. • Dependence on privileged simulator information: BeliefRL relies on simulator-grounded reference beliefs for offline BSM initialization, online BSM adaptation and the computation of the reliability score that weights policy advantages. This requires access to the underlying simulator state and a task-specific reference construction, which limits applicability to POMDP settings where such information is unavailable during training. Moreover, the experiments do not establish a comparison under equivalent privileged supervision, so it is difficult to disentangle the benefit of the proposed belief-centric mechanisms from that of the additional privileged signal. A controlled comparison under comparable supervision would help clarify this. • Lack of a self-contained methodology: Essential components of the framework are described only in the appendices, in particular the action selection mechanism, the construction of the reference beliefs and the offline dataset. Moreover, some elements, such as the critic’s parametrization and how the reference mapping handles observability constraints, remain insufficiently specified even there. This makes the method hard to follow and to reproduce, and the main paper should provide a more complete and coherent description of the framework. • Limited empirical support for key claims: The main comparisons rely on returns collected during training rather than on independent evaluation of the final policies, and the frozen-policy evaluation covers only one of the four main tasks. In addition, the ablations show only marginal improvements from reliability weighting on some tasks, and no direct statistical comparison or interaction analysis is provided to establish the robustness and complementarity of the two proposed mechanisms. Stronger evidence would require frozen-policy evaluation on all tasks, across multiple training seeds, and a more rigorous analysis of the ablations.

Questions for authors
  1. Reference belief construction: Could the authors clarify how simulator states and observations are used to assign True, False, and Unknown values to individual predicates? How is privileged information prevented from revealing facts that should remain unknown under partial observability? 2. Offline initialization and supervision: Have the authors evaluated BeliefRL without the initial supervised fine-tuning, while retaining online BSM adaptation and reliability-weighted optimization? This would clarify how strongly the method depends on offline belief supervision. 3. Comparison with belief-tracking baselines: Agent-BRACE is discussed in the Related Work but not included among the baselines. Could the authors explain this choice and clarify the main methodological differences between Agent-BRACE and BeliefRL? Additionally, the w/o BSM ablation in Figure 6 reports exactly the same AUC values as POAD in Table 5 across all four tasks. Is BeliefRL w/o BSM equivalent to POAD? 4. Frozen-policy evaluation: The frozen-checkpoint evaluation is reported only for Tomato Lettuce Salad. Could the authors report the same evaluation for the other three main tasks and specify how many independently trained checkpoints were evaluated? 5. BSM accuracy on policy trajectories: Belief estimation accuracy is measured on trajectories generated by uniformly sampled actions. Have the authors also evaluated the BSM on trajectories generated by the final learned policy? 6. Reproducibility: Do the authors plan to release the implementation, together with the POMDP adaptations and the configurations needed to reproduce the results? If so, when is the release planned? 7. Please elaborate on the weaknesses.
8 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/5)

We thank the reviewer for the thoughtful and constructive feedback! To address your concerns, we have substantially strengthened the experimental evaluation and clarified the methodological contributions of BeliefRL. Specifically, (1) we clarify our novelty relative to Agent-BRACE and ReBel, emphasizing the combination of online belief-state adaptation and reliability-aware policy optimization; (2) we provide more comprehensive methodological and implementation details, including observation-consistent reference belief construction, prompt templates, and baseline configurations; (3) we introduce GRU-memory and fully observable (FOMDP) baselines, extend frozen-policy evaluation to all four tasks using independently trained checkpoints, and demonstrate 95–100% belief estimation accuracy on learned-policy trajectories; (4) we add a no-offline-SFT ablation and statistical analyses of component contributions, further examining the roles of belief adaptation and reliability weighting; and (5) we clarify the assumptions and limitations of our theoretical analysis, supported by additional empirical investigations. We believe these revisions substantially address your concerns and strengthen the evidence supporting BeliefRL. We would appreciate your reconsideration of the assessment in light of these new results and clarifications.

C1 & Q3. Novelty and Originality

We agree with the reviewer's observation that BeliefRL's distinctive contribution lies in the specific combination of BSM supervision and reference-agreement-based advantage weighting. Our contribution is not the three-valued predicate schema itself, but how belief estimation and actor optimization are coupled through observation-consistent reference beliefs during RL training. Specifically, BeliefRL (i) continually adapts the BSM on policy-generated trajectories using reference-belief supervision (Eq. 10), and (ii) reweights token-level actor advantages according to the agreement between predicted and reference predicate tuples (Eqs. 11–15), with alternating BSM/PPO updates (Eq. 16). Section 1 now states this distinction explicitly, and Section 5 expands the comparison with the closest methods.

Comparison with Agent-BRACE and ReBel. Recent research has focused on different aspects of our approach. Agent-BRACE emphasizes uncertainty-aware belief representation and jointly trains belief and policy models through PPO. However, its belief learning relies on multiple heterogeneous reward components, introducing complexity in reward design and optimization. ReBel approaches belief modeling from a credit-assignment perspective, using belief–observation consistency rewards and belief-anchored advantage estimation. However, its reliance on agent-predicted belief anchors makes trajectory comparisons potentially sensitive to belief estimation errors. In contrast, BeliefRL investigates reliability-guided belief–policy co-adaptation. It uses shared observation-consistent reference beliefs to supervise continual BSM adaptation and assess belief reliability for weighting token-level actor advantages. This directly connects belief estimation quality with policy optimization, distinguishing BeliefRL from representation-oriented and credit-assignment-oriented approaches.

Why no Agent-BRACE baseline? Our evaluated baselines share the same task interfaces and Llama-3.1-8B backbone, enabling controlled comparisons of the implemented RL optimizers (Appendix E.4). We agree that a direct and carefully aligned comparison with Agent-BRACE would be valuable. Following your suggestion, we have completed a direct comparison with Agent-BRACE on Food Preparation (Appendix B.7, Fig. 10 and Table 11). BeliefRL achieves a normalized training AUC of 0.4710 ± 0.0311, compared with 0.4133 ± 0.0371 for Agent-BRACE, representing an absolute improvement of +14.0% relative. These results provide additional empirical evidence of BeliefRL's effectiveness compared with a closely related belief-learning approach, complementing our original baseline evaluations.

Is w/o BSM equivalent to POAD? Yes, in the evaluated configuration. BeliefRL uses POAD's token-level PPO optimizer; removing the BSM and belief-dependent mechanisms recovers its observation-only configuration. Thus the identical AUC values in the Table 1 and Table 3 (original Figure 6 and Table 5) are expected, not independent replication. The observed gains over this control reflect the added belief-centric framework; the separate ablations in revised Table 1 further examine online BSM adaptation and reliability weighting.

AuthorsOpenReview ↗

Rebuttal (Part 2/5)

C2 & Q6. Reproducibility

We appreciate the reviewer's concerns regarding reproducibility, particularly the missing BSM prompt templates, predicate schemas, worked examples, and baseline configurations. We have substantially expanded the revised manuscript to address these issues.

(1) BSM prompts and predicate schemas. Appendix D.3 now provides the complete domain-specific BSM prompt templates for VirtualHome and Overcooked, including input fields, belief-update instructions, output constraints, and predicate formats. We also include a recorded BSM input/output example from Food Preparation to illustrate recursive belief updates. To resolve the notation ambiguity, revised Eq. (23) explicitly defines Prompt as the serialization function used in Eq. (3), replacing the previous Π\Pi notation.

(2) Implementation and baseline configurations. We have expanded Appendix E.4 with detailed configurations for all five baselines (GLAM, TWOSOME, ARCHER, NTPO, and POAD). Tables 18–23 document their policy inputs, action-scoring procedures, optimization methods, hyperparameters, training schedules, and interaction budgets. Appendix D.2 further details the offline data-collection procedure, including random exploration, MCTS-guided trajectories, and collection budgets (Table 14).

(3) Code and artifact release (Q6). We agree that publicly releasing the implementation and experimental artifacts is important for reproducibility. We plan to release the BeliefRL implementation and POMDP environments upon publication, including task configurations, observation rules, predicate schemas, reference annotation procedures, training scripts, and evaluation settings. Together with the expanded documentation in Appendices D and E, these artifacts will facilitate independent reproduction of our experiments.

C3. Theoretical Analysis

We thank the reviewer for highlighting the need for a clearer theoretical motivation and a closer connection between the analysis and the implemented BSM. We have substantially reorganized Appendix G to explain the underlying questions before introducing the formal assumptions and results.

(1) Motivation and theoretical interpretation. Our analysis addresses two connected challenges in belief-conditioned RL: belief-update errors can accumulate during recursive estimation, and subsequent belief-dependent value errors can influence the advantages assigned to earlier actions. Propositions 1–2 (Appendices G.2–G.3) characterize these mechanisms under explicit conditions, motivating continued BSM adaptation. Theorem 1 (Appendix G.4, Eq. 50) analyzes reliability weighting: under the stated calibration condition E[∥ζϵA∥∣q]≤C(1−q)\mathbb{E}[\|\zeta\epsilon_A\|\mid q]\leq C(1-q), a positive, nondecreasing agreement-based weight w(q)w(q) yields

\leq C\frac{\mathbb{E}[w(q)(1-q)]}{\mathbb{E}[w(q)]} \leq C\mathbb{E}[1-q].$$ This provides a **conditional upper-bound rationale** for assigning greater weight to actor updates supported by more reliable beliefs. Appendix G now clearly states the assumptions and the precise scope of the bound. **(2) Connection to the implemented BSM.** Equation (3) defines a learned, recursive LLM-based belief updater. Rather than relying on the prompt alone, BeliefRL **initializes the BSM with supervised belief transitions and continually adapts it on policy-generated trajectories** using observation-consistent reference beliefs (Sec. 3.1–3.2). Predicate agreement then supplies an explicit, measurable reliability signal for policy optimization (Eqs. 11–15). This connects the theoretical quantities to the implemented training mechanism. **(3) New empirical evidence for the reliability signal.** We added an **on-policy belief–advantage sensitivity analysis** using 100 episodes per task across five training seeds (Appendix B.2, Table 5). The table below reports Spearman correlations between predicate agreement and belief-conditioned advantage sensitivity, excluding perfect-agreement samples ($q=1$): | Task | Spearman $\rho$ | 95% CI | |---|---:|---:| | Food Preparation | -0.313 | [-0.407, -0.193] | | Entertainment | -0.403 | [-0.475, -0.317] | | Tomato Salad | -0.164 | [-0.226, -0.093] | | Tomato Lettuce Salad | -0.176 | [-0.199, -0.151] | **All four intervals exclude zero**, indicating a consistent negative association: higher predicate agreement tends to correspond to smaller belief-conditioned advantage discrepancies.
AuthorsOpenReview ↗

Rebuttal (Part 3/5)

C4 & Q5. Experimental Evaluation Quality

We appreciate your suggestions for strengthening the experimental evaluation, particularly regarding recurrent-memory baselines, full-observation references, and belief estimation on policy-generated trajectories. We have added the following experiments and statistical analyses.

(1) Additional GRU and FOMDP baselines. We added a GRU-based recurrent-memory baseline evaluated over five training seeds on all four tasks (Sec. 4.5, Fig. 7). The continuous-prefix training-AUC comparison is:

TaskGRU-Sup (continuous prefix)BeliefRLΔ AUC
Food Preparation0.146 ± 0.0250.435 ± 0.051+0.288
Entertainment0.095 ± 0.0350.235 ± 0.069+0.140
Tomato Salad0.150 ± 0.0520.382 ± 0.104+0.232
Tomato Lettuce Salad0.125 ± 0.0100.320 ± 0.095+0.194

BeliefRL achieves higher mean AUC across all four tasks, supporting the value of its explicit belief interface relative to recurrent memory in these settings. We also added a full-observation (FOMDP) reference, implemented as POAD conditioned on the oracle task state (Sec. 4.6, Fig. 8). It performs strongly on Food Preparation and Tomato Salad, while BeliefRL achieves higher observed training returns on Entertainment and Tomato Lettuce Salad. This compares learned policies under different information inputs; oracle-state access does not imply an oracle-optimal policy.

(2) BSM accuracy on learned-policy trajectories (Q5). We now evaluate the final BSM on trajectories generated by the trained BeliefRL policy, rather than only uniformly sampled actions (Appendix B.9, Table 13). Exact-match accuracy against observation-consistent reference beliefs is 97.53% (Food Preparation), 100.00% (Entertainment), 95.00% (Tomato Salad), and 100.00% (Tomato Lettuce Salad). This directly measures belief estimation on histories encountered by the learned policy.

(3) Statistical evaluation of performance gains. We have refined the statistical comparisons in Appendix B.1, Table 4, reporting paired-bootstrap 95% confidence intervals over five matched training seeds against the strongest baseline for each task. BeliefRL achieves higher observed mean final-window returns across all four tasks, with confidence intervals excluding zero on Food Preparation, Entertainment, and Tomato Salad. For Tomato Lettuce Salad, the positive estimated difference (+0.063) has a confidence interval [-0.008, 0.196], so we distinguish the observed improvement from a statistically established advantage. These analyses provide a more precise assessment of the reported performance gains.

C5 & Q1. Privileged Simulator Information and Reference Belief Construction

We apologize that our original description of simulator-grounded reference beliefs may have suggested that their construction requires access to the underlying hidden environment state. We would like to clarify that reference beliefs are constructed from ordinary environment observations rather than privileged knowledge of hidden states. We have revised Sec. 3.1 (Eq. 4) and Appendix D.1 to make this distinction explicit.

(1) Observation-consistent reference construction (Q1). Reference beliefs are generated recursively through

eˉt=Gref(eˉt−1,at−1,ot,g),\bar e_t=G_{\mathrm{ref}}(\bar e_{t-1},a_{t-1},o_t,g),

using the previous reference belief, previous action, current observation, and task goal, without requiring access to the complete latent environment state. Predicates are assigned True when supported by observable evidence or valid transitions, False when observable evidence rules them out, and Unknown otherwise. Appendix D.1 further details the visibility constraints and predicate-update rules for both environments.

Importantly, reference beliefs serve as supervision targets for learning belief updates, rather than oracle states revealing hidden information. Therefore, BeliefRL does not inherently require privileged simulator-state access and is not restricted to environments where the underlying hidden state is available. Reference supervision can be constructed from observable interaction histories and task-specific annotation rules. At deployment, the BSM and policy operate without reference annotations.

(2) Fair comparison under matched supervision. To address your concern that the improvements might arise primarily from additional supervision, we additionally evaluate GRU under matched reference-belief supervision, allowing us to compare explicit belief modeling with recurrent memory under the same supervision source. As reported in C4, BeliefRL achieves higher normalized training AUC across all four tasks. Furthermore, our controlled ablations provide additional evidence that BeliefRL's improvements reflect its belief-centric learning mechanisms rather than merely access to additional supervision.

Reviewer_WcP7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents the use of an LLM to infer a belief representation in partially observable tasks. This LLM-encoded belief is then fed to an RL algorithm to solve the task. The benchmarking problems are image-based, such as search of an item in different rooms.

Critical strengths

The paper presents a nice and useful application of LLMs in the POMDP/RL context.

Critical weaknesses

Besides being a neat application of LLMs, I do not think the paper presents any real methodological innovation, nor sufficient mathematical grounding of the belief-based method presented. From my understanding, the LLM-based framework as presented in the paper is also only applicable to a quite limited class of POMDP problems.

Questions for authors

What is exactly the methodological innovation? Which other types of problems could this framework solve, besides "image-based search of items" and similar?

8 comments

Thank you for your feedback. In light of the reviewers' comments, we recognize that our original presentation did not fully convey the key contributions of our method. We have therefore strengthened the articulation of our algorithmic contributions, clarified their distinction from the LLM-based implementation, and refined the discussion of our theoretical results and their scope of applicability. We have also incorporated additional targeted evidence to further substantiate these clarifications.

(1) What is the methodological innovation? We do not claim novelty for symbolic beliefs, three-valued predicates, or PPO individually. BeliefRL's contribution is a coupled training mechanism: (i) the belief-state model (BSM) is continually supervised on histories generated by the evolving policy, rather than remaining a fixed pretrained memory; and (ii) the same observation-consistent reference beliefs provide predicate-agreement scores that weight token-level actor advantages. The first adapts belief estimation to the evolving policy; the second makes actor updates sensitive to belief reliability. Alternating updates couple belief learning and policy optimization, without backpropagating through belief generation or using reference labels at deployment. The distinction from prior belief-based agents lies in this training mechanism, not merely the memory format (Sec. 3). In the component ablations (Table 1), removing either online BSM adaptation or reliability weighting reduces normalized training AUC on all four tasks. Our new GRU-Sup baseline is initialized on the same 275 labeled episodes and receives online supervised memory updates; BeliefRL has higher mean training AUC across all four tasks. This supports gains beyond access to extra labeled data.

(2) What is the mathematical basis? Appendix G now separates conditional failure-mode analyses from the reliability-weighting result. Predicted and reference belief tuples define an agreement score q∈[0,1]q\in[0,1]; a bounded, nondecreasing weight w(q)w(q) adjusts token-level actor advantages. With A^=A∗+ϵA\hat A=A^*+\epsilon_A and token-score gradient ζ\zeta, Theorem 1 assumes the calibration condition E[∥ζϵA∥∣q]≤C(1−q)\mathbb E[\|\zeta\epsilon_A\|\mid q]\le C(1-q) and establishes

\le C\frac{\mathbb E[w(q)(1-q)]}{\mathbb E[w(q)]} \le C\mathbb E[1-q].$$ Under this assumption, the weighted **upper bound on the belief-induced component of actor-gradient discrepancy** is no larger than its unweighted counterpart. It does not establish exact filtering, unconditional convergence, or higher return. We also examine the premise empirically: on-policy belief agreement correlates negatively with advantage-discrepancy magnitude across all four tasks (Spearman $\rho=-0.313,-0.403,-0.164,-0.176$ after excluding perfect matches; Appendix B.2). These associations support the reliability proxy. **(3) Which POMDPs beyond object search?** Our evaluation is **not limited to image-based search**: the agents act on text or structured partial observations. VirtualHome includes multi-step manipulation and simultaneous goals; Overcooked requires tracking ingredient preparation, containment, and multi-step dependencies under restricted visibility. More importantly, we added a **non-embodied active root-cause diagnosis experiment (RCA-100)** with Qwen3-14B, five seeds, 50 training episodes per seed, and matched interaction/PPO budgets (Appendix B.6, Table 10). More broadly, BeliefRL targets **sequential decisions with partial observations, persistent task-relevant uncertainty, and reference beliefs constructible from observable histories**—including diagnostic and troubleshooting agents. We hope this clarifies **what is methodologically new, what is mathematically justified, and where the present evidence supports transfer beyond the original tasks**. &nbsp; Thanks again for your valuable suggestions. We have made brief revisions to the manuscript accordingly and will continue to work to incorporate the above discussions and improvements into the final version. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and looking forward to further discussions with you. Best regards, All Authors

Thank you again for your valuable feedback. We would like to kindly follow up on our rebuttal, particularly the clarifications of BeliefRL's methodological contributions and theoretical foundations, as well as the new RCA-100 experiment demonstrating applicability beyond embodied object-search tasks.

We would greatly appreciate it if you could review these additions and let us know whether they address your main concerns. We would be happy to provide any further clarification during the discussion.

I thank the authors for their explanations. However, I remain of the view that the methodological novelty is indeed limited and that the contribution lies primarily in the application of existing methods rather than in new methodology. I also could not find a detailed description of the new added experiments (as, e.g., section C).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Request for Further Discussion

Dear Area Chair,

Thank you for coordinating the review of our paper and for your time and consideration throughout the review process.

We have carefully addressed the reviewers' concerns through detailed point-by-point responses, additional experiments, and substantial revisions to the manuscript. In particular, we have clarified BeliefRL's methodological novelty and theoretical foundations, expanded our experimental evaluation, and added further comparisons and analyses to strengthen the empirical validation of our proposed method.

We sincerely appreciate Reviewer DjNj's positive response and raise rating, as well as Reviewer ZvJS's acknowledgment that our explanations would be considered. We have also provided further response to Reviewer WcP7's remaining concerns. As the author discussion period approaches its conclusion, we would be grateful if you could help facilitate any remaining discussion, at your discretion, so that we have an opportunity to address outstanding questions. We sincerely hope that our revisions and additional evidence will be taken into account in the final assessment of our work. We remain available to provide any further clarification.

Thank you again for your time and thoughtful consideration.

Best regards,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.