- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BeliefRL for language agents in partially observable environments. A belief state model recursively maintains task-relevant predicates with True, False, and Unknown values. The model is initialized with simulator-grounded supervision and further adapted on policy-generated trajectories. Agreement between predicted and reference beliefs is then used to weight policy updates. Experiments on VirtualHome and Overcooked show improved returns over language-agent RL baselines, with additional evaluations of component contributions, deployment performance, generalization, and inference cost.
Critical strengths
S1. The framework addresses a practical problem with a clear separation between state estimation and action selection. The explicit Unknown value preserves unresolved information, and the training procedure keeps simulator references outside deployment.
S2. Several performance improvements are substantial. Table 5 reports returns of 0.447 versus 0.175 for POAD on Entertainment, and 0.703 versus 0.490 on Tomato Salad, averaged over five runs.
S3. The evaluation includes separate ablations of continued belief adaptation and reliability weighting. Recursive belief evaluation, frozen-policy testing, environment shifts, and measured inference costs provide useful additional evidence.
Critical weaknesses
W1. BeliefRL pretrains its belief model on additional simulator-labelled trajectories. A simpler memory model trained on the same data would provide a useful comparison, helping separate the benefit of the structured belief interface from that of extra supervision.
W2. Reliability weighting changes the strength of the learning signal, but keeps its sign. A useful earlier action can therefore still be penalized when a later belief error causes failure. This specific case needs to be tested directly to support the claim of better credit assignment.
W3. Most main results come from training returns, and frozen-policy testing covers only one task. The reported confidence intervals also compare against the average of five baselines. A stronger evaluation should test fixed policies across all tasks and training seeds, with uncertainty reported against the strongest individual baselines.
Questions for authors
Q1. How does BeliefRL compare with Agent-BRACE or another close belief-learning method under the same backbone and comparable supervision resources?
Q2. Does the agreement score predict belief-induced advantage errors on policy-generated trajectories? Can a controlled experiment show how weighting affects useful earlier actions when a later belief error causes failure?
Q3. How were checkpoints selected for the frozen-policy evaluation, and are the gains consistent across independently trained checkpoints and the other main tasks?
6 comments
Rebuttal (Part 2/3)
W2 & Q2. Reliability Weighting and Belief-Induced Credit Assignment Errors
Thank you for highlighting the distinction between modulating actor updates and correcting erroneous advantage signs. We have clarified the mechanism and added two targeted empirical analyses to examine belief-induced advantage errors and their propagation to earlier actions.
(1) Does belief agreement predict advantage errors? (Q2) We conducted an on-policy belief–advantage sensitivity analysis using 100 episodes per task across five training seeds (Appendix B.2). Holding the actor, BSM, critic, actions, and rewards fixed, we recomputed token-level advantages under predicted versus reference beliefs. The Spearman correlations between predicate agreement and advantage-discrepancy magnitude are:
| Task | Spearman () | 95% CI |
|---|---|---|
| Food Preparation | -0.313 | [-0.407, -0.193] |
| Entertainment | -0.403 | [-0.475, -0.317] |
| Tomato Salad | -0.164 | [-0.226, -0.093] |
| Tomato Lettuce Salad | -0.176 | [-0.199, -0.151] |
All 95% CIs exclude zero, demonstrating a consistent negative association: higher belief agreement corresponds to smaller belief-induced advantage discrepancies. This provides empirical evidence that predicate agreement serves as an informative reliability proxy for policy optimization.
(2) Controlled analysis of delayed credit-assignment errors. To directly examine whether later belief errors can penalize useful earlier actions, we analyzed naturally occurring belief discrepancies and controlled future critic-value replacements on Food Preparation (Appendix B.3). We measured cases where an earlier action token has a positive reference-conditioned advantage but a negative advantage under erroneous belief conditioning:
| Condition | Sign inversions | Rate [95% CI] |
|---|---|---|
| Natural belief discrepancies | 86/2,209 | 3.89% [2.78, 7.50] |
| Controlled critic-tail replacement | 21/306 | 6.86% [3.95, 10.25] |
Furthermore, 10 matched environment-replay cases showed positive discounted continuation margins for the selected earlier actions relative to alternatives under reference beliefs. These findings illustrate how subsequent belief-dependent value errors can propagate backward and distort credit assignment.
(3) Role of reliability weighting. We agree that positive reliability weights do not reverse advantage signs. Instead, BeliefRL adjusts the relative contribution of actor updates according to belief agreement, assigning lower relative weight to less reliable decisions. The controlled experiment characterizes advantage sensitivity rather than demonstrating policy-level correction of sign inversions. Our component ablation (Sec. 4.3) separately evaluates the learning contribution of reliability weighting. Thus, our claim is reliability-aware modulation of policy updates, rather than guaranteed correction of every delayed credit-assignment error.
Rebuttal (Part 3/3)
W3 & Q3. Frozen-Policy Evaluation, Checkpoint Selection, and Statistical Comparisons
Thank you for requesting broader frozen-policy evaluation, clearer checkpoint selection, and uncertainty estimates against the strongest individual baselines. We have substantially strengthened the evaluation to address these concerns.
(1) Frozen-policy evaluation across all four tasks (Q3). We extended frozen-policy evaluation from one task to all four benchmarks, using three independently trained final checkpoints per method, with greedy action selection on 30 shared held-out environment seeds per checkpoint. Both the policy and BSM (where applicable) remain frozen during evaluation. The resulting frozen-policy returns (mean ± SD) are:
| Method | Food Prep. | Entertainment | Tomato Salad | Tomato Lettuce Salad |
|---|---|---|---|---|
| GLAM | 0.164 ± 0.285 | 0.000 ± 0.000 | 0.150 ± 0.010 | 0.467 ± 0.318 |
| TWOSOME | 0.734 ± 0.000 | 0.000 ± 0.000 | 0.391 ± 0.409 | 0.288 ± 0.000 |
| ARCHER | 0.217 ± 0.188 | 0.000 ± 0.000 | 0.098 ± 0.099 | 0.192 ± 0.084 |
| NTPO | 0.535 ± 0.204 | 0.000 ± 0.000 | 0.391 ± 0.409 | 0.208 ± 0.076 |
| POAD | 0.535 ± 0.000 | 0.000 ± 0.000 | 0.391 ± 0.409 | 0.402 ± 0.354 |
| BeliefRL | 0.756 ± 0.000 | 0.192 ± 0.332 | 0.627 ± 0.409 | 0.608 ± 0.392 |
(2) Uncertainty against the strongest individual baseline. We further strengthened the statistical analysis by comparing BeliefRL against the strongest individual baseline on each task, rather than only the average of five baselines. Using five matched training seeds, we report paired-bootstrap 95% confidence intervals for final-window training returns (Appendix B.1, Table 4):
| Task | Strongest baseline | Δ Return [95% CI] |
|---|---|---|
| Food Preparation | TWOSOME | +0.078 [0.021, 0.151] |
| Entertainment | POAD | +0.273 [0.100, 0.408] |
| Tomato Salad | POAD | +0.213 [0.102, 0.309] |
| Tomato Lettuce Salad | GLAM | +0.063 [-0.008, 0.196] |
Together, these additional evaluations provide stronger evidence of BeliefRL's performance across independently trained checkpoints, all four tasks, and held-out environments.
Thanks again for your valuable suggestions. We have made brief revisions to the manuscript accordingly and will continue to work to incorporate the above discussions and improvements into the final version. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and looking forward to further discussions with you.
Best regards,
All Authors
Rebuttal (Part 1/3)
Thank you for your thoughtful and constructive feedback. Your comments have helped us strengthen the experimental rigor and clarify the scope of BeliefRL's contributions. To address your concerns, we have conducted several additional experiments and analyses: (1) Baseline fairness: We introduce a supervision-matched GRU-Sup baseline using the same labeled trajectories and demonstrate higher training AUC for BeliefRL across all four tasks. We also add a direct comparison with Agent-BRACE, achieving a 14.0% relative AUC improvement on Food Preparation. (2) Credit assignment: We conduct on-policy belief–advantage sensitivity analyses and controlled experiments on delayed belief errors, demonstrating that belief agreement is informative about advantage discrepancies while clarifying the precise role of reliability weighting. (3) Evaluation rigor: We extend frozen-policy evaluation to all four tasks using independently trained checkpoints and report paired-bootstrap confidence intervals against the strongest individual baselines, providing more comprehensive evidence of performance and uncertainty. We believe these additions substantially address your concerns and strengthen the empirical foundation of BeliefRL. We would greatly appreciate your reconsideration of the assessment in light of these new results and clarifications.
W1 & Q1. Baseline Fairness and Comparison with Belief-Learning Methods
Thank you for suggesting a supervision-matched memory baseline and a direct comparison with closely related belief-learning methods. To address your concerns, we have added two targeted comparisons to distinguish the benefits of BeliefRL's belief-centric mechanisms from those of additional supervision.
(1) GRU-Sup under matched supervision (W1). We introduce GRU-Sup, a recurrent-memory baseline using the same LLaMA-3.1-8B backbone and 275 labeled episodes used for BeliefRL's BSM initialization. Its GRU memory is trained to predict the same True/False/Unknown predicate labels and continues to receive supervised updates alongside PPO training. This provides a controlled comparison between explicit belief modeling and recurrent memory under matched supervision resources (Appendix E.4).
We evaluate both methods across five training seeds on all four tasks. The normalized training AUC results (mean ± SD) are:
BeliefRL achieves higher mean training AUC on all four tasks, despite both methods benefiting from the same labeled supervision source. These findings support the value of BeliefRL's explicit belief-centric learning framework beyond simply providing additional labeled trajectories.
(2) Direct comparison with Agent-BRACE (Q1). We have additionally conducted a direct comparison with Agent-BRACE on Food Preparation (Appendix B.7, Fig. 10 and Table 11), obtaining the following normalized training AUC:
BeliefRL achieves an absolute AUC gain of +0.0577 (+14.0% relative) in this comparison. Methodologically, Agent-BRACE jointly optimizes belief and policy models through separate reward objectives, whereas BeliefRL uses shared observation-consistent reference beliefs to supervise online BSM adaptation and explicitly weight token-level actor advantages according to belief reliability.
Together, these experiments provide direct empirical evidence against both a supervision-matched recurrent baseline and a closely related belief-learning method, strengthening the experimental support for BeliefRL's belief-centric training approach.