- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an RL framework for LLM agents in the partially observable environment to address the belief drift and insufficient process rewards. The main idea of the proposed method includes explicit belief tracking, belief-consistency reward design, and belief-based advantage estimation.Experiments on ALFWorld and WebShop shows gains over episode-level GRPO and step-level GiGPO using Qwen2.5-1.5B-Instruct.
Critical strengths
Using model's own belief and environment's feedback to supervise intermediate state understanding is a sample-efficient and less costly way to provide process rewards for on-policy RL.
Critical weaknesses
- The central belief-based step grouping is insufficiently specified and differs across descriptions. Section 3.3 defines groups through equality of belief predicates, whereas Appendix A.3 states that ALFWorld actually uses an environment-derived task-phase anchor augmented with unspecified structural flags. The paper does not explain how these anchors and flags are computed or reconcile the two formulations. It is crucial to determine precisely what makes steps comparable in order to assess whether the benefit comes from belief modeling rather than simpler task-progress grouping.
- Insufficient baselines and experiments:
- There are many step-level methods [1-3] but the paper only compares with GiGPO (not a recent baseline).
- ALFWorld and WebShop are too old and simple benchmarks for recent LLMs, where the state and action space are finite and closed. Other tasks like agentic coding[4,5] or complex gaming (e.g, Minecraft/Pokemon) are expected to evaluate the method, especially open-ended state-action spaces that LLMs are better to handle compared to traditional ML models.
- The closed-source frontier models are out-dated as baselines, and more recent LLMs (e.g., GPT-6/Gemini-3.8/Opus-5) should be compared.
- Only Qwen2.5-1.5B-Instruct as the backbone. A larger and more backbones (e.g.,qwen-3.8-27b) should be added to show the method's robustness across models.
References:
- Zhang, Zijing, et al. "Rlvmr: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents." ICLR, 2026.
- Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." ICLR, 2026.
- Zou, Jiaru, et al. "Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms." Advances in Neural Information Processing Systems 38 (2026): 26764-26802.
- https://openai.com/index/introducing-swe-bench-verified/
- Trivedi, Harsh, et al. "Appworld: A controllable world of apps and people for benchmarking interactive coding agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Questions for authors
- What exact inputs and rules produce the belief anchors used in the experiments, including task phases, structural flags, and canonicalization, and how does this implementation correspond to the predicate-equality grouping defined in Section 3.3?
- How are consistency checks implemented when an action changes a predicate’s truth value, how does delayed verification distinguish an initially incorrect belief from a subsequent state change, and what value of the consistency-reward weight was used?
- Are SFT initialization, training data, clipping, entropy protection, rollout budgets, and evaluation protocols held fixed across B1–B3, particularly between B2 and B3?
- What evidence supports the claim that ReBel reduces belief drift?