- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper considers the credit assignment problem in agentic reinforcement learning of large language models. Towards this problem, the authors propose MICO (Milestone-Informed Counterfactual Optimization). This algorithm introduces two main ideas: (1) use hand-designed milestones for partial credit assignment, and (2) use sibling continuations as counterfactual signal for advantage estimation. Notably, the methodology does not generalize across environments without hand-designing the milestones to use, which is not always possible for environments beyond this paper's main focus. The authors prove some theoretical claims about the algorithm, including properties about the optimum of the shaped objective and error bounds about the advantage estimation, whose value are unclear to me. Experiments are mainly conducted on a small agentic benchmark, TaskShop, devised by the authors. On this benchmark, the authors are able to see MICO improves over GRPO and other credit assignment methods when the rollouts per task is 2 with Qwen2.5-1.5B-Instruct. But this improvement does not generalize to when rollouts per task is 8 or other more standard benchmarks like WebShop and ALFWorld.
Critical strengths
- The problem of study in this paper is critical and relevant. If credit assignment properly is solved, we will be able to see a huge improvement in agentic reinforcement learning of large language models.
Critical weaknesses
- (Major) This manuscript fails to follow several important research practice.
- The manuscript exceeds the page limit for the main paper.
- The manuscript does not contain the required statement about AI usage.
- References to related work have major flaws. For example, the paper "IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents" was authored by Bo Ren, Yirong Mao, Yi Yang, Wenhui Que, while the authors cited this paper under "Yirong Mao et al.". Another example is, the paper "KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling" was authored by Peng Kuang, Haibo Jin, Xiaoyu Han, Yanli Wang, Xiaopeng Yuan, Ye Yu, Kaidi Xu, Haohan Wang, while the authors cited this paper under "Peng Kuang, Haibo Jin, and Haohan Wang".
- (Major) The results are notably weak.
- The proposed methodology highly relies on a hand-designed, usable miletone and has poor generalizability for new environments.
- The theoretical results are unclear in their value.
- The experiment results are mainly negative, with the only improvement case being a very restricted setting, where the model is 1.5B, the RL rollout size is 2, and the task is a non-standard small benchmark devised by the authors.
- (Major) The writing quality is poor. Throughout the paper, nested clauses and phrases that are unclear in their meanings often appear.
Questions for authors
- Q1: Feel free to respond to the weaknesses raised.
- Q2: Could the authors disclose their use of AI tools in preparing the manuscript?