- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper summary: This paper proposes , a framework that injects a Controlling state into the rollout of an RL agent. More specifically, injects controlling states after the agent proposes the next possible action and determines whether that action is appropriate for the current trajectory. The experiments evaluate the performance of with other settings, including ReAct and PlanAndExecute, which illustrate the importance of the proposed controlling states and action assessment in RL agent rollout.
Contributions: The paper's main contribution is the introduction of controlling states before deciding to follow the action for the inference of an RL agent. The controlling states include: EXPLORE, EXPLOIT, and RECOVER. The experiments illustrate that by equipping the agent with the Controlling states, the RL agent can achieve more terminal efficiency compared with other baselines.
Critical strengths
There are several strengths of the paper, listed below:
- Important problem statement: The problem is important, and the paper points out the framework like ReAct can repeat failed actions; therefore, a verification framework is needed to inject and avoid repetition.
- Simple plug-and-play method: The proposed method is simple and does not require any training or fine-tuning. Prompting is enough to inject the controlling state into the agent.
- Experimental results: Compared with baselines, e.g., ReAct and Vanilla, achieves better success and termination score.
Critical weaknesses
Several weaknesses should be included:
- Novelty and clarity: It seems that the improvement of comes from the feedback it receives from the environment and from previous episodes. However, I think this proposal is not new. Previous methods such as AdaPlanner [1] and SwiftSage [2] already do quite similar work. This paper should include a highlight to illustrate the difference between and other previous methods (comparison in the experiments)
- Need to include more baselines in the experiments: The baselines reported in Table 1, Table 2 should include AdaPlanner [1], Self Refine [3], and Reflecxion [4] at each step. These baselines already use trajectory feedback to revise the behavior of the agent, therefore, a quantitative comparison with them is necessary.
- Clarity in notation: What can be the value of budget for PB-AUC in Equation 13? And in Appendix B, do the Vanilla and Plan&Execute baselines receive the demonstration similar to ReAct?
- Different backbone consideration: The method only runs on a single backbone (DeepSeek). The prompting can be different with different backbones; therefore, can the authors run it on different LLM families, including Qwen3-8B or LLaMA-8B?
- Motivating failures are not supported in the main experiment: In Figure 1, the paper reports 36.8% of ReAct failures due to repetition failures. In Table 4, what are the percentages of Matching Reasoning and for this statistic?
- What are the average termination steps of ReAct compared with ? Also, can the paper report the token usage of ReAct compared with ?
References: [1] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. In Advances in Neural Information Processing Systems, volume 36, 2023.
[2] Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Advances in Neural Information Processing Systems, volume 36, 2023.
[3] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, 2023.
[4] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.
Questions for authors
See the weaknesses section
Response to Reviewer Wpy5[Part 1]
We thank the reviewer for raising this important question. We would like to further clarify the motivation behind E2R and provide additional experimental evidence addressing the suggested simple intervention.
Motivation. Long-horizon interaction requires agents to continuously determine how accumulated trajectory evidence should guide subsequent behavior. This involves three distinct control requirements: (i) acquiring missing evidence needed for further progress, (ii) advancing an evidence-supported continuation, and (iii) recovering from a continuation rendered ineffective by observed feedback. These requirements motivate our trajectory-conditioned control state zt=(mt,ℓt), where the mode specifies how to proceed and the target grounds this decision in the current trajectory.
Empirical evidence. Figure 1 illustrates a concrete manifestation of inappropriate continuation: an agent repeatedly attempts an ineffective action despite explicit negative feedback. Among 38 ReAct episodes containing the specified negative-feedback event, 14 (36.8%) exhibit exact action persistence. This example highlights one observable control failure, while the broader formulation also addresses information acquisition and evidence-supported execution.
Sparse intervention. The low redirect rates in Table 3 reflect E2R's selective regulation mechanism. The controller infers a control state at each step and modifies a proposal only when it conflicts with the inferred control objective. Accordingly, the reported redirect rates (0.4–6.8% of steps) and intervention coverage (5.2–23.6% of episodes) characterize the frequency of explicit action modification, rather than the frequency of control-state inference or all possible control failures.
Comparison with the suggested simple intervention. To directly evaluate the reviewer's suggestion, we additionally tested ReAct with an explicit instruction to avoid repeating failed actions:
The results are summarized below:
The No-Repeat instruction improves ReAct by 4.47 percentage points in SR and 0.0212 in PB-AUC, confirming that local failure avoidance is beneficial. E²R achieves a further 13.44 percentage-point improvement in SR and 0.0963 in PB-AUC over this baseline. These results indicate that the benefits of E²R extend beyond those obtained from the suggested failure-avoidance instruction.