- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Reflection-Guided Policy Optimization, a reinforcement learning method for error recovery in long-horizon tool-using agents. It mainly consists of two components: reflection-guided sampling and tree-structured credit assignment. The main contributions include: i) proposing ReflectPO, which uses directed graphs to extract recovery signals from prefix-tree branch differences and repeated visitation patterns, and guides agents to generate recovery trajectories through contrast-based and loop-based sampling; ii) proposing a tree-structured credit assignment method that provides branch-level advantages for original actions and paired reward-difference advantages for recovery actions, thereby more accurately measuring the impact of local decisions and the gains from recovery; and iii) conducting experiments on long-horizon tool-use benchmarks. Experiments train Qwen3 models with RL data from EnvScaler and compare against several policy optimization baselines. Results show that ReflectPO achieves a certain degree of improvement over the baselines.
Critical strengths
-
The paper tackles an important problem in long-horizon tool use. Errors can propagate through stateful interactions and cause task failure. Improving recovery in these settings is a meaningful direction for building more reliable LLM agents.
-
Its contrast-based and loop-based sampling target complementary patterns in agent execution, while the tree-structured objective provides a concrete way to incorporate these trajectories into policy optimization.
-
Experiments on BFCL-v3 Multi-Turn and τ²-Bench, along with comparisons to several RL baselines and ablations, provide a useful initial assessment of the approach across different settings.
Critical weaknesses
-
A guide model generates recovery hints, and the sampling procedure selects where to insert them. The paper does not show that the trained policy can independently identify errors and choose recovery anchors when these aids are unavailable at inference time.
-
Recovery trajectories are generated with hints, but the hints are removed during policy updates. The paper does not specify how the old-policy probabilities in Equation 6 are computed under this context change, or justify why the resulting importance-sampling ratio remains valid.
-
The parent-node reward already includes trajectories passing through its child, making the parent–child reward difference difficult to interpret as a counterfactual advantage. In addition, the same advantage is propagated across subsequent actions, including actions that may not have caused the outcome. The paper provides no bias analysis or comparison with alternative credit-assignment rules.
-
A single LLM judge identifies errors and assesses whether they were corrected, but the paper reports no human validation or agreement statistics. Since these judgments underpin the paper’s central recovery claims, their reliability remains unclear.
-
ReflectPO uses a strong guide model and generates extra recovery trajectories, but the paper does not fully report their costs or compare against resampling the same anchors without hints or with generic hints. It is therefore unclear how much of the gain comes from the proposed sampling strategy, the guide’s feedback, or the additional training data.
Questions for authors
See weaknesses.
7 comments
Response for Reviewer pWhG 2/2
W4: Reliability and Human Validation of the LLM Judge
Thank you for pointing out this issue. Existing works such as MT-Bench, TRAIL, and AgentDebug have widely adopted the LLM-as-a-judge approach to identify and localize errors in agent trajectories. To thoroughly validate our conclusions on error recovery, we conducted three additional experiments to analyze the evaluation of Error Step Rate and Step Recovery Rate:
- For the same set of results, we conducted independent repeated evaluations using GPT-5.6 Terra. Due to cost constraints, we independently evaluated each sample five times and reported the mean and standard deviation of the judge scores.
- Furthermore, using the same setup, we replaced the judge model with Gemini 3.8 Flash and re-ran the evaluation.
- Due to labor cost constraints, we randomly sampled 30 trajectories from each benchmark for human evaluation. We hired three professionally trained annotators to determine whether recovery behaviors were triggered in the trajectories produced by both the baseline method and our method.
The final results are shown in the table below. The variance across repeated evaluations is minimal, and even under different evaluation methods, ReflectPO consistently maintains a lower Error Step Rate and a higher Step Recovery Rate compared to the base model, demonstrating the effectiveness of ReflectPO in improving error recovery. We have added this experiment to Section 5.8 of the paper.
| Judge | Method | Error Step Rate ↓ | Step Recovery Rate ↑ |
|---|---|---|---|
| GPT-5.6 Terra | Qwen3-8B | 19.39 ± 0.15 | 17.58 ± 0.24 |
| GPT-5.6 Terra | ReflectPO | 15.19 ± 0.06 | 21.43 ± 0.04 |
| Gemini 3.8 Flash | Qwen3-8B | 14.94 | 25.81 |
| Gemini 3.8 Flash | ReflectPO | 10.91 | 27.31 |
| Human | Qwen3-8B | 17.8 | 17.7 |
| Human | ReflectPO | 11.52 | 21.33 |
W5: Computational Costs and Disentangling Sources of Gain
Thank you for the reviewer's suggestion. In terms of training cost, we analyzed the training at step 210. In this step, the guidance model was invoked 156 times, consuming 573,729 tokens, while the total number of rollout tokens was 5,426,021. The tokens consumed by the guidance model accounted for only 10.5% of the total tokens. We have added this analysis to Section 5.7 of the paper. To further analyze the impact of each component of ReflectPO on the final training performance, we conducted additional experiments. Based on the same anchor points, we performed resampling without hints and resampling with a generic prompt, respectively, and then re-ran the RL training. The generic hint we used is as follows:
Recheck the task requirements against your recent actions and the latest tool results. Verify that the tool you selected, the arguments you used, and any values or assumptions carried over from earlier steps are correct. If you notice an inconsistency or a repeated action that is not making progress, revise your next step accordingly; Otherwise, continue based on the verified information.
We obtained the following results:
| Model | -Bench (Airline) | -Bench (Retail) | -Bench (Telecom) | -Bench (Overall) | BFCL-v3 (Base) | BFCL-v3 (Long-Context) | BFCL-v3 (Miss-Func) | BFCL-v3 (Miss-Param) | BFCL-v3 (Overall) | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 32 | 43 | 24.8 | 33.6 | 47.5 | 36.5 | 48 | 35 | 41.8 | 39.6 |
| Reinforce++ | 38 | 46.9 | 21.9 | 35.1 | 51.5 | 35.5 | 47.5 | 39.5 | 43.5 | 41.3 |
| Generic Prompt | 42 | 40.7 | 13.2 | 29.7 | 60 | 45.5 | 51.5 | 38.5 | 48.9 | 43.9 |
| Without Hints | 38 | 46 | 24.6 | 35.8 | 56 | 35.5 | 57 | 39 | 46.9 | 44 |
| ReflectPO | 46 | 46.5 | 24.6 | 37.4 | 55 | 45.5 | 53.5 | 38.5 | 48.1 | 45.4 |
It can be seen that using a generic hint or no hint both yield better training performance than Reinforce++ alone, but still underperform the full version trained with ReflectPO. We have added this experiment to Table 2 of the paper.
Thank you for the detailed rebuttal and additional experiments. The probability-shift analysis is valuable and helps quantify the impact of removing hints. However, one concern remains: recovery trajectories are sampled with hints, whereas old-policy probabilities are recomputed after hint removal. Thus, the denominator in Eq. (6) does not reflect the actual sampling policy. Could the authors clarify whether this is an intentional approximation and discuss its implications for policy optimization?
Response for Reviewer pWhG 1/2
W1: Autonomous Error Identification and Recovery at Inference Time
In Figure 4, we use GPT-5.6 Terra to determine whether recovery behaviors are spontaneously triggered in hint-free inference trajectories. Specifically, this evaluates whether the post-trained model can spontaneously exhibit reflective and corrective behaviors when such auxiliary guidance is unavailable at inference time. Furthermore, the experimental results in Figure 4 demonstrate that, compared to the pre-training baseline, the trained model is able to identify and recover from errors to a certain extent. We will include a more detailed explanation of this part in the camera-ready version.
W2: Old-Policy Probabilities and Importance-Sampling Validity in Eq. (6)
Thank you for pointing out this detail. The old-policy log probability in Equation (6) is recomputed on the sequence without the hint. To quantify its impact, we performed paired teacher-forcing across all 171 recovery trajectories from the final training step, computing the log-probabilities of the exact same tokens under both the hint-conditioned and hint-free contexts and measuring their token-wise differences. The results are shown in the table below:
The shifts are concentrated in the first corrective action and decay rapidly thereafter, leaving the vast majority of tokens virtually unaffected; The changes primarily occur at a small number of key decision tokens. We have added this experiment to Section 5.10 of the paper.
W3: Counterfactual Advantage Formulation and Credit Assignment
The reward of a parent node is the average terminal reward of all trajectories passing through it; therefore, the reward difference between a child node and its parent actually represents the performance of the current child node relative to the average performance of all trajectories passing through that branching point. Furthermore, regarding advantage propagation, within an unbranched trajectory segment, the tree structure cannot further distinguish the contribution of individual actions. Since it is difficult to estimate the value of each single action, we uniformly assign the advantage of the subtree corresponding to this segment to all actions within it, serving as a heuristic for segment-level credit assignment. In Table 3, we compare the performance of Reinforce++ and tree credit assignment, where the former performs direct sequence-level credit assignment and the latter computes credit using the parent-child reward difference. The results also show that tree credit assignment generally outperforms Reinforce++.