acceptodds
Under review as a conference paper at ICLR 2027

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Abstract

With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent–child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.

Primary area:
reinforcement learning
Keywords:
LLM-based agent, Large Language Model, self-reflexion, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_pWhGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Reflection-Guided Policy Optimization, a reinforcement learning method for error recovery in long-horizon tool-using agents. It mainly consists of two components: reflection-guided sampling and tree-structured credit assignment. The main contributions include: i) proposing ReflectPO, which uses directed graphs to extract recovery signals from prefix-tree branch differences and repeated visitation patterns, and guides agents to generate recovery trajectories through contrast-based and loop-based sampling; ii) proposing a tree-structured credit assignment method that provides branch-level advantages for original actions and paired reward-difference advantages for recovery actions, thereby more accurately measuring the impact of local decisions and the gains from recovery; and iii) conducting experiments on long-horizon tool-use benchmarks. Experiments train Qwen3 models with RL data from EnvScaler and compare against several policy optimization baselines. Results show that ReflectPO achieves a certain degree of improvement over the baselines.

Critical strengths
  1. The paper tackles an important problem in long-horizon tool use. Errors can propagate through stateful interactions and cause task failure. Improving recovery in these settings is a meaningful direction for building more reliable LLM agents.

  2. Its contrast-based and loop-based sampling target complementary patterns in agent execution, while the tree-structured objective provides a concrete way to incorporate these trajectories into policy optimization.

  3. Experiments on BFCL-v3 Multi-Turn and τ²-Bench, along with comparisons to several RL baselines and ablations, provide a useful initial assessment of the approach across different settings.

Critical weaknesses
  1. A guide model generates recovery hints, and the sampling procedure selects where to insert them. The paper does not show that the trained policy can independently identify errors and choose recovery anchors when these aids are unavailable at inference time.

  2. Recovery trajectories are generated with hints, but the hints are removed during policy updates. The paper does not specify how the old-policy probabilities in Equation 6 are computed under this context change, or justify why the resulting importance-sampling ratio remains valid.

  3. The parent-node reward already includes trajectories passing through its child, making the parent–child reward difference difficult to interpret as a counterfactual advantage. In addition, the same advantage is propagated across subsequent actions, including actions that may not have caused the outcome. The paper provides no bias analysis or comparison with alternative credit-assignment rules.

  4. A single LLM judge identifies errors and assesses whether they were corrected, but the paper reports no human validation or agreement statistics. Since these judgments underpin the paper’s central recovery claims, their reliability remains unclear.

  5. ReflectPO uses a strong guide model and generates extra recovery trajectories, but the paper does not fully report their costs or compare against resampling the same anchors without hints or with generic hints. It is therefore unclear how much of the gain comes from the proposed sampling strategy, the guide’s feedback, or the additional training data.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response for Reviewer pWhG 1/2

W1: Autonomous Error Identification and Recovery at Inference Time

In Figure 4, we use GPT-5.6 Terra to determine whether recovery behaviors are spontaneously triggered in hint-free inference trajectories. Specifically, this evaluates whether the post-trained model can spontaneously exhibit reflective and corrective behaviors when such auxiliary guidance is unavailable at inference time. Furthermore, the experimental results in Figure 4 demonstrate that, compared to the pre-training baseline, the trained model is able to identify and recover from errors to a certain extent. We will include a more detailed explanation of this part in the camera-ready version.

W2: Old-Policy Probabilities and Importance-Sampling Validity in Eq. (6)

Thank you for pointing out this detail. The old-policy log probability in Equation (6) is recomputed on the sequence without the hint. To quantify its impact, we performed paired teacher-forcing across all 171 recovery trajectories from the final training step, computing the log-probabilities of the exact same tokens under both the hint-conditioned and hint-free contexts and measuring their token-wise differences. The results are shown in the table below:

ScopeMean Δ\DeltaMean ∣Δ∣\lvert\Delta\rvertp95(∣Δ∣)p_{95}\left(\lvert\Delta\rvert\right)Fraction with ∣Δ∣>0.5\lvert\Delta\rvert > 0.5
All recovery tokens−0.045-0.0450.1210.1210.5090.5095.10%5.10\%
1st action (first corrective action)−0.230-0.2300.4040.4042.052.0514.40%14.40\%
2nd action−0.065-0.0650.180.180.9380.9388.00%8.00\%
3rd action−0.042-0.0420.1410.1410.6530.6536.20%6.20\%
≥4\ge 4th action−0.011-0.0110.0650.0650.3160.3163.10%3.10\%

The shifts are concentrated in the first corrective action and decay rapidly thereafter, leaving the vast majority of tokens virtually unaffected; The changes primarily occur at a small number of key decision tokens. We have added this experiment to Section 5.10 of the paper.

W3: Counterfactual Advantage Formulation and Credit Assignment

The reward of a parent node is the average terminal reward of all trajectories passing through it; therefore, the reward difference between a child node and its parent actually represents the performance of the current child node relative to the average performance of all trajectories passing through that branching point. Furthermore, regarding advantage propagation, within an unbranched trajectory segment, the tree structure cannot further distinguish the contribution of individual actions. Since it is difficult to estimate the value of each single action, we uniformly assign the advantage of the subtree corresponding to this segment to all actions within it, serving as a heuristic for segment-level credit assignment. In Table 3, we compare the performance of Reinforce++ and tree credit assignment, where the former performs direct sequence-level credit assignment and the latter computes credit using the parent-child reward difference. The results also show that tree credit assignment generally outperforms Reinforce++.

AuthorsOpenReview ↗

Response for Reviewer pWhG 2/2

W4: Reliability and Human Validation of the LLM Judge

Thank you for pointing out this issue. Existing works such as MT-Bench, TRAIL, and AgentDebug have widely adopted the LLM-as-a-judge approach to identify and localize errors in agent trajectories. To thoroughly validate our conclusions on error recovery, we conducted three additional experiments to analyze the evaluation of Error Step Rate and Step Recovery Rate:

  • For the same set of results, we conducted independent repeated evaluations using GPT-5.6 Terra. Due to cost constraints, we independently evaluated each sample five times and reported the mean and standard deviation of the judge scores.
  • Furthermore, using the same setup, we replaced the judge model with Gemini 3.8 Flash and re-ran the evaluation.
  • Due to labor cost constraints, we randomly sampled 30 trajectories from each benchmark for human evaluation. We hired three professionally trained annotators to determine whether recovery behaviors were triggered in the trajectories produced by both the baseline method and our method.

The final results are shown in the table below. The variance across repeated evaluations is minimal, and even under different evaluation methods, ReflectPO consistently maintains a lower Error Step Rate and a higher Step Recovery Rate compared to the base model, demonstrating the effectiveness of ReflectPO in improving error recovery. We have added this experiment to Section 5.8 of the paper.

JudgeMethodError Step Rate ↓Step Recovery Rate ↑
GPT-5.6 TerraQwen3-8B19.39 ± 0.1517.58 ± 0.24
GPT-5.6 TerraReflectPO15.19 ± 0.0621.43 ± 0.04
Gemini 3.8 FlashQwen3-8B14.9425.81
Gemini 3.8 FlashReflectPO10.9127.31
HumanQwen3-8B17.817.7
HumanReflectPO11.5221.33

W5: Computational Costs and Disentangling Sources of Gain

Thank you for the reviewer's suggestion. In terms of training cost, we analyzed the training at step 210. In this step, the guidance model was invoked 156 times, consuming 573,729 tokens, while the total number of rollout tokens was 5,426,021. The tokens consumed by the guidance model accounted for only 10.5% of the total tokens. We have added this analysis to Section 5.7 of the paper. To further analyze the impact of each component of ReflectPO on the final training performance, we conducted additional experiments. Based on the same anchor points, we performed resampling without hints and resampling with a generic prompt, respectively, and then re-ran the RL training. The generic hint we used is as follows:

Recheck the task requirements against your recent actions and the latest tool results. Verify that the tool you selected, the arguments you used, and any values or assumptions carried over from earlier steps are correct. If you notice an inconsistency or a repeated action that is not making progress, revise your next step accordingly; Otherwise, continue based on the verified information.

We obtained the following results:

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
Qwen3-8B324324.833.647.536.5483541.839.6
Reinforce++3846.921.935.151.535.547.539.543.541.3
Generic Prompt4240.713.229.76045.551.538.548.943.9
Without Hints384624.635.85635.5573946.944
ReflectPO4646.524.637.45545.553.538.548.145.4

It can be seen that using a generic hint or no hint both yield better training performance than Reinforce++ alone, but still underperform the full version trained with ReflectPO. We have added this experiment to Table 2 of the paper.

Thank you for the detailed rebuttal and additional experiments. The probability-shift analysis is valuable and helps quantify the impact of removing hints. However, one concern remains: recovery trajectories are sampled with hints, whereas old-policy probabilities are recomputed after hint removal. Thus, the denominator in Eq. (6) does not reflect the actual sampling policy. Could the authors clarify whether this is an intentional approximation and discuss its implications for policy optimization?

Reviewer_riPyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReflectPO trains LLM agents to recover from errors in long tasks. Training starts with several ordinary rollouts. Branch comparisons locate decisions followed by different outcomes. Repeated action patterns provide another source of recovery points. DeepSeek model writes a hint at each selected point. The small model then continues from a saved environment state. The hint is removed before the policy update. Recovery actions are rewarded when they improve the original result. A rollout tree provides credit for earlier actions.

Critical strengths

The method creates recovery examples from failed trajectories rather than only sampling complete new rollouts. Reusing the saved state also avoids generating the whole prefix again.

The branch comparison and loop detection methods are intuitive. The paper tests two model sizes on two benchmarks.

Critical weaknesses

The key baseline is missing. The paper should restart from the same selected states with the same number of new rollouts, but without hints. Otherwise, it is unclear whether the gain comes from reflection, targeted resampling or knowledge from DeepSeek model.

The paper does not directly establish autonomous recovery. Each model produces a different set of errors. So their recovery rates are not comparable. One LLM judge labels all errors and recoveries. Testing every model from the same failed states would be more convincing.

The results have no confidence intervals or independent training runs. Some differences are small. The overall score is also not explained and appears to place much more weight on BFCL-v3.

The ablation is mixed. Tree credit hurts tau2-Bench. The full method is barely better than contrast-only sampling. The efficiency analysis ignores the cost of generating hints with the stronger model.

Questions for authors

What happens when the agent restarts from the same recovery points without any reflection hint?

Can all models be tested on the same failed states, with human checking of the error and recovery labels?

How is the overall score calculated?

Are the gains stable across independent training runs?

4 comments
AuthorsOpenReview ↗

Response for Reviewer riPy 1/2

W1: Missing Hint-Free Resampling Baseline from Selected States

Thank you for this important concern. We note that the proposed setting of not providing any hints is conceptually close to the rollout procedure in tree-credit methd, except that the rollout is restarted from a selected anchor point. To further investigate the role of hints in our method, we conduct an additional experiment in which no hints are provided. Instead, we restart the rollout process using the same number of new rollouts and repeat the RL training and evaluation under this setting. The results are reported below.

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
Qwen3-8B324324.833.647.536.5483541.839.6
Reinforce++3846.921.935.151.535.547.539.543.541.3
Without Hints384624.635.85635.5573946.944
ReflectPO4646.524.637.45545.553.538.548.145.4

The results show that the Without Hints setting still outperforms both the original Qwen3-8B model and the standard Reinforce++ baseline. However, its performance remains below that of the full ReflectPO model. We have added the results of this experiment to Table 2 of the revised paper.

W2: Controlled Recovery Evaluation on Shared Failed States and Judge Reliability

Thank you for this important concern. In this experiment, we identify the first erroneous step in each Qwen3-8B trajectory and truncate the trajectory at that point, retaining the action at the erroneous step and its corresponding observation. We then use the ReflectPO-trained model and Reinforce++-trained model to continue inference from the truncated trajectories and evaluate their performance. To further validate the recovery capability, we employ three professionally trained annotators to determine whether the continued trajectories exhibit recovery behaviors. Due to the cost of human annotation, we randomly sample 30 trajectories from each benchmark for manual evaluation. The results are shown below.

Task Performance:

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
Qwen3-8B324324.833.647.536.5483541.839.6
Reinforce++ (Continuation)32.748.219.333.65436.245.538.543.641
Reinforce++3846.921.935.151.535.547.539.543.541.3
ReflectPO (Continuation)34.754.521.137.255.539.747.53845.243.1
ReflectPO4646.524.637.45545.553.538.548.145.4

Human-Annotated Recovery Rate (%):

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
Qwen3-8B14.342.921.129.435.129.234.823.130.530.1
Reinforce++ (Continuation)22.946.9203240.431.230.425.931.731.8
ReflectPO (Continuation)25.754.122.136.443.637.533.92534.635.3

The results show that ReflectPO achieves better continuation performance and a higher error recovery rate than both Qwen3-8B and Reinforce++. This demonstrates that our method improves the agent's ability to recover from erroneous execution states.

W3: Overall Score Weighting

The Overall score is calculated by combining all 800 BFCL test samples and 278 τ²-Bench test samples and computing the overall success rate: Overall=(800×BFCL+278×τ2)/1078\text{Overall}=(800\times\text{BFCL}+278\times\tau^2)/1078

AuthorsOpenReview ↗

Response for Reviewer riPy 2/2

W4: Mixed Ablation Results and Hint-Generation Cost in Efficiency Analysis

Thank you for this important concern. In Table 3, ReflectPO shows a performance drop on BFCL-v3 Multi-Turn compared with the Contrast-only variant, while achieving a higher Overall score. We believe this is partly due to the characteristics of long-horizon tool-calling tasks. Given the substantial distribution differences across tasks and the various factors that can perturb long-horizon task execution, it is difficult for a single method to consistently outperform all baselines across every benchmark. Compared with the Contrast-only variant, ReflectPO introduces Loop-based Sampling to guide the model to escape from failed execution loops. To demonstrate the effectiveness of the full method, we measure the occurrence of repetitive loops across different methods on the evaluation benchmarks. The results are shown below.

MethodAvg. Loop Occurrences per Trajectory (BFCL)Avg. Loop Occurrences per Trajectory (τ2\tau^2)Overall
Qwen3-8B0.20.0250.155
Contrast-only0.1310.0140.101
Loop-only0.0240.0180.022
ReflectPO0.0280.0070.022

As shown, the base model is most prone to repetitive looping. Although Contrast-only alleviates this issue to some extent, its overall average number of loop occurrences remains as high as 0.101. In contrast, both Loop-only and the full ReflectPO model reduce this metric to 0.022. These results demonstrate that loop-based sampling effectively mitigates repetitive looping behaviors. We have added this experiment to Section 5.6 of the revised paper. Furthermore, we analyze the computational cost of using a stronger model during training. Our reported ReflectPO results are obtained at training step 210, which we use as an example for this analysis. At this step, the guiding model is invoked 156 times, consuming 573,729 tokens, while the total rollout token consumption is 5,426,021. Thus, the tokens consumed by the guiding model account for only 10.5% of the total token consumption. Due to current computational resource constraints, it is difficult for us to repeat the RL experiments with multiple random seeds. We commit to including the training results across different random seeds in the camera-ready version of the paper.

Thanks for the detailed response. The hint-free baseline (also requested by other reviewers) show that if we restart from the selected states without hints we reach 44.0 versus full reflect IPO score of 45.4. It shows that failure-specific hints provide additional improvement, but not a large gain. The shared state experiment with human labels show the reliability of recovery. The results across multiple seeds will be provided for the camera-ready version. I will keep my original rating.

Thank you for your constructive feedback and for raising your rating. We will incorporate the new results into the revised version and continue proofreading the manuscript to improve its clarity:)

Reviewer_yZ8YOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReflectPO trains tool-using LLM agents to recover from mistakes during multi-turn interactions. It identifies two opportunities to retry a trajectory: a shared prefix after which branches achieve different rewards, and a repeated sequence of actions and observations within one trajectory. A guide model supplies a hint, and the agent resumes from a saved environment state. For the original trajectories, training uses reward differences between parent and child nodes. For a recovery continuation, it uses the reward difference relative to the original trajectory. On BFCL-v3 Multi-Turn and τ²-Bench, ReflectPO achieves the highest reported overall score at both the 4B and 8B scales.

Critical strengths

1.ReflectPO addresses a practical issue in long-horizon tool use by letting agents practice recovery from mistakes made during their own rollouts.

2.The experiments report gains in average performance at both model sizes. A separate analysis examines how often agents correct an error once it occurs.

Critical weaknesses

1.Contrast-based sampling selects the lowest-reward trajectory at a branch as its target. A fresh continuation from the same snapshot can therefore improve on that target through ordinary sampling variation. Table 3 compares recovery strategies, but it does not show how much of the gain survives when the agent retries the same states without a hint. This matters because Eq. (5) assigns the observed improvement to all newly generated recovery actions.

2.Recovery actions are sampled and scored under different contexts. Section 4.3 says that the behavior policy generates a continuation with a hint, while the update removes the hint. As written, the denominator in Eq. (6) evaluates the old policy without the hint. If so, the ratio does not account for the change in action probabilities caused by the hint. The update may still be effective, but its interpretation as a policy optimization step for the hint-free agent needs justification.

Questions for authors

1.At the same contrast-selected snapshots, how often does an unguided retry improve on the selected target, and how does its mean reward gain compare with that of a guided retry? A comparison using the same anchors and number of continuations would help establish what the hints contribute.

2.For recovery tokens in Eq. (6), is the old-policy log probability cached during hint-conditioned generation or recomputed after removing the hint? If it is recomputed, how large is the change in token log probabilities, particularly for the first corrective action?

4 comments
AuthorsOpenReview ↗

Response for Reviewer yZ8Y

Q1: How do guided retries compare with unguided retries at the same snapshots

We thank the reviewer for their comments. For the original training data, we used Qwen3-8B to sample 88 original trajectories per task, and applied the contrast-based method described in this paper to select branch points, targets, and recovery anchors. Under the same environment snapshot, with identical prefixes and identical decoding settings, we sampled 44 continuations for each of the guided and unguided conditions. The results show that the mean reward of the target is 0.42260.4226, the mean reward of unguided resampling is 0.62060.6206 (gain +0.1980+0.1980; trajectory improvement rate 58.2%58.2\%), and the mean reward of guided resampling is 0.72510.7251 (gain +0.3025+0.3025; trajectory improvement rate 88.2%88.2\%). The paired difference between the two conditions at the anchor level is +0.1045+0.1045 (task-clustered bootstrap 95%95\% CI: [+0.0638, +0.1468][+0.0638,\, +0.1468]). We have added this experiment to Section 5.9 of the paper.

Q2: Are old-policy log probabilities in Eq. (6) cached or recomputed without hints

We thank the reviewer for pointing out this detail. The old-policy log probability in Equation (6) is recomputed on hint-free sequences. To quantify its impact, we took all 171171 recovery trajectories from the final training step and performed paired teacher-forcing: for the same set of tokens, we computed log-probabilities under both the hint-conditioned context and the hint-free context, and took the token-wise difference. The results are shown in the table below.

ScopeMean Δ\DeltaMean ∣Δ∣\lvert\Delta\rvertp95(∣Δ∣)p_{95}\left(\lvert\Delta\rvert\right)Fraction with ∣Δ∣>0.5\lvert\Delta\rvert > 0.5
All recovery tokens−0.045-0.0450.1210.1210.5090.5095.10%5.10\%
1st action (first corrective action)−0.230-0.2300.4040.4042.052.0514.40%14.40\%
2nd action−0.065-0.0650.180.180.9380.9388.00%8.00\%
3rd action−0.042-0.0420.1410.1410.6530.6536.20%6.20\%
≥4\ge 4th action−0.011-0.0110.0650.0650.3160.3163.10%3.10\%

The differences are concentrated at the first corrective action and decay rapidly; Moreover, most tokens are barely affected, and the tokens that are substantially rewritten are mainly a small number of key decision tokens. We have added this experiment to Section 5.10 of the paper.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer yZ8Y,

We have revised the paper and added additional results to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of ReflectPO

Thank you for the additional experiments, I will raise my rating to 3.

Thank you for your constructive feedback and for raising your rating! We will incorporate the new results into the revised version and continue proofreading the manuscript to improve its clarity:)

Reviewer_CpFbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReflectPO, a reinforcement learning framework for improving autonomous error recovery in long-horizon LLM agents. It consists of two main components: Reflection-Guided Sampling and Tree-Structured Credit Assignment. The sampling module identifies potentially problematic execution points through cross-trajectory branch comparisons and repeated interaction patterns, then uses an external guide model to generate recovery hints and sample additional trajectories. The credit assignment module estimates advantages for original trajectories using parent-child reward differences and assigns recovery advantages based on the reward improvement over the corresponding failed trajectories. Experiments on BFCL-v3 Multi-Turn and τ²-Bench with Qwen3-4B and Qwen3-8B demonstrate improvements over several RL baselines, with ablations supporting the contributions of the proposed components.

Critical strengths

The proposed method demonstrates innovation and effectiveness. ReflectPO introduces a novel framework that combines reflection-guided recovery sampling with tree-structured credit assignment to improve the error recovery capabilities of long-horizon LLM agents. By leveraging complementary recovery strategies and more fine-grained learning signals, the method provides an effective approach to addressing execution errors in multi-step agent interactions. Experimental results on both BFCL-v3 Multi-Turn and τ²-Bench demonstrate consistent improvements over existing methods across different model scales, achieving the best overall performance among the evaluated baselines.

Critical weaknesses

1.The effectiveness of combining multiple components is not sufficiently demonstrated. The full ReflectPO achieves only a marginal improvement over the contrast-only variant (45.36 vs. 45.27 in overall performance). Although combining both sampling strategies improves performance on τ²-Bench, it slightly degrades performance on BFCL-v3 Multi-Turn. This raises concerns about whether introducing multiple sampling components provides meaningful additional benefits or introduces unnecessary complexity. The authors should further justify the necessity of each component and demonstrate their complementary contributions. 2.The claimed autonomous error detection capability is not sufficiently validated. Although the paper reports overall benchmark performance and LLM-judge-based error recovery metrics, it does not directly establish that the trained agent can independently recognize its own errors and decide when to initiate recovery at inference time. Improvements in overall benchmark scores do not necessarily imply that the model has learned autonomous error detection. More direct evaluations are needed to demonstrate that the proposed training method genuinely improves this capability and consequently leads to higher task completion success.

Questions for authors

1.The full ReflectPO achieves only a marginal overall improvement over the contrast-only variant. Could the authors provide further ablation studies and statistical significance analyses to demonstrate that combining contrast-based and loop-based sampling provides consistent and complementary benefits? In particular, why are the additional components necessary rather than redundant? 2.How do the authors verify that ReflectPO genuinely enables agents to autonomously detect their own errors and decide when to initiate recovery, rather than simply improving general task-solving performance? Could the authors provide more direct evaluations, such as task completion success rates under erroneous execution conditions, to demonstrate that the trained agent can independently identify and recover from errors without external guidance at inference time?

1 comment
AuthorsOpenReview ↗

Response for Reviewer CpFb

Q1: Complementarity and Significance of Contrast- and Loop-Based Sampling

Thank you for this important concern. The purpose of introducing the loop-based variant is to help the agent break out of repetitive execution cycles and reduce the number of unproductive loops. To evaluate the effectiveness of this component, we measure the occurrence of repetitive loops in the evaluation trajectories under four settings: the Qwen3-8B base model, the model with only the contrast-based variant, the model with only the loop-based variant, and the full model incorporating both variants. Specifically, we construct a window node using the most recent two action–observation pairs. Revisiting the same window node is counted as one loop occurrence. The average number of loop occurrences per trajectory is reported below.

MethodAvg. Loop Occurrences per Trajectory (BFCL)Avg. Loop Occurrences per Trajectory (τ2\tau^2)Overall
Qwen3-8B0.20.0250.155
Contrast-only0.1310.0140.101
Loop-only0.0240.0180.022
ReflectPO0.0280.0070.022

The results show that the distribution of repetitive loops is largely consistent with our expectations. The base model exhibits the highest overall loop frequency. Although the contrast-only variant provides some improvement, its overall average number of loop occurrences remains relatively high at 0.101. In contrast, both the loop-only variant and the full ReflectPO model substantially reduce this metric to 0.022. These results demonstrate that loop-based sampling effectively mitigates repetitive looping behavior, supporting its role in helping agents escape unproductive execution cycles. We have added this experiment and the corresponding analysis to Section 5.6 of the revised paper.

Q2: Direct Verification of Autonomous Error Detection and Recovery

Thank you for this important concern. We conduct an additional experiment by manually injecting errors into the execution trajectories generated by ReflectPO-8B. Specifically, we consider three types of errors: parameter value perturbation or substitution, parameter omission, and repetitive invocation of read-only tools. And we use the ReflectPO-8B model to continue the reasoning process from the points where errors were injected, and then we recalculate the overall accuracy as well as the recovery rate from the injected errors. The results are as follows:

Methodτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
ReflectPO4646.524.637.45545.553.538.548.145.4
Performance after recovery42.352.926.340.151.742.35439.746.945.2
Recovery Rate (%)7570.873.77269.672.969.872.670.971.1

As shown in the results, ReflectPO demonstrates strong error recovery capability even when its original execution trajectories are deliberately corrupted. These findings suggest that ReflectPO enhances the agent's ability to autonomously detect and correct errors during execution.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response for Reviewer pPco 1/2

Q1: Guidance and compute controls

Thank you for this important concern. We conduct an additional ablation experiment. Specifically, we retain the same contrast-based sampling and loop-based sampling strategies while replacing our proposed credit assignment mechanism with the standard Reinforce++ algorithm. We then retrain and evaluate the resulting model, denoted as ReflectPO-R++. The experimental results are reported below.

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
Qwen3-8B324324.833.647.536.5483541.839.6
ReflectPO-R++46.949.117.535.85141.55141.546.343.5
ReflectPO4646.524.637.45545.553.538.548.145.4

The results show that replacing our tree-structured credit assignment mechanism with conventional trajectory-level reinforcement learning leads to a moderate performance degradation but it still outperforms the Qwen3-8B baseline. We have added the results of this experiment to Table 2 of the revised paper. We further analyze the computational overhead introduced by the use of the guiding model during training. Taking training step 210 as an example, the guiding model is invoked 156 times, consuming 573,729 tokens in total, while the overall rollout process consumes 5,426,021 tokens. Therefore, the token consumption of the guiding model accounts for only 10.5% of the total token consumption. We have added this analysis to Section 5.7 of the revised paper.

Q2: Validity of recovery-anchor selection

Thank you for this important concern. We conduct an additional experiment by manually injecting errors into the execution trajectories generated by ReflectPO-8B. Specifically, we consider three types of errors: parameter value perturbation or substitution, parameter omission, and repetitive invocation of read-only tools. And we use the ReflectPO-8B model to continue the reasoning process from the points where errors were injected, and then we recalculate the overall accuracy as well as the recovery rate from the injected errors. The results are as follows:

Modelτ2\tau^2-Bench (Airline)τ2\tau^2-Bench (Retail)τ2\tau^2-Bench (Telecom)τ2\tau^2-Bench (Overall)BFCL-v3 (Base)BFCL-v3 (Long-Context)BFCL-v3 (Miss-Func)BFCL-v3 (Miss-Param)BFCL-v3 (Overall)Overall
ReflectPO4646.524.637.45545.553.538.548.145.4
Performance after recovery42.352.926.340.151.742.35439.746.945.2
Recovery Rate (%)7570.873.77269.672.969.872.670.971.1

As shown in the results, ReflectPO demonstrates strong error recovery capability even when its original execution trajectories are deliberately corrupted. These findings suggest that ReflectPO enhances the agent's ability to autonomously detect and correct errors during execution.

AuthorsOpenReview ↗

Response for Reviewer pPco 2/2

Q3: Autonomous recovery and sampler contributions

Thank you for this important concern. To further validate the reliability of our LLM-based evaluation of error occurrence and recovery, we conduct three additional experiments.

  • We independently repeat the evaluation using GPT-5.6 Terra as the judge model. Due to computational cost constraints, we evaluate each sample five times and report the mean and standard deviation of the resulting evaluation metrics.
  • We replace the judge model with Gemini 3.8 Flash and repeat the evaluation under the same experimental settings to investigate the results.
  • Due to the associated annotation costs, we randomly sample 30 trajectories from each benchmark for manual evaluation. We employ three trained annotators to assess whether the trajectories generated by the baseline and our method exhibit recovery behaviors.

The results are presented below. The results demonstrate that the variance across the three repeated evaluations is relatively small. More importantly, ReflectPO consistently achieves a lower error step rate and a higher step recovery rate than the baseline across all three evaluation settings. This consistency provides further evidence of the effectiveness of ReflectPO. We have incorporated this additional experiment and the corresponding analysis into Section 5.8 of the revised paper.

JudgeMethodError Step Rate ↓Step Recovery Rate ↑
GPT-5.6 TerraQwen3-8B19.39 ± 0.1517.58 ± 0.24
GPT-5.6 TerraReflectPO15.19 ± 0.0621.43 ± 0.04
Gemini 3.8 FlashQwen3-8B14.9425.81
Gemini 3.8 FlashReflectPO10.9127.31
HumanQwen3-8B17.817.7
HumanReflectPO11.5221.33

The purpose of introducing the loop-based variant is to help the agent break out of repetitive execution cycles and reduce unproductive looping behaviors. To evaluate the effectiveness of this component, we measure the occurrence of repetitive loops in the evaluation trajectories under four settings: the Qwen3-8B base model, the model with only the contrast-based variant, the model with only the loop-based variant, and the full ReflectPO model incorporating both variants. Specifically, we construct a window node using the most recent two action–observation pairs. Revisiting the same window node is counted as one loop occurrence. The average number of loop occurrences per trajectory is reported below.

MethodAvg. Loop Occurrences per Trajectory (BFCL)Avg. Loop Occurrences per Trajectory (τ2\tau^2)Overall
Qwen3-8B0.20.0250.155
Contrast-only0.1310.0140.101
Loop-only0.0240.0180.022
ReflectPO0.0280.0070.022

The results show that the distribution of repetitive loops is largely consistent with our expectations. The base model exhibits the highest overall loop frequency. Although the contrast-only variant provides some improvement, its overall average number of loop occurrences remains relatively high at 0.101. In contrast, both the loop-only variant and the full ReflectPO model substantially reduce this metric to 0.022. These results demonstrate that loop-based sampling effectively mitigates repetitive looping behavior, supporting its role in helping agents escape unproductive execution cycles. We have added this experiment and the corresponding analysis to Section 5.6 of the revised paper.

Q4: Positioning relative to step-level self-critique and tree-based agent learning

We thank the reviewer for their valuable comments. Our prefix tree is assembled post hoc from a batch of natural rollouts; it is not searched over at decision time. In addition, we investigate how an agent can break out of failure loops, which is another point of novelty relative to SLSC-MCTS. As shown by the results in our response to Question 3, ReflectPO also effectively teaches the agent to escape such loops. Our hints exist only during the training phase: they are explicitly removed at the update step, and no hints are provided at inference time. Our goal is to internalize this reflective self-correction capability into the model parameters. We have added this comparison to Section 2 of the paper.

AuthorsOpenReview ↗

General Reponse of ReflectPO

General Response of ReflectPO

We thank all reviewers and the Area Chair for their constructive feedback. We are glad our core contributions were recognized: error propagation in long-horizon tool use is an important, realistic problem (CpFb, pPco, yZ8Y, riPy, pWhG); recovery data comes from failures arising naturally in the policy's own rollouts, needing no manual annotation and reusing saved snapshots (pPco, yZ8Y, riPy); contrast- and loop-based sampling target complementary patterns, and tree-structured credit assignment separates local action credit from recovery gain (CpFb, pPco, riPy, pWhG); and both Qwen3-4B and 8B improve consistently over multiple RL baselines (CpFb, yZ8Y, riPy, pWhG).

Concerns center on gain attribution, autonomous recovery at inference, judge reliability, the objective after hint removal, and cost. We added controlled experiments accordingly (Qwen3-8B; Overall sample-weighted).

Focus AreaReviewerRevised SectionOur Actions and Results
1. Hint controls under matched anchors and rollout countsriPy, pWhGTable 2Hints removed: Overall 44.0 vs. ReflectPO 45.4 (τ² 35.8 vs. 37.4). Generic hint: 43.9, but τ² collapses to 29.7, below the untrained 33.6 (Telecom 13.2 vs. 24.8). Both beat Reinforce++ (41.3) yet fall short; generic hints help only on BFCL and hurt dual-control settings
2. Guided vs. unguided retriesyZ8Y, riPy§5.9Four continuations each, same snapshot, prefix, and decoding: target 0.4226, unguided 0.6206 (+0.1980; 58.2% improved), guided 0.7251 (+0.3025; 88.2%); paired anchor-level difference +0.1045 (task-clustered bootstrap 95% CI [+0.0638, +0.1468])
3. Independent contribution of tree creditpPco, pWhGTable 2 / §5.7Sampling fixed, credit assignment replaced by trajectory-level Reinforce++: Overall 43.5 vs. 45.4 (τ² 35.8 vs. 37.4; BFCL 46.3 vs. 48.1) — about +1.9 from tree credit alone
4. Necessity of loop-based samplingCpFb, pPco, riPy§5.6Loop occurrences per trajectory: 0.155 (Qwen3-8B) → 0.101 (contrast-only) → 0.022 (ReflectPO; 0.007 on τ²). The Overall gap from contrast-only is 0.09, but the behavioral change is explicit
5. Autonomous recovery at inferenceCpFb, pPco, riPy, pWhG§5.3 / camera-ready(a) Errors injected into ReflectPO-8B trajectories, continued without hints: τ² recovery 72.0%, BFCL subsets 69.6–72.9%. (b) All models continued from the same first-error states: Overall 43.1 vs. Reinforce++ 41.0; human-annotated recovery 35.3% vs. 31.8% vs. 30.1%. Shared states make recovery rates comparable across models
6. Judge reliabilitypPco, riPy, pWhG§5.8Five repetitions with GPT-5.6 Terra (std ≤ 0.24), re-evaluation with Gemini 3.8 Flash, and three trained annotators (30 trajectories per benchmark). ReflectPO wins under all three (human: error-step 11.52 vs. 17.8; recovery 21.33 vs. 17.7)
7. Old-policy probabilities after hint removalyZ8Y, pWhG§5.10 / §4.3Recomputed on hint-free sequences — a deliberate approximation. Mean Δ is −0.045 overall, concentrated at the first corrective action (−0.230); 90% of exact importance weights lie in [0.78, 1.16], median indistinguishable from 1. Exact IS would attenuate positive-advantage corrective tokens (mean weight 0.835; 11.3% below 0.1) while amplifying the opposite sign to 3.115 — asymmetric and high-variance. Off-policyness is controlled by the KL term (β = 0.001) and one update per batch
8. Guide-model costpPco, riPy, pWhG§5.7At step 210 the guide model is invoked 156 times for 573,729 tokens — 10.5% of that step's rollout tokens (5,426,021). Inference uses no hint and no extra model
9. Overall definition and related workriPy, pPco§5.1 / §2Overall is sample-weighted: (800 × BFCL + 278 × τ²) / 1078. Versus SLSC-MCTS: our prefix tree is built post hoc from natural rollouts and never searched at decision time; we additionally handle loops; hints exist only in training
10. Stability across seedsriPy, pPcocamera-readyWe will report results across random seeds in the camera-ready version

Together these controls support one conclusion: the gain comes neither from more rollouts nor from the presence of a hint, but from anchoring the hint at the specific failure and separating local credit from recovery gain. Hint-free resampling and generic hints both beat Reinforce++ yet fall below the full method, and removing tree credit costs 1.9. The human-annotated recovery rate on shared first-error states (35.3% vs. 30.1%) further indicates the improvement occurs at the point of failure, not only in forward execution accuracy.

We have also replied to Reviewer pWhG's follow-up on importance sampling. If these results resolve the main concerns, we respectfully ask for a more favorable reassessment, and look forward to continuing the discussion.

Best regards, The Authors of ReflectPO

Area_Chair_WFnLOpenReview ↗

Check Authors' Response

Dear reviewers,

The authors have posted their responses. Please read them and update your scores if they address your concerns. If they don't change your view, a short note on what remains unresolved would help the discussion.

Thank you for your time.
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.