- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates exactly which capabilities of large language model agents are enhanced by multi-turn reinforcement learning. Through the Pass@(K, T) metric and step-level evaluation, the authors found that the benefits of reinforcement learning depend largely on the interaction budget, and that these improvements are more evident in error recovery and long-term task execution capabilities than in the quality of individual actions. To address this limitation, the authors propose BudgetMix RL, which enhances the model’s robustness under different interaction constraints by randomizing the interaction budget during training. Experiments on benchmarks validate its effectivenss.
Critical strengths
s1: The observation that multi-turn RL can improve trajectory-level success while degrading average step-level action quality seems interesting and practical. s2: BudgetMix is simple and compatible with multiple RL algorithms, achieving consistent gains under constrained interaction budgets.
Critical weaknesses
w1: The improved performance on error states does not establish that error recovery is the primary driver of trajectory-level gains. w2: The accumulation of likelihood ratios across turns is an expected consequence of policy divergence and does not necessarily indicate improved long-horizon coordination.
Questions for authors
q1: Can you quantify how much of the trajectory-level improvement is attributable to error recovery? q2: Can the authors provide additional evidence that the likelihood-ratio patterns reflect improved long-horizon coordination rather than simply accumulated policy divergence? q3: Does BudgetMix improve step-level decision quality, or primarily come from more efficient interactions and fewer unnecessary retries?
4 comments
Response 2
Question 1: Quantifying the Contribution of Error Recovery
Can you quantify how much of the trajectory-level improvement is attributable to error recovery?
During the rebuttal, we conduct a manual attribution analysis of tasks where RL succeeds but Base fails. The attribution categories and quantitative results are presented in our response to Weakness 1.
Question 2: LLR Interpretation
Can the authors provide additional evidence that the likelihood-ratio patterns reflect improved long-horizon coordination?
As explained in our response to Weakness 2, our main point is precisely that policy divergence accumulates over long trajectories, leading to substantial differences in trajectory-level generation probabilities. As a result, even when the likelihoods of individual actions are similar, the probability of generating an entire successful trajectory can still be very low in long-horizon agentic tasks.
Question 3: Does BudgetMix Improve Step-Level Action Quality?
Does BudgetMix improve step-level decision quality?
Interestingly, BudgetMix not only encourages more efficient interactions under constrained budgets but can also implicitly improve step-level decision quality. We would like to highlight that our current paper already provides several pieces of evidence for this finding, both in the main text and appendix.
-
Training Dynamics Analysis: In Sec. 5.4 and Fig. 8, we compare the training dynamics of BudgetMix RL and conventional Full-Budget RL. We track both trajectory-level rewards and step-level rewards, where the latter reflects changes in step-level decision quality during training. While conventional RL exhibits a gradual decline in step-level rewards, BudgetMix RL maintains and even slightly improves step-level action quality. This reveals how action quality evolves during training. Additional results in Appendix E.3 (Fig. 24) show similar patterns.
-
Direct Step-Level Action Quality Evaluation: We also directly evaluate the step-level action quality of the final BudgetMix RL model. Appendix E.3 (Figs. 21 and 22) presents experiments following the same setup as Sec. 3.3. Using the same state pools, we compare the action quality of BudgetMix RL on randomly matched states and error states. The results show that, unlike conventional RL, BudgetMix RL shows clear improvements in action quality on randomly matched states while also enhancing error recovery capabilities relative to Base.
Together, these results demonstrate that BudgetMix RL can indeed improve step-level action quality, beyond simply increasing interaction efficiency or reducing unnecessary retries.
[Final Remark] Thank you again for your constructive feedback and valuable suggestions. We hope our detailed responses and the additional clarifications provided above can effectively address your concerns and further strengthen our findings. We sincerely welcome you to engage with us during the discussion phase should you have any further questions.
Revised Paper Uploaded
Dear Reviewer VgrP,
Thank you once again for your constructive suggestions. In response to your feedback, we have updated the paper to include a quantitative attribution analysis of RL's trajectory-level gains and further clarifications.
Best regards,
Authors
A Gentle Follow-up as the Discussion Period Draws to a Close
Dear Reviewer VgrP,
We sincerely appreciate your time and thoughtful feedback throughout the review process. We wanted to follow up on our earlier rebuttal and would be grateful to hear whether our responses have adequately addressed your concerns. We sincerely hope you could consider raising your score based on our further experiments and clarifications. Thank you again for your time and support!
If there are any remaining questions or points you would like us to clarify, we would be very happy to discuss them further.
Best regards,
Authors
Response 1
Dear Reviewer VgrP,
We sincerely appreciate your constructive feedback and the time you dedicated to reviewing our work. We are very glad you found our core observation to be both interesting and practical. We also appreciate your positive assessment of BudgetMix, particularly its simplicity, effectiveness and compatibility with multiple RL algorithms. Below, we carefully address your concerns and questions, supported by additional analyses and experimental evidence.
Weakness 1: Contribution of Error Recovery to Trajectory-Level Gains
Thank you for this constructive suggestion. We would first like to clarify that our main conclusion is not that error recovery is the primary source of RL's trajectory-level gains. Rather, our finding is that RL does not consistently improve step-level action quality across all states, but its advantage becomes more pronounced in error states. Leveraging this advantage often requires sufficient interaction budgets.
Nevertheless, your suggestion is valuable. During the rebuttal phase, we conduct a manual attribution analysisof RL and Base trajectories to better understand the sources of RL's performance gains . Specifically, we select tasks where RL succeeds but Base fails, and compare their trajectories to identify the main reasons for the improvement. We categorize the gains into five types:
Based on these categories, we analyze the collected trajectory pairs. As shown below, error recovery accounts for the largest proportion of performance gains, followed by long-horizon execution and error avoidance.
Attribution of RL Trajectory-Level Gains
Weakness 2: Interpretation of Cumulative LLR
Thank you for your comment. This might be actually a misunderstanding, which we will clarify below.
In fact, your observation is consistent with what we aim to convey in Sec. 4.1. We would like to clarify that our likelihood analysis is not intended to establish stronger long-horizon coordination through cumulative LLR. Instead, it aims to explain an interesting phenomenon: the fact that individual actions are supported by the Base policy does not mean that Base can reliably sample the entire successful trajectory.
Specifically, Fig. 7 and Appendix Fig. 23 show that RL-generated actions have NLL values under the Base policy comparable to those of Base's own actions. This suggests that these local actions remain within the range of actions that Base model can generate. However, at the trajectory level, cumulative LLR reveals that the relative likelihoods assigned by the two policies can diverge substantially as interactions progress. Thus, being able to generate individual actions and reliably sampling an entire successful trajectory within a limited sampling budget are two different things.
This difference arises precisely from the long-horizon nature of the trajectories. The accumulation of policy divergence across turns, as you pointed out, is exactly the mechanism we use to explain this phenomenon. Even without substantial changes in the likelihood of individual actions, the probabilities of generating complete successful trajectories can differ significantly between the two policies.
This provides a possible probabilistic explanation for the Pass@K results in Sec. 3.1: RL can achieve substantial trajectory-level performance gains under limited sampling budgets without consistently producing better actions at every step.
Therefore, the main purpose of this analysis is to explain why local action-level behavior and overall trajectory-level performance can differ, rather than treating cumulative LLR itself as a measure of long-horizon coordination.