acceptodds
Under review as a conference paper at ICLR 2027

What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

Abstract

While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth and interaction depth via , we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, LLM Agents, Long-Horizon Agents, Environment Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VgrPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates exactly which capabilities of large language model agents are enhanced by multi-turn reinforcement learning. Through the Pass@(K, T) metric and step-level evaluation, the authors found that the benefits of reinforcement learning depend largely on the interaction budget, and that these improvements are more evident in error recovery and long-term task execution capabilities than in the quality of individual actions. To address this limitation, the authors propose BudgetMix RL, which enhances the model’s robustness under different interaction constraints by randomizing the interaction budget during training. Experiments on benchmarks validate its effectivenss.

Critical strengths

s1: The observation that multi-turn RL can improve trajectory-level success while degrading average step-level action quality seems interesting and practical. s2: BudgetMix is simple and compatible with multiple RL algorithms, achieving consistent gains under constrained interaction budgets.

Critical weaknesses

w1: The improved performance on error states does not establish that error recovery is the primary driver of trajectory-level gains. w2: The accumulation of likelihood ratios across turns is an expected consequence of policy divergence and does not necessarily indicate improved long-horizon coordination.

Questions for authors

q1: Can you quantify how much of the trajectory-level improvement is attributable to error recovery? q2: Can the authors provide additional evidence that the likelihood-ratio patterns reflect improved long-horizon coordination rather than simply accumulated policy divergence? q3: Does BudgetMix improve step-level decision quality, or primarily come from more efficient interactions and fewer unnecessary retries?

4 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer VgrP,

We sincerely appreciate your constructive feedback and the time you dedicated to reviewing our work. We are very glad you found our core observation to be both interesting and practical. We also appreciate your positive assessment of BudgetMix, particularly its simplicity, effectiveness and compatibility with multiple RL algorithms. Below, we carefully address your concerns and questions, supported by additional analyses and experimental evidence.

Weakness 1: Contribution of Error Recovery to Trajectory-Level Gains

The improved performance on error states does not establish that error recovery is the primary driver of trajectory-level gains.

  • Thank you for this constructive suggestion. We would first like to clarify that our main conclusion is not that error recovery is the primary source of RL's trajectory-level gains. Rather, our finding is that RL does not consistently improve step-level action quality across all states, but its advantage becomes more pronounced in error states. Leveraging this advantage often requires sufficient interaction budgets.

  • Nevertheless, your suggestion is valuable. During the rebuttal phase, we conduct a manual attribution analysisof RL and Base trajectories to better understand the sources of RL's performance gains . Specifically, we select tasks where RL succeeds but Base fails, and compare their trajectories to identify the main reasons for the improvement. We categorize the gains into five types:

    1. Error Recovery: Both models encounter the same or similar errors, but RL successfully recovers while Base fails to do so.
    2. Error Avoidance: RL avoids critical errors that cause Base to fail.
    3. Strategy Adaptation: RL model adopts a different solution strategy or switches to an effective alternative after the original approach fails.
    4. Long-Horizon Execution: The RL model continuously advances the task, whereas the Base model quits early, falls into invalid loops, or fails to utilize the budget reasonably.
    5. Others: Cases that do not clearly fit the above categories.

Based on these categories, we analyze the collected trajectory pairs. As shown below, error recovery accounts for the largest proportion of performance gains, followed by long-horizon execution and error avoidance.

Attribution of RL Trajectory-Level Gains

Error TypeProportion (%)
Error Recovery48.9
Error Avoidance19.5
Strategy Adaptation9.1
Long-Horizon Execution20.1
Others2.3

Weakness 2: Interpretation of Cumulative LLR

The accumulation of likelihood ratios across turns is an expected consequence of policy divergence and does not necessarily indicate improved long-horizon coordination.

Thank you for your comment. This might be actually a misunderstanding, which we will clarify below.

  • In fact, your observation is consistent with what we aim to convey in Sec. 4.1. We would like to clarify that our likelihood analysis is not intended to establish stronger long-horizon coordination through cumulative LLR. Instead, it aims to explain an interesting phenomenon: the fact that individual actions are supported by the Base policy does not mean that Base can reliably sample the entire successful trajectory.

  • Specifically, Fig. 7 and Appendix Fig. 23 show that RL-generated actions have NLL values under the Base policy comparable to those of Base's own actions. This suggests that these local actions remain within the range of actions that Base model can generate. However, at the trajectory level, cumulative LLR reveals that the relative likelihoods assigned by the two policies can diverge substantially as interactions progress. Thus, being able to generate individual actions and reliably sampling an entire successful trajectory within a limited sampling budget are two different things.

  • This difference arises precisely from the long-horizon nature of the trajectories. The accumulation of policy divergence across turns, as you pointed out, is exactly the mechanism we use to explain this phenomenon. Even without substantial changes in the likelihood of individual actions, the probabilities of generating complete successful trajectories can differ significantly between the two policies.

  • This provides a possible probabilistic explanation for the Pass@K results in Sec. 3.1: RL can achieve substantial trajectory-level performance gains under limited sampling budgets without consistently producing better actions at every step.

Therefore, the main purpose of this analysis is to explain why local action-level behavior and overall trajectory-level performance can differ, rather than treating cumulative LLR itself as a measure of long-horizon coordination.

AuthorsOpenReview ↗

Response 2

Question 1: Quantifying the Contribution of Error Recovery

Can you quantify how much of the trajectory-level improvement is attributable to error recovery?

During the rebuttal, we conduct a manual attribution analysis of tasks where RL succeeds but Base fails. The attribution categories and quantitative results are presented in our response to Weakness 1.


Question 2: LLR Interpretation

Can the authors provide additional evidence that the likelihood-ratio patterns reflect improved long-horizon coordination?

As explained in our response to Weakness 2, our main point is precisely that policy divergence accumulates over long trajectories, leading to substantial differences in trajectory-level generation probabilities. As a result, even when the likelihoods of individual actions are similar, the probability of generating an entire successful trajectory can still be very low in long-horizon agentic tasks.


Question 3: Does BudgetMix Improve Step-Level Action Quality?

Does BudgetMix improve step-level decision quality?

Interestingly, BudgetMix not only encourages more efficient interactions under constrained budgets but can also implicitly improve step-level decision quality. We would like to highlight that our current paper already provides several pieces of evidence for this finding, both in the main text and appendix.

  • Training Dynamics Analysis: In Sec. 5.4 and Fig. 8, we compare the training dynamics of BudgetMix RL and conventional Full-Budget RL. We track both trajectory-level rewards and step-level rewards, where the latter reflects changes in step-level decision quality during training. While conventional RL exhibits a gradual decline in step-level rewards, BudgetMix RL maintains and even slightly improves step-level action quality. This reveals how action quality evolves during training. Additional results in Appendix E.3 (Fig. 24) show similar patterns.

  • Direct Step-Level Action Quality Evaluation: We also directly evaluate the step-level action quality of the final BudgetMix RL model. Appendix E.3 (Figs. 21 and 22) presents experiments following the same setup as Sec. 3.3. Using the same state pools, we compare the action quality of BudgetMix RL on randomly matched states and error states. The results show that, unlike conventional RL, BudgetMix RL shows clear improvements in action quality on randomly matched states while also enhancing error recovery capabilities relative to Base.

Together, these results demonstrate that BudgetMix RL can indeed improve step-level action quality, beyond simply increasing interaction efficiency or reducing unnecessary retries.

[Final Remark] Thank you again for your constructive feedback and valuable suggestions. We hope our detailed responses and the additional clarifications provided above can effectively address your concerns and further strengthen our findings. We sincerely welcome you to engage with us during the discussion phase should you have any further questions.

AuthorsOpenReview ↗

Revised Paper Uploaded

Dear Reviewer VgrP,

Thank you once again for your constructive suggestions. In response to your feedback, we have updated the paper to include a quantitative attribution analysis of RL's trajectory-level gains and further clarifications.

Best regards,
Authors

AuthorsOpenReview ↗

A Gentle Follow-up as the Discussion Period Draws to a Close

Dear Reviewer VgrP,

We sincerely appreciate your time and thoughtful feedback throughout the review process. We wanted to follow up on our earlier rebuttal and would be grateful to hear whether our responses have adequately addressed your concerns. We sincerely hope you could consider raising your score based on our further experiments and clarifications. Thank you again for your time and support!

If there are any remaining questions or points you would like us to clarify, we would be very happy to discuss them further.

Best regards,
Authors

Reviewer_JBCfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studied what has been learned/improved during agentic RL for model's agentic task-solving capability. From the experimental results and the author claimed that: multi-turn RL does not uniformly improve individual decisions; its gains are more pronounced in error recovery and long-horizon execution. Both of such gains rely on longer interactions between the agent and env, thus the pass@(K, T) diminishes as T decreases, which can be lower than the base model for small T. To this end, the author proposed to randomly select budget for each group during RL training, so that model will be aware of the number of turns used and prone to interact shorter. This can be viewed as implicit length penalty, i.e., over the randomly selected budget will be cut off and assign reward 0. Algorithms with BudgetMix improves its performance compared to their canonical baseline.

Critical strengths

A substantial coverage of aspects related to the agentic model's capability is studied empirically. I learned new things about agentic RL behavior from the experimental findings in this work and the design of the experiments, such as Fig. 5 for checking the error-state conditioned action quality, is neat. I find Fig. 6 particularly interesting, the model after RL will gradually drive the states to the ones with higher value (value as in average action quality), while in contrast the base model's quality decreases. The proposed BudgetMix is an intuitive approach and shows improvements.

Critical weaknesses

I don't find major weakness in this work. There is a minor concern. From my perspective, the overall mathematics when consider whether an action is good quality or bad quality should be better based on performance difference lemma instead of directly measure of optimal value in FrozenLake. J(\pi_{rl}) - J(\pi_{base}) = E_{\pi_rl}[ A^{\pi_{base}}(h, a) ], note that the advantage is w.r.t. \pi_{base}, which gives a more precise characterization of the "improvement". But given A^{base} requires estimation unlike Q^* directly computable, I can also accept the current Q^* formulation.

Questions for authors

In the Fig. 2 pass@k experiments. Pass@k improvement not necessarily mean capability boundary characterization given K is not large enough to observe the convergence. Do you observe for how many tasks that with accuracy 0 but with accuracy > 0 after RL for a large K? It would also be good to check the number of failed attempts the model had before solving the task, as an indicator of the actual long-horizon problem-solving capability improvement or merely reducing early quit.

In Fig. 7, Base success&failure LLR are both positive. This is weird to me as the trajectories sampled from the base model but its likelihood is not as high as an "off-policy" RL model? Could you please verify this or explain a bit more on why it could happen statistically?

Since the BudgetMix is a way to regulate large number of turns, should direct number of turn penalty resolves the issue?

5 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer JBCf,

We sincerely appreciate your thorough review and highly encouraging feedback. We greatly appreciate your recognition of the breadth of our empirical analyses and the new insights they provide into agentic RL behavior. We are glad that you highlighted our experimental design for error-state action quality (Fig. 5) and the interesting findings (Fig. 6). We also appreciate your positive assessment of BudgetMix and your encouraging overall evaluation of our work. Below, we carefully address your questions and suggestions to further refine the paper.

Weakness: Performance Difference Lemma for Action Quality Evaluation

  • Thank you for this constructive suggestion. In Sec. 3.3, we use the exactly computable Q∗Q^* to measure absolute action quality, aiming for a direct and interpretable metric that does not require additional value estimation under a reference policy. We agree that the Performance Difference Lemma (PDL) provides a complementary perspective by characterizing relative improvements.

  • Following your suggestion, we further evaluate the relative advantage based on the PDL formulation during the rebuttal phase: Aπbase(h,a)=Qπbase(h,a)−Vπbase(h).A^{\pi_{\mathrm{base}}}(h,a) = Q^{\pi_{\mathrm{base}}}(h,a) - V^{\pi_{\mathrm{base}}}(h).

    Using the same state pools as in Sec. 3.3, we calculated the average relative advantage and the proportion of states where the RL advantage is positive. In the table below, the numbers in parentheses represent the proportion of states with a positive RL relative advantage:

PDL-Based Relative Advantage Analysis

State PoolFrozenLakeDeepCoder-MT
Matched States−0.009 (43.4%)−0.004 (46.1%)
Error States+0.081 (64.2%)+0.104 (66.5%)
  • The results indicate that on random matched states, the average relative advantage of RL actions is slightly negative, and approximately 44.8% states do not outperform the base model own actions. However, on error states, the relative advantage of RL actions is clearly positive. This aligns with our main finding in Sec. 3.3: multi-turn RL does not consistently improve action quality across all states, but exhibits stronger decision-making capabilities after errors occur. We will include this additional analysis in the revised manuscript. Thank you again for your valuable suggestion. We believe that combining absolute action quality evaluation based on Q∗Q^* with PDL-based relative improvement analysis provides more comprehensive evidence for our findings.
AuthorsOpenReview ↗

Response 2

Question 1: Empirical Capability Boundary

1. Empirical Capability Boundary

Pass@k improvement not necessarily mean capability boundary characterization given K is not large enough to observe the convergence.

  • We agree that Pass@K cannot perfectly characterize the true capability boundary. This is why we consistently refer to it as the empirical capability boundary throughout our paper. This is primarily based on the settings from previous works[1]. Due to computational and time constraints, it is impractical to increase K indefinitely. Instead, we use a relatively large K as an approximate measure to investigate this phenomenon.

... how many tasks that with accuracy 0 but with accuracy > 0 after RL for a large K?

  • Following your suggestion, we further analyze the overlap between the sets of tasks solvable by RL and Base model at a larger K during the rebuttal phase. Specifically, we use K=512 to strengthen the empirical characterization. The table below reports the percentages of tasks with Pass@K > 0 for both models, only RL, and only Base model. We find that the proportion of tasks solvable only by RL remains substantially higher than that solvable only by base model across all benchmarks. This provides empirical evidence that RL extends beyond the empirical capability boundary of the base model.

Solvable Task Set Overlap at K=512 (%)

TaskBoth (%)RL-only (%)Base-only (%)
FrozenLake77.915.20.7
DeepCoder-MT55.117.21.5
InterCode-SQL40.922.10.9
BFCL-v331.37.31.1
Terminal-Bench17.28.40.6

2. Failed Attempts Before Success

It would also be good to check the number of failed attempts ... as an indicator of the actual long-horizon problem-solving capability ....

Following your suggestion, we also count the number of failed attempts before successfully solving a task. Specifically, we select tasks that both RL and Base eventually solve correctly, examine their trajectories, and annotate the failed attempts before success. As shown below, RL generally makes fewer failed attempts before success. This indirectly demonstrates that the RL model indeed possesses better error recovery capabilities and can ultimately solve tasks with fewer redundant retries. It is not merely a reduction in early quit.

Average Failed Attempts Before Success

TaskBaseRL
FrozenLake8.45.1
BFCL7.26.8

[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

AuthorsOpenReview ↗

Response 3

Question 2: Positive LLR on Base Trajectories

Base success&failure LLR are both positive. ... Could you please verify this or explain a bit more on why it could happen statistically?

  • Thank you for this insightful observation. This is a very interesting question. We agree with your intuition that, under ideal conditions, the expected LLR of Base-generated trajectories should be non-positive. According to the standard KL divergence property, if trajectories are sampled extensively from the original Base policy distribution and likelihoods are computed using the same original distributions, the expected LLR should be non-positive. Indeed, our BFCL analysis in Appendix Fig. 23 exhibits this expected behavior, with negative LLR values for Base-generated trajectories.

  • However, this theoretical expectation may not hold for finite empirical samples under different sampling conditions. Specifically, Fig. 7 uses a finite number of trajectories generated with Temperature=0.6 and Top-p=0.9, as specified in the appendix, while likelihoods are computed using the original model probability distributions. We also group trajectories by success and failure. Consequently, the empirical LLR estimates may deviate from the ideal KL expectation and can exhibit slightly positive values.

  • Furthermore, the confidence interval of the LLR analysis on Terminal-Bench is [−8.2, 31.4], which includes zero. This further helps explain the observed result. Compared with BFCL, Terminal-Bench involves much more complex and heterogeneous interaction processes, which can further increase the variance of this statistic.


Question 3: Direct Penalty on Interaction Turns

Since the BudgetMix is a way to regulate large number of turns, should direct number of turn penalty resolves the issue?

Thank you for this valuable question. In fact, we also explored directly penalizing interaction turns during our research, as this seems intuitive and easy to implement. However, contrary to our initial expectations, we found that the penalty coefficient is difficult to tune and can easily lead to reward hacking. Specifically:

  • Sensitivity to the penalty coefficient. If the penalty is too large, the model may prefer early termination over task completion to avoid penalties, leading to reward hacking. If it is too small, it fails to effectively encourage the model to complete tasks with fewer interactions.

  • Task-dependent tuning. In our preliminary tries, we found that different tasks may require different penalty coefficients. This introduces additional tuning for each new task and limits the transferability of the approach.

  • Difficulty in defining the optimal interaction length. Ideally, the penalty should apply only to interaction turns beyond the minimum required to solve a task, encouraging earlier completion when possible. In other words, it should penalize extra steps only when the task is solvable. However, determining this minimum number of turns is often difficult for realistic tasks.

Overall, BudgetMix is simpler to implement, more effective, and more stable during training. Moreover, BudgetMix can be readily adapted to and combined with existing RL algorithms (such as GiGPO or PPO) and various agentic tasks.

[Final Remark] Thank you again for your valuable feedback and constructive suggestions. We hope our responses and new experimental results have fully clarified your questions and strengthen our findings and address your questions. We would be more than happy to engage in further discussion if you have any additional questions.

Reviewer_UP98OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what capabilities multi-turn reinforcement learning (RL) improves in LLM agents by examining trajectory-level success, interaction-budget sensitivity, and step-level action quality. The authors find that RL gains become more pronounced with larger interaction budgets, while improvements in individual decisions are not consistent and appear concentrated in error recovery and later trajectory stages. Based on these observations, the paper proposes BudgetMix RL, which randomizes interaction budgets during training and demonstrates improved performance under constrained budgets across multiple benchmarks and RL algorithms.

Critical strengths
  1. The paper examines RL behavior through Pass@(K,T), matched-state action-quality comparisons, error recovery, and likelihood analysis. These complementary evaluations provide useful insights beyond conventional task success rates.
  2. BudgetMix consistently improves performance under tighter interaction budgets while generally preserving full-budget performance.
Critical weaknesses
  1. The step-quality analysis (Sec. 3.3) only covers FrozenLake and DeepCoder-MT, where each step can be scored. For tasks like SWE-bench or Terminal-Bench, actions such as reading files or running commands are much harder to evaluate, so the conclusion may not generalize.
  2. The authors attribute RL's gains to error recovery but never report how many turns RL actually uses. Since RL performs worse at small turn budgets (Fig. 3), the gains might simply come from taking more turns rather than making better decisions.
  3. BudgetMix is similar to existing approaches that vary interaction budgets during training[1-2]. Without fixed short-budget baselines or compute-matched comparisons, it is unclear whether mixing budgets actually provides additional benefits.

[1]. Time Limits in Reinforcement Learning
[2]. Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

Questions for authors
  1. The paper focuses on long-horizon agents, but DeepCoder-MT and InterCode-SQL can be solved in a single turn (L917–919), while FrozenLake involves simple actions. It would be more convincing to include step-quality, error-recovery, and BudgetMix results on Terminal-Bench, which better represents realistic long-horizon tasks.
  2. Figure 5 does not clearly report how many error states or continuation trajectories were sampled.
5 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer UP98,

We sincerely appreciate your careful review and constructive feedback. We are deeply encouraged by your positive remarks on our comprehensive experiments, including Pass@(K,T), action quality and log-likelihood analysis. We are especially glad that you recognize our work provides insightful conclusions beyond conventional task success rates, and that you acknowledge the effectiveness of BudgetMix. Below, we are glad to address your concerns point-by-point and provide additional experimental evidence to support our claims.

Weakness 1: Step-Level Action Quality Analysis on Terminal-Bench

The step-quality analysis only covers FrozenLake and DeepCoder-MT ... tasks like SWE-bench or Terminal-Bench ... the conclusion may not generalize.

  • First, we would like to clarify that FrozenLake and DeepCoder-MT were specifically selected in Sec. 3.3 to ensure our analysis is in-depth, controllable, and rigorous. Our main experiments in Secs. 3.1 and 3.2 already cover more realistic and challenging long-horizon benchmarks, including BFCL and Terminal-Bench (Figs. 2, 3, 14, and 18). Thus, our primary conclusions have been validated across diverse environments. Section 3.3 aims to provide a more in-depth analytical perspective on the main conclusion; therefore, a more controlled environment is chosen.

  • Second, we also analyze step-level decision on BFCL, a more realistic tool-use benchmark, in Appendix E.4. We find that models trained with agentic RL exhibit stronger error recovery capabilities when errors occur during tool calls and file-reading operations.

  • Finally, during rebuttal phase, we further extend our step-level action quality analysis to Terminal-Bench using a strong continuation policy to estimate the action value Q. In FrozenLake, Q* evaluates an action assuming optimal subsequent decisions. For Terminal-Bench, we use GPT-6-Astra as a strong continuation policy to approximate high-quality future decisions to evaluate the value of one step-level decision, while keeping the remaining hyperparameters consistent with our Q* evaluation in Sec. 3.3.

  • Based on this approach, we evaluate step-level action quality before and after RL on Terminal-Bench. As shown in the table below, RL does not consistently outperform base model on randomly matched states (win in only 43.1% of cases). However, on error states, RL shows substantially better action quality in 69.3% of cases, with a 9.4% improvement in continuation success rate. These results further support the generalizability of our findings in Sec. 3.3 to realistic long-horizon tasks.

Terminal-BenchBaseRLAQBase BetterRL Better
Matched States0.4350.428-0.00756.9%43.1%
Error States0.2560.342+0.08630.7%69.3%
Error State Continue Success Rate (%)15.224.6+9.4

Weakness 2: Interaction Turns and the Source of RL Gains

The authors attribute RL's gains to error recovery but never report how many turns RL actually uses.

Thank you for this valuable suggestion. Following your recommendation, we measure the average number of interaction turns before and after RL training, using Qwen3-4B on BFCL as an example.

  • Specifically, we categorize tasks based on whether both models succeed, only RL model succeeds, only base model succeeds, or both fail. As shown in the table below, RL and Base models generally use a similar number of interaction turns. When both models succeed, RL tends to use slightly more turns. However, when RL succeeds but Base fails, which directly corresponds to the "RL gains" you mentioned, RL actually uses fewer turns on average (17.2 vs. 20.3).

  • By examining the failed trajectories of base model, we find that base model often repeatedly attempts incorrect actions at the same failure point, whereas RL can recover from errors more quickly and eventually complete the task. These results indicate that the gains of RL are not simply due to taking more turns. We will incorporate these statistics into the revised paper to further strengthen our conclusions.

Average Interaction Turns (Qwen3-4B on BFCL):

ScenarioBaseRL
Both Success14.015.6
Only RL Success20.317.2
Only Base Success9.810.9
Both Failed19.716.4
AuthorsOpenReview ↗

Response 2

Weakness 3: Fixed short-budget baseline for BudgetMix

BudgetMix is similar to existing approaches that vary interaction budgets during training[1-2].

  • First, we would like to clarify that the primary goal of Sec. 5 is to discuss potential improvement strategies based on our core findings. Since the gains from standard agentic RL depend on sufficient interaction budgets, we investigate how to achieve more robust improvements. Specifically, we discuss process supervision and varying training budgets as two possible directions, aiming to provide initial analysis and insights.

  • Second, thank you for highlighting the relevant work, which we have actually cited and discussed in our paper. The core idea of TTI [1] is to start with short interaction trajectories and gradually increase the interaction horizon through curriculum learning, aiming to enable longer interactions. In contrast, BudgetMix focuses on achieving consistent improvements even under constrained interaction budgets. Thus, the two approaches differ in both their objectives and implementations.

Without fixed short-budget baselines or compute-matched comparisons ....

  • Third, we are very grateful for your suggestion. During the rebuttal phase, we have supplemented a fixed short-budget baseline. Specifically, we fix the training interaction budget at 50% of Tmax⁡T_{\max}, while keeping the amount of training data consistent. As shown in the table below, the Fixed Short-Budget RL outperforms Full-Budget RL under tighter interaction budgets, with an average improvement of 3.98% at the 50% budget. However, its performance at the full budget is worse than Full-Budget RL, with an average decrease of 4.30 %. More interestingly, Fixed Short-Budget RL still underperforms BudgetMix RL even at the 50% interaction budget. From our observations, models may need sufficient interaction budgets to explore possible solutions before learning effective problem-solving strategies. Training exclusively with short budgets may therefore hinder the learning of complex tasks. As discussed in Sec. 5.2, BudgetMix combines tighter budgets that discourage excessive redundant retries with larger budgets that help preserve long-horizon execution capabilities.

Fixed Short-Budget RL Baseline (Pass@8, %)

BenchmarkMethod25%50%75%100%
FrozenLake (Tmax⁡=70T_{\max}=70)Full-Budget RL28.755.168.172.4
Fixed Short RL (50%)33.859.664.265.9
BudgetMix RL37.661.172.073.5
Fixed Short - Full+5.1+4.5−3.9−6.5
BudgetMix - Full+8.9+6.0+3.9+1.1
DeepCoder-MT (Tmax⁡=10T_{\max}=10)Full-Budget RL29.248.056.059.0
Fixed Short RL (50%)34.951.853.954.6
BudgetMix RL38.154.360.561.2
Fixed Short - Full+5.7+3.8−2.1−4.4
BudgetMix - Full+8.9+6.3+4.5+2.2
InterCode-SQL (Tmax⁡=10T_{\max}=10)Full-Budget RL23.132.237.446.0
Fixed Short RL (50%)26.436.036.942.3
BudgetMix RL28.937.742.947.5
Fixed Short - Full+3.3+3.8−0.5−3.7
BudgetMix - Full+5.8+5.5+5.5+1.5
BFCL (Tmax⁡=24T_{\max}=24)Full-Budget RL2.517.627.030.2
Fixed Short RL (50%)5.821.425.927.6
BudgetMix RL8.523.030.530.8
Fixed Short - Full+3.3+3.8−1.1−2.6
BudgetMix - Full+6.0+5.4+3.5+0.6

[1] Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

AuthorsOpenReview ↗

Response 3

Question 1: Terminal-Bench results

  • First, we would like to clarify that, beyond the relatively simple tasks, our core experiments also cover BFCL and Terminal-Bench. These two are benchmarks for tool calling and terminal agents respectively, which are widely used in recent works of agents and better reflect realistic long-horizon settings. We conducted multiple experiments on these two benchmarks, including Pass@(K,T) evaluation, BudgetMix training, and mechanistic analysis, to enhance the reliability of our findings on realistic tasks.

  • Second, following your suggestion, we conduct additional experiments on Terminal-Bench during the rebuttal phase. We first extend our step-level action quality analysis to Terminal-Bench (see Weakness 1), validating that our conclusions can generalize to real long-horizon tasks. Furthermore, we supplemented the experimental results of BudgetMix RL on Terminal-Bench, as shown in the table below. The results demonstrate that BudgetMix RL consistently improves performance across different interaction budgets. In particular, it outperforms conventional Full-Budget RL by 3.9% and 4.2% at the 25% and 50% budgets, respectively.

BudgetMix RL on Terminal-Bench

Pass@8 (%)25%50%75%100%
Base6.110.011.012.0
Full-Budget RL6.011.414.716.0
BudgetMix RL9.915.616.116.4
BudgetMix - Full+3.9+4.2+1.4+0.4

Question 2: Number of Error States and Continuation Trajectories

Thank you for the reminder. We now supplement the statistical sample sizes used for this part of the analysis. For step-level action quality evaluation, we sample 16 actions per state and average their scores. For error recovery evaluation, we sample 4 complete continuation trajectories per error state and average their outcomes. The numbers of matched states and error states are reported below. We will include these details in the revised manuscript.

TaskMatched-States CountsError-States Counts
FrozenLake320300
DeepCoder-MT280260

[Final Remark] Thank you again for reviewing our paper and providing such meaningful suggestions. We hope the new experiments and results based on your suggestions can fully address your concerns. We sincerely invite you to engage in further discussion with us if you have any remaining questions.

Comments on the paper

2 comments

Further Experimental Results and Follow-up on Rebuttal

Dear Reviewer UP98,

Thank you once again for your constructive feedback and valuable time. As the discussion period draws to a close, we would like to provide one final update on our experimental results.

1. Our Previous Rebuttal: Sincerely Addressing Your Concerns with Additional Evidence

In our previous rebuttal and revised paper, we have already provided additional experiments and clarifications directly addressing all three weaknesses and both questions you raised, including:

  1. W1: Extending step-level action-quality analysis to Terminal-Bench, showing substantially stronger RL performance on error states (69.3% RL wins; +9.4 percentage points in continuation success).
  2. W2: Reporting interaction-turn statistics on BFCL, showing that RL's gains cannot be explained simply by taking more turns.
  3. W3: Introducing Fixed Short-Budget RL baselines across four benchmarks, demonstrating the benefits of BudgetMix over fixed short-budget training.
  4. Q1: Adding step-quality and BudgetMix experiments on Terminal-Bench, supporting our conclusions in realistic long-horizon settings.
  5. Q2: Explicitly reporting the numbers of matched/error states and continuation trajectories, together with the sampling protocols.

Over the past two days, we have further conducted two sets of additional experiments to strengthen our responses to W2 and W3, particularly on realistic long-horizon benchmarks.

2. Additional Interaction-Turn Statistics on DeepCoder-MT and Terminal-Bench (W2)

To further address your concern that RL's performance gains might simply result from taking more turns, we have extended our interaction-turn analysis from BFCL to DeepCoder-MT and Terminal-Bench.

Following the same evaluation protocol as our previous BFCL analysis, we categorize tasks into four groups based on the success or failure of the Base and RL models.

Average Interaction Turns:

BenchmarkScenarioBaseRL
DeepCoder-MTBoth Success5.25.6
Only RL Success8.07.0
Only Base Success5.56.1
Both Failed8.17.3
Terminal-BenchBoth Success26.925.7
Only RL Success41.238.5
Only Base Success22.320.1
Both Failed42.943.4

The new results are fully consistent with our previous findings on BFCL: Base and RL generally use comparable numbers of turns, and, crucially, RL uses fewer turns on tasks where it succeeds but Base fails. This consistent pattern across three benchmarks directly addresses your W2 concern: RL's trajectory-level advantages cannot be explained simply by taking more interaction turns.

3. Additional Fixed Short-Budget RL Baseline on Terminal-Bench (W3 & Q1)

To further strengthen our response to your concerns about BudgetMix and its generalizability to realistic long-horizon tasks, we have additionally conducted Fixed Short-Budget RL experiments on Terminal-Bench.

Following the same setting as our previous four-benchmark experiments, we fix the training interaction budget at 50% of Tmax⁡T_{\max}.

Terminal-Bench: Fixed Short-Budget RL Comparison (Pass@8, %)

Method25%50%75%100%
Base6.110.011.012.0
Full-Budget RL6.011.414.716.0
Fixed Short-Budget RL (50%)7.514.215.015.3
BudgetMix RL9.915.616.116.4
BudgetMix − Full+3.9+4.2+1.4+0.4
BudgetMix − Fixed Short+2.4+1.4+1.1+1.1

The results again confirm that BudgetMix consistently outperforms Fixed Short-Budget RL across all interaction budgets, including both constrained and full-budget settings. Together with our previous results on FrozenLake, DeepCoder-MT, InterCode-SQL, and BFCL, this provides consistent evidence across five benchmarks that mixing interaction budgets offers benefits beyond exclusively training with a fixed short budget, including on realistic long-horizon tasks such as Terminal-Bench.


In summary, we believe that our previous rebuttal and revised manuscript have already provided rich evidence for all the weaknesses and questions you raised. The two additional sets of experiments conducted over the past two days further strengthen these responses.

We sincerely appreciate the valuable suggestions that motivated these additional analyses.

We would be very grateful to hear whether you have any remaining questions. If you find that your original concerns have now been resolved, we would sincerely appreciate it if you could reconsider your initial rating in light of the substantial additional experimental evidence.

Thank you again for your time, consideration, and constructive feedback!

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Rebuttal, Revisions, and Discussion Status

Dear Area Chair, Senior Area Chair, and Program Chairs,

Thank you very much for your time and dedication in overseeing the review and rebuttal process of our paper. As the discussion period is drawing to a close, we would like to provide a concise summary of strengths the reviewers highlighted and our responses to assist with your final assessment and meta-review.


1. Summary of Core Strengths Recognized by Reviewers

Across all three reviews, there is a strong consensus on the novelty, empirical rigor, and practical value of our work:

  1. Insightful and Practical Findings Beyond Conventional Success Rates (UP98, JBCf, VgrP): All three reviewers recognized that our work uncovers fresh and practically important insights into multi-turn agentic RL. RL gains often relies on a substantial interaction budget, and it does not consistently improve the quality of every step-level action.
  2. Comprehensive and Well-Designed Evaluation Framework (UP98, JBCf): Reviewers praised the substantial empirical coverage and thoughtful experimental design, including our two-dimensional Pass@(K,T)\text{Pass@}(K,T) evaluation, matched-state and error-state action-quality analyses (Fig. 5), trajectory-depth dynamics (Fig. 6), and likelihood analysis.
  3. Simple, Intuitive, and Effective Mitigation via BudgetMix RL (UP98, JBCf, VgrP): All three reviewers highlighted BudgetMix RL as a simple, intuitive, and algorithm-agnostic approach (compatible with GRPO, GiGPO, and PPO) that consistently improves performance under tight interaction budgets while preserving full-budget capability.

2. Summary of Our Responses

We have carefully addressed every weakness and question raised across all reviews and incorporated the new results and clarifications into the revised paper:

  1. Strengthened Step-Level Action Analysis (UP98 W1 & Q1; JBCf W1): We extended our step-level action-quality evaluation to Terminal-Bench and supplemented a PDL-based relative-advantage analysis on FrozenLake and DeepCoder-MT. All new analyses consistently confirm our core finding: RL does not uniformly improve matched-state action quality, but delivers substantially stronger decisions and higher continuation success on error states.

  2. Mechanistic Attribution of RL Gains (UP98 W2; VgrP W1 & Q1; JBCf Q1): We measured average interaction turns, showing that on tasks where only RL succeeds, RL actually uses fewer turns, which rules out the hypothesis that gains come merely from more turns. We further added a manual attribution analysis (confirming error recovery as the largest driver of RL's trajectory-level gains), counted failed attempts before success (showing fewer redundant retries for RL), and evaluated solvable-task set overlap at K=512K=512 across all five benchmarks.

  3. New Baselines & Realistic-Task Validation for BudgetMix RL (UP98 W3 & Q1; JBCf Q3; VgrP Q3): We trained and evaluated a Fixed Short-Budget RL (50% Tmax⁡T_{\max}) baseline across all five benchmarks. Across all benchmarks, BudgetMix consistently outperforms both Full-Budget RL and Fixed Short-Budget RL across constrained and full budgets. We also clarified how mixing budgets differs from progressively increasing the training horizon, why direct turn penalties can encourage early stopping, and where existing evaluations show that BudgetMix preserves or improves step-level action quality.

  4. Clarifications on LLR & Experimental Protocols (VgrP W2 & Q2; JBCf Q2; UP98 Q2): We clarified that cumulative LLR is used to explain how modest per-step policy divergences compound across turns into large trajectory-level probability shifts, and explicitly documented all state counts and sampling protocols in the revised appendix.


3. Note on Discussion Status

We would like to respectfully draw your attention to the status of inactive Reviewer UP98 (Rating 2). Reviewer UP98 explicitly acknowledged our core contributions and evaluation insights, and all of the weaknesses and questions were concrete requests for additional experiments and details. We have completely resolved every single point of the concerns of reviewer UP98 in our rebuttal and revised manuscript. Besides, over the past two days, we further strengthened our responses to UP98 by adding two additional sets of experiments on realistic long-horizon benchmarks.

Despite our responses and follow-ups, Reviewer UP98 has not participated in the discussion since our rebuttal was posted. Because their original concerns have been comprehensively addressed with solid experimental results across five benchmarks, we kindly ask the AC, SAC, and PC to take our complete resolution of UP98's concerns into careful consideration when making the final decision.

Thank you once again for your valuable time, thoughtful consideration, and service to the ICLR community!

Best regards,

The Authors of Paper #5422

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.