- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies budget-aware RL training for multi-turn LLM agents. The authors compare different reward functions to improve task success and budget compliance under dynamically specified resource constraints. Experiments show generalization to unseen budgets and out-of-domain tasks, with simple gating rewards achieving competitive performance.
Critical strengths
- Important and practical problem. Budget-aware training for multi-turn agents is an important and practical problem that remains relatively underexplored compared to single-turn reasoning efficiency.
- Systematic comparison of reward designs. The paper provides a comprehensive evaluation of different reward objectives, with the finding that simple gating can achieve competitive performance being particularly interesting.
Critical weaknesses
- Lack of a stronger RL baseline. The improvements over the base model may partly come from general RL fine-tuning on AppWorld rather than budget-aware training. A standard RL baseline without budget penalties would help isolate the contribution.
- Limited generalization to arbitrary constraints. Separate policies are trained for each resource type, and only a limited range of unseen budgets is evaluated, weakening the claim of generalization to arbitrary test-time constraints.
- Weak budget adaptation for API calls. Figure 3(a) shows that API-call usage changes little across different budgets, suggesting that the model may learn a relatively fixed tool-use strategy rather than dynamically adapting to the given budget.
- Presentation needs improvement. The notation in Section 3.1 is confusing and difficult to follow. A clearer and more consistent presentation would improve readability.
Questions for authors
See Weaknesses
3 comments
Follow-up on new baselines and generalisation results
Thank you again for your constructive feedback. Your comments have been very helpful in strengthening the paper, and we've made significant additions and revisions in response your feedback.
In particular, your concern about distinguishing budget-aware learning from general RL improvements was important, and we've now addressed this directly with the new standard RL baseline (revised Fig. 3). The results provide stronger evidence for the benefits of budget-aware training, particularly under tighter budgets. We've also added BabyAI experiments and further out-of-distribution comparisons (revised Figs. 3 and 5) which include the new standard RL baseline in this context, improved the mathematical presentation, and clarified the scope of our claims.
We hope these additions help address your main reservations, and we'd be very interested to hear whether they have changed your assessment of the paper. We are very happy to clarify any remaining concerns and greatly appreciate your time and consideration in the review process.
Follow-up as the reviewer–author discussion period comes to a close
We appreciate the time you've taken to review our paper and for your helpful comments. As the reviewer–author discussion period draws to a close, we wanted to share a brief update with you. Following a suggestion from Reviewer txUo, we've now highlighted the revisions, additions, and new experiments in red throughout the updated manuscript on OpenReview to make them easier to locate and assess. Hopefully this makes checking our additions based on your suggestions easier now.
In particular, in response to your comments, we've added a standard RL baseline demonstrating the benefits of budget-aware training beyond general RL fine-tuning, particularly under tighter budgets (Fig. 3). We've strengthened the generalisation evidence with new BabyAI experiments introducing a different budgeted resource (turn budgets) and further out-of-distribution comparisons (Figs. 3 and 5). We've also expanded the analysis of API-call budgeting, showing improved within-budget utility and compliance over standard RL GRPO training despite API spending being somewhat less responsive to the stated budget and more closely tied to task difficulty, and discussing why this may be the case (Figs. 3, 4, and Section 4.1). Finally, we've simplified the notation and presentation in Section 3.1 and added a new diagram (Fig. 2) to provide a clearer overview of our setup.
We hope these additions help address the concerns in your initial assessment. We'd greatly value your updated thoughts on the paper and would be very happy to clarify any remaining points. Thank you once again for your time and consideration.
Thank you for recognising the practical importance of the paper and the value of our reward comparisons. Your comments have been very useful in strengthening the paper, and we have added experiments and improved the presentation to address your concerns.
W1: Standard RL baseline. We have added GRPO-Std, trained with the same setup as our budget-aware objectives, but rewarding task success regardless of budget compliance. While GRPO-Std improves over Base, the strongest budget-aware objectives achieve substantially higher within-budget utility and natural compliance, particularly at tighter budgets (revised Fig. 3). This demonstrates the additional benefit of budget-aware training over standard RL post-training, addressing the important distinction you raised between budget-aware learning and general improvements in task-solving ability from simply being exposed to the environment.
W2: Generalisation. We would like to clarify that our existing experiments already demonstrate both interpolation and extrapolation to budgets above and below those seen during training (revised Fig. 3). Our existing evaluations on AppWorld test-challenge and ASTRA provide further evidence under distribution shift, with harder tasks requiring more steps and a different task and tool domain with very different budget ranges, respectively. We have now added GRPO-Std comparisons to both evaluations, showing that the stronger budget-aware objectives retain higher within-budget utility and natural compliance than Base and standard RL training, even under these shifts, at these unseen budgets (revised Fig. 5).
We have also trained and evaluated on BabyAI, adding a fourth budget setting—turn budgets—in a different environment from the API-based tasks in AppWorld and ASTRA, where agents must balance batching navigation moves against the risk of mistakes. We observe similar benefits from budget-aware training over standard RL at both model scales (revised Fig. 3), further strengthening our evidence for generalisation.
Finally, our focus is on establishing whether agents can learn to manage dynamically specified budgets for different resources, which was not clear a priori, particularly for token budgets, and comparing which rewards best teach this behaviour. Training a single policy across resource types is an interesting and important extension that our framework could accommodate by conditioning on the resource type. However, this introduces further choices about which resources to train on, whether to specify one budget at a time or impose multiple constraints simultaneously, and how to sample budgets effectively during training. These choices depend on the downstream application, and learning to budget across multiple resource types could require significantly more training and compute. Given the scope of our experiments and space constraints, we believe this deserves a more thorough investigation than we can provide here. We thank the reviewer for this valuable suggestion and have highlighted it as an important direction for future work. We have also clarifed the scope of our claims in the final title of the paper which is now changed to ``Rewards for budget-aware multi-turn agents that adapt to dynamically specified budgets’’’ to reflect this.
W3: API-call adaptation. It is correct that Fig. 4(a) shows that API-call spending varies more strongly with task difficulty than with the stated budget. However, the same figure also shows that step and token spending increase with the permitted budget, even within each difficulty group. One possible explanation is that API calls are less flexible to budget, since tasks may require a minimum number of API interactions depending on their difficulty, whereas token and step usage can be reduced more flexibly through concise reasoning or batching. Regardless, budget-aware training still achieves higher within-budget utility and natural compliance than both Base and the newly added GRPO-Std baseline under API-call constraints (revised Fig. 3), demonstrating meaningful benefits over standard RL training even when API-call spending is less responsive to the stated budget.
W4: Presentation. We have addressed your concerns about Section 3.1 by revising and simplifying the mathematical notation and presentation of the framework. In addition, we have improved the notation table in Appendix A and added a new diagram of the budget-augmented POMDP (Fig. 2), providing an easy at-a-glance overview of how the agent interacts with the environment and receives budget feedback. We would be very happy to clarify anything else that remains unclear.
We thank the reviewer again for their constructive comments. We hope the new GRPO-Std comparisons and BabyAI experiments address the main concerns about budget-aware learning and generalisation, while the additional analysis and presentation changes help clarify our methodology and results. We are very happy to respond to any comments.