- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduce Allocbench, a benchmark for evaluating how LLM agents allocate tool-creation budget across sequential tasks with initially unknown recurrence frequencies. It compares four settings, ranging from abstract allocation to executable script construction, using the same underlying task sequences.
Critical strengths
- Well motivated study concerning a practical aspect of current LLM agents’ behavior: deciding when to invest in reusable tools under a limited budget and uncertain future demand.
- The method contributes a controlled experimental framework for measuring the capabilities of tool allocation, under the joint view of budget and resulting benefit.
Critical weaknesses
- Although acknowledged in the limitations, the synthetic environment is narrow, consisting of eight predefined mathematical task families with each script bound to an individual class. This restricts the variety of relationships between tasks and reusable tools. It remains unclear how well performance on AllocBench reflects allocation ability in settings involving tools with broader functionality, overlapping capabilities, or less structured task distributions. Evaluation across more diverse tool-use scenarios would strengthen its relevance as a benchmark for practical agents.
- Only small fraction of frontier models are tested, and both GPT models used are tested without additional reasoning, leaving the sensitivity of allocation behavior to reasoning budget unexplored. Also, leaving SOTA open-source model’s performance unclear. Broader model coverage and reasoning-enabled evaluations would help establish a more comprehensive assessment of tool-allocation capabilities.
Questions for authors
- How sensitive are the benchmark results to the task-stream construction, particularly the requirement that a low-frequency class appear among the first three tasks? Do the observed framing gaps and relative model performance remain consistent on streams sampled without this constraint, or with less sharply separated recurrence frequencies?
- Could the authors supplement the current results with a comparison of task utility and actual token consumption over matched task streams in the script-construction setting? This would help assess whether better allocation of tool-creation slots is associated with improved resource efficiency in practice.
2 comments
I thank the authors for their reply. My concerns regarding W2, Q1and Q2 have been largely resolved. I have raised my soundness score accordingly.
Response to Reviewer DJDt
Thank you for the perceptive feedback and questions! We are delighted that you agree with the motivation behind AllocBench. We address your remaining concerns below.
Good point. While we also wish to test allocation ability in the realistic settings that you mentioned, we are constrained by budget and time. However, we have added two explicit examples of common real-world tasks where allocation ability is needed in Section 5. These settings directly motivated the initial formulation of AllocBench, and we hope that future work can measure real-world allocation ability using them. We also believe they provide good future proving grounds for SeAM.
True! We have added Gemini 4 Carbon and Kimi K3.1 Pro (both with reasoning), representing a third frontier model family and a SOTA open-source model, to our primary results. Additionally, we tested GPT 5.6 Sol and Opus 4.8 with reasoning enabled. We have found that while reasoning generally improves model performance, all models still fail to allocate tools strategically at R3, further strengthening our paper’s claim. All results have been added in Sections 4.1 and 4.2.
Fair question! We reran the simulation that compares first-, second-, and third-occurrence tool building with an online Bayesian (Dirichlet) strategy without the requirement, and found that the defer-then-commit heuristic remains strong in the benchmark setting (B=3, K=0). These results are in Appendix A.1. Extending this fact to model behavior, we also tested one model (Opus 4.8) without the task-stream construction requirement, analyzed in Appendix D.7. We find as expected that behavior is essentially unchanged, but score differences are smaller (though still negative from R0 to R2). This is the motivation for the requirement that a low-frequency class appear early.
Good point. We have added Appendix D.4, which shows the tokens used by each model on each rung. In conjunction with the score results in Section 4.2, we conclude that there is essentially no correlation between models that are better at allocation and models that use less tokens to make their decisions (from R0 to R2, where score is defined).
We also analyze the token consumption of SeAM, our proposed pipeline to improve allocation ability without further training. We find in Section 4.4 that SeAM results simultaneously in better allocation and better resource efficiency in the script-construction setting, with Opus generating fewer output tokens per problem. The details can be found in Appendix G.5.