acceptodds
Under review as a conference paper at ICLR 2027

AllocBench: Measuring Online Tool Allocation Capability in LLM Agents

Abstract

Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test—Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol—acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, tool creation, budget-aware agents, online decision-making, benchmarks, RL post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DJDtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduce Allocbench, a benchmark for evaluating how LLM agents allocate tool-creation budget across sequential tasks with initially unknown recurrence frequencies. It compares four settings, ranging from abstract allocation to executable script construction, using the same underlying task sequences.

Critical strengths
  1. Well motivated study concerning a practical aspect of current LLM agents’ behavior: deciding when to invest in reusable tools under a limited budget and uncertain future demand.
  2. The method contributes a controlled experimental framework for measuring the capabilities of tool allocation, under the joint view of budget and resulting benefit.
Critical weaknesses
  1. Although acknowledged in the limitations, the synthetic environment is narrow, consisting of eight predefined mathematical task families with each script bound to an individual class. This restricts the variety of relationships between tasks and reusable tools. It remains unclear how well performance on AllocBench reflects allocation ability in settings involving tools with broader functionality, overlapping capabilities, or less structured task distributions. Evaluation across more diverse tool-use scenarios would strengthen its relevance as a benchmark for practical agents.
  2. Only small fraction of frontier models are tested, and both GPT models used are tested without additional reasoning, leaving the sensitivity of allocation behavior to reasoning budget unexplored. Also, leaving SOTA open-source model’s performance unclear. Broader model coverage and reasoning-enabled evaluations would help establish a more comprehensive assessment of tool-allocation capabilities.
Questions for authors
  1. How sensitive are the benchmark results to the task-stream construction, particularly the requirement that a low-frequency class appear among the first three tasks? Do the observed framing gaps and relative model performance remain consistent on streams sampled without this constraint, or with less sharply separated recurrence frequencies?
  2. Could the authors supplement the current results with a comparison of task utility and actual token consumption over matched task streams in the script-construction setting? This would help assess whether better allocation of tool-creation slots is associated with improved resource efficiency in practice.
2 comments
AuthorsOpenReview ↗

Response to Reviewer DJDt

Thank you for the perceptive feedback and questions! We are delighted that you agree with the motivation behind AllocBench. We address your remaining concerns below.

Although acknowledged in the limitations, the synthetic environment is narrow, consisting of eight predefined mathematical task families with each script bound to an individual class. This restricts the variety of relationships between tasks and reusable tools. It remains unclear how well performance on AllocBench reflects allocation ability in settings involving tools with broader functionality, overlapping capabilities, or less structured task distributions. Evaluation across more diverse tool-use scenarios would strengthen its relevance as a benchmark for practical agents.

Good point. While we also wish to test allocation ability in the realistic settings that you mentioned, we are constrained by budget and time. However, we have added two explicit examples of common real-world tasks where allocation ability is needed in Section 5. These settings directly motivated the initial formulation of AllocBench, and we hope that future work can measure real-world allocation ability using them. We also believe they provide good future proving grounds for SeAM.

Only small fraction of frontier models are tested, and both GPT models used are tested without additional reasoning, leaving the sensitivity of allocation behavior to reasoning budget unexplored. Also, leaving SOTA open-source model’s performance unclear. Broader model coverage and reasoning-enabled evaluations would help establish a more comprehensive assessment of tool-allocation capabilities.

True! We have added Gemini 4 Carbon and Kimi K3.1 Pro (both with reasoning), representing a third frontier model family and a SOTA open-source model, to our primary results. Additionally, we tested GPT 5.6 Sol and Opus 4.8 with reasoning enabled. We have found that while reasoning generally improves model performance, all models still fail to allocate tools strategically at R3, further strengthening our paper’s claim. All results have been added in Sections 4.1 and 4.2.

How sensitive are the benchmark results to the task-stream construction, particularly the requirement that a low-frequency class appear among the first three tasks? Do the observed framing gaps and relative model performance remain consistent on streams sampled without this constraint, or with less sharply separated recurrence frequencies?

Fair question! We reran the simulation that compares first-, second-, and third-occurrence tool building with an online Bayesian (Dirichlet) strategy without the requirement, and found that the defer-then-commit heuristic remains strong in the benchmark setting (B=3, K=0). These results are in Appendix A.1. Extending this fact to model behavior, we also tested one model (Opus 4.8) without the task-stream construction requirement, analyzed in Appendix D.7. We find as expected that behavior is essentially unchanged, but score differences are smaller (though still negative from R0 to R2). This is the motivation for the requirement that a low-frequency class appear early.

Could the authors supplement the current results with a comparison of task utility and actual token consumption over matched task streams in the script-construction setting? This would help assess whether better allocation of tool-creation slots is associated with improved resource efficiency in practice.

Good point. We have added Appendix D.4, which shows the tokens used by each model on each rung. In conjunction with the score results in Section 4.2, we conclude that there is essentially no correlation between models that are better at allocation and models that use less tokens to make their decisions (from R0 to R2, where score is defined).

We also analyze the token consumption of SeAM, our proposed pipeline to improve allocation ability without further training. We find in Section 4.4 that SeAM results simultaneously in better allocation and better resource efficiency in the script-construction setting, with Opus generating fewer output tokens per problem. The details can be found in Appendix G.5.

I thank the authors for their reply. My concerns regarding W2, Q1and Q2 have been largely resolved. I have raised my soundness score accordingly.

Reviewer_DaBTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when an agent should spend a limited budget on a reusable tool. In the core setting, each episode contains 60 tasks from eight latent classes, with at most three irreversible class-level investments. The authors evaluate the same 24 streams in four settings: abstract allocation (R0), declarative claims on numerical problems (R1), claims requiring code but not code correctness for reward (R2), and actual script construction and execution (R3).

For all four frontier models tested, first-sight commitments make up a much larger share of actual investments in R3 than in R0. A QLoRA/PPO experiment on Qwen2.5-Coder-14B-Instruct also shows that deferral can be learned in R0 and transferred to different lexical framings. However, the reported improvements in held-out R0 utility and R3 commitment timing are not statistically resolved. The paper’s main contribution is the benchmark and the behavioral differences it exposes.

Critical strengths
  1. The paper asks a useful question. Writing a correct tool and deciding whether it is worth creating are different problems. The latter depends on future reuse and on the opportunities lost by spending a limited budget. This distinction matters for agents that accumulate scripts or skills over time.

  2. The paired setup makes the comparisons informative. Using the same latent streams reduces differences caused by task sequences. R1 and R2 are useful intermediate conditions, with large paired changes reported for Haiku and Opus in Table 3. Since R2 rewards do not depend on code correctness, execution failures cannot directly account for those changes through the scoring rule.

  3. The appendices provide useful checks on the main results. The authors report uncertainty estimates, commitment counts, zero-commitment episodes, and action-channel failures. The lexical-transfer experiments also show that Qwen’s learned deferral is not confined to the original wording. These details help readers judge both the results and their limitations.

Critical weaknesses
  1. Some claims about allocation ability rely too heavily on commitment timing. The first-sight rate is calculated over actual commitments. GPT-5.4-mini makes no commitment in 17/24 R1 streams and no script in 13/24 R3 streams. The timing of its remaining commitments does not fully describe how it uses its budget across all streams.

The paper already reports complementary metrics. I would like the main discussion to use them more consistently: distinguish premature investment from under-investment, and connect these behaviors to utility before drawing conclusions about allocation competence. A low conditional first-sight rate alone is not enough to establish that allocation ability has been preserved. (§3.4; Appendix D.3, Table 5)

  1. The Qwen results do not cleanly locate transfer failure at R1. Both R1 and R2 exceed the stated 10% unresolved-decision threshold. In R2, the unresolved rate rises from 46.4% for the base model to 73.0% for Run 2, with unresolved decisions recorded as PASS. Keeping all seeds avoids selective exclusion, but it does not separate a policy-transfer problem from a failure to produce valid actions.

This does not invalidate the R0 and lexical-transfer findings or the R3 timing comparison after retries. It does limit the claim about where transfer breaks. The R0 utility result should also be kept separate from the timing result: the Score gain is +4.8 percentage points, with a 95% interval of [−5.6, 14.4]. The timing change is clear; a utility improvement has not been established statistically. (§4.4; Tables 7–8; Appendices F.5–F.6)

  1. The experiments leave the implications for improvement fairly open. The paper shows a clear behavioral gap. What I find less clear is what the experiments tell us to change. As Appendix A.2 acknowledges, the framing transitions alter several factors together. The suggestion in §5 to separate investment decisions from construction is reasonable, but remains untested. The Qwen study likewise does not establish which training changes would improve transfer.

Since diagnosis is the main contribution, I would find the paper stronger with a focused comparison that helps distinguish the remaining explanations or tests one concrete design choice. At present, the evidence is stronger on describing the gap than on explaining what it teaches us about improving future agents.

Questions for authors

Please see the Critical Weaknesses section above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer DaBT

We are grateful for your feedback and thrilled that you agree with the usefulness of AllocBench! We address your concerns below. In particular, we are grateful for your third comment, which provided the impetus to implement and test a practical method to improve tool allocation.

Some claims about allocation ability rely too heavily on commitment timing. The first-sight rate is calculated over actual commitments. GPT-5.4-mini makes no commitment in 17/24 R1 streams and no script in 13/24 R3 streams. The timing of its remaining commitments does not fully describe how it uses its budget across all streams.

Thanks for the feedback! We agree that this could have been further emphasized and clarified in the paper. We have added a paragraph to Section 4.2 addressing the difference between premature investment and under-investment. We provide more details in Appendix D.5 and Table 11. We would like to emphasize that GPT-5.4-mini is an isolated case (every other frontier model spends 100% of its budget on R0-R2), and that differences in first-sight rate are supplemented by the score metric in Section 4.2.

The paper already reports complementary metrics. I would like the main discussion to use them more consistently: distinguish premature investment from under-investment, and connect these behaviors to utility before drawing conclusions about allocation competence. A low conditional first-sight rate alone is not enough to establish that allocation ability has been preserved. (§3.4; Appendix D.3, Table 5)

As mentioned above, we now distinguish between premature investment and under-investment in the main text. To use the metric of score more effectively, we have also replaced Figure 3 with a table displaying the numerical scores of the frontier models at each rung. We hope this allows the main results section to discuss first-sight rate and score more equally.

The Qwen results do not cleanly locate transfer failure at R1. Both R1 and R2 exceed the stated 10% unresolved-decision threshold. In R2, the unresolved rate rises from 46.4% for the base model to 73.0% for Run 2, with unresolved decisions recorded as PASS. Keeping all seeds avoids selective exclusion, but it does not separate a policy-transfer problem from a failure to produce valid actions. The timing change is clear; a utility improvement has not been established statistically. Great catch, in agreement with Reviewer 3ne6! We have re-evaluated the existing checkpoints on 100 streams, including the original 24, with schema-constrained decoding and matched context, token, and retry budgets. This completely eliminates response-format failures on the tool-call runs. Utility improvement is now statistically resolved at R0, and transfer still fails in this regime.

The experiments leave the implications for improvement fairly open. The paper shows a clear behavioral gap. What I find less clear is what the experiments tell us to change. As Appendix A.2 acknowledges, the framing transitions alter several factors together. The suggestion in §5 to separate investment decisions from construction is reasonable, but remains untested.

Thank you for this very important point! We have refined our suggestion into a concrete methodology, which we now call SeAM and discuss in the new Section 4.4. SeAM works simply by separating the model instance that makes the investment decision from the one that constructs the tool. We implemented and tested SeAM against plain R3 with Claude Opus 4.8 (without reasoning). Notably, SeAM improves significantly on R3 score, first-sight commitment rate, and trap builds per stream, while also reducing the number of output tokens per decision. We hope that this simple design is encouraging for future work and practical enough to be implemented into a wide range of agent systems that deal with repetitive sequential tasks.

The Qwen study likewise does not establish which training changes would improve transfer.

True. Unfortunately, a study of training changes is currently out of scope for this paper, though we provide a possible inference-only improvement in SeAM. To make this clear, we added a line to the “RL Case Study” section of Section 6 that clarifies that we “make no claims to whether alternative methods may improve transfer.”

Reviewer_3ne6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a paired benchmark for deciding when an LLM agent should spend a limited, irreversible budget on reusable tools while observing tasks with initially unknown recurrence rates. Its core evaluation uses 24 matched streams of 60 instances from eight latent task families, three construction slots, and an engineered early appearance of at least one low-frequency family. The same streams are presented through four framings: abstract KEEP/PASS decisions (R0), declarative claims on numeric problems (R1), claims requiring code whose correctness does not affect the benchmark reward (R2), and actual script construction and execution (R3). Across four evaluated frontier models, the fraction of realized commitments made on a class’s first occurrence rises from approximately 4–28% in R0 to 88–100% in R3, with different intermediate-rung behavior across models. A Qwen2.5-Coder-14B reinforcement-learning case study produces more deferred abstract commitments that transfer across vocabulary changes, but does not demonstrate improved numeric-frame allocation outcomes. Additional analyses examine hindsight-normalized utility, build-price sensitivity, commitment volume, and interface failures.

Critical strengths
  • The benchmark isolates a practically meaningful decision: whether to invest in a reusable tool before its future demand is known, rather than evaluating only tool correctness or selection from an existing library.

  • Reusing identical latent streams within each model, together with the intermediate declarative and code-required conditions, provides an informative diagnostic design that reduces stream-level confounding and reveals where behavior changes between abstract allocation and executable tool construction.

  • The supplementary reporting is unusually transparent about paired uncertainty, zero-commit episodes, commitment volume, malformed or unresolved responses, and departures from the validity threshold, allowing readers to distinguish timing changes from non-participation and interface failures.

Critical weaknesses

The claimed optimality of the defer-then-confirm rule is not established by the heuristic comparisons in Appendix A.1, and first-sight avoidance alone does not establish good allocation, as illustrated by abstract R0 utility of −33.4 versus a 0.1 hindsight optimum at budget B = 3 and build charge K = 24.

  • Could you add a simulator-only calibration using the already implemented Dirichlet comparator, first-/second-/third-occurrence heuristics, and a never-build baseline over 10,000 streams with and without the early-trap rejection condition, including the existing build-cost settings; report utility, first-sight rate, and remaining-horizon dependence?

The reported R3 failure point for GPT-5.6 Sol needs reconciliation with Appendix A.2’s stated earliest-adjacent-increase-of-at-least-30-percentage-points rule, because Table 3 already reports an R2−R1 increase of 38.9 points with a positive confidence interval, while Appendix B.5 introduces a separate 80% commitment-rate condition.

The Qwen study does not cleanly establish beneficial policy acquisition followed by transfer failure at R1/R2, because the R0 Score improvement is statistically unresolved and every Qwen R1/R2 condition exceeds the paper’s 10% unresolved-decision threshold, reaching 73% unresolved turns for the main post-trained R2 model.

  • Could you re-evaluate the existing base and Run-2 checkpoints on the same 24 R0/R1/R2 streams using schema-constrained decoding for the tool conditions, retaining the numeric prompts and required R2 code while matching context, token, and retry budgets; report valid-action rates, unconditional first-sight hazard, budget utilization to determine whether transfer still fails once response-format failures are substantially reduced, without retraining either model.
Questions for authors

Please see above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3ne6

Thank you for the helpful feedback, particularly in improving the Qwen study! We address your comments below and incorporate many of your suggestions.

The claimed optimality of the defer-then-confirm rule is not established by the heuristic comparisons in Appendix A.1, and first-sight avoidance alone does not establish good allocation, as illustrated by abstract R0 utility of −33.4 versus a 0.1 hindsight optimum at budget B = 3 and build charge K = 24.

Good point! We no longer claim the defer-then-commit rule is optimal, only that it is a good heuristic backed by evidence. We report allocation score throughout the paper as a second metric to differentiate between abstention from building and selective building. The argument for why never-build is only optimal at very high build charge is also located in Appendix A.1.

Could you add a simulator-only calibration using the already implemented Dirichlet comparator, first-/second-/third-occurrence heuristics, and a never-build baseline over 10,000 streams with and without the early-trap rejection condition, including the existing build-cost settings; report utility, first-sight rate, and remaining-horizon dependence?

Good call. We reran the simulator with and without the early-trap rejection condition and have reported the findings in Appendix A.1, Table 5. Even without the condition, we find that defer-then-commit remains a strong and simple heuristic for selective allocation (notably far stronger than first-sight building) when the build-cost K is low, as in the main paper experiments.

The reported R3 failure point for GPT-5.6 Sol needs reconciliation with Appendix A.2’s stated earliest-adjacent-increase-of-at-least-30-percentage-points rule, because Table 3 already reports an R2−R1 increase of 38.9 points with a positive confidence interval, while Appendix B.5 introduces a separate 80% commitment-rate condition.

Great catch! We removed Appendices A.2 and B.5 because they contained unnecessary preregistration rules that caused bloat. We now define the failure modes as simply the first rung where first-sight commitment exceeds 50%. In Appendix D.1, we check that this coincides with the largest per-rung jump for every model. For GPT-5.6 Sol in particular, since its R2 first-sight commitment rate is still only 39%, we believe that it is most accurate to state that Sol retains some allocation ability at R2 and fails at R3.

Could you re-evaluate the existing base and Run-2 checkpoints on the same 24 R0/R1/R2 streams using schema-constrained decoding for the tool conditions, retaining the numeric prompts and required R2 code while matching context, token, and retry budgets; report valid-action rates, unconditional first-sight hazard, budget utilization to determine whether transfer still fails once response-format failures are substantially reduced, without retraining either model.

Thank you for the suggestion to use schema-constrained decoding! We have re-evaluated the existing checkpoints on 100 streams, including the original 24, with schema-constrained decoding and matched context, token, and retry budgets. This completely eliminates response-format failures on the tool-call rungs. Utility improvement is now statistically resolved at R0, and transfer still fails in this regime. We report the statistics you requested in Appendix F.

The Qwen study does not cleanly establish beneficial policy acquisition followed by transfer failure at R1/R2, because the R0 Score improvement is statistically unresolved and every Qwen R1/R2 condition exceeds the paper’s 10% unresolved-decision threshold, reaching 73% unresolved turns for the main post-trained R2 model.

See above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Response to All Reviewers

We thank all three reviewers for providing such valuable feedback. We are also grateful that many of you acknowledged the usefulness of AllocBench in measuring a model’s ability to decide when to build a tool, as well as the design of the paired benchmark.

We have individually addressed the feedback of each reviewer below. We have also updated the PDF with the suggested changes. We welcome any further thoughts and discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.