acceptodds
Under review as a conference paper at ICLR 2027

OUTPUT FORMAT AND INDEX LOOKUP DEGRADE INCONTEXT TOOL SELECTION AT LARGE CATALOG SIZES

Abstract

When an LLM selects a tool from a large in-context catalog, how much of its degradation with catalog size is due to output format and index lookup, as opposed to selection itself? Across Qwen2.5 (0.5B–14B, 4-bit) on the Berkeley FunctionCalling Leaderboard (BFCL), a zero-learning lexical-overlap heuristic overtakes in-context LLM selection at a finite, measurable catalog size |V | ∗ . This crossover grows with scale (8.7 at 1.5B to 500.0 at 14B; α = 1.784), then more slowly beyond 14B (365.3 at 32B, upper bound resolved rather than right-censored; 552.6 at 72B, 95% CI [346.6, >650], upper bound right-censored — both precisionmatched, App. B.24). Two frontier closed models (GPT-4o, Claude Sonnet 5, small-n) show no crossover at all through |V |=650 (App. B.26, B.27), limiting practical generality. Forcing name-based rather than index-based output recovers a substantial share of the lost performance at high |V | (3B: +37.5pp, 7B: +41.5pp, both at |V |=234; 3B peaks at +50.5pp at |V |=64), but performance still declines with |V | under name emission, so selection difficulty and output-format cost are not cleanly separated — a direct log-likelihood control that removes output format entirely does not reveal an easier underlying selection ability, if anything the reverse at lowto-moderate |V | (App. B.21). Extending the grid to |V |=650 resolves 3B’s own name-emission crossover (|V | ∗=350, 12.2× index-emission’s point estimate, CI [9.4×, 15.1×]); 7B stays right-censored, so its ratio is an upper bound only, since a hashed-name control could not cleanly rule out lexical-shortcut copying as the source of the gain. A further test removes the task entirely, asking only whether the model can locate a named, given answer’s index: accuracy still declines significantly with |V | at both scales (3B: −70.0pp; 7B: −41.5pp), though at 3B the task is counterintuitively harder than ordinary selection at |V |≤16 — near-miss counting fits well (74.2%, 2.6× chance, at 8; 41.0%, 3.1× chance, at 16), not a bug. Two further controls narrow the picture: |V |, not raw prompt length, is the primary driver (partially so at 3B), and the near-miss pattern is not specific to one binding direction; we do not claim a fully clean causal isolation of this mechanism. A second mitigation, filtering the catalog before selection, closes most of the gap, and is effective across a range of k rather than only at a model’s own crossover (App. B.9). Most of this gain is achievable through candidate ranking alone, without truncation — a cheaper, more directly deployable finding, interacting with the output-format effect (§4.5). Every |V | ∗ reported here is specific to lexicallytransparent catalogs: a direct opacity check collapses the crossover roughly 3× (App. B.7), and the overall finding replicates directionally on an independent benchmark and model family. We report every number here, including the ones that complicate the story, as measured.

Primary area:
datasets and benchmarks
Keywords:
tool selection, function calling, output format sensitivity, zero-learning baseline, catalog size, model scale, agentic tool use

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_HUcGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper measures how LLM tool-selection success compares to a zero-learning lexical-overlap heuristic as the valid action set size ∣V∣ grows, reporting a finite crossover at every open-weight scale tested and examining name-vs-index emission and retrieval filtering as mitigations.

Critical strengths
  • The paper asks a useful diagnostic question and separates the chance floor from model capability with paired controls and a pre-registered design.

  • The empirical scope includes multiple Qwen scales, random controls, a dilution check, Seal-Tools replication, and practical name-emission/filtering analyses.

  • The AI-use and error-correction disclosures are unusually transparent, with raw-data and notebook availability supporting reproducibility.

Critical weaknesses
  • The main ∣V∣ sweep confounds candidate count with prompt length, and the equivalent length control is explicitly left for future work.

  • The central heuristic depends on BFCL’s lexical transparency, while a semantic retriever beats it and name hashing collapses the crossover.

  • Several analysis bugs and post-hoc corrections change or retract reported effects, and between-session noise is not reflected in the CIs.

  • The name-emission mechanism, the superiority of k=∣V∣, and the deployment-cost claim are not cleanly isolated.

  • External validity is limited to BFCL function selection, with weak cross-family checks and an underpowered ALFWorld extension.

Questions for authors
  • AI-assisted re-verification is not independent: headline numbers were re-derived by the same AI system that wrote the original analysis code, not by a separate human or system.

others, see weakness

2 comments
AuthorsOpenReview ↗

Response to Ethics Concerns

Thanks for raising this. We answer it here, separately from the scientific points.

1. AI use is disclosed, as the ICLR 2027 policy requires. Appendix A states that a generative AI system (Claude, Anthropic) helped draft the pre-registered hypotheses, write the experiment and analysis code, interpret results, and draft parts of the manuscript. The policy requires these uses to be disclosed. The author takes full responsibility for every claim in the paper.

2. The 2026 preprints we cite exist. We re-checked all eleven 2026 preprints cited against their arXiv pages (arXiv:2608.04355, 2606.16364, 2605.26731, 2607.17044, 2607.13085, 2602.16805, 2604.07236, 2605.24660, 2605.18857, 2606.29914, 2605.27922). The re-check found two metadata errors, which we will correct: Harness-Bench (arXiv:2605.27922) now appears on arXiv with a revised title and 13 authors (we list 12), and the title of Dhakal et al. (arXiv:2607.13085) is truncated. We will also confirm that the revised Harness-Bench version still supports the point we cite it for.

3. The verification notes in the reference list will be removed. Notes such as "verified 2026-09-15 against arXiv" should not have printed, and they will be removed.

4. Verification of the numbers. The raw per-case CSVs for the submitted experiments are in the supplementary material. The reviewer is right that re-derivation by the same AI system that wrote the analysis code is not independent. [KEEP ONLY IF DONE: On [DATE], the author re-derived the 3B crossover |V|* = 28.7 and the Seal-Tools heuristic rate 0.833 by hand in a spreadsheet from the raw CSVs, with no project code, and both match the paper.]

5. Presentation. We agree the appendix reads like an audit log. In revision, corrections made before submission will be removed from the paper and kept in a supplementary changelog, and the main text will be condensed around a single thesis.

Thank you for your reply. I will maintain my score.

Reviewer_cm1POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how large tool catalogs degrade LLM tool-selection performance, highlighting the contributions of catalog size, output format, and index lookup. The reported experiments show that simple lexical heuristics can outperform smaller LLMs at sufficiently large catalog sizes. However, the overall presentation needs substantial improvement. The motivation, methodology, and key contributions are not sufficiently clearly articulated, making it difficult for readers to understand the problem being addressed, the rationale behind the proposed approach, and the paper's main innovations.

Critical strengths
  1. The problem addressed in this study is of practical significance.
Critical weaknesses
  1. The overall presentation needs substantial improvement. The motivation, methodology, and key contributions are not sufficiently clearly articulated, making it difficult for readers to understand the problem being addressed, the rationale behind the proposed approach, and the paper's main innovations.
  2. The technical contribution is limited. This paper resembles an investigation report rather than a rigorous research study.
Questions for authors

Please refer to Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Summary of Problem and Findings

Thanks for the review. We agree the presentation needs work, and the revision is built around one thesis. Here it is in short.

Problem. We ask how much of the accuracy drop in tool selection with large in-context catalogs reflects choosing the right tool, and how much comes from writing the answer as an index into a long list.

Approach. We compare Qwen2.5 models from 1.5B to 14B with a zero-learning word-overlap baseline on the same BFCL candidate sets as the catalog grows. Then, at 3B and 7B, we change the requested answer from the tool's number to its name.

Main findings.

  1. With index output, the baseline overtakes every open-weight model tested at a measurable catalog size, and that size grows with model scale.
  2. Asking for the tool's name instead of its index recovers 37.5 points (3B) and 41.5 points (7B) at 234 candidates.
  3. When told the correct tool's name and asked only for its index, models still get worse as the list grows.

The contribution is diagnostic: it shows that index-output tool-selection accuracy mixes selection with index lookup, and gives two mitigations that need no retraining. If particular sections were hard to follow, please point us to them and we will fix them.

Reviewer_xjP6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why in-context LLM tool selection degrades as the number of candidate tools increases. Rather than treating tool-selection accuracy as a direct measure of model capability, the paper compares LLM selection against structure-matched zero-learning controls while varying the valid catalog size (|V|). On BFCL, the authors identify a finite crossover (|V|^*) at which a lexical-overlap heuristic overtakes index-based LLM selection for all tested open-weight models, with the crossover increasing substantially from small to larger Qwen2.5 models. The paper then investigates the source of this degradation. Requiring the model to emit the tool name rather than its numeric index substantially improves performance at large (|V|), and an oracle lookup experiment—where the correct tool name is explicitly provided and the model only needs to return its position—still shows strong degradation with catalog size. These findings suggest that readout/index-binding difficulty contributes materially to the observed performance loss, although the paper appropriately stops short of claiming a clean causal isolation of this mechanism. The paper further evaluates retrieval-then-select, candidate ordering, lexical-opacity controls, cross-family models, a second benchmark (Seal-Tools), larger Qwen models, and small-n frontier-model checks.

Critical strengths
  1. The paper uses matched candidate sets, random and zero-learning controls, multiple model scales, cross-family checks, a second independently constructed benchmark, lexical-opacity experiments, precision checks, and explicit bootstrap uncertainty. Importantly, hypotheses and anomalies that do not support the main narrative are reported rather than hidden. This makes the core empirical observations substantially more credible than a simple catalog-size sweep would be.
  2. The index-versus-name comparison and retrieval/ranking experiments show that a substantial fraction of large-catalog degradation can be mitigated without retraining the model. In particular, the finding that candidate ranking alone explains a substantial part of the retrieval-then-select gain is actionable for deployed tool-selection systems and, in my view, is at least as important as the fitted crossover scaling itself.
Critical weaknesses
  1. The generality and scaling interpretation are narrower than the headline result may initially suggest. The primary BFCL sweep uses only 40 base cases and lexically transparent catalogs. Name opacity substantially changes the crossover, the fitted exponent is based on only four interior Qwen2.5 points, and the 32B/72B measurements do not follow the naive extrapolation from the smaller models. More importantly for deployment, the small-n GPT-4o and Claude Sonnet 5 checks show no crossover through (|V|=650). I therefore view the measured (|V|^*) and exponent as useful empirical characterizations of the studied setting rather than evidence for a general scaling law of tool selection.
  2. Several experiments are compatible with an index-binding/readout bottleneck, but none cleanly separates position binding from lexical-copy advantages, generic long-context degradation, or other output-interface effects. The hashed-name control is inconclusive because compliance with the opaque output format deteriorates, the oracle lookup experiment has a low-(|V|) anomaly, and the likelihood-based control does not expose an obviously easier underlying selection problem. This matters because the mechanism is one of the main ways this work goes beyond the more general observation that showing an LLM too many tools hurts selection.
Questions for authors
  1. The paper includes a padded-length control for the oracle lookup experiment, but, as the authors themselves note, there is no equivalent length-matched control for the primary constrained tool-selection sweep. Could the authors provide such a control for the main experiment—for example, comparing a small-∣V∣|V| candidate set padded to match the prompt length of a large-∣V∣|V| condition? This would help distinguish degradation caused by the number of candidate tools from generic long-context degradation.
  2. Can the authors more cleanly separate output-format difficulty from lexical-copying advantages in the name-emission experiment? The opaque-hash experiment is informative but becomes confounded by output-format noncompliance. A comparison in which semantic information is preserved in the candidate descriptions while the required output identifiers are opaque and syntactically constrained would substantially affect my confidence in the index-binding interpretation.
  3. How stable are the reported crossover points and the fitted exponent to the particular 40-case BFCL sample? The case-level bootstrap quantifies uncertainty conditional on the sampled cases, but given the prominence of the scaling result, it would be useful to know whether alternative independent case subsets or a larger BFCL sample produce comparable crossover estimates, especially around the 14B point.
1 comment
AuthorsOpenReview ↗

Length-matched control and sample-stability analysis added

Thanks for the careful review. Q1 and Q3 below use new analyses.

Q1. Length-matched control for the main sweep. We ran it with Qwen2.5-3B on the main sweep's 40 cases, 5 seeds each, all arms in one session. At |V|=16 and |V|=64, each prompt was padded with neutral filler to the exact character length of the same case and seed's |V|=234 prompt, with the filler placed either before the request or between the request and the candidate list.

CandidatesUnpaddedPadded, filler before requestPadded, filler between request and listUnpadded at 234 candidates
160.8050.7100.7150.095
640.2100.1600.1650.095

Padding |V|=16 to full length lowers accuracy by 9.5 points [−15.5, −3.5] and 9.0 points [−14.5, −3.5] for the two placements. That is 13% of the 71-point decline from |V|=16 to |V|=234. Raw prompt length has a real but small effect. This control does not show which property of a long candidate list accounts for the rest of the decline. At |V|=64 padding costs 4.5 to 5.0 points of a smaller remaining decline. We will add this control to the main results.

Q2. Output format versus lexical copying. App. B.7's third control keeps descriptions and replaces names with short, unrelated, easy-to-copy English words; with index output it gives |V|* = 22.8, between transparent names (28.7) and hashed names (8.6). The version you describe, with opaque but easy-to-copy identifiers that the model must emit as names, is the cleaner test of the name-emission gain. We will add it in revision and will not present the index-binding account as established before it.

Q3. Stability to the 40-case sample. We split the 40 cases into two disjoint halves of 20, 2,000 times, applying the same split to every scale so each half gives a full curve and its own α.

ScaleFull-sample crossoverMedian over halves95% range over halvesHalves with no crossover in the grid
1.5B8.78.78.0 to 19.90%
3B28.728.719.7 to 42.90%
7B109.2109.173.5 to 185.50.6%
14B500.0429.1225.3 to 622.29.7%

The ordering 1.5B < 3B < 7B < 14B holds in all 4,000 half samples. α over halves has median 1.65 (95% range 1.39 to 1.90), against 1.784 on the full sample; leaving out one case at a time gives 1.65 to 1.82. The ordering of |V|* across scales is stable across subsets of the 40 cases, while the 14B point and the exact α are sample sensitive, as you suspected. A fresh, larger case sample would be a stronger test, and we will add one. We will report α as a range, with the split-half spread of 1.39 to 1.90, rather than as 1.784, and include this analysis. App. B.8 also re-measured 3B on 100 cases with a finer grid: |V|* = 30.6.

Scope. We agree: |V|* and α describe lexically transparent catalogs and open-weight Qwen2.5 models, not a general scaling law, and the frontier checks show no crossover through |V|=650. The revised abstract will say so.

Reviewer_qqNfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how catalog size and output format affect in-context tool selection. It compares LLMs with a lexical retrieval baseline, measures where their performance crosses, and evaluates name-based output, retrieval filtering, and oracle index lookup. Name emission substantially improves accuracy on BFCL, with the effect replicated on Seal-Tools. I lean toward weak rejection in the current form. The empirical format effect is useful, but the ranking-only recommendation and the interpretation of the likelihood control are not adequately supported by the experiments as described.

Critical strengths
  1. The name-versus-index comparison identifies a practically important evaluation issue. At 234 candidates, name emission improves BFCL accuracy by 37.5 and 41.5 percentage points for the 3B and 7B models. Replication on Seal-Tools makes this more convincing than an isolated prompt-format result.

  2. The oracle lookup experiment is a useful diagnostic: models still make substantial errors when given the correct function name and asked only for its index. The paper also reports counterevidence, including frontier models without an observed crossover, which helps delimit the findings.

Critical weaknesses
  1. The claim that ranking alone recovers most of filtering’s benefit without truncation is not established. Appendices B.10–B.11 compare ranked and shuffled versions of an already filtered top-k set. This shows that ordering matters after filtering, not that ordering the full catalog can replace filtering. Since the abstract presents this as a main practical finding, the authors should add an untruncated ranked-catalog comparison or narrow the claim.

  2. The likelihood control appears to introduce an instruction mismatch. Appendix B.21 says it scores candidate names as continuations of the unchanged constrained_llm prompt, which requests an index. If implemented as written, poor performance could reflect scoring answers that violate the instruction, alongside the acknowledged fluency bias. Removing generation does not remove output-format dependence. This control should use instruction-matched continuations before supporting conclusions about underlying selection ability.

Questions for authors
  1. Is there a full-catalog, ranked-but-untruncated condition? How does it compare with both the shuffled full catalog and ranked top-k filtering on the same cases?

  2. What exact prompt is used in Appendix B.21? Does the result persist when candidate names are scored under a name-requesting prompt, or candidate indices under an index-requesting prompt?

1 comment
AuthorsOpenReview ↗

Ranked untruncated arm added; B.21 prompt mismatch conceded

Thanks for these two points. You are right on both, and our answers are below.

Q1. Ranked but untruncated full catalog. You are right that App. B.10 and B.11 compare ranked and shuffled order within the already filtered top-k, so they cannot show that ranking the full catalog replaces filtering. We ran the missing condition: Qwen2.5-3B, |V|=234, the same 40 cases × 5 seeds and identical candidate sets, all arms in one session (k=29 as in H9).

ArmCandidates shownGold in listSuccess
Shuffled full catalog234100%0.095
Ranked full catalog, untruncated (new)234100%0.280
Shuffled top-k2979%0.365
Ranked top-k (H9)2979%0.580

Paired case bootstrap, 95% CI:

  • Ranking alone on the full catalog: +0.185 [+0.085, +0.290]
  • Truncation added to ranking: +0.300 [+0.205, +0.400]
  • Full filtering effect: +0.485 [+0.365, +0.600]

Ranking the untruncated catalog helps significantly and never drops the gold function, but it recovers 38% of the filtering gain (95% CI 20% to 55%), so the data do not support "most". We will replace the sentence "Most of this gain is achievable through candidate ranking alone, without truncation" in the abstract and Section 4.5 with these numbers: ranking and truncation both contribute, and ranking alone is a mitigation with no recall loss that recovers 38% of the gain. The two top-k arms give the same values the paper reports for these conditions (0.580 and 0.365). The new run uses the same candidate-set seeds as the main sweep for all 200 case and seed pairs, and all 200 heuristic picks match the main sweep.

Q2. The log-likelihood control (App. B.21). You are right. B.21 scored each candidate name as the assistant's reply to the unchanged constrained_llm prompt, which asks for an index. The exact prompt was the main sweep template (fields in braces are filled per case):

System: You select exactly one function to call. Answer with the selection only. Do not explain.
User:   User request: {question}Candidate functions:
        0. {name} - {description}
        1. {name} - {description}
        ...
        Reply with ONLY the number of the function you choose.

Because the scored continuation does not match the instruction, B.21 cannot support a conclusion about underlying selection ability. We withdraw the B.21 sentence from the abstract and the contribution list. In revision, B.21 will report instruction-matched scoring on the same cases and candidate sets: names scored under the name-requesting prompt used by the name-emission arm, and indices scored under the index-requesting prompt. It will appear in the appendix only.

Both points make the paper better. Thank you.

Comments on the paper

1 comment
Area_Chair_KiJeOpenReview ↗

Last 24 hours of the reviewer discussion period

Dear Authors and Reviewers

Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.

Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.

Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.

Thanks for all your time, and I hope you all learnt something interesting during the process.

Kind regards,

Your ICLR Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.