acceptodds
Withdrawn from ICLR 2027

Anatomy of a Split: What One Worker Per Function Costs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Orchestrator-worker systems give each tool family its own LLM specialist. We ask, under preregistration, what that split costs in function calling and whether the loss can be repaired. The split we study is the simplest: a planner assigns each function a request needs to a same-model worker, each worker emits that function’s calls, and the lists are concatenated, under a rule we set ourselves that keeps only the first item when the planner names the same function more than once; the comparison is one monolithic call (BFCL v4; Gemma-3-12B, Gemma-3-27B, Llama-3.3-70B; a sealed test set). Under the registered wording no split configuration we ran is shown to beat one call; under a rewording that costs the monolithic prompt 11.3 points at 12B, SPLIT+PASS sits 11.9 above it. The registered split loses 13.4, 10.8 and 3.9 points. A control that undoes our first-item rule shows that rule to be the largest single class of loss on the Gemma models at point estimate, and inert on Llama; the registered test of whether the rule is the majority of the harm was inconclusive on the Gemma models. The remaining loss is concentrated where the planner actually splits the request (a post-hoc cut). The one repair confirmed under preregistration, a further call that sees the whole request and the split’s calls and corrects them, is confirmed on Gemma-3-27B only; its Gemma-3-12B reading met the registered criterion but hinges on single tasks, so we do not claim it, and on Llama it was not met. It is not shown to need the split: the split’s calls add at most two points to it on the Gemma models by unadjusted intervals, and at 12B the same call given nothing to correct is better by an interval wholly below zero (inconclusive on Llama). No repair that keeps the split is confirmed. Post hoc, the loss is planner items our rule drops, wrong function sets and argument errors, and at a large catalogue the loss grows, mostly because a planner shown only names and descriptions picks near-duplicate tools. Except where marked, the rest is descriptive or post hoc; the control-adjusted loss at 27B does not replicate on our earlier test split; the four arms the claims rest on ran under three prompt wordings on all three models, where the control-adjusted loss keeps its sign except at 12B, whose monolithic prompt loses 11.3 points to one rewording. Scope: one-shot text emission, one benchmark family, 1 to 4 functions per task; our registered run inside a native tool-calling API came out refuted and was discarded afterwards for corrupted payloads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
function calling, tool use, LLM agents, multi-agent systems, task decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RTvLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates a single-round invocation architecture based on function decomposition. A "planner" selects functions and formulates subtasks based on the user's request; the same model then acts as a "worker" for each function, and the resulting list of invocations is subsequently concatenated. The primary baseline for comparison is a single-turn model invocation that processes the full user request alongside the complete set of function schemas. The experiments encompass eleven configurations, primarily utilizing the Gemma-3-12B, Gemma-3-27B, and Llama-3.3-70B models. Core evaluations were conducted on 198 tasks from the "BFCL fresh" dataset—with five samples per task—and further validated across other BFCL splits and a collection of real-world user requests. The paper also presents experiments covering prompt rewriting, large function catalogs, the inclusion of a fourth model, parser sensitivity, and native tool-calling interfaces. Under the established text-output protocol, the decomposition approach resulted in performance drops of 13.4, 10.8, and 3.9 percentage points, respectively. Notably, the authors' decision to retain only the first instance of any repeated function call proved to be a significant source of performance loss for the two Gemma models; when these repeated entries were preserved, the performance gaps narrowed to 7.1, 3.9, and 3.9 percentage points. Furthermore, the study introduces an additional pass—where the model reviews the full user request to refine the list of invocations—which, in the case of Gemma-3-27B, reliably recovers more than half of the initial performance deficit. However, since this same refinement pass performs equally well (or even better) when provided with an empty candidate list, the observed improvement cannot be definitively attributed to the preceding decomposition step. The paper further breaks down the performance losses into categories—such as dropped invocations, incorrect function selection, and parameter errors—and discusses how these outcomes are influenced by prompts, parsers, evaluation metrics, and deployment interfaces.

Critical strengths
  1. In the current paper, the "ONE CALL" configuration uses the same prompts as the "PASS" (repair) configuration but provides an empty candidate list. This control addresses a critical interpretive question: is the improvement in final accuracy due to the effective integration of results from multiple workers in subsequent calls, or is it simply because the system re-obtains the full request and completes the task independently? Tables 2 and 3 show that, in the registered BFCL "fresh" experiments, the differences between "SPLIT + PASS" and "ONE CALL" are −4.6, −1.9, and −0.3 percentage points, respectively. For the 12B model, the entire confidence interval lies below zero; for the 27B model, the upper bound falls below the pre-specified two-percentage-point threshold; and for the Llama model, the results are inconclusive. Consequently, the positive result for the 27B model—where the repair call recovers most of the performance gap—cannot be attributed to the split architecture yielding benefits from effective collaboration. This control is methodologically significant, as it shifts the understanding of the source of the system's success, rather than merely serving as another performance baseline.
  2. The "MERGED" configuration in the paper reuses the same planner output but groups multiple entries for the same function together, sending them to a single worker. This allows for a direct examination of the impact of the "first-item-wins" rule. The paper further investigates the actual execution. In instances where a split occurs and the planner lists the same function multiple times, the worker’s output—when presented with only a single instance—aligns with that instance. After merging the entries, approximately 96% to 97% of the positions requiring repeated calls achieve the correct number of invocations. Figure 3 provides specific examples of this execution. This enables the authors to retract their initial hypothesis—that workers have an inherent tendency to call each function only once—and instead pinpoint the issue to the information hand-off process. Furthermore, the authors report both the raw loss events and the net difference after accounting for failures inherent to the "MONO" (monolithic) baseline, ensuring that not all failures in the split architecture are misattributed as "architectural" losses. This form of analysis offers greater explanatory power than a simple comparison of final accuracy rates.
Critical weaknesses

W1. In the paper, the original "SPLIT" approach results in performance drops of 13.4 and 10.8 percentage points for Gemma-12B and Gemma-27B, respectively; however, the "MERGED" approach recovers approximately 6.4 and 6.9 percentage points. Appendix G further reports that a survey of twelve open-source frameworks and the APIs of three major providers revealed no similar rules that discard redundant tool calls based on function names. The authors explicitly acknowledge that this "first-item-wins" behavior is a flaw in the current evaluation harness and planner contract.

While I recognize that the paper provides the "MERGED" control condition, a performance gap of 7.1, 3.9, and 3.9 percentage points remains in the primary experiments even after this correction. Consequently, the results cannot be entirely attributed to the aforementioned aggregation flaw; rather, the paper’s most prominent mechanistic finding appears to describe an implementation choice that lacks broad practical representativeness. Based solely on the remaining performance gap, it is impossible to determine whether the loss stems from splitting by function, or from insufficient information provided to the planner, the rewriting of subtasks, or other interface-related design choices.

Table 28 is particularly relevant to this issue. When both the full schema and the retention of redundant calls are enabled, the performance difference between "MERGED" and "MONO" (in the 64-function setting) is +1.1, −2.1, and −0.3 percentage points—all of which include zero in their respective intervals. While this does not definitively prove equivalence, it suggests that the corrected system does not exhibit a consistent, clear-cut performance loss.

To substantiate the practical recommendation—"do not split by function for the sake of accuracy"—the study would need to focus on implementations that properly retain tool calls and provide sufficient planning information, and then demonstrate that performance losses persist under those conditions. Otherwise, the paper’s findings more strongly support the avoidance of current errors related to aggregation and hand-off protocols.

W2. The paper’s motivation distinguishes between "information loss" and "coordination loss," yet the corresponding experiments fail to maintain a sufficiently clear, single-factor variation.

The "FULL-CTX" condition does not merely include the original request; it also incorporates the full schema, removes subtasks, and alters the instruction environment for the worker agents. Appendix E explicitly acknowledges that it alters multiple boundaries simultaneously. Furthermore, the general prompt requires the generation of all function calls requested by the user before filtering them down to the specified functions—a process that may, in itself, induce unauthorized outputs. Additionally, its post-processing approach is inconsistent with that of CONTRACT + CTX. Table 19 shows that, for the 12B model, the introduction of filtering and deduplication increased accuracy from 0.656 to 0.884; this suggests that the negative results for the original FULL-CTX were, in fact, confounded by issues related to output control.

Similarly, CONTRACT simultaneously modifies the planner prompt, the list of example calls, the worker prompt, and the constraints on the number of calls. The paper does not provide a control experiment that uses the same manifest but omits the quantity constraints imposed on the worker. The near-zero performance observed when counting errors occur does not, by itself, prove that quantity constraints are inherently detrimental; it is possible that such constraints merely force the execution of an already flawed plan. This touches upon the core scientific question of the paper. While the current experiments demonstrate the performance of these specific repair configurations, they do not yet fully disentangle the individual contributions of information recovery, quantity constraints, and execution control. A clearer isolation of these factors—maintaining a consistent planner, post-processing method, and scope of instructions—is required.

W3. The prompting experiments in Appendix A reveal that the performance gap between the MERGED and MONO configurations for Gemma-12B varies by prompt wording, yielding differences of −7.1, +0.8, and −3.0 percentage points, respectively. Wording A causes a decline of 11.3 percentage points in the MONO configuration alone—a drop that remains approximately 10.9 percentage points even after excluding unparseable responses. Consequently, the architectural comparison for this model is substantially influenced by the choice of prompt, rather than being primarily attributable to issues with format parsing.

A fourth model also exhibits a distinct form of interface dependency. As shown in Table 29, the gap between the MERGED and MONO configurations for Qwen2.5-72B narrows from approximately −32.0 percentage points (with a rigid parser) to −1.8 percentage points (when allowing for flexible delimiters and Python-style calls)—a range that includes zero. This suggests that, once different configurations trigger varying output formats, the scoring implementation can become the primary driver of the observed architectural performance gaps.

The experiments involving native tool calling lack robust external validation. For Gemma, the relevant units exhibit clear payload anomalies; meanwhile, for the Llama "required" units—the only ones the authors deem "clean"—the observed difference is −1.4 percentage points, with a confidence interval of [−4.7, +1.8].

I acknowledge that the 27B model and Llama continue to show a negative performance gap across the three text-based prompts, and I also note that the authors have disclosed these limitations. However, taken together, these results suggest that the current evidence only supports conditional conclusions—specific to certain text-based protocols—and is not yet sufficient to establish reliable, generalizable rules for architectural selection.

Questions for authors

Q1. After correcting for major implementation issues, what is the remaining gap attributable to the core architecture? Is it possible to conduct the primary comparison using a setup that includes the full schema, preserves all duplicate entries, employs a parser that adheres to the actual output format, and maintains consistent control over the scope of function calls? Specifically, for tasks where the correct answer necessitates the use of multiple functions, what is the performance delta (and its confidence interval) between this configuration and the MONO (monolithic) baseline? This result will be the deciding factor in whether I view the residual performance loss as a substantive cost of using a split architecture. Q2. Can the effects of information recovery and quantity constraints be isolated within the same planning outcome? Regarding information recovery: can we isolate this by keeping the original subtasks, providing only the relevant schema and a unified post-processing step, while simply adding the original user request? Regarding quantity constraints: can we reuse the same manifest to compare "workers" (agents/models) with and without explicit quantity constraints? The single-factor experiment on information mentioned in the appendix is ​​derived from the training set of an older evaluation harness; therefore, it cannot serve as a substitute for this primary comparison. Q3. To which specific, controlled system configuration do the practical recommendations apply? In Table 28, the three results under the "full schema + MERGED" category are inconclusive. Furthermore, in Appendix A, the sign of the performance gap for the 12B model shifts depending on the prompt used. Do the authors’ recommendations—specifically the advice "do not split for the sake of accuracy"—apply primarily to the original, "as-registered" configuration, or do they also extend to configurations that preserve function calls, provide sufficient planning information, and enforce consistent output formatting? What is the key evidence supporting the latter, broader scope? Q4. For tasks that genuinely require multiple functions, what magnitude of potential gains or losses can the existing data rule out? Could the primary analysis subset be defined based on the correct set of functions required for a task, and the performance difference between the corrected split architecture and the MONO baseline be reported directly—rather than relying on the heuristic of taking half the original "SPLIT" gap as a proxy for the "corrected" gap?

Reviewer_11ipOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the performance cost of decomposing LLM function calling across multiple specialized workers compared to using a single monolithic LLM. The authors study a planner-worker architecture in which a planner assigns functions to individual workers, and each worker independently generates the required function calls. They evaluate this approach on the BFCL benchmark using several LLMs and investigate different strategies to recover the performance degradation, including providing workers with full context, enforcing invocation counts, and adding a final correction pass. Their experiments show that the split architecture generally underperforms the monolithic approach, with a substantial portion of the degradation on some models caused by the system's aggregation rule that discards repeated function invocations. Although a correction pass can recover much of the lost accuracy, the authors find that a single full-context call without intermediate worker outputs performs similarly or better. Further analysis identifies incorrect function selection, missing invocations, and argument errors as major sources of failure. Overall, the authors conclude that partitioning function calling across specialized workers offers no demonstrated accuracy advantage over a single LLM call in their evaluated setting, while introducing additional computational costs and coordination challenges.

Critical strengths
  • Detailed empirical analysis: The paper provides a comprehensive empirical investigation of the performance degradation caused by splitting function calling across specialized workers. The authors evaluate several configurations and repair strategies, with detailed ablations and failure analyses that help identify where and why performance is lost.

  • Careful experimental methodology and transparency: The paper includes controlled comparisons, statistical significance tests, and multiple evaluation sets. The authors also openly discuss implementation issues, unsuccessful hypotheses, and limitations of their findings, which strengthens the credibility of the empirical analysis.

Critical weaknesses
  • Limited novelty and practical relevance: The main finding that decomposing tasks across multiple LLM agents can introduce information loss, coordination failures, and additional costs is already well established. Although the paper provides detailed empirical analysis, its differentiation from existing work is not sufficiently clear. More importantly, the evaluated architecture is highly simplified, and a substantial portion of the reported performance degradation comes from an aggregation rule introduced by the authors, which they acknowledge is not used by the surveyed agent frameworks. This raises concerns about both the novelty and practical relevance of the findings.

  • Limited generalizability: The main experiments involve only three models from two families, while the additional model evaluations are affected by parsing or implementation issues. Although the authors use four evaluation sets, all belong to the BFCL benchmark family and primarily involve relatively simple, single-turn function-calling tasks. This limited diversity makes it difficult to determine whether the findings extend to other models, more complex tasks, or realistic multi-agent systems.

Additionally, the paper appears to exceed the page limit slightly.

Questions for authors
  • Given that information loss, coordination failures, and additional costs in multi-agent systems have been investigated in prior work, what do the authors consider the main novel insights of this study beyond existing findings? In particular, how do the identified failure mechanisms and proposed repair strategies differ from those already discussed in related work?

  • How representative is the simplified, one-shot planner-worker architecture of practical multi-agent systems? In particular, since a substantial portion of the performance degradation comes from the first-item-wins aggregation rule, which Appendix G acknowledges is not used by the surveyed frameworks, to what extent do the authors expect the identified failure mechanisms to generalize to real-world systems?

  • Given that the main experiments involve only three models from two families and all evaluation sets belong to BFCL, how confident are the authors that the findings generalize to other model families, particularly more capable or function-calling-specialized models? Would the same conclusions hold for more complex, multi-step tasks involving dependencies between tool calls?

  • Several experimental configurations change multiple factors simultaneously. For example, FULL-CTX modifies both the information available to workers and their prompts, while CONTRACT changes planner and worker behavior. How do the authors distinguish the contributions of these individual changes? Could simpler, one-factor ablations provide stronger evidence for the proposed failure mechanisms?

  • Appendix A shows that the merge-controlled performance gap on Gemma-12B changes substantially under different prompt wordings, while the per-call F1 evaluation reduces the reported performance gaps and changes the ordering of failure categories. How do the authors interpret this sensitivity, and which of their main conclusions do they consider robust to these experimental choices? How might the parsing issues observed in additional model and native tool-calling experiments affect this interpretation?

  • The paper recommends avoiding function-call partitioning across workers for accuracy, although several corrected split configurations have small or statistically inconclusive differences from the monolithic baseline. Given the limited experimental setting and the lack of conclusive native tool-calling results, what evidence supports extending this recommendation beyond the particular architecture and tasks evaluated in the paper?

Reviewer_KHDiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes what happens when you give different "tool families" specialized agents which know how to use them when orchestrating function calls. They study what happens when a planner breaks down an input into different function calls and passes them to subagents on different function calling datasets. They have several hypothesis and findings that they describe throughout by analyzing the performance and behavior of different agentic architectures and harnesses. The main finding is that delegating ("splitting") to different specialized agents seems to decrease performance compared to just having 1 model do everything.

Critical strengths

The paper is very thorough and includes lot of experimental details that would otherwise typically be lost in a research paper, including bugs and implementation issues which may help future practitioners in the future.

Critical weaknesses

This paper is extremely difficult to read and understand overall. Every single page contains giant paragraphs of text and numbers and it is extremely unclear to even read the setup. The naming convention is also extremely unclear, as the results read from Hypothesis G0-G9 as well as H10-H12, and many of these terms are all defined one after another in a giant paragraph in section 3. (lines 200-213). The formatting and writing will need to be significantly reformatted in order to be much more readable and concise. The paper does not even have section headers such as 3.1, 3.2, only bolded paragraph headers.

This manuscript seems to document bugs and implementation issues discovered during the research process in the main paper, which makes the paper extremely difficult to read. For instance, in the Introduction under Contributions, line 69-70, it states that "The registered split loses 13.4, 10.8 and 3.9 points. Against a merge control that undoes our first-item rule it loses 7.1, 3.9 and 3.9." This seems to refer to a merge control which is described only on line 178 which states "that keeps every repeated item", which I interpreted to be that the merge control just stops repeated function calls from being deleted. These seem like ordinary bugs and fixes that can maybe be documented in the appendix, but are inordinately scattered through the main paper, making it very difficult to read. Other strange things are like on line 72-73: "The registered test of whether the rule is the majority of the harm (G0) is inconclusive on the Gemma models." which is simply very difficult to read.

Questions for authors

In the paper, you use the term "preregistration", what exactly does this mean in the context of this manuscript?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.