- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates a single-round invocation architecture based on function decomposition. A "planner" selects functions and formulates subtasks based on the user's request; the same model then acts as a "worker" for each function, and the resulting list of invocations is subsequently concatenated. The primary baseline for comparison is a single-turn model invocation that processes the full user request alongside the complete set of function schemas. The experiments encompass eleven configurations, primarily utilizing the Gemma-3-12B, Gemma-3-27B, and Llama-3.3-70B models. Core evaluations were conducted on 198 tasks from the "BFCL fresh" dataset—with five samples per task—and further validated across other BFCL splits and a collection of real-world user requests. The paper also presents experiments covering prompt rewriting, large function catalogs, the inclusion of a fourth model, parser sensitivity, and native tool-calling interfaces. Under the established text-output protocol, the decomposition approach resulted in performance drops of 13.4, 10.8, and 3.9 percentage points, respectively. Notably, the authors' decision to retain only the first instance of any repeated function call proved to be a significant source of performance loss for the two Gemma models; when these repeated entries were preserved, the performance gaps narrowed to 7.1, 3.9, and 3.9 percentage points. Furthermore, the study introduces an additional pass—where the model reviews the full user request to refine the list of invocations—which, in the case of Gemma-3-27B, reliably recovers more than half of the initial performance deficit. However, since this same refinement pass performs equally well (or even better) when provided with an empty candidate list, the observed improvement cannot be definitively attributed to the preceding decomposition step. The paper further breaks down the performance losses into categories—such as dropped invocations, incorrect function selection, and parameter errors—and discusses how these outcomes are influenced by prompts, parsers, evaluation metrics, and deployment interfaces.
Critical strengths
- In the current paper, the "ONE CALL" configuration uses the same prompts as the "PASS" (repair) configuration but provides an empty candidate list. This control addresses a critical interpretive question: is the improvement in final accuracy due to the effective integration of results from multiple workers in subsequent calls, or is it simply because the system re-obtains the full request and completes the task independently? Tables 2 and 3 show that, in the registered BFCL "fresh" experiments, the differences between "SPLIT + PASS" and "ONE CALL" are −4.6, −1.9, and −0.3 percentage points, respectively. For the 12B model, the entire confidence interval lies below zero; for the 27B model, the upper bound falls below the pre-specified two-percentage-point threshold; and for the Llama model, the results are inconclusive. Consequently, the positive result for the 27B model—where the repair call recovers most of the performance gap—cannot be attributed to the split architecture yielding benefits from effective collaboration. This control is methodologically significant, as it shifts the understanding of the source of the system's success, rather than merely serving as another performance baseline.
- The "MERGED" configuration in the paper reuses the same planner output but groups multiple entries for the same function together, sending them to a single worker. This allows for a direct examination of the impact of the "first-item-wins" rule. The paper further investigates the actual execution. In instances where a split occurs and the planner lists the same function multiple times, the worker’s output—when presented with only a single instance—aligns with that instance. After merging the entries, approximately 96% to 97% of the positions requiring repeated calls achieve the correct number of invocations. Figure 3 provides specific examples of this execution. This enables the authors to retract their initial hypothesis—that workers have an inherent tendency to call each function only once—and instead pinpoint the issue to the information hand-off process. Furthermore, the authors report both the raw loss events and the net difference after accounting for failures inherent to the "MONO" (monolithic) baseline, ensuring that not all failures in the split architecture are misattributed as "architectural" losses. This form of analysis offers greater explanatory power than a simple comparison of final accuracy rates.
Critical weaknesses
W1. In the paper, the original "SPLIT" approach results in performance drops of 13.4 and 10.8 percentage points for Gemma-12B and Gemma-27B, respectively; however, the "MERGED" approach recovers approximately 6.4 and 6.9 percentage points. Appendix G further reports that a survey of twelve open-source frameworks and the APIs of three major providers revealed no similar rules that discard redundant tool calls based on function names. The authors explicitly acknowledge that this "first-item-wins" behavior is a flaw in the current evaluation harness and planner contract.
While I recognize that the paper provides the "MERGED" control condition, a performance gap of 7.1, 3.9, and 3.9 percentage points remains in the primary experiments even after this correction. Consequently, the results cannot be entirely attributed to the aforementioned aggregation flaw; rather, the paper’s most prominent mechanistic finding appears to describe an implementation choice that lacks broad practical representativeness. Based solely on the remaining performance gap, it is impossible to determine whether the loss stems from splitting by function, or from insufficient information provided to the planner, the rewriting of subtasks, or other interface-related design choices.
Table 28 is particularly relevant to this issue. When both the full schema and the retention of redundant calls are enabled, the performance difference between "MERGED" and "MONO" (in the 64-function setting) is +1.1, −2.1, and −0.3 percentage points—all of which include zero in their respective intervals. While this does not definitively prove equivalence, it suggests that the corrected system does not exhibit a consistent, clear-cut performance loss.
To substantiate the practical recommendation—"do not split by function for the sake of accuracy"—the study would need to focus on implementations that properly retain tool calls and provide sufficient planning information, and then demonstrate that performance losses persist under those conditions. Otherwise, the paper’s findings more strongly support the avoidance of current errors related to aggregation and hand-off protocols.
W2. The paper’s motivation distinguishes between "information loss" and "coordination loss," yet the corresponding experiments fail to maintain a sufficiently clear, single-factor variation.
The "FULL-CTX" condition does not merely include the original request; it also incorporates the full schema, removes subtasks, and alters the instruction environment for the worker agents. Appendix E explicitly acknowledges that it alters multiple boundaries simultaneously. Furthermore, the general prompt requires the generation of all function calls requested by the user before filtering them down to the specified functions—a process that may, in itself, induce unauthorized outputs. Additionally, its post-processing approach is inconsistent with that of CONTRACT + CTX. Table 19 shows that, for the 12B model, the introduction of filtering and deduplication increased accuracy from 0.656 to 0.884; this suggests that the negative results for the original FULL-CTX were, in fact, confounded by issues related to output control.
Similarly, CONTRACT simultaneously modifies the planner prompt, the list of example calls, the worker prompt, and the constraints on the number of calls. The paper does not provide a control experiment that uses the same manifest but omits the quantity constraints imposed on the worker. The near-zero performance observed when counting errors occur does not, by itself, prove that quantity constraints are inherently detrimental; it is possible that such constraints merely force the execution of an already flawed plan. This touches upon the core scientific question of the paper. While the current experiments demonstrate the performance of these specific repair configurations, they do not yet fully disentangle the individual contributions of information recovery, quantity constraints, and execution control. A clearer isolation of these factors—maintaining a consistent planner, post-processing method, and scope of instructions—is required.
W3. The prompting experiments in Appendix A reveal that the performance gap between the MERGED and MONO configurations for Gemma-12B varies by prompt wording, yielding differences of −7.1, +0.8, and −3.0 percentage points, respectively. Wording A causes a decline of 11.3 percentage points in the MONO configuration alone—a drop that remains approximately 10.9 percentage points even after excluding unparseable responses. Consequently, the architectural comparison for this model is substantially influenced by the choice of prompt, rather than being primarily attributable to issues with format parsing.
A fourth model also exhibits a distinct form of interface dependency. As shown in Table 29, the gap between the MERGED and MONO configurations for Qwen2.5-72B narrows from approximately −32.0 percentage points (with a rigid parser) to −1.8 percentage points (when allowing for flexible delimiters and Python-style calls)—a range that includes zero. This suggests that, once different configurations trigger varying output formats, the scoring implementation can become the primary driver of the observed architectural performance gaps.
The experiments involving native tool calling lack robust external validation. For Gemma, the relevant units exhibit clear payload anomalies; meanwhile, for the Llama "required" units—the only ones the authors deem "clean"—the observed difference is −1.4 percentage points, with a confidence interval of [−4.7, +1.8].
I acknowledge that the 27B model and Llama continue to show a negative performance gap across the three text-based prompts, and I also note that the authors have disclosed these limitations. However, taken together, these results suggest that the current evidence only supports conditional conclusions—specific to certain text-based protocols—and is not yet sufficient to establish reliable, generalizable rules for architectural selection.
Questions for authors
Q1. After correcting for major implementation issues, what is the remaining gap attributable to the core architecture? Is it possible to conduct the primary comparison using a setup that includes the full schema, preserves all duplicate entries, employs a parser that adheres to the actual output format, and maintains consistent control over the scope of function calls? Specifically, for tasks where the correct answer necessitates the use of multiple functions, what is the performance delta (and its confidence interval) between this configuration and the MONO (monolithic) baseline? This result will be the deciding factor in whether I view the residual performance loss as a substantive cost of using a split architecture. Q2. Can the effects of information recovery and quantity constraints be isolated within the same planning outcome? Regarding information recovery: can we isolate this by keeping the original subtasks, providing only the relevant schema and a unified post-processing step, while simply adding the original user request? Regarding quantity constraints: can we reuse the same manifest to compare "workers" (agents/models) with and without explicit quantity constraints? The single-factor experiment on information mentioned in the appendix is derived from the training set of an older evaluation harness; therefore, it cannot serve as a substitute for this primary comparison. Q3. To which specific, controlled system configuration do the practical recommendations apply? In Table 28, the three results under the "full schema + MERGED" category are inconclusive. Furthermore, in Appendix A, the sign of the performance gap for the 12B model shifts depending on the prompt used. Do the authors’ recommendations—specifically the advice "do not split for the sake of accuracy"—apply primarily to the original, "as-registered" configuration, or do they also extend to configurations that preserve function calls, provide sufficient planning information, and enforce consistent output formatting? What is the key evidence supporting the latter, broader scope? Q4. For tasks that genuinely require multiple functions, what magnitude of potential gains or losses can the existing data rule out? Could the primary analysis subset be defined based on the correct set of functions required for a task, and the performance difference between the corrected split architecture and the MONO baseline be reported directly—rather than relying on the heuristic of taking half the original "SPLIT" gap as a proxy for the "corrected" gap?