acceptodds
Under review as a conference paper at ICLR 2027

Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs

Abstract

Function calling, also known as tool use, is a core capability of modern LLM agents but is typically constrained by synchronous execution semantics. Under these semantics, LLM decoding is blocked until each function call completes, resulting in increasing end-to-end latency. In this work, we introduce AsyncFC, a pure execution-layer framework that decouples LLM decoding from function execution, enabling overlap between model decoding and function execution as well as inter-function parallelism when dependencies permit. AsyncFC layers over existing models and unmodified function implementations, requiring no fine-tuning or changes to the standard synchronous function-calling protocol. Across function-calling and software engineering benchmarks, AsyncFC significantly reduces end-to-end task completion time while preserving task accuracy. Furthermore, these results reveal that LLMs possess a native capability to reason over symbolic futures that represent unresolved execution results, enabling an asynchronous paradigm for model-tool interaction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool Use, Function Calling, Efficiency, Agent Systems, Concurrency, Asynchronous Execution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vNscOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper prents AsyncFC that uses symbolic futures, turn-boundary result integration, and resource-access annotations to overlap model decoding with tool execution and enable concurrent calls without model fine-tuning. Evaluation spans BFCL, SWE-bench Lite, and HotpotQA. The experimental results suggest practical value, but accuracy preservation and the advance over prior work need stronger support.

Critical strengths
  1. Standard-API compatibility, explicit resource-conflict handling, and conservative fallback for unannotated tools provide a practical deployment path. More precise annotations can progressively expose parallelism.

  2. On 300 SWE-bench Lite issues, AsyncFC achieves a reported 1.40x speedup over the original SWE-agent, with resolution rates of 79.0% versus 79.3%, and a significant latency improvement over the parallel-executor variant (Figure 5). The no-annotation ablation and latency sweep further support the proposed mechanisms.

Critical weaknesses
  1. Accuracy preservation is not established. Non-significant McNemar results do not demonstrate equivalence. BFCL v3 parallel-API accuracy changes from 67.3% to 66.0%. On BFCL v4, overflow exclusions leave only 31 and 29 of 50 composed workloads in the sequential and parallel settings; sequential-API accuracy changes from 54.8% to 53.2%. Matched-subset comparisons are useful, but full-set reliability remains unclear. Paired confidence intervals, a justified non-inferiority margin, and complete overflow/failure outcomes are needed.

  2. The incremental contribution is insufficiently validated. Contrary to Section 2, lines 126-132, LLMCompiler's streamed planner already overlaps plan generation with execution and supports output placeholders and runtime dependency resolution. API compatibility and state-conflict scheduling remain useful distinctions, but comparisons only against synchronous baselines do not establish their added value. A matched streamed-planner comparison or targeted ablation would clarify the advance.

Ref: Kim et al., An LLM Compiler for Parallel Function Calling, ICML 2024, Sections 3.1-3.3 and 4.2. https://arxiv.org/html/2312.04511v3

Questions for authors
  1. What are the accuracy and overflow/failure outcomes on all 50 BFCL v4 workloads? Is accuracy scored per original task or composed workload, and what accuracy loss can the paired confidence intervals exclude?

  2. What benefit remains over LLMCompiler's streamed planner under matched models and tools, particularly from API compatibility and resource-conflict scheduling?

  3. Given the 3.8% missed-dependency rate in Table 2, are generated annotations validated? What assumptions and tests support serial-equivalent state behavior, including unresolved session paths, missed conflicts, and failures after partial side effects?

  4. Do all SWE-bench variants share the same executor, shell/session behavior, resource limits, and model settings? Please report absolute latency, token counts, and tool-call counts to separate scheduling benefits from executor changes and additional computation.

Reviewer_eHKZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AsyncFC makes function calling asynchronous without touching the model or the functions: schemas are rewritten so every call returns a future, decoding continues, resolved values arrive as user messages at turn boundaries, and an await_future helper requests a value explicitly. An optional annotation scheme, compiled onto a state tree with read/write region analysis, decides which calls may overlap. Across BFCL v3 with injected latency, BFCL v4, SWE-bench Lite and HotpotQA it reports 1.2x-1.4x speedups at parity accuracy, with a bound (Eq. 1) and a decomposition of the savings.

Critical strengths

The protocol adaptation is the strongest part. A future placeholder satisfies the call-return contract, input schemas are extended to accept futures so an unresolved result can be passed to a later call, and structured futures derived from output-schema examples expose pending fields. Passing a future as an argument is the concrete advance over prior asynchronous tool-use work.

The safety default is well chosen: unannotated functions fall back to root-level read and write sets and are serialized, so correctness does not hinge on developer diligence. Reporting is careful: paired one-sided tests, McNemar, bootstrap intervals, a baseline-favourable convention for Parallel FC, and disclosure of the one non-significant result.

Critical weaknesses

The mechanism is not new. Futures as handles, read/write sets to derive dependencies, and independent nodes running concurrently are what a DAG scheduler or a build system already does, and await exists in every async runtime; carrying the pattern into the function-calling protocol is a protocol adaptation rather than a mechanism, a narrower claim than the paper makes. Nor do the baselines support even that narrower claim: they are all synchronous — Sequential FC, Parallel FC, the unmodified SWE-agent — and the two asynchronous approaches (Gim et al., 2024; Ginart et al., 2024) together with the planning-based LLMCompiler (Kim et al., 2024) appear only in Section 2, set aside in prose as requiring model-side changes, without a mechanism-level comparison or an experiment.

The most informative baseline is missing, and it costs almost nothing to build. Coding agents already obtain decode-execution overlap with no model or protocol change: launch a command in the background, get an identifier at once, keep decoding, read the output later. That is the same manoeuvre as the turn-boundary injection here (Section 3.2), with no schema rewrite, no annotations and no state tree. Since Figures 3 and 4 credit the overlap to the runtime, a reader cannot tell how much of 1.26x on BFCL v4 or 1.24x on HotpotQA comes from the futures rather than from background execution. Dependency-safe concurrency and futures as arguments are what the runtime adds, and that is the part I would want argued and measured.

The "models natively reason over futures" framing is not established by the evidence offered: placeholders and async-await are dense in pretraining data, so decoding past an unresolved handle is in-distribution instruction following, and parity accuracy shows only that nothing breaks.

The headline numbers, as constructed, measure throughput rather than single-request latency. BFCL v4 composes 100 cases pairwise and HotpotQA composes 100 tasks into 50 pairs, so unrelated requests share a context, parallelism that two independent processes would also obtain. Single-request speedup is never isolated, and the overflow filter leaves n=31. BFCL v3 injects a constant 5s per call into a backend the authors call negligible-latency, which is the case most favourable to overlap, while one slow call on a heavy-tailed backend absorbs the slack.

Questions for authors

See weakness

Reviewer_QC8jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the AsyncFC technique that allows parallel asynchronous execution of function-calling and LLM decoding, as well as multiple parallel execution of function calls when there are no dependency issues. This is enabled by using future placeholders and hence can work without any further fine-tuning or changes to LLMs as long as they possess the ability to reason over symbolic futures. They implement this technique and show its benefits across 3 tasks and their corresponding benchmarks. Their best improvement is on coding tasks, where they achieved 1.4x e2e latency speedup without any statistically significant change in task accuracy.

Critical strengths
  • e2e latency optimization is a valuable problem to look at and the technique doesn't reply on pre-training techniques or fine-tuning LLMs to achieve it, making it a very viable idea for use.
  • In the reported limited validation, this shows promising results. Is there a simple way for developers to plug-in and use it for their tasks, releasing something like that would be valuable.
Critical weaknesses
  • For each task, one agent and one benchmark was evaluated. It would be good to validate it further across multiple benchamrks for a given task, using multiple agents for the task, and not just a high performing one. -Similarly, the cross-model evaluation just evaluated with a single other model.
Questions for authors
  • In sec. 2, you mentioned Planning-based FC that uses dependency graphs. How would combining that with AsyncFC behave? Does it add significantly more benefits? would be good to evaluate that.
  • Clearly AsyncFC adds a lot of benefits. What is the tradeoff? One can say the annotation expected from developers upfront is some minor inconvenience added (which could be worth the benefits) but as you mention, it works even without the annotation by treating everything to be dependent. We still are expected to get gains on the decoding-function execution parallelism, so are we not losing anything at all in doing this?
  • Fix typo in Sec 5.3 (double use of equation)
  • Your results seem to show that the benefits are highest for coding tasks, do you have an intuition on why that is the case? Also, to dig further into that, would be beneficial to validate that observation by experimenting with a sub-optimal coding agent OR on other coding benchmarks as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.