- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper prents AsyncFC that uses symbolic futures, turn-boundary result integration, and resource-access annotations to overlap model decoding with tool execution and enable concurrent calls without model fine-tuning. Evaluation spans BFCL, SWE-bench Lite, and HotpotQA. The experimental results suggest practical value, but accuracy preservation and the advance over prior work need stronger support.
Critical strengths
-
Standard-API compatibility, explicit resource-conflict handling, and conservative fallback for unannotated tools provide a practical deployment path. More precise annotations can progressively expose parallelism.
-
On 300 SWE-bench Lite issues, AsyncFC achieves a reported 1.40x speedup over the original SWE-agent, with resolution rates of 79.0% versus 79.3%, and a significant latency improvement over the parallel-executor variant (Figure 5). The no-annotation ablation and latency sweep further support the proposed mechanisms.
Critical weaknesses
-
Accuracy preservation is not established. Non-significant McNemar results do not demonstrate equivalence. BFCL v3 parallel-API accuracy changes from 67.3% to 66.0%. On BFCL v4, overflow exclusions leave only 31 and 29 of 50 composed workloads in the sequential and parallel settings; sequential-API accuracy changes from 54.8% to 53.2%. Matched-subset comparisons are useful, but full-set reliability remains unclear. Paired confidence intervals, a justified non-inferiority margin, and complete overflow/failure outcomes are needed.
-
The incremental contribution is insufficiently validated. Contrary to Section 2, lines 126-132, LLMCompiler's streamed planner already overlaps plan generation with execution and supports output placeholders and runtime dependency resolution. API compatibility and state-conflict scheduling remain useful distinctions, but comparisons only against synchronous baselines do not establish their added value. A matched streamed-planner comparison or targeted ablation would clarify the advance.
Ref: Kim et al., An LLM Compiler for Parallel Function Calling, ICML 2024, Sections 3.1-3.3 and 4.2. https://arxiv.org/html/2312.04511v3
Questions for authors
-
What are the accuracy and overflow/failure outcomes on all 50 BFCL v4 workloads? Is accuracy scored per original task or composed workload, and what accuracy loss can the paired confidence intervals exclude?
-
What benefit remains over LLMCompiler's streamed planner under matched models and tools, particularly from API compatibility and resource-conflict scheduling?
-
Given the 3.8% missed-dependency rate in Table 2, are generated annotations validated? What assumptions and tests support serial-equivalent state behavior, including unresolved session paths, missed conflicts, and failures after partial side effects?
-
Do all SWE-bench variants share the same executor, shell/session behavior, resource limits, and model settings? Please report absolute latency, token counts, and tool-call counts to separate scheduling benefits from executor changes and additional computation.