acceptodds
Under review as a conference paper at ICLR 2027

SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

Abstract

LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model's output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.37× improvement in end-to-end throughput over autoregressive decoding.

Primary area:
optimization
Keywords:
LLM agents, tool calling, structured generation, speculative decoding, slot-parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vEotOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents SchemaFill, a speculative decoding method specifically for tool calling. It uses a forecast-draft-then-verify loop, exploiting the inherent structure of tool calls to predict possible function call slots and fields, to draft field values, and to verify the drafted tokens. It presents the algorithm and the KV cache management scheme to make the methodology possible. It evaluates the method and demonstrates an increase in throughput over baselines.

Critical strengths

The paper is generally easy to follow, with the methodology clearly explained and illustrated. The evaluation is sound and demonstrates a substantive increase in throughput over baselines. The ablations are appropriate and demonstrates where the gains in performance comes from.

Critical weaknesses

The application of the methodology is narrow by design to target tool calling. It does not have broader application to code generation etc. It is also missing related work to parallel decoding techniques such as ASPD[1], APAR[2], PASTA[3], APR[4], Multiverse[5].

Algorithm 1 is a bit difficult to understand. The purpose and operation of the helper functions are not always obvious (e.g. PreparedSIDE, AlignForecast).

[1] Chen, Keyu, et al. "Aspd: Unlocking adaptive serial-parallel decoding by exploring intrinsic parallelism in llms." arXiv preprint arXiv:2508.08895 (2025).

[2] Liu, Mingdao, et al. "Apar: Llms can do auto-parallel auto-regressive decoding." arXiv preprint arXiv:2401.06761 (2024).

[3] Jin, Tian, et al. "Learning to keep a promise: Scaling language model decoding parallelism with learned asynchronous decoding." arXiv preprint arXiv:2502.11517 (2025).

[4] Pan, Jiayi, et al. "Learning adaptive parallel reasoning with language models." arXiv preprint arXiv:2504.15466 (2025).

[5] Yang, Xinyu, et al. "Multiverse: Your language models secretly decide how to parallelize and merge generation." Advances in Neural Information Processing Systems 38 (2026): 100773-100804.

Questions for authors
  • How do different syntax formats of the tool calls (e.g. XML vs JSON) affect the performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vEot:

We thank the reviewer for recognizing that “the evaluation is sound and demonstrates a substantive increase in throughput over baselines.” Below, we explain the design choices behind these improvements and present additional results supporting their applicability across structured output formats.

W1 / Q1: Applicability to structured generation

W1: “narrow by design to target tool calling.”

SchemaFill targets lossless acceleration of schema-guided structured generation, with tool calling as the evaluated application. Schemas expose candidate fields, types, and nesting before their values are generated, enabling speculative preparation of future content. This principle applies across schema-defined outputs, an important workload as agents increasingly interact with tools and services through machine-readable interfaces. General code generation presents a different setting, where the concrete program structure must itself be synthesized.

To substantiate this broader applicability, we extend our evaluation beyond JSON to an additional structured format, XML, directly addressing the reviewer’s question:

Q1: “How do different syntax formats of the tool calls (e.g. XML vs JSON) affect the performance?”

DatasetFormatAR TPSSchemaFill TPSRelative TPSAL
BFCLJSON92.70406.124.381×6.9721
BFCLXML89.66383.494.277×7.3666
GlaiveJSON91.36313.723.434×5.8244
GlaiveXML87.36313.933.593×6.3623

Each speedup uses the corresponding same-format AR baseline. XML uses a format-adapted drafter; the target remains unchanged. SchemaFill retains substantial acceleration on XML—4.28× on BFCL and 3.59× on Glaive—supporting its applicability across structured serialization formats.

W2: Distinction from parallel generation methods

“missing related work to parallel decoding techniques such as ASPD[1], APAR[2], PASTA[3], APR[4], Multiverse[5].”

The central distinction is lossless acceleration of the original target. These methods change the generation protocol and the context available to individual branches, which generate without some content that would precede them in serial decoding. Whenever that omitted context affects target predictions, distributional deviation is unavoidable without an exact correction mechanism. Comparable task quality does not imply preservation of the original target distribution.

Our motivation experiment illustrates this issue: even with an oracle call-and-field layout, the fraction of responses containing independently generated values that deviate from greedy AR rises from 3.3% to 57.5% across eight slots, and from 7.6% to 47.1% across three calls.

SchemaFill keeps parallel values provisional. Every value and structural token is verified under the actual serialized prefix, with speculative acceptance and correction preserving the target distribution. Parallel preparation therefore accelerates generation without requiring slot independence.

We thank the reviewer for highlighting that our related-work discussion did not make this distinction sufficiently clear. We will include all five references and refine the discussion and presentation of our method.

W3: Clarifying Algorithm 1

“Algorithm 1 is a bit difficult to understand. The purpose and operation of the helper functions are not always obvious (e.g. PreparedSIDE, AlignForecast).”

The algorithm follows four steps:

  1. Forecast: Identify candidate calls and argument slots from public metadata, then initialize MAIN and speculative SIDE branches.
  2. Draft and pack: PreparedSIDE collects eligible candidates available before the current round. MAIN and SIDE draft concurrently, and available candidates are packed with proposed structural tokens into a MAIN continuation.
  3. Verify and commit: The target verifies proposals with isolated branch attention. Only the accepted MAIN prefix and target correction become output. SIDE results remain provisional until verified under the actual MAIN prefix.
  4. Update: AlignForecast reconciles forecast slots with newly verified output, updating assignments and retiring incompatible predictions.

We thank the reviewer for identifying this clarity issue. We will revise Algorithm 1 and its explanation to make the execution order and helper functions explicit.

We thank the reviewer again for the constructive feedback and welcome further discussion. We would be happy to address any additional questions or clarify any aspect of our work.

Thanks, my concerns and questions have been addressed.

Thank you for your thoughtful feedback and support. We’re glad that our response has addressed your concerns and questions. If the additional results have further strengthened your confidence in the paper, we would be grateful if you could consider a higher rating.

Reviewer_XoFQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SchemaFill targets structured tool calls, where a response interleaves schema-predictable structure with typed value slots that depend on earlier fields and calls, so AR decoding serializes partly predictable work. It forecasts call positions and fields with BM25 over tool metadata, drafts future values on isolated SIDE branches with a shared EAGLE-3 drafter, packs them with the intervening structural text across calls, and verifies the result as one MAIN proposal under the actual prefix; MAIN and SIDE share a single target tree forward on a branch-isolated KV runtime. On Glaive and BFCL v3 with Qwen2.5-7B-Instruct it reports 3.45x and 4.37x over AR, 1.39x and 1.98x over EAGLE-3, 1.56x and 1.58x over ToolSpec, and a greedy-consistency proof (Appendix B).

Critical strengths

The motivation is measured, not asserted: Section 3.3 and Figure 2 quantify AR deviation from independently drafted slots on BFCL (3.3% of responses after one slot to 57.5% after eight). EAGLE-3 and ToolSpec are re-run in-house, and EAGLE-3 shares SchemaFill's drafter head (Appendix A), removing the usual confound. Measurement is above average: three launches with standard deviation (Table 1), KV overhead (Table 4), phase-level timing (Table 5), temperature 1 (Appendix C.5), four targets (Appendix D).

Critical weaknesses
  1. The verification budget is not matched. Nodes per verification are reported for SchemaFill only (23.43 to 42.71; Table 3, Figure 4); EAGLE-3's and ToolSpec's tree sizes and draft budgets are never given. Because acceptance length and throughput scale with candidate compute, the 1.98x over EAGLE-3 (Table 1) cannot be separated from a larger tree. EAGLE-3 at matched nodes per verification is the missing control.

  2. The title mechanism explains a minority of the gain. In Table 2, stage B (structural packing, no SIDE branches) supplies about 54% of the delta from EAGLE-3 to full SchemaFill, SIDE slot drafting 18%, cross-call packing 28%; the speedup therefore comes mostly from predictable structural text, closest to grammar-constrained acceleration. Appendix C.4 also reports B and C as single runs, so the 11.62% credited to SIDE drafting has no error bar.

  3. Output validity is proved, not measured. Appendix B assumes each MAIN verification yields serial AR's logits at that prefix, which BF16 tree attention does not guarantee, yet Section 5.1 defines only throughput and acceptance length: no schema-validity, executability or accuracy number, no token-level agreement rate with AR. "Target AR agreement (%)" is an undefined Figure 1 axis label; failed requests are pooled but never counted.

  4. The regime is narrow for a serving claim: batch one, a single GPU, 7-8B targets (Appendix A), 650 Glaive examples, three of BFCL's 25 categories, no latency percentiles or TTFT, no constrained-decoding baseline. The call density the method needs is also not the common one: Section 5.2 puts the broader pool at 1.00 call and 1.88 top-level fields per decision, the setting in which cross-call packing has least to do, and the two datasets differ by 0.6x in reported speedup (1.39x against 1.98x). Worst cases are unprobed: Table 3 varies only call count and is count-informed (Section 5.4), with no result by field count, no tools outside the drafter's corpora, no BM25 forecast hit rate.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer XoFQ [1/3]:

We thank the reviewer for recognizing our empirically grounded motivation and the quality of our experimental measurements. Below, we present additional matched-budget results and clarify the contributions of the proposed mechanisms, numerical consistency, and evaluation scope.

W1. Verification budget

“EAGLE-3 at matched nodes per verification is the missing control.”

We evaluated all three methods under the same verification-node budget, and SchemaFill maintains a substantial throughput advantage. Here, nodes per verification measures the average tree size processed by the target model in each verification pass. We matched this average tree size across SchemaFill, EAGLE-3, and ToolSpec.

DatasetMethodAverage tree size (nodes / verification)Acceptance lengthThroughput (tokens/s)Speedup over AR
BFCLEAGLE-335.403.05209.872.26×
BFCLToolSpec35.403.31247.062.67×
BFCLSchemaFill35.406.97406.124.38×
GlaiveEAGLE-348.243.50227.422.49×
GlaiveToolSpec48.242.76198.692.18×
GlaiveSchemaFill48.245.82313.723.43×

At the same average tree size, SchemaFill achieves 1.94× / 1.38× EAGLE-3’s throughput and 1.64× / 1.58× ToolSpec’s throughput on BFCL / Glaive. Its higher acceptance length shows that SchemaFill commits more output tokens from the same verification-node budget, translating into higher end-to-end throughput.


W2. Contribution of the proposed mechanisms

“stage B … supplies about 54% of the delta” / “the speedup therefore comes mostly from predictable structural text”

We agree that exploiting predictable structure contributes substantially to the speedup. However, our original ablation grouped structural prediction with mechanisms for dynamically correcting forecast errors and realigning slots within configuration B, obscuring their distinct roles. We added a structural-prediction-only control to provide a finer breakdown.

ConfigurationDescriptionAcceptance lengthThroughput (tokens/s)Speedup over A
AEAGLE-32.87205.111.00×
A + structural predictionStructural prediction only4.43285.941.39×
BSchema-guided MAIN path4.82313.451.53×
CB + SIDE slot drafting5.85349.881.71×
DC + cross-call packing6.97406.511.98×

The added control confirms the benefit of structural prediction. Configuration B additionally corrects forecast errors and realigns slots dynamically as generation proceeds. Forecasted tools and fields may differ from those ultimately generated by the target, leaving future-slot candidates associated with an incorrect anticipated context. These mechanisms adjust the forecasts and slot alignment to follow the target’s committed output, allowing subsequent candidates to be verified under the actual prefix.

Dynamically correcting forecast errors and realigning slots enables SIDE drafting and cross-call packing to operate despite these forecast mismatches. Compared with structural prediction alone, these mechanisms, together with SIDE drafting and cross-call packing, increase throughput from 285.94 to 406.51 tokens/s (+42.2%). The ablation therefore measures incremental gains from mechanisms that operate together, rather than independent contributions. We will revise the framing and configuration definitions to clarify these roles and their relationship.

The need for configuration B’s dynamic correction and slot realignment is directly reflected in BM25’s prediction results. This analysis also addresses the “no BM25 forecast hit rate” concern in W4. BM25 retrieves relevant tools with high accuracy: Top-1 hit rate measures whether the highest-ranked tool appears in the target’s committed tool set, and Macro Recall@3 measures the average fraction of that set covered by the three highest-ranked tools.

DatasetTop-1 hit rateMacro Recall@3
BFCL99.67%99.22%
Glaive97.89%100.00%

Retrieving the correct tools leaves substantial uncertainty about their execution order. A separate BFCL call-order diagnostic shows:

Evaluation subsetOrdered coverageExact sequence match
BFCL overall55.78%17.33%
BFCL multi-tool subset35.50%6.50%

Ordered coverage requires the required tools to appear in the correct order while allowing additional tools between them. Exact sequence matching requires the entire predicted tool-call sequence to match.

BM25 retrieves useful tools but frequently mispredicts their order, placing future-slot candidates in an incorrect anticipated context. Configuration B dynamically corrects forecasts and realigns the slots with the target’s committed output, enabling SIDE drafting and cross-call packing to continue under the actual prefix.


AuthorsOpenReview ↗

Response to Reviewer XoFQ [2/3]

W3. Output agreement and numerical verification

“‘Target AR agreement (%)’ is an undefined Figure 1 axis label”

Target AR agreement requires exact equality of the entire generated token sequence: the same sequence length and the same token ID at every position. Our supplementary evaluation achieves 100.00% exact AR agreement. We will state this definition and precision setting alongside Figure 1.

“Output validity is proved, not measured.” / “which BF16 tree attention does not guarantee”

We measured both complete output-token agreement and conditional-distribution differences against target AR. Every request in the supplementary evaluation produces the same output under SchemaFill and AR. Deterministic schema-validity, executability, and accuracy evaluations consequently return the same results for those outputs.

To examine numerical differences directly, we compared actual MAIN verification logits and serial AR logits at identical prefixes, with MAIN, SIDE drafting, and cross-call packing enabled.

The diagnostic covers two BFCL requests. We compute DKL(PAR∥PSchemaFill)D_{\mathrm{KL}}(P_{\mathrm{AR}}\Vert P_{\mathrm{SchemaFill}}) over the full 152,064-token vocabulary. Within each precision, the rows correspond to the same two requests in order.

Target precisionMean KL (nats)Maximum KL (nats)
BF165.03e−84.09e−6
BF162.68e−61.40e−4
FP649.75e−197.02e−17
FP64−2.27e−181.13e−16

Both requests produce identical complete token sequences under AR and SchemaFill in both precision settings. Increasing target precision to FP64 reduces the measured distribution differences to approximately numerical roundoff. The small negative KL estimate is roundoff in the direct calculation and is reported without clipping.


AuthorsOpenReview ↗

Response to Reviewer XoFQ [3/3]

W4. Evaluation setting and generalization

“batch one, a single GPU”

Larger-batch or multi-GPU evaluations introduce additional implementation-dependent factors, including batching policies, request scheduling, and inter-device communication, that also affect end-to-end performance. To focus on per-request decoding acceleration, we follow the batch-one evaluation setting used in Medusa (ICML 2024), PEARL (ICLR 2025), Accelerating Blockwise Parallel Language Models with Draft Refinement (NeurIPS 2024), Speculative Streaming (EMNLP 2025), and EAGLE-3 (NeurIPS 2025), and conduct our experiments on a single GPU. We will clarify this evaluation scope in the revised manuscript.

“The regime is narrow for a serving claim: … three of BFCL’s 25 categories” / “the two datasets differ by 0.6x in reported speedup”

We chose two substantially different workloads to evaluate how SchemaFill benefits from different amounts of available parallelism. The selected BFCL multi-call categories exercise both slot-level parallelism and cross-call packing, while Glaive primarily exercises parallel argument generation within a single call. This contrast lets us test whether SchemaFill remains effective with a single tool call and how much additional acceleration multiple calls provide. In the matched-budget evaluation, SchemaFill achieves 1.94× EAGLE-3’s throughput on BFCL and 1.38× on Glaive, demonstrating substantial acceleration in both settings.

“Section 5.2 puts the broader pool at 1.00 call and 1.88 top-level fields per decision, the setting in which cross-call packing has least to do” / “Worst cases are unprobed” / “no latency percentiles”

We deliberately included a predominantly single-call dataset to test whether SchemaFill still provides substantial acceleration when cross-call packing has little opportunity to contribute. For a request containing only one tool call, SchemaFill exploits parallelism across argument slots within that call.

Even in this less favorable workload, the acceleration remains substantial. On Glaive, SchemaFill achieves 1.38× EAGLE-3’s throughput and 1.58× ToolSpec’s throughput under the same verification-node budget, and 3.43× the throughput of AR decoding.

We also report end-to-end request latency percentiles on both evaluation cohorts, including P95 and P99 to characterize tail latency:

DatasetP50 latency (ms)P95 latency (ms)P99 latency (ms)
BFCL186.28573.67833.40
Glaive135.02300.11396.35

Together, these results quantify acceleration in the low-call-density workload and the latency distribution across both datasets.

“no constrained-decoding baseline”

SchemaFill and constrained decoding have different objectives: SchemaFill accelerates generation, while constrained decoding restricts outputs to satisfy a grammar or schema. Token masking alone does not reduce sequential target-model decoding steps.

ToolSpec provides a constrained-decoding-like drafting baseline. Its finite-state machine restricts structural candidates to a small, schema-defined token set for target verification. At the same verification-node budget, SchemaFill achieves 1.64× its throughput on BFCL and 1.58× on Glaive.


We thank the reviewer for the constructive feedback and for recognizing the measured motivation and care taken in our evaluation. Our additional experiments and analyses directly address the concerns you raised about verification budgets, individual mechanism contributions, numerical agreement, and evaluation coverage. We will incorporate these results and clarifications into the revised manuscript.

In light of this additional evidence, we would appreciate your reconsideration of the score. If any concerns remain that prevent a more favorable assessment, we would welcome the opportunity to address them.

Dear Reviewer XoFQ,

Thank you again for your time and thoughtful feedback. We would like to follow up on our response and ask whether you have any remaining questions or concerns. We welcome further discussion and would be glad to provide any additional clarification.

If our response and additional results have addressed your concerns, we would be grateful if you could consider a more favorable score in light of this evidence.

Thank you for your consideration.

Reviewer_TkNSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SchemaFill, a speculative decoding framework for LLM tool calling. The key idea is to exploit the structured nature of tool-call responses: instead of decoding argument values strictly left-to-right, SchemaFill forecasts possible future tool/slot positions, drafts slot values in parallel on side branches, and then packs these candidates with structural text for verification by the target model under the actual committed prefix. The system uses BM25 over tool metadata to rank candidate tools, expands slots, and manages a shared target-KV arena with branch-isolated attention. Experiments on Glaive and BFCL v3 with Qwen2.5-7B-Instruct report up to 4.37× end-to-end throughput over autoregressive decoding, outperforming EAGLE-3 and ToolSpec. The paper includes a greedy consistency proof and cross-target evaluations (Qwen3, Mistral, Hermes).

Critical strengths
  1. Principled verification framework. The core mechanism—keeping all SIDE drafts provisional and re-verifying every token under the actual output prefix—is sound and well-argued. The greedy consistency proof in Appendix B is a meaningful formal guarantee: the method cannot change the target's output distribution. This matters because it isolates the contribution to a pure latency/throughput problem rather than trading off quality.

  2. Cross-target evaluation adds credibility. Showing gains on four different model families (Qwen2.5, Qwen3, Mistral, Hermes) with per-target EAGLE-3 heads is more than many speculative decoding papers do. It suggests the packing/verification machinery is not tied to one model's quirks.

Critical weaknesses
  • The headline "slot-parallel" contribution is marginal; most of the gain comes from simpler structural packing. Table 2 is revealing: Stage B (schema-guided MAIN path with tool/slot forecasting and structural concatenation, no SIDE branches) already achieves 1.53× TPS over EAGLE-3. Adding SIDE parallel drafting (Stage C) contributes only +11.6% TPS, and cross-call packing (Stage D) adds +16.2%. The paper's central claim—parallel speculation over future slots—accounts for roughly 28% of the total 1.98× speedup over EAGLE-3. The framing overstates the novelty: this is primarily a schema-aware packing method with a parallel drafting add-on, not "slot-parallel speculative decoding" as the title suggests.

  • Forecast accuracy is unvalidated, and scaling results undercut the motivation. BM25 tool ranking is a heuristic; the paper never reports how often the forecasted tool is actually the one the target calls. If forecasts are wrong, SIDE computation is pure overhead. More troubling, Table 3 and Figure 4 show throughput degrades at 4 reference calls (410.72 TPS) vs. 2–3 calls (~440 TPS), even when the forecast budget is set to the oracle call count B=C. This is exactly the regime—many calls, many fields—where the paper claims the method should shine. The scaling story is backwards, and the paper attributes it vaguely to "slot-drafter quality" without analysis.

Questions for authors

What is the tool-level forecast accuracy of BM25 on your evaluation sets? Specifically, what fraction of SIDE drafting compute is spent on tools that the target never selects? Without this number, the efficiency accounting is incomplete.

Why does throughput drop at 4 calls in Table 3 despite B=C? Is the drafter's acceptance rate falling on harder multi-call requests, or is there a packing/scheduling bottleneck? This directly affects the claim that SchemaFill is particularly suited to complex multi-call requests.

In Table 2, Stage B (schema-guided MAIN without SIDE) already delivers 1.53× over EAGLE-3. What is the precise division of labor you intend between the "slot-parallel" claim in the title/abstract and the structural packing that seems to drive most of the speedup?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TkNS [1/2]

We thank the reviewer for recognizing that our verification mechanism is “sound and well-argued” and that our cross-target evaluation adds credibility. Below, we clarify each mechanism’s contribution and present additional results on forecast accuracy and the trade-off between verification budgets and throughput.


W1 & Q3 — Contribution of slot-parallel drafting

W1: “The headline "slot-parallel" contribution is marginal; most of the gain comes from simpler structural packing.”

Q3: “In Table 2, Stage B (schema-guided MAIN without SIDE) already delivers 1.53× over EAGLE-3. What is the precise division of labor you intend between the "slot-parallel" claim in the title/abstract and the structural packing that seems to drive most of the speedup?”

Response:

Thank you for highlighting this distinction. Our original ablation grouped structural prediction with dynamic forecast correction and slot realignment within Stage B, obscuring their distinct roles. We added a structural-prediction-only control to provide a finer breakdown on the submission’s BFCL evaluation cohort.

ConfigurationDescriptionAL ↑TPS ↑Speedup over A
AEAGLE-32.87205.111.00×
A + structural predictionStructural prediction only4.43285.941.39×
BSchema-guided MAIN path4.82313.451.53×
CB + SIDE slot drafting5.85349.881.71×
DC + cross-call packing6.97406.511.98×

Stage B’s key contribution is correcting forecast errors and maintaining slot alignment as generation proceeds. When predicted tools, call multiplicities, or execution order differ from the target’s output, Stage B updates the forecast-to-slot alignment so that candidates can be verified under the actual prefix.

The need for this mechanism is reflected in a separate BFCL call-order diagnostic:

Evaluation subsetOrdered coverageExact sequence match
BFCL overall55.78%17.33%
BFCL multi-tool subset35.50%6.50%

Ordered coverage requires the realized tool-call sequence to appear in the correct order within the forecast, while allowing additional predicted calls between them. Exact matching requires the entire predicted sequence to agree.

The frequent sequence mismatches highlight the need for dynamic realignment, which allows SIDE drafting and cross-call packing to continue as the target’s output diverges from the forecast.

These stages jointly support slot-parallel speculative decoding through interdependent roles: structural prediction connects value slots; Stage B maintains the alignment needed for Stage C’s parallel SIDE drafting; and Stage D packs the resulting candidates across calls for joint verification. Together, Stages B–D increase throughput from 285.94 to 406.51 tokens/s (+42.2%) over structural prediction alone.

We will revise the title/abstract framing and ablation definitions to present SchemaFill as an integrated forecast–draft–align–pack–verify framework and clearly identify each mechanism’s contribution.


W2 & Q1 — Forecast accuracy and unused predictions

W2: “Forecast accuracy is unvalidated, and scaling results undercut the motivation. BM25 tool ranking is a heuristic; the paper never reports how often the forecasted tool is actually the one the target calls. If forecasts are wrong, SIDE computation is pure overhead.”

Q1: “What is the tool-level forecast accuracy of BM25 on your evaluation sets? Specifically, what fraction of SIDE drafting compute is spent on tools that the target never selects? Without this number, the efficiency accounting is incomplete.”

Response:

Thank you for requesting this breakdown. BM25 retrieves relevant tools with high accuracy, as shown below:

DatasetTop-1 hit rateMacro Recall@3
BFCL99.67%99.22%
Glaive97.89%100.00%

Top-1 hit rate measures whether the highest-ranked tool appears in the target’s committed tool set. Macro Recall@3 measures the average fraction of that set covered by the three highest-ranked tools.

The main forecasting challenge is predicting tool execution order. As explained in W1 & Q3, dynamic forecast correction and slot realignment allow SchemaFill to handle sequence mismatches and reuse applicable SIDE candidates.

A supplementary BFCL diagnostic shows that 14.37% of predicted call positions are unused, primarily due to overpredicting repeated calls to relevant tools. SIDE drafts for these positions are not used, resulting in redundant computation. Better prediction of call order and multiplicity can reduce this overhead and prepare more useful candidates. Improved predictors can replace BM25 within the same drafting and verification framework.


AuthorsOpenReview ↗

Response to Reviewer TkNS [2/2]

W2 & Q2 — Verification budgets and throughput

W2: “More troubling, Table 3 and Figure 4 show throughput degrades at 4 reference calls (410.72 TPS) vs. 2–3 calls (~440 TPS), even when the forecast budget is set to the oracle call count B=C.”

Q2: “Why does throughput drop at 4 calls in Table 3 despite B=C? Is the drafter's acceptance rate falling on harder multi-call requests, or is there a packing/scheduling bottleneck? This directly affects the claim that SchemaFill is particularly suited to complex multi-call requests.”

Response:

Thank you for raising this question. Small-batch autoregressive decoding is often memory-bound, leaving compute capacity underutilized. Speculative decoding uses this available capacity to trade additional computation for faster generation: it verifies multiple candidates in a single target forward pass, amortizing memory-access costs and reducing serial target calls when candidates are accepted.

This trade-off has diminishing returns. As the candidate budget grows, drafting and verification require more computation, while the additional accepted output may grow more slowly or plateau. Beyond a favorable operating point, processing more candidates can therefore reduce end-to-end throughput. SchemaFill follows the same principle: parallel drafting across slots creates more opportunities to prepare useful candidates, and its efficiency depends on how much of that preparation contributes to committed output.

Figure 4 shows that verification work in the early stages of inference increases with the number of tool calls: the average number of verified nodes in the first pass increases from 61.78 → 91.99 → 127.34 → 151.98 as the reference call count grows from one to four, then drops sharply in subsequent rounds. However, the additional computation at four calls does not translate into an end-to-end throughput gain: throughput falls from 443.54 to 410.72 TPS when moving from three to four calls.

Our supplementary EAGLE-3 sweep on BFCL illustrates the same trade-off:

MethodVerified nodesTPS ↑AL ↑
EAGLE-37.78199.892.835
EAGLE-331.08202.473.014
EAGLE-362.08179.153.155
EAGLE-3124.08167.733.228

Verified nodes measures the average number of candidate-token nodes processed together in a target verification pass, indicating the speculative computation workload per pass.

Compared with the highest-throughput configuration, increasing average verified nodes from 31.08 to 124.08 raises AL from 3.014 to 3.228, but reduces throughput from 202.47 to 167.73 TPS. Thus, a larger speculative workload can deliver more accepted tokens per target call while still slowing end-to-end generation.

A straightforward engineering approach to controlling this overhead is to cap the verification workload—in our setting, by limiting the number of tool calls processed concurrently to balance computation cost against speculative gains. Our current implementation does not apply this concurrency cap. Adding and tuning it is expected to improve throughput by reducing redundant speculative work. Even before this optimization, Table 1 shows that SchemaFill achieves 406.51 TPS versus 205.11 TPS for EAGLE-3—a 1.98× throughput advantage.


We thank you for the constructive feedback and for recognizing the soundness of our verification framework. Our additional experiments and analyses clarify how the individual mechanisms contribute to acceleration, quantify forecast accuracy and unused predictions, and explain the throughput drop at higher call counts through the trade-off between speculative computation and accepted output, with a concrete engineering path to reduce redundant work. We will incorporate these results and clarifications into the revised manuscript.

In light of this additional evidence, we would appreciate your reconsideration of the score. We would also welcome the opportunity to address any remaining concerns.

Dear Reviewer TkNS,

Thank you again for your time and thoughtful feedback. We would like to follow up on our response and ask whether you have any remaining questions or concerns. We welcome further discussion and would be glad to provide any additional clarification.

If our response and additional results have addressed your concerns, we would be grateful if you could consider a more favorable score in light of this evidence.

Thank you for your consideration.

Comments on the paper

4 comments
~Mujie_Xu1OpenReview ↗

Concern Regarding Closely Related Prior Work and Incremental Novelty [1/2]

I would like to raise a serious concern regarding the substantial methodological overlap between this submission and the earlier publicly available work OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling (arXiv:2608.00814, August 1, 2026).

OoO-Spec had already introduced parallel speculation of future tool-argument slots and integrated these predictions into target-verified speculative decoding. However, this submission presents closely related mechanisms as central methodological contributions without adequately acknowledging or distinguishing the earlier work.

This concern goes beyond a missing reference: it directly affects the assessment of the submission's incremental novelty.

1. Substantially Overlapping Core Mechanisms

(a) Predicting unseen argument values beyond ToolSpec.

OoO-Spec (§3–4) identifies the limitation of ToolSpec in proposing request-specific argument values absent from schemas or historical calls, and introduces learned semantic prediction to address it.

This submission (§1–2, §4.1) addresses the same limitation through speculative generation of future argument values.

(b) Parallel speculation of future argument slots.

OoO-Spec (§4.3) constructs schema-defined function and argument slots, predicting their values in parallel before their serialized positions are reached.

This submission (§4.1–4.2) similarly forecasts future argument slots and drafts their values concurrently.

Both approaches exploit the same fundamental observation: argument values that must eventually be serialized sequentially can nevertheless be prepared speculatively and in parallel, removing unnecessary autoregressive dependencies.

This is the most significant methodological overlap.

(c) Target-verified use of speculative semantic values.

OoO-Spec (§4.4–4.5) incorporates prepared argument values into speculative candidates while retaining target-authoritative verification and commitment.

This submission (§3.2, §4.2) similarly treats drafted values as provisional candidates that must be verified under the actual target prefix.

Target verification itself is established in earlier speculative decoding research. The relevant overlap is its combination with parallel, out-of-order semantic-slot prediction.

2. Related Objectives with Different Implementations

Several mechanisms also pursue closely related objectives through different implementations:

  • Speculation before tool selection: OoO-Spec prepares argument slots across candidate functions without waiting for the selected function. This submission forecasts possible tools and fields across multiple call positions.
  • Structured candidate construction: OoO-Spec combines predicted values with schema-defined structural text through a reusable hint bank. This submission introduces candidate packing across fields and calls.
  • Decoupled speculation: OoO-Spec uses an independent asynchronous semantic sidecar, while this submission employs MAIN/SIDE speculative branches with shared verification.

These mechanisms are not implementation-identical, but their underlying objectives and dependency-elimination principles substantially overlap.

~Mujie_Xu1OpenReview ↗

Concern Regarding Closely Related Prior Work and Incremental Novelty [2/2]

3. Genuinely Distinct Contributions

There are also meaningful technical differences:

  • Drafter architecture: OoO-Spec employs a target-agnostic Qwen3-0.6B semantic sidecar, whereas this submission uses EAGLE-3-based drafting.
  • Cross-call speculation: This submission extends forecasting and candidate packing across multiple tool calls.
  • Shared verification and KV-cache management: This submission introduces branch-specific verification, attention isolation, and KV management mechanisms distinct from OoO-Spec's architecture.

These differences deserve independent evaluation.

The central novelty question is therefore not whether the two implementations are identical, but which contributions of this submission remain genuinely new beyond the slot-parallel semantic speculation mechanism already established in OoO-Spec.

4. Missing Acknowledgment and Requested Clarification

In the publicly available version of this work, ToolSpec is identified as the closest prior work, while OoO-Spec is not discussed.

This omission is particularly concerning because OoO-Spec was publicly available before the ICLR submission deadline and directly addresses the methodological limitation identified in this submission.

I would appreciate clarification from the authors regarding:

  1. How the proposed slot-parallel speculation mechanism differs fundamentally from OoO-Spec, beyond the implementation differences discussed above.
  2. Which components constitute the independent algorithmic contributions of this submission, as opposed to extensions or runtime optimizations of previously disclosed mechanisms.
  3. Whether the authors will explicitly acknowledge OoO-Spec and clarify the relationship between the two methods in their manuscript.
  4. Whether an additional comparison with OoO-Spec is feasible, given their closely related objectives.

I recognize that recent concurrent work may not require direct experimental comparison under ICLR's policies. Nevertheless, the absence of a mandatory comparison does not eliminate the importance of accurately positioning related methodological contributions.

I respectfully ask the Area Chair and reviewers to consider OoO-Spec when assessing this submission's originality and technical significance, and to distinguish the previously established mechanisms from its independent advances.

The purpose of this comment is to support an accurate scientific record and a fair, evidence-based evaluation.

Relevant prior work: OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling, arXiv:2608.00814 (August 1, 2026).
https://arxiv.org/abs/2608.00814

~Mujie_Xu1OpenReview ↗

What do realignment and shared verification add to parallel slot drafting?

The revised BFCL ablation gives a clear result for cross-call packing. Enabling it raises throughput from 349.88 to 406.51 tokens/s (C→D, +16.2%). The broader 42.2% gain over structural prediction alone includes several changes, among them parallel drafting of future argument values for target verification, a mechanism also used in OoO-Spec.

The contribution of dynamic slot realignment is harder to infer. Stage B introduces realignment while SIDE drafting is still disabled. By the time SIDE drafting is enabled in C, realignment is already active. Could the authors rerun C on the same BFCL requests with the same initial forecasts and draft budget, fixing each draft to its original slot and discarding it if that assignment becomes invalid? The throughput difference would quantify the value of realignment. Reporting how many reassigned drafts MAIN subsequently accepts would also show when it helps.

For the shared verifier, could the authors report the time spent verifying the same MAIN and SIDE proposals jointly versus in separate target passes? That would give readers a direct estimate of the runtime saving attributable to shared verification.

AuthorsOpenReview ↗

Technical comparison of SchemaFill and OoO-Spec

1. OoO and SchemaFill differ substantially in their underlying design.

ToolSpec (Xia et al., 2026) constructs speculative candidates through schema-driven structural filling and retrieval of historical calls. OoO extends this framework with an independent small model that asynchronously predicts the function and argument-slot values for the current request in parallel. These predictions are assembled into a fixed hint bank and incorporated into ToolSpec’s subsequent candidate-retrieval rounds. Candidate-tree construction, verification, KV-cache updates, and recovery remain with ToolSpec. Thus, OoO’s additional design primarily concerns semantic candidate generation and asynchronous integration, extending ToolSpec’s existing inference framework.

SchemaFill is a speculative decoding framework we propose for structured-data generation. It systematically organizes speculative generation through the ongoing progression of MAIN/SIDE branches, cross-call candidate packing, dynamic realignment and rollback, shared verification, and multilevel cache management. The two methods differ substantially in how speculative generation is organized, states are reused, and verification is performed.

2. SchemaFill’s independent contributions span generation organization, execution flow, and cache management.

For generation organization, we decompose structured generation into four levels: the main stream, cross-call, tool-call, and slot levels. This enables speculation and candidate reuse across individual argument slots and tool calls.

For execution flow, we use BM25 to produce initial forecasts and introduce dynamic realignment and rollback to address its limitations in predicting tool-call order. When forecasts diverge from actual generation, these mechanisms coordinate candidate and state reuse across levels, allowing MAIN/SIDE branches to continue progressing.

For cache management, we design three levels of caches for structured generation: main-stream, slot-level, and tool-level caches, together with corresponding index-rollback and prefix-management mechanisms. Tree attention organizes computation and shared verification across branches and call candidates. Our method overview illustrates the coordination between these generation processes and cache states.

These mechanisms for generation organization, execution flow, and cache management are core methods we designed for structured generation and constitute SchemaFill’s independent technical contributions.

3. Why the initial manuscript did not specifically discuss OoO, and how we will address this concurrent work.

At our submission deadline, OoO had been publicly available on arXiv for only 46 days, and we had been unable to obtain its official implementation. Given its recent release, the unavailability of an accessible implementation, and the substantial differences in underlying inference organization and state management, we did not specifically discuss this concurrent work in the initial manuscript.

We are willing to add a citation in the revised manuscript and accurately explain the relationship between the two methods. However, we disagree with Mujie Xu’s characterization of SchemaFill’s core contributions and originality. SchemaFill’s cross-call organization, dynamic state management, shared verification, and cache-management mechanisms are core technical contributions that we developed independently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.