- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents SchemaFill, a speculative decoding method specifically for tool calling. It uses a forecast-draft-then-verify loop, exploiting the inherent structure of tool calls to predict possible function call slots and fields, to draft field values, and to verify the drafted tokens. It presents the algorithm and the KV cache management scheme to make the methodology possible. It evaluates the method and demonstrates an increase in throughput over baselines.
Critical strengths
The paper is generally easy to follow, with the methodology clearly explained and illustrated. The evaluation is sound and demonstrates a substantive increase in throughput over baselines. The ablations are appropriate and demonstrates where the gains in performance comes from.
Critical weaknesses
The application of the methodology is narrow by design to target tool calling. It does not have broader application to code generation etc. It is also missing related work to parallel decoding techniques such as ASPD[1], APAR[2], PASTA[3], APR[4], Multiverse[5].
Algorithm 1 is a bit difficult to understand. The purpose and operation of the helper functions are not always obvious (e.g. PreparedSIDE, AlignForecast).
[1] Chen, Keyu, et al. "Aspd: Unlocking adaptive serial-parallel decoding by exploring intrinsic parallelism in llms." arXiv preprint arXiv:2508.08895 (2025).
[2] Liu, Mingdao, et al. "Apar: Llms can do auto-parallel auto-regressive decoding." arXiv preprint arXiv:2401.06761 (2024).
[3] Jin, Tian, et al. "Learning to keep a promise: Scaling language model decoding parallelism with learned asynchronous decoding." arXiv preprint arXiv:2502.11517 (2025).
[4] Pan, Jiayi, et al. "Learning adaptive parallel reasoning with language models." arXiv preprint arXiv:2504.15466 (2025).
[5] Yang, Xinyu, et al. "Multiverse: Your language models secretly decide how to parallelize and merge generation." Advances in Neural Information Processing Systems 38 (2026): 100773-100804.
Questions for authors
- How do different syntax formats of the tool calls (e.g. XML vs JSON) affect the performance?
3 comments
Thanks, my concerns and questions have been addressed.
Thank you for your thoughtful feedback and support. We’re glad that our response has addressed your concerns and questions. If the additional results have further strengthened your confidence in the paper, we would be grateful if you could consider a higher rating.
Response to Reviewer vEot:
We thank the reviewer for recognizing that “the evaluation is sound and demonstrates a substantive increase in throughput over baselines.” Below, we explain the design choices behind these improvements and present additional results supporting their applicability across structured output formats.
W1 / Q1: Applicability to structured generation
SchemaFill targets lossless acceleration of schema-guided structured generation, with tool calling as the evaluated application. Schemas expose candidate fields, types, and nesting before their values are generated, enabling speculative preparation of future content. This principle applies across schema-defined outputs, an important workload as agents increasingly interact with tools and services through machine-readable interfaces. General code generation presents a different setting, where the concrete program structure must itself be synthesized.
To substantiate this broader applicability, we extend our evaluation beyond JSON to an additional structured format, XML, directly addressing the reviewer’s question:
Each speedup uses the corresponding same-format AR baseline. XML uses a format-adapted drafter; the target remains unchanged. SchemaFill retains substantial acceleration on XML—4.28× on BFCL and 3.59× on Glaive—supporting its applicability across structured serialization formats.
W2: Distinction from parallel generation methods
The central distinction is lossless acceleration of the original target. These methods change the generation protocol and the context available to individual branches, which generate without some content that would precede them in serial decoding. Whenever that omitted context affects target predictions, distributional deviation is unavoidable without an exact correction mechanism. Comparable task quality does not imply preservation of the original target distribution.
Our motivation experiment illustrates this issue: even with an oracle call-and-field layout, the fraction of responses containing independently generated values that deviate from greedy AR rises from 3.3% to 57.5% across eight slots, and from 7.6% to 47.1% across three calls.
SchemaFill keeps parallel values provisional. Every value and structural token is verified under the actual serialized prefix, with speculative acceptance and correction preserving the target distribution. Parallel preparation therefore accelerates generation without requiring slot independence.
We thank the reviewer for highlighting that our related-work discussion did not make this distinction sufficiently clear. We will include all five references and refine the discussion and presentation of our method.
W3: Clarifying Algorithm 1
The algorithm follows four steps:
PreparedSIDEcollects eligible candidates available before the current round. MAIN and SIDE draft concurrently, and available candidates are packed with proposed structural tokens into a MAIN continuation.AlignForecastreconciles forecast slots with newly verified output, updating assignments and retiring incompatible predictions.We thank the reviewer for identifying this clarity issue. We will revise Algorithm 1 and its explanation to make the execution order and helper functions explicit.
We thank the reviewer again for the constructive feedback and welcome further discussion. We would be happy to address any additional questions or clarify any aspect of our work.