SpecSplice: Dynamic Composition of Parametric and Retrieval-based Drafts for Agentic Speculative Decoding

Agentic workloads interleave novel text with repeated spans, making parametric and retrieval drafters complementary for speculative decoding. Within a single round, parametric draft tokens can expose retrieval matches unavailable from the committed context alone. Existing hybrids limit these opportunities through round-level source selection or prescribed attachment rules. We introduce SpecSplice, which jointly selects candidates from both sources under a shared node budget, treating the root and every node of a supplied parametric draft as potential splice points. A unified contribution score estimates each candidate's survival probability, accounting for the parametric prefix leading to a retrieved node. Score bounds enable a best-first algorithm to interleave retrieval with node selection, skipping queries that cannot affect the selected tree. We prove that the resulting tree maximizes the total contribution score over this candidate space under the node budget, exactly as if retrieval were queried at every splice point. Across four agentic workloads and three target models, SpecSplice achieves up to a 6.91× speedup over autoregressive decoding and 13–37% higher throughput than the strongest baseline on every agentic model–workload pair.