APEX: Speculate Smarter, Not Deeper

Speculative decoding is a leading approach for reducing large language model inference latency, but existing systems still treat speculation as a mostly static configuration: choose a proposal mechanism, choose a depth, and apply it uniformly. This static treatment is mismatched to dynamic generation, where the value of speculation changes across prompts, prefixes, entropy regimes, repetition structure, verifier outcomes, and proposal cost. When the first rejection occurs early, deeper speculation does not accelerate decoding; it simply converts compute into wasted draft tokens. We introduce APEX, a learned controller that balances acceleration and draft-token waste through request-level expert selection and block-level depth adaptation. APEX-Router selects the speculative expert for each request, choosing among existing baselines like EAGLE-3, n-gram, and draft-model speculation, while APEX-Depth adapts the number of draft tokens mid-generation at each verifier block. Its key technical idea is to model accepted draft length as censored survival feedback. Instead of predicting a coarse acceptance rate, APEX learns position-wise rejection hazards and block-level cost, then selects the action that maximizes expected accepted progress per unit cost. We instantiate APEX inside vLLM and evaluate it with Qwen3-8B across several workloads. APEX improves the speed-waste tradeoff over strong fixed speculative configurations, achieving up to 5.24X speedup over autoregressive decoding and reducing wasted speculative tokens to 40% compared against fixed-depth baselines. APEX shows that the next step in speculative decoding is not deeper drafting, but adaptive control.