- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper characterizes how proposal mechanisms and depths trade speed against draft-token waste in speculative decoding across workloads, entropy, repetition, and proposal costs. The authors observe that the best mechanism (EAGLE-3, n-gram, or draft-model SD) and the best depth vary by workload. Motivated by this, they propose APEX, a learned hierarchical controller that adaptively selects both the mechanism and the depth. APEX-Router selects a mechanism and initial depth once per request, and APEX-Depth adapts at each verifier block from causal features such as prefix statistics and previous verifier outcomes. Evaluated on six workloads against fixed-depth baselines, APEX is reported to reach up to speedup over autoregressive decoding and to reduce waste to as low as 40%.
Critical strengths
-
The paper’s motivation is strong. A fixed speculation configuration is poorly suited to real-world serving, where workloads and decoding states vary constantly. Through an empirical study in Section 3, the authors decompose this limitation into several concrete observations, which in turn justify each component of the proposed design. This direction has clear practical value for existing speculative decoding serving stacks.
-
The authors conduct reasonable ablation study and candid reporting. Table 10 shows that removing the survival loss or the verifier-history features substantially hurts the controller, which supports the claim that recent verifier feedback is the key signal for block-level adaptation. The authors also openly report failure cases (code generation, n-gram performs better than APEX-B), which increases the credibility of the remaining results.
Critical weaknesses
-
The paper is hard to follow in several places. For example, Equation 5 is for router, but it appears at the start of section 4.3 for APEX-depth. The paper also introduces a large number of mathematical notations. It would be better if the authors can add a notation table summarizing all notations.
-
The paper mentions various adaptive speculation methods in related work, but none of them is evaluated in the main experiment. The only baselines are fixed-depth configurations, which can not thoroughly show the superity of APEX.
-
The method APEX has too many hyperparameters, which hinders its practical deployment. Additionally, the utility weights for three APEX variants appear to be set by hand, and it is not mentioned how they are chosen.
-
Many key details of the method are missed. The mathematical definition of the scoring functions in Equation 3 and 6, the definition of , and the loss function for training are not specified. Besides, the network architecture and the training setup are not provided.
Questions for authors
-
Can you compare with various adaptive speculation methods mentioned in related work?
-
What are the definition of the scoring functions in Equation 3 and 6, , and the loss function for training? Could you provide the network architecture and detailed training setup?
-
How do you handle the mismatch between fixed-k training traces and dynamic-k deployment?
4 comments
Thank you for the detailed response, which addresses some of my concerns. However, the training loss has five hyper-parameters, and the utility target in Eq. (7) has four. Such a large number of hyper-parameters makes the method hard to deploy in practical scenarios. Moreover, the paper does not provide a concrete ablation study on these hyper-parameters. Therefore, I would like to maintain my original score.
Response to Reviewer (summary of edits done and answers to the questions)
Thank you for clarifying the concern. We have uploaded a new revised manuscript in order to answer the concerns adn have added a table with new results and new text explantions have been added. Here is our explanation to the reviewer's conerns. The four utility coefficients in Eq.7 define the training targets from recorded decoding outcomes. The five loss coefficients determine how strongly each training objective contributes when fitting the depth controller. To deploy APEX-B, we load a trained checkpoint and a fixed serving configuration. The router selects a drafting method and initial depth for each request, and the depth controller adjusts the draft depth using verifier feedback. We used the same APEX-B checkpoint and serving configuration across all six workloads in Table 2, with zero deployment-time tuning of these nine coefficients.
Appendix A.7 now quantifies the cost of changing the operating preference. We reuse the existing decoding traces and fitted preprocessing and refit only the depth controller. A 12-epoch refit on 6.26 million block examples took 22.2–24.0 minutes on one A100 GPU. In the recorded benchmark, approximately 5 minutes and 15 seconds elapsed from the first logged engine initialization to the first dispatched prompt, including loading the cached models and preparing the drafting engines.
We also added a direct utility-weight sensitivity experiment in Table 17 (please ignore the table numbering concern here, we wnat to keep it consistent with our response to other reviwers, we will correct it in the final submission). We retrained APEX-B after increasing or decreasing each of the four utility coefficients by 25%, one at a time, and repeated the experiment with two seeds. In both repeats, raising the waste penalty reduced recorded draft waste and throughput, with the default between the two tested values on both metrics. For the training losses, Table 14 reports serving ablations on 120 requests. Full APEX-B achieved 2.40× speedup with 54.62% waste. Removing the survival loss gave 1.53× with 69.38% waste, and removing cost and throughput supervision together gave 2.33× with 57.83% waste. These results show why those training signals are useful and make the cost of changing the controller’s operating preference explicit.
Edits: line 961-967; line 1238-1241; line 125-173
Follow-up to the reviewer
Dear Reviewer,
Thank you again for clarifying your concerns about hyperparameter sensitivity and practical deployment. In our latest response and revised manuscript, we added utility-weight sensitivity experiments (Table 17), training-loss ablations (Table 14), and measurements of controller refitting cost (Appendix A.7). We also clarified that the same checkpoint and serving configuration were used across all six workloads without deployment-time tuning of these coefficients.
As the discussion period draws to a close, we would appreciate your feedback on whether these additional results and clarifications address your concerns. Please let us know if any questions remain.
Thank you for your time and consideration.
Response to Reviewer (summary of edits done and answers to the questions)
Thank you for your detailed feedback. The revised manuscript reorganizes the methods section, adds a notation table and comparisons with adaptive-depth baselines, and explains our choice of utility weights. The revision also specifies the controller architecture and training objectives and clarifies how we train on fixed-depth traces and adapt depth during inference. We address each comment below in the order metioned in the comments.
1- Organization and notation- We agree that the arrangement of equations in previous version could create confusions, In the revised version, we moved Eq. 5( router training objective) into section 4.2, immediately after the router's action-selection rule. The revised equation explicitly defines the router’s training target.
We also added a notation table in Appendix A.3 (Table 5) and refer to it at the start of the methods section. Edits in the manuscript: line 262-269; line 756-796
2- Comparison with adaptive-depth methods - We added Section-5.5 and Table 4 in the revised manuscript. It compare APEX-Depth with BanditSpec, an AdaEAGLE-style predictor, and DynamicProposer (this was realeased by vLLM directly in its system) on 60 test prompts with both n-gram and EAGLE-3 drafting. We also evaluate the DSDE-style method with EAGLE-3, which provides the draft-token probabilities needed by its KL-divergence rule. With n-gram drafting, APEX-Depth achieved a 1.57× speedup over autoregressive decoding, the lowest draft-token waste (62.0%), and the highest reported WAS (2.53). With EAGLE-3, APEX-Depth achieved the highest WAS (5.81) at 2.40× speedup and 41.3%. These comparisons show where APEX-Depth improves the speed–waste trade-off against controllers that already adapt depth. To have an apple-to-apple comparison we set the speculative method and run only APEX-Depth as none of the other adaptive methods have the feature to adapt between different speculative methods. Edits in the manuscript: line 487-494 and line 504 -524
3- Utility weights and deployment - We chose the utility weights by hand to explore different preferences for throughput, accepted draft progress, and draft-token waste (we essentially want to explore if having different wieghts cause the kind of effect we expect). Section 5.1 gives the weights and the rationale for each variant. APEX-S emphasizes speed, APEX-E penalizes waste more heavily during training, and APEX-B is the default checkpoint for balancing these objectives. The variants share the same architecture, training data, and serving configuration. When deploying a trained checkpoint, its weights are fixed and the training utility weights require no per-request adjustment.
4- Scoring, losses, architecture, and training setup- We revised Sections 4.2 and 4.3 to explain how both controllers make decisions and make the equation clearer. The router uses an ExtraTrees regressor to predict output tokens per second for each candidate drafting method and initial depth. We train it on throughput measured in completed generation runs. At inference, it selects the pair with the highest predicted throughput. The depth controller evaluates each candidate depth using the current decoding state and the selected drafting method. It ranks candidates using predicted throughput, expected draft acceptance and waste, and a learned utility that reflects the trade-off between speed and draft usage.
We also spell out what each training loss teaches the depth controller. The survival loss learns from where the first rejection occurs or from the fact that the entire draft was accepted. Three regression losses train the model to predict block cost, throughput, and utility from the recorded outcomes. The auxiliary ranking loss favors the candidate with the highest observed utility in each training group, using each candidate’s own recorded state. These losses are detailed in Appendices A.4 and A.7.
Table 6 now gives the architecture of both controllers, including the router's 500 trees and the depth controller's three shared layers and four prediction heads. It also reports the optimizer settings, batch size, and 12-epoch training schedule. Appendix A.7 explains how we split the traces by prompt for training and checkpoint selection. Edits in the manuscript: lines 262-269; lines 918-960
5- Fixed-depth training and adaptive deployment. As clarified in section 5.2, we train on individual blocks from separate fixed-depth runs at k= {1,2,4,8,16}. The requested depth is an input, and each block provides its own observed outcome. During generation, the frozen controller evaluates candidate depths using the current decoding state and updates that state with feedback from completed verifier blocks. Changing depth within a request can produce histories absent from the fixed-depth training data. We now state this limitation explicitly. The end-to-end results were measured with depth adaptation enabled, so they reflect the states the controller encounters under its own decisions.