acceptodds
Under review as a conference paper at ICLR 2027

APEX: Speculate Smarter, Not Deeper

Abstract

Speculative decoding is a leading approach for reducing large language model inference latency, but existing systems still treat speculation as a mostly static configuration: choose a proposal mechanism, choose a depth, and apply it uniformly. This static treatment is mismatched to dynamic generation, where the value of speculation changes across prompts, prefixes, entropy regimes, repetition structure, verifier outcomes, and proposal cost. When the first rejection occurs early, deeper speculation does not accelerate decoding; it simply converts compute into wasted draft tokens. We introduce APEX, a learned controller that balances acceleration and draft-token waste through request-level expert selection and block-level depth adaptation. APEX-Router selects the speculative expert for each request, choosing among existing baselines like EAGLE-3, n-gram, and draft-model speculation, while APEX-Depth adapts the number of draft tokens mid-generation at each verifier block. Its key technical idea is to model accepted draft length as censored survival feedback. Instead of predicting a coarse acceptance rate, APEX learns position-wise rejection hazards and block-level cost, then selects the action that maximizes expected accepted progress per unit cost. We instantiate APEX inside vLLM and evaluate it with Qwen3-8B across several workloads. APEX improves the speed-waste tradeoff over strong fixed speculative configurations, achieving up to 5.24X speedup over autoregressive decoding and reducing wasted speculative tokens to 40% compared against fixed-depth baselines. APEX shows that the next step in speculative decoding is not deeper drafting, but adaptive control.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Dynamic speculation depth, Large Language Model inference, Workload-aware routing, Inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_abHMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper characterizes how proposal mechanisms and depths trade speed against draft-token waste in speculative decoding across workloads, entropy, repetition, and proposal costs. The authors observe that the best mechanism (EAGLE-3, n-gram, or draft-model SD) and the best depth kk vary by workload. Motivated by this, they propose APEX, a learned hierarchical controller that adaptively selects both the mechanism and the depth. APEX-Router selects a mechanism and initial depth once per request, and APEX-Depth adapts kk at each verifier block from causal features such as prefix statistics and previous verifier outcomes. Evaluated on six workloads against fixed-depth baselines, APEX is reported to reach up to 5.24×5.24\times speedup over autoregressive decoding and to reduce waste to as low as 40%.

Critical strengths
  1. The paper’s motivation is strong. A fixed speculation configuration is poorly suited to real-world serving, where workloads and decoding states vary constantly. Through an empirical study in Section 3, the authors decompose this limitation into several concrete observations, which in turn justify each component of the proposed design. This direction has clear practical value for existing speculative decoding serving stacks.

  2. The authors conduct reasonable ablation study and candid reporting. Table 10 shows that removing the survival loss or the verifier-history features substantially hurts the controller, which supports the claim that recent verifier feedback is the key signal for block-level adaptation. The authors also openly report failure cases (code generation, n-gram performs better than APEX-B), which increases the credibility of the remaining results.

Critical weaknesses
  1. The paper is hard to follow in several places. For example, Equation 5 is for router, but it appears at the start of section 4.3 for APEX-depth. The paper also introduces a large number of mathematical notations. It would be better if the authors can add a notation table summarizing all notations.

  2. The paper mentions various adaptive speculation methods in related work, but none of them is evaluated in the main experiment. The only baselines are fixed-depth configurations, which can not thoroughly show the superity of APEX.

  3. The method APEX has too many hyperparameters, which hinders its practical deployment. Additionally, the utility weights for three APEX variants appear to be set by hand, and it is not mentioned how they are chosen.

  4. Many key details of the method are missed. The mathematical definition of the scoring functions in Equation 3 and 6, the definition of ΨR\Psi_R, and the loss function for training are not specified. Besides, the network architecture and the training setup are not provided.

Questions for authors
  1. Can you compare with various adaptive speculation methods mentioned in related work?

  2. What are the definition of the scoring functions in Equation 3 and 6, ΨR\Psi_R, and the loss function for training? Could you provide the network architecture and detailed training setup?

  3. How do you handle the mismatch between fixed-k training traces and dynamic-k deployment?

4 comments
AuthorsOpenReview ↗

Response to Reviewer (summary of edits done and answers to the questions)

Thank you for your detailed feedback. The revised manuscript reorganizes the methods section, adds a notation table and comparisons with adaptive-depth baselines, and explains our choice of utility weights. The revision also specifies the controller architecture and training objectives and clarifies how we train on fixed-depth traces and adapt depth during inference. We address each comment below in the order metioned in the comments.

1- Organization and notation- We agree that the arrangement of equations in previous version could create confusions, In the revised version, we moved Eq. 5( router training objective) into section 4.2, immediately after the router's action-selection rule. The revised equation explicitly defines the router’s training target.

We also added a notation table in Appendix A.3 (Table 5) and refer to it at the start of the methods section. Edits in the manuscript: line 262-269; line 756-796

2- Comparison with adaptive-depth methods - We added Section-5.5 and Table 4 in the revised manuscript. It compare APEX-Depth with BanditSpec, an AdaEAGLE-style predictor, and DynamicProposer (this was realeased by vLLM directly in its system) on 60 test prompts with both n-gram and EAGLE-3 drafting. We also evaluate the DSDE-style method with EAGLE-3, which provides the draft-token probabilities needed by its KL-divergence rule. With n-gram drafting, APEX-Depth achieved a 1.57× speedup over autoregressive decoding, the lowest draft-token waste (62.0%), and the highest reported WAS (2.53). With EAGLE-3, APEX-Depth achieved the highest WAS (5.81) at 2.40× speedup and 41.3%. These comparisons show where APEX-Depth improves the speed–waste trade-off against controllers that already adapt depth. To have an apple-to-apple comparison we set the speculative method and run only APEX-Depth as none of the other adaptive methods have the feature to adapt between different speculative methods. Edits in the manuscript: line 487-494 and line 504 -524

3- Utility weights and deployment - We chose the utility weights by hand to explore different preferences for throughput, accepted draft progress, and draft-token waste (we essentially want to explore if having different wieghts cause the kind of effect we expect). Section 5.1 gives the weights and the rationale for each variant. APEX-S emphasizes speed, APEX-E penalizes waste more heavily during training, and APEX-B is the default checkpoint for balancing these objectives. The variants share the same architecture, training data, and serving configuration. When deploying a trained checkpoint, its weights are fixed and the training utility weights require no per-request adjustment.

4- Scoring, losses, architecture, and training setup- We revised Sections 4.2 and 4.3 to explain how both controllers make decisions and make the equation clearer. The router uses an ExtraTrees regressor to predict output tokens per second for each candidate drafting method and initial depth. We train it on throughput measured in completed generation runs. At inference, it selects the pair with the highest predicted throughput. The depth controller evaluates each candidate depth using the current decoding state and the selected drafting method. It ranks candidates using predicted throughput, expected draft acceptance and waste, and a learned utility that reflects the trade-off between speed and draft usage.

We also spell out what each training loss teaches the depth controller. The survival loss learns from where the first rejection occurs or from the fact that the entire draft was accepted. Three regression losses train the model to predict block cost, throughput, and utility from the recorded outcomes. The auxiliary ranking loss favors the candidate with the highest observed utility in each training group, using each candidate’s own recorded state. These losses are detailed in Appendices A.4 and A.7.

Table 6 now gives the architecture of both controllers, including the router's 500 trees and the depth controller's three shared layers and four prediction heads. It also reports the optimizer settings, batch size, and 12-epoch training schedule. Appendix A.7 explains how we split the traces by prompt for training and checkpoint selection. Edits in the manuscript: lines 262-269; lines 918-960

5- Fixed-depth training and adaptive deployment. As clarified in section 5.2, we train on individual blocks from separate fixed-depth runs at k= {1,2,4,8,16}. The requested depth is an input, and each block provides its own observed outcome. During generation, the frozen controller evaluates candidate depths using the current decoding state and updates that state with feedback from completed verifier blocks. Changing depth within a request can produce histories absent from the fixed-depth training data. We now state this limitation explicitly. The end-to-end results were measured with depth adaptation enabled, so they reflect the states the controller encounters under its own decisions.

Thank you for the detailed response, which addresses some of my concerns. However, the training loss has five hyper-parameters, and the utility target in Eq. (7) has four. Such a large number of hyper-parameters makes the method hard to deploy in practical scenarios. Moreover, the paper does not provide a concrete ablation study on these hyper-parameters. Therefore, I would like to maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer (summary of edits done and answers to the questions)

Thank you for clarifying the concern. We have uploaded a new revised manuscript in order to answer the concerns adn have added a table with new results and new text explantions have been added. Here is our explanation to the reviewer's conerns. The four utility coefficients in Eq.7 define the training targets from recorded decoding outcomes. The five loss coefficients determine how strongly each training objective contributes when fitting the depth controller. To deploy APEX-B, we load a trained checkpoint and a fixed serving configuration. The router selects a drafting method and initial depth for each request, and the depth controller adjusts the draft depth using verifier feedback. We used the same APEX-B checkpoint and serving configuration across all six workloads in Table 2, with zero deployment-time tuning of these nine coefficients.

Appendix A.7 now quantifies the cost of changing the operating preference. We reuse the existing decoding traces and fitted preprocessing and refit only the depth controller. A 12-epoch refit on 6.26 million block examples took 22.2–24.0 minutes on one A100 GPU. In the recorded benchmark, approximately 5 minutes and 15 seconds elapsed from the first logged engine initialization to the first dispatched prompt, including loading the cached models and preparing the drafting engines.

We also added a direct utility-weight sensitivity experiment in Table 17 (please ignore the table numbering concern here, we wnat to keep it consistent with our response to other reviwers, we will correct it in the final submission). We retrained APEX-B after increasing or decreasing each of the four utility coefficients by 25%, one at a time, and repeated the experiment with two seeds. In both repeats, raising the waste penalty reduced recorded draft waste and throughput, with the default between the two tested values on both metrics. For the training losses, Table 14 reports serving ablations on 120 requests. Full APEX-B achieved 2.40× speedup with 54.62% waste. Removing the survival loss gave 1.53× with 69.38% waste, and removing cost and throughput supervision together gave 2.33× with 57.83% waste. These results show why those training signals are useful and make the cost of changing the controller’s operating preference explicit.

Edits: line 961-967; line 1238-1241; line 125-173

AuthorsOpenReview ↗

Follow-up to the reviewer

Dear Reviewer,

Thank you again for clarifying your concerns about hyperparameter sensitivity and practical deployment. In our latest response and revised manuscript, we added utility-weight sensitivity experiments (Table 17), training-loss ablations (Table 14), and measurements of controller refitting cost (Appendix A.7). We also clarified that the same checkpoint and serving configuration were used across all six workloads without deployment-time tuning of these coefficients.

As the discussion period draws to a close, we would appreciate your feedback on whether these additional results and clarifications address your concerns. Please let us know if any questions remain.

Thank you for your time and consideration.

Reviewer_q5ZqOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors design APEX, a learned controller for speculative decoding. The proposal is that in speculative decoding, the best proposal mechanism and speculation depth depend on the request and how the generation evolves. APEX makes decisions at two timescales where APEX-Router selects a speculative mechanism and initial draft depth before generation and APEX-Depth adjusts draft depth after each verification block. The controller then predicts position-wise rejection hazards along with other factors like block cost. Three weight settings produce the speed-oriented, balanced, and efficiency-oriented profiles: APEX-S, APEX-B, and APEX-E. The paper evaluates on Qwen3-8B and reports up to a 5x speed up over AR.

Critical strengths
  • Practical and well-motivated problem: The paper distinguishes between producing plausible draft tokens and accelerating generation. The controllers aims to account for both is very reasonable.
  • Survival likelihood: Appendix A.4 was very useful and the survival likelihood as a whole was very appropriately defined.
  • Analyses and Component Ablations: Section 4 was very interesting, showing the division between request routing and block-depth control. Changing speculative mechanisms can involve model placement and batching decisions. Similarly, ablations are included in the appendix in Table 10.
Critical weaknesses
  • Figure 1: The results in Figure 1 and in tables have fairly major differences. What is going on there? For example, in Figure 1, the Code EAGLE-3, k=4 result reports 4.17x sped up but Table 4 reports 2.78x speedup. Which is it? These differences change the magnitude of the gains and the apparent relative strength.
    • Figure 1’s caption says each cell reports the highest speedup and its depth. So there is no identification of a different aggregation rule.
    • This just seems to be a reporting problem but it makes me worried about the readiness of the submission.
  • Comparison over other adaptive methods: The central contribution is adaptive control but the experimental contributions are against AR and fixed-depth speculative configurations. The paper cites several adaptive approaches in section 2.2 but omits them from the evaluation. Just doing a quick search, BandItSpec, AdaEAGLE, etc. are all very very close. APEX may be distinct but these comparisons are still needed.
  • Appendix A.7: This states that candidate rows are grouped by prompt, speculative method, and block index. The highest utility row becomes the preferred depth. The loss written treats these alternatives as evaluation at a shared state. But what if rows come from independently executed fixed-depth traces? an observed utility difference can reflect differences in the decoding states rather than which depth would have been better at a common state. This is a substantive training problem but the text does not establish whether it occurred. The authors may have replayed candidate depths from synchronized states.
  • Metrics: Why were metrics like wall-clock time, energy, GPU work, etc. not measured and reported?
Questions for authors
  • See weaknesses
  • Could the authors state the router training objective more clearly?
3 comments
AuthorsOpenReview ↗

Response to Reviewer (summary of edits done and answers to the questions)

Thank you for raising these three questions. The revision clarifies how speedup is summarized in Figure 1 and the workload tables, adds comparisons with adaptive-depth methods, explains how the ranking loss handles blocks from independent runs, reports GPU time and energy, and defines the router's training objective. We have edited the manuscript and submitted the revised version with all the new experiments. The answers below are in order of teh questions posed.

1- Figure 1- The two values measure different quantities. We agree that the original caption did not make the aggregation clear and thus we have made the required changes in the revised manuscript. In Figure 1, we randomly sampled up to 50 test prompts per workload and ran each drafting method at k = {1,2,4,8,16}. Each cell in Figure 1 reports the largest per-prompt speedup observed for that method across the sampled prompts and tested depths, along with the depth that produced it. For code generation, EAGLE-3 reached a peak of 4.17× on one prompt at (k=4). The 2.78× in the original Table 4, now Table 8 (due to revisions), is the mean speedup at (k=4) across the full code-generation test set (which is essentially the deafult test split for the dataset used). We revised the caption and section 3 to better say the details about the peak calculation and clarified the test-set distinction in section 5.2. Table 8 is the reference for comparing average code-generation performance. Edits in the manuscrupt: lines 119-123; lines 150-161; lines 410 -416.

2- Comparison with adaptive-depth methods - We added Section-5.5 and Table 4 in the revised manuscript. It compare APEX-Depth with BanditSpec, an AdaEAGLE-style predictor, and DynamicProposer (this was realeased by vLLM directly in its system) on 60 test prompts with both n-gram and EAGLE-3 drafting. We also evaluate the DSDE-style method with EAGLE-3, which provides the draft-token probabilities needed by its KL-divergence rule. With n-gram drafting, APEX-Depth achieved a 1.57× speedup over autoregressive decoding, the lowest draft-token waste (62.0%), and the highest reported WAS (2.53). With EAGLE-3, APEX-Depth achieved the highest WAS (5.81) at 2.40× speedup and 41.3%. These comparisons show where APEX-Depth improves the speed–waste trade-off against controllers that already adapt depth. To have an apple-to-apple comparison we set the speculative method and run only APEX-Depth as none of the other adaptive methods have the feature to adapt between different speculative methods. Edits in the manuscript: line 487-494 and line 504 -524

3- Ranking loss and decoding states- The reviewer's concern applies to our training data. The rows come from independent fixed-depth runs. Two runs can accept different numbers of tokens in earlier blocks, so the same block index can refer to different generated prefixes. We corrected Appendix A.7 to use the state \s_{g,k}\ (latex format), recorded for each candidate depth. The auxiliary ranking loss favors the row with the highest observed utility in its group, which can reflect both the depth and the state visited in that run. The survival, cost, throughput, and utility losses use the observed outcome of each individual block. At inference, APEX evaluates candidate depths at the current decoding state. We now state the limitation explicitly and identify replaying candidate depths from the same saved state as future work in section 6. Edits in the manuscript: line 936 - 953; lines 534-539; lines 363-365

4- Wall-clock time and GPU cost. We would like to clarify that the times used in the paper are wall-clock time, the mention cna be found at line 422. The revision makes that definition explicit in section 5.2 and adds direct GPU measurements in Appendix A.10. We report that the router runs once per request and takes 250 ms on average in the 60-prompt evaluation. The depth controller runs on a separate CPU thread so its decisions can overlap with GPU decoding. We also measured GPU time and energy directly in a controlled n-gram replay experiment on one H100. At the same output volume of 184,320 tokens, APEX's depth choices reduced GPU time from 50.89 to 33.99 seconds and energy from 35.08 to 20.72 kJ relative to fixed depth 16, reductions of 33.2% and 40.9%, respectively (Table 16). Edits in the Manuscript: lines 450-455 and line 1229 - 1253

5- Router training objective- We have included a router training paragrpgh in the revised manuscript. For each training request and candidate pair, the target is the throughput observed in the completed run (Eq. 5). We compute throughput as generated output tokens divided by the wall-clock duration of the generation call. An ExtraTrees regressor with a squared-error split criterion predicts this target from the prompt features and candidate pair. At inference, Eq. 4 selects the drafting method and initial depth with the highest predicted throughput. Edits: lines 262-269

Thanks for the clarifications and additional experiments! The clarification on Figure 1 is particularly helpful and resolves my reporting concern. Also the router objective and training details are now substantially clearer. I will increase my score from a 1 to a 3.

Some concerns remain though so I will be cautious about this increase.

  • The revised ranking loss could possibly provide valid auxiliary supervision over observsed state action utilities but it doesn't directly supervise depth comparisons at the same decoding state. An ablation without this loss would help establish its contribution and unfortunately, acknowledging the limitation does not establish that the term benefits from the deployed policy.
  • Table 4 provides useful speed-waste tradeoffs. But AdaEAGLE style predictor remains faster with both drafters. The n-gram WAS differencs is small without uncertainty estimates. These depth-only estimatees also leave the routing contribution unresolved.
  • I'm not super familiar with the related work but I do think the authors need to correct the statement that competing methods cannot select among speculative mechanisms. BanditSpec seems to evaluate this in their own paper.

Thank you for recognizing the improvements to the reporting and training description adn for increasing the score. We address the adaptive comparisons and related work below.

  1. Comparison with the AdaEAGLE-style predictor

We agree that the AdaEAGLE-style predictor is faster with both drafters. APEX-Depth offers a useful balance between speed and draft-token waste. With EAGLE-3, it achieves 2.40× versus 2.46× speedup while reducing waste from 47.7% to 41.3%. Thus, approximately 2.5% more decoding time accompanies 6.4 percentage points less waste, raising WAS from 5.16 to 5.81. With n-gram drafting, APEX-Depth achieves 1.57× versus 1.71× speedup and reduces waste from 68.4% to 62.0%. Its WAS is slightly higher, at 2.53 versus 2.50. These results support APEX’s intended speed–waste balance, although the n-gram WAS advantage is marginal. We agree that uncertainty estimates are needed before drawing a stronger conclusion about that difference.

  1. BanditSpec and the routing comparison

You are correct that BanditSpec already selects among speculative mechanisms. We should have examined this aspect more carefully, and we withdraw our earlier statement. BanditSpec evaluates selection among PLD, REST, Suffix Tree, and EAGLE-2 during generation.

Following your comment, we added Appendix A.11, testing a released-selector adaptation of BanditSpec over n-gram, EAGLE-3, and Draft-SD on the same prompt set used for the APEX comparison. The adaptation preserves the released selector’s reward, score calculation, running updates, and initialization while replacing depth choices with drafting-method choices.

Across 120 requests, all three methods are exercised. After initialization, EAGLE-3 accounts for 88.03% of rounds, n-gram for 11.52%, and Draft-SD for 0.45%. The selector makes 98 further method switches across 34 requests, confirming that it continues to reconsider the drafting method during generation. On this prompt set, APEX-B reports 54.62% draft waste compared with 73.63% for the tested BanditSpec adaptation—a reduction of 19.01 pp—while APEX-B achieves 2.40× speedup over AR. This supports APEX’s goal of retaining acceleration while discarding fewer draft tokens.

The design difference is that APEX uses learned prompt features to choose the method before generation, then adapts depth using verifier feedback. It therefore retains adaptation during generation without trying every drafting method on each request. These results concern the complete systems and do not assign the entire difference to routing alone. We will continue working on adding more experiments to strengthen the Router study, but we wanted to show some substantial results before the rebuttal period ends.

edits: lines 1293-1334

Reviewer_TuPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce APEX, a controller for adaptive speculative decoding. For each request, the controller selects a drafting model and dynamically adjusts the number of draft tokens generated. The controller models the likelihood of rejection for each draft token and combines this with the runtime cost of generating the draft token to determine the optimal draft length. It implemented APEX in vLLM and evaluated it using Qwen3-8B. In the best case, APEX can achieve up to 5.24× speedup over the standard autoregressive decoding. In terms of average performance, the speed-oriented configuration of APEX achieves similar speed to the fastest fixed-depth baseline while reducing the number of discarded draft tokens from 79.90% to 55.30%. Finally, they present ablation studies for the rejection modeling and the recent verification feedback, and both show benefits for APEX, although it does not outperform fixed configurations on all workloads.

Critical strengths
  1. The paper addresses a practical problem of wasted drafting tokens during speculative decoding and try to improve it.

  2. The overall writing structure and method are clear.

  3. The experiments show a meaningful speed–waste trade-off and demonstrates the power of APEX in reaching a better trade-off point.

Critical weaknesses
  1. The paper discusses several other methods that adjust the speculation depth, yet the main comparison is made with fixed-depth baselines.

  2. It is unclear how much computation is saved by reducing the number of discarded draft tokens. The controller used in APEX also adds some overhead in terms of computation used to determine the drafting method and the draft length. It would be good to report the overhead of the controller as well as the total GPU cost used in the experiments to see if the reduction in wasted tokens is actually saving resources.

  3. The controller was trained on traces from Qwen3-8B and the experiments were conducted using the same model. It is unknown whether the controller can be used with other target models without retraining. Thus, the evidence presented in the paper for APEX being useful is somewhat limited.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer (summary of edits done and answers to the questions)

Thank you for raising these three questions. The revision adds direct comparisons with adaptive controllers, measurements of GPU time and energy, and an evaluation of the same trained controller on a second target model. We have edited the manuscript and submitted the revised version with all the new experiments. The answers below are in order of the questions posed.

1- Comparison with adaptive-depth methods - We added Section-5.5 and Table 4 in the revised manuscript. It compare APEX-Depth with BanditSpec, an AdaEAGLE-style predictor, and DynamicProposer (this was realeased by vLLM directly in its system) on 60 test prompts with both n-gram and EAGLE-3 drafting. We also evaluate the DSDE-style method with EAGLE-3, which provides the draft-token probabilities needed by its KL-divergence rule. With n-gram drafting, APEX-Depth achieved a 1.57× speedup over autoregressive decoding, the lowest draft-token waste (62.0%), and the highest reported WAS (2.53). With EAGLE-3, APEX-Depth achieved the highest WAS (5.81) at 2.40× speedup and 41.3%. These comparisons show where APEX-Depth improves the speed–waste trade-off against controllers that already adapt depth. To have an apple-to-apple comparison we set the speculative method and run only APEX-Depth as none of the other adaptive methods have the feature to adapt between different speculative methods. Edits in the manuscript: line 487-494 and line 504 -524

2- Controller overhead and GPU cost - We added Appendix A.10 and a small description in section 5.2. We report that the router runs once per request and takes 250 ms on average in the 60-prompt evaluation. The depth controller runs on a separate CPU thread so its decisions can overlap with GPU decoding. We also measured GPU time and energy directly in a controlled n-gram replay experiment on one H100. At the same output volume of 184,320 tokens, APEX's depth choices reduced GPU time from 50.89 to 33.99 seconds and energy from 35.08 to 20.72 kJ relative to fixed depth 16, reductions of 33.2% and 40.9%, respectively (Table 16).

Additionally, we compute speedup from measured wall-clock latency. For the same generation workload on one dedicated GPU, finishing sooner (higher speedup) reduces the time that GPU is occupied by the request. Accepting more draft tokens per verification pass can contribute to this saving by reducing the number of target-model passes needed to finish generation. Thus a higher speedup in most workloads, is also an indication of reduced time on GPU. Edits in the Manuscript: lines 450-455 and line 1229 - 1253

3- Transfer without retraining - We evaluated the Qwen3-8B-trained APEX-B controller on both Qwen3-8B and Llama-3.1-8B-Instruct using the same 30 test prompts, with the controller weights and fitted preprocessing frozen. Against each target's own autoregressive baseline, APEX-B achieved 1.14× speedup and 67.5% draft waste on Qwen3-8B, and 1.13× speedup and 68.2% draft waste on Llama-3.1-8B-Instruct. WAS was 1.70 and 1.66, respectively (Table 3). This experiment shows that the same controller can retain a speedup over autoregressive decoding on a second target model without retraining. Edits in the manuscript : lines 481- 485 and lines 497 -502

AuthorsOpenReview ↗

Follow-up to the reviewer

Dear Reviewer,

We would appreciate your feedback on our response and the additional experiments addressing your three main concerns: comparisons with adaptive-depth methods, controller overhead and GPU resource use, and transfer to another target model without retraining. These results are included in the revised manuscript and summarized in our response above.

As the discussion period draws to a close, could you please let us know whether these additions address your concerns, or whether any questions remain? We would be happy to provide further clarification.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Update to the Reviewer

Dear Reviewer,

We wanted to share a further update relevant to your concern about comparisons with adaptive methods. The latest manuscript now includes a mechanism-selection study in Appendix A.11, evaluating a released-selector adaptation of BanditSpec that chooses among n-gram, EAGLE-3, and Draft-SD across 120 requests. We report its method choices, switching behavior, and draft-token waste, and clarify how its online selection differs from APEX’s request-level routing and subsequent depth adaptation.

This addition complements the adaptive-depth comparisons in Table 4, GPU time and energy measurements in Table 16, and transfer-without-retraining results in Table 3 described in our earlier response.

We would appreciate your feedback on whether the revised manuscript and additional results address your concerns, and would be happy to clarify any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.