acceptodds
Under review as a conference paper at ICLR 2027

T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

Abstract

Speculative decoding significantly accelerates autoregressive text generation in Large Language Models (LLMs), but its efficacy depends largely on two key factors: the distributional alignment between the *draft* and the *target* model, and the *draft* model's latency during inference. However, existing approaches typically utilize independently pretrained small draft models or seek to prune the target model to a manually chosen architecture, thereby failing to explicitly balance the trade-off between draft model alignment and token generation efficiency. To address this gap, we formulate draft model design as an *optimization* problem and propose T2D, a framework that constructs draft models directly from the target model using *latency-aware structured pruning* and *knowledge distillation*. In this manner, T2D obtains draft models that are jointly optimized for both measured inference latency and cross-entropy loss on the target model's responses, resulting in superlative speculative decoding performance. Through extensive experiments on multi-domain speculative decoding benchmarks and across several target model families of differing scale, we demonstrate how T2D prunes up to 90% of the target model to obtain highly performant draft models compared to competitive baselines. Moreover, draft models obtained via T2D can also be utilized by other speculative decoding token-alignment and knowledge distillation methods in a plug-and-play fashion, leading to further gains. Our findings underscore the importance of jointly navigating the draft model architecture-alignment frontier to unlock the full potential of speculative decoding for LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Structured Pruning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9gwqOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes T2D, which constructs draft models for speculative decoding through learnable structured pruning of the target model. It jointly optimizes cross-entropy on target-generated data and predicted latency, with a reference-based penalty to limit alignment degradation. On Llama-2-7B-Chat and Vicuna-13B-v1.5, the resulting drafts achieve overall speedups of 1.74× and 1.89×, outperforming the evaluated pruning baselines.

Critical strengths

Jointly choosing the draft architecture and aligning it with the target is a sensible and practically relevant direction. The improvements in Table 1 hold across all 11 domains, rather than being driven by a few tasks. The method also produces compact dense models that can be used directly for inference.

Critical weaknesses
  1. The benefit of automatic architecture selection is not yet fully established. In Appendix D, the main comparison uses the five-layer Wide architecture adopted from prior work, whereas T2D produces three- and four-layer drafts. The ShortGPT sweep in Section 5.2 shows that the pruning ratio substantially affects speedup. However, a baseline architecture selected on validation data is not directly compared with T2D under consistent training settings. The results support improvement over the selected baselines, but the additional benefit and construction cost relative to a simple architecture search remain unclear.
  2. The central assumption behind the surrogate objective is not directly validated. T2D optimizes CE and predicted single-step latency, whereas the actual objective depends on consecutive acceptance length and end-to-end speedup. Figure 3 illustrates the trade-off for ShortGPT, and Figure 4 shows training dynamics. Neither establishes whether T2D's surrogate reliably ranks candidate architectures. Appendix H also provides no latency prediction error.
Questions for authors
  1. How much advantage remains after tuning the depth and width of Wide/ShortGPT on validation data under matched data and training budgets? How does a baseline that fixes T2D's discovered architecture before pruning and distillation perform?
  2. Could the authors report CE, predicted and measured latency, acceptance length, and actual speedup for the same candidate architectures? Ablations removing the latency term or reference penalty would also help establish which components improve architecture selection.
3 comments

Dear Reviewer 9gwq,

Thank you for sharing your observations on our paper. We ran additional experiments on matched architectures, latency/speedup consistency, and the CE-reference ablation, to address your queries. We describe our efforts in detail below.



W-1: The benefit of automatic ... architecture search remain unclear.

Q-1: How much advantage remains ... pruning and distillation perform?


WR-1 & QR-1: Thank you for the excellent observation. In the main results, the baselines use the 5-layer architecture chosen by the Wide authors' own architecture search, while T2D discovers its architecture in a single run. We ran the baselines with T2D's 3-layer architecture under the same data and training budget:

TargetDraftCodeHum.MathMultilin.QARAGReasonRole.STEMSumm.WritingOverall
Llama-2-7B-ChatShortGPT (3L)1.881.521.821.921.331.711.811.271.641.531.521.64
Wide (3L)1.961.521.841.881.371.671.861.341.671.451.561.66
T2D1.981.591.922.031.461.791.901.351.721.601.621.74
  • Even with T2D's architecture fixed before pruning and distillation, both baselines remain below T2D (1.64 / 1.66 vs. 1.74). Independent pruning and distillation underperforms joint optimization at the same architecture, so the gain does not come only from the architecture search.
  • Depth sweep: we swept ShortGPT over 11, 9, 7, 5, 3, and 1 layers (Figure 3); all are below T2D. This sweep was selected on the SPEED-Bench evaluation itself (an oracle choice that favors the baselines), and T2D still wins.
  • The Wide authors found their 5-layer architecture from a Sheared-LLaMA-style search over 1/3/5 layers, and T2D outperforms it at 5 layers as well (Table 1).
  • Cost: Searching k architectures costs k times as much (e.g., 50 candidates cost 50x T2D) and gives no joint-optimization benefit.


W-2: The central assumption behind the surrogate objective ... no latency prediction error.

Q-2: Could the authors report ... establish which components improve architecture selection.


WR-2 & QR-2: Thank you for your observation. We provide the following clarifications:

  • Why CE: The draft is trained on target-generated responses, so the CE is the draft's negative log-likelihood on the target's own outputs. Lower CE means better draft-target agreement and therefore higher acceptance. To assess whether cross-entropy (CE) reliably correlates with Acceptance Length (AL), we calculated the Pearson correlation between AL and CE loss across the candidate Short, Wide, and T2D models. We obtained a correlation coefficient of −0.93, indicating a strong negative correlation: higher AL is strongly associated with lower CE loss on target model generated responses.
  • Why the surrogate is reliable: Speedup equation shows that speedup depends only on acceptance length (AL) and the latency ratio. Latency is covered by the predictor (4.09% error), and AL is covered by CE, since lower CE means better agreement with the target's tokens. Figure 3 shows this trade-off directly for the ShortGPT sweep: as pruning increases, latency and AL fall together, and speedup peaks where the latency gain outweighs the acceptance.
  • Latency prediction error: the predictor has a low average error of ~4.09%. We report this error in Appendix J.
  • End-to-end consistency (Llama-2-7B-Chat, T2D draft, averages over the benchmark):
QuantityValue
Measured draft latency1.35 ms
Predicted draft latency1.39 ms
Target model latency11.45 ms
Draft length (gamma)4
Observed acceptance length2.57
Speedup (predicted latency)1.73
Speedup (measured latency)1.75
Observed speedup1.74
  • Plugging the predicted latency into Speedup equation reproduces the observed speedup (1.73 vs. 1.74), so the predicted latency is reliable for estimating speedup.
  • CE reference ablation: without the CE reference, T2D prunes 31 of 32 layers with no control and speedup drops from 1.74 to 1.38:
Reference CEOverall Speedup
0.51.74
none1.38
  • Latency term: unfortunately, it cannot be removed, since it is the only force driving pruning. With CE alone, the trivial/degenerate solution is to keep the full target model.


Thank you for the additional experiments. This shows that giving the 3-layer architecture by itself does not recover the full gain. It largely addresses my first concern. For the second concern, I have one remaining question: how many draft architectures are used to compute the correlation (−0.93 ) between CE and acceptance length? Overall, the new results have improved my assessment of the paper.

Dear Reviewer 9gwq,

Thank you for your feedback and for increasing your assessment score. We are happy to hear that your first concern has been addressed.

Regarding your second concern:

  • We have used 8 draft architectures to compute the correlation (−0.93) between CE and acceptance length.
  • Overall, the correlation reinforces that decreasing CE on target-model-generated responses can increase the acceptance length.

Thank you once again for your thoughtful feedback. We truly appreciate the time and care you have put into reviewing our work. We hope your remaining concern is addressed, and we are happy to discuss further or provide any additional information if needed.

Kind regards,

Authors

Reviewer_rC2wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

T2D builds a draft model by pruning the target model’s layers, attention heads, and FFN channels during distillation. Learnable gates and a differentiable latency predictor balance draft speed against prediction quality. The method learns the draft’s size, though this choice still depends on a manually set cross-entropy threshold and latency weight.

Critical strengths

Optimizes for measured hardware latency rather than parameter count alone. Learns which components to retain while distilling from the target. Treats draft size as an optimization problem: smaller drafts run faster but can reduce token acceptance.

Critical weaknesses

A main concern is that the components in the method section all come from existing work, and part of this is not cited. For example, augmented-Lagrangian penalty is introduced in Sheared-LLaMA, and using MLP to predict latency and return gradient is used in hardware-aware architecture search works such as HAT 2020,proxylessNAS, and FBNet. Also, the target models are old (Llama-2-7B and vicuna-13B). The paper also does not report current speculative decoding methods such as EAGLE-2/3, Medusa, HASS, etc. The main comparisons use drafts of different sizes, leaving the benefit of component selection unclear. For instance, on Llama-2-7B, T2D retains three layers, while Wide and ShortGPT retain five. The paper does not establish whether T2D beats a simple size sweep under the matched training budgets. The method highly rely on hyperparameters. Sensitivity to the cross-entropy threshold is not measured, and the latency weights are not fully specified. The reported Vicuna configuration appears to require a hidden-size reduction that the described pruning mechanism does not explain.

Questions for authors

How does T2D compare with three-layer Wide and ShortGPT drafts trained with the same budget?

How sensitive are draft size and speedup to the cross-entropy threshold and latency weight? 1 Does jointly pruning heads and FFN channels improve over pruning layers alone? It seems most of the benefits come from pruning layers. 2 How is Vicuna’s reported hidden-size reduction implemented? 3 What is the total search and training cost compared with a simple sweep over draft sizes?

5 comments
AuthorsOpenReview ↗

Reviewer rC2w - 1/2

Dear Reviewer rC2w,

Thank you for your time and effort in reviewing our paper. Based on your observations, we ran several new experiments to address queries, which we describe in detail below.



W-1: A main concern ... HAT 2020, proxylessNAS, and FBNet.


WR-1: Thank you for pointing out these references. We have cited Sheared-LLaMA in Sections 1 and 2, and we have cited it again at the augmented-Lagrangian penalty in our revised manuscript. We have also added HAT (2020), ProxylessNAS, and FBNet in Section 4. We also clarify how T2D differs from these works, and will add a "relation to prior work" paragraph in Section 4:

  • Sheared-LLaMA uses the Lagrangian to force the pruned model to match a prespecified target architecture. In T2D there is no target architecture. The constraint is placed on the alignment (a CE reference), the objective is latency, and the architecture is the output of optimization. This is what allows T2D to find the draft shape instead of assuming it.
  • HAT, ProxylessNAS, and FBNet use latency predictors or lookup tables to search a designed supernet/space for models trained from scratch. T2D uses a hardware-measured predictor to prune a pretrained target through continuous gates, with inherited weights and joint distillation.
  • Our contribution is the formulation. To our knowledge, T2D is the first to treat draft-model construction as a joint latency-alignment optimization where the draft architecture emerges from training. Our experiments show that this formulation, rather than any single component, gives the gain over fixed-architecture baselines (please see WR-3 as well).


W-2: Also, the target models are old (Llama-2-7B and vicuna-13B).


WR-2: Llama-2-7B-Chat is widely used in existing pruning and speculative decoding work (Lines 343-345), enabling direct comparison. To address your concern, we evaluated three newer and larger targets: Qwen2.5-7B-Instruct, Qwen3-14B (reasoning), and Nemotron-OpenReasoning-32B (reasoning).

Target ModelShortGPTWideT2D
Llama-2-7B-Chat1.451.591.74
Vicuna-13B-v1.51.741.781.89
Qwen2.5-7B-Instruct1.241.281.42
Qwen3-14B1.071.101.20
Nemotron-OpenReasoning-32B1.431.721.79
  • T2D achieves the highest overall speedup across all five target models on the SPEED-Bench benchmark, demonstrating its generalizability across different model architectures.


W-3: The main comparisons use drafts of different sizes ... matched training budgets.

Q-1: How does T2D compare with three-layer Wide and ShortGPT drafts trained with the same budget?


WR-3 & QR-1: Thank you for this excellent suggestion. The baselines need the architecture to be specified, so in the main results we used the 5-layer architecture found by the Wide authors through their own architecture search. T2D then discovers a different architecture in a single run. Following your suggestion, we re-ran ShortGPT and Wide with T2D's 3-layer architecture under the same data and training budget (2 joint + 1 fine-tuning epoch, as in Appendix E):

TargetDraftCodeHum.MathMultilin.QARAGReasonRole.STEMSumm.WritingOverall
Llama-2-7B-ChatShortGPT (3L)1.881.521.821.921.331.711.811.271.641.531.521.64
Wide (3L)1.961.521.841.881.371.671.861.341.671.451.561.66
T2D1.981.591.922.031.461.791.901.351.721.601.621.74
  • With the architecture matched, T2D still outperforms both baselines (1.74 vs. 1.66 / 1.64). Fixing T2D's architecture does not recover the gain, so it comes from joint optimization of pruning and distillation and not only from the architecture choice.
  • We also swept ShortGPT over 11, 9, 7, 5, 3, and 1 layers (Figure 3 of the manuscript). Here, the speedup of every setting is below T2D. Note that this sweep was selected on the SPEED-Bench evaluation itself, which favors the baselines.
  • The Wide authors found their 5-layer architecture from a Sheared-LLaMA-style search over 1/3/5 layers. T2D also outperforms Wide at 5 layers (Table 1).
  • Each Wide baseline run costs the same as one T2D run (Appendix E). Finding the best architecture by sweep therefore costs k times as much for k candidates, and a sweep over 50 candidate architectures costs 50x T2D.
  • We have added these results to Appendix G.


AuthorsOpenReview ↗

Reviewer rC2w - 2/2

W-4: The method highly rely on hyperparameters ... fully specified.


WR-4: Thank you for the suggestion. We ran T2D with different reference CE thresholds on Llama-2-7B-Chat:

Reference CEOverall Speedup
0.5 (default)1.74
1.01.36
2.01.38
none (penalty off)1.38
  • Speedup drops as the threshold increases. A loose threshold activates the penalty late, so pruning proceeds too far before alignment is retained, and results approach the unconstrained case (1.38, where 31 of 32 layers are pruned). The reference CE is what keeps pruning and distillation balanced. We have discussed the overall training progression in Section 5.3.
  • We use the same threshold (0.5) for both Llama-2 and Vicuna (Table 6), so no per-model tuning was needed. In practice, the threshold should be set close to the CE that the target model achieves on its own generated data (approximately 0.5). For the Llama-2-7B-Chat model the latency weight is 200 and for Vicuna-13B-v1.5 the latency weight is 400. We will explicitly mention the hyperparameters for the models in our revised manuscript.
  • We have added these results to Appendix F of the updated PDF.


W-5: The reported Vicuna configuration appears to require a hidden-size reduction that the described pruning mechanism does not explain.


WR-5: Thank you for this question. The hidden size is tied to the head count (Line 274, Algorithm 1 Line 18). T2D optimization prunes Vicuna-13B from 40 to 35 heads which gives 35 ×\times 128 = 4480, which is the reported hidden size.



Q-2: How sensitive are draft size ... with a simple sweep over draft sizes?


QR-2:

  • Analysis on CE threshold is provided in WR-4.
  • Latency weight is provided in WR-4.
  • Layer pruning is the dominant component for latency reduction, as also observed by existing work [1] and in our Figure 4. However, T2D discovers this automatically, whereas prior work needed manual sweeps to find it. For Llama-2-7B (hidden 4096, FFN 10752 vs. 11008), T2D keeps almost all heads/FFN, which matches this finding. For Vicuna-13B, T2D also prunes heads and FFN because they then contribute a portion of latency (Section 5.3).
  • Vicuna's hidden size reduction is clarified in WR-5.
  • A sweep over k wide architectures costs k times as much as T2D, and gives no joint-optimization benefit, as shown by the matched-architecture results in WR-3.

—

References:

  • [1] Yan et al., Decoding Speculative Decoding, 2025
AuthorsOpenReview ↗

Reviewer rC2w - 3

W-6: The paper also ... such as EAGLE-2/3, Medusa, HASS, etc.


WR-6: Thank you for pointing out these references. We cited EAGLE and Medusa in Section 2 and will also include HASS. T2D constructs the draft model used in standard draft-model-based speculative decoding. Therefore, it is complementary to methods that modify the drafting or verification algorithms rather than directly competing with them.


Reviewer_Jfi2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

T2D proposes a latency-aware structured pruning and knowledge distillation joint optimization framework for constructing speculative decoding draft models directly from a target model. The core idea is to formalize draft model construction as a joint optimization problem over inference latency and alignment quality. Discrete pruning decisions are made differentiable via Hard-Concrete gates, and an offline-trained lightweight MLP latency predictor converts hardware-measured inference latency into a differentiable objective, enabling architecture selection and parameter alignment to be co-optimized within a single training pipeline. The method maintains strong alignment quality even at 90% pruning of the target model, and the resulting draft models are plug-and-play compatible with distillation methods such as AdaSpec.

Critical strengths
  1. Highly novel method: first unified formulation of draft model architecture search and alignment optimization as a single objective The paper explicitly formulates draft model construction as a latency-alignment joint optimization problem (Equation 2), representing a fundamental departure from existing approaches that select architectures before distillation or prune before fine-tuning. By continuouslizing discrete keep/remove decisions through Hard-Concrete gates and pairing them with a differentiable latency predictor, architecture selection and parameter alignment are co-optimized in a unified pipeline. This escapes the limitations of manually predefined architectures, and experiments show the automatically discovered architecture outperforms heuristic designs such as Wide and ShortGPT, validating the joint optimization paradigm.
  2. Pragmatic latency predictor design bridging hardware measurements and gradient-based optimization The paper trains an offline 3-layer MLP to predict single-token decoding latency, converting non-differentiable hardware latency into a differentiable objective. This avoids expensive online latency measurement while ensuring gradients flow back to the gate parameters. The latency predictor is trained on the target hardware (B200) and then frozen, providing a continuous latency signal that is critical for the method to automatically converge to the latency-alignment Pareto frontier.
Critical weaknesses
  1. The "plug-and-play with AdaSpec" claim is validated only on the GSM8K single domain, lacking cross-domain evaluation.

  2. The paper evaluates only two models — Llama-2-7B-Chat and Vicuna-13B-v1.5 — with no involvement of MoE (Mixture-of-Experts), SSM (State Space Models), or other modern architectures. The applicability to 70B+, 100B+, or trillion-parameter-scale models remains entirely unknown.

  3. The paper reports only greedy decoding (T=0) speedup ratios, with no presentation of acceptance length or speedup results under sampling mode (T>0) — yet sampling is the core configuration for real-world LLM deployment. All experiments are conducted on a single-request, single-GPU setup, with no evaluation of batch size> 1.

Questions for authors

See Weakness.

3 comments

Dear Reviewer Jfi2,

Thank you for your time and effort in reviewing our paper, and for your constructive suggestions. Based on your feedback, we ran multiple new experiments (cross-domain AdaSpec, three additional target models, and sampling/batched decoding) to address the queries, as described below.



W-1: The "plug-and-play with AdaSpec" claim is validated only on the GSM8K single domain, lacking cross-domain evaluation.


WR-1: Thank you for the suggestion. We performed AdaSpec distillation on the Short, Wide, and T2D draft models (derived from Llama-2-7B-Chat) on two additional datasets: MBPP (coding) and XSum (summarization).

MethodGSM8KMBPPXSum
ShortGPT1.431.561.09
Wide1.591.841.19
T2D1.792.011.37
  • T2D achieves the highest speedup in all three domains: +12.6% / +9.2% / +15.1% over Wide and +25% / +29% / +26% over ShortGPT on GSM8K / MBPP / XSum.
  • This shows that the benefit of T2D draft models carries over to AdaSpec across math, code, and summarization, and is not specific to a single domain.
  • We have added these results to Table 2 of the revised manuscript.


W-2: The paper evaluates only two models ... models remains entirely unknown.


WR-2: Llama-2-7B-Chat is widely used in existing speculative decoding and pruning work (Lines 343-345), which allows direct comparison with prior methods. Vicuna-13B-v1.5 was added to test a larger scale. To further evaluate T2D on newer families, larger scale, and reasoning models, we ran three additional targets: Qwen2.5-7B-Instruct, Qwen3-14B (reasoning), and Nemotron-OpenReasoning-32B (reasoning). Overall speedups are provided in following table:

Target ModelShortGPTWideT2D
Llama-2-7B-Chat1.451.591.74
Vicuna-13B-v1.51.741.781.89
Qwen2.5-7B-Instruct1.241.281.42
Qwen3-14B1.071.101.20
Nemotron-OpenReasoning-32B1.431.721.79
  • T2D achieves the highest overall speedup on all five targets, spanning three model families, 7B to 32B parameters, and both instruction-tuned and reasoning models. The gain over the best baseline (+0.07x to +0.15x) remains positive across all five targets.
  • On MoE/SSM and trillion-parameter-scale models: The largest model we could train within our compute budget is 32B, which is larger than those studied by existing baselines, and the trend from 7B to 32B is satisfactory compared to the baselines. It shows with scale the performance of T2D retains. T2D's gates act on layers, attention heads, and FFN channels. Extending to MoE would require expert-level gates, and SSM/hybrid blocks would require a layer-type-aware latency predictor. We view both as natural extensions for future work.
  • We have updated Table-1 with category-wise speedups for all models in the revised manuscript PDF.


W-3: The paper reports only greedy decoding ... evaluation of batch size> 1.


WR-3: Thank you for suggesting a real-world deployment setting. We evaluated all methods with temperature = 0.7 and batch size = 4.

TargetDraftCodeHum.MathMultilin.QARAGReasonRole.STEMSumm.WritingOverall
Llama-2-7B-ChatShortGPT1.711.291.551.511.051.581.621.211.411.431.391.44
Wide1.861.461.701.821.321.651.681.211.561.431.491.57
T2D1.991.591.872.031.441.721.921.351.691.631.601.73
Vicuna-13B-v1.5ShortGPT1.661.381.581.881.201.551.561.131.411.501.351.48
Wide1.711.401.581.851.261.561.601.131.421.531.381.50
T2D1.831.521.702.071.301.651.791.191.571.691.461.63
  • T2D achieves the highest speedup in every domain under sampling and batching.

  • Acceptance length (AL): under greedy decoding, the T2D Llama-2-7B-Chat draft achieves AL = 2.57 with draft length gamma = 4. On the other hand with temperature = 0.7 and batch size = 4 AL = 2.52. On the other hand, under greedy decoding the T2D Vicuna-13B-v1.5 draft achieves AL = 2.59 and under real-world settings it achieves AL=2.50. Thus, under temperature sampling, when the generated tokens can be uncertain, T2D draft models' acceptance length does not drop significantly which leads to overall best speedup even under temperature sampling.

  • We have added these results to Appendix D of the revised manuscript.



Dear Reviewer Jfi2,

Thank you once again for your insightful review. With the deadline approaching in just one day, we would be happy to answer any remaining questions.

If our rebuttal has addressed your concerns, we would also kindly request for a reconsideration of the score.

Kind Regards,

Authors.

Dear Reviewer Jfi2,

We would like to thank you once again for taking the time to review our work and share your valuable feedback. We are happy to answer any additional questions you may have.

As the rebuttal period is ending in a few hours, if our responses have addressed your main concerns, we would be grateful if you would reconsider your assessment.

Kind Regards,

Authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Updates

Dear Chairs and Reviewers,

Thank you for your time and effort in reviewing our paper. We have conducted additional experiments to address the concerns raised and updated the manuscript accordingly. All updates are highlighted in blue font in the revised PDF.

The major additions and improvements are summarized below:

  • Newer and larger target models: We evaluated T2D on Qwen2.5-7B-Instruct, Qwen3-14B, and Nemotron-OpenReasoning-32B. T2D achieves the highest overall speedup on all five targets.
  • Extended experiment with AdaSpec: We extended AdaSpec experiments to MBPP (coding) and XSum (summarization). T2D is the best in all three domains.
  • Real-world deployment: We evaluated with temperature 0.7 and batch size 4, and T2D remains the best method in every domain.
  • Matched-architecture baselines: We re-ran ShortGPT and Wide with T2D's 3-layer architecture under the same budget. T2D still outperforms both baselines (1.74 vs. 1.64/1.66).
  • CE sensitivity: We added a reference-CE sensitivity analysis. The default threshold (0.5) performs best and is shared across models.
  • Validation of surrogate objective: We report the latency predictor error (~4.09%) and show that predicted latency reproduces the observed speedup (1.73 vs. 1.74). We also report the CE-acceptance length correlation.

All corresponding experiments, results, and analyses have been incorporated into the revised manuscript and its appendices.

We sincerely appreciate the reviewers' constructive feedback, which has helped us strengthen our work.

Kind Regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.