acceptodds
Under review as a conference paper at ICLR 2027

Calibrate Before You Branch: Feedback-Calibrated, Cost-Aware Trees for Block-Diffusion Speculative Decoding

Abstract

Speculative decoding (SD) accelerates large language model inference by using a lightweight drafter to propose future tokens that are verified in parallel by the target model. Block-diffusion speculative decoding further reduces drafting latency by predicting multiple future positions in a single forward pass. To exploit alternative candidates within such parallel drafts, recent methods organize them into draft trees for joint verification. However, existing tree construction primarily ranks candidate paths using factorized draft probabilities, which may not reflect the target model’s actual acceptance preferences. We propose FACT, a cost-aware draft-tree allocation framework that jointly addresses candidate prioritization and budget selection. Specifically, FACT uses historical target-verification outcomes to construct a frozen correction table that calibrates candidate priorities without additional inference-time model calls. It further combines draft-coverage estimates with profiled full-round costs to adaptively determine whether expanding the tree is worthwhile, while an incremental builder reuses the search frontier across budget checkpoints and materializes only the selected tree. Experiments on six benchmarks with Qwen3-8B and Qwen3-4B show that FACT consistently increases tokens per verification round over DDTree-B128. On Qwen3-8B, FACT achieves 4.17–6.80× speedup over autoregressive decoding and reduces generation time by 0.27–2.73% relative to DDTree-B128. On Qwen3-4B, it reduces generation time by 1.24–3.05%. Code is available at https://anonymous.4open.science/r/fact-anonymous-review-B853/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, block diffusion, draft trees, verification feedback, feedback calibration, cost-aware inference, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LwQLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FACT, a framework that uses historical target verification feedback to improve candidate prioritization in block-diffusion speculative decoding. It also introduces cost-aware budget control to dynamically select the verification tree size and improve decoding efficiency.

Critical strengths
  1. The paper identifies a reasonable limitation of existing tree-based speculative decoding methods and proposes using historical target feedback to improve candidate prioritization.

  2. FACT is training-free and introduces no additional inference-time model calls. Its cost-aware budget control also considers the trade-off between acceptance length and decoding latency.

Critical weaknesses
  1. The paper does not sufficiently justify the necessity of the proposed method. In practice, target tokens are often among the top-ranked draft candidates (e.g., Top-3), which existing tree-based methods already exploit. The authors should analyze how frequently correct paths are missed by existing methods and how much improvement is possible.

  2. The reported latency improvements over DDTree are modest (0.27%–3.05%), and the ablation studies are insufficient. The paper does not isolate the contributions of Feedback Calibration and Cost-aware Budget Control. More detailed experiments are needed to explain the performance gains and justify both components.

  3. The paper lacks a comparison with DARTree, which also improves candidate prioritization and verification budget utilization through score correction. Since DARTree reports substantial speedups and shares similar objectives, this comparison is important to evaluate FACT's competitiveness.

Questions for authors
  1. How often do target tokens fall outside the top-ranked draft candidates (e.g., Top-3, Top-5, or Top-10)? How frequently does DDTree miss correct paths, and what is the potential upper bound of improvement under the same verification budget?

  2. Does the frequency of these missed candidates depend on the capability gap between the draft and target models? How does this gap affect the effectiveness of FACT?

  3. Could the authors provide separate ablation studies for Feedback Calibration and Cost-aware Budget Control? How much does each component contribute to the reported speedup?

  4. Could the authors compare FACT with DARTree under the same experimental settings? Both methods share similar objectives, and such a comparison would help establish FACT's competitiveness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer LwQL — Part 1/4: Candidate Ranks and Path Coverage (W1/Q1)

We thank the reviewer for the valuable questions regarding candidate rankings and path coverage.

Q1. How often do target tokens fall outside the top-ranked draft candidates (e.g., Top-3, Top-5, or Top-10)? How frequently does DDTree miss correct paths, and what is the potential upper bound of improvement under the same verification budget?

W1. The paper does not sufficiently justify the necessity of the proposed method. In practice, target tokens are often among the top-ranked draft candidates (e.g., Top-3), which existing tree-based methods already exploit. The authors should analyze how frequently correct paths are missed by existing methods and how much improvement is possible.

To quantify the limitations of existing draft-tree allocation, we have conducted additional analyses of target-token rankings across different draft depths.

Table 1. Draft-token ranks

Draft DepthOutside Top-3Valid PositionsRate (%)
1105711.75
36557111.38
513757123.99
1024456643.11
1531456355.77

As shown in the table, the Top-3 omission rate increases substantially with draft depth, reaching 55.77% at depth 15. Across 8,513 reference positions, the overall omission rates for Top-3, Top-5, and Top-10 are 31.81%, 25.35%, and 17.47%, respectively. These results reveal the increasing discrepancy between draft candidate rankings and target-accepted continuations at deeper positions.

Furthermore, we have evaluated reference-path coverage under the same verification budget of 128 nodes. The results are summarized below.

Table 2. Reference-path coverage

Method (Budget 128)Total Coverage ↑Mean Coverage ↑
Raw DDTree4,3307.583
Calibrated Tree4,3497.616
Reference-chain Oracle7,80413.667

The reference-chain oracle incorporates the available target-reference prefix, limited to 15 unknown positions, and achieves longer coverage than DDTree on 464 of 571 prefixes. This demonstrates substantial untapped potential in allocating a fixed verification budget to complete reference paths.

Under the same budget, Feedback Calibration increases coverage from 4,330 to 4,349, demonstrating that historical verification feedback can improve candidate selection without increasing the tree size. Together, these findings establish the practical motivation for FACT's feedback-informed allocation mechanism.

AuthorsOpenReview ↗

Response to Reviewer LwQL — Part 2/4: Component Contributions and Decoding Gains (W2/Q3)

We thank the reviewer for suggesting additional component ablations.

Q3. Could the authors provide separate ablation studies for Feedback Calibration and Cost-aware Budget Control? How much does each component contribute to the reported speedup?

W2. The reported latency improvements over DDTree are modest (0.27%–3.05%), and the ablation studies are insufficient. The paper does not isolate the contributions of Feedback Calibration and Cost-aware Budget Control. More detailed experiments are needed to explain the performance gains and justify both components.

To quantify the independent contributions of Feedback Calibration and Cost-aware Budget Control, we have conducted a controlled study under the same C++ implementation by enabling or disabling the two components separately. We have extended the original two timing repeats to six timing blocks. The results are presented below.

Table 3. Component ablation

ConfigurationTime (s) ↓Speedup ↑Time Saved (%) ↑
Raw fixed 1282.32705.207×0.000
Calibrated fixed 1282.31365.237×0.576
Raw adaptive2.22775.439×4.266
Full FACT2.22595.443×4.344

As shown in the table, Feedback Calibration independently reduces decoding latency by 0.58% at a fixed budget, while adaptive budgeting achieves a 4.27% reduction without calibration. Combining both components yields the lowest latency, achieving a 4.34% reduction over the baseline.

These results demonstrate the complementary functions of the two components. Feedback Calibration refines which candidate paths receive verification resources, whereas Cost-aware Budget Control determines the appropriate verification budget. Their integration enables FACT to optimize candidate selection and verification cost within a unified framework.

We have further evaluated FACT against CaDDTree and BASTION under matched Qwen3-8B/DFlash checkpoints, a single RTX 4090, and six timing blocks across 182 prompts. The results are summarized below.

Table 4. Six-domain speedups

DomainCaDDTreeBASTIONFACT
HumanEval6.472×6.201×6.541×
GSM8K5.811×5.661×5.827×
MBPP5.823×5.647×5.898×
MATH-5006.628×6.286×6.677×
Alpaca3.179×3.138×3.228×
AIME256.029×5.618×6.154×
Geometric Mean5.505×5.292×5.569×

All values represent speedups over matched autoregressive decoding. FACT achieves the highest speedup across all six domains, with a geometric mean of 5.569×, compared with 5.505× for CaDDTree and 5.292× for BASTION. These results further support the effectiveness of FACT's feedback-informed allocation and cost-aware budget selection across diverse workloads.

AuthorsOpenReview ↗

Response to Reviewer LwQL — Part 3/4: Draft–Target Agreement and Calibration Gains (Q2)

We thank the reviewer for the insightful question regarding the relationship between draft–target agreement and FACT's effectiveness.

Q2. Does the frequency of these missed candidates depend on the capability gap between the draft and target models? How does this gap affect the effectiveness of FACT?

To investigate this relationship, we have conducted a controlled experiment by fixing the Qwen3-8B target, draft architecture, and verification budget of 128, while varying the draft training checkpoints at steps 0, 50, and 200.

The evaluation uses 40 development prompts, 160 test prompts, and six timing blocks, covering 8,640 requests across the evaluated configurations. Training here is used to construct different draft states for this controlled experiment; FACT itself requires no additional neural-network training. The results are summarized below.

Table 5. Draft–target agreement

Draft StepTop-1 Agreement (%)Raw CoverageOriginal Table CoverageRefitted Table CoverageOriginal Table Saved (%)Refitted Table Saved (%)
040.751,0061,0101,0090.6780.648
5041.551,0231,0251,0220.096-0.422
20046.751,0921,0881,0940.751-0.075

Top-1 agreement measures how often the AR reference token has the maximal draft score, including ties, over 2,385 common reference positions. Coverage counts contiguous reference tokens across the 160 test prompts. Original and refitted refer to the historical table and a table refitted on development prompts, respectively. Time saved is 100 × (1 − method time / raw time), calculated within each checkpoint under greedy EOS/512 stopping, with native output lengths retained.

As shown in the table, Top-1 agreement increases from 40.75% to 46.75%, accompanied by an increase in raw reference-path coverage from 1,006 to 1,092. This indicates that improved draft–target agreement is associated with better baseline path coverage in the evaluated checkpoints.

Notably, the original calibration table achieves positive latency reductions at all three checkpoints, while its benefits vary non-monotonically with agreement. These results suggest that FACT's effectiveness depends not only on draft–target agreement but also on the distribution of candidate-ranking errors and the paths already retained by the uncalibrated tree. This further highlights the importance of feedback-informed candidate allocation across different draft states.

Reviewer_nFVROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FACT, which uses historical target-verification feedback to adjust candidate priorities when constructing speculative decoding trees. It combines this correction with a cost-aware tree-budget controller. Experiments with Qwen3-4B and Qwen3-8B on six benchmarks report modest reductions in generation time over DDTree.

Critical strengths
  1. Historical verification feedback is a reasonable, lightweight signal for improving tree allocation without additional drafter calls. The candidate-level examples help explain how it can recover continuations missed by raw draft scores.

  2. The matched C++ comparisons are useful: they show that the reported gains cannot be attributed entirely to replacing the Python builder.

Critical weaknesses
  1. The additional benefit over existing methods is not yet convincing. With matched C++ builders, generation time decreases by 0.36–2.53% on 8B and 0.97–4.01% on 4B, measured on 182 prompts with two timing repeats. The paper lacks comparisons with CaDDTree and BASTION and does not isolate calibration from adaptive budgeting. DSpark also combines confidence calibration with hardware-aware verification scheduling. Although it selects draft prefixes rather than tree nodes, the paper does not analyze this connection or compare their benefits. Given the modest gains over DDTree, these missing comparisons and controls make it difficult to assess the added value of FACT.

  2. The calibration data and evaluation protocol need more detail. Appendix A.1 gives broad data sources and 8B sample counts, but the source mixture, trajectory collection policy, and separation from evaluation data are insufficiently specified. Section 5.4 selects the correction strength using the reported development workloads, without reporting a separate held-out test. Since historical feedback is central to the method, these gaps leave uncertainty about whether the gains generalize beyond the workloads used during development.

Questions for authors
  1. How much improvement remains with adaptive budgeting but no calibration? Conversely, how much does calibration help at a fixed tree budget?

  2. How does FACT compare with CaDDTree or BASTION under matched conditions? Please also clarify its relationship to DSpark’s calibration and hardware-aware scheduling.

  3. Could you specify the calibration data mixture, trajectory collection settings, and overlap checks with evaluation data for both model sizes? Were any results obtained on a test set not used to select the correction strength?

6 comments
AuthorsOpenReview ↗

Response to Reviewer nFVR — Part 1/3: Component Contributions (W1/Q1)

We thank the reviewer for suggesting a more detailed component analysis.

Q1. How much improvement remains with adaptive budgeting but no calibration? Conversely, how much does calibration help at a fixed tree budget?

W1:The additional benefit over existing methods is not yet convincing. With matched C++ builders, generation time decreases by 0.36–2.53% on 8B and 0.97–4.01% on 4B, measured on 182 prompts with two timing repeats. The paper lacks comparisons with CaDDTree and BASTION and does not isolate calibration from adaptive budgeting. DSpark also combines confidence calibration with hardware-aware verification scheduling. Although it selects draft prefixes rather than tree nodes, the paper does not analyze this connection or compare their benefits. Given the modest gains over DDTree, these missing comparisons and controls make it difficult to assess the added value of FACT.

To quantify the individual contributions of Feedback Calibration and Cost-aware Budget Control, we conduct a controlled ablation study by independently enabling or disabling each component under the same C++ implementation. We have extended the original two timing repeats to six timing blocks. The results are summarized below.

Table 1. Component ablation

ConfigurationTime (s) ↓Speedup ↑Time Saved (%) ↑
Raw fixed 1282.32705.207×0.000
Calibrated fixed 1282.31365.237×0.576
Raw adaptive2.22775.439×4.266
Full FACT2.22595.443×4.344

As shown in the table, Feedback Calibration reduces decoding latency by 0.58% at a fixed budget, while Cost-aware Budget Control independently achieves a 4.27% reduction. Integrating both components yields the lowest decoding latency, with a 4.34% reduction over the baseline and an additional 0.08% reduction relative to adaptive budgeting alone. These results highlight their complementary roles. Feedback Calibration determines which candidates to prioritize, while Budget Control determines how much verification work to perform, supporting the joint allocation design of FACT.

AuthorsOpenReview ↗

Response to Reviewer nFVR — Part 2/3: Closest Baselines and Relation to DSpark (W1/Q2)

We appreciate the reviewer for highlighting these closely related methods.

Q2. How does FACT compare with CaDDTree or BASTION under matched conditions? Please also clarify its relationship to DSpark’s calibration and hardware-aware scheduling.

W1:The additional benefit over existing methods is not yet convincing. With matched C++ builders, generation time decreases by 0.36–2.53% on 8B and 0.97–4.01% on 4B, measured on 182 prompts with two timing repeats. The paper lacks comparisons with CaDDTree and BASTION and does not isolate calibration from adaptive budgeting. DSpark also combines confidence calibration with hardware-aware verification scheduling. Although it selects draft prefixes rather than tree nodes, the paper does not analyze this connection or compare their benefits. Given the modest gains over DDTree, these missing comparisons and controls make it difficult to assess the added value of FACT.

To further evaluate FACT against existing cost-aware approaches, we conduct additional comparisons with CaDDTree and BASTION using matched Qwen3-8B/DFlash checkpoints, a single RTX 4090, BF16/SDPA, greedy EOS/512 stopping, and 182 prompts across six domains with six timing blocks. CaDDTree uses its official allocation policy with a shared verifier, while BASTION uses an adapted official generator. The results are summarized below.

We will add CaDDTree, BASTION, and DARTree to Related Work and explain how FACT differs from these methods.

Table 2. Six-domain speedups

DomainCaDDTreeBASTIONFACT
HumanEval6.472×6.201×6.541×
GSM8K5.811×5.661×5.827×
MBPP5.823×5.647×5.898×
MATH-5006.628×6.286×6.677×
Alpaca3.179×3.138×3.228×
AIME256.029×5.618×6.154×
Geometric mean5.505×5.292×5.569×

As shown in the table, FACT consistently achieves the highest speedup across all six domains, with a geometric mean of 5.569×, outperforming CaDDTree (5.505×) and BASTION (5.292×). These correspond to 1.15% and 4.96% geometric-mean reductions in decoding latency, respectively, demonstrating the effectiveness of FACT's feedback-informed tree allocation under the evaluated conditions.

Regarding DSpark, both methods incorporate confidence calibration and hardware-aware optimization, but differ in their underlying mechanisms. DSpark trains a prefix-confidence head. DSpark applies Sequential Temperature Scaling to calibrate prefix-survival probabilities while preserving their relative ranking, enabling load-aware verification-length scheduling. In contrast, FACT leverages historical target-verification feedback to recalibrate candidate priorities within the draft tree, directly guiding the allocation of verification resources across competing paths. Furthermore, FACT combines feedback-based prioritization with profiled full-round costs to adapt the verification budget, without additional neural-network training or inference-time model calls. FACT reuses the existing target and DFlash weights: its setup consists of feedback collection, table fitting, and hardware profiling. Our evaluation runs on a single RTX 4090. We have clarified these distinctions in the revised Appendix D.8.

AuthorsOpenReview ↗

Response to Reviewer nFVR — Part 3/3: Calibration Data and Independent Evaluation (W2/Q3)

We thank the reviewer for the detailed questions regarding calibration data and evaluation independence.

Q3. Could you specify the calibration data mixture, trajectory collection settings, and overlap checks with evaluation data for both model sizes? Were any results obtained on a test set not used to select the correction strength?

W2. The calibration data and evaluation protocol need more detail. Appendix A.1 gives broad data sources and 8B sample counts, but the source mixture, trajectory collection policy, and separation from evaluation data are insufficiently specified. Section 5.4 selects the correction strength using the reported development workloads, without reporting a separate held-out test. Since historical feedback is central to the method, these gaps leave uncertainty about whether the gains generalize beyond the workloads used during development.

We provide additional information on data composition, trajectory collection, and overlap checks for both model sizes.

Calibration Data and Collection. For Qwen3-8B, the calibration table aggregates 5,426,051 candidate observations from 8,923 unique conversations. The detailed data distribution is summarized below.

Table 3. Calibration sources

SourceObservationsShare
ShareGPT1,232,18122.71%
MathInstruct1,181,48621.77%
Mixed1,915,58835.30%
Code1,096,79620.21%
Total5,426,051100%

Trajectory collection uses a fixed tree budget of 128, block size 16, greedy decoding, a maximum of 128 generated tokens, and BF16/SDPA. Candidates are labeled only when their parents lie on the accepted path, while unvisited candidates remain unlabeled. The saved source streams enable byte-for-byte reproduction of the frozen calibration table.

For Qwen3-4B, we have used 32 GSM8K and 32 MBPP training prompts, yielding 1,877 verification rounds and 238,816 observations. Collection uses a fixed budget of 128, greedy decoding, and a maximum of 256 generated tokens, guided by the 8B table at strength 0.25. The fitted 4B table uses a correction strength of 0.15, with each reached parent's top-16 candidates recorded once per round, including all-positive and all-negative groups.

Data Independence and Held-out Evaluation. We have verified zero exact-key overlap between the 8B calibration identities and all six original evaluation sets. The 4B test identities are also disjoint from its 64 training prompts and eight profiling prompts.

To further assess generalization, we have excluded 9,454 historical identities and 531 available normalized-text hashes, constructing an additional split of 40 development and 160 test prompts across five domains.

Table 4. Development/test split

DomainDevelopmentTest
HumanEval832
GSM8K832
MBPP832
MATH-500832
Alpaca832
Total40160

Importantly, the correction strength is fixed at 0.25 before evaluating the additional test set, ensuring that the test results are not used for its selection. Calibration-size experiments use only development prompts for fitting, and the evaluation employs six timing blocks instead of the original two repeats. These additional experiments evaluate the transfer of a previously selected correction rule to a separate test workload, providing further evidence of FACT's effectiveness beyond the development data.

The decoding performance on the additional five-domain evaluation is summarized below.

Table 5. Held-out speedups

ConfigurationTime (s) ↓AR Speedup ↑
Raw fixed 1282.32705.207×
Calibrated fixed 1282.31365.237×
Raw adaptive2.22775.439×
Full FACT2.22595.443×

FACT achieves the highest speedup of 5.443×, compared with 5.207× for uncalibrated fixed-budget allocation. The calibrated configurations also improve decoding efficiency under both fixed and adaptive budgets, supporting the effectiveness of the previously selected correction rule on the additional evaluation workload.

Reviewer_bxLpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FACT, a cost-aware draft-tree allocation framework that jointly addresses candidate prioritization and budget selection. It stores past verification outcomes in a frozen table indexed by draft depth, rank and confidence, uses the table to reweight candidate priorities, and selects the tree budget with a profiled cost model. On Qwen3-8B and Qwen3-4B with DFlash drafters over six benchmarks on one RTX 4090. On Qwen3-8B, FACT achieves 4.17–6.80× speedup over autoregressive decoding and reduces generation time by 0.27–2.73% relative to DDTree-B128. On Qwen3-4B, it reduces generation time by 1.24–3.05%.

Critical strengths
  1. Using past verification outcomes to correct draft-tree priorities is a reasonable idea, and it adds no drafter or target calls at inference time.
  2. The paper is detailed and clear, including matched C++ and Python builders, per-repeat timing and the case analysis where FACT does not help.
Critical weaknesses
  1. Limited novelty and missing closest baselines. Cost-aware tree budgets are already proposed by CaDDTree and BASTION, and building trees from offline acceptance statistics is close to Sequoia and the Medusa tree. None of them is compared; the only tree baseline on the same drafter is DDTree with a fixed budget.

  2. The time saving over DDTree is at most about 3%, measured over two runs, while the method needs a correction table fitted on millions of logged verification outcomes and a profiled cost table for every target/drafter pair.

Questions for authors

na

4 comments
AuthorsOpenReview ↗

Response to Reviewer bxLp — Part 1/3: Contributions and Closest Baselines (W1)

We thank the reviewer for highlighting these relevant studies.

W1. Limited novelty and missing closest baselines. Cost-aware tree budgets are already proposed by CaDDTree and BASTION, and building trees from offline acceptance statistics is close to Sequoia and the Medusa tree. None of them is compared; the only tree baseline on the same drafter is DDTree with a fixed budget.

CaDDTree optimizes tree structure and budget by maximizing estimated decoding throughput, while BASTION combines path-confidence-based acceptance estimation with hardware-aware latency modeling and adaptive tree expansion. Both methods make important contributions to cost-aware allocation.

The key distinction of FACT lies in how historical target-verification feedback is incorporated into the prioritization of current draft candidates. Specifically, parallel block-diffusion predictions lack sequential conditioning within candidate paths, causing draft scores to deviate from actual target-acceptance behavior. To address this discrepancy, FACT collects verification outcomes from candidates whose parents lie on the accepted path and aggregates them by depth, rank, and confidence into 320 frozen coefficients. Unlike approaches that primarily rely on raw draft probabilities or aggregate acceptance statistics for tree allocation, FACT uses these feedback-derived corrections to recalibrate candidate priorities during each tree-construction round.

Furthermore, FACT integrates feedback-calibrated candidate allocation with cost-aware incremental budget selection, jointly determining which candidates to prioritize and how much verification work to perform. By incorporating profiled full-round costs and reusing selected nodes and the search frontier, FACT improves allocation efficiency without additional neural-network training or inference-time model calls.

We will add CaDDTree, BASTION, and DARTree to Related Work and explain their contributions and relationship to FACT.

To further evaluate FACT against closely related methods, we have added comparisons with CaDDTree and BASTION using matched Qwen3-8B/DFlash checkpoints, a single RTX 4090, BF16/SDPA, greedy decoding with EOS/512-token stopping, and 182 prompts across six domains with six timing blocks. CaDDTree uses its official allocation policy with a shared verifier, while BASTION uses an adapted official generator. The results are summarized below.

Table 1. Six-domain speedups

DomainCaDDTreeBASTIONFACT
HumanEval6.472×6.201×6.541×
GSM8K5.811×5.661×5.827×
MBPP5.823×5.647×5.898×
MATH-5006.628×6.286×6.677×
Alpaca3.179×3.138×3.228×
AIME256.029×5.618×6.154×
Geometric mean5.505×5.292×5.569×

As shown in the table, FACT consistently achieves the highest speedup across all six domains, with a geometric mean of 5.569×, outperforming CaDDTree (5.505×) and BASTION (5.292×). This translates into 1.15% and 4.96% reductions in decoding latency, respectively, demonstrating the effectiveness of FACT's feedback-informed candidate prioritization and cost-aware tree allocation.

AuthorsOpenReview ↗

Response to Reviewer bxLp — Part 2/3: Comparison with Medusa and Sequoia (W1)

We thank the reviewer for highlighting these relevant studies.

W1. Limited novelty and missing closest baselines. Cost-aware tree budgets are already proposed by CaDDTree and BASTION, and building trees from offline acceptance statistics is close to Sequoia and the Medusa tree. None of them is compared; the only tree baseline on the same drafter is DDTree with a fixed budget.

Meanwhile, Medusa exploits empirical prediction accuracies across decoding heads and candidate ranks to optimize tree construction, whereas Sequoia employs dynamic programming to optimize tree topology based on acceptance-rate statistics.

Additionally, we further evaluate Medusa-style path-product greedy allocation and Sequoia-style dynamic programming adapted to the same DFlash candidate setting. The results are summarized below.

Table 2. Allocation strategies

DomainMedusa-style 192Sequoia-style 192FACT
HumanEval5.669×5.635×6.523×
GSM8K5.115×5.227×5.833×
MBPP5.172×5.293×5.916×
MATH-5005.874×5.788×6.690×
Alpaca2.869×2.945×3.234×
AIME255.294×5.376×6.154×
Overall4.864×4.916×5.558×

FACT achieves higher speedups across all six domains, reducing overall decoding latency by 12.48% and 11.55% relative to the Medusa-style and Sequoia-style implementations, respectively. The consistent improvements across diverse workloads highlight the effectiveness of FACT's feedback-informed candidate prioritization and cost-aware tree allocation in improving decoding efficiency.

We have clarified the relationship to Medusa and Sequoia in the revised Appendix D, distinguishing their acceptance-statistics-based tree construction from FACT's feedback-conditioned prioritization of current DFlash candidates.

AuthorsOpenReview ↗

Response to Reviewer bxLp — Part 3/3: Component Gains and Calibration Cost (W2)

We thank the reviewer for the valuable comments.

W2. The time saving over DDTree is at most about 3%, measured over two runs, while the method needs a correction table fitted on millions of logged verification outcomes and a profiled cost table for every target/drafter pair.

To further substantiate the performance gains, we have extended the original two timing repeats to six timing blocks and conducted additional ablation studies to evaluate the contributions of Feedback Calibration and Cost-aware Budget Control. The results are summarized below.

Table 3. Component ablation

ConfigurationTime (s) ↓Speedup (vs AR)↑Time Saved (%) ↑
Raw fixed 1282.32705.207×—
Calibrated fixed 1282.31365.237×0.576
Raw adaptive2.22775.439×4.266
Full FACT2.22595.443×4.344

As shown in the table, Feedback Calibration reduces decoding latency by 0.58% at a fixed budget, while Cost-aware Budget Control achieves a 4.27% reduction. By integrating both components, FACT achieves the lowest decoding latency, with a 4.34% reduction over the uncalibrated fixed-budget baseline. These results demonstrate the effectiveness of combining feedback-informed candidate prioritization with adaptive verification budget allocation.

To further investigate calibration overhead, we have evaluated the impact of calibration data size on decoding efficiency. The results are presented below.

Table 4. Calibration size

Calibration PromptsObservationsLatency Reduction (%) ↑
515,3600.235
1032,1920.618
2071,5040.286
40145,3280.647
Historical5,426,0510.650

Notably, calibration with 40 development prompts achieves a 0.647% latency reduction, closely matching the 0.650% obtained from the historical table containing 5.43 million observations. This indicates that FACT can achieve comparable calibration benefits with approximately 37× fewer observations in the evaluated setting. Furthermore, collecting the 40-prompt calibration data requires 59.88 seconds, and fitting the correction table takes only 0.038 seconds. Hardware profiling requires an additional 1,650 seconds, while the full historical data collection takes 4.10 GPU-hours. Once constructed, both the calibration and cost tables can be reused for a fixed model pair and hardware configuration, amortizing the preparation cost across subsequent requests without introducing additional inference-time model calls.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.