acceptodds
Under review as a conference paper at ICLR 2027

Branch-Jacobi: Prefix-Conditioned Multi-Pass Drafting for Speculative Decoding

13 views

Abstract

Block-diffusion drafters predict token blocks in parallel, but later predictions cannot condition on tokens selected earlier in the same pass. We introduce Branch-Jacobi, which reuses one DFlash-style drafter for multi-pass prefix-conditioned refinement. Each pass packs prefix-MASK branches into one forward call. Consecutive first-MASK checks identify the proposal's common prefix with a drafter-autoregressive path; selecting one complete branch extends certification before linear target verification. We prove sound prefix certification for deterministic greedy drafting under fixed cycle conditions and equivalent branch evaluation. Moving-anchor training and causal prefix supervision align the drafter with variable-prefix queries. Across seven math, code, and chat benchmarks on Qwen3-4B and Qwen3-8B at temperatures 0 and 1, a three-layer drafter using three passes per cycle achieves – geometric-mean decode-stage speedups over target autoregressive decoding, exceeding DFlash by – while increasing average acceptance length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Block Diffusion, Jacobi Decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZvvwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the lack of within-block causal conditioning in parallel speculative drafting, which limits the acceptance length of DFlash-style drafters. Motivated by the observation that later draft tokens can benefit from conditioning on previously predicted tokens, this paper proposes Branch-Jacobi, a multi-pass refinement framework that uses packed prefix-conditioned branches and AR-path consistency checks to progressively refine draft proposals. It further introduces moving-anchor training and causal prefix supervision to improve the drafter's ability to handle variable-length prefixes. Experiments on Qwen3-4B and Qwen3-8B across seven benchmarks demonstrate 4.93–6.40× decoding speedups over autoregressive decoding, outperforming DFlash by 25.3–29.2% while improving average acceptance length.

Critical strengths

The paper demonstrates substantial and consistent improvements in decoding efficiency across seven benchmarks on Qwen3-4B and Qwen3-8B. Branch-Jacobi achieves 25.3–29.2% higher speedups than DFlash while improving average acceptance length, demonstrating the effectiveness of the proposed approach.

Critical weaknesses
  1. The core motivation of this paper has been extensively explored in recent studies, such as DSpark, xPress, and DFlash 2, which similarly aim to improve parallel speculative drafting by recovering within-block token dependencies. Although Branch-Jacobi introduces a different refinement mechanism, its conceptual novelty remains a bit limited. Furthermore, the lack of direct experimental comparisons with these closely related methods makes it difficult to assess the significance of the proposed design.

  2. Considering the computational resources required for training, the evaluation on Qwen3-4B and Qwen3-8B is understandable. However, the paper lacks direct experimental comparisons with closely related methods, such as DSpark and xPress. At least one representative comparison under comparable experimental settings is necessary to establish whether the proposed refinement mechanism offers meaningful advantages over existing approaches.

If the paper provides stronger evidence of its advantages over existing methods, I may reconsider my rating.

Questions for authors
  1. What are the key conceptual and technical advantages of Branch-Jacobi over closely related methods such as xPress,DSpark or Domino?

  2. In Table 1, for Qwen3-4B at T=0, the average speedup decreases from 6.35× at F=3 to 6.11× at F=4, but slightly increases to 6.13× at F=5. What explains this non-monotonic behavior, and how can the optimal number of refinement passes be determined across different tasks and model configurations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Zvvw (Part 1 / 1)

W.1–2 / Q.1: Technical Contributions and Recent Baselines

We thank you for your careful review and for recognizing the substantial and consistent improvements in decoding speed and acceptance over DFlash across seven benchmarks on Qwen3-4B and Qwen3-8B.

Thank you for asking us to clarify the technical contribution. On Qwen3-4B at T=0T=0, with the same 100K training examples, Branch-Jacobi (F=3F=3) achieves a seven-benchmark geometric-mean speedup of 4.72×, compared with 4.30× for DSpark, the fastest recent baseline.

Relative to DSpark, Domino, DFlash2, and XPress, Branch-Jacobi reruns the same drafter backbone on packed moving-prefix rows to recompute suffix representations. First-MASK checks identify the current proposal's common-prefix boundary with the fixed greedy drafter-AR reference and select one already-computed complete branch for a single linear target verification. General Response #1 provides the mechanism distinctions and direct comparisons.

Moving-anchor training and causal prefix supervision adapt the drafter to recomputation under different prefix lengths, providing a training design matched to this inference procedure.

The theoretical guarantee concerns prefix consistency with the drafter path, rather than target acceptance of the entire suffix. In the separate complete-branch versus first-MASK assembly comparison (pKWA W2; corresponding table in the revised Appendix), we use the same drafter checkpoint and packed-branch evaluation protocol, changing only the proposal-update rule while keeping target verification unchanged. Based on these tabulated values, complete-branch selection has higher reported speedup in all 24 configurations, with relative differences of 4.1%–26.3%. These results provide empirical evidence concerning the update design alongside the recent-baseline comparisons.

Q.2: Refinement Pass Budget

Thank you for pointing out the non-monotonic trend in Table 1 and asking about the choice of FF. General Response #2 reports our timing re-evaluation of the submitted Tables 1, 3, and 4 and updated measurements. For Qwen3-4B at T=0T=0, recomputing the F=4F=4 seven-benchmark geometric mean from the reported benchmark values gives approximately 6.2×, between 6.35× at F=3F=3 and 6.13× at F=5F=5. The updated aggregate therefore no longer shows the F=4F=4-to-F=5F=5 rebound seen in the submitted table. The corresponding aggregate AAL values remain 8.27, 8.41, and 8.45.

Choosing FF. Higher AAL alone does not determine the preferred budget; we select FF using measured response-mean TPOT. We recommend F=3F=3 as a practical default for the main-table settings, with a small latency sweep for further task-specific optimization. The preferred budget also depends on serving load: our concurrency sweep favors F=3F=3 at C=1C=1–4 and F=2F=2 at C=8C=8–16.

Execution cost and extended pass-budget sweep. The fixed-shape implementation in Appendix D executes all scheduled draft forwards, including after certification is complete; extra passes therefore continue to incur draft computation.

To check whether the decline in seven-benchmark geometric-mean speedup observed from F=3F=3 to F=5F=5 continues at larger budgets, we extended the sweep to F=6F=6 and F=7F=7 using the same Qwen3-4B checkpoint and timing protocol as Table 1, at T=0T=0 and block size 16. Results are reported as mean ± sample standard deviation over four runs with a shared AR baseline; other reporting conventions follow Table 1. The full results of the extended sweep have been added to Appendix E of the revised manuscript.

BenchmarkF=6F=6 SpeedupF=6F=6 τ\tauF=7F=7 SpeedupF=7F=7 τ\tau
GSM8K6.50±0.12×9.00±0.006.39±0.04×9.00±0.00
MATH-5007.17±0.15×9.99±0.007.01±0.09×9.99±0.00
AIME257.66±0.16×9.24±0.007.45±0.21×9.25±0.00
HumanEval5.85±0.07×8.18±0.005.61±0.10×8.18±0.00
MBPP5.67±0.14×7.86±0.005.57±0.07×7.86±0.00
LCB6.72±0.08×9.14±0.006.59±0.06×9.14±0.00
MT-Bench3.45±0.02×5.81±0.003.37±0.06×5.81±0.00
Avg.5.98±0.07×8.46±0.005.84±0.03×8.46±0.00

Average acceptance length remains approximately 8.46, while average speedup falls from 5.98±0.07× to 5.84±0.03×. These measurements illustrate diminishing returns: additional draft forwards incur computation while yielding little further acceptance gain.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Revisions

Dear Reviewer Zvvw,

We appreciate your constructive feedback on the technical contribution and refinement budget. Following your comments, we have revised the manuscript with clearer distinctions from related methods, comparisons against recent baselines, and an extended pass-budget study. If you have time, could you please let us know if any concerns remain?

Best regards,
Authors of Branch-Jacobi

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Reviewer_n57VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes branch-Jacobi that improves parallel drafting through prefix-conditioned refinement. Multiple branches are evaluated in parallel to identify a consistent prefix and update the remaining draft. The target then verifies only one final draft sequence. Overall, the paper is well-written and the presentation is mostly clear. I've given an initial score of 3 due to insufficient justifications, the authors have completed the rebuttal and addressed most of my comments so I'd maintain my score and consider this paper acceptable.

Critical strengths
  1. I have explored a similar idea of multi-step denoising myself, but found the gains limited. This makes the branch-based refinement particularly interesting to me. The combination of moving-anchor training and prefix-conditioned inference seems to offer a useful way to make those extra passes pay off.
  2. Improving parallel draft quality without adding target verification passes is practically relevant. The reported gains over DFlash make this worth reproducing and exploring further.
  3. The controlled ablations are helpful. In particular, the root-only versus moving-anchor comparison shows why the training changes matter, rather than attributing everything to extra inference compute.
Critical weaknesses

There're some insufficiencies in the original manuscript, the authors have added more justifications and I believe the authors have done a good job to address most of my concerns. For the record, I still list my original questions below:

  1. The evaluation is limited to Qwen3-4B and 8B. I would like to see newer and larger targets, such as Qwen3.5 or later models, and comparisons with stronger refinement methods such as Domino, DSpark, and DFlash2. How sensitive are the gains to block size? A block-size-8 experiment would help establish whether the method remains useful with shorter drafts.

  2. Reproducibility is also important. However, the results depend on both a custom training recipe and an optimized packed-refinement implementation. I would appreciate a release of the training and inference code, including checkpoints and evaluation scripts, so that others can verify the gains and build on the method.

  3. The batch-1 gains are encouraging, but what happens at higher concurrency? Although the target only verifies one final draft, refinement expands each request into multiple drafter branches. This increases the effective drafting batch and may introduce substantial compute and memory overhead when many requests run together. A concurrency sweep would help show whether the acceptance gains still outweigh the extra drafting cost.

  4. I would also like more discussion of recent work such as DSpark [1], D-cut [2], and DOUBLE [3]. These address different aspects of speculative decoding, so a direct comparison is not always necessary. Still, explaining the differences and potential combinations would help clarify where Branch-Jacobi fits.

[1] Xin Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026.

[2] Tianyu Liu et al. D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding. arXiv:2607.14647, 2026.

[3] Yuhao Shen et al. Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.

Questions for authors

Same as the weaknesses above. I am positive about the core idea. If the authors address these concerns with convincing evidence and clarify their code-release plans, I would be willing to raise my score to 4.

Final comments: I've maintained my score of 3 and considered this work acceptable to ICLR.

4 comments
AuthorsOpenReview ↗

Response to Reviewer n57V (Part 1 / 2)

W.1: Broader Evaluation and Block Size

We thank you for your thoughtful review and for recognizing the practical relevance of improving parallel draft quality without additional target verification passes. We also appreciate your interest in the combination of moving-anchor training and prefix-conditioned inference, and your positive assessment of the controlled ablations.

Following your suggestion for broader evaluation, we briefly discuss the findings from the new target-model and block-size experiments in Section 4 and provide the full results in Appendix E. Unless otherwise stated, we use the evaluation metrics and benchmark subsets in Section 4.1 and Appendix D of the paper. BJ denotes Branch-Jacobi.

Additional target: Qwen3.6-35B-A3B. DFlash and Branch-Jacobi are trained on the same 100K examples for six epochs, following the paper's ablation setup. We use block size 16 and disable thinking at both T=0T=0 and T=1T=1.

TTMethodMATH-500 Speedupτ\tauHumanEval Speedupτ\tauMT-Bench Speedupτ\tau
0DFlash5.21×5.816.33×7.252.59×3.69
0BJ F15.66×5.957.13×7.642.85×3.74
0BJ F26.42×7.338.56×10.022.97×4.37
0BJ F36.38×7.798.70×10.772.83×4.54
0BJ F46.30×7.948.60×10.952.81×4.63
0BJ F56.24×7.978.45×11.002.73×4.63
1DFlash4.48×5.025.44×6.392.55×3.31
1BJ F15.27×5.726.52×7.352.66×3.43
1BJ F26.01×6.888.55×9.912.87×3.97
1BJ F36.29×7.408.62×10.542.82×4.21
1BJ F45.83×7.358.10×10.592.63×4.22
1BJ F55.79×7.417.94×10.482.63×4.33

Branch-Jacobi improves both acceptance length and speedup over DFlash on all three benchmarks at both temperatures across the reported pass budgets.

Recent baselines. General Response #1 reports comparisons with Domino, DSpark, DFlash2, and XPress on Qwen3-4B at T=0T=0 under matched training and inference settings.

Block size. These experiments use Qwen3-4B at T=0T=0 with 100K training examples. BtrainB_{train} and BtestB_{test} denote training and inference block sizes. The Btrain=Btest=16B_{train}=B_{test}=16 rows reuse the corresponding Branch-Jacobi measurements in General Response #1, with identical checkpoints, prompts, and inference implementation.

FFBtrainB_{train}BtestB_{test}MATH-500 Speedupτ\tauHumanEval Speedupτ\tauMT-Bench Speedupτ\tau
116164.96×5.724.09×4.672.58×3.48
11684.05×4.883.55×4.252.45×3.27
18164.18×5.103.54×4.262.40×3.28
1884.16×5.023.67×4.382.50×3.36
216165.60×7.074.63×5.822.74×4.14
21684.31×5.423.84×4.832.57×3.63
28164.96×6.444.13×5.292.56×3.90
2884.39×5.523.93×4.922.62×3.74
316165.78×7.504.82×6.242.77×4.33
31684.22×5.543.81×4.982.52×3.73
38165.13×6.984.26×5.772.54×4.13
3884.29×5.633.87×5.042.56×3.82

With matched training and inference block size 8, two passes improve speedup over one pass by 5.5%, 7.2%, and 4.8% on MATH-500, HumanEval, and MT-Bench, respectively.

The following counts report responses whose per-response mean τ>8\tau>8, measured with Btrain=Btest=16B_{train}=B_{test}=16. MT-Bench contributes 160 responses from 80 two-turn dialogues.

FFMATH-500HumanEvalMT-BenchTotal
14/128 (3.13%)0/164 (0%)4/160 (2.50%)8/452 (1.77%)
230/128 (23.44%)6/164 (3.66%)12/160 (7.50%)48/452 (10.62%)
343/128 (33.59%)11/164 (6.71%)12/160 (7.50%)66/452 (14.60%)

At block size 8, refinement yields smaller speedup gains than at block size 16 because the short-block limit constrains our higher acceptance lengths, leaving some long-prefix benefits unrealized. With block size 16 and three draft passes, 14.60% of responses have a mean acceptance length above 8, rising to 33.59% on MATH-500.

AuthorsOpenReview ↗

Response to Reviewer n57V (Part 2 / 2)

W.2: Reproducibility

Thank you for emphasizing reproducibility. We will release the materials needed to reproduce both the training recipe and the packed-refinement implementation according to the following schedule:

  • Within two weeks of acceptance: all training and inference code, checkpoints, evaluation scripts, the detailed training-data recipe, and preprocessing code supporting the main results in Table 1.
  • Within four weeks of acceptance: the corresponding code, checkpoints, evaluation scripts, data recipes, and preprocessing code for the remaining ablation studies.

W.3: Higher Concurrency

Thank you for highlighting the importance of concurrent serving. We added a concurrency sweep to evaluate whether the acceptance gains continue to offset the additional drafting cost. We briefly discuss the findings in Section 4 and provide the full results in Appendix E.

We evaluate Qwen3-4B with block size 16 at T=0T=0 on the same fixed 128-prompt MATH-500 subset, following the precision, sampling, and stopping settings in Section 4.1 and Appendix D of the paper. Contexts range from 36 to 375 tokens (mean approximately 94). Speedup is the ratio of mean AR TPOT to the method's mean TPOT at the same concurrency CC.

Decoding speedup over AR decoding.

Concurrency CCDFlashBJ F=1F=1BJ F=2F=2BJ F=3F=3BJ F=4F=4
15.99×6.01×7.23×7.51×7.26×
26.37×6.50×7.55×7.72×7.48×
46.46×6.38×7.28×7.58×7.03×
85.84×5.80×6.34×5.84×5.90×
165.50×5.47×6.08×5.73×5.18×

B-Jacobi with F=2F=2 outperforms DFlash throughout C=1C=1–16. The best tested pass budget is F=3F=3 for C=1C=1–4 and F=2F=2 for C=8C=8–16. At C=8C=8 and 16, F=2F=2 improves speedup over DFlash by 8.6% and 10.5%, respectively. Thus, refinement remains beneficial at higher concurrency, with a smaller pass budget providing the best performance in these settings.

PyTorch peak GPU memory (GiB), including resident model memory. Each entry is allocated / reserved.

Concurrency CCDFlashBJ F=1F=1BJ F=2F=2BJ F=3F=3BJ F=4F=4
19.02 / 9.338.89 / 9.189.17 / 9.609.41 / 10.009.41 / 9.99
29.38 / 9.768.97 / 9.349.31 / 9.809.33 / 9.819.34 / 9.82
49.54 / 10.109.13 / 9.669.49 / 10.389.52 / 10.389.51 / 10.34
810.14 / 11.329.64 / 10.8110.10 / 11.8410.14 / 11.8310.04 / 11.92
1611.22 / 13.7910.84 / 13.5911.41 / 14.2511.35 / 14.1411.45 / 14.07

W.4: Recent Related Work and Potential Combinations

Thank you for this suggestion. We have expanded Section 2 to discuss DSpark's verification scheduling, D-cut's cross-request prefix pruning, and DOUBLE's retrieval-guided overlap between drafting and verification, including potential combinations with our prefix-conditioned drafter recomputation. General Response #1 provides the drafting-mechanism distinctions and direct comparisons.

We are particularly interested in combining our approach with D-cut. In our concurrency experiments, Branch-Jacobi's gains over DFlash are more pronounced at lower concurrency (see W.3), while D-cut addresses high-concurrency verification costs. This motivates jointly adapting refinement and verification budgets to serving load. Within the limited rebuttal period, we preliminarily tested a D-cut-inspired pruning rule based on drafter-certified prefix length, but have not observed consistent speedup gains. In our view, this preliminary test leaves open how much can be gained by jointly allocating refinement and verification budgets. We thank the reviewer for suggesting this direction and would like to study how the two budgets can be adapted together as serving load changes.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Revisions

Dear Reviewer n57V,

Thank you again for your helpful suggestions. We have added experiments with a larger target model, block size 8, and higher concurrency to the revised paper, and outlined a code-release schedule in our response. When convenient, we would be grateful to hear whether these updates address your concerns.

Best regards,
Authors of Branch-Jacobi

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Reviewer_pKWAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

Branch-Jacobi method first generates a block using the draft model and then refines the draft before target verification. It addresses a limitation of DFlash's parallel drafting where later predictions do not observe the actual tokens chosen earlier in the same pass. Each refinement pass constructs branches retain prefixes of different lengths from the current proposal and mask the remaining positions. A common prefix is identified by comparing the first MASK predictions from the branches with the current proposal through a recursively defined reference drafting path.

The algorithm selects the complete branch whose first masked position extends the certified prefix. The target after a fixed number of draft passes verifies one linear proposal.

The drafter is trained using moving-anchor suffix queries and auxiliary causal prefix supervision. Experiments compare Qwen3 4/8B models on math, code and chat benchmarks. The results show improvements over DFlash. Across the evaluated settings, 5 passes achieve the highest average acceptance length while 3 passes achieve highest speedup.

Critical strengths
  1. Different budget pass experiments shows that the refinement improves target acceptance. The authors show that in Figure 4, longer certified prefixes show higher target acceptance. The paper distinguises this correlation from theoretical guarantee of agreement with draft's AR path.

  2. Each refinement pass computes both the first MASK consistency checks and complete suffix predictions. Once the method chooses a branch, its suffix is already available, so additional draft forward is not required. The target model checks only one sequence which makes the computational design simple.

Critical weaknesses
  1. Figure 2, examines conditional acceptance by draft position, but its DSpark curve uses a different target model and sampling setup and therefore supports only a qualitative comparison. DSpark’s lightweight Markov head provides a relevant alternative to repeated suffix refinement by conditioning logit corrections on the immediately preceding sampled token. A matched comparison of position-wise acceptance, average acceptance length, and latency would help clarify the trade-off between these approaches. I suggest also including comparison with DSpark in Table 1.

  2. Comparing one branch with multiple branches answers whether multiple branches help. It does not answer whether selecting a whole branch is better than combining the predictions already computed. The assembled first-MASK vector VV also agrees with the reference drafting path through the newly certified position l+1l+1. The selected branch and VV differ in their provisional suffixes. The authors provide a plausible rationale for selecting a complete branch that its suffix avoids conditioning on the known mismatching token. But the theorem does not establish whether this produces better target acceptance or lower latency. A controlled comparison of these two update rules, using identical branch evaluations and checkpoints, would clarify whether complete-branch selection improves target acceptance and latency.

  3. Packing branches into one forward call still requires processing multiple versions of the proposal, even though the branches share target-context keys and values. The paper reports latency and stage costs, but peak memory usage and performance across context lengths would clarify the practical cost of this design.

  4. In Table 4, on GSM8K at F=4F=4, increasing λ\lambda from 0 to 0.2 raises average acceptance length from 6.66 to 6.82 but reduces speedup from 5.70 to 5.02. This effect differs on HumanEval, where speedup increases slightly. These results suggest that improved acceptance does not consistently translate into lower latency. The authors should explain how λ=0.2\lambda=0.2 was selected for the main model and which validation criterion guided that choice.

Questions for authors

Kindly refer to points in weakness 1-4.

8 comments
AuthorsOpenReview ↗

Response to Reviewer pKWA (Part 1 / 3)

W.1: Baseline Comparisons

Thank you for recognizing the acceptance gains from refinement and our distinction between the drafter-path guarantee and the observed correlation with target acceptance. We also appreciate your positive assessment of the packed forward that supplies both consistency checks and suffix predictions while retaining simple, single-sequence target verification.

Thank you for pointing out the mismatch in the original DSpark curve. We agree that the cross-setting comparison supports only a qualitative interpretation. Following your suggestion, we have updated the position-wise acceptance figure with the measurements below. General Response #1 reports average acceptance length and latency-based speedup at T=0T=0 for drafters trained on the same 100K examples.

Conditional acceptance by draft position (%). We compare B-Jacobi (F=1F=1–5) with DSpark, DFlash, and DFlash2. For each method, we reuse its 100K-trained checkpoint from General Response #1 with the same block size (B=16B=16). All methods use Qwen3-4B at T=0T=0 on the same 128 MATH-500 prompts. We report qi=Pr⁡(a≥i∣a≥i−1)q_i=\Pr(a\ge i\mid a\ge i-1), where aa counts target-accepted draft tokens. BJ denotes Branch-Jacobi.

PositionBJ F=1F=1BJ F=2F=2BJ F=3F=3BJ F=4F=4BJ F=5F=5DSparkDFlashDFlash2
186.1585.5185.2485.3185.2888.1587.6488.28
282.5186.9486.7886.7686.7286.9882.9987.10
383.1487.0588.3288.3188.3387.0283.4786.65
482.9087.2589.0189.2589.2786.7583.0486.90
583.5187.7889.4189.8589.7887.0083.8786.88
684.5888.2789.7690.2090.4687.1784.0987.20
783.3387.5089.2889.7989.9686.8883.4187.36
882.8588.9490.3591.1091.3886.6283.5187.52
982.2088.1089.7390.3390.7886.5482.7686.89
1079.8688.4890.1990.7591.0885.4381.6885.36
1178.3288.1189.6190.1490.3884.7281.7985.46
1273.0587.6390.4191.3691.3184.0580.6084.46
1370.6186.8289.2090.7991.3883.4878.6884.35
1465.9186.0189.4190.5290.8383.8476.4184.19
1564.2984.7089.7490.4691.1081.5576.5282.59

At position 15, B-Jacobi with F=3F=3–5 reaches 89.74%–91.10%, versus 81.55% for DSpark, a gain of 8.19–9.55 percentage points.

Released-checkpoint comparison. We additionally evaluated the official DSpark checkpoints for Qwen3-4B/8B and have added these results to Table 1. These checkpoints use B=8B=8, whereas B-Jacobi uses B=16B=16, F=3F=3. Computational limits prevented full-data training of a DSpark B=16B=16 checkpoint. This comparison therefore measures the released systems with different block sizes; General Response #1 provides the controlled 100K-data comparison at T=0T=0.

The B-Jacobi entries reuse the corresponding results in Table 1 and follow its metric and aggregation conventions.

Qwen3TTMethodGSM8K Speedupτ\tauMATH Speedupτ\tauAIME25 Speedupτ\tauHumanEval Speedupτ\tauMBPP Speedupτ\tauLCB Speedupτ\tauMT-Bench Speedupτ\tauAvg. Speedupτ\tau
4B0B-Jacobi7.04×8.807.59×9.798.11×9.016.18×7.975.95×7.617.16×9.023.63×5.666.35×8.27
4B0DSpark5.50×6.535.47×6.435.77×5.815.07×5.714.96×5.594.97×5.543.70×4.475.02×5.73
4B1B-Jacobi6.18×8.015.91×7.853.93×5.385.57×7.125.28×6.846.30×8.483.20×5.095.06×6.97
4B1DSpark5.46×6.365.15×6.024.35×5.095.02×5.594.79×5.444.47×5.373.41×4.314.62×5.45
8B0B-Jacobi7.15×9.007.92×10.057.62×9.476.25×8.275.96×7.817.10×9.423.86×5.516.40×8.50
8B0DSpark4.58×6.694.57×6.544.40×5.894.08×5.884.14×5.664.11×5.812.97×4.404.09×5.84
8B1B-Jacobi5.95×7.735.74×7.624.28×5.785.03×6.475.00×6.516.45×8.802.99×4.784.93×6.81
8B1DSpark4.57×6.405.03×6.163.34×5.164.49×5.584.40×5.464.21×5.602.75×4.214.04×5.51

Average speedup improves over the released DSpark checkpoints by 26.5%/9.5% for 4B at T=0/1T=0/1, and 56.5%/22.0% for 8B.

AuthorsOpenReview ↗

Response to Reviewer pKWA (Part 2 / 3)

W.2: Proposal-Update Rule

Thank you for suggesting a controlled comparison of the two proposal-update rules. We briefly discuss the findings in Section 4 and provide the full results in Appendix E.

Complete-branch selection versus first-MASK assembly. We agree that the theorem establishes prefix consistency, rather than superiority of the provisional suffix. Both the assembled first-MASK vector VV and the selected complete branch CℓC_\ell agree with the reference drafting path through the newly certified position. Their remaining suffixes require empirical comparison.

Following your suggestion, we compare complete-branch selection with first-MASK assembly using the same Qwen3-4B drafter checkpoint as Table 1. Both methods use the same packed-branch evaluation protocol and differ only in the proposal-update rule; the target-verification procedure is unchanged. We use the evaluation metrics and benchmark subsets defined in Section 4.1 and Appendix D of the paper.

Based on the tabulated values, complete-branch selection has higher reported τ\tau and speedup than first-MASK assembly across all 24 configurations, with relative differences of 3.2%–13.4% and 4.1%–26.3%, respectively. These results complement the theoretical prefix-consistency guarantee with an empirical comparison of the two update rules.

Qwen3-4B, T=0T=0.

TaskFFSpeedup VVSpeedup CℓC_\ellτ\tau (VV)τ\tau (CℓC_\ell)
MATH26.14×7.35×8.199.27
MATH36.07×7.59×8.649.79
MATH46.23×7.65×8.989.94
MATH56.55×7.20×9.279.97
HumanEval24.95×5.99×6.617.49
HumanEval34.95×6.18×7.037.97
HumanEval44.98×5.94×7.348.13
HumanEval55.12×6.08×7.578.17
MT-Bench23.14×3.55×4.815.34
MT-Bench33.21×3.63×5.095.66
MT-Bench43.11×3.64×5.275.77
MT-Bench52.96×3.48×5.405.80

Qwen3-4B, T=1T=1.

TaskFFSpeedup VVSpeedup CℓC_\ellτ\tau (VV)τ\tau (CℓC_\ell)
MATH24.72×5.81×6.817.55
MATH34.86×5.91×7.207.85
MATH45.02×5.66×7.367.81
MATH55.31×5.53×7.577.81
HumanEval24.29×5.39×5.976.76
HumanEval34.41×5.57×6.357.12
HumanEval44.71×5.51×6.677.26
HumanEval54.58×5.36×6.747.28
MT-Bench22.62×3.18×4.294.81
MT-Bench32.79×3.20×4.645.09
MT-Bench42.71×3.10×4.735.04
MT-Bench52.78×3.08×4.885.08

W.3: Context Length and GPU Memory

Thank you for asking about the practical cost of packed refinement. We added a context-length sweep on Qwen3-4B at T=0T=0, with thinking disabled, using the first 32 MATH-500 problems. Each context-length setting is repeated three times. All runs reuse the same set of 100K-trained checkpoints, with block size B=16B=16, concurrency C=1C=1, and a fixed output length of 128 tokens. We measure peak allocated GPU memory with PyTorch. We briefly discuss the findings in Section 4 and provide the full results in Appendix E.

Decoding speedup over AR decoding.

Context lengthDFlashBJ F=1F=1BJ F=2F=2BJ F=3F=3BJ F=4F=4BJ F=5F=5
5124.31×4.30×5.17×5.22×5.37×5.22×
10244.08×3.98×4.75×4.85×4.79×4.87×
20484.19×4.40×5.22×5.43×5.55×5.42×
40964.09×3.55×4.28×4.42×4.59×4.48×

PyTorch peak allocated GPU memory (GiB). The minimum in each row is bolded.

Context lengthDFlashBJ F=1F=1BJ F=2F=2BJ F=3F=3BJ F=4F=4BJ F=5F=5
5128.8098.6888.9619.2089.2089.208
10249.0018.8909.1419.3929.3929.392
20489.3519.2409.4849.7269.7269.726
409610.11810.00510.24910.49110.49110.491

Across the tested context lengths, all multi-pass settings (F=2F=2–5) remain faster than DFlash. For F=3F=3–5, the additional peak allocated memory over DFlash is only 0.373–0.399 GiB (3.7%–4.5%). Over the tested 512–4096-token range, this measured increment remains approximately constant rather than growing linearly with context length, while total memory increases.

AuthorsOpenReview ↗

Response to Reviewer pKWA (Part 3 / 3)

W.4: Hyperparameter Selection and Decoding Efficiency

Thank you for highlighting the result in submitted Table 4 and asking how λ\lambda was selected. After re-evaluation, the GSM8K speedups at F=4F=4 are similar for λ=0\lambda=0 and 0.20.2 (4.99× and 5.02×, respectively), while AAL increases from 6.66 to 6.82. The λ=0.2\lambda=0.2 entry retains its submitted value. General Response #2 describes the timing re-evaluation of the submitted Tables 1, 3, and 4.

Selection of λ\lambda. We chose 0.2 as our default in part because of the roles of the two objectives. The proposed prefix tokens used during refinement must themselves be generated by the drafter at inference time, so we supervise prefix generation as an auxiliary objective. Masked-suffix prediction remains the main task, which favors a modest auxiliary weight. We have clarified this motivation in the revised manuscript's “Prefix-loss weight” paragraph. The training measurements also support this choice. Over the final 1,000 steps on the 100K training subset, λ=0.2\lambda=0.2 and 0.40.4 gave similar mean acceptance lengths (both 4.28 when rounded to two decimal places).

λ\lambda00.20.40.60.81.0
Mean acceptance length4.264.284.284.284.244.20

The GSM8K and HumanEval ablations provide a further check. At F=4F=4, every tested nonzero weight, including 0.2, gives higher AAL than λ=0\lambda=0 on both benchmarks. These results support using a modest nonzero weight; the highest GSM8K AAL in this sweep occurs at λ=0.6\lambda=0.6.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response 1: Recent Baselines and Mechanism Differences

We thank Reviewers pKWA, n57V, and Zvvw for their constructive suggestions to strengthen our comparisons with recent methods. We trained the drafters on the same 100K examples and compared DSpark, DFlash2, Domino, XPress, and Branch-Jacobi on the same seven benchmarks using Qwen3-4B at T=0T=0 with block size B=16B=16 for all methods. These comparisons have been added to Appendix E of the revised manuscript.

With 100K training examples, Branch-Jacobi (F=3F=3) achieves a 9.8% higher geometric-mean speedup across the seven benchmarks than DSpark, the fastest recent baseline (4.72× versus 4.30×). Mean acceptance length increases from 5.47 to 6.38.

MethodWhere within-block dependencies enterHow the proposal is formed or updated
DSparkLightweight sequential headConditions predictions from the parallel backbone on preceding tokens.
DFlash2In-block convolutions and candidate-path selectionUses adjacent-token compatibility to select a path through per-position candidates.
DominoGRU prefix encoder and residual headApplies a low-rank, prefix-dependent correction to backbone logits.
XPressLightweight causal refinerIteratively refines logits across the block with backbone features held fixed.
Branch-JacobiPrefix-conditioned draft backbonePacks candidate prefixes of different lengths into the same backbone, recomputes suffix representations, and uses first-MASK consistency checks to select one complete branch.

Metrics and aggregation follow Section 4.1 and Appendix D of the paper. We report raw τ\tau before EOS/length-cap clipping, including the target bonus token and averaging cycles within each response before averaging responses equally. FF includes the initial draft pass. Best averages are bold.

Qwen3-4B, T=0T=0.

MethodGSM8K Speedupτ\tauMATH Speedupτ\tauAIME25 Speedupτ\tauHumanEval Speedupτ\tauMBPP Speedupτ\tauLCB Speedupτ\tauMT-Bench Speedupτ\tauAvg. Speedupτ\tau
DFlash4.13×5.254.66×5.985.03×5.513.84×4.983.42×4.574.57×5.862.43×3.593.92×5.11
DSpark4.71×5.755.09×6.385.43×5.904.20×5.353.92×4.964.81×6.152.64×3.784.30×5.47
DFlash24.42×5.615.11×6.445.39×6.004.16×5.283.88×4.914.83×6.102.56×3.774.23×5.45
Domino4.47×5.495.08×6.385.34×5.924.06×5.133.88×4.844.78×6.102.49×3.624.19×5.36
XPress4.59×5.754.99×6.415.31×5.934.17×5.323.86×4.984.83×6.152.55×3.744.23×5.47
B-Jacobi (F=1F=1)4.29×4.994.96×5.725.27×5.364.09×4.673.61×4.274.54×5.582.58×3.484.10×4.87
B-Jacobi (F=2F=2)4.98×6.215.60×7.075.25×6.574.63×5.824.14×5.225.35×6.862.74×4.144.56×5.98
B-Jacobi (F=3F=3)5.20×6.675.78×7.505.47×7.034.82×6.244.30×5.605.53×7.302.77×4.334.72×6.38
B-Jacobi (F=4F=4)5.18×6.825.73×7.625.44×7.194.82×6.404.29×5.715.46×7.432.72×4.394.69×6.51
B-Jacobi (F=5F=5)5.08×6.855.63×7.675.35×7.234.73×6.454.22×5.755.33×7.462.67×4.414.60×6.54
AuthorsOpenReview ↗

General Response 2: Timing Protocol and Updated Speedup Measurements

We thank the reviewers for prompting a closer examination of our timing results. We measure a separate AR reference for each experimental batch to reduce sensitivity to changes in GPU operating conditions over time, such as variations in GPU temperature. Short-term timing variation may nevertheless affect the relative latencies of AR and speculative decoding within a batch.

For GSM8K, the AR reference was 30.1265 ms/token for the λ=0\lambda=0, F=2F=2–5 group and 28.795226 ms/token for the λ=1\lambda=1, F=2F=2–5 group, compared with 26.2768 ± 0.3688 ms/token (mean ± SD across the six other GSM8K AR reference batches). This contrast motivated a closer re-examination of the timing results.

We reran every configuration in the submitted Tables 1, 3, and 4 three times under the original settings. The updates below use the first rerun, and the other two runs were used to assess timing variability. For most configurations, speedup was within 2% of the submitted value. The additional reruns corroborated both the corrected speedups and those retained unchanged in the revised tables.

All updates below concern Qwen3-4B at T=0T=0.

Submitted Table 4: Prefix-loss ablation. Each cell shows submitted speedup → rerun speedup.

FFGSM8K, λ=0\lambda=0HumanEval, λ=0.2\lambda=0.2GSM8K, λ=1\lambda=1HumanEval, λ=1\lambda=1
25.40× → 4.73×4.14× → 4.63×5.25× → 4.77×4.94× → 4.33×
35.49× → 4.81×4.53× → 4.82×5.55× → 5.05×5.08× → 4.45×
45.70× → 4.99×4.38× → 4.82×5.44× → 4.95×5.21× → 4.56×
55.60× → 4.90×4.43× → 4.73×5.38× → 4.89×5.00× → 4.38×

Following the full re-evaluation, we retain all originally reported AAL values, the F=1F=1 entries, and the GSM8K λ=0.2\lambda=0.2 entries in the submitted Table 4.

Table 1: GSM8K, F=4F=4. The rerun speedup is 6.82×, compared with 6.11× in the submitted table; the reported AAL remains 8.95. Recomputing the corresponding seven-benchmark geometric-mean speedup from the reported benchmark values gives approximately 6.2×. The highest aggregate speedup in this setting remains 6.35× at F=3F=3.

We have incorporated these timing updates and the corresponding aggregate recalculation into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.