- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a speculative entropy decoding (SpecED) framework for accelerating autoregressive entropy models in neural compression. Instead of computing the full entropy model sequentially for every symbol, SpecED uses lightweight draft models to predict future distributions. It then performs speculative arithmetic decoding and verifies the results with batched evaluations of the original entropy model. Multi-head lookahead drafting, speculative state branching and budgeted tree search mechanisms are introduced for efficient parallel speculation. The paper further provides a Rényi-divergence-based bound on speculation success and evaluates the method across lossless and lossy compression tasks.
Critical strengths
- The paper addresses an interesting and practical problem in autogressive neural compression, where high decoding latency is caused by sequential entropy-model evaluations. SpecED reduces such a sequential bottleneck while preserving bit-exact reconstruction and leaving the original bitstream and base entropy model unchanged.
- The proposed method is broadly evaluated across different benchmarks with various modalities and codecs to show notable speedups.
Critical weaknesses
- Table 1 compares two ALB schedules against an exhaustive oracle, but does not include single-path speculation or SLB for BOA Constrictor and OctAttention. Although related comparisons appear in the audio experiments, their benefits should also be demonstrated on these main benchmarks.
- Report measured latency for each baseline. Expected acceptance counts alone cannot show whether additional branching is worthwhile, because exploring and verifying more candidates introduces overhead.
Overall the experimental results are not sufficiently strong to make a case that the proposed branching and multi-draft architecture directly translates into improved latency.
Questions for authors
- Please address the weaknesses sections above.
- Could the authors provide a theoretical characterization or at least clarify how the current theoretical success bound should be interpreted for the actual multi-head, multi-branch budgeted tree search algorithm? Table 1 suggests that these mechanisms have a substantial effect on performance.
3 comments
Thank you for the detailed response and the additional experimental results.
The inclusion of the single-path speculation baseline and the measured latency comparisons addresses my main experimental concerns. In particular, Table 8 demonstrates that increasing the branching budget yields meaningful improvements in end-to-end latency for both BOA Constrictor and OctAttention. I agree that these results provide stronger evidence for the practical benefits of the proposed approach.
We thank the reviewer for the recognition of our work. The valuable suggestions improve the completeness of our paper.
We thank the reviewer for the constructive feedback. We updated to include the single-path speculation results (
Table 1) and clarify why existing experiments show additional branching is worthwhile. We also provide a brief characterization on how Theorem 1 extends to multi-head, multi-branch designs.Detailed responses follow.
W1. Results on single-path speculation
We have updated to include the following results of the single-path speculation in the revised paper under the same hardware and seeds (Table R3.1).
Table R3.1 E[K] of single-path speculation and ALB (2,2,1,1) in revised
Table 1(physics data, point cloud).As shown in Table R3.1, a trend consistent with original
Table 1of the manuscript can be observed, where accepted draft tokens increase with the leaf budgets, further supporting our conclusions.W2. How reported results support value of additional branching
W2.1 Baseline latency
We would like to clarify that measured baseline latency (BOA Constrictor, OctAttention and LLMZip, Encodec) has already been reported in
Figure 6, 7andTable 8. To make it clearer, we also report the detailed datapoints ofFigure 6in Table R3.2:Table R3.2 Measured latency for decoding 104 symbols of SpecED and baselines (physics data, point cloud).
W2.2 Effect of additional branching
We would like to clarify that we can already deduct that additional branching is worthwhile from existing experiment results, by realizing the physical significance of average acceptance E[K] and compare with measured latency speedup.
We note that 1+E[K] is an inherent upper bound of the latency speedup. For example, in
Fig. 6(b), ALB (6,6,4,2) strategy achieves an end-to-end latency speedup of 1.8×, which is already surpassing the upper bound 1+E[K] of single-path speculation and ALB (2,2,1,1), which are 1.43× and 1.58× respectively, as shown inTable 1. This shows that the proposed branching architecture directly translates into improved latency.We specify this property for clarity in
Section 5.1.Q1 See the responses to W1, W2.
Q2.
Theorem 1's extension to multi-head, multi-branch designTheorem 1covers multi-head drafting by letting Q to be the joint distribution induced by the K heads. It can extend to tree search. The search, including SLB/ALB pruning, is a deterministic function of the codepoint and retains at most Bn leaves at depth n. Let Q~n(z)=1/Bn∑l∣Al(z)∣, where Al(z) is the set of codepoints for which leaf l decodes block z. A union bound over leaves, followed by the argument ofTheorem 1, gives Pn(success)≤infλ∈(0,1)exp(−λ[D1−λ(P∥Q~n)−logBn]), which reduces toTheorem 1when Bn=1. The leaf budget therefore offsets logBn nats of draft–base divergence. The bound holds for any pruning rule with at most Bn leaves.We record this extension in
Remark 2,Section 4. While there exists tighter bound for data-dependent pruning rule, the derivation would be non-trivial. Exact characterization is also open in standard speculative decoding, and closed-form optimal acceptance is known only for restricted cases such as two identical drafts at the token level (Khisti et al. "Multi-draft speculative sampling: Canonical decomposition and theoretical limits." ICLR 2025). Multi-step and tree-based analyses rely on simplifying assumptions, such as i.i.d. acceptance decisions across positions (Leviathan et al., 2023), or a candidate's acceptance probability depending only on its child position under an already accepted parent (Chen et al. "Sequoia: Scalable and robust speculative decoding." NeurIPS 2024).We thank the reviewer again for the constructive feedback, allowing us to improve the completeness of the paper. We welcome further questions to enhance the contribution of our paper.