- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SMC-SD for approximate decoding, replacing token-level rejection with importance-weighted resampling. The method provides error bounds, speed analysis, and a shared-KV-cache implementation. Experiments show higher throughput with small average accuracy losses.
Critical strengths
- Replacing rejection with particle resampling offers an interesting approach to inference acceleration. The connection between the algorithm and GPU utilization provides a useful perspective on the speed–quality tradeoff.
- The implementation addresses practical KV-cache sharing and resampling costs. The paper also reports the high-batch crossover where an exact decoder becomes faster, helping identify the method’s operating regime.
Critical weaknesses
-
The motivation for choosing Sequential Monte Carlo is not sufficiently established. While the paper explains how SMC addresses the limitations of rejection-based speculative decoding, it does not clearly justify why SMC is preferable to other approximate decoding approaches. In particular, the benefits of sequential resampling over simpler alternatives remain unclear.
-
The accuracy evaluation leaves important reliability questions unresolved. Although the paper evaluates four benchmarks, it does not analyze how accuracy degradation varies with task difficulty or generation length. Long-context performance is also not systematically evaluated, and the fixed-horizon error bound does not fully address these concerns.
-
The experimental comparisons with existing speculative decoding methods are insufficient. The comparisons with EAGLE3 and DSpark lack sufficient configuration details to assess their fairness. The lossy decoding comparisons report throughput without corresponding quality measurements, while other approximate methods, such as FSD and FLy, are not evaluated.
Questions for authors
-
Why is SMC particularly suitable for approximate speculative decoding compared with other approaches? What specific benefits does sequential resampling provide? Could the authors compare it with simpler alternatives, such as importance sampling without intermediate resampling?
-
How does accuracy degradation relative to the target model vary with task difficulty? Could the authors provide results on more challenging tasks to assess whether the reported accuracy retention generalizes?
-
At a fixed particle count, how does accuracy degradation change with generation length? How do accuracy and throughput behave when both input contexts and generated outputs are long?
-
Could the authors provide more details on the configurations and parameter selection for EAGLE3 and DSpark? What quality degradation accompanies the reported speedups of lossy methods, including Cactus? How does SMC-SD compare with FSD or FLy under similar accuracy constraints?