- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FuseHead accelerates CPU-based greedy decoding by combining HNSW initialization, a batch-shared scan of 8-bit vocabulary representations with conservative score bounds, and exact FP32 rescoring of surviving candidates. It preserves the outputs of a specified FP32 reference while reducing full-precision output-head computation. However, the evaluation is restricted to one modified Qwen2.5-0.5B checkpoint, one CPU platform, and a small set of batch sizes and decoding traces. Practical end-to-end gains, applicability to other models and hardware, and support for stochastic decoding remain unverified.
Critical strengths
- The idea behind this paper is clearly presented: find cheap threshold through HNSW, use 8 bit computation to compare with this threshold in order to get the candidates and finally use exact FP32 computation to determine the final chosen.
- Under the experimental settings FuseHead gets the best performance.
Critical weaknesses
- The method/kernel is applied to a narrow scope: cpu, top-1 greedy sampling. It remains unclear whether the method can benefit from other senarios like gpu and topk/topp sampling.
- The experiment scope is also limited to Qwen2.5-0.5B checkpoint, one CPU platform, and a small set of batch sizes and decoding traces.
- More discussion and ablation study about UNSW and 8-bit computation is needed. The paper acknowledges that the stored error radius is not guaranteed to upper-bound the exact reconstruction error. Consequently, the current implementation does not establish output equivalence for every input.
- Lack of practical end-to-end accelration results under different workloads.
Questions for authors
- Demonstrate the results on gpu and topk/topp sampling.
- Implement on different model sizes among different model families: such as llama-3.2-3b, qwen2.5-7b, llama-3.1-8b.
- Please analyze the trade-off between HNSW initialization quality and pruning efficiency by varying ef and the number of initial candidates.
- A bit-width sensitivity study, together with a breakdown of quantization and rounding margins would clarify why 8-bit is chosen.
- Provide practical end-to-end accelaration results under different workloads on at least a model.