- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a lossless entropy-coding framework, namely MINE, for memory-efficient LLM inference. MINE adapts existing GPU entropy coding techniques to enable fused decoding and matrix-vector multiplication, together with hardware-aware auto-tuning to optimize the memory–latency trade-off. Experiments on Llama 3.1 8B demonstrate up to 2.52× and 1.48× single-stream decoding speedups over the fastest baselines for the same checkpoint on RTX 4060 Ti and RTX 5090.
Critical strengths
The paper makes traditionally GPU-unfriendly entropy-coded weight representations more efficient for single-stream LLM inference through optimized GPU kernels. The reported performance improvements demonstrate the practical potential of this approach on the evaluated configurations.
Critical weaknesses
- The system novelty and algorithm novelty are limited. The fundamental idea of GPU-efficient entropy decoding fused with matrix-vector multiplication has already been explored by dtANS (IPDPS 2026). Furthermore, ZipServ (ASPLOS 2026) has demonstrated lossless compressed LLM inference through hardware-aware fused decompression and computation. While MINE introduces useful GPU-specific optimizations, its technical contributions appear largely incremental relative to these existing works.
2.Kernel-level ablations remain insufficient. The paper introduces several GPU optimizations but does not adequately quantify their individual contributions. In particular, I would expect comparisons with the original dtANS design and ablations of introduced components (asynchronous chunk buffering, sparse escape handling, and simplified decoder state management). Without these experiments, it is difficult to determine which optimizations are responsible for the reported improvements or what new GPU specific design insights can be generalized.
- The empirical evaluation was limited in scope. First, all experiments are conducted on a single 8B model. The evaluation also primarily relies on synthetic short-context decoding benchmarks rather than representative end-to-end inference workloads. Given the paper's positioning as a memory-efficient inference engine, I would expect larger-scale evaluations (e.g., 32B/70B) and experiments under realistic inference workloads.
Questions for authors
-
What are the main technical insights introduced by MINE beyond dtANS and ZipServ?
-
Could the authors provide controlled kernel-level ablations to quantify the contributions for each of these components: asynchronous buffering, sparse escape handling, and simplified decoder state management?
-
Could the authors provide end-to-end evaluations on larger compressed models (e.g., 32B/70B) and real single-stream inference workloads with varying input and output lengths?
Thank you for your review. We added an ablation against dtANS and evaluated MINE on a larger model. All numbers are updated (maximum speedups now 2.53× / 1.59×).
Q1 (insights). Over dtANS, MINE adds the shared-memory chunk buffer, the Pair / Quad modes, the dense / sparse split of the values by entropy (sparse escape records), the
E5M2payload for EntQuant, side streams that free the table size, and an auto-tuner that builds each unit's speed–size Pareto front from a compressed-size prediction (Section 4). ZipServ is limited to BF16 and assumes a particular distribution of values; MINE uses generic entropy coding on all 19 checkpoints and is faster than ZipServ on BF16 (Table 1).Q2 (ablations). Appendix C (Table 4) now ablates from a faithful dtANS implementation (D0) to MINE (D5) on five checkpoints and both GPUs: asynchronous buffering and sparse escape handling in D2 and the simplified decoder state in D3 (stored instead of decremented base). Column slicing (D1) gives the largest gain, D2 helps all checkpoints except BF16, and D3 gives a small gain.
Q3 (larger models). Appendix F adds Qwen3.8-27B: on the RTX 5090, MINE is 1.87× faster than the fastest baseline on EntQuant 2.1 bit (1.02× on FP8), and on the RTX 4060 Ti only MINE fits the model. Since we report steady-state per-token rates of decode and prefill (Section 6), the time of any mix of input and output lengths can be computed from them without further experiments. We chose Qwen3.8-27B over 32B / 70B models because it is the largest model whose FP8 checkpoint still fits our 32 GB GPU, so the speedup over an uncompressed baseline can be measured; for a 32B or 70B model, no baseline fits.