acceptodds
Under review as a conference paper at ICLR 2027

MINE: Entropy-coded Memory-efficient Inference for Large Language Models

Abstract

We present MINE (Memory-efficient INference Engine), a lossless entropy coding approach that accelerates single-stream inference of Large Language Models (LLMs), where existing lossless compression methods yield speedups only for specific data types such as BF16 and under additional assumptions. Single-stream decoding is strongly memory-bound, since every weight must be streamed from memory once per generated token and requests cannot be batched in local or low-latency deployments. At the core of MINE is dtANS-LLM, a GPU entropy coding scheme adapted from the prior dtANS to the needs of LLM inference. It exploits the non-uniform distribution of weights, which persists even after quantization, and exposes hyper-parameters that trade compression ratio against decoding cost. We auto-tune these parameters for each tensor and GPU to obtain the Pareto front between model size and decoding latency. MINE integrates into vLLM as a plugin and selects the best kernel configuration for a given memory budget. On Llama 3.1 8B Instruct with a 2.1-bit EntQuant checkpoint, MINE achieves speedups of 2.52 and 1.48 over the fastest baseline on an RTX 4060 Ti and an RTX 5090, at effective bitrates of 2.24 and 3.43 bits per weight. MINE is available as open source.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Entropy coding, Asymmetric Numeral Systems (ANS), LLM inference, Quantization, Compression, Low-latency inference, Consumer GPUs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P7sVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a lossless entropy-coding framework, namely MINE, for memory-efficient LLM inference. MINE adapts existing GPU entropy coding techniques to enable fused decoding and matrix-vector multiplication, together with hardware-aware auto-tuning to optimize the memory–latency trade-off. Experiments on Llama 3.1 8B demonstrate up to 2.52× and 1.48× single-stream decoding speedups over the fastest baselines for the same checkpoint on RTX 4060 Ti and RTX 5090.

Critical strengths

The paper makes traditionally GPU-unfriendly entropy-coded weight representations more efficient for single-stream LLM inference through optimized GPU kernels. The reported performance improvements demonstrate the practical potential of this approach on the evaluated configurations.

Critical weaknesses
  1. The system novelty and algorithm novelty are limited. The fundamental idea of GPU-efficient entropy decoding fused with matrix-vector multiplication has already been explored by dtANS (IPDPS 2026). Furthermore, ZipServ (ASPLOS 2026) has demonstrated lossless compressed LLM inference through hardware-aware fused decompression and computation. While MINE introduces useful GPU-specific optimizations, its technical contributions appear largely incremental relative to these existing works.

2.Kernel-level ablations remain insufficient. The paper introduces several GPU optimizations but does not adequately quantify their individual contributions. In particular, I would expect comparisons with the original dtANS design and ablations of introduced components (asynchronous chunk buffering, sparse escape handling, and simplified decoder state management). Without these experiments, it is difficult to determine which optimizations are responsible for the reported improvements or what new GPU specific design insights can be generalized.

  1. The empirical evaluation was limited in scope. First, all experiments are conducted on a single 8B model. The evaluation also primarily relies on synthetic short-context decoding benchmarks rather than representative end-to-end inference workloads. Given the paper's positioning as a memory-efficient inference engine, I would expect larger-scale evaluations (e.g., 32B/70B) and experiments under realistic inference workloads.
Questions for authors
  1. What are the main technical insights introduced by MINE beyond dtANS and ZipServ?

  2. Could the authors provide controlled kernel-level ablations to quantify the contributions for each of these components: asynchronous buffering, sparse escape handling, and simplified decoder state management?

  3. Could the authors provide end-to-end evaluations on larger compressed models (e.g., 32B/70B) and real single-stream inference workloads with varying input and output lengths?

1 comment

Thank you for your review. We added an ablation against dtANS and evaluated MINE on a larger model. All numbers are updated (maximum speedups now 2.53× / 1.59×).

Q1 (insights). Over dtANS, MINE adds the shared-memory chunk buffer, the Pair / Quad modes, the dense / sparse split of the values by entropy (sparse escape records), the E5M2 payload for EntQuant, side streams that free the table size, and an auto-tuner that builds each unit's speed–size Pareto front from a compressed-size prediction (Section 4). ZipServ is limited to BF16 and assumes a particular distribution of values; MINE uses generic entropy coding on all 19 checkpoints and is faster than ZipServ on BF16 (Table 1).

Q2 (ablations). Appendix C (Table 4) now ablates from a faithful dtANS implementation (D0) to MINE (D5) on five checkpoints and both GPUs: asynchronous buffering and sparse escape handling in D2 and the simplified decoder state in D3 (stored instead of decremented base). Column slicing (D1) gives the largest gain, D2 helps all checkpoints except BF16, and D3 gives a small gain.

Q3 (larger models). Appendix F adds Qwen3.8-27B: on the RTX 5090, MINE is 1.87× faster than the fastest baseline on EntQuant 2.1 bit (1.02× on FP8), and on the RTX 4060 Ti only MINE fits the model. Since we report steady-state per-token rates of decode and prefill (Section 6), the time of any mix of input and output lengths can be computed from them without further experiments. We chose Qwen3.8-27B over 32B / 70B models because it is the largest model whose FP8 checkpoint still fits our 32 GB GPU, so the speedup over an uncompressed baseline can be measured; for a 32B or 70B model, no baseline fits.

Reviewer_pA9rOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

S1. Clear speedups where quantization quality is not a confound. BF16 on the RTX 5090: 130.3 tok/s versus 100.6 for vLLM (1.30×). FP8 on the RTX 4060 Ti: 37.4 versus 32.4 (1.15×) at 6.86 bpw (Figure 3). On the 16 GB card, MINE fits BF16 at 10.70 bpw (25.6 tok/s), where vLLM does not fit. Why this matters to my assessment: fused entropy decoding can pay off at batch 1.

S2. Transparent, reproducible reporting. Appendix A maps every number of §6 to a script (Table 1), pins versions, and lists every change to third-party code, including a fix that makes ZipServ compute at batch 1 (l. 966-972). The authors disclose unfavorable facts themselves (l. 404-405; prefill at 94% of vLLM, l. 413-414). Why this matters to my assessment: the numbers are checkable, and my concerns rest on the paper's own disclosures.

Critical strengths

S1. Clear speedups where quantization quality is not a confound. BF16 on the RTX 5090: 130.3 tok/s versus 100.6 for vLLM (1.30×). FP8 on the RTX 4060 Ti: 37.4 versus 32.4 (1.15×) at 6.86 bpw (Figure 3). On the 16 GB card, MINE fits BF16 at 10.70 bpw (25.6 tok/s), where vLLM does not fit. Why this matters to my assessment: fused entropy decoding can pay off at batch 1.

S2. Transparent, reproducible reporting. Appendix A maps every number of §6 to a script (Table 1), pins versions, and lists every change to third-party code, including a fix that makes ZipServ compute at batch 1 (l. 966-972). The authors disclose unfavorable facts themselves (l. 404-405; prefill at 94% of vLLM, l. 413-414). Why this matters to my assessment: the numbers are checkable, and my concerns rest on the paper's own disclosures.

Critical weaknesses

W1. Lack of novelty, the method is given only in prose, and the increment over dtANS is not measured.

  • The authors write "We adapt an existing method (Schätzle et al., 2026)" (§1, l. 074) and cite it in related work as "a fast fused entropy decoding method" for "another workload" (§5, l. 336-337).
  • BF16 handling follows "the established idea of only encoding the exponent" (Hershcovitch et al., 2025; §4.1, l. 211-212); budget selection is a standard multiple-choice knapsack (Kellerer et al., 2004; §2, l. 116-117); in-kernel shared-memory decompression is cited as existing (nvCOMPDx, §5, l. 336) without a comparison.
  • The changes to dtANS (§4.1) are justified qualitatively: "suffers from high latency" (l. 194), "reduces overhead drastically" (l. 243). dtANS is not a baseline in Figure 3, and no per-change ablation is reported.
  • The method is specified only in prose. The 19-page paper contains no numbered equation; §4 (l. 179-340, about 2,300 words) states the coding constraints only inline (M≥tkM \ge t^{k}, t2≤Mt^{2} \le M). The compressed-size predictor that drives tuning is described as "closely predicted by the cross-entropy between the distribution and its approximation" (l. 267-271), without defining either distribution or the estimate. The speed-size tuning objective (l. 259-261) is given in words, and the tuning of the remaining kernels is "omitted for brevity" (l. 265).

Why this matters to my assessment: the contribution is the engineering increment over dtANS; without a formal specification it cannot be checked, and no change is measured.

W2. The headline speedup is a maximum, and the remaining claims lack isolating evidence.

  • 2.52× / 1.48× is the "maximum speedup" (§6, l. 392), on one EntQuant 2.1 checkpoint whose baseline is the same checkpoint in native E4M3 (8.00 bpw); the abstract reports only this. Computing from Figure 3, the median gain over the fastest baseline (17 checkpoints) is 1.11× (4060 Ti) and 1.06× (5090).
  • On the RTX 5090, "For many checkpoints, MINE never chooses its own kernels"; speedups come "exclusively from combining native/Marlin/Humming kernels and compressing the lm head" (l. 404-405). Since bpw counts decoder linears only (Appendix A, l. 936-937), I infer 12 such checkpoints, gaining 3.4%-8.2%. vLLM builds also differ between MINE and baselines (l. 902-905).
  • Serving under a memory budget is claimed (abstract, l. 022-023; l. 085-087; §7, l. 420-421), but Table 1 marks it "optional, no reported result" and "The evaluation serves every checkpoint with its fastest tuned selection" (l. 892).
  • MINE uses FP16 activations for quantized checkpoints where vLLM uses BF16 (l. 186-187) and accumulates in FP16 (l. 250-251). ZipServ is gated by perplexity (l. 938-943); MINE has no reported output check.

Why this matters to my assessment: typical gains are small, on the 5090 partly not from entropy coding, and the budget tradeoff and MINE's output correctness are unreported.

Questions for authors
  1. What share of launches or bytes do MINE's own kernels serve, per checkpoint and GPU? Please give throughput for (i) vLLM kernels only, (ii) plus lm_head compression, (iii) full MINE, on one vLLM build.
  2. How does each dtANS-LLM change compare with the original dtANS? Cold-cache mvm latency and bitrate on BF16, FP8 and integer launches would suffice.
  3. Is MINE's output equivalent to the corresponding vLLM path? Please report perplexity difference and greedy-token agreement, as for ZipServ.
  4. Does the budgeted selection deliver the predicted tradeoff? Please report predicted and measured throughput at several budgets for a few checkpoints.
  5. How does the §7 premise hold on the RTX 5090? EntQuant 2.1 (3.43 bpw) reaches 242.5 tok/s, below MINE-served RTN, GPTQ and AutoRound 2-bit rows (319.3-329.4), unlike "inference at equivalent speeds" (l. 427). Please report perplexity, and what the metric adds over EntQuant, an "entropy-regularized E4M3 quantizer" (Appendix A, l. 917).
  6. Please state the encoding, the compressed-size predictor and the tuning objective formally: the distribution and its approximation in l. 267-271, the estimate computed from them, and the objective and constraints of the speed-size search.
1 comment

Thank you for your very thorough review. In our revised paper, we state the method more formally, isolate where the speedup comes from, and validate MINE's decoded outputs. The former Figure 3 is now in Appendix D; all numbers are updated (maximum speedups 2.53× / 1.59×).

W1.1 (novelty). Our contributions over dtANS are: the shared-memory chunk buffer, the Pair / Quad modes, side streams that free the table size from 4096 entries, and an efficient construction of the Pareto front from the compressed-size prediction (Section 4). Beyond dtANS, two parts are new in general: the dense / sparse split of the values by entropy (sparse escape records) and the E5M2 payload for EntQuant. To our knowledge, nvCOMPDx has not been used for LLM inference, so there is no engine to compare against.

W1.2 (dtANS ablation). See Q2 (Appendix C, Table 4).

W1.3 (formalization). Sections 2 and 4.2 now state the budgeted selection and the size predictor formally (Equations 2 and 3); see Q6. The remaining kernels (dequant, encode, decode) are not on the single-stream decode path, so only their launch geometry is tuned (Section 4.2).

W2.1 (maximum speedup). The speedup is bounded by the compression a checkpoint allows (Equation 1), and most checkpoints have little entropy headroom, so a median over checkpoints mostly reflects the checkpoints rather than MINE. Table 1 now gives each checkpoint's maximum compression / speedup next to MINE's achieved ones, and Figure 3 (Equation 4) shows how the speedup follows the achieved compression.

W2.2 (RTX 5090 gains). These checkpoints have little headroom (maximum speedup 1.09–1.27×, Table 1), so on the RTX 5090 baseline kernels are often faster; Appendix B (Tables 2, 3) shows per checkpoint which launches use MINE's kernels and what the lm_head contributes. MINE restricted to baseline kernels on its source build of vLLM 0.27.1 is marginally slower than the baseline on the official wheel (Table 2), so the build does not explain the speedup.

W2.3 (memory budget). Appendix A (Figure 4) now reports the served throughput under up to five budgets per checkpoint and GPU; see Q4. The reproduction table in Appendix G (Table 7) now lists this experiment (run/14_budget.sh) instead of marking it optional.

W2.4 (output check). See Q3 (Appendix E, Table 5: all checkpoints, both GPUs).

Q1 (kernel shares). Appendix B: Table 3 gives per checkpoint and GPU the share of launches of each kind served by MINE's kernels and the share of the achievable traffic savings it exploits. Table 2 gives the throughput of the baseline and of MINE with baseline kernels only, without lm_head compression, and in full, all MINE columns on one vLLM build.

Q2 (dtANS ablation). Appendix C (Table 4) ablates from a faithful dtANS implementation (D0) to MINE (D5) on multiple checkpoints, reported as decode throughput and bits per weight (with and without baseline fallback) rather than per-launch latency. Column slicing (D1) gives the largest gain, the shared-memory buffer (D2) helps all checkpoints except BF16, and storing the actual base (D3) gives a small gain.

Q3 (output equivalence). Yes. Appendix E (Table 5) compares MINE and every baseline with an FP32 CPU reference (top-1 agreement with the reference's greedy tokens, KL divergence) and reports WikiText-2 perplexity through each engine's decode path. MINE is of the same order as the baselines on all checkpoints (e.g., BF16 on the RTX 5090: 99.9% vs. 100.0% top-1, perplexity 7.37 for both).

Q4 (budget tradeoff). Yes. Appendix A: across up to five budgets per checkpoint, the measured throughput follows the tuned latency-based prediction within 0.6 / 2.3% (median 0.1 / 0.4%) on the RTX 4060 Ti / RTX 5090.

Q5 (EntQuant premise). On the RTX 5090 the premise holds only partly: MINE serves EntQuant 2.1 bit at 259.8 tok/s versus 310–323 tok/s for the 2-bit RTN, GPTQ and AutoRound checkpoints (Appendix D), as MINE's constant overhead weighs more on the faster GPU (Equation 4); on the RTX 4060 Ti it is on par (82.0 vs. 86.6–90.6 tok/s). We now restrict the claim of comparable speed (Section 7) to the RTX 4060 Ti. Through MINE's decode path, EntQuant 2.1 bit reaches perplexity 26.22 on both GPUs, versus 26.15–26.24 for vLLM and Humming (Table 5). EntQuant minimizes entropy for storage; its own stack decodes slower than the uncompressed model (0.44–0.86×, Table 1). With MINE, entropy also determines decode speed; Section 7 now says our work makes entropy a metric for the speed of quantization methods, rather than motivating a new metric.

Q6 (formal definitions). Section 4.2 now defines the distribution P of the (tuples of) weights, its approximation Q, and the predicted size σ_min (Equation 3), extended by the escape records for FP8 and BF16. Section 2 states the speed–size search as a multiple-choice knapsack (Equation 2).

Reviewer_3dYuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Memory-efficient INference Engine (MINE), a lossless entropy coding approach that accelerates single-stream inference of LLMs, a strongly memory-bound setting where existing lossless compression methods yield speedups only for specific data types such as BF16 and under additional assumptions. Its core is dtANS-LLM, a GPU entropy coding scheme adapted from the prior dtANS that exploits the non-uniform distribution of weights. MINE integrates into vLLM as a plugin that selects the best kernel configuration for a given memory budget. Validation is on Llama 3.1 8B Instruct across 19 checkpoints and two GPUs.

Critical strengths
  1. This paper claims that MINE is much faster than other entropy coding baselines.
  2. The authors integrate dtANS-LLM into vLLM as a plugin that picks the fastest codings under a memory budget from an auto-tuned speed–size Pareto front.
Critical weaknesses
  1. All results use a single old model on two consumer GPUs. The paper motivates MINE by fitting larger models into less RAM, but no larger model, other model family, or datacenter GPU is evaluated. It would be good to have all those aspects addressed.

  2. The decode throughput versus memory budget along the Pareto front is not provided.

  3. The paper suggests that for many checkpoints MINE never chooses its own kernels on the RTX 5090 and speedups come exclusively from native/Marlin/Humming kernels plus compressing the lm_head. For each checkpoint, it would be good to know the fraction of launches served by MINE's mvm versus fallback kernels, and the speedup attributable to each.

  4. dtANS-LLM introduces several changes (shared memory chunk buffer, sparse escape correction stream, companion-stream innermost loop, stored base in merged tables, fused mvm epilogue), but none is ablated individually against the original dtANS. It would be good to see an ablation study. Which dtANS-LLM modifications contribute most to the mvm speedup over the original dtANS design?

  5. No results are provided for larger batches and no information on compression ratios.

  6. On the RTX 5090, which launches and checkpoints actually use MINE's mvm, and what share of the reported speedup comes from compressing the lm_head alone?

  7. DFloat11, NeuZip, and EntQuant (nvCOMP) run under HF Transformers stacks while MINE runs in vLLM 0.27.1. How much of the "at least 3.52× slower" gap is engine overhead rather than decoding cost?

Questions for authors

See weaknesses

2 comments

Thank you for your detailed review. We added the speed–size fronts under a memory budget, a breakdown of where the speedup comes from, an ablation against dtANS, and results on a second, larger model in our revised paper.

W1 (model scope). We added Qwen3.8-27B, a larger model of another family (Appendix F): on the RTX 5090, MINE is 1.87× faster than the fastest baseline on EntQuant 2.1 bit, and on the RTX 4060 Ti only MINE fits it. We chose Llama 3.1 8B because single-stream speed depends on the tensor shapes rather than on the model's quality, and because most related works report on it. Datacenter GPUs are out of scope: they offer fewer ALU operations per bit of bandwidth, leaving too little compute for fast decoding, and single-stream decoding is rarely done on them.

W2 (budget front). Appendix A (Figure 4) now shows the decode throughput along the speed–size Pareto front of every checkpoint on both GPUs, under up to five budgets each.

W3 (kernel shares). Appendix B now gives, per checkpoint and GPU, the share of launches of each kind served by MINE's kernels (Table 3) and the throughput with baseline kernels only, without lm_head compression, and with full MINE (Table 2). MINE's own kernels add at least 12% on EntQuant (both GPUs), BF16 (RTX 5090) and FP8 (RTX 4060 Ti).

W4 (ablation). Appendix C (Table 4) now ablates the steps from a faithful dtANS implementation (D0) to MINE (D5) on five checkpoints and both GPUs. Slicing the columns to tuned widths (D1) gives the largest gain, followed by the shared-memory buffer (D2) on all checkpoints except BF16. Per change: the chunk buffer and the sparse escapes enter in D2, the stored base in D3, the fused epilogue in D3e, and the companion-stream inner loop with MINE's own kernel in D4.

W5 (batches, compression). Table 1 now states the achieved compression ratio of every engine and checkpoint next to the maximum. Larger batches are out of scope: MINE targets batch size 1, the most memory-bound setting, and serves larger batches with dequant and a GEMM (Prefill paragraph, Section 6).

W6 (RTX 5090 kernels). See W3: Table 3 lists per checkpoint which launches use MINE's kernels on the RTX 5090, and Table 2 shows that compressing the lm_head adds 3–11% throughput.

W7 (engine overhead). Table 1 now also gives each related work's speedup over its own stack with compression disabled ("own"). Even there, DFloat11, NeuZip and EntQuant stay below 1× (0.17–0.86×), so decoding makes them slower. The remaining engine overhead is the ratio of the two rows: our baseline is 1.1–1.25× faster than their own uncompressed stacks, and 2.4–2.5× faster than EntQuant's on the RTX 5090.

Dear authors

Thank you for the detailed rebuttal and additional experiments. The new results address several of my concerns.

After careful consideration, I am inclined to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.