acceptodds
Under review as a conference paper at ICLR 2027

HEPH: 3-Bit Grid-Aligned Trellis Quantization of MXFP4 Mixture-of-Experts for Accurate, Hadamard-Free Serving

Abstract

A growing class of mixture-of-experts models, including gpt-oss, Kimi K3, and DeepSeek-V4, ships routed experts natively in the MXFP4 hardware microscaling format after quantization-aware post-training. HEPH is a trellis-coded codec for these weights. Existing methods such as QTIP and QuIP# achieve strong rate–distortion performance but decode to arbitrary FP16 values in a randomized-Hadamard-rotated basis. Serving them requires a Hadamard transform on every forward pass and cannot use the native FP4 tensor-core kernel. HEPH instead targets the hardware grid: it reuses each checkpoint block’s UE8M0 exponent as its exact scale and restricts the trellis codebook to the 15 distinct E2M1 values, making every decoded weight a valid MXFP4 element. This design improves both accuracy and serving speed. On gpt-oss-20b, HEPH achieves **80.03** mean accuracy across five zero-shot benchmarks at 3.04 bits per weight, versus **77.72** for a rate-matched QTIP baseline and within 0.76 points of the 4.25-bpw MXFP4 checkpoint. When both compressed streams remain resident and decode weights in-kernel on each forward pass, HEPH reaches **96.1** tokens/s for batch-1 decode on a Grace Blackwell GPU in SGLang, versus **74.2** for QTIP (); it remains – faster across batches of 1–32. HEPH also supports a second serving mode: its approximately 3-bpw storage representation can be expanded into a resident 4.25-bpw MXFP4 tensor and executed by the unmodified production fused grouped-MoE kernel. This trades resident memory for throughput while retaining the storage saving. In this mode, HEPH reaches **382** tokens/s at batch 1, versus **72.8** for QTIP (), with speedups of – across batches of 1–32. Aligning the codec with the native FP4 grid thus yields both higher accuracy at a matched compression rate and faster mixture-of-experts serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
weight quantization, post-training quantization, trellis-coded quantization, mixture-of-experts, MXFP4, microscaling formats, LLM serving, inference throughput, hardware-aligned quantization, Blackwell GPU

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hr3xOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a dedicated trellis-based weight quantization method tailored for MXFP4 checkpoints. A specific form of grid (MagStrat with exponent reuse) is chosen that can be mapped exactly to the MXFP4 format. The optimal weight encoding is found via the Hessian-aware Viterbi algorithm. The efficacy of the proposed method is validated on the gpt-oss-20b model on several tasks from the lm-eval harness.

Critical strengths
  • The paper acknowledges the problem of quantization of already quantized models and the need to develop a dedicated method and weight representation that conforms with the base model precision (FP4 or some other low-precision format). The proposed HEPH grid addresses this problem by proposing a trellis encoding tailored for MXFP4 model quantization that, on the one hand, preserves accuracy as much as possible and, on the other hand, is efficient to decode and allows for large-batch serving via dequantizing the weights back to the original precision.
  • HEPH provides efficient inference both in low-batch and high-batch serving scenarios.
Critical weaknesses
  • The evaluation is limited—it involves only a single model (gpt-oss-20b) and a limited set of benchmarks (that can be considered outdated as of 2026). To fully appreciate the efficacy of the method, it would be great to consider more models published in MXFP4 precision (DeepSeek-V4, Kimi-K2 / 3). In case it is challenging due to compute constraints, one can consider public MXFP4 quants for Qwen-3.x models released on Hugging Face and measure performance relative to them. In addition, it would be insightful to measure the quality on reasoning benchmarks (GPQA-Diamond, AIME, MATH-500, LiveCodeBench) or agentic benchmarks, as they are more representative measures of the model capabilities.
  • The claim that Hadamard transforms incur significant overhead during inference is not accurate. One can fuse small rotations [1] into the dequantization + GEMM kernel without noticeable inference slowdown.

[1] Egiazarian, V., et al. (2025). Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization. arXiv:2509.23202. Google Scholar.

Questions for authors
  • In the comparison with QTIP [2], does one project the QTIP-quantized weights onto the MXFP4 grid or perform BF16 quantization? In case it is mapped to the MXFP4 format, is the reason for HEPH providing higher accuracy due to the grid being better suited for the MXFP4 base checkpoint or due to the additional error from QTIP conversion to the MXFP4 format?
  • Does the proposed method work for NVFP4-quantized models?
  • The 3-bit QTIP models for Llama-3 (even 8B) are almost lossless compared to the BF16 checkpoint, whereas the results in this work show a significant accuracy decline for gpt-oss-20b. Do you have an intuition about why this could be the case?

[2] Tseng, A., Sun, Q., Hou, D., & De Sa, C. (2024). QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024. Google Scholar.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hr3x: gpt-oss-120b and reasoning benchmarks, format vs. rotation cost, QTIP scoring, NVFP4

Thank you for the expert review, for recognizing the need for a dedicated representation that conforms to the base model's precision, and for the pointer to MR-GPTQ. Changes are marked in blue in the revised PDF.

W1 One model, older benchmarks. The revision adds gpt-oss-120b (36 layers × 128 experts) and reasoning benchmarks you suggested (GPQA-Diamond, AIME 2024+2025), plus GSM8K. On gpt-oss-120b, HEPH is within 0.8 points of the source on the five zero-shot tasks (78.37 vs 79.16; calibration text doubled because each of its 128 experts receives 4× fewer routed tokens), 96.20 vs 96.25 on GSM8K and 71.40 vs 70.52 on GPQA-Diamond. On gpt-oss-20b it leads QTIP on GSM8K (+1.10 [0.25, 2.00]) and AIME 2024+2025 (+8.75 [3.75, 14.17]). DeepSeek-V4 releases its routed experts in exactly the same E2M1 × UE8M0/32 format. We measured the released DeepSeek-V4-Flash experts (atom entropy 3.77 bits, exponent-delta entropy 0.047 bpw, both matching gpt-oss), so the codec and its 3.07-bpw rate carry over unchanged.

W2 Hadamard overhead. We agree that small rotations can be fused into the quantization and GEMM kernels at little cost, as MR-GPTQ shows, and we removed the claims that attribute serving speed to avoiding the Hadamard. However, the obstacle HEPH removes is the format, not the arithmetic. MR-GPTQ quantizes a high-precision model into a rotated MXFP4 basis. For a model released only in MXFP4, a rotated reconstruction is no longer in the checkpoint's own E2M1 × UE8M0 set (Proposition 1), so it needs either its own kernels or a second rounding back onto the grid. We measured that second rounding for QTIP: it costs a further 1.5 points (77.72 → 76.20). HEPH reconstructs inside the checkpoint's grid. This is what lets it run on the unmodified vendor kernel in two modes (§4.3): expanded at load, at native speed, or decoded on demand from the compressed stream, which uses 20% less GPU memory and gives 2.7× the KV-cache capacity with 16 GiB of GPU memory, the minimum OpenAI states for gpt-oss-20b. The related-work and introduction text now make this argument.

Q1 How QTIP is scored. QTIP is evaluated with its own BF16 reconstruction (rotation undone), not projected onto MXFP4, which favors QTIP. So the 2.3-point gap is not conversion error. Projecting QTIP onto MXFP4, which it needs in order to run on the FP4 kernel, widens the gap to 3.8 points (Table 1).

Q2 NVFP4. Yes. The construction needs only E2M1 elements and a stored block scale: NVFP4 weights divided by their E4M3 block scale are exact atoms, and the scale bytes are reused and entropy-coded. The scale stream is larger (one scale per 16 weights), so the rate overhead is higher than for MXFP4.

Q3 Why 3-bit QTIP is near-lossless on Llama-3 but not here. The source is different. A BF16 Llama-3 checkpoint carries far more precision than its accuracy needs, so a 3-bit code mostly removes noise. gpt-oss's experts were trained (quantization-aware) to live on the MXFP4 grid, and their E2M1 atoms carry 3.74 bits of empirical entropy per weight. Even lossless coding needs ≈3.78 bpw with the exponents, so any 3-bit code must discard about 0.7 bits of information per weight that training placed there. Rotating this discrete source into a continuous one discards its structure; HEPH keeps it, which we believe is why it degrades less than QTIP, most visibly on AIME (§4.2).

Thank you for the response. My concerns and questions were mostly resolved. After reading the answers addressed to me and other reviewers I decide to raise the score.

Reviewer_QRZhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HEPH compresses MXFP4 routed-expert weights using a trellis with E2M1 codewords, retaining the native per-32-weight UE8M0 scales. It uses a magnitude-stratified codebook, sensitivity-weighted Viterbi encoding, and approximate tail-biting. On gpt-oss-20b, the five-task zero-shot mean is 80.03, versus 80.79 for the source checkpoint and 77.72 for QTIP. Serving tests cover both decoding compressed weights each forward pass and expanding them to resident MXFP4 for the fused MoE kernel.

Critical strengths
  1. E2M1 codewords with native scales give a valid MXFP4 reconstruction, allowing the existing fused serving path after expansion. This addresses a concrete constraint that off-grid quantizers face.

  2. Tables 3 and 4 distinguish compressed-resident and expanded-resident serving. Appendix A includes correctness checks and a ported multi-batch QTIP kernel. The paper also acknowledges that HEPH's isolated operator is slower than QTIP's, which helps interpret the whole-model speedups.

Critical weaknesses
  1. The described format gives 3.25 bpw, not 3.04. Each 64-weight block stores 64 three-bit labels and two eight-bit scales: (64*3 + 2*8)/64 = 3.25 bpw before other overhead (Figure 2; Sections 3.2-3.3). Reusing scales does not remove their storage cost; the two-bit variant similarly needs at least 2.25 bpw. The MX specification, Table 1, confirms the scale overhead. Unless further scale compression or a different denominator is intended, the comparisons with roughly 3.03-bpw QTIP are not rate-matched, affecting the equal-rate and equal-memory claims in Tables 1-3.

  2. Equation (4) appears to omit the squared scale. It defines h_i as a diagonal entry of the original activation second moment but uses h_i (c(s_i)-z_i)^2, with z_i=w_i/s_g. The corresponding diagonal output-error proxy is h_i s_g^2 (c(s_i)-z_i)^2. Since one trellis spans two independently scaled groups, this factor can change the selected path. It may already be absorbed in the implementation, but the paper needs to say so. Figure 3 measures grid membership and exact reconstruction, not the benefit of sensitivity weighting, leaving the accuracy attribution unclear.

Questions for authors
  1. What byte count and denominator produce 3.04 bpw, including labels, UE8M0 scales, metadata, and any scale compression? What resident weight bytes underlie Table 3's equal-memory comparison, and how would a 3.25-bpw rate change the conclusions?

  2. Does Equation (4)'s implementation include squared native scales, directly or through scale-adjusted activations? Please state the objective used for Tables 1 and 2 and how it handles the two scales within a trellis block.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QRZh: exact rate (3.07 bpw), resident bytes, and the objective of Eq. (4)

Thank you for checking the rate arithmetic and the objective, and for noting that the E2M1 reconstruction lets HEPH use the existing fused serving path. Both points made the paper more precise. Changes are marked in blue in the revised PDF.

W1 / Q1 3.25 vs 3.04 bpw. You are right that reusing the scale does not make it free to store, and our wording suggested it did. However, HEPH does not store raw UE8M0 bytes. Along the input dimension the exponents change slowly (64.6% of adjacent deltas are 0, 99.2% are in {−1, 0, 1}), so each row stores its first byte and the deltas with a canonical Huffman code. Measured over all 2,304 expert matrices: labels 3.000 + exponent code 0.051 + per-row bit offsets for random access 0.022 = 3.073 bpw; 3.04 was the entropy of the exponent deltas. QTIP is 3.022 bpw including its own metadata, so the comparison is rate-matched to within 0.05 bpw (1.7%), and the K = 2 variant is 2.07 bpw, not 2.25. All tables and Figure 2 now say 3.07 / 2.07, and §3.2 describes the exponent code.

Resident bytes of Table 3. That table compared the two in-kernel decoders at matched coding rate (3.07 vs 3.02 bpw). Both integrations attached their stream to the stock loader without releasing its MXFP4 buffers, so the process footprint (24.65 / 23.58 GiB) did not reflect the stream; the table measured decode cost, not memory. To give the equal-memory comparison you asked for, the revision adds a decode-on-demand mode that releases them (§4.3, Table 5). The GPU holds 11.32 GiB in total, of which 6.83 GiB is the HEPH stream and 0.43 GiB one staging buffer shared by all layers, against 14.23 GiB for native MXFP4. Before each MoE layer the experts selected by the router are decoded into the buffer, and the unmodified vendor kernel runs at 285 / 1502 / 3567 tok/s for batch 1 / 8 / 32. With 16 GiB of GPU memory, the minimum OpenAI states for gpt-oss-20b ("as little as 16GB memory"), this leaves 2.7× the KV-cache capacity of the native model (112K vs 41K tokens).

At 3.25 bpw the accuracy would be unchanged, because the weights are the same. Keeping the raw exponent bytes resident is in fact a useful speed/memory trade-off (3.27 bpw with the kernel's layout padding): 11.90 GiB and 325 / 1643 / 5133 tok/s.

W2 / Q2 Eq. (4) and the squared scale. The implementation is Eq. (4) as printed: the trellis encodes zi=wi/sgz_i = w_i / s_g with branch cost hi (c(si)−zi)2h_i\,(c(s_i) - z_i)^2, so the two 32-weight halves of a block are weighted equally in the z-domain. The output-error proxy you derive, hi sg2 (c(si)−zi)2h_i\,s_g^2\,(c(s_i) - z_i)^2, differs on the 35% of blocks whose two groups have different exponents, so we re-encoded all 768 experts with it. It is 0.74 [0.43, 1.05] points worse on the five-task mean. The printed objective is therefore also the better one; §3.3 now states it explicitly and Table 3 reports both.

Attribution. The new ablations (Table 3) separate the components. Grid alignment carries most of the gain: the same trellis with re-derived RMS scales and a Gaussian codebook is worse even at a higher rate (3.25 bpw), and against the stratified codebook, uniformly drawn atoms cost 0.90 points and atoms drawn from the source distribution 1.60. Sensitivity weighting adds 1.07 [0.68, 1.47] points on top with our calibration set (80.03 vs 78.96 for h = 1). With a different WikiText-2 calibration set HEPH still leads QTIP by 1.29 [0.87, 1.73], and the calibration-free variant already leads it by 1.24 [0.77, 1.70].

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

The author has addressed my concerns and I will raise my score to 3.

Reviewer_4Dd1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compresses the expert weights of MoE models that are released in MXFP4, such as gpt-oss, from 4.25 to about 3 bits per weight. Existing trellis methods such as QTIP produce weights that are not valid MXFP4 values, so they cannot use the fast MXFP4 kernels. The proposed method, HEPH, keeps the original MXFP4 scales and only allows the 15 values of the E2M1 format in its codebook, so every decoded weight is still a valid MXFP4 value. On gpt-oss-20b, HEPH reaches 80.03% average accuracy on five tasks, which is comparable to the 80.79% achieved by the original model and higher than the 77.72% of QTIP. It also runs 1.1-1.5x faster than QTIP when both keep 3-bit weights in memory.

Critical strengths
  1. The paper addresses a practical problem: mainstream GPUs do not support formats below 4 bits, so weights compressed below 4 bits must be decoded before computation, which adds significant overhead. Keeping every decoded weight a valid MXFP4 value is a reasonable way to make such weights compatible with existing 4-bit hardware.
  2. At about 3 bits per weight, HEPH achieves 80.03% average accuracy on gpt-oss-20b, higher than QTIP (77.72%) and QuIP# (75.25%) at a similar bit-width and close to the original MXFP4 model (80.79%). When both methods keep 3-bit weights in memory, it also decodes 1.1-1.5x faster than QTIP.
Critical weaknesses
  1. HEPH stores 3-bit trellis labels rather than FP4 codes, so it still has to decode the weights before computation. When it keeps 3-bit weights in memory, it runs about 4x slower than the original MXFP4 model (96.1 vs. 382 tok/s at batch 1); when it is fast, the weights are converted back to 4.25-bit MXFP4, so it simply matches the original model and saves no memory. The paper therefore does not show a setting where HEPH both saves memory and keeps the speed of native FP4 kernels.
  2. The accuracy results are limited. The paper tests only one model (gpt-oss-20b) and only five zero-shot benchmarks (ARC-Challenge, ARC-Easy, HellaSwag, PiQA, MMLU), with no generation tasks such as math or code. It also reports no variance and no ablation, so it is unclear whether the gains are reliable and which part of the method brings them.
Questions for authors
  1. What calibration data is used to compute the Hessian diagonal, and how many tokens does each expert receive? Since only routed tokens are used, rarely selected experts may get very few tokens. How long does encoding all 768 experts take?
  2. The presentation could be improved. The abstract is long and hard to follow, and the main text repeats the same results in several sections. A more focused abstract and structured main text would make the contribution clearer.
  3. The evaluation could be extended to more benchmarks and models. In particular, results on generation tasks and on other MXFP4 models would show whether the accuracy gains hold beyond the current setting.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4Dd1 (1/2): saving memory on the native FP4 kernel, and where it pays

Thank you for the detailed review and for recognizing that keeping every decoded weight a valid MXFP4 value is a reasonable way to make sub-4-bit weights compatible with existing 4-bit hardware. We answer in two parts: this part covers serving (W1); part 2/2 covers accuracy, calibration and presentation (W2, Q1–Q3). New results are marked in blue in the revised PDF.

W1 A setting that both saves memory and keeps the native FP4 kernel. We agree that the submission did not show one. However, this is precisely the setting that only an on-grid codec can offer, and the revision adds it (new §4.3, Table 5). Because every decoded HEPH weight is a valid MXFP4 element, the stream decodes into exactly the bytes that the unmodified vendor kernel reads. We now compare three serving modes of the same 3.07-bpw file:

  • Storage mode: the stream is expanded once at load; the GPU holds MXFP4 and runs at native speed.
  • Decode-on-demand (new): the GPU holds only the stream (6.83 GiB) plus one staging buffer laid out exactly like one layer's experts after the vendor loader (0.43 GiB, shared by all 24 layers). Before each MoE layer, one CUDA kernel decodes the experts selected by the router into that buffer, and the unmodified FlashInfer MXFP4 fused-MoE kernel runs on it. At load, every layer is decoded in full and compared byte for byte with the stock layout; generations are token-identical to storage mode.
  • In-kernel decode: the submission's custom kernel.

Setup: gpt-oss-20b, one GB200, SGLang TP1, CUDA graphs, 512-token prompt, median decode throughput. KV capacity: KV-cache tokens SGLang allocates, with the same server settings, when the server has 16 GiB of GPU memory in total.

modeGPU memory after loadKV tokens @ 16 GiBtok/s, batch 1 / 8 / 32
native MXFP4 (public checkpoint)14.23 GiB41K461 / 3044 / 9721
HEPH storage mode14.23 GiB41K464 / 2987 / 9627
HEPH decode-on-demand11.32 GiB112K (2.7×)285 / 1502 / 3567

At batch 1, decode-on-demand reaches 62% of native speed, against 25% for the submission's in-kernel decoder and 19% for the QTIP integration (each relative to native MXFP4 in its own software stack). Its decode kernel is a first version (44 µs per layer for the four experts of a batch-1 step).

Where each mode pays. Decoding is memory-bound, and on the devices gpt-oss-20b is built for, memory decides what can run at all.

  • 16 GiB of memory. OpenAI positions gpt-oss-20b "for lower latency, and local or specialized use cases" and states that it runs "on systems with as little as 16GB memory". With 16 GiB the native weights leave room for 41K KV-cache tokens, about one 32K-token reasoning trace (the budget of our AIME runs). HEPH leaves room for 112K: three such traces at once, or one context 2.7× longer. With 24 GiB the gain is 1.3×.
  • Larger MXFP4 MoEs. The 28% saving applies to every routed expert. Over all 13,824 expert matrices of gpt-oss-120b, the HEPH stream is 41.0 GiB instead of 56.7 GiB of MXFP4 experts; DeepSeek-V4 and Kimi K3 release their routed experts in the same format.
  • Memory-rich, large-batch serving. When nearly every expert is selected at every step, storage mode is the right choice: it runs at native speed and still cuts expert storage and download by 28%.

The operator picks the mode per deployment from the same file, and in both storage and decode-on-demand modes the compute runs on the unmodified vendor FP4 kernel.

AuthorsOpenReview ↗

Response to Reviewer 4Dd1 (2/2): reasoning benchmarks, gpt-oss-120b, variance and ablations, calibration

W2 One model, five zero-shot tasks, no generation tasks, no variance, no ablation. The revision adds all four.

Generation tasks (new Table 2). Pass@1 with reasoning effort medium, temperature 1 and a 32K-token budget, averaged over 4 samples per item (GSM8K) or 8 (GPQA-Diamond, AIME); 95% item-level paired-bootstrap intervals. All methods are served from BF16 weights here, because QTIP has no MXFP4 form:

gpt-oss-20bGSM8KGPQA-DiamondAIME 2024+2025
MXFP4 source (4.25 bpw)95.8067.0578.54
HEPH (3.07 bpw)95.3564.0874.17
QTIP 3inst (3.02 bpw)94.2563.3865.42
HEPH − QTIP+1.10 [0.25, 2.00]+0.69 [−2.27, 3.66]+8.75 [3.75, 14.17]

HEPH stays within 0.5 points of the source on GSM8K and leads QTIP on every task. The lead is largest on AIME, where long reasoning chains amplify per-token weight errors.

Second model (Tables 1–2): gpt-oss-120b, 36 layers × 128 experts = 4,608 experts. Each of its experts receives about 4× fewer routed calibration tokens than on gpt-oss-20b, so we doubled the calibration text (WikiText-2 + C4). HEPH scores 78.37 vs 79.16 for the source on the five zero-shot tasks (within 0.79 [0.36, 1.23]), 96.20 vs 96.25 on GSM8K and 71.40 vs 70.52 on GPQA-Diamond.

Variance. Every comparison now carries a 95% paired-bootstrap interval; HEPH − QTIP on the five tasks is +2.31 [1.86, 2.75]. We also re-encoded with a different WikiText-2 calibration set: HEPH still leads QTIP by 1.29 [0.87, 1.73]. Without any calibration data (h = 1), HEPH is deterministic and still +1.24 [0.77, 1.70] above QTIP, which itself needs routed Hessians.

Ablations (new Table 3; one design choice at a time, all 768 experts re-encoded at the same rate). Against the stratified codebook, uniformly drawn atoms cost 0.90 [0.44, 1.37] points and atoms drawn from the source distribution 1.60 [1.17, 2.02]. The squared-scale metric costs 0.74, a larger trellis state (L = 14) 0.74, and the block length changes the mean by at most 0.35. The grid-aligned codebook carries most of the gain.

Q1 Calibration data, tokens per expert, encoding time. 112 WikiText-2 windows of 2,048 tokens (229K tokens, 917K routed assignments per layer). Routed tokens per expert: median 14,846, maximum 209,582. h is estimated from the first 256 routed tokens of each expert: 707 of 768 experts reach 256, 24 receive fewer than 64, and the 2 that receive none use their layer's pooled samples. Rare experts are not a weak point of HEPH. Its targets are the checkpoint's exact E2M1 atoms and calibration only reweights them, which is why even the calibration-free variant above beats QTIP. Encoding all 768 experts takes 82 GPU-minutes on one GB200 (6.4 s per expert; layers are independent and parallelize across GPUs), against 10.2 GPU-hours for QTIP with its released code.

Q2 Presentation. We rewrote the abstract around the two results (accuracy at 3 bits, serving on the vendor kernel), made §1 and §5 state each result once, and merged the two serving tables and their repeated discussion into one table of serving modes (§4.3, Table 5).

Q3 More benchmarks and models. Covered under W2: three reasoning benchmarks and gpt-oss-120b. DeepSeek-V4 releases its routed experts in the same E2M1 × UE8M0/32 format. On the released DeepSeek-V4-Flash experts, the atom entropy (3.77 bits) and the exponent-delta entropy (0.047 bpw) match those of gpt-oss, so the codec and its 3.07-bpw rate carry over unchanged.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Thanks for the detailed response and additional experiments. Most of my concerns (W2, Q1, Q2, Q3) have been resolved. For W1, the method still cannot save memory and keep the native speed at the same time: decoding in advance brings the memory back to 4.25 bits, and decode-on-demand saves memory but is still 1.6-2.7× slower than the original model. So I do not see a clear benefit of using HEPH over simply serving the original MXFP4 model. Therefore, I raise my rating to 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision: memory-saving serving on the vendor FP4 kernel, reasoning benchmarks, gpt-oss-120b, ablations

We thank the reviewers for their careful reading. The revised PDF is uploaded, with all changes marked in blue. It adds what the reviews asked for:

  1. A serving mode that keeps the ~3-bit memory saving and still runs the native FP4 kernel (§4.3, Table 5). Every decoded HEPH weight is a valid MXFP4 element, so the stream decodes into exactly the bytes that the unmodified vendor fused-MoE kernel reads. In the new decode-on-demand mode the GPU keeps only the HEPH stream; before each MoE layer, the experts selected by the router are decoded into one staging buffer shared by all layers. gpt-oss-20b then occupies 11.32 GiB instead of 14.23 GiB. With 16 GiB of GPU memory, the minimum OpenAI states for this model ("as little as 16GB memory"), that is 2.7× the KV-cache capacity (112K vs 41K tokens), which decides how long a reasoning trace, or how many requests, the GPU can hold.
  2. Reasoning benchmarks (GSM8K, GPQA-Diamond, AIME 2024+2025) and a second model, gpt-oss-120b (§4.2, Tables 1–2). HEPH leads QTIP on all three tasks, significantly on GSM8K and AIME (+8.75 points on AIME). On gpt-oss-120b it stays within 0.8 points of the source on the five zero-shot tasks and matches it on GSM8K and GPQA-Diamond.
  3. Variance and ablations (§4.2, Table 3): paired-bootstrap intervals for every comparison, results with a second calibration set and without calibration, and one ablation per design choice.
  4. Exact rate accounting (§3.2–3.3): 3.07 bits per weight including the entropy-coded exponents, against 3.02 for QTIP.

The same 3.07-bpw file serves both regimes: expanded at load time when GPU memory is ample (native speed, 28% fewer expert bytes to store and download), decoded on demand when memory is the constraint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.