- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies hardware-aware weight-only quantization for LLM inference on ET-SoC-1, aiming to improve the Pareto frontier of accuracy, weight storage, and decoding throughput. The authors profile how quantization formats interact with the hardware, select viable formats, and assign them to tensors using an output-sensitive Fisher objective under storage and decode-time budgets. Activations remain unquantized, with FP32 operands on the decode path. Across two dense models and one MoE model, the method achieves about 1.07-1.72x the decode throughput of K-quant recipes with similar storage, generally with modest accuracy changes.
Critical strengths
- The detailed hardware measurements are the strongest part of the paper. The breakdown of weight fetching and dequantization, together with the controlled comparisons of format components, helps explain why some apparently attractive formats perform poorly on this accelerator. This is useful information for deploying quantized models on hardware without native support for the chosen low-bit formats.
- The evaluation includes both dense and MoE models, and the throughput gains over K-quant recipes are measured in complete model runs at similar storage. This makes the practical benefit clear. I see value in this engineering work, particularly for existing hardware whose numerical capabilities cannot be changed.
Critical weaknesses
- Hardware-aware precision allocation has been studied in HAWQ-V3 and MxMoE. This paper differs in its format choices and output-based Fisher objective, but I would like a clearer picture of the benefit of the allocation itself. Figure 4b and the Q4_0 results provide some evidence. However, Table 16 compares the other uniform alternatives only through predicted objectives and timings. Appendix C.6.2 also notes that at the K-quant budgets the storage-only program already gives the same allocation as the full program, so the speedup over K-quants comes from pruning the format set in Section 3.4, not from the decode-time constraint. Measured accuracy and throughput for one feasible alternative are needed to show how much is gained by mixed-format allocation.
- The performance evaluation is limited to ET-SoC-1, whose decode path converts weights to FP32. Section 3.2 explains why format rankings change with hardware support, and the authors describe how to repeat the procedure after re-profiling. However, there is no quantitative analysis or experiment showing whether mixed-format allocation would still improve on a suitable single-format baseline on another target. Even on ET-SoC-1, the kernels come from an in-house generation stack that the paper does not describe, and Q4_0 got extra tuning, so part of the format ranking may reflect kernel quality rather than the formats themselves. The engineering work is useful, but I am less convinced by the added value of the allocation method and the evidence beyond this hardware. I therefore lean toward weak rejection.
Questions for authors
- Could you report measured accuracy and end-to-end throughput for one of the uniform alternatives in Table 16 that fits both the storage and decode budgets? One representative model is enough.
- How well does the method generalize beyond ET-SoC-1? On an accelerator with native low-bit support, would mixed-format allocation still improve on a suitable single-format baseline at a comparable storage budget?
- For Llama 3.1 8B, the recipe optimized at Q4_0's budgets reaches 10.5 tok/s, versus Q4_0's 11.6 (Table 19). Can the mean task accuracy gain be retained after adjusting the allocation to reach 11.6 tok/s within Q4_0's storage budget?
4 comments
Response to Reviewer uRRz (Part 2)
W1: At the Q4_0 budget of each model, we quantized, evaluated, and timed the best uniform alternative of Table 20 (previously Table 16). It is the single format the allocator selects under the same objective and both budgets, so it isolates the gain from mixing formats:
| Model | Configuration | Storage (GB) | tok/s | WikiText-2 | C4 | Reas. |
|---|---|---|---|---|---|---|
| Qwen3.5-4B | Uniform Q4_1_FP8_G32 | 2.367 | 15.30 | 10.13 | 14.45 | 77.9 |
| Allocation | 2.688 | 13.79 | 9.83 | 14.11 | 81.2 | |
| Llama-3.1-8B | Uniform Q4_0 | 4.517 | 12.48 | 7.85 | 12.51 | 58.7 |
| Allocation | 4.990 | 10.91 | 7.42 | 11.77 | 59.4 | |
| Qwen3.5-35B-A3B | Uniform IQ4_NL | 19.53 | 14.50 | 6.63 | 10.94 | 89.3 |
| Allocation* | 20.035 | 13.50 | 6.64 | 10.79 | 88.6 |
*Re-solved at a corrected decode budget; the submitted allocation was solved with the budget set incorrectly (73.9 instead of 69.2 ms). The submitted allocation gives WikiText-2 6.62, C4 10.74, and reasoning 88.9.
The uniform format is less accurate than the allocation on both dense models. On Qwen3.5-35B-A3B, where the task benchmarks are saturated, the difference appears only in C4 perplexity. The uniform formats decode faster because uniform formats come in discrete widths, and the best one that fits leaves part of the budget unused (0.32 GB on Qwen3.5-4B). On Qwen3.5-35B-A3B, uniform IQ4_NL also stores the output head, the largest dense tensor read per token, in 4 bits, while the allocation is constrained to Q4_0's speed and keeps the head at Q5_0. A uniform choice that does decode at Q4_0's speed on Qwen3.5-4B is Q4_1_FP8_G32 with a Q8_0 embedding and head (14.10 tok/s). An allocation at the same storage and predicted speed scores 2.7 points higher in reasoning, while measuring 2.8% slower (13.70 tok/s).
At the K-quant budgets, the speedup comes from the pruned format set, not from the decode constraint. To test whether pruning costs accuracy, we re-solved Qwen3.5-4B with all 27 kernel-timed formats as candidates, including the K-quants, IQ4_XS, MXFP4 and NVFP4. At the Q4_0 budget, where the decode budget is active, 2.3% of the parameters change format and measured accuracy does not change. At the Q4_K_M budget, where it is not active, the allocator moves 80% of the parameters to IQ4_XS and K-quants and decodes 28% slower (9.66 vs 13.49 tok/s) with no measurable accuracy gain (WikiText-2 9.74 vs 9.76, reasoning 81.1 vs 81.0). Pruning therefore costs no measurable accuracy (Appendix C.6.6).
HAWQ-V3 and MxMoE cannot be applied as they are on this stack. HAWQ-V3 estimates each layer's Hessian trace with Hessian–vector products, which the fused gated-delta-rule kernels of Qwen3.5 do not support, and it chooses among bit-widths of one integer quantizer. MxMoE assigns a format to each expert, which llama.cpp cannot execute because all experts of a layer share one format, and its cost model describes GPU tensor cores. For our KL objective, HAWQ's sensitivity is the Fisher trace times the quantization error, which we can compute. With candidates, budget, and solver fixed (Qwen3.5-4B, Q3_K_M budget), we replaced our score with it and with simpler scores, and quantized and evaluated each allocation:
| Per-tensor score | WikiText-2 | C4 | Reas. |
|---|---|---|---|
| Ours | 10.28 | 14.59 | 77.7 |
| Fisher trace (HAWQ-V2/V3 sensitivity) | 10.37 | 14.68 | 76.7 |
| Weight MSE, relative to ‖W‖² | 10.85 | 15.40 | 79.1 |
| Output MSE (GPTQ layer loss) | 11.38 | 15.55 | 65.5 |
The Fisher trace ranks tensors almost as our score does and trails our results. Relative weight MSE has higher perplexity, and output MSE is 12 points lower in reasoning.
Response to Reviewer uRRz (Part 3)
W2: The reviewer rightly points out that the format rankings, the viable set, and the decode speeds are properties of ET-SoC-1 and its kernels, and the paper motivates its method from them and we do not claim that they carry over to other hardware (Section 1 and 3.4). What carries over is the procedure: measure the target's formats, keep those it executes efficiently, and allocate under its storage and decode budgets. Applying it to a second accelerator requires a kernel set for that accelerator, which is beyond the scope of this paper. Please see below for further details.
Part of the question can be answered without a second device. On hardware that consumes its formats natively (the first regime of Section 3.2), single-stream decode is bandwidth-bound, and its cost follows stored bits. A decode budget then coincides with a storage budget, and our program reduces to its storage-only form. The question becomes whether, at a fixed storage, mixing formats from a candidate set improves on the best single format from that set. We measured this on all three models:
| Model | Best uniform format | WikiText-2 | C4 | Reas. |
|---|---|---|---|---|
| Qwen3.5-4B | Q4_1_FP8_G32 | 10.13 / 9.81 | 14.45 / 13.99 | 77.9 / 81.3 |
| Llama-3.1-8B | IQ4_NL | 7.50 / 7.33 | 11.93 / 11.64 | 58.3 / 59.0 |
| Qwen3.5-35B-A3B | IQ4_NL | 6.63 / 6.66 | 10.94 / 10.74 | 89.3 / 88.7 |
Values are uniform / storage-only mixed assignment at equal storage.
The mixed assignment is more accurate on both dense models: 3.4 points of reasoning on Qwen3.5-4B and lower perplexity on Llama-3.1-8B. On Qwen3.5-35B-A3B, where the task benchmarks are saturated, it has lower C4 perplexity, while WikiText-2 and reasoning are within resolution. These assignments use no timing information, so this is the comparison a bandwidth-bound target would face for its own candidate set. The gain follows from where the model's sensitivity lies: at uniform Q4_0, the 10% most sensitive parameters carry 67%, 54% and 84% of the predicted loss on the three models (Appendix C.6.4), so a single format spends most of its bits where they matter little. Prior work reports gains from mixed precision on GPUs with native formats (HAWQ-V3, MxMoE), and llama.cpp's K-quant recipes are themselves mixed. Where the decode budget binds, as on ET-SoC-1, we fitted per-format corrections to the latency model from a few end-to-end timings (Appendix B7 in the revised manuscript) and re-solved; the allocations then decode within 3.0–3.6% of Q4_0 on all three models and keep lower perplexity than it.
On kernel quality, Section 3.4 now separates the findings that follow from the datapath from those that depend on the kernels: Fetch cost linear in stored width, per-element conversion for FP4/FP8 elements and per-element lookup for codebook formats (the vector units accept only FP32/INT32), and two-level scale decoding for K-quants follow from the datapath. The Q5_0/Q5_1 expert ordering and orderings within a few percent depend on the kernels. All dense decode kernels share one matrix–vector structure (Appendix F.2) and differ mainly in the block-decode routine. Q4_0 was implemented first and tuned most extensively; the other kernels were derived from it, adopted its optimizations, and were then further optimized. Exclusions decided by decode cost have margins (1.37–2.24× Q4_0) far larger than any implementation difference, and the 27-format re-solve in our answer to W1 shows that under a binding decode budget, the allocations do not depend on the pruning step. Q4_0 carries no parameters in 11 of the 12 reported allocations and serves as the baseline, so its additional tuning actually makes our comparisons against it conservative.
Q1. Yes. The table in our answer to W1 reports measured accuracy and end-to-end throughput for the best uniform format within both budgets on all three models. On Qwen3.5-4B at the Q4_0 budget, uniform Q4_1_FP8_G32 decodes at 15.30 tok/s against 13.79 for the allocation, and is less accurate: WikiText-2 10.13 vs 9.83, C4 14.45 vs 14.11, reasoning 77.9 vs 81.2. It uses 2.367 GB of the 2.688 GB budget, because the next uniform format does not fit.
Q2. Our answer to W2 addresses this. On an accelerator with native low-bit support, single-stream decode cost follows stored bits, so at a comparable storage budget our program reduces to its storage-only form. At equal storage, that mixed assignment improves on the best single format on all three models: 3.4 points of reasoning on Qwen3.5-4B, lower perplexity on Llama-3.1-8B (WikiText-2 7.33 vs 7.50), and lower C4 perplexity on Qwen3.5-35B-A3B (10.74 vs 10.94). The candidate formats on such hardware would differ and must be measured, which is the first step of the procedure.
Response to Reviewer uRRz (Part 4)
Q3. Re-solved with corrected decode costs, the Llama-3.1-8B allocation reaches 11.54 tok/s within Q4_0's storage (Table 3 in the revised manuscript). It keeps the perplexity gain over Q4_0, but its mean task accuracy equals Q4_0's rather than exceeding it.
The solver predicts decode time from kernels timed in isolation, which underprices some formats inside the full model. We fitted per-format corrections to these costs from a few end-to-end timings and re-solved at Q4_0's storage and decode budgets (Appendix B7 in the revised manuscript). For the revision, all quants were re-timed with the same operator fusions applied to every model graph, which were missing in the earlier submission. Q4_0 measures 11.90 tok/s on this basis (11.6 in Table 19), and the published allocation is 10.91 (10.5 in Table 19):
| Llama-3.1-8B | Storage (GB) | tok/s | WikiText-2 | C4 | Mean accuracy (% of BF16) |
|---|---|---|---|---|---|
| Q4_0 + Q8_0 | 5.041 | 11.90 | 7.56 | 12.09 | 97.7 |
| Allocation, published | 4.990 | 10.91 | 7.42 | 11.77 | 98.8 |
| Allocation, re-solved with corrections | 4.991 | 11.54 | 7.46 | 11.84 | 97.6 |
The re-solved allocation keeps the perplexity gain (WikiText-2 7.46 vs 7.56, C4 11.84 vs 12.09), and its mean task accuracy equals Q4_0's (97.6% vs 97.7%). The margin is narrow because Q4_0 is among the fastest formats on this datapath. On the dense path, every format that decodes as fast or faster stores fewer bits per weight, and all but one are markedly less accurate. The exception, Q4_0_FP8, matches Q4_0's error at 0.25 fewer bits, but on Llama-3.1-8B its kernel-level advantage does not survive in the full model (11.60 vs 11.90 tok/s with a Q8_0 embedding and head, at 4.3% less storage). Every format more accurate than Q4_0 is slower, by 5–76%.
On Qwen3.5-4B, Appendix C.6.5 compares the allocation with a stronger reference that already decodes at Q4_0's speed, Q4_1_FP8_G32 with a Q8_0 embedding and head; at equal storage and predicted speed, the allocation is more accurate.
Response to Review uRRz (Part 1)
We thank the reviewer for the careful review and for recognizing the value of the hardware measurements and of the end-to-end evaluation on dense and MoE models. To address your concerns, we made the following additions, detailed below: