acceptodds
Under review as a conference paper at ICLR 2027

Hardware-Aware Language Model Quantization For A Many-Core AI Inference Accelerator

Abstract

Edge accelerators for ML/AI workloads are designed around trade-offs between peak performance, memory capacity and bandwidth, power, and cost, with vendor economics driven by volume rather than per-unit margin. Because deployed silicon cannot be refreshed on the cadence of model releases, gains must increasingly come from the algorithmic side: pushing out the Pareto frontier of accuracy, memory footprint, and throughput within the constraints of hardware already in the field. Given the extensive work on post-training quantization, we choose it as the primary compression lever and treat it as a full inference-stack optimization problem, jointly optimizing accuracy, storage, and throughput rather than minimizing quantization error alone. We account for hardware features such as supported data formats, computational units, and the instruction set architecture, alongside structural constraints imposed by the runtime and compiler. Beyond the formats natively supported by the hardware, we treat the format design space itself — element and scale datatypes, group size, and hierarchical scales — as part of the optimization process, covering standardized block formats such as MXFP and NVFP as well as non-standard constructions. Algorithmically, we evaluate mixed-precision allocation and codebook-based quantization across dense and mixture-of-experts models from 4B to 35B parameters. We share with the wider community insights on what works well on paper and what actually improves the Pareto frontier across accuracy, memory footprint, and throughput for models deployed on edge silicon. We share with the wider community insights and trade-offs on what works well on paper and what actually improves the Pareto frontier across accuracy, memory footprint, and throughput for a model deployed on edge silicon. We demonstrate our hardware-aware quantization methodology on a many-core edge accelerator with over a thousand RISC-V cores, achieving 34%, 67%, and 24% decode speedups over the llama.cpp K-quant of equal storage at 4.3, 3.7 and 3.3 weight-memory reduction over BF16, within one point of its mean accuracy, on Qwen3.5-35B-A3B, Qwen3.5-4B, and Llama-3.1-8B respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
post-training quantization, mixed-precision quantization, hardware-aware quantization, edge accelerators, mixture-of-experts, llama.cpp

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uRRzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hardware-aware weight-only quantization for LLM inference on ET-SoC-1, aiming to improve the Pareto frontier of accuracy, weight storage, and decoding throughput. The authors profile how quantization formats interact with the hardware, select viable formats, and assign them to tensors using an output-sensitive Fisher objective under storage and decode-time budgets. Activations remain unquantized, with FP32 operands on the decode path. Across two dense models and one MoE model, the method achieves about 1.07-1.72x the decode throughput of K-quant recipes with similar storage, generally with modest accuracy changes.

Critical strengths
  1. The detailed hardware measurements are the strongest part of the paper. The breakdown of weight fetching and dequantization, together with the controlled comparisons of format components, helps explain why some apparently attractive formats perform poorly on this accelerator. This is useful information for deploying quantized models on hardware without native support for the chosen low-bit formats.
  2. The evaluation includes both dense and MoE models, and the throughput gains over K-quant recipes are measured in complete model runs at similar storage. This makes the practical benefit clear. I see value in this engineering work, particularly for existing hardware whose numerical capabilities cannot be changed.
Critical weaknesses
  1. Hardware-aware precision allocation has been studied in HAWQ-V3 and MxMoE. This paper differs in its format choices and output-based Fisher objective, but I would like a clearer picture of the benefit of the allocation itself. Figure 4b and the Q4_0 results provide some evidence. However, Table 16 compares the other uniform alternatives only through predicted objectives and timings. Appendix C.6.2 also notes that at the K-quant budgets the storage-only program already gives the same allocation as the full program, so the speedup over K-quants comes from pruning the format set in Section 3.4, not from the decode-time constraint. Measured accuracy and throughput for one feasible alternative are needed to show how much is gained by mixed-format allocation.
  2. The performance evaluation is limited to ET-SoC-1, whose decode path converts weights to FP32. Section 3.2 explains why format rankings change with hardware support, and the authors describe how to repeat the procedure after re-profiling. However, there is no quantitative analysis or experiment showing whether mixed-format allocation would still improve on a suitable single-format baseline on another target. Even on ET-SoC-1, the kernels come from an in-house generation stack that the paper does not describe, and Q4_0 got extra tuning, so part of the format ranking may reflect kernel quality rather than the formats themselves. The engineering work is useful, but I am less convinced by the added value of the allocation method and the evidence beyond this hardware. I therefore lean toward weak rejection.
Questions for authors
  1. Could you report measured accuracy and end-to-end throughput for one of the uniform alternatives in Table 16 that fits both the storage and decode budgets? One representative model is enough.
  2. How well does the method generalize beyond ET-SoC-1? On an accelerator with native low-bit support, would mixed-format allocation still improve on a suitable single-format baseline at a comparable storage budget?
  3. For Llama 3.1 8B, the recipe optimized at Q4_0's budgets reaches 10.5 tok/s, versus Q4_0's 11.6 (Table 19). Can the mean task accuracy gain be retained after adjusting the allocation to reach 11.6 tok/s within Q4_0's storage budget?
4 comments
AuthorsOpenReview ↗

Response to Review uRRz (Part 1)

We thank the reviewer for the careful review and for recognizing the value of the hardware measurements and of the end-to-end evaluation on dense and MoE models. To address your concerns, we made the following additions, detailed below:

  • Measured uniform alternative (W1, Q1). The best uniform format within both budgets, previously compared only by predicted objective (Table 20, previously Table 16), is now quantized, evaluated, and timed on all three models (Table 21). It decodes faster but is less accurate than the allocation on the dense models, and has higher C4 perplexity on Qwen3.5-35B-A3B.
  • Format pruning (W1). With all 27 kernel-timed formats as candidates, the allocator changes 2.3% of the parameters at the Q4_0 budget with no measurable change in accuracy. At the Q4_K_M budget, it decodes 28% slower with no measurable accuracy gain (Appendix C.6.6).
  • Prior allocators (W1). Our response to Reviewer 1cf9 explains why HAWQ-V3 and MxMoE cannot be applied as they are on this stack, and compares our score with HAWQ-based sensitivity and simpler scores, with formats and budgets fixed.
  • Beyond ET-SoC-1 (W2, Q2). On hardware with native formats, decode cost follows stored bits, and our program reduces to its storage-only form. At equal storage, that mixed assignment is more accurate than the best single format on both dense models, and has lower C4 perplexity on Qwen3.5-35B-A3B.
  • Kernel quality (W2). Section 3.4 now separates the findings that follow from the datapath from those that depend on the kernels, and describes how the kernels were tuned.
  • Llama-3.1-8B at Q4_0's speed (Q3). Re-solved with per-format corrections fitted from a few end-to-end timings, the allocation decodes at 11.54 tok/s (Q4_0: 11.6 in Table 19, 11.90 re-timed). It keeps its perplexity gain over Q4_0, and its mean task accuracy equals Q4_0's.
AuthorsOpenReview ↗

Response to Reviewer uRRz (Part 2)

W1: At the Q4_0 budget of each model, we quantized, evaluated, and timed the best uniform alternative of Table 20 (previously Table 16). It is the single format the allocator selects under the same objective and both budgets, so it isolates the gain from mixing formats:

ModelConfigurationStorage (GB)tok/sWikiText-2C4Reas.
Qwen3.5-4BUniform Q4_1_FP8_G322.36715.3010.1314.4577.9
Allocation2.68813.799.8314.1181.2
Llama-3.1-8BUniform Q4_04.51712.487.8512.5158.7
Allocation4.99010.917.4211.7759.4
Qwen3.5-35B-A3BUniform IQ4_NL19.5314.506.6310.9489.3
Allocation*20.03513.506.6410.7988.6

*Re-solved at a corrected decode budget; the submitted allocation was solved with the budget set incorrectly (73.9 instead of 69.2 ms). The submitted allocation gives WikiText-2 6.62, C4 10.74, and reasoning 88.9.

The uniform format is less accurate than the allocation on both dense models. On Qwen3.5-35B-A3B, where the task benchmarks are saturated, the difference appears only in C4 perplexity. The uniform formats decode faster because uniform formats come in discrete widths, and the best one that fits leaves part of the budget unused (0.32 GB on Qwen3.5-4B). On Qwen3.5-35B-A3B, uniform IQ4_NL also stores the output head, the largest dense tensor read per token, in 4 bits, while the allocation is constrained to Q4_0's speed and keeps the head at Q5_0. A uniform choice that does decode at Q4_0's speed on Qwen3.5-4B is Q4_1_FP8_G32 with a Q8_0 embedding and head (14.10 tok/s). An allocation at the same storage and predicted speed scores 2.7 points higher in reasoning, while measuring 2.8% slower (13.70 tok/s).

At the K-quant budgets, the speedup comes from the pruned format set, not from the decode constraint. To test whether pruning costs accuracy, we re-solved Qwen3.5-4B with all 27 kernel-timed formats as candidates, including the K-quants, IQ4_XS, MXFP4 and NVFP4. At the Q4_0 budget, where the decode budget is active, 2.3% of the parameters change format and measured accuracy does not change. At the Q4_K_M budget, where it is not active, the allocator moves 80% of the parameters to IQ4_XS and K-quants and decodes 28% slower (9.66 vs 13.49 tok/s) with no measurable accuracy gain (WikiText-2 9.74 vs 9.76, reasoning 81.1 vs 81.0). Pruning therefore costs no measurable accuracy (Appendix C.6.6).

HAWQ-V3 and MxMoE cannot be applied as they are on this stack. HAWQ-V3 estimates each layer's Hessian trace with Hessian–vector products, which the fused gated-delta-rule kernels of Qwen3.5 do not support, and it chooses among bit-widths of one integer quantizer. MxMoE assigns a format to each expert, which llama.cpp cannot execute because all experts of a layer share one format, and its cost model describes GPU tensor cores. For our KL objective, HAWQ's sensitivity is the Fisher trace times the quantization error, which we can compute. With candidates, budget, and solver fixed (Qwen3.5-4B, Q3_K_M budget), we replaced our score with it and with simpler scores, and quantized and evaluated each allocation:

Per-tensor scoreWikiText-2C4Reas.
Ours10.2814.5977.7
Fisher trace (HAWQ-V2/V3 sensitivity)10.3714.6876.7
Weight MSE, relative to ‖W‖²10.8515.4079.1
Output MSE (GPTQ layer loss)11.3815.5565.5

The Fisher trace ranks tensors almost as our score does and trails our results. Relative weight MSE has higher perplexity, and output MSE is 12 points lower in reasoning.

AuthorsOpenReview ↗

Response to Reviewer uRRz (Part 3)

W2: The reviewer rightly points out that the format rankings, the viable set, and the decode speeds are properties of ET-SoC-1 and its kernels, and the paper motivates its method from them and we do not claim that they carry over to other hardware (Section 1 and 3.4). What carries over is the procedure: measure the target's formats, keep those it executes efficiently, and allocate under its storage and decode budgets. Applying it to a second accelerator requires a kernel set for that accelerator, which is beyond the scope of this paper. Please see below for further details.

Part of the question can be answered without a second device. On hardware that consumes its formats natively (the first regime of Section 3.2), single-stream decode is bandwidth-bound, and its cost follows stored bits. A decode budget then coincides with a storage budget, and our program reduces to its storage-only form. The question becomes whether, at a fixed storage, mixing formats from a candidate set improves on the best single format from that set. We measured this on all three models:

ModelBest uniform formatWikiText-2C4Reas.
Qwen3.5-4BQ4_1_FP8_G3210.13 / 9.8114.45 / 13.9977.9 / 81.3
Llama-3.1-8BIQ4_NL7.50 / 7.3311.93 / 11.6458.3 / 59.0
Qwen3.5-35B-A3BIQ4_NL6.63 / 6.6610.94 / 10.7489.3 / 88.7

Values are uniform / storage-only mixed assignment at equal storage.

The mixed assignment is more accurate on both dense models: 3.4 points of reasoning on Qwen3.5-4B and lower perplexity on Llama-3.1-8B. On Qwen3.5-35B-A3B, where the task benchmarks are saturated, it has lower C4 perplexity, while WikiText-2 and reasoning are within resolution. These assignments use no timing information, so this is the comparison a bandwidth-bound target would face for its own candidate set. The gain follows from where the model's sensitivity lies: at uniform Q4_0, the 10% most sensitive parameters carry 67%, 54% and 84% of the predicted loss on the three models (Appendix C.6.4), so a single format spends most of its bits where they matter little. Prior work reports gains from mixed precision on GPUs with native formats (HAWQ-V3, MxMoE), and llama.cpp's K-quant recipes are themselves mixed. Where the decode budget binds, as on ET-SoC-1, we fitted per-format corrections to the latency model from a few end-to-end timings (Appendix B7 in the revised manuscript) and re-solved; the allocations then decode within 3.0–3.6% of Q4_0 on all three models and keep lower perplexity than it.

On kernel quality, Section 3.4 now separates the findings that follow from the datapath from those that depend on the kernels: Fetch cost linear in stored width, per-element conversion for FP4/FP8 elements and per-element lookup for codebook formats (the vector units accept only FP32/INT32), and two-level scale decoding for K-quants follow from the datapath. The Q5_0/Q5_1 expert ordering and orderings within a few percent depend on the kernels. All dense decode kernels share one matrix–vector structure (Appendix F.2) and differ mainly in the block-decode routine. Q4_0 was implemented first and tuned most extensively; the other kernels were derived from it, adopted its optimizations, and were then further optimized. Exclusions decided by decode cost have margins (1.37–2.24× Q4_0) far larger than any implementation difference, and the 27-format re-solve in our answer to W1 shows that under a binding decode budget, the allocations do not depend on the pruning step. Q4_0 carries no parameters in 11 of the 12 reported allocations and serves as the baseline, so its additional tuning actually makes our comparisons against it conservative.

Q1. Yes. The table in our answer to W1 reports measured accuracy and end-to-end throughput for the best uniform format within both budgets on all three models. On Qwen3.5-4B at the Q4_0 budget, uniform Q4_1_FP8_G32 decodes at 15.30 tok/s against 13.79 for the allocation, and is less accurate: WikiText-2 10.13 vs 9.83, C4 14.45 vs 14.11, reasoning 77.9 vs 81.2. It uses 2.367 GB of the 2.688 GB budget, because the next uniform format does not fit.

Q2. Our answer to W2 addresses this. On an accelerator with native low-bit support, single-stream decode cost follows stored bits, so at a comparable storage budget our program reduces to its storage-only form. At equal storage, that mixed assignment improves on the best single format on all three models: 3.4 points of reasoning on Qwen3.5-4B, lower perplexity on Llama-3.1-8B (WikiText-2 7.33 vs 7.50), and lower C4 perplexity on Qwen3.5-35B-A3B (10.74 vs 10.94). The candidate formats on such hardware would differ and must be measured, which is the first step of the procedure.

AuthorsOpenReview ↗

Response to Reviewer uRRz (Part 4)

Q3. Re-solved with corrected decode costs, the Llama-3.1-8B allocation reaches 11.54 tok/s within Q4_0's storage (Table 3 in the revised manuscript). It keeps the perplexity gain over Q4_0, but its mean task accuracy equals Q4_0's rather than exceeding it.

The solver predicts decode time from kernels timed in isolation, which underprices some formats inside the full model. We fitted per-format corrections to these costs from a few end-to-end timings and re-solved at Q4_0's storage and decode budgets (Appendix B7 in the revised manuscript). For the revision, all quants were re-timed with the same operator fusions applied to every model graph, which were missing in the earlier submission. Q4_0 measures 11.90 tok/s on this basis (11.6 in Table 19), and the published allocation is 10.91 (10.5 in Table 19):

Llama-3.1-8BStorage (GB)tok/sWikiText-2C4Mean accuracy (% of BF16)
Q4_0 + Q8_05.04111.907.5612.0997.7
Allocation, published4.99010.917.4211.7798.8
Allocation, re-solved with corrections4.99111.547.4611.8497.6

The re-solved allocation keeps the perplexity gain (WikiText-2 7.46 vs 7.56, C4 11.84 vs 12.09), and its mean task accuracy equals Q4_0's (97.6% vs 97.7%). The margin is narrow because Q4_0 is among the fastest formats on this datapath. On the dense path, every format that decodes as fast or faster stores fewer bits per weight, and all but one are markedly less accurate. The exception, Q4_0_FP8, matches Q4_0's error at 0.25 fewer bits, but on Llama-3.1-8B its kernel-level advantage does not survive in the full model (11.60 vs 11.90 tok/s with a Q8_0 embedding and head, at 4.3% less storage). Every format more accurate than Q4_0 is slower, by 5–76%.

On Qwen3.5-4B, Appendix C.6.5 compares the allocation with a stronger reference that already decodes at Q4_0's speed, Q4_1_FP8_G32 with a Q8_0 embedding and head; at equal storage and predicted speed, the allocation is more accurate.

Reviewer_1cf9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes hardware-aware language-model quantization for the ET-SoC-1 accelerator. It profiles quantization formats, retains 14 candidates, and assigns formats to tensors using a Fisher-based sensitivity objective under separate memory and latency budgets. Across three dense and mixture-of-experts models, selected configurations achieve 24–67% faster decoding than K-quant baselines at approximately matched storage and similar accuracy.

Critical strengths
  1. The analysis shows why similar bit widths can yield very different execution times, connecting performance to unpacking, scale reconstruction, and lookup costs.

  2. Results include actual end-to-end accelerator measurements, multiple model architectures, and diverse accuracy benchmarks.

  3. Separating storage from execution cost is particularly appropriate for sparsely activated MoE models.

Critical weaknesses
  1. The allocator’s added value is insufficiently isolated. Appendix shows that latency constraints are inactive for the headline K-quant comparisons. The ablations report predicted objectives rather than measured accuracy, leaving unclear how much benefit comes from the sensitivity method versus simply choosing faster formats.

  2. Candidates are removed using aggregate timing and weight-reconstruction error, whereas allocation uses tensor-specific output sensitivity. Aggregate dominance does not establish dominance under that objective.

Questions for authors
  1. With the candidate formats and budgets fixed, how does the proposed Fisher-based allocator compare with simpler sensitivity scores and existing mixed-precision allocators on measured accuracy and throughput?

  2. Can end-to-end timing feedback correct the Q4_0 budget violations, and how stable are the resulting allocations across different prompt and generation lengths?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1cf9 (Part 1)

We thank the reviewer for the careful review and for recognizing the value of the cost decomposition and of separate storage and decode budgets. To address your concerns, we made the following additions, detailed below:

  • Measured alternatives. The best uniform format and the storage-only assignment, previously compared only by predicted objective (Table 20, previously Table 16), are now quantized, evaluated, and timed on all three models (Table 21). At equal storage, the storage-only assignment is as accurate as ours but decodes 4.5–16.8% slower than Q4_0. The uniform format is less accurate on the dense models and has higher C4 perplexity on Qwen3.5-35B-A3B.
  • Sensitivity scores. With candidates, budget, and solver fixed, our score gives lower perplexity than relative-weight MSE, output MSE, and the Fisher trace used by HAWQ-V2/V3.
  • Prediction accuracy. Per-format offsets fitted from a few end-to-end timings bring the kernel-level decode prediction within 2% on the dense models and 3.7% on Qwen3.5-35B-A3B (Appendix B.7).
  • Re-solved allocations. Re-solved with these offsets, and for Qwen3.5-35B-A3B at a corrected decode budget, the allocations decode within 3.0–3.6% of Q4_0 and keep lower perplexity than it.
  • Candidate pruning. With pruning removed, the allocator choosing among all 27 timed formats changes 2.3% of the parameters at the Q4_0 budget, with no measurable change in accuracy (Appendix C.6.6).

W1. We agree that the headline K-quant comparisons do not isolate the allocator. At those budgets the decode constraint is inactive: the K-quants cost 1.47–2.24× Q4_0 to decode on this hardware (Table 1), because their two-level scales are decoded for every block, so any assignment from the viable set is already faster at equal storage. The speedup there comes from the viable set, and the allocator determines accuracy. We ran three experiments to measure the allocator's contribution apart from the choice of formats.

1. Uniform and storage-only assignments. We quantized, evaluated and timed the alternatives that Table 20 (previously Table 16) compared only by predicted objective (Table 21):

ModelConfigurationStorage (GB)WikiText-2C4Non-reas.Reas.tok/s
Qwen3.5-4BBest uniform format (Q4_1_FP8_G32)2.36710.1314.4563.577.915.30
Storage only2.6889.8113.9963.481.313.52
Allocation2.6889.8314.1163.281.213.79
Llama-3.1-8BBest uniform format (Q4_0)4.5177.8512.5166.158.712.48
Storage only5.0417.3311.6467.459.09.90
Allocation4.9907.4211.7766.759.410.91
Qwen3.5-35B-A3BBest uniform format (IQ4_NL)19.536.6310.9471.989.314.50
Storage only20.0356.6610.7471.788.712.00
Allocation*20.0356.6410.7971.988.613.50

*Re-solved at the corrected decode budget (69.2 ms instead of 73.9 ms). The published allocation gives WikiText-2 6.62, C4 10.74, and reasoning 88.9.

The comparisons separate the gain from mixing formats from the effect of the decode budget. The best uniform format is the single format the allocator selects under the same objective and budgets, so it isolates the gain from mixing formats. It is less accurate than the allocation on Qwen3.5-4B and Llama-3.1-8B; on Qwen3.5-4B it also leaves 0.32 GB of the budget unused, because the next uniform format does not fit. On Qwen3.5-35B-A3B, where the task benchmarks are saturated, the difference appears only in C4 perplexity (10.94 vs 10.79). Uniform IQ4_NL is faster there because of its 4-bit output head, the largest dense tensor read per token; the allocation is constrained to Q4_0's speed and keeps the head at Q5_0. The storage-only assignments have the same reasoning accuracy as the allocations on all three models (within 0.4 points), but they are slower: relative to Q4_0, they decode 4.5–16.8% slower and the allocations 2.6–8.3% slower.

2. Sensitivity score. We replaced D, the predicted KL divergence from the BF16 model, with simpler per-tensor scores, with candidates, budget, and solver unchanged. The results are in our answer to Q1.

3. All 27 timed formats as candidates. We re-solved Qwen3.5-4B with every kernel-timed format as a candidate, including the K-quants and IQ4_XS (Appendix C.6.6). At the Q4_0 budget, where the decode budget is active, 2.3% of the parameters change format, accuracy does not improve (WikiText-2 9.79 vs 9.83, reasoning 79.7 vs 81.2), and decode is 13.60 vs 13.79 tok/s. At the Q4_K_M budget, where it is not active, 80% of the parameters move to IQ4_XS and K-quants and D falls by 24%, but decode is 28% slower (9.66 vs 13.49 tok/s) with no measurable accuracy gain (WikiText-2 9.74 vs 9.76, reasoning 81.1 vs 81.0). Restricting the candidates to the viable set therefore costs no measurable accuracy.

AuthorsOpenReview ↗

Response to Reviewer 1cf9 (Part 2)

W2. We agree that a format that is worse on average can still be the best choice for a particular tensor. We tested this with the 27-format re-solve above. The allocator picks pruned formats for only a few tensors, with no gain in accuracy, when the decode budget is active. When it is not active, it trades a large amount of speed for reductions in D that do not improve measured accuracy. Pruning on aggregate measures therefore costs no measurable accuracy, and it keeps the allocator from giving up speed where the decode budget does not bind.

Q1. HAWQ and MxMoE cannot be run as they are on this model and hardware, so we kept our candidates, budget, and solver fixed and replaced only the per-tensor score, including the HAWQ sensitivity. Our score gives the lowest perplexity; the HAWQ score comes close, and the simpler scores lose either perplexity or reasoning accuracy.

HAWQ-V2/V3 scores each layer by its Hessian trace, estimated with Hutchinson's method, times its quantization error. The Hutchinson estimate needs Hessian–vector products, which the fused gated-delta-rule kernels of Qwen3.5 do not support, and HAWQ chooses among bit-widths of one integer quantizer, whereas our candidates differ in scale datatype, group size, zero point, and codebook at similar widths. For the KL objective, the Hessian at the full-precision weights equals the Fisher information (Proposition 1), so HAWQ's sensitivity for this loss is Tr(F_t)/n_t · ‖Q(W_t) − W_t‖², which our probe computes from per-position gradients. HAWQ-V3's integer program with a latency constraint belongs to the same class as ours. MxMoE assigns a precision to each expert and generates mixed-precision group-GEMM kernels for GPUs. llama.cpp requires all experts of a layer to share one format (Section 1), and MxMoE's cost model describes GPU tensor cores, while our target has no sub-INT8 datapath. It also optimizes one weighted combination of accuracy and throughput, which cannot match a baseline's storage and decode budgets separately. We keep its central idea, pricing throughput alongside memory, as a decode-time constraint priced from profiled kernels, in which an expert block counts fully toward storage but only k/E toward per-token weight traffic. The K-quant baselines of Table 2 are themselves llama.cpp's rule-based mixed-precision recipes, compared on measured accuracy and throughput.

Each allocation below was quantized, evaluated, and timed (Qwen3.5-4B, Q3_K_M budget):

Per-tensor scoreRank correlation with D across tensorsWikiText-2C4Non-reas.Reas.tok/s
Q3_K_M (llama.cpp)—9.9014.3762.077.99.00
D (ours)110.2814.5963.677.714.97
Fisher trace (HAWQ-V2/V3)0.9710.3714.6863.676.714.70
Weight MSE, relative to ‖W‖²−0.4910.8515.4062.279.114.80
Output MSE (GPTQ layer loss)−0.0311.3815.5560.465.514.24

All allocations use the full 2.283 GB storage budget, and the decode budget does not bind at this budget, so the speed differences reflect the formats each score selects. Every allocation decodes 1.6–1.7× faster than Q3_K_M. For any single tensor, all scores order the candidate formats almost identically (mean rank correlation 0.98); they differ in how they weigh one tensor against another, which decides where the allocator spends bits. Relative weight MSE is the inverse weight SQNR, whose sum over layers is the output-SQNR model of Lin et al. (2016); it gives larger tensors fewer bits, here dropping the tied embedding and output head to 3 bits, and has higher perplexity than D. Output MSE is 12 points lower in reasoning. The Fisher trace ranks tensors almost as D does and gives similar results.

AuthorsOpenReview ↗

Response to Reviewer 1cf9 (Part 3)

Q2. End-to-end timings can correct most of the Q4_0 shortfall: with per-format corrections fitted from a few timings and a re-solve, the allocations decode within 3.0–3.6% of Q4_0 on all three models and keep their perplexity advantage. The speedups hold across prompt and generation lengths, varying by about 3%.

We fitted per-format offsets from a few end-to-end timings per model, added them to the kernel costs and re-solved (Appendix B.7). The shortfall against Q4_0 falls from 8.3% to 3.0% on Llama-3.1-8B and, together with the corrected decode budget, from 10.1% to 3.6% on Qwen3.5-35B-A3B. On Qwen3.5-4B, where the published allocation was already 2.6% short, the re-solve does not improve it (3.2%). Perplexity stays below the Q4_0 recipe on all three models (WikiText-2 7.46 vs 7.56 on Llama-3.1-8B). The remaining gap of about 3% arises because a solver optimizing against estimated costs favours the formats those costs understate.

The assignments differ only in weight formats; activations and the KV cache are unquantized and identical across assignments. The format-dependent cost of a decoded token is therefore independent of length, while the attention cost, which grows with length, is common to all assignments. We measured decode across prompt and generation lengths for four published allocations against their baselines:

Allocation vs baseline512 / 2562048 / 2562048 / 10244096 / 2048
Qwen3.5-4B, Q4_0 budget, vs Q4_00.970.970.970.97
Qwen3.5-4B, vs Q4_K_M1.441.431.431.41
Llama-3.1-8B, vs Q4_K_S1.201.191.191.17
Qwen3.5-35B-A3B, vs Q3_K_M1.381.351.341.34

Values are decode speedups (×); columns are prompt / generated tokens.

Each allocation's decode speedup varies by at most about 3% across lengths, and its per-token time difference from the baseline stays nearly constant (19.2–19.7 ms on Llama-3.1-8B). The Q4_0-budget shortfall is the same 3% at every length, so a correction calibrated at one length carries over to the others. Prefill runs on the tensor units and varies little with format for the dense versions and are generally stable across prompt length for both types of models.

Reviewer_grCyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hardware-aware weight quantization on the ET-SoC-1 many-core inference accelerator. It characterizes quantization formats along several dimensions, including element encoding, scale datatype, group size, and zero-point representation, and profiles their execution costs on the target device. After selecting 14 viable formats, the paper assigns formats to individual tensors using a Fisher/KL-based sensitivity objective and a binary optimization program with separate storage and predicted decode-time constraints. The evaluation covers two dense models and one MoE model. At matched storage, the resulting configurations achieve 1.10–1.72× higher end-to-end decode throughput than the K-quant baselines.

Critical strengths
  1. The hardware characterization provides useful empirical knowledge beyond nominal bit-width comparisons. The paper connects format-dependent performance to concrete costs such as weight fetching, unpacking, and dequantization. This helps explain why formats with similar storage requirements can have different execution costs on an accelerator without native sub-INT8 arithmetic. These observations are valuable for practitioners selecting quantization formats for deployment.

  2. The main performance results include real end-to-end measurements. Table 2 shows throughput improvements in all six matched K-quant comparisons, with relatively small accuracy differences in most configurations. The paper also reports less favorable results and discrepancies between isolated-kernel estimates and full-model execution. This makes the evaluation informative and supports the practical value of hardware-aware format selection.

Critical weaknesses
  1. The incremental contribution of the dual-constraint allocator is not established by the reported end-to-end gains. Appendix C.6.2 and Table 16 state that storage-only and storage-plus-decode optimization return the same assignments at all K-quant budgets. Consequently, the improvements in Table 2 support the value of the candidate formats and hardware-aware selection, but do not isolate a benefit from adding the decode-time constraint.

At the Q4_0 budgets where that constraint becomes active, the realized configurations have approximately 3%, 9%, and 13% lower throughput than Q4_0 in Table 19. This indicates a discrepancy between the predicted budget and actual execution, rather than demonstrating that the constrained allocator is ineffective. However, without measured quality and throughput for the corresponding storage-only assignments, the practical value of the second constraint remains unresolved. Likewise, Table 16 compares some alternatives only through the predicted objective, explicitly without quantization or evaluation. A limited measured comparison using these existing assignments would substantially strengthen the central methodological claim.

  1. Some format-ranking conclusions are difficult to separate from differences in kernel implementation. Section 3.4 describes an expert-guided kernel development process whose details are outside the manuscript, and notes additional tuning for Q4_0. Appendix B.5 attributes the Q5_0/Q5_1 ordering in one expert path to an implementation difference involving activation reuse. These observations do not invalidate the measured performance, but they limit how confidently the rankings can be attributed to format properties themselves.

Because the measured rankings determine which formats enter the allocator, this distinction affects both the interpretation and reproducibility of the proposed methodology. The paper should identify which findings follow from the target datapath and which depend on the particular kernel implementations, and provide sufficient implementation detail to assess the key comparisons.

Questions for authors
  1. At representative Q4_0 budgets, what are the measured task accuracy and end-to-end throughput of the existing storage-only assignments, compared with the published dual-constraint assignments at matched storage? Please also report their predicted-versus-measured timing discrepancies.

  2. Can the authors provide measured accuracy for the best uniform feasible format in Table 16 at representative budgets? This would help establish whether the predicted sensitivity objective correctly identifies the practically preferable assignments.

  3. Which format-ranking conclusions remain supported after accounting for the disclosed kernel optimization differences? Please clarify the implementation and tuning procedures for the comparisons that determine the viable set.

Additional clarification, not an independent reason for rejection: Section 6 states an average throughput improvement of “more than 50%.” The six matched comparisons in Table 2 give an unweighted arithmetic mean improvement of approximately 41.7%. Please specify the subset or weighting used for the aggregate statement, or revise it accordingly. The individual improvements remain meaningful.

3 comments
AuthorsOpenReview ↗

Response to Reviewer grCy (Part 1)

We thank the reviewer for their thoughtful review and for recognizing the major contributions of the work. The holistic picture constructed with end-to-end measurement and data should help any practitioner understand the subtleties of deployment on hardware. To address your concerns, we made the following additions, detailed below:

  • Measured alternatives (W1, Q1, Q2). The storage-only assignments and the best uniform formats, previously compared only by predicted objective (Table 20, previously Table 16), are now quantized, evaluated, and timed on all three models (Table 21). At equal storage, the storage-only assignment is as accurate as ours but decodes 4.5–16.8% slower than Q4_0; the uniform format is less accurate, consistent with its several-times-higher D.
  • Predicted versus measured decode (W1, Q1). Per-format offsets fitted from a few end-to-end timings bring the kernel-level prediction within 2% on the dense models and 3.7% on Qwen3.5-35B-A3B (Appendix B.7). Re-solved with them, and for Qwen3.5-35B-A3B at a corrected decode budget, the allocations decode within 3.0–3.6% of Q4_0 and keep lower perplexity than it.
  • Datapath versus kernel findings (W2, Q3). Section 3.4 now separates the findings that follow from the datapath from those that depend on the kernels, and describes how the kernels were tuned. With pruning removed, the allocator choosing among all 27 timed formats changes 2.3% of the parameters at the Q4_0 budget, with accuracy within resolution (Appendix C.6.6).

W1. We agree that, at the K-quant budgets, the decode speedup comes from the candidate set, not from the decode constraint. K-quants cost 1.47–2.24× Q4_0 to decode on this hardware, so any assignment from the viable set at the same storage is already faster, and the constraint does not bind. At higher decode throughput, the allocations match or exceed the K-quants in accuracy on Qwen3.5-4B and Qwen3.5-35B-A3B and are slightly below them on Llama-3.1-8B (Table 28, previously Table 19). Appendix C.6.2 now discusses this, and Table 21 adds measured accuracy and decode for the alternatives of Table 20 (previously Table 16).

The decode constraint binds at the Q4_0 budget, where predicted and measured decode differ. The solver sums kernel times measured in isolation; inside the model, each format runs at a consistent, format-dependent ratio to its isolated time. For one Llama-3.1-8B matmul (FFN gate projection, 14336×4096):

FormatIsolated (ms)In-model (ms)Difference
Q4_00.5000.476−5%
IQ4_NL0.6140.582−5%
Q4_0_FP8_G320.4770.531+11%
Q5_0_FP8_G320.6380.688+8%

When the budget binds, the solver favours formats whose cost is underestimated, so the allocation decodes slower than predicted. The ratios repeat across runs, so we fitted per-format offsets from a few end-to-end timings per model and added them to the kernel costs (Appendix B.7). Without the offsets, the published allocations decode 2.9%, 9.2% and 4.4% slower than predicted; with them, the timed files are predicted within 2% on the dense models and within 3.7% on Qwen3.5-35B-A3B (table in Q1).

The 13% gap of Qwen3.5-35B-A3B to Q4_0 in Table 19 has two causes. Its timing was taken before operator fusions were made consistent across files; re-timed, it decodes at 12.58 tok/s, 10% below Q4_0. Its decode budget was also set incorrectly (73.9 instead of 69.2 ms on the solver's basis); re-solved at the corrected budget, it decodes at 13.50 tok/s, 3.6% below Q4_0's 14.00.

AuthorsOpenReview ↗

Response to Reviewer grCy (Part 2)

W2. We agree that the measured rankings reflect the format, its datapath on ET-SoC-1 and the kernel implementation together. The method does not depend on rankings transferring: it consumes measured costs, which are a property of the target. Section 3.4 now separates the two kinds of findings:

  • Datapath: fetch cost linear in stored width; per-element conversion for FP4/FP8 elements and per-element lookup for codebook formats, since the vector units accept only FP32/INT32; two-level scale decoding for K-quants.
  • Kernel-dependent: the Q5_0/Q5_1 expert ordering, and orderings within a few percent.

To test whether the pruning step or kernel-level differences affect the result, Appendix C.6.6 removes pruning and lets the allocator choose among all 27 timed formats, including MXFP4, NVFP4, IQ4_XS and the K-quants (Qwen3.5-4B):

BudgetCandidatesParameters outside the 14Dtok/sWiki2C4Non-reas.Reas.
Q4_0 (decode binds)14—0.078813.799.8314.1163.281.2
272.3%0.077913.69.7914.1263.279.7
Q4_K_M (decode does not bind)14—0.066813.499.7613.9863.081.0
2780.8%0.05109.669.7413.9263.081.1

At the Q4_0 budget, the allocator moves 2.3% of the parameters, with accuracy within resolution; at the Q4_K_M budget, the added formats bring no measurable accuracy gain at 1.40× the decode time. Q4_0 itself carries no parameters in 11 of the 12 allocations of Table 28 and serves as the baseline, so its additional tuning only makes those comparisons conservative.

Q1. We quantized, evaluated, and timed the storage-only assignments at each model's Q4_0 budget (Table 21):

ModelConfigurationDWiki2C4Non-reas.Reas.tok/sDiscrepancy, kernel-levelDiscrepancy, with offsets
Qwen3.5-4BQ4_0 + Q8_00.9099.8814.2462.876.814.16––
Uniform Q4_1_FP8_G320.58910.1314.4563.577.915.30+0.4%+0.3%
Storage only0.07289.8113.9963.481.313.52+0.4%−0.1%
Ours0.07889.8314.1163.281.213.79+2.9%0.0%
Ours, re-solved0.08039.8414.1263.479.913.70+3.0%+3.4%
Llama-3.1-8BQ4_0 + Q8_00.1187.5612.0967.058.311.90––
Uniform Q4_00.1387.8512.5166.158.712.48+0.3%+0.3%
Storage only0.02627.3311.6467.459.09.90+2.4%−1.9%
Ours0.04507.4211.7766.759.410.91+9.2%−1.0%
Ours, re-solved0.05637.4611.8466.958.211.54+3.2%+3.1%
Qwen3.5-35B-A3BQ4_0 + Q8_00.0936.7410.9771.489.114.00––
Uniform IQ4_NL0.06116.6310.9471.989.314.50−0.4%−3.7%
Storage only0.01506.6610.7471.788.712.00+2.1%+0.3%
Ours, re-solved*0.02576.6410.7971.988.613.50+4.3%+3.7%

*At the corrected decode budget. The previous assignment: D 0.0168, Wiki2 6.62, C4 10.74, Non-reas. 72.1, Reas. 88.9, 12.58 tok/s.

At equal storage, the storage-only and dual-constraint assignments are equally accurate within run-to-run resolution, but storage-only decodes 4.5%, 16.8% and 14.3% slower than Q4_0. The kernel-level prediction places the storage-only assignments within 2.4% of measured decode, and within 1.9% of the offsets. The re-solved allocations remain 3.1–3.7% slower than their corrected prediction. This is a selection effect: an allocation optimized against estimated costs favors the formats those costs understate. Narrowing the gap further requires exact costs for every format and shape inside the running model, which means building and profiling one model per format.

Q2. The table in Q1 includes the best uniform format within both budgets: the single-format assignment with the lowest objective under the same constraints, which isolates the gain from mixing formats. It has several times the allocation's D, and measured accuracy follows: it is lower in reasoning on Qwen3.5-4B and has higher perplexity on Llama-3.1-8B and on the saturated Qwen3.5-35B-A3B.

AuthorsOpenReview ↗

Response to Reviewer grCy (Part 3)

Q3. Section 3.4 now separates datapath findings from kernel-dependent ones (W2). The implementation details of the kernel optimization process are part of a parallel publication under review at another venue and cannot be covered here. All dense decode kernels share the structure of Appendix F.2 (activation staging, shape-dependent work distribution and a vectorized per-row dot product); they differ mainly in the block-decode routine, and in which optional fast paths, such as streaming weights directly from memory, a format uses. Each kernel exposes tuning parameters, such as prefetch distance, work splitting across cores, and staging sizes. Each kernel went through several optimization iterations of its own, starting from the Q4_0 structure. As Section 3.4 notes, some costs may still improve; this affects the conclusions only where formats are close:

  • Exclusions decided by decode cost have margins (1.37–2.24× Q4_0) far larger than any implementation difference.
  • Exclusions decided by reconstruction error do not depend on the kernels.
  • Orderings within about 10% depend on the implementation and have little effect at the binding budget.

An improved kernel enters the allocator directly as a new timing and a re-solve.

Clarification (Section 6). We thank the reviewer for pointing out the discrepancy in the conclusion. The unweighted mean over the six matched comparisons in Table 2 is indeed 42%, and we have revised Section 6 accordingly.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your efforts so far. As we are at the end of the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.